Updated on

MongoDB has no JOIN, but it has $lookup, an aggregation stage that pulls matching documents from a second collection into each document of the first. In the C# driver we reach it through Aggregate() and chained stage methods.

A pipeline is an ordered list of those stages, each one taking the previous stage’s output as its input. Joining two collections is usually two of them: $lookup to bring the related documents in, then $project to drop the raw identifiers we no longer need.

To download the source code for this article, you can visit our GitHub repository.

Working with NoSQL databases is often a complex task, especially when dealing with multiple collections. However, MongoDB has made these challenging tasks easier with the aggregation pipelines, which simplify the process of transforming documents through multiple stages.

How Do We Prepare Our Environment to Join MongoDB Collections?

To move forward with our example, we have to initialize our environment so that our aggregation pipeline can run smoothly and return our expected data. First, we need to create a MongoDB instance to host our data. Then we need to create our entities that will represent a model for our MongoDB collections. Finally, we need to seed our database with our data. We cover the MongoDB integration itself in a previous article, so here we only set up as much as the pipeline needs.

Spinning Off Our MongoDB Container

Let’s go ahead and create a new console application project and configure it with MongoDB. For this example, we will be using the MongoDB Test Container as our database, which is one way of running dependencies in Testcontainers.

To kick things off, let’s create a new MongoDB container by using the MongoDbBuilder class provided by the Testcontainers.MongoDb package, pinning the image tag so the sample does not drift onto whatever mongo tag happens to be current:

await using var mongoDbContainer = new MongoDbBuilder("mongo:8.0").Build();

The await using is what stops and disposes the container once the program ends.

Next, let’s start our MongoDb container by calling the StartAsync() method of the mongoDbContainer variable:

await mongoDbContainer.StartAsync();

Now we have a MongoDB container running in Docker. Next, we need to create a MongoClient instance so we can use it in our code:

var mongoClient = new MongoClient(mongoDbContainer.GetConnectionString());

We call the GetConnectionString() method on the mongoDbContainer variable to get the connection string of the MongoDB service and pass it as an argument to the MongoClient class.

Subsequently, we can now initialize our database by using the GetDatabase() method in the MongoClient class:

var database = mongoClient.GetDatabase(DatabaseConfiguration.DatabaseName);

Creating Our Entities

Our next step is to create our entities. We will need two entities, a Course class:

public class Course
{
    [BsonElement("_id")]
    [BsonRepresentation(BsonType.ObjectId)]
    public string Id { get; set; } = string.Empty;

    [BsonElement("Name")]
    public string Name { get; set; } = string.Empty;

    [BsonElement("Code")]
    public string Code { get; set; } = string.Empty;

    public override bool Equals(object? obj)
    {
        if (obj is not Course course) return false;
        return Name == course.Name
                && Code == course.Code;
    }

    public override int GetHashCode()
    {
        return HashCode.Combine(Name, Code);
    }
}

And a Student class:

public class Student
{
    [BsonElement("_id")]
    [BsonRepresentation(BsonType.ObjectId)]
    public string Id { get; set; } = string.Empty;

    [BsonElement("FirstName")]
    public string FirstName { get; set; } = string.Empty;

    [BsonElement("LastName")]
    public string LastName { get; set; } = string.Empty;

    [BsonElement("Major")]
    public string Major { get; set; } = string.Empty;

    [BsonElement("StudentCourses")]
    public List<Course> StudentCourses { get; set; } = [];

    public override bool Equals(object? obj)
    {
        if (obj is not Student student) return false;
        return FirstName == student.FirstName
            && LastName == student.LastName
            && StudentCourses.SequenceEqual(student.StudentCourses);
    }

    public override int GetHashCode()
    {
        var hash = new HashCode();
        hash.Add(FirstName);
        hash.Add(LastName);

        foreach (var course in StudentCourses)
            hash.Add(course);

        return hash.ToHashCode();
    }
}

We add the StudentCourses property as a list of the Course type in the Student class instead of the ObjectId list. We do that because we need to pull this property from the Course collection in the aggregation pipeline.

The two field names are the whole trick, and the article’s pipeline depends on it. The stored student document carries a Courses array of ObjectId values, and Student does not map Courses at all: it maps StudentCourses. Since the driver rejects elements it has no member for, a Student cannot be deserialized from the raw collection, which is why the pipeline has to fill StudentCourses and drop Courses before anything materializes.

Seeding Our Database

Our final step is to seed our database:

var courseCollection = database.GetCollection<BsonDocument>("Courses");
var studentCollection = database.GetCollection<BsonDocument>("Students");
await courseCollection.InsertManyAsync(new List<BsonDocument>
{
    new()
    {
        { "_id", new ObjectId("655e134180c300fcdd067d24") } ,
        { "Name", "Networks" },
        { "Code", "ECEN 474" }
    },
    new()
    {
        { "_id", new ObjectId("655e134180c300fcdd067d25") } ,
        { "Name", "Power Systems" },
        { "Code", "ECEN 485" }
    }
});

await studentCollection.InsertManyAsync(new List<BsonDocument>
{
    new()
    {
        { "_id", new ObjectId("656623db682962fa62ad75ba") } ,
        { "FirstName", "John" },
        { "LastName", "Doe" },
        { "Major", "Electrical Engineering" },
        { "Courses", new BsonArray {
            new ObjectId("655e134180c300fcdd067d24"),
            new ObjectId("655e134180c300fcdd067d25")
        } }
    }
});

We use the InsertManyAsync() method on the Course and Student collections and pass a list of entities as arguments to populate our collections with data. The identifiers we hand-write here are the same values the driver generates on insert, and there is more to say about querying by ObjectId in the C# driver.

We are now ready to start building our aggregation pipeline.

Is this material useful to you? Consider subscribing and get ASP.NET Core Web API Best Practices eBook for FREE!

How Do We Build the Aggregation Pipeline to Join MongoDB Collections?

An aggregation pipeline is an ordered sequence of stages. Each stage takes the documents the previous one produced and hands its own output to the next.

In the C# driver, collection.Aggregate() starts an empty pipeline, and every stage is a method chained onto it. Nothing runs until we call ToListAsync() or one of its siblings at the end.

The order matters more than anything else. A $match before a $lookup cuts the number of documents the join has to touch; a $match after it does the same work over a set we already paid to build.

For a join, two stages are usually enough. $lookup brings the related documents in, and $project removes the identifier fields we joined on but no longer want in the result.

The pipeline is built as an object and executed once, so composing it across several lines costs nothing.

The reason this advice is narrower than it sounds is that the server already reorders some stages for us. MongoDB’s pipeline-optimization reference says “MongoDB moves any filters in the $match stage that do not require values computed in the projection stage to a new $match stage before the projection”. It does not do that for $lookup, which is why placing $match first still matters here.

Let’s create a StudentRepository class to hold the pipeline:

public class StudentRepository
{
    private readonly IMongoCollection<Student> _studentCollection;

    public StudentRepository(MongoClient client)
    {
        var database = client.GetDatabase(DatabaseConfiguration.DatabaseName);
        _studentCollection = database.GetCollection<Student>("Students");
    }
}

First, we create a StudentRepository class and in the constructor of the class, we initialize our collections from the MongoClient we already built against the container.

The pipeline will consist of two stages: the $lookup and $project stages provided by MongoDB. Each stage will help in transforming our data until it matches our model.

We can start by creating an empty aggregation pipeline on our Students collection by using the Aggregate() method on the _studentCollection field:

var studentAggregationPipeline = _studentCollection.Aggregate();

With this, we can continue joining our course collection with the student collection.

Join MongoDB Collections With $lookup

$lookup performs a left outer join. Every document from the input collection comes through, whether or not anything matched, and, as MongoDB’s aggregation reference puts it, “[t]he new array field contains the matching documents from the foreign collection”.

It takes four things: the collection to join to, the field on the local document, the field on the foreign document, and the name of the array field to create.

The array part is what catches people out. Even when exactly one document matches, $lookup produces a one-element array; it does not flatten. Getting a single object instead means adding $unwind after it, or projecting the first element.

The local field can hold a single value or an array of values, and both work. An array of identifiers matches every foreign document whose key appears in it, which is what makes a many-to-many relationship a single stage.

Nothing is enforced. There are no foreign keys here, so a local value pointing at nothing simply produces an empty array.

In our case, we can use it to create an outer left join between our Students and Courses collections. We can achieve this by using the Lookup() method of the studentAggregationPipeline we created earlier.

This method accepts four arguments, foreign collection name, local field name, foreign primary key, and the desired field name:

var studentsJoinedWithCoursesPipeline = studentAggregationPipeline
    .Lookup<Student, Student>("Courses", "Courses", "_id", "StudentCourses");

In our example, our foreign collection is the Courses collection, our local field name is Courses, our foreign primary key is _id, and the field we need to create is StudentCourses.

Is this material useful to you? Consider subscribing and get ASP.NET Core Web API Best Practices eBook for FREE!

That is the whole join, seen on a single document:

Before the join the Courses field holds two ObjectId values; after the lookup and project stages StudentCourses holds the two full course documents

This is a join in the aggregation framework, not in LINQ. If we are after the LINQ shapes instead, we cover LINQ inner joins in C# and LINQ outer joins in C# separately.

Projection With $project

Our second stage in the pipeline is $project which we can use to include, exclude, or transform certain fields in our documents.

The purpose of this stage in our example is to exclude the Courses field from our Student documents since it will be replaced by the whole course document from the Courses collection:

var projection = Builders<Student>.Projection.Exclude("Courses");
var studentsWithoutCourseIdsPipeline = studentsJoinedWithCoursesPipeline.Project<Student>(projection);

As we can see, we use the Exclude() method on the Builders<Student>.Projection property to exclude the Courses field from the result. Then, we call the Project() method on the studentsJoinedWithCoursesPipeline variable and pass our projection as an argument.

Let’s put that all together:

public class StudentRepository
{
    private readonly IMongoCollection<Student> _studentCollection;

    public StudentRepository(MongoClient client)
    {
        var database = client.GetDatabase(DatabaseConfiguration.DatabaseName);
        _studentCollection = database.GetCollection<Student>("Students");
    }

    public async Task<List<Student>> GetAllStudentsAsync()
    {
        //Empty Pipeline
        var studentAggregationPipeline = _studentCollection.Aggregate();

        //Lookup
        var studentsJoinedWithCoursesPipeline = studentAggregationPipeline
            .Lookup<Student, Student>("Courses", "Courses", "_id", "StudentCourses");

        //Project
        var projection = Builders<Student>.Projection
            .Exclude("Courses");

        var studentsWithoutCourseIdsPipeline = studentsJoinedWithCoursesPipeline.Project<Student>(projection);

        var students = await studentsWithoutCourseIdsPipeline.ToListAsync();

        return students;
    }
}

Finally, we convert our result as a list using the ToListAsync() method on our setup pipeline studentsWithoutCourseIdsPipeline.

Now let’s use that code to get our students with courses:

var repository = new StudentRepository(mongoClient);

var students = await repository.GetAllStudentsAsync();
foreach (var student in students)
{
    Console.WriteLine(student.ToJson());
}

By iterating through our students and printing them as JSON string with the ToJson() method to our console output, we can see that we have successfully joined the Students and Courses collections into a single result that is similar to our model class:

{
    "_id" : ObjectId("656623db682962fa62ad75ba"),
    "FirstName" : "John",
    "LastName" : "Doe",
    "Major" : "Electrical Engineering",
    "StudentCourses" : [
        {
            "_id" : ObjectId("655e134180c300fcdd067d24"),
            "Name" : "Networks",
            "Code" : "ECEN 474"
        },
        {
            "_id" : ObjectId("655e134180c300fcdd067d25"),
            "Name" : "Power Systems",
            "Code" : "ECEN 485"
        }
    ]
}

In this result, we can see that the courses with object identifiers ObjectId("655e134180c300fcdd067d24") and ObjectId("655e134180c300fcdd067d25") from the Courses collection have been joined with the Student document from the Students collection.

What Aggregation Stages Can We Use in the C# Driver?

The C# driver exposes MongoDB’s aggregation stages as fluent methods on the object returned by Aggregate(), so a pipeline reads as a chain rather than as a document.

Match(), Project(), Group(), SortBy(), Skip(), Limit() and Lookup() cover most work. Each one appends a stage and returns the pipeline, so they compose in any order the server accepts.

Two of them matter beyond this article. Unwind() turns an array field into one document per element, which is how a $lookup result becomes a flat row. Group() is where aggregation stops being a join tool and starts being a reporting one.

For anything the fluent methods do not cover, AppendStage() takes a raw stage document, so an operator the driver has no method for is still reachable.

The full stage list is much longer than this, and the ones here are the ones that appear in real pipelines.

StageC# fluent methodWhat it does
$matchMatch()Filters documents; the pipeline's WHERE
$projectProject()Chooses, renames, or computes fields
$lookupLookup()Pulls in matching documents from another collection
$unwindUnwind()Turns an array field into one document per element
$groupGroup()Groups documents and computes aggregates
$sortSortBy() / SortByDescending()Orders the results
$skipSkip()Skips a number of documents
$limitLimit()Caps the number of documents
$countCount()Replaces the results with a count
$outOut()Writes the results into a collection

Once the documents are back in memory, the array a $lookup leaves behind is an ordinary nested collection, and the same problem turns up in plain C# when we look at flattening nested collections in C#.

Conclusion

Whether for small applications or enterprise applications, MongoDB is proving to be efficient and performant on all levels. With the great power that can be leveraged using the aggregation pipelines provided by MongoDB, there seems to be no limit to what can be achieved using NoSQL databases.

Tested with .NET 10.0.10 and MongoDB.Driver 3.10.0.