Running MongoDB Aggregation Operation in Azure Data Factory.

Question

Hi,

I want to query my MongoDB Atlas (Not Cosmos DB) to fetch data and process it in Azure Data Factory. I have successfully created a connection to MongoDB and can run filter operations.

I want to perform aggregation operations like $lookup with the MongoDB connector (MongoDbV2). There is some reference to running such operation over there (https://learn.microsoft.com/en-us/azure/data-factory/connector-mongodb?tabs=data-factory#upgrade-the-mongodb-linked-service) but using those queries in MongoDB source (in filter field) didn't work.

I can't find other related documentation online. How will I be able to run aggregation pipeline in ADF?

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A.

To perform aggregation operations, such as $lookup, on your MongoDB Atlas data within ADF, you can utilize the MongoDB Atlas connector. This connector supports advanced aggregation pipelines, enabling complex data transformations directly within your ADF pipelines.

Here's how you can set this up:

Create a Linked Service to MongoDB Atlas:

• In ADF, navigate to the Manage tab and select Linked Services.
  • Click on New and search for MongoDB Atlas.
    • Configure the connection by providing your MongoDB Atlas connection string and specifying the target database.

1. Configure the MongoDB Atlas Source with Aggregation:
   • In your pipeline, add a Copy Data activity.
   • Set the source to the MongoDB Atlas linked service you created.
   • Within the source settings, locate the Filter field.
   • Input your aggregation pipeline as a JSON array.

          [
            { "$match": { "field": "value" } },
            { "$lookup": {
                "from": "otherCollection",
                "localField": "localField",
                "foreignField": "foreignField",
                "as": "joinedResult"
              }
            }
          ]
          
          
     

2. Ensure that your aggregation stages are correctly formatted and aligned with MongoDB's aggregation framework.

Important things to note:

• Connector Version: Sstandard MongoDB connector (MongoDbV2) in ADF supports MongoDB versions up to 4.2. For versions beyond this, the MongoDB Atlas connector is recommended.
• Aggregation Support: While the MongoDB Atlas connector facilitates aggregation operations, ensure that your pipeline JSON is correctly structured. If you encounter issues, verify the syntax and structure of your aggregation stages.

By configuring the MongoDB Atlas connector with your desired aggregation pipeline, you can effectively perform complex data transformations within Azure Data Factory, leveraging the full capabilities of MongoDB's aggregation framework.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.