Best way to read from MongoDB change streams from within Azure

Question

Hello! We have an application that needs events from our MongoDB change stream and insert the change event into Data Lake storage. We cannot use MongoDB Triggers to call an Azure Function because we have no way of authenticating the Trigger (we have been denied using any sort of passwords, etc in MongoDB). Thus, we need a way, from inside azure, to listen to the MongoDB change streams and put the events into json files in our data lake.

Ideally, we'd like to use event driven architecture for this.

I know Data Factory has a MongoDB connector, but this is only for copying data from the DB, not for getting the change events.

Any suggestions on this? Thanks!

Answer 1

Hello Jaime Becker,

Thanks for posting your question in the Microsoft Q&A forum.

The best approach to read from MongoDB change streams within Azure and store the events in Data Lake storage would be to use Azure Functions with a custom implementation.

Create an Azure Function App, implement the MongoDB change stream listener in C# or Python.

Use the official MongoDB driver for .NET or Python, connect to your MongoDB instance securely using connection strings stored in Azure Key Vault.

As the function receives change events, process them and convert them to JSON format, use the Azure Data Lake Storage Gen2 SDK to write the JSON files to your Data Lake storage

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful