Azure Stream Analytics Job to save data to CosmosDB for Mongo

Question

Hi

I am creating an IoT solution

I have connected the Azure Stream Analytics job with CosmosDB NoSQL and it si working fine, but I would like to connect to the CosmosDB for Mongo since we are used to work with Mongo DB. More over The CosmosDB is saving alot of more add info in the message

{
    "id": "22d4sff5-2818f47a437",
    "_rid": "dw5tAMoGAdsfdfsd",
    "_self": "dbs/dw5tAA==/colls/dw5fsl8=/docs/dw5tAMoG6l8DAQAAAAAAAA==/",
    "_etag": "\"02-0000-0d00-0000-67sdfsdfs9520000\"",
    "_attachments": "attachments/",
    "_ts": 173218f402
}

that is increasing the storage space. While the Mongo DB is not based on the Properties of an item

I have tried to do it, but it seems that the Azure Stream Analytics Output cannot connect to CosmosDB for Mongo (i.e. Mongo API not supported). When I try to do it, I get

My questions are?

1. Is it correct that Azure Stream Analytics cannot have a CosmosDB for Mongo as output?
2. if yes for the 1) how can I achieve to save the Streaming data to CosmosDB for Mongo? Here I was thinking to use Function app with Mongo DB API but I am not sure.

Thanks for you help

BR

Alberto

Answer 1

Hello Alberto Morando,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like your Azure Stream Analytics Job to save data to CosmosDB for Mongo.

You're correct that Azure Stream Analytics currently does not support output to Azure Cosmos DB for MongoDB API - https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-documentdb-output It only supports output to Azure Cosmos DB using the SQL API - https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/stream-analytics/azure-cosmos-db-output.md

To achieve your goal, you can use an intermediary approach involving Azure Event Hub and Azure Functions. This method allows you to route data from Azure Stream Analytics to CosmosDB for MongoDB API.

1. Configure your Azure Stream Analytics job to output data to an Azure Event Hub. This step involves setting up an Event Hub namespace and creating an Event Hub within it. You can then configure your Stream Analytics job to send data to this Event Hub.
2. Create an Azure Function that triggers on new events in the Event Hub. This function will process the incoming data and write it to CosmosDB for MongoDB using the MongoDB API. Below is an example of what the Azure Function might look like in C#:

using System;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using MongoDB.Bson;
using MongoDB.Driver;
public static class StreamToMongoFunction
{
    private static readonly string mongoConnectionString = Environment.GetEnvironmentVariable("MongoDB_ConnectionString");
    private static readonly string databaseName = Environment.GetEnvironmentVariable("MongoDB_DatabaseName");
    private static readonly string collectionName = Environment.GetEnvironmentVariable("MongoDB_CollectionName");
    private static readonly MongoClient mongoClient = new MongoClient(mongoConnectionString);
    private static readonly IMongoDatabase database = mongoClient.GetDatabase(databaseName);
    private static readonly IMongoCollection<BsonDocument> collection = database.GetCollection<BsonDocument>(collectionName);
    [FunctionName("StreamToMongoFunction")]
    public static async Task Run([EventHubTrigger("your-event-hub-name", Connection = "EventHubConnectionString")] string myEventHubMessage, ILogger log)
    {
        log.LogInformation($"C# Event Hub trigger function processed a message: {myEventHubMessage}");
        var document = BsonDocument.Parse(myEventHubMessage);
        await collection.InsertOneAsync(document);
    }
}

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.