Azure Function cannot process data streaming at 10 Hz

Question

Hi

I have the following architecture

I am using the Azure Function to be able to save data in CosmosDB for Mongo since this is not supported by the Azure Streaming Analytics API and we need to use MongoDB.

All is working but when I test with incoming messages at 10 Hz I can see that Azure Function cannot save all the messages in the CosmosDB for Mongo (i.e. some of the messages are missing). It seems that the performance of the Azure Faction is not good enough.

The Azure Function is Azure Service Plan with P2v3 pricing plan

The Azure Function is

import azure.functions as func
import logging
import pymongo
import os

app = func.FunctionApp(http_auth_level=func.AuthLevel.FUNCTION)

@app.route(route="SaveToMongoDB")
def SaveToMongoDB(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    client = pymongo.MongoClient(os.environ['cosmosdbformongodb_connectionstring'])

    db = client["db"]
    json_data = req.get_json()[0]
    db["events"].insert_one(json_data)
    
    return func.HttpResponse(f"This HTTP triggered function executed successfully.")

My questions are:

1. Is this the correct setup for saving the streaming data to CosmosDB for Mongo?
2. if yes to the 1) why is the Azure Function not performing to save all the data? Do I miss some setting?

Please let me know

Thanks
Alberto

Answer 1

Hello @Alberto Morando,

welcome to this moderated Azure community forum.

Although I use C#, Azure Functions are reliable in my situation.

10Hz could be a challenge if you have to do the scaling yourself because you need to monitor the resource usage.

Can you first test the Azure Function being triggered while no actual logic being executed?

So you only log the separate calls and show 10Hz is not an issue?

This way, is this works well, the challenge could be the DB connection.

I'm not an expert regarding Python, but it seems you are creating and destroying a DB connection with EVERY call, 10 times a second.

Can you reuse an existing connection that survives the stateless Function calls?

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.