How can I scale my mobile application using Azure Event Hub to support more than 5 concurrent connections per consumer group for real-time sensor data processing?

Question

I have a mobile application that reads sensor data, and we need it to support multiple users simultaneously. Currently, we use Azure Event Hub to handle the data, with consumer groups connecting to it. However, one consumer group only allows 5 connections at a time, which limits our ability to scale and support more users in real-time. What are some other ways or solutions we can implement to overcome this limitation and allow more connections efficiently?

Answer 1

Hello @Keerthana Merugu ,

welcome to this moderated Azure community forum.

If I understand correctly, you have a mobile application that consumes messages coming from an Azure Event Hub.

To consume messages, a consumer needs to be part of a consumer group.

The limit for an consumer group is up to five consumers, otherwise, you can expect errors.

So, the only way to cope with more users is scaling up:

Depending on the Event Hub tier, 1, 20, 100, 1000 consumer groups are supported (I have no experience with what 'No limit per CU' means).

Notice that The Kafka endpoint of an Event Hub also knows the concept of consumer groups (per namespace) and the number is always 1000. Please check the documentation of this endpoint and what a consumer groups means there.

As an alternative, it is highly recommended to check out the EventGrid Namespace MQTT support because this service can serve 10.000 clients per throughput unit out of the box.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Answer 2

Hello Keertha, Welcome to MS Q&A

To overcome the connection limitation of Azure Event Hub consumer groups, you can implement the following solution

The job engine can use multiple readers per query from that same consumer group. When there are too many references to the same consumer group, the job can exceed the limit of five and throw an error. In those circumstances, you can further divide by using multiple inputs across multiple consumer groups.

Scenarios in which the number of readers per partition exceeds the Event Hubs limit of five include:

Multiple SELECT statements: If you use multiple SELECT statements that refer to the same event hub input, each SELECT statement causes a new receiver to be created.

UNION: When you use UNION, it's possible to have multiple inputs that refer to the same event hub and consumer group.

SELF JOIN: When you use a SELF JOIN operation, it's possible to refer to the same event hub multiple times.

The following best practices can help mitigate scenarios in which the number of readers per partition exceeds the Event Hubs limit of five.

For more details , please check this doc --> https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-troubleshoot-input?wt.mc_id=knwlserapi_inproduct_azportal#readers-per-partition-exceed-the-event-hubs-limit

Kindly let us now if any further questions

Please accept answer if it helps

Thanks

Deepanshu