Azure Event Hubs Timeout Issue with High Throughput

Koemuer, Taha Yusuf (148) 0

Hello, a proof of concept is being conducted with Azure Event Hubs to ingest real-time data streams into storage. According to the documentation, a single Throughput Unit (TU) can handle up to 1,000 messages per second. However, in practice, with 10 TUs, even 100 messages per second eventually start timing out.

The goal is to ensure that events sent by external services are reliably ingested by Event Hubs without timeouts. Each message size is approximately 50 bytes.

Here’s what has been tried so far:

Creating multiple Event Hubs and using a round-robin approach for sending requests.
Configuring partitions to distribute the load.
Increasing the number of TUs.
Enabling auto-scaling.
Keeping the Capture feature always enabled.

phemanth 13,625 Reputation points Microsoft Vendor

2025-01-03T17:53:44.6533333+00:00
@Koemuer, Taha Yusuf (148)

Welcome to the Microsoft Q&A and thank you for posting your questions here

It Seems like you've already taken several important steps to address the timeout issue with Azure Event Hubs. Here are a few additional Steps please check and confirm us

Check for Throttling: Ensure that your Event Hubs namespace isn't being throttled. Throttling can occur if the ingress rate exceeds the capacity of the purchased throughput units. You can monitor this using Azure Monitor metrics

Optimize Client Configuration: Adjust the client-side configurations, such as increasing the prefetchCount to reduce the number of round trips between the client and Event Hubs. This can help improve throughput.

Batch Processing: Use batch processing for sending events. This reduces the overhead of sending individual messages and can significantly improve throughput

Review Network Latency: Ensure that there are no network latency issues between your data sources and Event Hubs. Network latency can cause timeouts and affect performance.

Partition Load Balancing: Make sure that the load is evenly distributed across partitions. Uneven load distribution can lead to some partitions being overloaded while others are underutilized

Auto-Inflate Feature: Verify that the auto-inflate feature is correctly configured to automatically scale the number of throughput units based on the load

Event Processor Optimization: If you're using EventProcessorClient, consider using processEventBatch instead of processEvent to handle high traffic more efficiently

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you
phemanth 13,625 Reputation points Microsoft Vendor

2025-01-06T03:49:35.03+00:00

@Koemuer, Taha Yusuf (148) We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Koemuer, Taha Yusuf (148) 0 Reputation points

2025-01-06T09:49:33.3733333+00:00
Hi! Thank you for the response. Let me answer all the list one by one.

1- No throttling of any requests in 100k's of requests

2- I am not using client SDK but only the Rest API of

servicebus.windows.net

3- Not sure how to set up batches with Rest API

4- How can I ensure network latency?

5- Does partition have an effect on ingestion rate? We are not going to consume the ingested data by other service but we just want to put them on the bucket

6- It is enabled

7- We are not using the SDK

Thanks in advance.
Corey Wall (Dexian Inc) 0 Reputation points Microsoft Vendor

2025-01-07T19:48:57.6633333+00:00

Hi @Koemuer, Taha Yusuf (148)

We have seen a number of cases related to intermittent producer timeouts occurring seemingly at random in high throughput scenarios.

This behavior is related to a known issue with older versions of the SDK / Maven Repository for Kafka.

I recommend that you update to the latest versions and re-test.

Please try this and let me know if you are still seeing the behavior!
phemanth 13,625 Reputation points Microsoft Vendor

2025-01-09T04:25:27.49+00:00
@Koemuer, Taha Yusuf (148) Thanks for the detailed responses

Setting Up Batches with REST API: You can send batch events using the REST API by creating a JSON payload that includes multiple messages. Here’s a brief example:

[ {"Body": "Message1"}, {"Body": "Message2"}, {"Body": "Message3"} ]

Ensure the Content-Type header is set to application/vnd.microsoft.servicebus.json

Ensuring Network Latency: To check for network latency, you can:

Use tools like Azure Monitor to track network performance.

Ensure your data sources and Event Hubs are in the same Azure region to minimize latency.

Use Azure Network Watcher to diagnose and visualize network issues.

Effect of Partitions on Ingestion Rate: Yes, partitions can affect the ingestion rate. Each partition can handle a certain amount of throughput, so distributing the load evenly across partitions can help improve performance.Even if you're not consuming the data immediately, proper partitioning ensures efficient data ingestion.

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you

1 answer

Koemuer, Taha Yusuf (148) 0 Reputation points

2025-01-09T08:26:13.6266667+00:00

Thank you for all the responses, we aborted the idea of using Event Hubs for another reason. The reason is there is no way to add a custom header to the requests generated by Jira Webhooks, so basically SAS authentication is not working. This post can be closed.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Event Hubs Timeout Issue with High Throughput

1 answer

Your answer