Kafka streams being throttled in Event hubs standard tier

Question

I am trying to run my kafka streams client and it feels like every message sent is being instantly rejected and after retry it succeeds. So I am thinking that there is some kind of throttling going on here. It also failed to create internal topics automatically so I had to manually do it. It took 17 minutes to join few topics and produce an output of 10k records.

Is it happening because of standard tier? I am planning to upgrade to premium for my production so wondering if it’s not an issue with premium as well!

Answer 1

Hi @Sid

Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.

According to this documentation: Throughput units

The throughput capacity of event hubs is controlled by throughput units. Throughput units are prepurchased units of capacity. A single throughput unit lets you: Ingress: Up to 1 MB per second or 1,000 events per second (whichever comes first) Egress: Up to 2 MB per second or 4,096 events per second.

Kafka Streams, especially during initial setup and processing of large datasets, can generate significant traffic. If your application's throughput exceeds the provisioned TUs, you'll experience throttling. The behavior you're seeing (rejections, retries) is precisely how Event Hubs signals that it's exceeding capacity.

Upgrade to Premium - The Premium tier offers dedicated resources (instead of shared ones) and much better support for high-throughput scenarios. This tier also provides better handling for Kafka Streams applications, with improved throughput, more partitions, and fewer throttling issues.

Upgrading to the Premium tier should mitigate many of the issues you're seeing with the Standard tier, especially as you're scaling your production workload. The Premium tier also supports features like guaranteed throughput and lower latency, which would likely result in better performance and fewer failures.

For more details refer to these documentations: Apache Kafka troubleshooting guide for Event Hubs

Overview of Event Hubs Premium

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.