I am consuming events from a consumer group into ClickHouse via a Kafka endpoint for Event Hubs. After consuming data with a consumer group and re-reading it from the beginning, I cannot delete the consumer group and reuse the same name, as it seems to be persistent. Even after deleting the group, the offsets remain persisted. Is there a way to use the same consumer group name and simply reset the offset?

Hi @ Sid Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query. To reset the offsets for a consumer group in Azure Event Hubs when using the Kafka endpoint, you typically need to manage the offsets manually. While you can delete a consumer group, the offsets may remain persisted, and you cannot reuse the same name if the group still exists. To effectively reset the offset, you can use the Kafka consumer group APIs to manipulate the offsets directly. This allows you to set the offsets to a specific point in the event stream, enabling you to re-read the events from that position. However, keep in mind that offsets are stored in the Event Hubs service, and you will need to ensure that your consumer group is configured correctly to manage these offsets. If you want to start consuming from the beginning of the stream again, you can also specify the starting offset as the earliest available offset when you create your consumer or when you reconnect. Reference: Frequently asked questions - Event Hubs for Apache Kafka Features and terminology in Azure Event Hubs Hope this helps. Do let us know if you any further queries. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Deleting a Consumer Group Without Persisting Offsets

Accepted answer

Ganesh Gurram 3,765 Reputation points Microsoft Vendor

2025-01-30T11:08:37.4233333+00:00

Hi @Sid

Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.

To reset the offsets for a consumer group in Azure Event Hubs when using the Kafka endpoint, you typically need to manage the offsets manually. While you can delete a consumer group, the offsets may remain persisted, and you cannot reuse the same name if the group still exists.

To effectively reset the offset, you can use the Kafka consumer group APIs to manipulate the offsets directly. This allows you to set the offsets to a specific point in the event stream, enabling you to re-read the events from that position. However, keep in mind that offsets are stored in the Event Hubs service, and you will need to ensure that your consumer group is configured correctly to manage these offsets.

If you want to start consuming from the beginning of the stream again, you can also specify the starting offset as the earliest available offset when you create your consumer or when you reconnect.

Reference: Frequently asked questions - Event Hubs for Apache Kafka

Features and terminology in Azure Event Hubs

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.

2 people found this answer helpful.
Sid 60 Reputation points

2025-01-30T17:16:59.7933333+00:00

Is there any way to do it either using azure cli or using the azure portal?

Ganesh Gurram 3,765 Reputation points Microsoft Vendor

2025-01-30T18:49:49.5666667+00:00

@Sid - Currently, Azure Event Hubs does not provide a direct option via the Azure CLI or Azure Portal to reset offsets for a specific consumer group using the Kafka endpoint. However, you can achieve this by managing your consumer group offsets programmatically using Kafka consumer APIs or by implementing custom logic in your consumer application.

Here's a general approach you can take:

Using Kafka Consumer API - You can use the Kafka consumer APIs to manually set the offsets for your consumer group. This involves creating a consumer with the same group ID and seeking the desired offset (e.g., earliest or latest).

Delete and Recreate the Consumer Group - If you truly need to delete the consumer group and start fresh, you might have to stop all consumers in that group and wait for the group to be removed from the Event Hubs service, which can take some time. After that, you can recreate the consumer group.

Create a New Consumer Group - As an alternative, you can create a new consumer group with a different name and start consuming from the beginning of the stream. This does not reset the original consumer group's offsets but provides a fresh start with a new group.

Programmatically Reset Offsets - If you're developing an application, you can implement logic to programmatically manage offsets by storing them externally and applying them when your application connects to consume events.

Unfortunately, the Azure CLI and Azure Portal do not currently offer direct support for these specific operations regarding Kafka offsets in Event Hubs. Keeping an eye on Azure updates and documentation is a good practice to see if such features become available in the future.

If you have further questions or need more detailed guidance, feel free to ask!

Thank you.

Sid 60 Reputation points

2025-01-30T19:27:07.95+00:00

Delete and Recreate the Consumer Group - If you truly need to delete the consumer group and start fresh, you might have to stop all consumers in that group and wait for the group to be removed from the Event Hubs service, which can take some time. After that, you can recreate the consumer group.

I did wait for the consumer group to be deleted, its instant for me. But I cannot use the same name to recreate as its somehow persisting the offsets even after I delete and wait for 12 hours+ I cannot use the same name.

Ganesh Gurram 3,765 Reputation points Microsoft Vendor

2025-01-31T14:17:13.2+00:00

@Sid - Apologize for the inconvenience caused.

Here are some steps and considerations that might help:

Use Kafka's consumer API to manually set the consumer group's offsets to the earliest (or a specific point) when reusing the same group. This gives you more control over where you begin reading from, even if the offsets are persisting in Event Hubs.

If you're looking to start fresh with the same configuration but cannot reuse the old consumer group name, the easiest workaround would be to create a new consumer group with a different name. You can specify earliest as the offset when you start consuming to read the data from the beginning.

Hope this helps.

Thank you.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Deleting a Consumer Group Without Persisting Offsets

0 additional answers

Your answer