Freigeben über

Data Retention in Event Hubs

  • Artikel

This is a common question so I'm going to put it on the blog before it works its way into our more formal documentation. Event Hubs have a message retention setting that is chosen at creation time; it can also be changed later.

 

The setting is shown in the portal below.

 

The default value is one day. This, however, leads to confusion in that Service Bus Messaging (Queues & Topics) have a Default Message Time to Live (TTL) on a Queue, Topic, or Subscription. These concepts do not mean the same thing - though they are both defaults and involve time. The similarities end there.

UPDATE: This logic has changed and we now delete data at the specified retention window no matter how much or little is in the partition. We still delete the entire extent, but we no longer require it to be closed.

This leads to the common confusion "I see events older than one day in my Event Hub - what is wrong". The short answer is nothing - read on for more information.

Service Bus Messaging (again Queues & Topics) provides per message control and lifetime management. Event Hubs do not. Event Hubs are much more coarse grained. This means on Service Bus Messaging entities (Queue / Topic / Subscription) you can specify lifetime / expiration on a single message or on an entity. When that specific message has expired, it is no longer in the entity - at least from the users' viewpoint.

The same is not true of Event Hubs - the EventData class does not expose such a concept as TTL and the only time based value to be specified is the Message Retention interval at the Event Hub level. Here is where the confusion comes into play. Event Hubs are a massive scale service - we don't track very much on a per event basis. Each partition is basically a sequence of event packages or blocks. In a way you could think of this like shipping containers on a transport ship. We load events into the container then, when it is full, load it on the ship; marking the time at which this happens (OK - it doesn't really work that way, but use your imagination a bit). Every now and then we look at the fully loaded containers, which have their time stamp on them, and unload or discard the ones that are outside the message retention window.

There are two implications here: the first is that retention happens in a fairly large block. Again, Event Hubs are made for big scale - if you don't need scale a Service Bus Queue may be better for you. Secondly we don't look at the containers until they're full. So if you put only a few items into an Event Hubs - even a few tens of thousands - there is a high likelihood that you will see events from beyond the retention period. 

Nothing is wrong - keep reading events and if you're very concerned about older events keep careful track of your offsets or use a DateTime value when connecting to only read events from a specific point in time.

Comments

  • Anonymous
    March 09, 2015
    Great Work

  • Anonymous
    March 10, 2015
    I'm not able to find the content when message expired. Is there any retrieval method available.

  • Anonymous
    March 10, 2015
    I'm sorry I don't understand. Once the retention interval has elapsed we will, on our own time, clean up the data. It will be gone forever.

  • Anonymous
    March 23, 2015
    Fantastic Dan!

  • Anonymous
    August 24, 2015
    Nice Summary. Any pointers on where this is explained technically?

  • Anonymous
    December 03, 2015
    This architectural choice abuot the eventhub non applying retention is very arguementable, badly documented and definetely unclear speaking at interface level (in both azure manager and portal) I don't care/mind if applying the retention time to eventhubs is technically not possible due to the eventhub massive nature, making an interface that lets me set a parameter that is ignored with no warining, no documentation and no alert is NOT ACCEPTABLE. "Do it good or avoid doing it".

  • Anonymous
    February 03, 2016
    Agreed with above comment by Manjia. Of course it was a common question! If Dan's explanation were included in the pop up help message for this field, then everything would have been fine. Also, it would not take much to fix it, right? It's been almost a year now after the original post.

  • Anonymous
    May 03, 2016
    Hi Dan,Now I see clearly, whats the problem behind that. I was also very confused, that there is no deleting of records going on.A more obvious documentation of the feature would be desirable.Best Regards,Tarek