Understanding MQTT Session Overflow in Azure Event Grid

Question

I have a question about the buffering capabilities of Azure Event Grid in relation to MQTT.

The documentation states the following about Session Overflow:
MQTT broker maintains a queue of messages for each active MQTT session that isn't connected until the client reconnects to receive the messages in the queue. If a client doesn't connect to receive the queued QOS1 messages, the session queue starts accumulating messages until it reaches its limit: 100 messages or 1 MB. Once the queue reaches its limit during the lifespan of the session, the session is terminated.

When reading this, it seems that the MQTT broker can only store 100 messages or up to 1 MB. However, during testing, I found that if a client is offline, it can receive more messages (tested up to 1000) when coming online again.

Additionally, when testing with large messages (256 KB), the client does not receive messages upon reconnecting if the "receiveMaximum" is set to more than 1. I expected that it could be at least 3 since 256 x 3 = 768 KB, which is less than 1 MB, so the client should collect all 3 messages at once.

This behavior of Event Grid raises some questions. Could someone explain how Session Overflow works? Moreover, what is the maximum number of messages that can be stored by Event Grid?

Thank you in advance!

Answer 1

@Anouk ,

Thanks for reaching out. Azure Event Grid's MQTT broker manages a queue for each active session to store QoS 1 messages for offline clients. The queue can hold up to 100 messages or 1 MB of data, whichever limit is reached first. If the queue reaches its limit, the session is terminated to maintain system stability

Regarding your observations:

1. Receiving More Than 100 Messages Upon Reconnection: If your client receives more than 100 messages after reconnecting, it might be due to multiple sessions being utilized. Azure Event Grid allows for multiple sessions per client, each with its own queue. By using different ClientId values, a single client can establish multiple sessions, effectively increasing the total number of messages it can receive upon reconnection
2. Issues with Large Messages and receiveMaximum Setting: The receiveMaximum parameter in MQTT v5 specifies the maximum number of unacknowledged PUBLISH messages the client is willing to process concurrently. Azure Event Grid enforces a maximum message size of 512 KB. If you're sending messages of 256 KB each and set receiveMaximum to a value greater than 1, the combined size of unacknowledged messages could exceed the in-flight bandwidth limit, leading to message delivery issues.

In summary, while Azure Event Grid's MQTT broker has default session queue limits of 100 messages or 1 MB, these can be managed by utilizing multiple sessions or adjusting client configurations. It's also important to consider other related limits, such as in-flight message bandwidth, to ensure reliable message delivery, especially when dealing with large messages.
Reference:

https://learn.microsoft.com/en-us/azure/event-grid/mqtt-support

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for "Was this answer helpful." And if you have any further questions, let us know.