Topic Size and Partitioning on Service Bus

Azure Service Bus is a fabulous vehicle to transport data from one end to the other. Since the incoming traffic data cannot be returned to the sender once it is in, one outstanding risk with the transportation channel is, whether or not we have enough room to host a large amount data, before they can be consumed, even temporarily, One easy option is, to minimize the TTL time for every message in Service Bus. But, what if we have a big list of subscribers to the same topic, and some of these subscribers are lazy to consume their copy of data, or they do receive data timely, but they forget to "receive and delete" programmatically? More and more data will be stuck inside Service Bus. The related topic used size will reach the maximum. As a result, Service Bus becomes unavailable. We have actually seen this symptom in the high load / longhaul stress test lab. It will become a nightmare to deal with such situation in the business world. It is unavoidable in an unmanaged Service Bus hosting environment.

Recently I shared a story about one day (or night) I was suddenly getting 80 GB topic size. I thought it was an accidental discount by operational glitch, but it happened again last night. I have learned the "root cause" from an expert. Thanks to the recent released partitioning feature for Service Bus Topic, it "boosts up" Topic size dramatically. If it is turned on, when you create a new Topic, you will get 16 times the max size you are requesting. So, if you want to configure 5 GB with partition on (set "true"), you get 80 GB. I tried it with 3 GB plus partition equal true, and I've got 48 GB max size in return, as expected. Very cool!

 
   
  

Here is a code snippet just for reference.

        private static void CreateEventTopic(string connectionString, string topicName, long topicMaxSizeInMB, bool partitionEnabled)
        {
            if (string.IsNullOrEmpty(topicName))
            {
                throw new Exception("Error: Topic name is needed!");
            }

            NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
            // this method (TopicExists) is NOT case sensitive for topicName string
            if (!namespaceManager.TopicExists(topicName))
            {
                namespaceManager.CreateTopic(
                    new TopicDescription(topicName)
                    {
                        DefaultMessageTimeToLive = TimeSpan.FromMilliseconds(60 * 1000),
                        EnableBatchedOperations = true,
                        DuplicateDetectionHistoryTimeWindow = TimeSpan.FromMilliseconds(60 * 1000),
                        RequiresDuplicateDetection = true,
                        MaxSizeInMegabytes = topicMaxSizeInMB,
                        EnablePartitioning = partitionEnabled
                    });
            }
        }

With 80 GB (per topic) on hand, I feel much more comfortable to do more things with the toy. The partitioning feature looks promising in terms of messaging handling for both send and receive. You may find detailed description from https://msdn.microsoft.com/library/azure/dn520246.aspx.  With excitement and hope, I have enabled this feature for all topics. We will try and see how it is working in practice.