How do I know if my Event Hubs (Cloud Gateway) is working?

 

Hi all, Event Hubs is one of the ways of ingesting data into Microsoft Azure cloud. It is a temporary means of messaging storage under service bus. It essentially works as a parallel queue, and uses a “producer consumer” like principle. Event hubs is one of the early modules that enables strong IoT scenarios in Azure and is being used widely in many data scenarios. It also is capable of delivering high bandwidth server to server communication bridging. In any case, one of the most common questions I receive is “How do I make sure the module is working?” Being one the very first steps to connect your IoT device to the cloud, you may want some verification around if it actually received your data or not.

 

Code Side Verification

One of the first steps should be code side verification. Below is a C# sample that uses the service bus sdk. It implements a try,catch structure to make sure the EventHubClient.Send() returns properly.

 

private EventHubClient _eventHubClient;

try
            {

                var serialisedString = JsonConvert.SerializeObject(TwitterPayloadData);
                EventData data = new EventData(Encoding.UTF8.GetBytes(serialisedString)) { PartitionKey = TwitterPayloadData.Topic };
                _eventHubClient.Send(data);
              
               
                Console.ForegroundColor = ConsoleColor.Yellow;
                Console.WriteLine("Sending" + serialisedString + " at: " + TwitterPayloadData.CreatedAt.ToString() );
                               
            }

catch (Exception ex)
            {
               
            }

Event Hubs Monitoring

Let’s take a look at the cloud side of things. In the Azure Portal under Service Bus > EventHubs , under the dashboard tab, you should see if it receives any messages. It may take a minute or two for the graph to update.

Sampling Data Using Stream Analytics

One other way I like using personally is using Stream Analytics as a consumer for the Event Hubs and see some actual data.

Couple of words around Stream Analytics, it is a PaaS module on Azure for real-time data analytics and event processing. Chances are you might want to use one if you are doing any sort of IoT implementation on Azure. For the scope of this blog ,we will use SA for testing and verifying the messages that the Event Hubs receive.

You can find Stream Analytics under Data Services in the “new” menu bottom left. After following the wizard, you’ll have a SA created. When you navigate to the SA, you’ll see three distinct tabs, Inputs, Query, Output. Inputs is the one we are interested right now. It enables us to add inputs to our SA.

 

The next step is to add our Event Hubs as and input to the Stream Analytics. On the bar at the bottom, click on the “ADD INPUT” button.

 

We will use the Data stream option as we are trying to get live data from the Event Hubs. When you follow the wizard, it will automatically identify the EventHubs that you have in your account. At this point you may choose to get data from an Event Hub that is located on a different account. Give your input an alias name and finish the wizard. The input name is important for identification purposes in the query section. Query is used to do actual processing on the data which will be covered in a different post.

 

Once you’ve created your input for the SA, on the bottom bar you will see a “SAMPLE DATA” button. Click on it and then specify a time period, that you think that you were sending data, then the SA will go and pull the data from you Event Hub for the range of the timestamps that you’ve specified. When the process is finished, you can opt to download the data as a CSV file and not only verify the ingestion is working but also get the actual data and check if it is received the way you like and in the correct format.

 

 

Shalll you need more information on Event Hubs and Stream Analytics, feel free to dive in to the links!