Debugging Data Loss in Stream Analytics to Event Hubs

Question

A stream analytics job has been set up to transfer data from a blob storage account to Event Hubs and subsequently into a Kusto database. There is an observed data loss of 10% compared to the original blob storage account.

What methods are available to debug and verify the complete amount of data being sent from blob storage to Event Hubs through the stream analytics query?

Answer 1

Hi @Annie Zhou

Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.

To debug and verify the complete amount of data being sent from blob storage to Event Hubs through the Stream Analytics query, you can follow the below MS documentations:

• Troubleshoot Azure Stream Analytics queries
• Data-driven debugging by using the job diagram
• Troubleshoot Azure Stream Analytics queries

Here are some methods you can follow:

Use Activity and Resource Logs - Utilize the Activity Logs to filter and identify any errors that may have occurred during the data transfer process. Additionally, job resource logs can help you debug errors related to the Stream Analytics job.

Test Queries Locally - On the Azure portal, you can test your query using the sample data downloaded from the blob storage. This allows you to examine any errors and correct them before running the job in production.

Debug Queries Step by Step -You can debug queries locally using the job diagram in Azure Stream Analytics tools for Visual Studio Code. This diagram shows how data flows through the query steps, allowing you to view metrics and data at each intermediate result set.

Check for Common Pitfalls - Ensure that there are no filtering issues in your query that could be excluding data. For instance, a WHERE clause could inadvertently filter out all events, or JOIN conditions might not match, resulting in zero output.

Monitor Input Events - Check the "Input Events Backlogged" metric to see if your job is processing data as fast as it is coming in. If this number is greater than 0, you may need to increase the number of Streaming Units or optimize your job for parallel processing.

By following these methods, you can identify where data loss may be occurring and ensure that all data is being correctly transferred from blob storage to Event Hubs.

Hope this helps. Do let us know if you have any further queries.