Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,068 questions
Debugging Data Loss in Stream Analytics to Event Hubs
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 28.999999999999996%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAZ%3C/text%3E%3C/svg%3E)
Annie Zhou
20
Reputation points Microsoft Employee
A stream analytics job has been set up to transfer data from a blob storage account to Event Hubs and subsequently into a Kusto database. There is an observed data loss of 10% compared to the original blob storage account.
What methods are available to debug and verify the complete amount of data being sent from blob storage to Event Hubs through the stream analytics query?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram 3,845 Reputation points Microsoft Vendor2025-02-06T22:03:25.26+00:00 Hi @Annie Zhou
Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.
To debug and verify the complete amount of data being sent from blob storage to Event Hubs through the Stream Analytics query, you can follow the below MS documentations:
Here are some methods you can follow:
Use Activity and Resource Logs - Utilize the Activity Logs to filter and identify any errors that may have occurred during the data transfer process. Additionally, job resource logs can help you debug errors related to the Stream Analytics job.
Test Queries Locally - On the Azure portal, you can test your query using the sample data downloaded from the blob storage. This allows you to examine any errors and correct them before running the job in production.
Debug Queries Step by Step -You can debug queries locally using the job diagram in Azure Stream Analytics tools for Visual Studio Code. This diagram shows how data flows through the query steps, allowing you to view metrics and data at each intermediate result set.
Check for Common Pitfalls - Ensure that there are no filtering issues in your query that could be excluding data. For instance, a WHERE clause could inadvertently filter out all events, or JOIN conditions might not match, resulting in zero output.
Monitor Input Events - Check the "Input Events Backlogged" metric to see if your job is processing data as fast as it is coming in. If this number is greater than 0, you may need to increase the number of Streaming Units or optimize your job for parallel processing.
By following these methods, you can identify where data loss may be occurring and ensure that all data is being correctly transferred from blob storage to Event Hubs.
Hope this helps. Do let us know if you have any further queries.
-
Ganesh Gurram 3,845 Reputation points Microsoft Vendor
2025-02-07T12:42:53.6566667+00:00 @Annie Zhou - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
-
Ganesh Gurram 3,845 Reputation points Microsoft Vendor
2025-02-10T10:44:02.8633333+00:00 @Annie Zhou - Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Sign in to comment
Sign in to answer