Debugging Data Loss in Stream Analytics to Event Hubs and ADF

Question

My goal is to stream telemetry data from blob storage files into a Azure Data Factory database.

I have set up a stream analytics job that takes data from blob storage, bins it by hour, and then passes it through an event hub with two partitions in order to pass it into a table inside a database. However, while some of the corresponding data is similar, the database is missing 10% of the expected data on a daily basis. Some data points are not being streamed into the database, without rhyme or reason.

1. What are possible failure points to debug?
2. What are ways to monitor and find errors in the process that could be leading to dropped data?

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A. You need to systematically analyze each component in the pipeline. Here are the possible failure points and steps to monitor/debug:

1. Diagnostics and Monitoring:
   • Enable diagnostic settings for Stream Analytics, Event Hub, and ADF to collect logs and metrics.
   • Use Azure Monitor to set up alerts for unusual patterns or dropped messages.
2. Data Consistency Verification:
   • Implement a reconciliation process to compare source and destination data.
   • Use checksum/hash validation to detect mismatches between Blob, Event Hub, and the database.
3. Partitioning Strategy:
   • Review partition keys in Event Hub and database to ensure even data distribution.
   • Align Stream Analytics query outputs with Event Hub partitioning.
4. Error Handling and Retention:
   • Enable error-handling policies in Stream Analytics (e.g., send malformed data to a separate output sink for inspection).
   • Increase Event Hub retention time to allow for delayed processing by ADF.
5. Scaling and Performance Tuning:
   • Scale up throughput units for Event Hub and Stream Analytics if metrics show performance bottlenecks.
   • Optimize ADF pipelines for parallelism and batch size to improve throughput.

By investigating each stage and implementing monitoring, you should be able to identify and resolve the root cause of data loss.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.