Azure Data Streaming Issue: Switching from Block Blobs to Append Blobs

Question

I've recently made a change to my telemetry stream architecture that's causing some data loss, and I'm hoping to get some insights from those with experience.

Current Setup & Issue

• Original configuration: Telemetry stream using Block Blobs
• New configuration: Switched to Append Blobs
• Data size change: Increased from 1 KB to ~10 KB per message
• Streaming method: Using Stream Analytics to process data into Event Hub
• Problem: Since the change, I'm losing approximately 10% of my data

Questions

1. Is Append Blob the right storage type to use with Stream Analytics for this scale of data?
2. Could the increased message size be contributing to the data loss?
3. Would reverting back to Block Blobs likely resolve this issue?
4. Are there any configuration changes I should make in Stream Analytics to better handle Append Blobs? Has anyone encountered similar issues when changing blob types with Stream Analytics pipelines? Any advice would be appreciated. Thanks!

Answer 1

Hi @Annie Zhou

Welcome to Microsoft Q&A platform and thanks for posting your query here.

It seems like you are facing data loss issues after switching from using Block Blobs to Append Blobs in your telemetry stream architecture. Let's address your questions one by one:

Is Append Blob the right storage type to use with Stream Analytics for this scale of data?

It depends on the use case. Append Blobs are best suited for log-like data that grows over time, but they may not be the best fit for high-frequency streaming scenarios where latency and consistency are critical. Block Blobs are typically recommended for Stream Analytics due to their optimized write performance.

Could the increased message size be contributing to the data loss?

• Since your message size increased from 1 KB to ~10 KB, this could be affecting ingestion performance.
• Stream Analytics has input limits (such as throughput constraints) that might be causing some messages to be dropped.

Would reverting back to Block Blobs likely resolve this issue?

• If your original setup with Block Blobs was working without data loss, reverting back may be the simplest fix.
• Block Blobs allow more efficient writes and are commonly used for batch processing scenarios in Stream Analytics.

Are there any configuration changes I should make in Stream Analytics to better handle Append Blobs?

If you want to continue using Append Blobs, consider:

• Increasing Stream Analytics Streaming Units (SUs) to handle larger message sizes.
• Checking Event Hub partitioning strategy to distribute messages efficiently.
• Enabling diagnostic logs in Stream Analytics to see if events are being dropped due to throttling.

If you have encountered similar behavior before, I would recommend testing a temporary switch back to Block Blobs to compare performance.

For more information you can refer the below articles:

• Understanding block blobs, append blobs, and page blobs
• Understand and adjust Stream Analytics streaming units
• Performance and scalability checklist for Blob storage
• Understand and adjust Stream Analytics streaming units

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.