Ingestion response takes too long

Question

Hello ADX experts.

I have a background worker that are doing calculations in near real-time. The results are then ingested to ADX using the .NET SDK.

It's important that the ingestion operation doesn't take too long, or it will accumulate a delay which causes the worker to "get behind". When inserting data to Azure SQL, the operation takes 10-30 ms which is acceptable.

However, ingesting to ADX takes much longer. When the traffic is low, streaming ingestion takes about 300 ms to complete, and queued ingestion about 100 ms. When the traffic increases, streaming ingestion takes 500-1000 ms, while queued ingestion takes 300-500 ms. The ReportLevel is set to default FailuresOnly.

Is this expected? What is a normal response time? I have tried increasing the SKU, but it didn't make much difference. It's the call to IKustoQueuedIngestClient.IngestFromDataReaderAsync that takes time.

I need to wait for the call to return to know whether the data was ingested successfully or not, since the next action depends on it.

There are multiple threads ingesting data simultaneously, and I'm using a single instance of the client, as recommended here. The application is a Function App running in the same Azure region as the ADX cluster.

I'm not sure if the delay is caused by the thread-safety mechanisms in the client SDK or if the ingest API itself is slow.

Do you have any recommandations on how I can reduce the ingest response time?

Answer 1

To reduce ADX ingestion time:

1. Adjust Batching Policy: Lower MaximumBatchingTimeSpan and MaximumNumberOfItems.
2. Optimize Client: Use a single client instance for multiple threads.
3. Use Streaming Ingestion: Fine-tune settings for better performance.
4. Check Network: Ensure minimal latency between Function App and ADX.
5. Monitor and Scale: Continuously monitor and scale ADX resources.
6. Update SDK: Use the latest .NET SDK for potential performance improvements.

Answer 2

Hello @Pål Kristian Halle,

welcome to this moderated Azure community forum.

Azure Data Explorer has separate data management set up next to the compute cluster.

Scaling the cluster will increase the availability of the data slightly.

I expect you enabled streaming ingest already because this skips the standard batching mechanism (1mb of data, 1000 rows or 5 minutes).

Making use of a partitioning policy will also speed up retrieval but it not sure if this helps you with your initial availability problem.

We faced a similar challenge for live dashboard updates.

What we did when we needed the stream of data while inserted in Azure Data Explorer, we started to read the streaming data and query ADX only if we need historical.

I hope you can speed up data retrieval from ADX but perhaps this new architecture approach is more viable.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.