This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
My setup consists of several databases in a single Kusto cluster, each having a landing table with ingestion mapping configured from IoT Hubs. The data in the landing table is then split with update policies into specialized tables.
I have no ingestion batching policy defined, so things run on defaults.
To me it looks like the cluster is not keeping up with ingesting data, ingestion utilization is high at 100%, but strangely upon restarting the cluster, it somehow manages to ingest everything quickly, after which things start accumulating again. Metrics also suggest nothing is ingested for days, then a whole lot is ingested at once (see spikes in metrics).
This leads me to believe I should change some configuration to help with ingestion - but what?
Another strange thing is that data queries run fine, but some management queries time out
Here are some of the metrics:
All recommendations green, cluster health green.
Thanks for all the support!
Hello @Sander van de Velde | MVP ,
Scaling up does help, it also does a system flush as a side effect which helps the same way as a restart. Right now the cluster is a small dual core D series SKU. I have not tried scaling out yet.
The fact that it can process all accumulated queues without issues on restart, leads me to believe that it is possible to manage with a smaller cluster, if configured wisely.
I am now trying to increase the largest database batching policy to 30 minutes since I saw all ingestion events were triggered by Time and none by Rows or Size (I guess this is normal for IoT Hub ingestion with small JSON payloads).
It is strange how the metrics report zero ingestion for several days, then all of them land at once. Latency stays at zero and then jumps to days. Do the 700 throttled commands refer to ingestion commands or data queries?
@Kovács, Ferenc - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
@Kovács, Ferenc - Did you get a chance to open a support ticket?
Hello @Kovács, Ferenc,
welcome to this moderated Azure community forum.
Can you share the size and kind of your cluster?
Have you tried scaling out already for a couple of days?
If so, did this give another result?
Explicitly setting the ingestion batch policy to 30 m decreased the load and avoids this deadlock - it is a workaround.