Hi. Recently we experienced a weird phenomenon of a sudden and very short dip in our ADX Cluster Is Alive this didn't impact performance/data what-so-ever (as ingestion continued as usual and restored itself automatically) but I'm interested to learn: Is it common? unique? when do these take place? I read the docu for Support + troubleshooting re CPU/workload, but interested in understanding from real examples, when this takes place during the work on the cluster (e.g. heavy queries, long ingestions, many users simultaneously, etc) how can I monitor and understand the reason? Can it be avoided? We're using our cluster in many ways with CPU fluctuating between 10% to 80% usually Glad to read about any experience re this :) Tnx!

Hello Ori Bandel , Do you mean sudden dip in the "Keep alive" metrics? The "Keep alive" metric in Azure Data Explorer is a key indicator of cluster health, and a dip in this metric suggests a temporary issue with the cluster. It should ideally remain close to 1 for a healthy cluster Can you check the below? Use the .show diagnostics command to check if the cluster is healthy and identify any issues. Monitor the "Keep Alive" metric in the Azure portal, which should be close to 1 for a healthy cluster. Review the resource and ingestion metrics listed in cluster metrics and review the recommendations in the metric description column. Reference document: https://learn.microsoft.com/en-us/azure/data-explorer/check-cluster-health I hope this helps.

ADX Cluster Is Alive sudden dip

Ori Bandel 30

Hi.

Recently we experienced a weird phenomenon of a sudden and very short dip in our ADX Cluster Is Alive this didn't impact performance/data what-so-ever (as ingestion continued as usual and restored itself automatically) but I'm interested to learn:

Is it common? unique?
when do these take place? I read the docu for Support + troubleshooting re CPU/workload, but interested in understanding from real examples, when this takes place during the work on the cluster (e.g. heavy queries, long ingestions, many users simultaneously, etc)
how can I monitor and understand the reason?
Can it be avoided?

We're using our cluster in many ways with CPU fluctuating between 10% to 80% usually

Glad to read about any experience re this :)

Tnx!

Bhargava-MSFT 31,121 Reputation points Microsoft Employee

2024-08-09T18:38:59.6433333+00:00

Hello Ori Bandel,

I am checking to see if you had a chance to look into my above response. Please let me know if you have any further questions. Thank you.

1 answer

Bhargava-MSFT 31,121 Reputation points Microsoft Employee

2024-08-05T18:43:55.97+00:00
Hello Ori Bandel,

Do you mean sudden dip in the "Keep alive" metrics? The "Keep alive" metric in Azure Data Explorer is a key indicator of cluster health, and a dip in this metric suggests a temporary issue with the cluster. It should ideally remain close to 1 for a healthy cluster

Can you check the below?

Use the .show diagnostics command to check if the cluster is healthy and identify any issues.

Monitor the "Keep Alive" metric in the Azure portal, which should be close to 1 for a healthy cluster.

Review the resource and ingestion metrics listed in cluster metrics and review the recommendations in the metric description column.

Reference document:

https://learn.microsoft.com/en-us/azure/data-explorer/check-cluster-health

I hope this helps.
Please sign in to rate this answer.
Ori Bandel 30 Reputation points

2024-10-05T16:49:10.9266667+00:00

Hi Bhargava-MSFT

deep apologies for the delay

I'm not familiar with the '.show diagnostics' and I haven't found the relevant docu page -can you please share it?

re the rest -- I do use it and monitor it, nevertheless, we are following all/most KQL best practices and we don't see what caused this

tnx!

Bhargava-MSFT 31,121 Reputation points Microsoft Employee

2024-10-07T22:14:39.8633333+00:00

Can you please check this below document:

https://learn.microsoft.com/en-us/azure/data-explorer/check-cluster-health

Ori Bandel 30 Reputation points

2024-10-09T09:18:05.5533333+00:00

Hi

tnx!

I checked the .show diagnostics operator and all seems OK (and we're familiar with and monitoring all the cluster health metrics in the Azure portal daily)

nevertheless, we experience small/minor issues with the cluster

Can you please recommend a technical Azure/ADX course that can assist with a better understanding of how to optimize the ADX Cluster?

Is it normal/regular for the cluster to have small availability "hiccups"? I know is CAN happen but does it happen?

Bhargava-MSFT 31,121 Reputation points Microsoft Employee

2024-10-09T17:57:55.4666667+00:00

Did you consider scaling up your cluster? you might need to consider increasing cluster size, adjusting scaling settings, or analyzing your workload to optimize specific queries that might be causing pressure on the cluster.

https://learn.microsoft.com/en-us/azure/data-explorer/manage-cluster-horizontal-scaling

https://learn.microsoft.com/en-us/azure/data-explorer/manage-cluster-vertical-scaling

also, you can check out the Azure Data Explorer documentation here:

https://learn.microsoft.com/en-us/azure/data-explorer/

Regarding the availability "hiccups", It's not unusual for Azure Data Explorer clusters to experience minor availability issues from time to time. These "hiccups" can be caused by various factors, including temporary network issues, resource contention, or maintenance activities.

If you are not able to figure out the issues with availability, I would suggest opening a support case so that support engineer can further troubleshoot the issues with your cluster.

Ori Bandel 30 Reputation points

2024-10-12T20:55:26.3566667+00:00

Hi

Tnx for the updates and suggestions!

We have already used many and varied solutions, part of which you listed above

Will indeed consider that some more :)

Tnx
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADX Cluster Is Alive sudden dip

1 answer

Your answer