Azure HDInsight
An Azure managed cluster service for open-source analytics.
217 questions
Azure HDInsight: Managing Unhealthy Nodes and Sessions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 91%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mahesh Sanga
0
Reputation points Microsoft Employee
Hi Team,
We are currently facing an issue with Unhealthy Nodes on Azure HDInsight. We have one centralized HDInsight cluster shared between Data Teams, where multiple Spark and Hive jobs are being submitted continuously. However, due to either overload or code-related issues, many HDInsight nodes are becoming unhealthy.
Do you have a solution to terminate those unhealthy nodes and sessions once they reach a 90% threshold?
Regards,
Mahesh
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla 5,755 Reputation points Microsoft Vendor2024-10-22T00:33:39.3666667+00:00 Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!
As I understand you're encountering unhealthy nodes in your centralized HDInsight cluster due to potential overload or code-related issues. While automatically terminating nodes based on a specific threshold isn't directly supported in HDInsight.
Here are some strategies you can consider:
Monitoring and Alerting: Implement a monitoring system to track node health and set up alerts when a node's health falls below a certain threshold (e.g., 90%). You can use Azure Monitor, Azure Log Analytics, or third-party tools like Prometheus and Grafana for monitoring. This way, you'll receive timely notifications to take corrective action.
For more details, please refer: Monitoring on Azure HDInsight Cluster health and availability
Cluster Sizing and Optimization: Review your cluster's configuration and optimize it to handle the workload more efficiently. This might involve adjusting the number of nodes, node types, or scaling up/down based on demand.
For additional information please refer: Automatically scale Azure HDInsight clusters
https://learn.microsoft.com/en-us/azure/hdinsight/cluster-management-best-practices
I hope this information helps. Please do let us know if you have any further queries.
Thank you.
-
Chandra Boorla 5,755 Reputation points Microsoft Vendor
2024-10-23T19:12:42.9733333+00:00 We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Thank you.
-
Chandra Boorla 5,755 Reputation points Microsoft Vendor
2024-10-24T19:54:07.18+00:00 Just checking in to see if the above response helped. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Thank you.
Sign in to comment
Sign in to answer