Manage clusters
Note
We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.
Only basic support will be available until the retirement date.
Important
This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.
Clusters are individual compute workloads such as Apache Spark, Apache Flink, and Trino, which can be created rapidly in few minutes with preset configurations and few clicks.
This article describes how to manage a cluster using Azure portal.
Note
You are required to have an operational cluster, Learn how to create a cluster.
Get started
Sign in to Azure portal.
In the Azure portal search bar, type "HDInsight on AKS clusters" and select "Azure HDInsight on AKS clusters" from the drop-down list.
Select your cluster name from the list page.
View cluster details
You can view the cluster details in the "Overview" blade of your cluster. It provides general information and easy access to the tools that are part of the cluster.
| Property | Description |
|---|---|
| Resource group | The resource group in which cluster is created. |
| Cluster pool name | Cluster pool name inside which the cluster is created. |
| Cluster type | The type of the cluster such as Spark, Trino, or Flink. |
| HDInsight on AKS version | HDInsight on AKS cluster version. For more information, see versioning. |
| Cluster endpoint | The endpoint of the cluster. |
| Cluster package | Component versions associated with the cluster. |
| Subscription details | Subscription name and subscription ID. |
| Location | Region in which the cluster is deployed. |
| Cluster size details | Node size, node type, and number of nodes. |
Manage cluster size
You can check and modify the number of worker nodes for your cluster using "Cluster size" blade in the Azure portal. There are two options to scale up/down your cluster:
Manage cluster access
HDInsight on AKS provides a comprehensive and fine-grained access control at both control plane and data plane, which allows you to manage cluster resources and provide access to cluster data plane.
Learn how to manage access to your cluster.
Configure secure shell (SSH)
Secure shell (SSH) allows you to submit jobs and queries to your cluster directly. You can enable or disable SSH using "Secure shell (SSH)" blade in the Azure portal.
Note
Enabling SSH will create additional VMs in the cluster. The maximum allowed secure shell nodes are 5.
Manage cluster configuration
HDInsight on AKS allows you to tweak the configuration properties to improve performance of your cluster with certain settings. For example, usage or memory settings. In the Azure portal, use "Configuration management" blade of your cluster to manage the configurations.
You can do the following actions:
- Update the existing service configurations or add new configurations.
- Export the service configurations using RestAPI.
Learn how to manage the cluster configuration.
View service details
In the Azure portal, use "Services" blade in your cluster to check the health of the services running in your cluster. It includes the collection of the services and the status of each service running in the cluster. You can drill down on each service to check instance level details.
Learn how to check service health.
Enable integration with Azure services
In the Azure portal, use "Integrations" blade in your cluster pool to configure the supported Azure services. Currently, we support Log Analytics and Azure managed Prometheus and Grafana, which has to be configured at cluster pool before you can enable at cluster level.
- Learn more about Azure Monitor Integration.
- For more information, see how to enable Log Analytics.
- For more information, see how to enable Azure managed Prometheus and Grafana.
Delete cluster
Deleting a cluster doesn't delete the default storage account nor any linked storage accounts. You can re-create the cluster by using the same storage accounts and the same metastores.
From the "Overview" blade in the Azure portal:
Select Delete from the top menu.
Status can be checked on Notification icon
.