Default and minimum virtual machine size recommendations and capacity planning for HDInsight on AKS
Note
We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.
Only basic support will be available until the retirement date.
Important
This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.
This article discusses default and recommended node configurations for Azure HDInsight on AKS clusters.
Cluster pools
For creating cluster pools, the currently available VM Options are F4s_v2, D4a_v4, D4as_v4, and E4s_v3.
Clusters
HDInsight on AKS currently supports Virtual Machines (VMs) from the Memory Optimized and General Purpose categories for cluster creation. Memory optimized VM sizes offer a high Memory-to-CPU ratio, which is great for relational database servers, medium to large caches, and in-memory analytics. General purpose VM sizes provide balanced Memory-to-CPU ratio and are ideal for testing and development, small to medium databases, and low to medium traffic web servers.
If your use case requires the usage of Memory Optimized VMs, the default recommendation is to use VMs from the families Eadsv5 or Easv5. For use cases requiring the usage of General Purpose VMs, we recommend VMs from the families Dadsv5 or Ddsv5.
If there is insufficient capacity/quota on the recommended VM Families, look for alternatives in the table that lists all the VM Families currently supported in HDInsight on AKS for cluster creation:
| Type | Virtual machine family | Temp storage | Premium storage support |
|---|---|---|---|
| Memory optimized | Eadsv5 | Yes | Yes |
| Easv5 | No | Yes | |
| Edsv5 | Yes | Yes | |
| Edv5 | Yes | No | |
| Easv4 | Yes | Yes | |
| Eav4 | Yes | No | |
| Edv4 | Yes | No | |
| General purpose | Dadsv5 | Yes | Yes |
| Ddsv5 | Yes | Yes | |
| Ddv5 | Yes | No | |
| Dasv4 | Yes | Yes | |
| Dav4 | Yes | No |
The minimum VM specifications recommended (regardless of the chosen VM Family for the cluster) is 8vCPUs and 32 GiB RAM. Higher vCPU and GiB variants may be chosen as per the workload being processed.
Capacity planning
The Virtual Machines used in HDInsight on AKS clusters requires the same Quota as Azure VMs. This is unlike the original version of HDInsight, where users had to request a dedicated quota to create HDInsight clusters by selecting 'Azure HDInsight' from the quota selection dropdown as shown in the image. For HDInsight on AKS, customers need to select ‘Compute’ from the Quota selection dropdown in order to request extra capacity for the VMs they intend to use in their clusters. Find detailed instructions for increasing your quota here.
