Microsoft.HDInsight clusters 2023-08-15-preview

Article
12/09/2024

Bicep resource definition

The clusters resource type can be deployed with operations that target:

Resource groups - See resource group deployment commands

For a list of changed properties in each API version, see change log.

Resource format

To create a Microsoft.HDInsight/clusters resource, add the following Bicep to your template.

resource symbolicname 'Microsoft.HDInsight/clusters@2023-08-15-preview' = {
  identity: {
    type: 'string'
    userAssignedIdentities: {
      {customized property}: {
        tenantId: 'string'
      }
    }
  }
  location: 'string'
  name: 'string'
  properties: {
    clusterDefinition: {
      blueprint: 'string'
      componentVersion: {
        {customized property}: 'string'
      }
      configurations: any(Azure.Bicep.Types.Concrete.AnyType)
      kind: 'string'
    }
    clusterVersion: 'string'
    computeIsolationProperties: {
      enableComputeIsolation: bool
      hostSku: 'string'
    }
    computeProfile: {
      roles: [
        {
          autoscale: {
            capacity: {
              maxInstanceCount: int
              minInstanceCount: int
            }
            recurrence: {
              schedule: [
                {
                  days: [
                    'string'
                  ]
                  timeAndCapacity: {
                    maxInstanceCount: int
                    minInstanceCount: int
                    time: 'string'
                  }
                }
              ]
              timeZone: 'string'
            }
          }
          dataDisksGroups: [
            {
              disksPerNode: int
            }
          ]
          encryptDataDisks: bool
          hardwareProfile: {
            vmSize: 'string'
          }
          minInstanceCount: int
          name: 'string'
          osProfile: {
            linuxOperatingSystemProfile: {
              password: 'string'
              sshProfile: {
                publicKeys: [
                  {
                    certificateData: 'string'
                  }
                ]
              }
              username: 'string'
            }
          }
          scriptActions: [
            {
              name: 'string'
              parameters: 'string'
              uri: 'string'
            }
          ]
          targetInstanceCount: int
          virtualNetworkProfile: {
            id: 'string'
            subnet: 'string'
          }
          VMGroupName: 'string'
        }
      ]
    }
    diskEncryptionProperties: {
      encryptionAlgorithm: 'string'
      encryptionAtHost: bool
      keyName: 'string'
      keyVersion: 'string'
      msiResourceId: 'string'
      vaultUri: 'string'
    }
    encryptionInTransitProperties: {
      isEncryptionInTransitEnabled: bool
    }
    kafkaRestProperties: {
      clientGroupInfo: {
        groupId: 'string'
        groupName: 'string'
      }
      configurationOverride: {
        {customized property}: 'string'
      }
    }
    minSupportedTlsVersion: 'string'
    networkProperties: {
      outboundDependenciesManagedType: 'string'
      privateLink: 'string'
      resourceProviderConnection: 'string'
    }
    osType: 'string'
    privateLinkConfigurations: [
      {
        name: 'string'
        properties: {
          groupId: 'string'
          ipConfigurations: [
            {
              name: 'string'
              properties: {
                primary: bool
                privateIPAddress: 'string'
                privateIPAllocationMethod: 'string'
                subnet: {
                  id: 'string'
                }
              }
            }
          ]
        }
      }
    ]
    securityProfile: {
      aaddsResourceId: 'string'
      clusterUsersGroupDNs: [
        'string'
      ]
      directoryType: 'string'
      domain: 'string'
      domainUsername: 'string'
      domainUserPassword: 'string'
      ldapsUrls: [
        'string'
      ]
      msiResourceId: 'string'
      organizationalUnitDN: 'string'
    }
    storageProfile: {
      storageaccounts: [
        {
          container: 'string'
          enableSecureChannel: bool
          fileshare: 'string'
          fileSystem: 'string'
          isDefault: bool
          key: 'string'
          msiResourceId: 'string'
          name: 'string'
          resourceId: 'string'
          saskey: 'string'
        }
      ]
    }
    tier: 'string'
  }
  tags: {
    {customized property}: 'string'
  }
  zones: [
    'string'
  ]
}

Property values

Autoscale

Name	Description	Value
capacity	Parameters for load-based autoscale	AutoscaleCapacity
recurrence	Parameters for schedule-based autoscale	AutoscaleRecurrence

AutoscaleCapacity

Name	Description	Value
maxInstanceCount	The maximum instance count of the cluster	int
minInstanceCount	The minimum instance count of the cluster	int

AutoscaleRecurrence

Name	Description	Value
schedule	Array of schedule-based autoscale rules	AutoscaleSchedule[]
timeZone	The time zone for the autoscale schedule times	string

AutoscaleSchedule

Name	Description	Value
days	Days of the week for a schedule-based autoscale rule	String array containing any of: 'Friday' 'Monday' 'Saturday' 'Sunday' 'Thursday' 'Tuesday' 'Wednesday'
timeAndCapacity	Time and capacity for a schedule-based autoscale rule	AutoscaleTimeAndCapacity

AutoscaleTimeAndCapacity

Name	Description	Value
maxInstanceCount	The maximum instance count of the cluster	int
minInstanceCount	The minimum instance count of the cluster	int
time	24-hour time in the form xx:xx	string

ClientGroupInfo

Name	Description	Value
groupId	The AAD security group id.	string
groupName	The AAD security group name.	string

ClusterCreateParametersExtendedTags

Name	Description	Value

ClusterCreatePropertiesOrClusterGetProperties

Name	Description	Value
clusterDefinition	The cluster definition.	ClusterDefinition
clusterVersion	The version of the cluster.	string
computeIsolationProperties	The compute isolation properties.	ComputeIsolationProperties
computeProfile	The compute profile.	ComputeProfile
diskEncryptionProperties	The disk encryption properties.	DiskEncryptionProperties
encryptionInTransitProperties	The encryption-in-transit properties.	EncryptionInTransitProperties
kafkaRestProperties	The cluster kafka rest proxy configuration.	KafkaRestProperties
minSupportedTlsVersion	The minimal supported tls version.	string
networkProperties	The network properties.	NetworkProperties
osType	The type of operating system.	'Linux' 'Windows'
privateLinkConfigurations	The private link configurations.	PrivateLinkConfiguration[]
securityProfile	The security profile.	SecurityProfile
storageProfile	The storage profile.	StorageProfile
tier	The cluster tier.	'Premium' 'Standard'

ClusterDefinition

Name	Description	Value
blueprint	The link to the blueprint.	string
componentVersion	The versions of different services in the cluster.	ClusterDefinitionComponentVersion
configurations	The cluster configurations.	any
kind	The type of cluster.	string

ClusterDefinitionComponentVersion

Name	Description	Value

ClusterIdentity

Name	Description	Value
type	The type of identity used for the cluster. The type 'SystemAssigned, UserAssigned' includes both an implicitly created identity and a set of user assigned identities.	'None' 'SystemAssigned' 'SystemAssigned, UserAssigned' 'UserAssigned'
userAssignedIdentities	The list of user identities associated with the cluster. The user identity dictionary key references will be ARM resource ids in the form: '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}'.	ClusterIdentityUserAssignedIdentities

ClusterIdentityUserAssignedIdentities

Name	Description	Value

ComputeIsolationProperties

Name	Description	Value
enableComputeIsolation	The flag indicates whether enable compute isolation or not.	bool
hostSku	The host sku.	string

ComputeProfile

Name	Description	Value
roles	The list of roles in the cluster.	Role[]

DataDisksGroups

Name	Description	Value
disksPerNode	The number of disks per node.	int

DiskEncryptionProperties

Name	Description	Value
encryptionAlgorithm	Algorithm identifier for encryption, default RSA-OAEP.	'RSA-OAEP' 'RSA-OAEP-256' 'RSA1_5'
encryptionAtHost	Indicates whether or not resource disk encryption is enabled.	bool
keyName	Key name that is used for enabling disk encryption.	string
keyVersion	Specific key version that is used for enabling disk encryption.	string
msiResourceId	Resource ID of Managed Identity that is used to access the key vault.	string
vaultUri	Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net	string

EncryptionInTransitProperties

Name	Description	Value
isEncryptionInTransitEnabled	Indicates whether or not inter cluster node communication is encrypted in transit.	bool

HardwareProfile

Name	Description	Value
vmSize	The size of the VM	string

IPConfiguration

Name	Description	Value
name	The name of private link IP configuration.	string (required)
properties	The private link ip configuration properties.	IPConfigurationProperties

IPConfigurationProperties

Name	Description	Value
primary	Indicates whether this IP configuration is primary for the corresponding NIC.	bool
privateIPAddress	The IP address.	string
privateIPAllocationMethod	The method that private IP address is allocated.	'dynamic' 'static'
subnet	The subnet resource id.	ResourceId

KafkaRestProperties

Name	Description	Value
clientGroupInfo	The information of AAD security group.	ClientGroupInfo
configurationOverride	The configurations that need to be overriden.	KafkaRestPropertiesConfigurationOverride

KafkaRestPropertiesConfigurationOverride

Name	Description	Value

LinuxOperatingSystemProfile

Name	Description	Value
password	The password.	string
sshProfile	The SSH profile.	SshProfile
username	The username.	string

Microsoft.HDInsight/clusters

Name	Description	Value
identity	The identity of the cluster, if configured.	ClusterIdentity
location	The location of the cluster.	string
name	The resource name	string (required)
properties	The cluster create parameters.	ClusterCreatePropertiesOrClusterGetProperties
tags	Resource tags	Dictionary of tag names and values. See Tags in templates
zones	The availability zones.	string[]

NetworkProperties

Name	Description	Value
outboundDependenciesManagedType	A value to describe how the outbound dependencies of a HDInsight cluster are managed. 'Managed' means that the outbound dependencies are managed by the HDInsight service. 'External' means that the outbound dependencies are managed by a customer specific solution.	'External' 'Managed'
privateLink	Indicates whether or not private link is enabled.	'Disabled' 'Enabled'
resourceProviderConnection	The direction for the resource provider connection.	'Inbound' 'Outbound'

OsProfile

Name	Description	Value
linuxOperatingSystemProfile	The Linux OS profile.	LinuxOperatingSystemProfile

PrivateLinkConfiguration

Name	Description	Value
name	The name of private link configuration.	string (required)
properties	The private link configuration properties.	PrivateLinkConfigurationProperties (required)

PrivateLinkConfigurationProperties

Name	Description	Value
groupId	The HDInsight private linkable sub-resource name to apply the private link configuration to. For example, 'headnode', 'gateway', 'edgenode'.	string (required)
ipConfigurations	The IP configurations for the private link service.	IPConfiguration[] (required)

ResourceId

Name	Description	Value
id	The azure resource id.	string

Role

Name	Description	Value
autoscale	The autoscale configurations.	Autoscale
dataDisksGroups	The data disks groups for the role.	DataDisksGroups[]
encryptDataDisks	Indicates whether encrypt the data disks.	bool
hardwareProfile	The hardware profile.	HardwareProfile
minInstanceCount	The minimum instance count of the cluster.	int
name	The name of the role.	string
osProfile	The operating system profile.	OsProfile
scriptActions	The list of script actions on the role.	ScriptAction[]
targetInstanceCount	The instance count of the cluster.	int
virtualNetworkProfile	The virtual network profile.	VirtualNetworkProfile
VMGroupName	The name of the virtual machine group.	string

ScriptAction

Name	Description	Value
name	The name of the script action.	string (required)
parameters	The parameters for the script provided.	string (required)
uri	The URI to the script.	string (required)

SecurityProfile

Name	Description	Value
aaddsResourceId	The resource ID of the user's Azure Active Directory Domain Service.	string
clusterUsersGroupDNs	Optional. The Distinguished Names for cluster user groups	string[]
directoryType	The directory type.	'ActiveDirectory'
domain	The organization's active directory domain.	string
domainUsername	The domain user account that will have admin privileges on the cluster.	string
domainUserPassword	The domain admin password.	string
ldapsUrls	The LDAPS protocol URLs to communicate with the Active Directory.	string[]
msiResourceId	User assigned identity that has permissions to read and create cluster-related artifacts in the user's AADDS.	string
organizationalUnitDN	The organizational unit within the Active Directory to place the cluster and service accounts.	string

SshProfile

Name	Description	Value
publicKeys	The list of SSH public keys.	SshPublicKey[]

SshPublicKey

Name	Description	Value
certificateData	The certificate for SSH.	string

StorageAccount

Name	Description	Value
container	The container in the storage account, only to be specified for WASB storage accounts.	string
enableSecureChannel	Enable secure channel or not, it's an optional field. Default value is false when cluster version < 5.1 and true when cluster version >= 5.1 ,	bool
fileshare	The file share name.	string
fileSystem	The filesystem, only to be specified for Azure Data Lake Storage Gen 2.	string
isDefault	Whether or not the storage account is the default storage account.	bool
key	The storage account access key.	string
msiResourceId	The managed identity (MSI) that is allowed to access the storage account, only to be specified for Azure Data Lake Storage Gen 2.	string
name	The name of the storage account.	string
resourceId	The resource ID of storage account, only to be specified for Azure Data Lake Storage Gen 2.	string
saskey	The shared access signature key.	string

StorageProfile

Name	Description	Value
storageaccounts	The list of storage accounts in the cluster.	StorageAccount[]

UserAssignedIdentity

Name	Description	Value
tenantId	The tenant id of user assigned identity.	string

VirtualNetworkProfile

Name	Description	Value
id	The ID of the virtual network.	string
subnet	The name of the subnet.	string

Quickstart samples

The following quickstart samples deploy this resource type.

Bicep File	Description
Deploy a Linux-based HBase cluster in HDInsight	This template allows you to create a Linux-based HBase cluster in Azure HDInsight.
Deploy a Spark cluster in Azure HDInsight	This template allows you to create a Spark cluster in Azure HDInsight.
Deploy an Interative Hive cluster in HDInsight	This template allows you to create an Interative Hive (LLAP) cluster in HDInsight and the dependent Azure Storage account. The SSH authentication method for the cluster is username and password. For a template using SSH public key authentication, see /samples/azure/azure-quickstart-templates/hdinsight-linux-ssh-publickey
Deploy HDInsight cluster with Storage and SSH password	This template allows you to create a Linux-based Hadoop cluster in HDInsight and the dependent Azure Storage account. The SSH authentication method for the cluster is username and password. For a template using SSH public key authentication, see /samples/azure/azure-quickstart-templates/hdinsight-linux-ssh-publickey/
Deploy Kafka on HDInsight in a virtual network	This template allows you to create an Azure Virtual Network and a Kafka on HDInsight cluster in the virtual network. The SSH authentication method for the cluster is username and password. For a template using SSH public key authentication, see /samples/azure/azure-quickstart-templates/hdinsight-linux-ssh-publickey/

ARM template resource definition

The clusters resource type can be deployed with operations that target:

Resource groups - See resource group deployment commands

For a list of changed properties in each API version, see change log.

Resource format

To create a Microsoft.HDInsight/clusters resource, add the following JSON to your template.

{
  "type": "Microsoft.HDInsight/clusters",
  "apiVersion": "2023-08-15-preview",
  "name": "string",
  "identity": {
    "type": "string",
    "userAssignedIdentities": {
      "{customized property}": {
        "tenantId": "string"
      }
    }
  },
  "location": "string",
  "properties": {
    "clusterDefinition": {
      "blueprint": "string",
      "componentVersion": {
        "{customized property}": "string"
      },
      "configurations": {},
      "kind": "string"
    },
    "clusterVersion": "string",
    "computeIsolationProperties": {
      "enableComputeIsolation": "bool",
      "hostSku": "string"
    },
    "computeProfile": {
      "roles": [
        {
          "autoscale": {
            "capacity": {
              "maxInstanceCount": "int",
              "minInstanceCount": "int"
            },
            "recurrence": {
              "schedule": [
                {
                  "days": [ "string" ],
                  "timeAndCapacity": {
                    "maxInstanceCount": "int",
                    "minInstanceCount": "int",
                    "time": "string"
                  }
                }
              ],
              "timeZone": "string"
            }
          },
          "dataDisksGroups": [
            {
              "disksPerNode": "int"
            }
          ],
          "encryptDataDisks": "bool",
          "hardwareProfile": {
            "vmSize": "string"
          },
          "minInstanceCount": "int",
          "name": "string",
          "osProfile": {
            "linuxOperatingSystemProfile": {
              "password": "string",
              "sshProfile": {
                "publicKeys": [
                  {
                    "certificateData": "string"
                  }
                ]
              },
              "username": "string"
            }
          },
          "scriptActions": [
            {
              "name": "string",
              "parameters": "string",
              "uri": "string"
            }
          ],
          "targetInstanceCount": "int",
          "virtualNetworkProfile": {
            "id": "string",
            "subnet": "string"
          },
          "VMGroupName": "string"
        }
      ]
    },
    "diskEncryptionProperties": {
      "encryptionAlgorithm": "string",
      "encryptionAtHost": "bool",
      "keyName": "string",
      "keyVersion": "string",
      "msiResourceId": "string",
      "vaultUri": "string"
    },
    "encryptionInTransitProperties": {
      "isEncryptionInTransitEnabled": "bool"
    },
    "kafkaRestProperties": {
      "clientGroupInfo": {
        "groupId": "string",
        "groupName": "string"
      },
      "configurationOverride": {
        "{customized property}": "string"
      }
    },
    "minSupportedTlsVersion": "string",
    "networkProperties": {
      "outboundDependenciesManagedType": "string",
      "privateLink": "string",
      "resourceProviderConnection": "string"
    },
    "osType": "string",
    "privateLinkConfigurations": [
      {
        "name": "string",
        "properties": {
          "groupId": "string",
          "ipConfigurations": [
            {
              "name": "string",
              "properties": {
                "primary": "bool",
                "privateIPAddress": "string",
                "privateIPAllocationMethod": "string",
                "subnet": {
                  "id": "string"
                }
              }
            }
          ]
        }
      }
    ],
    "securityProfile": {
      "aaddsResourceId": "string",
      "clusterUsersGroupDNs": [ "string" ],
      "directoryType": "string",
      "domain": "string",
      "domainUsername": "string",
      "domainUserPassword": "string",
      "ldapsUrls": [ "string" ],
      "msiResourceId": "string",
      "organizationalUnitDN": "string"
    },
    "storageProfile": {
      "storageaccounts": [
        {
          "container": "string",
          "enableSecureChannel": "bool",
          "fileshare": "string",
          "fileSystem": "string",
          "isDefault": "bool",
          "key": "string",
          "msiResourceId": "string",
          "name": "string",
          "resourceId": "string",
          "saskey": "string"
        }
      ]
    },
    "tier": "string"
  },
  "tags": {
    "{customized property}": "string"
  },
  "zones": [ "string" ]
}

Property values

Autoscale

Name	Description	Value
capacity	Parameters for load-based autoscale	AutoscaleCapacity
recurrence	Parameters for schedule-based autoscale	AutoscaleRecurrence

AutoscaleCapacity

Name	Description	Value
maxInstanceCount	The maximum instance count of the cluster	int
minInstanceCount	The minimum instance count of the cluster	int

AutoscaleRecurrence

Name	Description	Value
schedule	Array of schedule-based autoscale rules	AutoscaleSchedule[]
timeZone	The time zone for the autoscale schedule times	string

AutoscaleSchedule

Name	Description	Value
days	Days of the week for a schedule-based autoscale rule	String array containing any of: 'Friday' 'Monday' 'Saturday' 'Sunday' 'Thursday' 'Tuesday' 'Wednesday'
timeAndCapacity	Time and capacity for a schedule-based autoscale rule	AutoscaleTimeAndCapacity

AutoscaleTimeAndCapacity

Name	Description	Value
maxInstanceCount	The maximum instance count of the cluster	int
minInstanceCount	The minimum instance count of the cluster	int
time	24-hour time in the form xx:xx	string

ClientGroupInfo

Name	Description	Value
groupId	The AAD security group id.	string
groupName	The AAD security group name.	string

ClusterCreateParametersExtendedTags

Name	Description	Value

ClusterCreatePropertiesOrClusterGetProperties

Name	Description	Value
clusterDefinition	The cluster definition.	ClusterDefinition
clusterVersion	The version of the cluster.	string
computeIsolationProperties	The compute isolation properties.	ComputeIsolationProperties
computeProfile	The compute profile.	ComputeProfile
diskEncryptionProperties	The disk encryption properties.	DiskEncryptionProperties
encryptionInTransitProperties	The encryption-in-transit properties.	EncryptionInTransitProperties
kafkaRestProperties	The cluster kafka rest proxy configuration.	KafkaRestProperties
minSupportedTlsVersion	The minimal supported tls version.	string
networkProperties	The network properties.	NetworkProperties
osType	The type of operating system.	'Linux' 'Windows'
privateLinkConfigurations	The private link configurations.	PrivateLinkConfiguration[]
securityProfile	The security profile.	SecurityProfile
storageProfile	The storage profile.	StorageProfile
tier	The cluster tier.	'Premium' 'Standard'

ClusterDefinition

Name	Description	Value
blueprint	The link to the blueprint.	string
componentVersion	The versions of different services in the cluster.	ClusterDefinitionComponentVersion
configurations	The cluster configurations.	any
kind	The type of cluster.	string

ClusterDefinitionComponentVersion

Name	Description	Value

ClusterIdentity

Name	Description	Value
type	The type of identity used for the cluster. The type 'SystemAssigned, UserAssigned' includes both an implicitly created identity and a set of user assigned identities.	'None' 'SystemAssigned' 'SystemAssigned, UserAssigned' 'UserAssigned'
userAssignedIdentities	The list of user identities associated with the cluster. The user identity dictionary key references will be ARM resource ids in the form: '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}'.	ClusterIdentityUserAssignedIdentities

ClusterIdentityUserAssignedIdentities

Name	Description	Value

ComputeIsolationProperties

Name	Description	Value
enableComputeIsolation	The flag indicates whether enable compute isolation or not.	bool
hostSku	The host sku.	string

ComputeProfile

Name	Description	Value
roles	The list of roles in the cluster.	Role[]

DataDisksGroups

Name	Description	Value
disksPerNode	The number of disks per node.	int

DiskEncryptionProperties

Name	Description	Value
encryptionAlgorithm	Algorithm identifier for encryption, default RSA-OAEP.	'RSA-OAEP' 'RSA-OAEP-256' 'RSA1_5'
encryptionAtHost	Indicates whether or not resource disk encryption is enabled.	bool
keyName	Key name that is used for enabling disk encryption.	string
keyVersion	Specific key version that is used for enabling disk encryption.	string
msiResourceId	Resource ID of Managed Identity that is used to access the key vault.	string
vaultUri	Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net	string

EncryptionInTransitProperties

Name	Description	Value
isEncryptionInTransitEnabled	Indicates whether or not inter cluster node communication is encrypted in transit.	bool

HardwareProfile

Name	Description	Value
vmSize	The size of the VM	string

IPConfiguration

Name	Description	Value
name	The name of private link IP configuration.	string (required)
properties	The private link ip configuration properties.	IPConfigurationProperties

IPConfigurationProperties

Name	Description	Value
primary	Indicates whether this IP configuration is primary for the corresponding NIC.	bool
privateIPAddress	The IP address.	string
privateIPAllocationMethod	The method that private IP address is allocated.	'dynamic' 'static'
subnet	The subnet resource id.	ResourceId

KafkaRestProperties

Name	Description	Value
clientGroupInfo	The information of AAD security group.	ClientGroupInfo
configurationOverride	The configurations that need to be overriden.	KafkaRestPropertiesConfigurationOverride

KafkaRestPropertiesConfigurationOverride

Name	Description	Value

LinuxOperatingSystemProfile

Name	Description	Value
password	The password.	string
sshProfile	The SSH profile.	SshProfile
username	The username.	string

Microsoft.HDInsight/clusters

Name	Description	Value
apiVersion	The api version	'2023-08-15-preview'
identity	The identity of the cluster, if configured.	ClusterIdentity
location	The location of the cluster.	string
name	The resource name	string (required)
properties	The cluster create parameters.	ClusterCreatePropertiesOrClusterGetProperties
tags	Resource tags	Dictionary of tag names and values. See Tags in templates
type	The resource type	'Microsoft.HDInsight/clusters'
zones	The availability zones.	string[]

NetworkProperties

Name	Description	Value
outboundDependenciesManagedType	A value to describe how the outbound dependencies of a HDInsight cluster are managed. 'Managed' means that the outbound dependencies are managed by the HDInsight service. 'External' means that the outbound dependencies are managed by a customer specific solution.	'External' 'Managed'
privateLink	Indicates whether or not private link is enabled.	'Disabled' 'Enabled'
resourceProviderConnection	The direction for the resource provider connection.	'Inbound' 'Outbound'

OsProfile

Name	Description	Value
linuxOperatingSystemProfile	The Linux OS profile.	LinuxOperatingSystemProfile

PrivateLinkConfiguration

Name	Description	Value
name	The name of private link configuration.	string (required)
properties	The private link configuration properties.	PrivateLinkConfigurationProperties (required)

PrivateLinkConfigurationProperties

Name	Description	Value
groupId	The HDInsight private linkable sub-resource name to apply the private link configuration to. For example, 'headnode', 'gateway', 'edgenode'.	string (required)
ipConfigurations	The IP configurations for the private link service.	IPConfiguration[] (required)

ResourceId

Name	Description	Value
id	The azure resource id.	string

Role

Name	Description	Value
autoscale	The autoscale configurations.	Autoscale
dataDisksGroups	The data disks groups for the role.	DataDisksGroups[]
encryptDataDisks	Indicates whether encrypt the data disks.	bool
hardwareProfile	The hardware profile.	HardwareProfile
minInstanceCount	The minimum instance count of the cluster.	int
name	The name of the role.	string
osProfile	The operating system profile.	OsProfile
scriptActions	The list of script actions on the role.	ScriptAction[]
targetInstanceCount	The instance count of the cluster.	int
virtualNetworkProfile	The virtual network profile.	VirtualNetworkProfile
VMGroupName	The name of the virtual machine group.	string

ScriptAction

Name	Description	Value
name	The name of the script action.	string (required)
parameters	The parameters for the script provided.	string (required)
uri	The URI to the script.	string (required)

SecurityProfile

Name	Description	Value
aaddsResourceId	The resource ID of the user's Azure Active Directory Domain Service.	string
clusterUsersGroupDNs	Optional. The Distinguished Names for cluster user groups	string[]
directoryType	The directory type.	'ActiveDirectory'
domain	The organization's active directory domain.	string
domainUsername	The domain user account that will have admin privileges on the cluster.	string
domainUserPassword	The domain admin password.	string
ldapsUrls	The LDAPS protocol URLs to communicate with the Active Directory.	string[]
msiResourceId	User assigned identity that has permissions to read and create cluster-related artifacts in the user's AADDS.	string
organizationalUnitDN	The organizational unit within the Active Directory to place the cluster and service accounts.	string

SshProfile

Name	Description	Value
publicKeys	The list of SSH public keys.	SshPublicKey[]

SshPublicKey

Name	Description	Value
certificateData	The certificate for SSH.	string

StorageAccount

Name	Description	Value
container	The container in the storage account, only to be specified for WASB storage accounts.	string
enableSecureChannel	Enable secure channel or not, it's an optional field. Default value is false when cluster version < 5.1 and true when cluster version >= 5.1 ,	bool
fileshare	The file share name.	string
fileSystem	The filesystem, only to be specified for Azure Data Lake Storage Gen 2.	string
isDefault	Whether or not the storage account is the default storage account.	bool
key	The storage account access key.	string
msiResourceId	The managed identity (MSI) that is allowed to access the storage account, only to be specified for Azure Data Lake Storage Gen 2.	string
name	The name of the storage account.	string
resourceId	The resource ID of storage account, only to be specified for Azure Data Lake Storage Gen 2.	string
saskey	The shared access signature key.	string

StorageProfile

Name	Description	Value
storageaccounts	The list of storage accounts in the cluster.	StorageAccount[]

UserAssignedIdentity

Name	Description	Value
tenantId	The tenant id of user assigned identity.	string

VirtualNetworkProfile

Name	Description	Value
id	The ID of the virtual network.	string
subnet	The name of the subnet.	string

Quickstart templates

The following quickstart templates deploy this resource type.

Template	Description
Create HDInsight Linux Cluster and run a script action	Template creates an HDInsight Linux cluster in a virtual network and then runs a custom script action on every node and sets environment var.
Creates an HDInsight cluster running ADAM	Creates an HDInsight linux cluster running the genomics analysis platform ADAM
Creates an HDInsight cluster running Apache Spark 1.4.1	Creates an HDInsight linux cluster running Apache Spark 1.4.1.
Deploy a HDInsight cluster and a SQL database	This template allows you to create a HDInsight cluster and a SQL Database for testing Sqoop.
Deploy a HDInsight cluster with an edge node	This template allows you to create an HDInsight cluster running Linux with an empty edge node. For more information, see /azure/hdinsight/hdinsight-apps-use-edge-node
Deploy a Linux-based HBase cluster in HDInsight	This template allows you to create a Linux-based HBase cluster in Azure HDInsight.
Deploy a secure VNet and a HDInsight cluster within the VNet	This template allows you to create an Azure VNet and an HDInsight Hadoop cluster running Linux within the VNet.
Deploy a Spark cluster in a VNet	This template allows you to create an Azure VNet and an HDInsight Spark cluster within the VNet.
Deploy a Spark cluster in Azure HDInsight	This template allows you to create a Spark cluster in Azure HDInsight.
Deploy a VNet, and a HBase cluster within the VNet	This template allows you to create an Azure VNet and an HDInsight HBase cluster running Linux within the VNet.
Deploy an Azure VNet and two HBase clusters within the VNet	This template allows you to configure an HBase environment with two HBase clusters within a VNet for configuring HBase replication.
Deploy an Interative Hive cluster in HDInsight	This template allows you to create an Interative Hive (LLAP) cluster in HDInsight and the dependent Azure Storage account. The SSH authentication method for the cluster is username and password. For a template using SSH public key authentication, see /samples/azure/azure-quickstart-templates/hdinsight-linux-ssh-publickey
Deploy an R-server HDInsight cluster	This template allows you to create an HDInsight cluster running Linux with R Server for HDInsight. This template also creates an Azure Storage account. The SSH authentication method for the cluster is username / password.
Deploy HBase replication with two VNets in one region	This template allows you to configure aN HBase environment with two HBase clusters within two VNets in the same region for configuring HBase replication.
Deploy HDInsight cluster + Confluent Schema Registry node	This template allows you to create an HDInsight cluster running Linux with a schema registry edge node. For more information, see /azure/hdinsight/hdinsight-apps-use-edge-node
Deploy HDInsight cluster with existing default storage	This template allows you to create an Hadoop cluster in HDInsight. The cluster uses an existing storage account as the default storage accout.
Deploy HDInsight cluster with existing linked storage	This template allows you to create an Hadoop cluster in HDInsight and the dependent default storage account. The template also links an existing storage account. The linked storage account usually contains the business data.
Deploy HDInsight cluster with Storage and SSH password	This template allows you to create a Linux-based Hadoop cluster in HDInsight and the dependent Azure Storage account. The SSH authentication method for the cluster is username and password. For a template using SSH public key authentication, see /samples/azure/azure-quickstart-templates/hdinsight-linux-ssh-publickey/
Deploy HDInsight on Linux (w/ Azure Storage, SSH key)	This template allows you to create an HDInsight cluster running Linux. This template also creates an Azure Storage account. The SSH authentication method for the cluster is username / public key.
Deploy HDInsight on new Data Lake Store and Storage	This template allows you to deploy a new Linux HDInsight cluster with new Data Lake Store and Storage accounts.
Deploy Kafka on HDInsight in a virtual network	This template allows you to create an Azure Virtual Network and a Kafka on HDInsight cluster in the virtual network. The SSH authentication method for the cluster is username and password. For a template using SSH public key authentication, see /samples/azure/azure-quickstart-templates/hdinsight-linux-ssh-publickey/
Deploy Linux HBase cluster with enhanced writes in HDInsight	This template allows you to create a Linux-based HBase cluster with enhanced writes in Azure HDInsight.
HDInsight (Linux on existing Hive metastore, SSH, vnet)	This template allows you to create an HDInsight cluster running Linux, on an existing Hive metastore and virtual network. The SSH authentication method for the cluster is username / password.
HDInsight cluster with TLS version 1.2 or newer	This template allows you to create a HDInsight cluster with TLS protocol enforced to 1.2 or newer.
HDInsight with custom Ambari + Hive Metastore DB in VNET	This template allows you to create an HDInsight cluster in an existing virtual network with a new SQL DB that serves as both a custom Ambari DB and Hive Metastore. You must have an existing SQL Sever, storage account, and VNET.
HDInsight with Load-based Autoscale Enabled	This template allows you to create an HDInsight Spark cluster with load-based Autoscale enabled.
HDInsight with schedule-based Autoscale Enabled	This template allows you to create an HDInsight Spark cluster with schedule-based Autoscale enabled.

Terraform (AzAPI provider) resource definition

The clusters resource type can be deployed with operations that target:

Resource groups

For a list of changed properties in each API version, see change log.

Resource format

To create a Microsoft.HDInsight/clusters resource, add the following Terraform to your template.

resource "azapi_resource" "symbolicname" {
  type = "Microsoft.HDInsight/clusters@2023-08-15-preview"
  name = "string"
  identity = {
    type = "string"
    userAssignedIdentities = {
      {customized property} = {
        tenantId = "string"
      }
    }
  }
  location = "string"
  tags = {
    {customized property} = "string"
  }
  zones = [
    "string"
  ]
  body = jsonencode({
    properties = {
      clusterDefinition = {
        blueprint = "string"
        componentVersion = {
          {customized property} = "string"
        }
        configurations = ?
        kind = "string"
      }
      clusterVersion = "string"
      computeIsolationProperties = {
        enableComputeIsolation = bool
        hostSku = "string"
      }
      computeProfile = {
        roles = [
          {
            autoscale = {
              capacity = {
                maxInstanceCount = int
                minInstanceCount = int
              }
              recurrence = {
                schedule = [
                  {
                    days = [
                      "string"
                    ]
                    timeAndCapacity = {
                      maxInstanceCount = int
                      minInstanceCount = int
                      time = "string"
                    }
                  }
                ]
                timeZone = "string"
              }
            }
            dataDisksGroups = [
              {
                disksPerNode = int
              }
            ]
            encryptDataDisks = bool
            hardwareProfile = {
              vmSize = "string"
            }
            minInstanceCount = int
            name = "string"
            osProfile = {
              linuxOperatingSystemProfile = {
                password = "string"
                sshProfile = {
                  publicKeys = [
                    {
                      certificateData = "string"
                    }
                  ]
                }
                username = "string"
              }
            }
            scriptActions = [
              {
                name = "string"
                parameters = "string"
                uri = "string"
              }
            ]
            targetInstanceCount = int
            virtualNetworkProfile = {
              id = "string"
              subnet = "string"
            }
            VMGroupName = "string"
          }
        ]
      }
      diskEncryptionProperties = {
        encryptionAlgorithm = "string"
        encryptionAtHost = bool
        keyName = "string"
        keyVersion = "string"
        msiResourceId = "string"
        vaultUri = "string"
      }
      encryptionInTransitProperties = {
        isEncryptionInTransitEnabled = bool
      }
      kafkaRestProperties = {
        clientGroupInfo = {
          groupId = "string"
          groupName = "string"
        }
        configurationOverride = {
          {customized property} = "string"
        }
      }
      minSupportedTlsVersion = "string"
      networkProperties = {
        outboundDependenciesManagedType = "string"
        privateLink = "string"
        resourceProviderConnection = "string"
      }
      osType = "string"
      privateLinkConfigurations = [
        {
          name = "string"
          properties = {
            groupId = "string"
            ipConfigurations = [
              {
                name = "string"
                properties = {
                  primary = bool
                  privateIPAddress = "string"
                  privateIPAllocationMethod = "string"
                  subnet = {
                    id = "string"
                  }
                }
              }
            ]
          }
        }
      ]
      securityProfile = {
        aaddsResourceId = "string"
        clusterUsersGroupDNs = [
          "string"
        ]
        directoryType = "string"
        domain = "string"
        domainUsername = "string"
        domainUserPassword = "string"
        ldapsUrls = [
          "string"
        ]
        msiResourceId = "string"
        organizationalUnitDN = "string"
      }
      storageProfile = {
        storageaccounts = [
          {
            container = "string"
            enableSecureChannel = bool
            fileshare = "string"
            fileSystem = "string"
            isDefault = bool
            key = "string"
            msiResourceId = "string"
            name = "string"
            resourceId = "string"
            saskey = "string"
          }
        ]
      }
      tier = "string"
    }
  })
}

Property values

Autoscale

Name	Description	Value
capacity	Parameters for load-based autoscale	AutoscaleCapacity
recurrence	Parameters for schedule-based autoscale	AutoscaleRecurrence

AutoscaleCapacity

Name	Description	Value
maxInstanceCount	The maximum instance count of the cluster	int
minInstanceCount	The minimum instance count of the cluster	int

AutoscaleRecurrence

Name	Description	Value
schedule	Array of schedule-based autoscale rules	AutoscaleSchedule[]
timeZone	The time zone for the autoscale schedule times	string

AutoscaleSchedule

Name	Description	Value
days	Days of the week for a schedule-based autoscale rule	String array containing any of: 'Friday' 'Monday' 'Saturday' 'Sunday' 'Thursday' 'Tuesday' 'Wednesday'
timeAndCapacity	Time and capacity for a schedule-based autoscale rule	AutoscaleTimeAndCapacity

AutoscaleTimeAndCapacity

Name	Description	Value
maxInstanceCount	The maximum instance count of the cluster	int
minInstanceCount	The minimum instance count of the cluster	int
time	24-hour time in the form xx:xx	string

ClientGroupInfo

Name	Description	Value
groupId	The AAD security group id.	string
groupName	The AAD security group name.	string

ClusterCreateParametersExtendedTags

Name	Description	Value

ClusterCreatePropertiesOrClusterGetProperties

Name	Description	Value
clusterDefinition	The cluster definition.	ClusterDefinition
clusterVersion	The version of the cluster.	string
computeIsolationProperties	The compute isolation properties.	ComputeIsolationProperties
computeProfile	The compute profile.	ComputeProfile
diskEncryptionProperties	The disk encryption properties.	DiskEncryptionProperties
encryptionInTransitProperties	The encryption-in-transit properties.	EncryptionInTransitProperties
kafkaRestProperties	The cluster kafka rest proxy configuration.	KafkaRestProperties
minSupportedTlsVersion	The minimal supported tls version.	string
networkProperties	The network properties.	NetworkProperties
osType	The type of operating system.	'Linux' 'Windows'
privateLinkConfigurations	The private link configurations.	PrivateLinkConfiguration[]
securityProfile	The security profile.	SecurityProfile
storageProfile	The storage profile.	StorageProfile
tier	The cluster tier.	'Premium' 'Standard'

ClusterDefinition

Name	Description	Value
blueprint	The link to the blueprint.	string
componentVersion	The versions of different services in the cluster.	ClusterDefinitionComponentVersion
configurations	The cluster configurations.	any
kind	The type of cluster.	string

ClusterDefinitionComponentVersion

Name	Description	Value

ClusterIdentity

Name	Description	Value
type	The type of identity used for the cluster. The type 'SystemAssigned, UserAssigned' includes both an implicitly created identity and a set of user assigned identities.	'None' 'SystemAssigned' 'SystemAssigned, UserAssigned' 'UserAssigned'
userAssignedIdentities	The list of user identities associated with the cluster. The user identity dictionary key references will be ARM resource ids in the form: '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}'.	ClusterIdentityUserAssignedIdentities

ClusterIdentityUserAssignedIdentities

Name	Description	Value

ComputeIsolationProperties

Name	Description	Value
enableComputeIsolation	The flag indicates whether enable compute isolation or not.	bool
hostSku	The host sku.	string

ComputeProfile

Name	Description	Value
roles	The list of roles in the cluster.	Role[]

DataDisksGroups

Name	Description	Value
disksPerNode	The number of disks per node.	int

DiskEncryptionProperties

Name	Description	Value
encryptionAlgorithm	Algorithm identifier for encryption, default RSA-OAEP.	'RSA-OAEP' 'RSA-OAEP-256' 'RSA1_5'
encryptionAtHost	Indicates whether or not resource disk encryption is enabled.	bool
keyName	Key name that is used for enabling disk encryption.	string
keyVersion	Specific key version that is used for enabling disk encryption.	string
msiResourceId	Resource ID of Managed Identity that is used to access the key vault.	string
vaultUri	Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net	string

EncryptionInTransitProperties

Name	Description	Value
isEncryptionInTransitEnabled	Indicates whether or not inter cluster node communication is encrypted in transit.	bool

HardwareProfile

Name	Description	Value
vmSize	The size of the VM	string

IPConfiguration

Name	Description	Value
name	The name of private link IP configuration.	string (required)
properties	The private link ip configuration properties.	IPConfigurationProperties

IPConfigurationProperties

Name	Description	Value
primary	Indicates whether this IP configuration is primary for the corresponding NIC.	bool
privateIPAddress	The IP address.	string
privateIPAllocationMethod	The method that private IP address is allocated.	'dynamic' 'static'
subnet	The subnet resource id.	ResourceId

KafkaRestProperties

Name	Description	Value
clientGroupInfo	The information of AAD security group.	ClientGroupInfo
configurationOverride	The configurations that need to be overriden.	KafkaRestPropertiesConfigurationOverride

KafkaRestPropertiesConfigurationOverride

Name	Description	Value

LinuxOperatingSystemProfile

Name	Description	Value
password	The password.	string
sshProfile	The SSH profile.	SshProfile
username	The username.	string

Microsoft.HDInsight/clusters

Name	Description	Value
identity	The identity of the cluster, if configured.	ClusterIdentity
location	The location of the cluster.	string
name	The resource name	string (required)
properties	The cluster create parameters.	ClusterCreatePropertiesOrClusterGetProperties
tags	Resource tags	Dictionary of tag names and values.
type	The resource type	"Microsoft.HDInsight/clusters@2023-08-15-preview"
zones	The availability zones.	string[]

NetworkProperties

Name	Description	Value
outboundDependenciesManagedType	A value to describe how the outbound dependencies of a HDInsight cluster are managed. 'Managed' means that the outbound dependencies are managed by the HDInsight service. 'External' means that the outbound dependencies are managed by a customer specific solution.	'External' 'Managed'
privateLink	Indicates whether or not private link is enabled.	'Disabled' 'Enabled'
resourceProviderConnection	The direction for the resource provider connection.	'Inbound' 'Outbound'

OsProfile

Name	Description	Value
linuxOperatingSystemProfile	The Linux OS profile.	LinuxOperatingSystemProfile

PrivateLinkConfiguration

Name	Description	Value
name	The name of private link configuration.	string (required)
properties	The private link configuration properties.	PrivateLinkConfigurationProperties (required)

PrivateLinkConfigurationProperties

Name	Description	Value
groupId	The HDInsight private linkable sub-resource name to apply the private link configuration to. For example, 'headnode', 'gateway', 'edgenode'.	string (required)
ipConfigurations	The IP configurations for the private link service.	IPConfiguration[] (required)

ResourceId

Name	Description	Value
id	The azure resource id.	string

Role

Name	Description	Value
autoscale	The autoscale configurations.	Autoscale
dataDisksGroups	The data disks groups for the role.	DataDisksGroups[]
encryptDataDisks	Indicates whether encrypt the data disks.	bool
hardwareProfile	The hardware profile.	HardwareProfile
minInstanceCount	The minimum instance count of the cluster.	int
name	The name of the role.	string
osProfile	The operating system profile.	OsProfile
scriptActions	The list of script actions on the role.	ScriptAction[]
targetInstanceCount	The instance count of the cluster.	int
virtualNetworkProfile	The virtual network profile.	VirtualNetworkProfile
VMGroupName	The name of the virtual machine group.	string

ScriptAction

Name	Description	Value
name	The name of the script action.	string (required)
parameters	The parameters for the script provided.	string (required)
uri	The URI to the script.	string (required)

SecurityProfile

Name	Description	Value
aaddsResourceId	The resource ID of the user's Azure Active Directory Domain Service.	string
clusterUsersGroupDNs	Optional. The Distinguished Names for cluster user groups	string[]
directoryType	The directory type.	'ActiveDirectory'
domain	The organization's active directory domain.	string
domainUsername	The domain user account that will have admin privileges on the cluster.	string
domainUserPassword	The domain admin password.	string
ldapsUrls	The LDAPS protocol URLs to communicate with the Active Directory.	string[]
msiResourceId	User assigned identity that has permissions to read and create cluster-related artifacts in the user's AADDS.	string
organizationalUnitDN	The organizational unit within the Active Directory to place the cluster and service accounts.	string

SshProfile

Name	Description	Value
publicKeys	The list of SSH public keys.	SshPublicKey[]

SshPublicKey

Name	Description	Value
certificateData	The certificate for SSH.	string

StorageAccount

Name	Description	Value
container	The container in the storage account, only to be specified for WASB storage accounts.	string
enableSecureChannel	Enable secure channel or not, it's an optional field. Default value is false when cluster version < 5.1 and true when cluster version >= 5.1 ,	bool
fileshare	The file share name.	string
fileSystem	The filesystem, only to be specified for Azure Data Lake Storage Gen 2.	string
isDefault	Whether or not the storage account is the default storage account.	bool
key	The storage account access key.	string
msiResourceId	The managed identity (MSI) that is allowed to access the storage account, only to be specified for Azure Data Lake Storage Gen 2.	string
name	The name of the storage account.	string
resourceId	The resource ID of storage account, only to be specified for Azure Data Lake Storage Gen 2.	string
saskey	The shared access signature key.	string

StorageProfile

Name	Description	Value
storageaccounts	The list of storage accounts in the cluster.	StorageAccount[]

UserAssignedIdentity

Name	Description	Value
tenantId	The tenant id of user assigned identity.	string

VirtualNetworkProfile

Name	Description	Value
id	The ID of the virtual network.	string
subnet	The name of the subnet.	string

Share via

Microsoft.HDInsight clusters 2023-08-15-preview

Bicep resource definition

Resource format

Property values

Autoscale

AutoscaleCapacity

AutoscaleRecurrence

AutoscaleSchedule

AutoscaleTimeAndCapacity

ClientGroupInfo

ClusterCreateParametersExtendedTags

ClusterCreatePropertiesOrClusterGetProperties

ClusterDefinition

ClusterDefinitionComponentVersion

ClusterIdentity

ClusterIdentityUserAssignedIdentities

ComputeIsolationProperties

ComputeProfile

DataDisksGroups

DiskEncryptionProperties

EncryptionInTransitProperties

HardwareProfile

IPConfiguration

IPConfigurationProperties

KafkaRestProperties

KafkaRestPropertiesConfigurationOverride

LinuxOperatingSystemProfile

Microsoft.HDInsight/clusters

NetworkProperties

OsProfile

PrivateLinkConfiguration

PrivateLinkConfigurationProperties

ResourceId

Role

ScriptAction

SecurityProfile

SshProfile

SshPublicKey

StorageAccount

StorageProfile

UserAssignedIdentity

VirtualNetworkProfile

Quickstart samples

ARM template resource definition

Resource format

Property values

Autoscale

AutoscaleCapacity

AutoscaleRecurrence

AutoscaleSchedule

AutoscaleTimeAndCapacity

ClientGroupInfo

ClusterCreateParametersExtendedTags

ClusterCreatePropertiesOrClusterGetProperties

ClusterDefinition

ClusterDefinitionComponentVersion

ClusterIdentity

ClusterIdentityUserAssignedIdentities

ComputeIsolationProperties

ComputeProfile

DataDisksGroups

DiskEncryptionProperties

EncryptionInTransitProperties

HardwareProfile

IPConfiguration

IPConfigurationProperties

KafkaRestProperties

KafkaRestPropertiesConfigurationOverride

LinuxOperatingSystemProfile

Microsoft.HDInsight/clusters

NetworkProperties

OsProfile

PrivateLinkConfiguration

PrivateLinkConfigurationProperties

ResourceId

Role

ScriptAction

SecurityProfile

SshProfile