Cluster REST API in Azure HDInsight

Article
05/10/2022

Create

Creates a cluster in the specified subscription.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
PUT	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}?api-version={api-version}`

The following example shows the request body for creating a Linux-based hadoop cluster. For examples of creating clusters in other ways, see the Examples section below.

{  
    "id":"/subscriptions/{subscription-id}/resourceGroups/myresourcegroup1/providers/Microsoft.HDInsight/clusters/mycluster",  
  "name":"mycluster",   
  "type":"Microsoft.HDInsight/clusters",  
  
    "location": "location-name",  
    "tags": { "tag1": "value1", "tag2": "value2" },  
    "properties": {  
        "clusterVersion": "3.2",  
        "osType": "Linux",  
        "clusterDefinition": {  
            "kind": "hadoop",  
  
            "configurations": {  
                "gateway": {  
                    "restAuthCredential.isEnabled": true,  
                    "restAuthCredential.username": "http-user",  
                    "restAuthCredential.password": "password"  
                },  
  
                "core-site": {  
                    "fs.defaultFS": "wasb://container@storageaccount.blob.core.windows.net",  
                    "fs.azure.account.key.storageaccount.blob.core.windows.net": storage-account-key"  
                }  
            }  
        },
  
        "computeProfile": {  
            "roles": [  
                {  
                    "name": "headnode",  
  
                    "targetInstanceCount": 2,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    },  
  
                    "osProfile": {  
                        "linuxOperatingSystemProfile": {  
                            "username": "username",  
                            "sshProfile": {  
                                "publicKeys": [   
                                    { "certificateData": "ssh-rsa key" }  
                                ]  
                            }  
                        }  
                    }  
                },  
                {  
                    "name": "workernode",  
  
                    "targetInstanceCount": 1,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    },  
  
                    "osProfile": {  
                        "linuxOperatingSystemProfile": {  
                            "username": "username",  
                            "sshProfile": {  
                                "publicKeys": [  
                                    { "certificateData": " ssh-rsa key" }  
                                ]  
                            }  
                        }  
                    }  
                },  
                {  
                    "name": "zookeepernode",  
  
                    "targetInstanceCount": 3,  
  
                    "hardwareProfile": {  
                        "vmSize": "Small"  
                    },  
  
                    "osProfile": {  
                        "linuxOperatingSystemProfile": {  
                            "username": "username",  
                            "sshProfile": {  
                                "publicKeys": [   
                                    { "certificateData": "ssh-rsa key" }  
                                ]  
                            }  
                        }  
                    }  
                }  
            ]  
        }  
    }  
}

Element name	Required	Type	Description
id	Yes	String	Specifies the resource identifier of the cluster.
name	Yes	String	Specifies the name of the cluster.
type	Yes	String	Specifies the type of the cluster.
location	Yes	String	Specifies the supported Azure location where the cluster should be created. For more information, see List all of the available geo-locations.
tags	No	String	Specifies the tags that will be assigned to the cluster. For more information about using tags, see Using tags to organize your Azure resources.
Properties	Yes	Complex Type	Specifies the properties of the cluster.

Properties

Element name	Required	Type	Description
clusterVersion	Yes	String	Specifies the cluster version
osType	Yes	String	Specifies the Operating system for the cluster. Valid values are Linux and Windows
clusterDefinition	Yes	Complex Type	Specifies information about the cluster type and configurations
computeProfile	Yes	Complex Type	Specifies information about the cluster topology and associated role properties

clusterDefinition

Element name	Required	Type	Description
kind	Yes	String	Specifies the cluster type. Valid values are hadoop, hbase, storm & spark
configurations	Yes	Dictionary	This element is a dictionary of configuration type and its associated value dictionary. gateway configuration type is used to configure the http user used for connecting to web APIs and the Ambari portal core-site configuration type is used to configure the default storage account for the cluster

computeProfile

Element name	Required	Type	Description
clusterVersion	Yes	String	Specifies the cluster version
role	Yes	Array of Complex Type (role)	Specifies information about roles in the cluster

role

Element name	Required	Type	Description
name	Yes	String	Specifies the role name
targetInstanceCount	Yes	Integer	Specifies the target instance count for the role
hardwareProfile	Yes	Complex Type	Specifies information about the hardware profile for the role
osProfile	Yes	Complex Type	Specifies information about the os profile for the role

hardwareProfile

Element name	Required	Type	Description
vmSize	Yes	String	Specifies the size of the VM. Refer to HDInsight configuration options (once on this link, scroll down to Node pricing tiers) for valid sizes

osProfile

Element name	Required	Type	Description
linuxOperatingSystemProfile	No	Complex Type	Specifies the linux OS-related settings
windowsOperatingSystemProfile	No	Complex Type	Specifies windows OS-related settings
virtualNetworkProfile	No	Complex Type	Specifies virtual network-related settings if the cluster is being deployed in a virtual network in the user’s subscription
scriptActions	No	Array of Complex Type	List of script actions to execute on the cluster

linuxOperatingSystemProfile

Element name	Required	Type	Description
Username	Yes	String	SSH user name
sshProfile	No	Complex Type	Specifies the SSH key. One of sshProfile or Password is required.
Password	No	String	Specifies the SSH password One of sshProfile or Password is required.

sshProfile

Element name	Required	Type	Description
publicKeys	Yes	Array	Contains a list of certificateData objects. The value is an ssh-rsa public key

windowsOperatingSystemProfile

Element name	Required	Type	Description
rdpSettings	No	Complex Type	Specifies RDP settings for windows clusters

rdpSettings

Element name	Required	Type	Description
username	Yes	String	Specifies the RDP user name
password	Yes	String	Specifies the password for the RDP user
expiryDate	Yes	Date	Expiry date for the RDP credentials

virtualNetworkProfile

Element name	Required	Type	Description
id	Yes	String	Virtual Network Resource Id
subnet	Yes	String	Specifies the subnet name

scriptActions

Element name	Required	Type	Description
name	Yes	String	Friendly name for the script action
uri	Yes	String	URL to the script action file
parameters	No	String	Arguments to pass when executing the script action file

Response

If validation is complete and the request is accepted, the operation will return 200 (OK).

Status code: 200 OK

Response body for a linux cluster creates using ssh key:

{  
    "id":"/subscriptions/{subscription-id}/resourceGroups/myresourcegroup1/providers/Microsoft.HDInsight/clusters/mycluster",  
  "name":"mycluster",   
  "type":"Microsoft.HDInsight/clusters",  
  
    "location": "location-name",  
    "tags": { "tag1": "value1", "tag2": "value2" },  
    "properties": {  
        "clusterVersion": "3.2",  
        "osType": "Linux",  
		"provisioningState": "InProgress",  
		"clusterState": "Accepted",  
		"createdDate": "2015-09-23",  
		"quotaInfo": {  
			"coresUsed": 20  
}  
        "clusterDefinition": {  
            "kind": "hadoop"  
        },  
  
        "computeProfile": {  
            "roles": [  
                {  
                    "name": "headnode",  
  
                    "targetInstanceCount": 2,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    }  
  
                },  
                {  
                    "name": "workernode",  
  
                    "targetInstanceCount": 1,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    }  
                },  
                {  
                    "name": "zookeepernode",  
  
                    "targetInstanceCount": 3,  
  
                    "hardwareProfile": {  
                        "vmSize": "Small"  
                    }  
                }  
            ]  
        }  
    }  
}

Element name	Type	Description
provisioningState	String	Indicates the current provisioning state.
clusterState	String	Indicates the more detailed HDInsight cluster state while provisioning is in progress.
createdDate	Date	Datetime when the cluster create request was received
quotaInfo	Complex Type	Specifies the coresUsed by the cluster
errors	Array of error messages	Contains the error message if provisioningState = ‘failed"
connectivityEndpoints	Complex Type	Specifies the public endpoints for the cluster

connectivityEndpoints

Element name	Type	Description
name	String	Friendly name for the connectivity endpoint
protocol	String	Specifies the Protocol to use (example: HTTPS, SSH)
location	String	Specifies the URL to connect
port	int	Specifies the port to connect

Create a premium, domain-joined HDInsight cluster (Linux only, preview)

Create a premium domain-joined cluster with Apache Ranger. User needs to provide SecurityProfile in the request body to create a secure cluster.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
PUT	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}?api-version={api-version}`

The following example shows the request body for creating a Linux based, premium, domain-joined Hadoop cluster.

{
	"id": "/subscriptions/{ subscription-id }/resourceGroups/myresourcegroup1/providers/Microsoft.HDInsight/ clusters/mycluster ", "
	name "
	: "mycluster",
	"type": "Microsoft.HDInsight/clusters",
	"location": "location-name",
	"tags": {
		"tag1": "value1",
		"tag2": "value2"
	},
	"properties": {
		"clusterVersion": "3.5",
		"osType": "Linux",
		"tier": "premium",
		"clusterDefinition": {
			"kind": "hadoop",
			"configurations": {
				"gateway": {
					"restAuthCredential.isEnabled": true,
					"restAuthCredential.username": "http-user",
					"restAuthCredential.password": "password"
				},
				"core-site": {
					"fs.defaultFS": "wasb://container@storageaccount.blob.core.windows.net",
					"fs.azure.account.key.storageaccount.blob.core.windows.net": "storage-account-key"
				}
			}
		},
		"securityProfile": {
			"directoryType": "ActiveDirectory",
			"domain": "mydomain.com",
			"organizationalUnitDN": "OU=Hadoop,DC=mydomain,DC=COM",
			"ldapsUrls": ["ldaps://mydomain.com:636"],
			"domainUsername": "clusteradmin@mydomain.com",
			"domainUserPassword": "password",
			"clusterUsersGroupDNs": ["ADGroup1", "ADGroup2"]
		},
		"computeProfile": {
			"roles": [
				{
					"name": "headnode",
					"targetInstanceCount": 2,
					"hardwareProfile": {
						"vmSize": "Large"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "username",
							"sshProfile": {
								"publicKeys": [
									{
										"certificateData": "ssh-rsa key"
									}
								]
							}
						}
					},
					"virtualNetworkProfile": {
						"id": "/subscriptions/mysubscriptionid/resourceGroups/myrresourcegroup/providers/Microsoft.Network/virtualNetworks/myvirtualnetwork",
						"subnet": "/subscriptions/mysubscriptionid /resourceGroups/myresourcegroup/providers/Microsoft.Network/virtualNetworks/myvirtualnetwork/subnets/mysubnet"
					}
				},
				{
					"name": "workernode",
					"targetInstanceCount": 1,
					"hardwareProfile": {
						"vmSize": "Large"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "username",
							"sshProfile": {
								"publicKeys": [
									{
										"certificateData": " ssh-rsa key"
									}
								]
							}
						}
					},
					"virtualNetworkProfile": {
						"id": "/subscriptions/mysubscriptionid/resourceGroups/myrresourcegroup/providers/Microsoft.Network/virtualNetworks/myvirtualnetwork",
						"subnet": "/subscriptions/mysubscriptionid /resourceGroups/myresourcegroup/providers/Microsoft.Network/virtualNetworks/myvirtualnetwork/subnets/mysubnet"
					}
				},
				{
					"name": "zookeepernode",
					"targetInstanceCount": 3,
					"hardwareProfile": {
						"vmSize": "Small"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "username",
							"sshProfile": {
								"publicKeys": [
									{
										"certificateData": "ssh-rsa key"
									}
								]
							}
						},
						"virtualNetworkProfile": {
							"id": "/subscriptions/mysubscriptionid/resourceGroups/myrresourcegroup/providers/Microsoft.Network/virtualNetworks/myvirtualnetwork",
							"subnet": "/subscriptions/mysubscriptionid /resourceGroups/myresourcegroup/providers/Microsoft.Network/virtualNetworks/myvirtualnetwork/subnets/mysubnet"
						}
					}
				}
			]
		}
	}
}

Element name	Required	Type	Description
id	Yes	String	Specifies the resource identifier of the cluster.
name	Yes	String	Specifies the name of the cluster.
type	Yes	String	Specifies the type of the cluster.
location	Yes	String	Specifies the supported Azure location where the cluster should be created. For more information, see List all of the available geo-locations.
tags	No	String	Specifies the tags that will be assigned to the cluster. For more information about using tags, see Using tags to organize your Azure resources.
Properties	Yes	Complex Type	Specifies the properties of the cluster.

Properties

Element name	Required	Type	Description
clusterVersion	Yes	String	Specifies the cluster version
osType	Yes	String	Specifies the Operating system for the cluster. Valid value is Linux because only Linux cluster types can join an Azure AD domain.
tier	No	String	Default value is standard. Valid values are standard and premium. If no value is specified, the value is assumed to be standard. Specifies the Tier for the cluster. Domain joined clusters are only supported in premium tier
clusterDefinition	Yes	Complex Type	Specifies information about the cluster type and configurations
computeProfile	Yes	Complex Type	Specifies information about the cluster topology and associated role properties
securityProfile	No	Complex Type	If a secure, domain-joined cluster is being created, this element specifies the Active Directory related settings

clusterDefinition

Element name	Required	Type	Description
kind	Yes	String	Specifies the cluster type. Valid values are hadoop, hbase, storm & spark
configurations	Yes	Dictionary	This element is a dictionary of configuration type and its associated value dictionary. gateway configuration type is used to configure the http user used for connecting to web APIs and the Ambari portal core-site configuration type is used to configure the default storage account for the cluster

computeProfile

Element name	Required	Type	Description
clusterVersion	Yes	String	Specifies the cluster version
role	Yes	Array of Complex Type (role)	Specifies information about roles in the cluster

securityProfile

Element name	Required	Type	Description
directoryType	Yes	String	Type of LDAP directory that is used. Currently "ActiveDirectory" is the only supported value.
domain	Yes	String	Active Directory domain for the cluster
organizationalUnitDN	Yes	String	Distinguished name of the organizational unit in the Active directory where user and computer accounts will be created
ldapsUrls	Yes	Array of String	URLs of one or multiple LDAPS servers for the Active Directory
domainUserName	Yes	String	A domain user account with sufficient permissions for creating the cluster. It should be in user@domain format
domainUserPassword	Yes	String	Password for the domain user account
clusterUsersGroupDNS	No	Array of String	Distinguished names of the Active Directory groups that will be available in Ambari and Apache Ranger

role

Element name	Required	Type	Description
name	Yes	String	Specifies the role name
targetInstanceCount	Yes	Integer	Specifies the target instance count for the role
hardwareProfile	Yes	Complex Type	Specifies information about the hardware profile for the role
osProfile	Yes	Complex Type	Specifies information about the os profile for the role

hardwareProfile

Element name	Required	Type	Description
vmSize	Yes	String	Specifies the size of the VM. Refer to HDInsight configuration options (once on this link, scroll down to Node pricing tiers) for valid sizes

osProfile

Element name	Required	Type	Description
linuxOperatingSystemProfile	No	Complex Type	Specifies the linux OS-related settings
virtualNetworkProfile	No	Complex Type	Specifies virtual network-related settings if the cluster is being deployed in a virtual network in the user’s subscription
scriptActions	No	Array of Complex Type	List of script actions to execute on the cluster

linuxOperatingSystemProfile

Element name	Required	Type	Description
Username	Yes	String	SSH user name
sshProfile	No	Complex Type	Specifies the SSH key. One of sshProfile or Password is required.
Password	No	String	Specifies the SSH password One of sshProfile or Password is required.

sshProfile

Element name	Required	Type	Description
publicKeys	Yes	Array	Contains a list of certificateData objects. The value is an ssh-rsa public key

virtualNetworkProfile

Element name	Required	Type	Description
id	Yes	String	Virtual Network Resource Id
subnet	Yes	String	Specifies the subnet name

scriptActions

Element name	Required	Type	Description
name	Yes	String	Friendly name for the script action
uri	Yes	String	URL to the script action file
parameters	No	String	Arguments to pass when executing the script action file

Response

If validation is complete and the request is accepted, the operation will return 200 (OK).

Status code: 200 OK

Response body for a linux cluster creates using ssh key:

{  
    "id":"/subscriptions/{subscription-id}/resourceGroups/myresourcegroup1/providers/Microsoft.HDInsight/clusters/mycluster",  
  "name":"mycluster",   
  "type":"Microsoft.HDInsight/clusters",  
  
    "location": "location-name",  
    "tags": { "tag1": "value1", "tag2": "value2" },  
    "properties": {  
        "clusterVersion": "3.2",  
        "osType": "Linux",  
		"provisioningState": "InProgress",  
		"clusterState": "Accepted",  
		"createdDate": "2015-09-23",  
		"quotaInfo": {  
			"coresUsed": 20  
}  
        "clusterDefinition": {  
            "kind": "hadoop"  
        },  
  
        "computeProfile": {  
            "roles": [  
                {  
                    "name": "headnode",  
  
                    "targetInstanceCount": 2,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    }  
  
                },  
                {  
                    "name": "workernode",  
  
                    "targetInstanceCount": 1,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    }  
                },  
                {  
                    "name": "zookeepernode",  
  
                    "targetInstanceCount": 3,  
  
                    "hardwareProfile": {  
                        "vmSize": "Small"  
                    }  
                }  
            ]  
        }  
    }  
}

Element name	Type	Description
provisioningState	String	Indicates the current provisioning state.
clusterState	String	Indicates the more detailed HDInsight cluster state while provisioning is in progress.
createdDate	Date	Datetime when the cluster create request was received
quotaInfo	Complex Type	Specifies the coresUsed by the cluster
errors	Array of error messages	Contains the error message if provisioningState = ‘failed"
connectivityEndpoints	Complex Type	Specifies the public endpoints for the cluster

connectivityEndpoints

Element name	Type	Description
name	String	Friendly name for the connectivity endpoint
protocol	String	Specifies the Protocol to use (example: HTTPS, SSH)
location	String	Specifies the URL to connect
port	int	Specifies the port to connect

Create a cluster with Azure Data Lake Store as the default filesystem

Creates a cluster in the specified subscription with Azure Data Lake Store as the default filesystem. Provide a ClusterIdentity object in the request body and configure the default-filesystem property with appropriate Data Lake Store URL.

Azure Data Lake can be configured as the default filesystem for cluster versions starting from 3.5 inclusive.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
PUT	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}?api-version={api-version}}`

The following example shows the request body for creating a Linux-based Hadoop cluster using Azure Data Lake Store as the default filesystem for the cluster.

{
	"id": "/subscriptions/{subscription-id}/resourceGroups/myresourcegroup1/providers/Microsoft.HDInsight/clusters/mycluster",
	"name": "mycluster",
	"type": "Microsoft.HDInsight/clusters",
	"location": "location-name",
	"tags": {
		"tag1": "value1",
		"tag2": "value2"
	},
	"properties": {
		"clusterVersion": "3.5",
		"osType": "Linux",
		"tier": "Standard",
		"clusterDefinition": {
			"kind": "hadoop",
			"configurations": {
				"gateway": {
					"restAuthCredential.isEnabled": true,
					"restAuthCredential.username": "http-user",
					"restAuthCredential.password": "password"
				},
				"clusterIdentity": {
					"clusterIdentity.applicationId": "applicationId",
					"clusterIdentity.certificate": "certificate-contents-in-base64-encoding",
					"clusterIdentity.aadTenantId": "aad-tenant-id",
					"clusterIdentity.resourceUri": "https://KonaCompute.net/",
					"clusterIdentity.certificatePassword": "certificate-password"
				},
				"core-site": {
					"fs.defaultFS": "adl://home",
					"dfs.adls.home.hostname": "yourstorageaccount.azuredatalakestore.net",
					"dfs.adls.home.mountpoint": "/path/to/cluster/root"
				}
			}
		},
		"computeProfile": {
			"roles": [
				{
					"name": "headnode",
					"targetInstanceCount": 2,
					"hardwareProfile": {
						"vmSize": "Large"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "username",
							"sshProfile": {
								"publicKeys": [
									{
										"certificateData": "ssh-rsa key"
									}
								]
							}
						}
					}
				},
				{
					"name": "workernode",
					"targetInstanceCount": 1,
					"hardwareProfile": {
						"vmSize": "Large"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "username",
							"sshProfile": {
								"publicKeys": [
									{
										"certificateData": " ssh-rsa key"
									}
								]
							}
						}
					}
				},
				{
					"name": "zookeepernode",
					"targetInstanceCount": 3,
					"hardwareProfile": {
						"vmSize": "Small"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "username",
							"sshProfile": {
								"publicKeys": [
									{
										"certificateData": "ssh-rsa key"
									}
								]
							}
						}
					}
				}
			]
		}
	}
}

Element name	Required	Type	Description
id	Yes	String	Specifies the resource identifier of the cluster.
name	Yes	String	Specifies the name of the cluster.
type	Yes	String	Specifies the type of the cluster.
location	Yes	String	Specifies the supported Azure location where the cluster should be created. For more information, see List all of the available geo-locations.
tags	No	String	Specifies the tags that will be assigned to the cluster. For more information about using tags, see Using tags to organize your Azure resources.
Properties	Yes	Complex Type	Specifies the properties of the cluster.

Properties

Element name	Required	Type	Description
clusterVersion	Yes	String	Specifies the cluster version
osType	Yes	String	Specifies the Operating system for the cluster. Valid value is Linux
tier	No	String	Specifies the tier for the cluster. Valid values are standard and premium. If tier is not specified, cluster will be of standard tier.
clusterDefinition	Yes	Complex Type	Specifies information about the cluster type and configurations
computeProfile	Yes	Complex Type	Specifies information about the cluster topology and associated role properties

clusterDefinition

Element name	Required	Type	Description
kind	Yes	String	Specifies the cluster type. Valid values are hadoop, hbase, storm & spark
configurations	Yes	Dictionary	This element is a dictionary of configuration type and its associated value dictionary. gateway configuration type is used to configure the http user used for connecting to web APIs and the Ambari portal core-site configuration type is used to configure the default storage account for the cluster

computeProfile

Element name	Required	Type	Description
clusterVersion	Yes	String	Specifies the cluster version
role	Yes	Array of Complex Type (role)	Specifies information about roles in the cluster

role

Element name	Required	Type	Description
name	Yes	String	Specifies the role name
targetInstanceCount	Yes	Integer	Specifies the target instance count for the role
hardwareProfile	Yes	Complex Type	Specifies information about the hardware profile for the role
osProfile	Yes	Complex Type	Specifies information about the os profile for the role

hardwareProfile

Element name	Required	Type	Description
vmSize	Yes	String	Specifies the size of the VM. Refer to HDInsight configuration options (once on this link, scroll down to Node pricing tiers) for valid sizes

osProfile

Element name	Required	Type	Description
linuxOperatingSystemProfile	No	Complex Type	Specifies the linux OS-related settings
virtualNetworkProfile	No	Complex Type	Specifies virtual network-related settings if the cluster is being deployed in a virtual network in the user’s subscription
scriptActions	No	Array of Complex Type	List of script actions to execute on the cluster

linuxOperatingSystemProfile

Element name	Required	Type	Description
Username	Yes	String	SSH user name
sshProfile	No	Complex Type	Specifies the SSH key. One of sshProfile or Password is required.
Password	No	String	Specifies the SSH password One of sshProfile or Password is required.

sshProfile

Element name	Required	Type	Description
publicKeys	Yes	Array	Contains a list of certificateData objects. The value is an ssh-rsa public key

virtualNetworkProfile

Element name	Required	Type	Description
id	Yes	String	Virtual Network Resource Id
subnet	Yes	String	Specifies the subnet name

scriptActions

Element name	Required	Type	Description
name	Yes	String	Friendly name for the script action
uri	Yes	String	URL to the script action file
parameters	No	String	Arguments to pass when executing the script action file

Response

If validation is complete and the request is accepted, the operation will return 200 (OK).

Status code: 200 OK

Response body for a linux cluster creates using ssh key:

{
	"id": "/subscriptions/{subscription ID}/resourceGroups/rg1/providers/Microsoft.HDInsightCurrent/clusters/mycluster",
	"name": "mycluster",
	"type": "Microsoft.HDInsightCurrent/clusters",
	"location": "East Asia",
	"etag": "{value}",
	"tags": null,
	"properties": {
		"clusterVersion": "3.5.1000.0",
		"osType": "Linux",
		"clusterDefinition": {
			"kind": "hadoop"
		},
		"computeProfile": {
			"roles": [
				{
					"name": "headnode",
					"targetInstanceCount": 2,
					"hardwareProfile": {
						"vmSize": "Standard_D3"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "myuser"
						}
					}
				},
				{
					"name": "workernode",
					"targetInstanceCount": 2,
					"hardwareProfile": {
						"vmSize": "Standard_D3"
					},
					"osProfile": {
						"linuxOperatingSystemProfile": {
							"username": "myuser"
						}
					}
				}
			]
		},
		"provisioningState": "InProgress",
		"clusterState": "Accepted",
		"createdDate": "2016-11-15T09:21:47.61",
		"quotaInfo": {
			"coresUsed": 16
		},
		"tier": "standard"
	}
}

Element name	Type	Description
provisioningState	String	Indicates the current provisioning state.
clusterState	String	Indicates the more detailed HDInsight cluster state while provisioning is in progress.
createdDate	Date	Datetime when the cluster create request was received
quotaInfo	Complex Type	Specifies the coresUsed by the cluster
errors	Array of error messages	Contains the error message if provisioningState = ‘failed"
connectivityEndpoints	Complex Type	Specifies the public endpoints for the cluster

connectivityEndpoints

Element name	Type	Description
name	String	Friendly name for the connectivity endpoint
protocol	String	Specifies the Protocol to use (example: HTTPS, SSH)
location	String	Specifies the URL to connect
port	int	Specifies the port to connect

Delete

Deletes an HDInsight cluster.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
DELETE	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}?api-version={api-version}`

Response

HTTP 202 (Accepted) to indicate that the operation will complete asynchronously. Async polling will return a 204 (NoContent) once the operation completes successfully.

Remarks

To track progress of a delete cluster request, see Asynchronous Operations (202 Accepted and Location header)

Get

Gets the details/properties of the specified cluster.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
GET	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}?api-version={api-version}`

Response

The operation will return 200 (OK) if the request is completed successfully

Status code: 200 OK

Response body is the same as create cluster.

Response body for linux cluster details

{  
    "id":"/subscriptions/{subscription-id}/resourceGroups/myresourcegroup1/providers/Microsoft.HDInsight/clusters/mycluster",  
  "name":"mycluster",   
  "type":"Microsoft.HDInsight/clusters",  
  
    "location": "location-name",  
    "tags": { "tag1": "value1", "tag2": "value2" },  
    "properties": {  
        "clusterVersion": "3.2",  
        "osType": "Linux",  
		"provisioningState": "InProgress",  
		"clusterState": "Accepted",  
		"createdDate": "2015-09-23",  
		"quotaInfo": {  
			"coresUsed": 20  
}  
        "clusterDefinition": {  
            "kind": "hadoop"  
        },  
  
        "computeProfile": {  
            "roles": [  
                {  
                    "name": "headnode",  
  
                    "targetInstanceCount": 2,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    },  
                    "osProfile": {  
                       "linuxOperatingSystemProfile": {  
                          "username": "sshuser"  
                       }  
                     }  
  
                },  
                {  
                    "name": "workernode",  
  
                    "targetInstanceCount": 1,  
  
                    "hardwareProfile": {  
                        "vmSize": "Large"  
                    },  
  
                    "osProfile": {  
                       "linuxOperatingSystemProfile": {  
                          "username": "sshuser"  
                       }  
                     }  
  
                },  
                {  
                    "name": "zookeepernode",  
  
                    "targetInstanceCount": 3,  
  
                    "hardwareProfile": {  
                        "vmSize": "Small"  
                    },  
  
                    "osProfile": {  
                       "linuxOperatingSystemProfile": {  
                          "username": "sshuser"  
                       }  
                     }  
                }  
            ]  
        }  
    }  
}

Element name	Type	Description
provisioningState	String	Indicates the current provisioning state.
clusterState	String	Indicates the more detailed HDInsight cluster state while provisioning is in progress.
createdDate	Date	Datetime when the cluster create request was received
quotaInfo	Complex Type	Specifies the coresUsed by the cluster
errors	Array of error messages	Contains the error message if provisioningState = ‘failed"
connectivityEndpoints	Complex Type	Specifies the public endpoints for the cluster

connectivityEndpoints

Element name	Type	Description
name	String	Friendly name for the connectivity endpoint
protocol	String	Specifies the Protocol to use (example: HTTPS, SSH)
location	String	Specifies the URL to connect
port	int	Specifies the port to connect

List configurations

Retrieves all cluster configuration details (including sensitive information).

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/configurations?api-version={api-version}`

Response

HTTP 200 (OK) on successful completion of the operation.

Example response:

"configurations":   
{  
     "gateway": {  
          "restAuthCredential.isEnabled": true,  
          "restAuthCredential.username": "username",  
          "restAuthCredential.password": "Password123!"     
      },  
  
      "core-site": {  
           "fs.defaultFS": "wasbs://mycontainter@mystorageaccount.blob.core.windows.net",
	   "fs.azure.account.key.mystorageaccount.blob.core.windows.net": "storagekey1value"  
      }  
}

Element name	Type	Description
configurations	Dictionary	This element is a dictionary of configuration type and its associated value dictionary. gateway configuration type is used to configure the http user used for connecting to web APIs and the Ambari portal core-site configuration type is used to configure the default storage account for the cluster

Get configuration

Gets details about a single configuration type (omits sensitive values).

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
GET	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/configurations/{configuration Type}?api-version={api-version}`

Response

HTTP 200 (OK) on successful completion of the operation.

Example response:

{
     "core-site": {  
          "fs.defaultFS": "wasbs://mycontainter@mystorageaccount.blob.core.windows.net",
	  "fs.azure.account.key.mystorageaccount.blob.core.windows.net": ""
     }
}

Get gateway settings

This operation retrieves cluster gateway HTTP credentials.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/getGatewaySettings?api-version={api-version}`

Response

HTTP 200 (OK) on successful completion of the operation.

Example response:

{  
     "restAuthCredential.isEnabled": true,  
     "restAuthCredential.username": "username",  
     "restAuthCredential.password": "Password123!"  
}

Update gateway settings

This operation allows users to change the cluster gateway HTTP credentials.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/updateGatewaySettings?api-version={api-version}`

{  
     "restAuthCredential.isEnabled": true,  
     "restAuthCredential.username": "username",  
     "restAuthCredential.password": "Password123!"  
}

Response

HTTP 202 (Accepted) on successful completion of the operation.

Change connectivity settings (DEPRECATED)

~~This operation allows users to enable/disable the HTTPS connectivity to the cluster.~~

Use updateGatewaySettings instead.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/configurations/{configuration Type}?api-version={api-version}`

To enable connectivity

{  	
   "restAuthCredential.isEnabled": true,  	
   "restAuthCredential.username": "user",  	
   "restAuthCredential.password": "password here"  
}

List by resource group

Lists all the clusters in the user’s subscription in the specified resource group.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
GET	`https://management.azure.com/subscriptions/{subscription Id}/resourcegroups/{resourceGroup Id}/providers/Microsoft.HDInsight/clusters?api-version={api-version}`

Response

The operation will return 200 (OK) if the request is completed successfully.

Status code: 200 OK

Response body is an array of cluster details. For more information, see Get cluster properties.

{  
  "value": [  
		{ Cluster details }  
    ]  
}

List by subscription

Lists all the clusters in the user’s subscription.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
GET	`https://management.azure.com/subscriptions/{subscription Id}/providers/Microsoft.HDInsight/clusters?api-version={api-version}`

Response

The operation will return 200 (OK) if the request is completed successfully.

Status code: 200 OK

Response body is an array of cluster details.

{  
  "value": [  
		{ Cluster details }  
    ]  
}

To disable connectivity

{  
   "restAuthCredential.isEnabled": false  
}

Element name	Required	Type	Description
restAuthCredential.isEnabled	Yes	Boolean	Specifies if connectivity should be enabled or disabled
restAuthCredential.username	No	String	Required if isEnabled=true Specifies the username for connectivity settings
restAuthCredential.password	No	String	Required if isEnabled=true Specifies the password for connectivity settings

Response

HTTP 202 (Accepted) to indicate that the operation will complete asynchronously. Async polling will return a 204 (NoContent) once the operation completes successfully.

Remarks

To track progress of a delete cluster request, see Asynchronous Operations (202 Accepted and Location header)

Change RDP settings (Windows cluster only)

This operation allows a user to enable/disable RDP. It applies to Windows-based clusters.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/changerdpsetting?api-version={api-version}`

Request Body

To enable RDP

{  
	"osProfile": {  
        "windowsOperatingSystemProfile": {  
        	"rdpSettings": {  
        	      "username": "username",  
            	      "password": "password here",  
            	      "expiryDate": "YYYY-MM-DD"  
        	}  
        }  
    }  
}

To disable RDP

{  
	"osProfile": {  
        "windowsOperatingSystemProfile": {  
        	"rdpSettings": null  
        }  
    }  
}

osProfile

Element name	Required	Type	Description
windowsOperatingSystemProfile	No	Complex Type	Specifies windows OS-related settings

windowsOperatingSystemProfile

Element name	Required	Type	Description
rdpSettings	No	Complex Type	Specifies RDP settings for windows clusters

rdpSettings

Element name	Required	Type	Description
username	Yes	String	Specifies the RDP user name
password	Yes	String	Specifies the password for the RDP user
expiryDate	Yes	Date	Expiry date for the RDP credentials

Response

The operation will return 200 (OK) if the request is completed successfully

Status code: 200 OK

Response body is the same as Create a cluster.

Run Script Actions on a running cluster (Linux cluster only)

Execute Script action on a running cluster.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/executeScriptActions?api-version={api-version}`

Request Body

{  
  "scriptActions": [  
    {  
      "name": "script-name",  
      "uri": "script-uri",  
      "parameters": "script-parameters",  
      "roles": [  
        "headnode",  
        "workernode"  
      ]  
    },  
    ...  
  ],  
  "persistOnSuccess": true  
}

scriptActions

Element name	Required	Type	Description
name	Yes	String	Specifies the name of the script action
uri	Yes	String	Specifies the URI of the script action
parameters	Yes	String	Specifies the parameters required by the script action
roles	Yes	Array of String	Specifies the target roles that the script action executes on
persistOnSuccess	Yes	Boolean	Specifies whether the script actions will be persisted after successful executions

Response

If validation is complete and the request is accepted, the operation will return 202 (Accepted).

Status code: 202 (Accepted)

List all persisted Script Actions for a cluster (Linux cluster only)

This operation returns all the persisted scripts actions of the specified cluster.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
GET	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/scriptActions?api-version={api-version}`

Response

HTTP 200 (OK) on successful completion of the operation.

Status code: 200 OK

Example response:

{  
"value":  
[  
  {  
    "name":"script-name",  
    "uri":"script-uri",  
    "parameters":"script-parameters",  
    "roles":["headnode","workernode"],  
    "applicationName":null  
  },  
  ...  
]  
}

Element name	Type	Description
name	String	Specifies the name of the script action.
uri	String	Specifies the URI of the script action.
parameters	String	Specifies the parameters required by the script action
roles	Array of String	Specifies the targeted roles that the script action executes on.
applicationName	String	Specifies the corresponding application that the script is associated with. applicationName is null if the script is provided by users

Remove a persisted Script Action for a cluster (Linux cluster only)

This operation removes an HDInsight persisted script action for a cluster.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
DELETE	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/scriptActions/{script Name}?api-version={api-version}`

Response

HTTP 200 (OK) to indicate that the script action has been removed from the list of persisted script actions.
HTTP 404 (NotFound) to indicate that there is no existing persisted script action with corresponding scriptName.

Get Script Action execution from history (Linux clusters only)

This operation returns latest scripts action execution of the specified cluster or execution details for an individual script execution.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
GET	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/scriptExecutionHistory/{script Execution Id}?api-version={api-version}`

Response

The operation will return 200 (OK) if the request is completed successfully. Response body is an array of script execution details or a single script execution details if scriptExecutionId is provided. Below is an example of a script execution detail.

Status code: 200 (OK)

Example response:

{  
  "scriptExecutionId":script-execution-id,  
  "name":"script-name",  
  "applicationName":null,  
  "uri":"script-uri",  
  "parameters":"script-parameters",  
  "roles":["headnode","workernode"],  
  "startTime":"2016-02-26T23:49:13.0773637Z",  
  "endTime":"2016-02-26T23:49:33.4964725Z",  
  "status":"Succeeded",  
  "operation":"PostClusterCreateScriptActionRequest",  
  "executionSummary":  
	[{"status":"COMPLETED",  
	 "instanceCount":4}],  
  "debugInformation": "debug-information"  
}

Element name	Type	Description
scriptExecutionId	Long	Specifies the execution id of the script action.
name	String	Specifies the name of the script action.
applicationName	String	Specifies the corresponding application that the script is associated with. applicationName is null if the script is provided by users
uri	String	Specifies the URI of the script action.
parameters	String	Specifies the parameters required by the script action
roles	Array of String	Specifies the targeted roles that the script action executes on.
startTime	DateTime	Specifies the start time of the script action execution
endTime	DateTime	Specifies the end time of the script action execution
status	String	Specifies the status of the script action execution
operation	String	Specifies the reason why the script action was executed. For example, ScaleUp means that the script action was executed during cluster scale up.
executionSummary	Array of complex type	Specifies the summary of execution in terms of how many hosts succeeded and how many hosts failed to execute the script.
debugInformation	String	Specifies detailed debug information for the script. debugInformation is returned only when a scriptExecutionId is provided in the request.

executionSummary

Element name	Type	Description
status	String	Specifies the status of the execution on individual hosts.
instanceCount	Int	Specifies the number of executions with corresponding status.

Promote a script from Script Action execution history to be persisted (Linux cluster only)

This operation promotes an HDInsight script from script execution history to be persisted.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/scriptExecutionHistory/{script Execution Id}/promote?api-version={api-version}`

Response

HTTP 200 (OK) to indicate that the script action has been promoted.
HTTP 404 (NotFound) to indicate that there is no existing script action execution with corresponding scriptExecutionId.

Resize

Resizes an existing HDInsight cluster.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
POST	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/roles/{role name}/resize?api-version={api-version}`

Following shows an example request to update tags for a cluster

{  
    "targetInstanceCount": 10  
}

Element name	Required	Type	Description
targetInstanceCount	Yes	Int	Specifies the new instance count for the role

Response

HTTP 202 (Accepted) to indicate that the operation will complete asynchronously. Async polling will return a 204 (NoContent) once the operation completes successfully.

Remarks

To track progress of a delete cluster request, see Asynchronous Operations (202 Accepted and Location header)

Update

Updates tags for a cluster.

All other updates are separate actions described separately.

Request

See Common parameters and headers for headers and parameters that are used by clusters.

Method	Request URI
PATCH	`https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}?api-version={api-version}`

Following shows an example request to update tags for a cluster

{   
    "tags": {"department": "finance"}  
}

Element name	Required	Type	Description
tags	Yes	String	Specifies the tags that will be assigned to the cluster. For more information about using tags, see Using tags to organize your Azure resources.

Response

The operation will return 200 (OK) if the request is completed successfully.

Status code: 200 OK

Response body is the same as Create a cluster.

Share via

Cluster REST API in Azure HDInsight

Create

Request

Properties

clusterDefinition

computeProfile

role

hardwareProfile

osProfile

linuxOperatingSystemProfile

sshProfile

windowsOperatingSystemProfile

rdpSettings

virtualNetworkProfile

scriptActions

Response

connectivityEndpoints

Create a premium, domain-joined HDInsight cluster (Linux only, preview)

Request

Properties

clusterDefinition

computeProfile

securityProfile

role

hardwareProfile

osProfile

linuxOperatingSystemProfile

sshProfile

virtualNetworkProfile

scriptActions

Response

connectivityEndpoints

Create a cluster with Azure Data Lake Store as the default filesystem

Request

Properties

clusterDefinition

computeProfile

role

hardwareProfile

osProfile

linuxOperatingSystemProfile

sshProfile

virtualNetworkProfile

scriptActions

Response

connectivityEndpoints

Delete

Request

Response

Remarks

Get

Request

Response

connectivityEndpoints

List configurations

Request

Response

Get configuration

Request

Response

Get gateway settings

Request

Response

Update gateway settings

Request

Response

Change connectivity settings (DEPRECATED)

Request

List by resource group

Request

Response

List by subscription

Request

Response

Response

Remarks

Change RDP settings (Windows cluster only)

Request

osProfile