Create Apache Hadoop clusters using the Azure REST API
Learn how to create an HDInsight cluster using an Azure Resource Manager template and the Azure REST API.
The Azure REST API allows you to perform management operations on services hosted in the Azure platform, including the creation of new resources such as HDInsight clusters.
Note
The steps in this document use the curl (https://curl.haxx.se/) utility to communicate with the Azure REST API.
Create a template
Azure Resource Manager templates are JSON documents that describe a resource group and all resources in it (such as HDInsight.) This template-based approach allows you to define the resources that you need for HDInsight in one template.
The following JSON document is a merger of the template and parameters files from https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.hdinsight/hdinsight-linux-ssh-password/azuredeploy.json, which creates a Linux-based cluster using a password to secure the SSH user account.
{
"properties": {
"template": {
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"clusterType": {
"type": "string",
"allowedValues": ["hadoop",
"hbase",
"spark"],
"metadata": {
"description": "The type of the HDInsight cluster to create."
}
},
"clusterName": {
"type": "string",
"metadata": {
"description": "The name of the HDInsight cluster to create."
}
},
"clusterLoginUserName": {
"type": "string",
"metadata": {
"description": "These credentials can be used to submit jobs to the cluster and to log into cluster dashboards."
}
},
"clusterLoginPassword": {
"type": "securestring",
"metadata": {
"description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
}
},
"sshUserName": {
"type": "string",
"metadata": {
"description": "These credentials can be used to remotely access the cluster."
}
},
"sshPassword": {
"type": "securestring",
"metadata": {
"description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
}
},
"clusterStorageAccountName": {
"type": "string",
"metadata": {
"description": "The name of the storage account to be created and be used as the cluster's storage."
}
},
"clusterWorkerNodeCount": {
"type": "int",
"defaultValue": 4,
"metadata": {
"description": "The number of nodes in the HDInsight cluster."
}
}
},
"variables": {
"defaultApiVersion": "2015-05-01-preview",
"clusterApiVersion": "2015-03-01-preview"
},
"resources": [{
"name": "[parameters('clusterStorageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"location": "[resourceGroup().location]",
"apiVersion": "[variables('defaultApiVersion')]",
"dependsOn": [],
"tags": {
},
"properties": {
"accountType": "Standard_LRS"
}
},
{
"name": "[parameters('clusterName')]",
"type": "Microsoft.HDInsight/clusters",
"location": "[resourceGroup().location]",
"apiVersion": "[variables('clusterApiVersion')]",
"dependsOn": ["[concat('Microsoft.Storage/storageAccounts/',parameters('clusterStorageAccountName'))]"],
"tags": {
},
"properties": {
"clusterVersion": "3.6",
"osType": "Linux",
"clusterDefinition": {
"kind": "[parameters('clusterType')]",
"configurations": {
"gateway": {
"restAuthCredential.isEnabled": true,
"restAuthCredential.username": "[parameters('clusterLoginUserName')]",
"restAuthCredential.password": "[parameters('clusterLoginPassword')]"
}
}
},
"storageProfile": {
"storageaccounts": [{
"name": "[concat(parameters('clusterStorageAccountName'),'.blob.core.windows.net')]",
"isDefault": true,
"container": "[parameters('clusterName')]",
"key": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('clusterStorageAccountName')), variables('defaultApiVersion')).key1]"
}]
},
"computeProfile": {
"roles": [{
"name": "headnode",
"targetInstanceCount": "2",
"hardwareProfile": {
"vmSize": "{}"
},
"osProfile": {
"linuxOperatingSystemProfile": {
"username": "[parameters('sshUserName')]",
"password": "[parameters('sshPassword')]"
}
}
},
{
"name": "workernode",
"targetInstanceCount": "[parameters('clusterWorkerNodeCount')]",
"hardwareProfile": {
"vmSize": "{}"
},
"osProfile": {
"linuxOperatingSystemProfile": {
"username": "[parameters('sshUserName')]",
"password": "[parameters('sshPassword')]"
}
}
}]
}
}
}],
"outputs": {
"cluster": {
"type": "object",
"value": "[reference(resourceId('Microsoft.HDInsight/clusters',parameters('clusterName')))]"
}
}
},
"mode": "incremental",
"Parameters": {
"clusterName": {
"value": "newclustername"
},
"clusterType": {
"value": "hadoop"
},
"clusterStorageAccountName": {
"value": "newstoragename"
},
"clusterLoginUserName": {
"value": "admin"
},
"clusterLoginPassword": {
"value": "changeme"
},
"sshUserName": {
"value": "sshuser"
},
"sshPassword": {
"value": "changeme"
}
}
}
}
This example is used in the steps in this document. Replace the example values in the Parameters section with the values for your cluster.
Important
The template uses the default number of worker nodes (4) for an HDInsight cluster. If you plan on more than 32 worker nodes, then you must select a head node size with at least 8 cores and 14-GB ram.
For more information on node sizes and associated costs, see HDInsight pricing.
Sign in to your Azure subscription
Follow the steps documented in Get started with Azure CLI and connect to your subscription using the az login command.
Create a service principal
Note
These steps are an abridged version of the Create service principal with password section of the Use Azure CLI to create a service principal to access resources document. These steps create a service principal that is used to authenticate to the Azure REST API.
From a command line, use the following command to list your Azure subscriptions.
az account list --query '[].{Subscription_ID:id,Tenant_ID:tenantId,Name:name}' --output tableIn the list, select the subscription that you want to use and note the Subscription_ID and Tenant_ID columns. Save these values.
Use the following command to create an application in Microsoft Entra ID.
az ad app create --display-name "exampleapp" --homepage "https://www.contoso.org" --identifier-uris "https://www.contoso.org/example" --password <Your password> --query 'appId'Replace the values for the
--display-name,--homepage, and--identifier-uriswith your own values. Provide a password for the new Active Directory entry.Note
The
--home-pageand--identifier-urisvalues don't need to reference an actual web page hosted on the internet. They must be unique URIs.The value returned from this command is the App ID for the new application. Save this value.
Use the following command to create a service principal using the App ID.
az ad sp create --id <App ID> --query 'objectId'The value returned from this command is the Object ID. Save this value.
Assign the Owner role to the service principal using the Object ID value. Use the subscription ID you obtained earlier.
az role assignment create --assignee <Object ID> --role Owner --scope /subscriptions/<Subscription ID>/
Get an authentication token
Use the following command to retrieve an authentication token:
curl -X "POST" "https://login.microsoftonline.com/$TENANTID/oauth2/token" \
-H "Cookie: flight-uxoptin=true; stsservicecookie=ests; x-ms-gateway-slice=productionb; stsservicecookie=ests" \
-H "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "client_id=$APPID" \
--data-urlencode "grant_type=client_credentials" \
--data-urlencode "client_secret=$PASSWORD" \
--data-urlencode "resource=https://management.azure.com/"
Set $TENANTID, $APPID, and $PASSWORD to the values obtained or used previously.
If this request is successful, you receive a 200 series response and the response body contains a JSON document.
The JSON document returned by this request contains an element named access_token. The value of access_token is used to authentication requests to the REST API.
{
"token_type":"Bearer",
"expires_in":"3599",
"expires_on":"1463409994",
"not_before":"1463406094",
"resource":"https://management.azure.com/","access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik1uQ19WWoNBVGZNNXBPWWlKSE1iYTlnb0VLWSIsImtpZCI6Ik1uQ19WWmNBVGZNNXBPWWlKSE1iYTlnb0VLWSJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuYXp1cmUuY29tLyIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI2Ny8iLCJpYXQiOjE0NjM0MDYwOTQsIm5iZiI6MTQ2MzQwNjA5NCwiZXhwIjoxNDYzNDA5OTk5LCJhcHBpZCI6IjBlYzcyMzM0LTZkMDMtNDhmYi04OWU1LTU2NTJiODBiZDliYiIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI0Ny8iLCJvaWQiOiJlNjgxZTZiMi1mZThkLTRkZGUtYjZiMS0xNjAyZDQyNWQzOWYiLCJzdWIiOiJlNjgxZTZiMi1mZThkLTRkZGUtYjZiMS0xNjAyZDQyNWQzOWYiLCJ0aWQiOiI3MmY5ODhiZi04NmYxLTQxYWYtOTFhYi0yZDdjZDAxMWRiNDciLCJ2ZXIiOiIxLjAifQ.nJVERbeDHLGHn7ZsbVGBJyHOu2PYhG5dji6F63gu8XN2Cvol3J1HO1uB4H3nCSt9DTu_jMHqAur_NNyobgNM21GojbEZAvd0I9NY0UDumBEvDZfMKneqp7a_cgAU7IYRcTPneSxbD6wo-8gIgfN9KDql98b0uEzixIVIWra2Q1bUUYETYqyaJNdS4RUmlJKNNpENllAyHQLv7hXnap1IuzP-f5CNIbbj9UgXxLiOtW5JhUAwWLZ3-WMhNRpUO2SIB7W7tQ0AbjXw3aUYr7el066J51z5tC1AK9UC-mD_fO_HUP6ZmPzu5gLA6DxkIIYP3grPnRVoUDltHQvwgONDOw"
}
Create a resource group
Use the following to create a resource group.
- Set
$SUBSCRIPTIONIDto the subscription ID received while creating the service principal. - Set
$ACCESSTOKENto the access token received in the previous step. - Replace
DATACENTERLOCATIONwith the data center you wish to create the resource group, and resources, in. For example, 'South Central US'. - Set
$RESOURCEGROUPNAMEto the name you wish to use for this group:
curl -X "PUT" "https://management.azure.com/subscriptions/$SUBSCRIPTIONID/resourcegroups/$RESOURCEGROUPNAME?api-version=2015-01-01" \
-H "Authorization: Bearer $ACCESSTOKEN" \
-H "Content-Type: application/json" \
-d $'{
"location": "DATACENTERLOCATION"
}'
If this request is successful, you receive a 200 series response and the response body contains a JSON document containing information about the group. The "provisioningState" element contains a value of "Succeeded".
Create a deployment
Use the following command to deploy the template to the resource group.
- Set
$DEPLOYMENTNAMEto the name you wish to use for this deployment.
curl -X "PUT" "https://management.azure.com/subscriptions/$SUBSCRIPTIONID/resourcegroups/$RESOURCEGROUPNAME/providers/microsoft.resources/deployments/$DEPLOYMENTNAME?api-version=2015-01-01" \
-H "Authorization: Bearer $ACCESSTOKEN" \
-H "Content-Type: application/json" \
-d "{set your body string to the template and parameters}"
Note
If you saved the template to a file, you can use the following command instead of -d "{ template and parameters}":
--data-binary "@/path/to/file.json"
If this request is successful, you receive a 200 series response and the response body contains a JSON document containing information about the deployment operation.
Important
The deployment has been submitted, but has not completed. It can take several minutes, usually around 15, for the deployment to complete.
Check the status of a deployment
To check the status of the deployment, use the following command:
curl -X "GET" "https://management.azure.com/subscriptions/$SUBSCRIPTIONID/resourcegroups/$RESOURCEGROUPNAME/providers/microsoft.resources/deployments/$DEPLOYMENTNAME?api-version=2015-01-01" \
-H "Authorization: Bearer $ACCESSTOKEN" \
-H "Content-Type: application/json"
This command returns a JSON document containing information about the deployment operation. The "provisioningState" element contains the status of the deployment. If this element contains a value of "Succeeded", then the deployment has completed successfully.
Troubleshoot
If you run into issues with creating HDInsight clusters, see access control requirements.
Next steps
Now that you've successfully created an HDInsight cluster, use the following to learn how to work with your cluster.