Edit

Share via

Manage Apache Hadoop clusters in Azure HDInsight by using PowerShell

  • Article

You can use Azure PowerShell to control and automate the deployment and management of your workloads in Azure. In this article, you learn how to manage Apache Hadoop clusters in Azure HDInsight by using the Az PowerShell module. For the list of the HDInsight PowerShell cmdlets, see the Az.HDInsight reference.

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

Note

We recommend that you use the Azure Az PowerShell module to interact with Azure. To get started, see Install Azure PowerShell. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.

The Az PowerShell module installed.

Create clusters

To create clusters, see Create Linux-based clusters in HDInsight by using Azure PowerShell.

List clusters

To list all clusters in the current subscription, use the following command:

Get-AzHDInsightCluster

Show clusters

To show details of a specific cluster in the current subscription, use the following command:

Get-AzHDInsightCluster -ClusterName <Cluster Name>

Delete clusters

To delete a cluster, use the following command:

Remove-AzHDInsightCluster -ClusterName <Cluster Name>

You can also delete a cluster by removing the resource group that contains the cluster. Deleting a resource group deletes all the resources in the group, including the default storage account.

Remove-AzResourceGroup -Name <Resource Group Name>

Scale clusters

You can use the cluster scaling feature to change the number of worker nodes that are used by a cluster that's running in HDInsight without having to re-create the cluster. To change the Hadoop cluster size by using PowerShell, run the following command from a client machine:

Set-AzHDInsightClusterSize -ClusterName <Cluster Name> -TargetInstanceCount <NewSize>

For more information about scaling clusters, see Scale HDInsight clusters.

Update HTTP user credentials

The Set-AzHDInsightGatewayCredential parameter sets the gateway HTTP credentials of an HDInsight cluster.

$clusterName = "CLUSTERNAME"
$credential = Get-Credential -Message "Enter the HTTP username and password:" -UserName "admin"

Set-AzHDInsightGatewayCredential -ClusterName $clusterName -HttpCredential $credential

Find the default storage account

The following PowerShell script demonstrates how to get the default storage account name and the related information:

#Connect-AzAccount
$clusterName = "<HDInsight Cluster Name>"

$clusterInfo = Get-AzHDInsightCluster -ClusterName $clusterName
$storageInfo = $clusterInfo.DefaultStorageAccount.split('.')
$defaultStorageType = $storageInfo[1]
$defaultStorageName = $storageInfo[0]

echo "Default Storage account name: $defaultStorageName"
echo "Default Storage account type: $defaultStorageType"

if ($defaultStorageType -eq "blob")
{
    $defaultBlobContainerName = $cluster.DefaultStorageContainer
    $defaultStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $defaultStorageAccountName)[0].Value
    $defaultStorageAccountContext = New-AzStorageContext -StorageAccountName $defaultStorageAccountName -StorageAccountKey $defaultStorageAccountKey

    echo "Default Blob container name: $defaultBlobContainerName"
    echo "Default Storage account key: $defaultStorageAccountKey"
}

Find the resource group

In the Azure Resource Manager mode, each HDInsight cluster belongs to an Azure resource group. To find the resource group, use the following command:

$clusterName = "<HDInsight Cluster Name>"

$cluster = Get-AzHDInsightCluster -ClusterName $clusterName
$resourceGroupName = $cluster.ResourceGroup

Submit jobs

To use the following products to submit jobs, follow the instructions in the references:

Upload data to Azure Blob Storage

To upload data to Azure Blob Storage, see Upload data to HDInsight.

Related content