Managing Your HDInsight Cluster using PowerShell – Update

Since writing my last post Managing Your HDInsight Cluster and .Net Job Submissions using PowerShell, there have been some useful modifications to the Azure PowerShell Tools.

The HDInsight cmdlets no longer exist as these have now been integrated into the latest release of the Windows Azure Powershell Tools. This integration means:

• You don’t need to specify Subscription parameter
• If needed, you can use AAD authentication to Azure instead of certificates

Also, the cmdlets are fully backwards compatible meaning you don’t need to change your current scripts. The Subscription parameter is now optional but if it specified then it is honoured.

As such the cluster creation script will now be:

1. Param($Hosts = 4, [string] $Cluster = $(throw "Cluster Name Required."), [string] $StorageContainer = "hadooproot")
2.  
3. # Get the subscription information and set variables
4. $subscriptionInfo = Get-AzureSubscription -Default
5. $subName = $subscriptionInfo | %{ $_.SubscriptionName }
6. $subId = $subscriptionInfo | %{ $_.SubscriptionId }
7. $cert = $subscriptionInfo | %{ $_.Certificate }
8. $storeAccount = $subscriptionInfo | %{ $_.CurrentStorageAccountName }
9.  
10. Select-AzureSubscription -SubscriptionName $subName
11.  
12. $key = Get-AzureStorageKey $storeAccount | %{ $_.Primary }
13.  
14. $storageAccountInfo = Get-AzureStorageAccount $storeAccount
15. $location = $storageAccountInfo | %{ $_.Location }
16.  
17. $hadoopUsername = "Hadoop"
18. $clusterUsername = "Admin"    
19. $clusterPassword = "myclusterpassword"
20.  
21. $secpasswd = ConvertTo-SecureString $clusterPassword -AsPlainText -Force
22. $clusterCreds = New-Object System.Management.Automation.PSCredential($clusterUsername, $secpasswd)
23.  
24. $clusterName = $Cluster
25. $numberNodes = $Hosts
26. $containerDefault = $StorageContainer
27. $blobStorage = "$storeAccount.blob.core.windows.net"
28.  
29. # tidyup the root to ensure empty
30. # Remove-AzureStorageContainer Name $containerDefault -Force
31. Write-Host "Deleting old storage container contents: $containerDefault" -f yellow
32. $blobs = Get-AzureStorageBlob -Container $containerDefault
33. foreach($blob in $blobs)
34. {
35.     Remove-AzureStorageBlob -Container $containerDefault -Blob ($blob.Name)
36. }
37.  
38. # Create the cluster
39. Write-Host "Creating '$numberNodes' Node Cluster named: $clusterName" -f yellow
40. Write-Host "Storage Account '$storeAccount' and Container '$containerDefault'" -f yellow
41. Write-Host "User '$clusterUsername' Password '$clusterPassword'" -f green
42.  
43. New-AzureHDInsightCluster -Certificate $cert -Name $clusterName -Location $location -DefaultStorageAccountName $blobStorage -DefaultStorageAccountKey $key -DefaultStorageContainerName $containerDefault -Credential $clusterCreds -ClusterSizeInNodes $numberNodes
44.  
45. Write-Host "Created '$numberNodes' Node Cluster: $clusterName" -f yellow

The only changes are the selection of the subscription and the removal of the Subscription option when creating the cluster. Of course all the other scripts can easily be modified along the same lines; such as the cluster deletion script:

1. Param($Cluster = $(throw "Cluster Name Required."))
2.  
3. # Get the subscription information and set variables
4. $subscriptionInfo = Get-AzureSubscription -Default
5. $subName = $subscriptionInfo | %{ $_.SubscriptionName }
6. $subId = $subscriptionInfo | %{ $_.SubscriptionId }
7. $cert = $subscriptionInfo | %{ $_.Certificate }
8. $storeAccount = $subscriptionInfo | %{ $_.CurrentStorageAccountName }
9.  
10. Select-AzureSubscription -SubscriptionName $subName
11.  
12. $clusterName = $Cluster
13.  
14. # Delete the cluster
15. Write-Host "Deleting Cluster named: $clusterName" -f yellow
16.  
17. Remove-AzureHDInsightCluster $clusterName -Subscription $subId -Certificate $cert
18.  
19. Write-Host "Deleted Cluster $clusterName" -f yellow

This cluster creation script also contains an additional section to ensure that the default storage container does not container any leftover files, from previous cluster creations:

1. $blobs = Get-AzureStorageBlob -Container $containerDefault
2. foreach($blob in $blobs)
3. {
4.     Remove-AzureStorageBlob -Container $containerDefault -Blob ($blob.Name)
5. }

The rationale behind this is to ensure that any files that may be left over from previous clusters creations/deletions are removed.