Get HDInsight Properties with PowerShell

Article
05/23/2014

Small Bites of Big Data from AzureCAT

You’ve created your HDInsight Hadoop clusters and now you want to know exactly what you have out there in Azure. Maybe you want to pull the key information into a repository periodically as a reference for future troubleshooting, comparisons, or billing. Maybe you just need to get a quick look at your overall HDInsight usage. This is something you can easily automate with PowerShell.

Environment

First, open Windows Azure PowerShell or powershell_ise.exe.

Set some values for your environment:

 $SubName = "YourSubscriptionName"

Select-AzureSubscription -SubscriptionName $SubName

Get-AzureSubscription -Current

$ClusterName = "HDInsightClusterName" #HDInsight cluster name

HDInsight Usage for the Subscription

Take a look at your overall HDInsight usage for this subscription:

 Get-AzureHDInsightProperties

Get-AzureHDInsightProperties returns the number of clusters for this subscription, the total HDInsight cores used and available (for head nodes and data nodes), the Azure regions where HDInsight clusters can be created, and the HDInsight versions available for new clusters:

 ClusterCount    : 2

CoresAvailable  : 122

CoresUsed       : 48

Locations       : {East US, North Europe, Southeast Asia, West Europe...}

MaxCoresAllowed : 170

Versions        : {1.6, 2.1, 3.0}

You can also pick out specific pieces of information and write them to a file, store them as variables, or use them elsewhere. This example simply outputs the values to the screen.

 write-host '== Max HDInsight Cores for Sub: ' (Get-AzureHDInsightProperties).MaxCoresAllowed

write-host '== Cores Available:             ' (Get-AzureHDInsightProperties).CoresAvailable

write-host '== Cores Used:                  ' (Get-AzureHDInsightProperties).CoresUsed

HDInsight Cluster Information

Get-AzureHDInsightCluster provides information about all existing HDInsight clusters for this subscription:

 Get-AzureHDInsightCluster

 As you can see this cmdlet tells you the size, connection information, and version.

 ClusterSizeInNodes    : 4

ConnectionUrl         : https://BigCAT.azurehdinsight.net

CreateDate            : 4/5/2014 3:37:23 PM

DefaultStorageAccount : sqlcatwomanwestus.blob.core.windows.net

HttpUserName          : Admin

Location              : West US

Name                  : BigCAT30

State                 : Running

StorageAccounts       : {}

SubscriptionId        : {YourSubID}

UserName              : Admin

Version               : 3.0.0.0.661685

VersionStatus         : Compatible

ClusterSizeInNodes    : 4

ConnectionUrl         : https://cgrosstest.azurehdinsight.net

CreateDate            : 5/5/2014 6:09:58 PM

DefaultStorageAccount : cgrosstest.blob.core.windows.net

HttpUserName          : Admin

Location              : West US

Name                  : cgrosstest

State                 : Running

StorageAccounts       : {sqlcatwomanwestus.blob.core.windows.net}

SubscriptionId        : {YourSubID}

UserName              : Admin

Version               : 3.0.2.0.727283

VersionStatus         : Compatible

You can also get information about just one HDInsight cluster at a time:

 Get-AzureHDInsightCluster  -name $ClusterName

Or you can get very granular and look at specific properties, even some that aren’t in the default values:

 write-host '== Default Storage Account:     ' `

(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageAccountName.split(".")[0]

write-host '== Default Container:           ' `

(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageContainerName

This information will be a valuable source of information for tracking past configurations, current usage, and planning. Enjoy your Hadooping!

Sample Script

 # Cindy Gross 2014

# Get HDInsight properties

$SubName = "YourSubscriptionName"

Select-AzureSubscription -SubscriptionName $SubName

Get-AzureSubscription -Current

$ClusterName        = "YourHDInsightClusterName" #HDInsight cluster name

Get-AzureHDInsightProperties 

Get-AzureHDInsightCluster 

Get-AzureHDInsightCluster  -name $ClusterName

write-host '== Default Storage Account:     ' `

(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageAccountName.split(".")[0]

write-host '== Default Container:           ' `

(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageContainerName

write-host '== Max HDInsight Cores for Sub: ' (Get-AzureHDInsightProperties).MaxCoresAllowed

write-host '== Cores Available:             ' (Get-AzureHDInsightProperties).CoresAvailable

write-host '== Cores Used:                  ' (Get-AzureHDInsightProperties).CoresUsed

Share via