Share via

Azure HDInsight: PowerShell Commands

  • Article

While working on PowerShell scripts, make it a habit to define/store the cluster name, storage name, etc. in some relevant/meaningful variable names. Use these variables as and when required.

  1. To associate a Windows Azure Subscription:
    1. Add-AzureAccount
  2. To see the already associated account
    1. Get-AzureAccount
  3. To define a variable and assign value to that variable
    1. $<VariableName> = "Value"
    2. For Example, $subscriptionName = “Free Trial”
  4. To see your subscription
    1. Get-AzureSubscription
  5. To assign a storage key to a variable:
    1. $storageAccountKey = Get-AzureStorageKey $storageAccountName | %{ $_.Primary }
  6. To select a particular subscription at the time of usage. It is mostly used when you have added multiple subscriptions:
    1. Select-AzureSubscription $subscriptionName
  7. To define a MapReduce Job, just define a variable and assign it a value using “New-AzureHDInsightMapReduceJobDefinition” function along with some parameters. A sample is shown below:
    1. $MapReduceDefinition = New-AzureHDInsightMapReduceJobDefinition -JarFile "<location of JAR file (MapReduce)>" -Arguments "<Path to source file>", "<Path to output folder>"
  8. To execute the above defined MapReduce job, use the following command:
    1. $executeJob = Start-AzureHDInsightJob -Cluster <clustername> -JobDefinition $MapReduceDefinition
  9. To view the status of the job execution, use:
    1. Wait-AzureHDInsightJob -Job $executeJob -WaitTimeoutInSeconds 3600
  10. To run a Hive Jobs thru PowerShell, use the following command:
    1. $CreateTableVar = "DROP TABLE WordCount;" + "CREATE EXTERNAL TABLE WordCount(c1 string, c2 int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE LOCATION '<Path to the source / part file>';"
  11. To define the HIVE query, use:
    1. $DefiningHive = New-AzureHDInsightHiveJobDefinition -Query $CreateTableVar
  12. To execute the HIVE job, use:
    1. $hiveJob = Start-AzureHDInsightJob -Cluster <name of the cluster> -JobDefinition $DefiningHive
  13. Once the table is created, you can start querying into the content of that table. Use the following command:
    1. Query without getting the status displayed
      1. invoke-Hive -Query "SELECT * FROM WordCount LIMIT 10; "
    2. Query with the status getting displayed

      1. invoke-Hive -Query @"

        SELECT * FROM WordCount LIMIT 10;

        "@

       

This is just a consolidated list of PowerShell commands developed by leveraging the Azure HDInsight guides.

This list of PowerShell command goes on and on. Let's try to cover as many HDInsight specific commands as possible in this article.

See Also