Edit

Share via

Invoke-AzHDInsightHiveJob

  • Reference
Module:
Az.HDInsight

Submits a Hive query to an HDInsight cluster and retrieves query results in one operation.

Syntax

Invoke-AzHDInsightHiveJob
      [-Arguments <String[]>]
      [-Files <String[]>]
      [-StatusFolder <String>]
      [-Defines <Hashtable>]
      [-File <String>]
      [-JobName <String>]
      [-Query <String>]
      [-RunAsFileJob]
      [-DefaultContainer <String>]
      [-DefaultStorageAccountName <String>]
      [-DefaultStorageAccountKey <String>]
      [-DefaultProfile <IAzureContextContainer>]
      [<CommonParameters>]

Description

The Invoke-AzHDInsightHiveJob cmdlet submits a Hive query to an Azure HDInsight cluster and retrieves query results in one operation. Use the Use-AzHDInsightCluster cmdlet before calling Invoke-AzHDInsightHiveJob to specify which cluster will be used for the query.

Examples

Example 1: Submit a Hive query to an Azure HDInsight cluster

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value


$storageContainer = "container001"

# Cluster info
$clusterName = "your-hadoop-001"
$clusterCreds = Get-Credential

# Hive job details
$statusFolder = "tempStatusFolder/"
$query = "SHOW TABLES"

Use-AzHDInsightCluster `
            -ClusterCredential $clusterCreds `
            -ClusterName $clusterName

Invoke-AzHDInsightHiveJob -StatusFolder $statusFolder `
            -Query $query `
            -DefaultContainer $storageContainer `
            -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" `
            -DefaultStorageAccountKey $storageAccountKey

This command submits the query SHOW TABLES to the cluster named your-hadoop-001.

Parameters

-Arguments

Specifies an array of arguments for the job. The arguments are passed as command-line arguments to each task.

Type:String[]
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-DefaultContainer

Specifies the name of the default container in the default Azure Storage account that an HDInsight cluster uses.

Type:String
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Type:IAzureContextContainer
Aliases:AzContext, AzureRmContext, AzureCredential
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-DefaultStorageAccountKey

Specifies the account key for the default storage account that the HDInsight cluster uses.

Type:String
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-DefaultStorageAccountName

Specifies the name of the default storage account that the HDInsight cluster uses.

Type:String
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-Defines

Specifies Hadoop configuration values to set when a job runs.

Type:Hashtable
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-File

Specifies the path to a file in Azure Storage that contains the query to run. You can use this parameter instead of the Query parameter.

Type:String
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-Files

Specifies a collection of files that are required for a Hive job.

Type:String[]
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-JobName

Specifies the name of a Hive job. If you do not specify this parameter, this cmdlet uses the default value: "Hive: <first 100 characters of Query>".

Type:String
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-Query

Specifies the Hive query.

Type:String
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-RunAsFileJob

Indicates that this cmdlet creates a file in the default Azure storage account in which to store a query. This cmdlet submits the job that references this file as a script to run. You can use this functionality to handle special characters such as percent sign (%) that would fail on a job submission through Templeton, because Templeton interprets a query with a percent sign as a URL parameter.

Type:SwitchParameter
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

-StatusFolder

Specifies the location of the folder that contains standard outputs and error outputs for a job.

Type:String
Position:Named
Default value:None
Required:False
Accept pipeline input:False
Accept wildcard characters:False

Inputs

None

Outputs

String