Add-AzureRmHDInsightConfigValues

Module:

AzureRM.HDInsight

Adds a Hadoop configuration value customization and/or a Hive shared library customization to a cluster configuration object.

Warning

The AzureRM PowerShell module has been officially deprecated as of February 29, 2024. Users are advised to migrate from AzureRM to the Az PowerShell module to ensure continued support and updates.

Although the AzureRM module may still function, it's no longer maintained or supported, placing any continued use at the user's discretion and risk. Please refer to our migration resources for guidance on transitioning to the Az module.

Syntax

Add-AzureRmHDInsightConfigValues
   [-Config] <AzureHDInsightConfig>
   [-Core <Hashtable>]
   [-HiveSite <Hashtable>]
   [-HiveEnv <Hashtable>]
   [-OozieSite <Hashtable>]
   [-OozieEnv <Hashtable>]
   [-WebHCat <Hashtable>]
   [-HBaseSite <Hashtable>]
   [-HBaseEnv <Hashtable>]
   [-Storm <Hashtable>]
   [-Yarn <Hashtable>]
   [-MapRed <Hashtable>]
   [-Tez <Hashtable>]
   [-Hdfs <Hashtable>]
   [-RServer <Hashtable>]
   [-SparkDefaults <Hashtable>]
   [-SparkThriftConf <Hashtable>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]

Add-AzureRmHDInsightConfigValues
   [-Config] <AzureHDInsightConfig>
   [-Core <Hashtable>]
   [-HiveSite <Hashtable>]
   [-HiveEnv <Hashtable>]
   [-OozieSite <Hashtable>]
   [-OozieEnv <Hashtable>]
   [-WebHCat <Hashtable>]
   [-HBaseSite <Hashtable>]
   [-HBaseEnv <Hashtable>]
   [-Storm <Hashtable>]
   [-Yarn <Hashtable>]
   [-MapRed <Hashtable>]
   [-Tez <Hashtable>]
   [-Hdfs <Hashtable>]
   [-RServer <Hashtable>]
   [-Spark2Defaults <Hashtable>]
   [-Spark2ThriftConf <Hashtable>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]

Description

The Add-AzureRmHDInsightConfigValues cmdlet adds a Hadoop configuration value customization, such as core-site.xml or hive-site.xml, and/or a Hive shared library customization to the HDInsight configuration object created by the New-AzureRmHDInsightClusterConfig cmdlet.

Examples

Example 1: Add a custom configuration value to the cluster configuration object

PS C:\># Primary storage account info
PS C:\> $storageAccountResourceGroupName = "Group"
PS C:\> $storageAccountName = "yourstorageacct001"
PS C:\> $storageAccountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value

PS C:\> $storageContainer = "container001"

# Cluster configuration info
PS C:\> $location = "East US 2"
PS C:\> $clusterResourceGroupName = "Group"
PS C:\> $clusterName = "your-hadoop-001"
PS C:\> $clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
#   New-AzureRmResourceGroup -Name $clusterResourceGroupName -Location $location

# Config values
PS C:\> $coreConfigs = @{"io.file.buffer.size"="300000"}
PS C:\> $mapRedConfigs = @{"mapred.map.max.attempts"="2"}

# Create the cluster
PS C:\> New-AzureRmHDInsightClusterConfig `
            | Add-AzureRmHDInsightConfigValues `
                -Core $coreConfigs `
                -MapRed $mapRedConfigs `
            | New-AzureRmHDInsightCluster `
                -ClusterType Hadoop `
                -OSType Windows `
                -ClusterSizeInNodes 4 `
                -ResourceGroupName $clusterResourceGroupName `
                -ClusterName $clusterName `
                -HttpCredential $clusterCreds `
                -Location $location `
                -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" `
                -DefaultStorageAccountKey $storageAccountKey `
                -DefaultStorageContainer $storageAccountContainer

This command adds a Hadoop configuration value to the cluster named your-hadoop-001.

Parameters

-Config

Specifies the HDInsight cluster configuration object that this cmdlet modifies. This object is created by the New-AzureRmHDInsightClusterConfig cmdlet.

Type:	AzureHDInsightConfig
Position:	0
Default value:	None
Required:	True
Accept pipeline input:	True
Accept wildcard characters:	False

-Core

Specifies the Core Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Type:	IAzureContextContainer
Aliases:	AzureRmContext, AzureCredential
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-HBaseEnv

Specifies the HBase Env configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-HBaseSite

Specifies the HBase Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Hdfs

Specifies the HDFS configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-HiveEnv

Specifies the Hive Env configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-HiveSite

Specifies the Hive Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-MapRed

Specifies the MapRed Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-OozieEnv

Specifies the Oozie Env configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-OozieSite

Specifies the Oozie Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-RServer

Specifies the RServer configurations. Valid only for RServer clusters.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Spark2Defaults

Specifies the Spark2 Defaults configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Spark2ThriftConf

Specifies the Spark2 Thrift SparkConf configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-SparkDefaults

Specifies the Spark Defaults configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-SparkThriftConf

Specifies the Spark Thrift SparkConf configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Storm

Specifies the Storm Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Tez

Specifies the Tez Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-WebHCat

Specifies the WebHCat Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Yarn

Specifies the YARN Site configurations of this HDInsight cluster.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

Inputs

AzureHDInsightConfig

Parameters: Config (ByValue)

Outputs

AzureHDInsightConfig

Related Links

• New-AzureRmHDInsightClusterConfig