Share via

az synapse spark pool

  • Reference

Manage Spark pools.

Commands

Name Description Type Status
az synapse spark pool create

Create a Spark pool.

 Core GA
az synapse spark pool delete

Delete a Spark pool.

 Core GA
az synapse spark pool list

List all Spark pools.

 Core GA
az synapse spark pool show

Get a Spark pool.

 Core GA
az synapse spark pool update

Update the Spark pool.

 Core GA
az synapse spark pool wait

Place the CLI in a waiting state until a condition of a Spark pool is met.

 Core GA

az synapse spark pool create

Edit

Create a Spark pool.

az synapse spark pool create --name
                             --node-count
                             --node-size {Large, Medium, None, Small, XLarge, XXLarge, XXXLarge}
                             --resource-group
                             --spark-version
                             --workspace-name
                             [--delay]
                             [--enable-auto-pause {false, true}]
                             [--enable-auto-scale {false, true}]
                             [--enable-dynamic-exec {false, true}]
                             [--max-executors]
                             [--max-node-count]
                             [--min-executors]
                             [--min-node-count]
                             [--no-wait]
                             [--node-size-family {HardwareAcceleratedFPGA, HardwareAcceleratedGPU, MemoryOptimized, None}]
                             [--spark-config-file-path]
                             [--spark-events-folder]
                             [--spark-log-folder]
                             [--tags]

Examples

Create a Spark pool.

az synapse spark pool create --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--spark-version 2.4 --node-count 3 --node-size Medium --spark-config-file-path 'path/configfile.txt'

Required Parameters

--name -n

The name of the Spark pool.

--node-count

The number of node.

--node-size

The level of compute power that each node in the Big Data pool has..

Accepted values: Large, Medium, None, Small, XLarge, XXLarge, XXXLarge
--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--spark-version

The supported Spark version is 2.4 now.

--workspace-name

The workspace name.

Optional Parameters

--delay

The delay time whose unit is minute.

--enable-auto-pause

The flag of enabling auto pause.

Accepted values: false, true
--enable-auto-scale

The flag of enabling auto scale.

Accepted values: false, true
--enable-dynamic-exec

Indicates whether Dynamic Executor Allocation is enabled or not.

Accepted values: false, true
--max-executors

The maximum number of executors alloted.

--max-node-count

The max node count.

--min-executors

The minimum number of executors alloted.

--min-node-count

The min node count.

--no-wait

Do not wait for the long-running operation to finish.

Default value: False
--node-size-family

The kind of nodes that the Big Data pool provides.

Accepted values: HardwareAcceleratedFPGA, HardwareAcceleratedGPU, MemoryOptimized, None
Default value: MemoryOptimized
--spark-config-file-path

Absolute path of Spark pool properties configuration file.

--spark-events-folder

The Spark events folder.

Default value: /events
--spark-log-folder

The default Spark log folder.

Default value: /logs
--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark pool delete

Edit

Delete a Spark pool.

az synapse spark pool delete [--ids]
                             [--name]
                             [--no-wait]
                             [--resource-group]
                             [--subscription]
                             [--workspace-name]
                             [--yes]

Examples

Delete a Spark pool.

az synapse spark pool delete --name testpool --workspace-name testsynapseworkspace --resource-group rg

Optional Parameters

--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

--name -n

The name of the Spark pool.

--no-wait

Do not wait for the long-running operation to finish.

Default value: False
--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--workspace-name

The workspace name.

--yes -y

Do not prompt for confirmation.

Default value: False
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark pool list

Edit

List all Spark pools.

az synapse spark pool list --resource-group
                           --workspace-name

Examples

List all Spark pools.

az synapse spark pool list --workspace-name testsynapseworkspace --resource-group rg

Required Parameters

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name

The workspace name.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark pool show

Edit

Get a Spark pool.

az synapse spark pool show [--ids]
                           [--name]
                           [--resource-group]
                           [--subscription]
                           [--workspace-name]

Examples

Get a Spark pool.

az synapse spark pool show --name testpool --workspace-name testsynapseworkspace --resource-group rg

Optional Parameters

--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

--name -n

The name of the Spark pool.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--workspace-name

The workspace name.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark pool update

Edit

Update the Spark pool.

az synapse spark pool update [--delay]
                             [--enable-auto-pause {false, true}]
                             [--enable-auto-scale {false, true}]
                             [--enable-dynamic-exec {false, true}]
                             [--force {false, true}]
                             [--ids]
                             [--library-requirements]
                             [--max-executors]
                             [--max-node-count]
                             [--min-executors]
                             [--min-node-count]
                             [--name]
                             [--no-wait]
                             [--node-count]
                             [--node-size {Large, Medium, None, Small, XLarge, XXLarge, XXXLarge}]
                             [--package]
                             [--package-action {Add, Remove}]
                             [--resource-group]
                             [--spark-config-file-path]
                             [--subscription]
                             [--tags]
                             [--workspace-name]

Examples

Update the Spark pool's tags.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--tags key1=value1

Update the Spark pool's auto scale configuration.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--enable-auto-scale --min-node-count 3 --max-node-count 100

Update the Spark pool's custom libraries.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--package-action Add --package package1.jar package2.jar

Update the Spark pool's configuration file.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--spark-config-file-path 'path/configfile.txt'

Update the Spark pool's dynamic executor allocation configuration.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--enable-dynamic-exec --min-executors 3 --max-executors 10

Optional Parameters

--delay

The delay time whose unit is minute.

--enable-auto-pause

The flag of enabling auto pause.

Accepted values: false, true
--enable-auto-scale

The flag of enabling auto scale.

Accepted values: false, true
--enable-dynamic-exec

Indicates whether Dynamic Executor Allocation is enabled or not.

Accepted values: false, true
--force

The flag of force operation.

Accepted values: false, true
Default value: False
--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

--library-requirements

The library requirements file.

--max-executors

The maximum number of executors alloted.

--max-node-count

The max node count.

--min-executors

The minimum number of executors alloted.

--min-node-count

The min node count.

--name -n

The name of the Spark pool.

--no-wait

Do not wait for the long-running operation to finish.

Default value: False
--node-count

The number of node.

--node-size

The level of compute power that each node in the Big Data pool has..

Accepted values: Large, Medium, None, Small, XLarge, XXLarge, XXXLarge
--package

List of workspace packages name.

--package-action

Package action must be specified when you add or remove a workspace package from a Apache Spark pool.

Accepted values: Add, Remove
--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--spark-config-file-path

Absolute path of Spark pool properties configuration file.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.

--workspace-name

The workspace name.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark pool wait

Edit

Place the CLI in a waiting state until a condition of a Spark pool is met.

az synapse spark pool wait --big-data-pool-name
                           [--created]
                           [--custom]
                           [--deleted]
                           [--exists]
                           [--ids]
                           [--interval]
                           [--resource-group]
                           [--subscription]
                           [--timeout]
                           [--updated]
                           [--workspace-name]

Required Parameters

--big-data-pool-name

Big Data pool name.

Optional Parameters

--created

Wait until created with 'provisioningState' at 'Succeeded'.

Default value: False
--custom

Wait until the condition satisfies a custom JMESPath query. E.g. provisioningState!='InProgress', instanceView.statuses[?code=='PowerState/running'].

--deleted

Wait until deleted.

Default value: False
--exists

Wait until the resource exists.

Default value: False
--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

--interval

Polling interval in seconds.

Default value: 30
--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--timeout

Maximum wait in seconds.

Default value: 3600
--updated

Wait until updated with provisioningState at 'Succeeded'.

Default value: False
--workspace-name

The workspace name.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.