Big Data Pools - Create Or Update

Create a Big Data pool.
Create a new Big Data pool.

PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}?api-version=2021-06-01-preview
PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}?api-version=2021-06-01-preview&force={force}

URI Parameters

Name In Required Type Description
bigDataPoolName
path True

string

Big Data pool name

resourceGroupName
path True

string

The name of the resource group. The name is case insensitive.

subscriptionId
path True

string

The ID of the target subscription.

workspaceName
path True

string

The name of the workspace.

api-version
query True

string

The API version to use for this operation.

force
query

boolean

Whether to stop any running jobs in the Big Data pool

Request Body

Name Required Type Description
location True

string

The geo-location where the resource lives

properties.autoPause

AutoPauseProperties

Spark pool auto-pausing properties
Auto-pausing properties

properties.autoScale

AutoScaleProperties

Spark pool auto-scaling properties
Auto-scaling properties

properties.cacheSize

integer

The cache size

properties.customLibraries

LibraryInfo[]

List of custom libraries/packages associated with the spark pool.

properties.defaultSparkLogFolder

string

The default folder where Spark logs will be written.

properties.dynamicExecutorAllocation

DynamicExecutorAllocation

Dynamic Executor Allocation

properties.isAutotuneEnabled

boolean

Enable Autotune
Whether autotune is required or not.

properties.isComputeIsolationEnabled

boolean

Whether compute isolation is required or not.

properties.libraryRequirements

LibraryRequirements

Spark pool library version requirements
Library version requirements

properties.nodeCount

integer

The number of nodes in the Big Data pool.

properties.nodeSize

NodeSize

The level of compute power that each node in the Big Data pool has.

properties.nodeSizeFamily

NodeSizeFamily

The kind of nodes that the Big Data pool provides.

properties.provisioningState

string

The state of the Big Data pool.

properties.sessionLevelPackagesEnabled

boolean

Whether session level packages enabled.

properties.sparkConfigProperties

SparkConfigProperties

Spark pool Config Properties
Spark configuration file to specify additional properties

properties.sparkEventsFolder

string

The Spark events folder

properties.sparkVersion

string

The Apache Spark version.

tags

object

Resource tags.

Responses

Name Type Description
200 OK

BigDataPoolResourceInfo

OK

202 Accepted

BigDataPoolResourceInfo

Other Status Codes

ErrorResponse

Examples

Create or update a Big Data pool

Sample request

PUT https://management.azure.com/subscriptions/01234567-89ab-4def-0123-456789abcdef/resourceGroups/ExampleResourceGroup/providers/Microsoft.Synapse/workspaces/ExampleWorkspace/bigDataPools/ExamplePool?api-version=2021-06-01-preview

{
  "tags": {
    "key": "value"
  },
  "location": "West US 2",
  "properties": {
    "sparkVersion": "3.3",
    "nodeCount": 4,
    "nodeSize": "Medium",
    "nodeSizeFamily": "MemoryOptimized",
    "autoScale": {
      "enabled": true,
      "minNodeCount": 3,
      "maxNodeCount": 50
    },
    "autoPause": {
      "enabled": true,
      "delayInMinutes": 15
    },
    "sparkEventsFolder": "/events",
    "libraryRequirements": {
      "content": "",
      "filename": "requirements.txt"
    },
    "defaultSparkLogFolder": "/logs",
    "isAutotuneEnabled": false
  }
}

Sample response

{
  "id": "/subscriptions/01234567-89ab-4def-0123-456789abcdef/resourceGroups/ExampleResourceGroup/providers/Microsoft.Synapse/workspaces/ExampleWorkspace/bigDataPools/ExamplePool",
  "type": "Microsoft.Synapse/workspaces/bigDataPools",
  "location": "West US 2",
  "name": "ExamplePool",
  "tags": {
    "key": "value"
  },
  "properties": {
    "provisioningState": "Provisioning",
    "sparkVersion": "3.3",
    "nodeCount": 4,
    "nodeSize": "Medium",
    "nodeSizeFamily": "MemoryOptimized",
    "autoScale": {
      "enabled": true,
      "minNodeCount": 3,
      "maxNodeCount": 50
    },
    "autoPause": {
      "enabled": true,
      "delayInMinutes": 15
    },
    "creationDate": "1970-01-01T00:00:00Z",
    "sparkEventsFolder": "/events",
    "libraryRequirements": {
      "time": "1970-01-01T00:00:00Z",
      "content": "",
      "filename": "requirements.txt"
    },
    "defaultSparkLogFolder": "/logs",
    "lastSucceededTimestamp": "1970-01-01T10:00:00Z",
    "isAutotuneEnabled": false
  }
}
{
  "id": "/subscriptions/01234567-89ab-4def-0123-456789abcdef/resourceGroups/ExampleResourceGroup/providers/Microsoft.Synapse/workspaces/ExampleWorkspace/bigDataPools/ExamplePool",
  "type": "Microsoft.Synapse/workspaces/bigDataPools",
  "location": "West US 2",
  "name": "ExamplePool",
  "tags": {
    "key": "value"
  },
  "properties": {
    "provisioningState": "Provisioning",
    "sparkVersion": "3.3",
    "nodeCount": 4,
    "nodeSize": "Medium",
    "nodeSizeFamily": "MemoryOptimized",
    "autoScale": {
      "enabled": true,
      "minNodeCount": 3,
      "maxNodeCount": 50
    },
    "autoPause": {
      "enabled": true,
      "delayInMinutes": 15
    },
    "creationDate": "1970-01-01T00:00:00Z",
    "sparkEventsFolder": "/events",
    "libraryRequirements": {
      "time": "1970-01-01T00:00:00Z",
      "content": "",
      "filename": "requirements.txt"
    },
    "defaultSparkLogFolder": "/logs",
    "isAutotuneEnabled": false
  }
}
{
  "error": {
    "code": "Error code",
    "message": "Error message"
  }
}

Definitions

Name Description
AutoPauseProperties

Spark pool auto-pausing properties

AutoScaleProperties

Spark pool auto-scaling properties

BigDataPoolResourceInfo

Big Data pool

ConfigurationType

The type of the spark config properties file.

DynamicExecutorAllocation

Dynamic Executor Allocation Properties

ErrorAdditionalInfo

The resource management error additional info.

ErrorDetail

The error detail.

ErrorResponse

Error response

LibraryInfo

Information about a library/package created at the workspace level.

LibraryRequirements

Spark pool library version requirements

NodeSize

The level of compute power that each node in the Big Data pool has.

NodeSizeFamily

The kind of nodes that the Big Data pool provides.

SparkConfigProperties

Spark pool Config Properties

AutoPauseProperties

Spark pool auto-pausing properties

Name Type Description
delayInMinutes

integer

Number of minutes of idle time before the Big Data pool is automatically paused.

enabled

boolean

Whether auto-pausing is enabled for the Big Data pool.

AutoScaleProperties

Spark pool auto-scaling properties

Name Type Description
enabled

boolean

Whether automatic scaling is enabled for the Big Data pool.

maxNodeCount

integer

The maximum number of nodes the Big Data pool can support.

minNodeCount

integer

The minimum number of nodes the Big Data pool can support.

BigDataPoolResourceInfo

Big Data pool

Name Type Description
id

string

Fully qualified resource ID for the resource. Ex - /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}

location

string

The geo-location where the resource lives

name

string

The name of the resource

properties.autoPause

AutoPauseProperties

Spark pool auto-pausing properties
Auto-pausing properties

properties.autoScale

AutoScaleProperties

Spark pool auto-scaling properties
Auto-scaling properties

properties.cacheSize

integer

The cache size

properties.creationDate

string

The time when the Big Data pool was created.

properties.customLibraries

LibraryInfo[]

List of custom libraries/packages associated with the spark pool.

properties.defaultSparkLogFolder

string

The default folder where Spark logs will be written.

properties.dynamicExecutorAllocation

DynamicExecutorAllocation

Dynamic Executor Allocation

properties.isAutotuneEnabled

boolean

Enable Autotune
Whether autotune is required or not.

properties.isComputeIsolationEnabled

boolean

Whether compute isolation is required or not.

properties.lastSucceededTimestamp

string

The time when the Big Data pool was updated successfully.

properties.libraryRequirements

LibraryRequirements

Spark pool library version requirements
Library version requirements

properties.nodeCount

integer

The number of nodes in the Big Data pool.

properties.nodeSize

NodeSize

The level of compute power that each node in the Big Data pool has.

properties.nodeSizeFamily

NodeSizeFamily

The kind of nodes that the Big Data pool provides.

properties.provisioningState

string

The state of the Big Data pool.

properties.sessionLevelPackagesEnabled

boolean

Whether session level packages enabled.

properties.sparkConfigProperties

SparkConfigProperties

Spark pool Config Properties
Spark configuration file to specify additional properties

properties.sparkEventsFolder

string

The Spark events folder

properties.sparkVersion

string

The Apache Spark version.

tags

object

Resource tags.

type

string

The type of the resource. E.g. "Microsoft.Compute/virtualMachines" or "Microsoft.Storage/storageAccounts"

ConfigurationType

The type of the spark config properties file.

Name Type Description
Artifact

string

File

string

DynamicExecutorAllocation

Dynamic Executor Allocation Properties

Name Type Description
enabled

boolean

Indicates whether Dynamic Executor Allocation is enabled or not.

maxExecutors

integer

The maximum number of executors alloted

minExecutors

integer

The minimum number of executors alloted

ErrorAdditionalInfo

The resource management error additional info.

Name Type Description
info

object

The additional info.

type

string

The additional info type.

ErrorDetail

The error detail.

Name Type Description
additionalInfo

ErrorAdditionalInfo[]

The error additional info.

code

string

The error code.

details

ErrorDetail[]

The error details.

message

string

The error message.

target

string

The error target.

ErrorResponse

Error response

Name Type Description
error

ErrorDetail

The error object.

LibraryInfo

Information about a library/package created at the workspace level.

Name Type Description
containerName

string

Storage blob container name.

creatorId

string

Creator Id of the library/package.

name

string

Name of the library.

path

string

Storage blob path of library.

provisioningStatus

string

Provisioning status of the library/package.

type

string

Type of the library.

uploadedTimestamp

string

The last update time of the library.

LibraryRequirements

Spark pool library version requirements

Name Type Description
content

string

The library requirements.

filename

string

The filename of the library requirements file.

time

string

The last update time of the library requirements file.

NodeSize

The level of compute power that each node in the Big Data pool has.

Name Type Description
Large

string

Medium

string

None

string

Small

string

XLarge

string

XXLarge

string

XXXLarge

string

NodeSizeFamily

The kind of nodes that the Big Data pool provides.

Name Type Description
HardwareAcceleratedFPGA

string

HardwareAcceleratedGPU

string

MemoryOptimized

string

None

string

SparkConfigProperties

Spark pool Config Properties

Name Type Description
configurationType

ConfigurationType

The type of the spark config properties file.

content

string

The spark config properties.

filename

string

The filename of the spark config properties file.

time

string

The last update time of the spark config properties file.