Big Data Pools - Create Or Update
Create a Big Data pool.
Create a new Big Data pool.
PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}?api-version=2021-06-01-preview
PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}?api-version=2021-06-01-preview&force={force}
URI Parameters
Name | In | Required | Type | Description |
---|---|---|---|---|
big
|
path | True |
string |
Big Data pool name |
resource
|
path | True |
string |
The name of the resource group. The name is case insensitive. |
subscription
|
path | True |
string |
The ID of the target subscription. |
workspace
|
path | True |
string |
The name of the workspace. |
api-version
|
query | True |
string |
The API version to use for this operation. |
force
|
query |
boolean |
Whether to stop any running jobs in the Big Data pool |
Request Body
Name | Required | Type | Description |
---|---|---|---|
location | True |
string |
The geo-location where the resource lives |
properties.autoPause |
Spark pool auto-pausing properties |
||
properties.autoScale |
Spark pool auto-scaling properties |
||
properties.cacheSize |
integer |
The cache size |
|
properties.customLibraries |
List of custom libraries/packages associated with the spark pool. |
||
properties.defaultSparkLogFolder |
string |
The default folder where Spark logs will be written. |
|
properties.dynamicExecutorAllocation |
Dynamic Executor Allocation |
||
properties.isAutotuneEnabled |
boolean |
Enable Autotune |
|
properties.isComputeIsolationEnabled |
boolean |
Whether compute isolation is required or not. |
|
properties.libraryRequirements |
Spark pool library version requirements |
||
properties.nodeCount |
integer |
The number of nodes in the Big Data pool. |
|
properties.nodeSize |
The level of compute power that each node in the Big Data pool has. |
||
properties.nodeSizeFamily |
The kind of nodes that the Big Data pool provides. |
||
properties.provisioningState |
string |
The state of the Big Data pool. |
|
properties.sessionLevelPackagesEnabled |
boolean |
Whether session level packages enabled. |
|
properties.sparkConfigProperties |
Spark pool Config Properties |
||
properties.sparkEventsFolder |
string |
The Spark events folder |
|
properties.sparkVersion |
string |
The Apache Spark version. |
|
tags |
object |
Resource tags. |
Responses
Name | Type | Description |
---|---|---|
200 OK |
OK |
|
202 Accepted | ||
Other Status Codes |
Examples
Create or update a Big Data pool
Sample request
PUT https://management.azure.com/subscriptions/01234567-89ab-4def-0123-456789abcdef/resourceGroups/ExampleResourceGroup/providers/Microsoft.Synapse/workspaces/ExampleWorkspace/bigDataPools/ExamplePool?api-version=2021-06-01-preview
{
"tags": {
"key": "value"
},
"location": "West US 2",
"properties": {
"sparkVersion": "3.3",
"nodeCount": 4,
"nodeSize": "Medium",
"nodeSizeFamily": "MemoryOptimized",
"autoScale": {
"enabled": true,
"minNodeCount": 3,
"maxNodeCount": 50
},
"autoPause": {
"enabled": true,
"delayInMinutes": 15
},
"sparkEventsFolder": "/events",
"libraryRequirements": {
"content": "",
"filename": "requirements.txt"
},
"defaultSparkLogFolder": "/logs",
"isAutotuneEnabled": false
}
}
Sample response
{
"id": "/subscriptions/01234567-89ab-4def-0123-456789abcdef/resourceGroups/ExampleResourceGroup/providers/Microsoft.Synapse/workspaces/ExampleWorkspace/bigDataPools/ExamplePool",
"type": "Microsoft.Synapse/workspaces/bigDataPools",
"location": "West US 2",
"name": "ExamplePool",
"tags": {
"key": "value"
},
"properties": {
"provisioningState": "Provisioning",
"sparkVersion": "3.3",
"nodeCount": 4,
"nodeSize": "Medium",
"nodeSizeFamily": "MemoryOptimized",
"autoScale": {
"enabled": true,
"minNodeCount": 3,
"maxNodeCount": 50
},
"autoPause": {
"enabled": true,
"delayInMinutes": 15
},
"creationDate": "1970-01-01T00:00:00Z",
"sparkEventsFolder": "/events",
"libraryRequirements": {
"time": "1970-01-01T00:00:00Z",
"content": "",
"filename": "requirements.txt"
},
"defaultSparkLogFolder": "/logs",
"lastSucceededTimestamp": "1970-01-01T10:00:00Z",
"isAutotuneEnabled": false
}
}
{
"id": "/subscriptions/01234567-89ab-4def-0123-456789abcdef/resourceGroups/ExampleResourceGroup/providers/Microsoft.Synapse/workspaces/ExampleWorkspace/bigDataPools/ExamplePool",
"type": "Microsoft.Synapse/workspaces/bigDataPools",
"location": "West US 2",
"name": "ExamplePool",
"tags": {
"key": "value"
},
"properties": {
"provisioningState": "Provisioning",
"sparkVersion": "3.3",
"nodeCount": 4,
"nodeSize": "Medium",
"nodeSizeFamily": "MemoryOptimized",
"autoScale": {
"enabled": true,
"minNodeCount": 3,
"maxNodeCount": 50
},
"autoPause": {
"enabled": true,
"delayInMinutes": 15
},
"creationDate": "1970-01-01T00:00:00Z",
"sparkEventsFolder": "/events",
"libraryRequirements": {
"time": "1970-01-01T00:00:00Z",
"content": "",
"filename": "requirements.txt"
},
"defaultSparkLogFolder": "/logs",
"isAutotuneEnabled": false
}
}
{
"error": {
"code": "Error code",
"message": "Error message"
}
}
Definitions
Name | Description |
---|---|
Auto |
Spark pool auto-pausing properties |
Auto |
Spark pool auto-scaling properties |
Big |
Big Data pool |
Configuration |
The type of the spark config properties file. |
Dynamic |
Dynamic Executor Allocation Properties |
Error |
The resource management error additional info. |
Error |
The error detail. |
Error |
Error response |
Library |
Information about a library/package created at the workspace level. |
Library |
Spark pool library version requirements |
Node |
The level of compute power that each node in the Big Data pool has. |
Node |
The kind of nodes that the Big Data pool provides. |
Spark |
Spark pool Config Properties |
AutoPauseProperties
Spark pool auto-pausing properties
Name | Type | Description |
---|---|---|
delayInMinutes |
integer |
Number of minutes of idle time before the Big Data pool is automatically paused. |
enabled |
boolean |
Whether auto-pausing is enabled for the Big Data pool. |
AutoScaleProperties
Spark pool auto-scaling properties
Name | Type | Description |
---|---|---|
enabled |
boolean |
Whether automatic scaling is enabled for the Big Data pool. |
maxNodeCount |
integer |
The maximum number of nodes the Big Data pool can support. |
minNodeCount |
integer |
The minimum number of nodes the Big Data pool can support. |
BigDataPoolResourceInfo
Big Data pool
Name | Type | Description |
---|---|---|
id |
string |
Fully qualified resource ID for the resource. Ex - /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName} |
location |
string |
The geo-location where the resource lives |
name |
string |
The name of the resource |
properties.autoPause |
Spark pool auto-pausing properties |
|
properties.autoScale |
Spark pool auto-scaling properties |
|
properties.cacheSize |
integer |
The cache size |
properties.creationDate |
string |
The time when the Big Data pool was created. |
properties.customLibraries |
List of custom libraries/packages associated with the spark pool. |
|
properties.defaultSparkLogFolder |
string |
The default folder where Spark logs will be written. |
properties.dynamicExecutorAllocation |
Dynamic Executor Allocation |
|
properties.isAutotuneEnabled |
boolean |
Enable Autotune |
properties.isComputeIsolationEnabled |
boolean |
Whether compute isolation is required or not. |
properties.lastSucceededTimestamp |
string |
The time when the Big Data pool was updated successfully. |
properties.libraryRequirements |
Spark pool library version requirements |
|
properties.nodeCount |
integer |
The number of nodes in the Big Data pool. |
properties.nodeSize |
The level of compute power that each node in the Big Data pool has. |
|
properties.nodeSizeFamily |
The kind of nodes that the Big Data pool provides. |
|
properties.provisioningState |
string |
The state of the Big Data pool. |
properties.sessionLevelPackagesEnabled |
boolean |
Whether session level packages enabled. |
properties.sparkConfigProperties |
Spark pool Config Properties |
|
properties.sparkEventsFolder |
string |
The Spark events folder |
properties.sparkVersion |
string |
The Apache Spark version. |
tags |
object |
Resource tags. |
type |
string |
The type of the resource. E.g. "Microsoft.Compute/virtualMachines" or "Microsoft.Storage/storageAccounts" |
ConfigurationType
The type of the spark config properties file.
Name | Type | Description |
---|---|---|
Artifact |
string |
|
File |
string |
DynamicExecutorAllocation
Dynamic Executor Allocation Properties
Name | Type | Description |
---|---|---|
enabled |
boolean |
Indicates whether Dynamic Executor Allocation is enabled or not. |
maxExecutors |
integer |
The maximum number of executors alloted |
minExecutors |
integer |
The minimum number of executors alloted |
ErrorAdditionalInfo
The resource management error additional info.
Name | Type | Description |
---|---|---|
info |
object |
The additional info. |
type |
string |
The additional info type. |
ErrorDetail
The error detail.
Name | Type | Description |
---|---|---|
additionalInfo |
The error additional info. |
|
code |
string |
The error code. |
details |
The error details. |
|
message |
string |
The error message. |
target |
string |
The error target. |
ErrorResponse
Error response
Name | Type | Description |
---|---|---|
error |
The error object. |
LibraryInfo
Information about a library/package created at the workspace level.
Name | Type | Description |
---|---|---|
containerName |
string |
Storage blob container name. |
creatorId |
string |
Creator Id of the library/package. |
name |
string |
Name of the library. |
path |
string |
Storage blob path of library. |
provisioningStatus |
string |
Provisioning status of the library/package. |
type |
string |
Type of the library. |
uploadedTimestamp |
string |
The last update time of the library. |
LibraryRequirements
Spark pool library version requirements
Name | Type | Description |
---|---|---|
content |
string |
The library requirements. |
filename |
string |
The filename of the library requirements file. |
time |
string |
The last update time of the library requirements file. |
NodeSize
The level of compute power that each node in the Big Data pool has.
Name | Type | Description |
---|---|---|
Large |
string |
|
Medium |
string |
|
None |
string |
|
Small |
string |
|
XLarge |
string |
|
XXLarge |
string |
|
XXXLarge |
string |
NodeSizeFamily
The kind of nodes that the Big Data pool provides.
Name | Type | Description |
---|---|---|
HardwareAcceleratedFPGA |
string |
|
HardwareAcceleratedGPU |
string |
|
MemoryOptimized |
string |
|
None |
string |
SparkConfigProperties
Spark pool Config Properties
Name | Type | Description |
---|---|---|
configurationType |
The type of the spark config properties file. |
|
content |
string |
The spark config properties. |
filename |
string |
The filename of the spark config properties file. |
time |
string |
The last update time of the spark config properties file. |