Spark Batch - Create Spark Batch Job

Service:

Synapse

API Version:

2020-12-01

Create new spark batch job.

POST {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches

With optional parameters:

POST {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches?detailed={detailed}

URI Parameters

Name	In	Required	Type	Description
endpoint	path	True	string	The workspace development endpoint, for example https://myworkspace.dev.azuresynapse.net.
livyApiVersion	path	True	string	Valid api-version for the request.
sparkPoolName	path	True	string	Name of the spark pool.
detailed	query		boolean	Optional query param specifying whether detailed response is returned beyond plain livy.

Request Body

Name	Required	Type	Description
file	True	string
name	True	string
archives		string[]
args		string[]
artifactId		string
className		string
conf		object
driverCores		integer
driverMemory		string
executorCores		integer
executorMemory		string
files		string[]
jars		string[]
numExecutors		integer
pyFiles		string[]
tags		object

Responses

Name	Type	Description
200 OK	SparkBatchJob	Success

Examples

Create new spark batch job.

Sample request

HTTP

POST myWorkspace.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/mySparkPool/batches?detailed=True

{
  "tags": null,
  "artifactId": "fill in here",
  "name": "jobname",
  "file": "abfss://",
  "className": "classname",
  "args": [],
  "jars": [],
  "files": [],
  "archives": [],
  "conf": null,
  "driverMemory": "4g",
  "driverCores": 4,
  "executorMemory": "2g",
  "executorCores": 4,
  "numExecutors": 2
}

Sample response

Status code:

200

{
  "livyInfo": null,
  "name": "jobname",
  "workspaceName": "myWorkspace",
  "sparkPoolName": "mySparkPool",
  "submitterName": "thetime",
  "submitterId": "thesubmitterid",
  "artifactId": "fill in here",
  "jobType": "SparkBatch",
  "result": "Failed",
  "schedulerInfo": null,
  "pluginInfo": null,
  "errorInfo": [],
  "tags": null,
  "id": 1,
  "appId": "fill in here",
  "appInfo": null,
  "state": "the state",
  "log": []
}

Definitions

Name	Description
LivyStates	The batch state
PluginCurrentState
SchedulerCurrentState
SparkBatchJob
SparkBatchJobOptions
SparkBatchJobResultType	The Spark batch job result.
SparkBatchJobState
SparkErrorSource
SparkJobType	The job type.
SparkRequest
SparkScheduler
SparkServiceError
SparkServicePlugin

LivyStates

The batch state

Name	Type	Description
busy	string
dead	string
error	string
idle	string
killed	string
not_started	string
recovering	string
running	string
shutting_down	string
starting	string
success	string

PluginCurrentState

Name	Type	Description
Cleanup	string
Ended	string
Monitoring	string
Preparation	string
Queued	string
ResourceAcquisition	string
Submission	string

SchedulerCurrentState

Name	Type	Description
Ended	string
Queued	string
Scheduled	string

SparkBatchJob

Name	Type	Description
appId	string	The application id of this session
appInfo	object	The detailed application info.
artifactId	string	The artifact identifier.
errorInfo	SparkServiceError[]	The error information.
id	integer	The session Id.
jobType	SparkJobType	The job type.
livyInfo	SparkBatchJobState
log	string[]	The log lines.
name	string	The batch name.
pluginInfo	SparkServicePlugin	The plugin information.
result	SparkBatchJobResultType	The Spark batch job result.
schedulerInfo	SparkScheduler	The scheduler information.
sparkPoolName	string	The Spark pool name.
state	LivyStates	The batch state
submitterId	string	The submitter identifier.
submitterName	string	The submitter name.
tags	object	The tags.
workspaceName	string	The workspace name.

SparkBatchJobOptions

Name	Type	Description
archives	string[]
args	string[]
artifactId	string
className	string
conf	object
driverCores	integer
driverMemory	string
executorCores	integer
executorMemory	string
file	string
files	string[]
jars	string[]
name	string
numExecutors	integer
pyFiles	string[]
tags	object

SparkBatchJobResultType

The Spark batch job result.

Name	Type	Description
Cancelled	string
Failed	string
Succeeded	string
Uncertain	string

SparkBatchJobState

Name	Type	Description
currentState	string	the Spark job state.
deadAt	string	time that at which "dead" livy state was first seen.
jobCreationRequest	SparkRequest
killedAt	string	the time that at which "killed" livy state was first seen.
notStartedAt	string	the time that at which "not_started" livy state was first seen.
recoveringAt	string	the time that at which "recovering" livy state was first seen.
runningAt	string	the time that at which "running" livy state was first seen.
startingAt	string	the time that at which "starting" livy state was first seen.
successAt	string	the time that at which "success" livy state was first seen.

SparkErrorSource

Name	Type	Description
Dependency	string
System	string
Unknown	string
User	string

SparkJobType

The job type.

Name	Type	Description
SparkBatch	string
SparkSession	string

SparkRequest

Name	Type	Description
archives	string[]
args	string[]
className	string
conf	object
driverCores	integer
driverMemory	string
executorCores	integer
executorMemory	string
file	string
files	string[]
jars	string[]
name	string
numExecutors	integer
pyFiles	string[]

SparkScheduler

Name	Type	Description
cancellationRequestedAt	string
currentState	SchedulerCurrentState
endedAt	string
scheduledAt	string
submittedAt	string

SparkServiceError

Name	Type	Description
errorCode	string
message	string
source	SparkErrorSource

SparkServicePlugin

Name	Type	Description
cleanupStartedAt	string
currentState	PluginCurrentState
monitoringStartedAt	string
preparationStartedAt	string
resourceAcquisitionStartedAt	string
submissionStartedAt	string