Batch Job
Common parameters and headers
The following information is common to all tasks related to clusters that you might do:
Replace {cluster-endpoint} with the url of your cluster’s management endpoint. (for example, myendpoint.azurehdinsight.net)
Replace {batch-id} with the batch ID returned on job submission.
Set the Content-Type header to application/json.
Authenticate with the endpoint using HTTP basic authentication.
Create a new batch job
Submit a new batch job from a jar.
Request
See Common parameters and headers for headers and parameters that are used by clusters.
Method | Request URI |
---|---|
POST | https://{cluster-endpoint}/livy/batches |
Following shows an example request to create a new batch job
{
"file" : "wasb://<BlobStorageContainerName>@<StorageAccountName>.blob.core.windows.net/sample.jar",
"args" : ["arg0", "arg1"],
"className" : "com.sample.Job1",
"jars" : ["wasb://<BlobStorageContainerName>@<StorageAccountName>.blob.core.windows.net/helper.jar"],
"files" : ["wasb://<BlobStorageContainerName>@<StorageAccountName>.blob.core.windows.net/config.xml"],
"driverMemory" : "1G",
"driverCores" : 2,
"executorMemory" : "1G",
"executorCores" : 10,
"numExecutors" : 10
}
Element Name | Required | Type | Description |
---|---|---|---|
proxyUser | No | String | The user to impersonate that will execute the job |
file | Yes | String | Path to the batch job’s jar. |
args | No | Array of String | Command-line arguments passed to the batch job. |
className | Yes | String | The class name of the main class. |
jars | No | Array of String | Files to be placed on the java classpath |
pyFiles | No | Array of String | Files to be placed on the PYTHONPATH |
files | No | Array of String | Files to be placed in executor working directory |
driverMemory | No | String | Memory for driver (1000M, 2G) |
driverCores | No | Integer | Number of cores used by driver |
executorMemory | No | String | Memory for executor (1000M, 2G) |
executorCores | No | Integer | Number of cores used by executor |
numExecutors | No | Integer | number of executors |
archives | No | Array of String | Archives to be uncompressed (YARN mode only) |
Response
The operation will return 200 (OK) if the request is completed successfully
Status code: 200 OK
Response body:
{
"id" : 1,
"state" : "starting",
"log" : ["logline"]
}
Element Name | Required | Type | Description |
---|---|---|---|
id | Yes | Integer | The batch session id |
log | Yes | Array of string | Array of log lines for this batch job. |
state | No | String | The session state. Possible value: [“starting”, “idle”, “error”] |
Get the full log of a batch job
Request
See Common parameters and headers for headers and parameters that are used by clusters.
Method | Request URI |
---|---|
GET | https://{cluster-endpoint}/livy/batches/{batch-id}/log |
Response
If validation is complete and the request is accepted, the operation will return 200 (OK).
Status code: 200 OK
Response body:
{
"id" : 1,
"from":0,
"total":54
“log" : ["logline1", “logline2”]
}
Element name | Required | Type | Description |
---|---|---|---|
id | Yes | Integer | The batch session id |
from | Yes | Integer | Offset |
size | Yes | Integer | Total number of lines |
log | Yes | Array of string | Array of log lines for this batch job. |
Get the state of a batch job
Request
See Common parameters and headers for headers and parameters that are used by clusters.
Method | Request URI |
---|---|
GET | https://{cluster-endpoint}/livy/batches/{batch-id} |
Response
If validation is complete and the request is accepted, the operation will return 200 (OK).
Status code: 200 OK
Response body:
{
"id" : 1,
"state" : "starting",
“log" : ["logline"]
}
Element name | Required | Type | Description |
---|---|---|---|
id | Yes | Integer | The batch session id |
log | Yes | Array of strings | Array of log lines for this batch job. |
state | Yes | String | The session state. Possible value: [“starting”, “idle”, “error”] |
Kill a batch job
Request
See Common parameters and headers for headers and parameters that are used by clusters.
Method | Request URI |
---|---|
DELETE | https://{cluster-endpoint}/livy/batches/{batch-id} |
Response
If validation is complete and the request is accepted, the operation will return 200 (OK).
Status code: 200 OK
Response body:
{
"msg" : "deleted"
}
Element name | Required | Type | Description |
---|---|---|---|
Message | Yes | String | The message of the delete. |
Interactive sessions
- Start, submit statement, retrieve statement results, and terminate an interactive session
Return all active batch jobs
Request
See Common parameters and headers for headers and parameters that are used by clusters.
Method | Request URI |
---|---|
GET | https://{cluster-endpoint}/livy/batches |
Response
If validation is complete and the request is accepted, the operation will return 200 (OK).
Status code: 200 OK
Response body:
{
"from" : 0,
"total" : 2,
"sessions" : [{
"id" : 1,
"state" : "starting",
"log" : ["logline"]
}, {
"id" : 0,
"state" : "idle",
"log" : ["logline"]
}
]
}
Element name | Required | Type | Description |
---|---|---|---|
from | Yes | String | Offset |
total | Yes | String | Number of batches to return |
session | Yes | Array of Complex Type (Batch) | A list of active batch jobs |
Batch
Element name | Required | Type | Description |
---|---|---|---|
id | Yes | Integer | The batch session id |
log | Yes | Array of strings | Array of log lines for this batch job. |
state | Yes | String | The session state. Possible value: [“starting”, “idle”, “error”] |