Jaa

Add email and system notifications for job events

  • Artikkeli

You can monitor the runs of a job and the tasks that are part of that job by configuring notifications when a run starts, completes successfully, fails, or its duration exceeds a configured threshold. Notifications can be sent to one or more email addresses or system destinations such as Slack, Microsoft Teams, PagerDuty, or any webhook-based service.

Configure system notifications

Note

  • For each job or task, you can configure a maximum of three system destinations for each notification event type.
  • An administrator must configure system destinations. System destinations are configured by selecting Create new destination in the Edit system notifications dialog or the admin settings page.
  • Notifications you set at the job level are not sent when failed tasks are retried. To receive a failure notification after every failed task (including every failed retry), use task notifications instead. To add system notifications for task runs, click + Add next to Notifications in the task panel when you add or edit a job task.
  • A job that has completed in a Succeeded with failures state is considered to be in a successful state. To receive alerts for jobs that complete in this state, you must select Success when you configure notifications.

System notifications integrate with popular notification tools, including:

To add one or more system notifications when runs of this job have a notifiable event such as a job start, completion, or failure:

  1. In the Job details panel for your job, click Edit notifications.
  2. Click Add Notification and select a system destination in Destination.
  3. In Select a system destination, select a destination, and click the checkbox for each notification type to send to that destination.
  4. To add another destination, click Add notification again.
  5. Click Confirm.

Important

The content of Slack and Microsoft Teams messages might change in future releases. You should not implement clients or processing that depend on the specific content or formatting of these messages. If you require a specific schema or formatting for notifications, Azure Databricks recommends configuring a user-defined webhook.

Configure email notifications

Note

  • Notifications you set at the job level are not sent when failed tasks are retried. To receive a failure notification after every failed task (including every failed retry), use task notifications instead. To add email notifications for task runs, click + Add next to Notifications in the task panel when you add or edit a job task.
  • A job that has completed in a Succeeded with failures state is considered to be in a successful state. To receive alerts for jobs that complete in this state, you must select Success when you configure notifications.

To add one or more email addresses to notify when runs of this job begin, complete, or fail:

  1. In the Job details panel for your job, click Edit notifications.
  2. Click Add Notification and select Email address in Destination.
  3. Enter an email address and click the checkbox for each notification type to send to that address.
  4. To enter another email address for notification, click Add notification again.
  5. Click Confirm.

You can use email notifications to integrate with tools such as Amazon SES and SNS.

Configure notifications when a threshold is exceeded

If you have configured a threshold for a job or a a task such as duration or streaming backlog metrics, you can add an email or system notification if the job or task exceeds the configured threshold. To receive a notification for a job or task that exceeds a duration threshold, click the checkbox for Duration Warning when you add or edit a notification. To receive a notification for a job or task that exceeds a streaming backlog metric, click the checkbox for Streaming backlog when you add or edit a notification.

The following applies to streaming backlog metrics:

  • Notifications are sent when the average backlog over a 10-minute period exceeds the defined threshold.
  • To prevent excessive messages, Azure Databricks waits 30 minutes before determining whether to send another message. While the backlog remains high, you’ll receive updates at 30-minute intervals.

Filter out notifications for skipped or canceled runs

You can reduce the number of notifications sent by filtering out notifications when a run is skipped or canceled. To filter notifications, check Mute notifications for skipped runs or Mute notifications for canceled runs when you add or modify email notifications or system notifications.

Note

Selecting Mute notifications for skipped runs or Mute notifications for canceled runs for a job does not filter out notifications configured for job tasks. To filter all notifications for skipped or canceled runs, you must also filter out any task-level notifications you have configured.

HTTP webhook payloads

If you use an HTTP webhook to send notifications, the following are example payloads sent by Azure Databricks to your configured endpoint.

Notification for a job run start event:

{
  "event_type": "jobs.on_start",
  "workspace_id": "your_workspace_id",
  "run": {
    "run_id": "run_id"
  },
  "job": {
    "job_id": "job_id",
    "name": "job_name"
  }
}

Notification for a task run start event:

{
  "event_type": "jobs.on_start",
  "workspace_id": "your_workspace_id",
  "task": {
    "task_key": "task_name"
  },
  "run": {
    "run_id": "run_id_of_task"
    "parent_run_id": "run_id_of_parent_job_run"
  },
  "job": {
    "job_id": "job_id",
    "name": "job_name"
  }
}

When configuring destinations, webhooks can be configured for the following event types:

Event code When is it sent?
jobs.on_start A run starts.
jobs.on_success A run stops and completes in a successful or succeeded with failures state.
jobs.on_failure A run stops in an unsuccessful state.
jobs.on_duration_warning_threshold_exceeded A run has been running for more than the configured expected duration.