Automating jobs with schedules and triggers
In Jobs, it is possible to configure jobs to automatically trigger in any of the following situations:
- On a time-based schedule
- On the arrival of files to a Unity Catalog storage location
- Continuously
You can also trigger job runs manually or through external orchestration tools.
Job schedules and triggers
| Trigger type | Behavior |
|---|---|
| Scheduled | Triggers a job run based on a time-based schedule. See Run jobs on a schedule. |
| File arrival | Triggers a job run when new files arrive in a monitored Unity Catalog storage location. See Trigger jobs when new files arrive. |
| Continuous | To keep the job always running, trigger another job run whenever a job run completes or fails. See Run jobs continuously. |
| None (manual) | Runs are triggered manually with the Run now button or programmatically using other orchestration tools. See Trigger a single job run |
By default, only a single run of a job can be active at a time. However, it is possible to increase this limit in the Advanced settings. Runs are skipped when they exceed the configured max concurrency for a job. See Configure maximum concurrent runs.
Configure a trigger on a job
- Open the job that you want to configure a trigger on.
- In the Job details pane, scroll down to the Schedules & Triggers section, and then click Add trigger.
- In Schedules & Triggers, select the type of trigger you want to configure: Scheduled, File arrival, or Continuous.
- For a file arrival trigger, select File arrival and enter the path in Storage location. Optionally, you can define a minimum time between triggers, or a delay between file arrival and the trigger starting the job. By default, there is a 60-second minimum time between triggers.
- Click Save. After you save your trigger, your job will start only when a new file arrives in the configured location.
Note
If one or more tasks in a job with multiple tasks are unsuccessful, you can re-run the subset of unsuccessful tasks. See Re-run failed and skipped tasks.
Pause and resume job triggers
You can pause and resume your jobs in the Job details pane for your job under Schedules & Triggers. The Pause and Resume buttons appear only for jobs that have a trigger configured.
To pause any active job trigger, click Pause. When you pause a trigger, any currently active runs continue, but the trigger no longer starts new runs.
To resume the trigger, click Resume. When you resume a trigger, the configured behavior resumes on the same schedule previously configured.
When creating or editing a trigger, you can also control these settings in the Schedules & Triggers dialog. Toggle between the Active and Paused to control the Trigger Status.
Note
If a run is active when a continuous trigger is resumed, the job scheduler waits until that run is complete to trigger a new run.