Production job scheduling cheat sheet

This article aims to provide clear and opinionated guidance for production job scheduling. Using best practices can help reduce costs, improve performance, and tighten security.

Best Practice	Impact	Docs
Use jobs clusters for automated workflows	Cost: Jobs clusters are billed at lower rates than interactive clusters.	- Create a cluster - All-purpose and job clusters.
Restart long-running clusters	Security: Restart clusters to take advantage of patches and bug fixes to the Databricks Runtime.	- Restart a cluster to update it with the latest images
Use service principals instead of user accounts to run production jobs	Security: If jobs are owned by individual users, when those users leave the org, these jobs may stop running.	- Manage service principals
Use Databricks Jobs for orchestration whenever possible	Cost: There’s no need to use external tools to orchestrate if you are only orchestrating workloads on Azure Databricks.	- Schedule and orchestrate workflows
Use latest LTS version of Databricks Runtime	Performance and cost: Azure Databricks is always improving Databricks Runtime for usability, performance, and security.	- Compute - Databricks support lifecycles
Don’t store production data in DBFS root	Security: When data is stored in the DBFS root, all users can access it.	- What is DBFS? - Recommendations for working with DBFS root