Attribute serverless usage with budget policies

Important

This feature is in Public Preview.

This article explains how to use budget policies to enforce cost attribution tags on serverless compute workloads.

Budget policies consist of tags that are applied to any serverless compute activity incurred by a user assigned to the policy. The tags are logged in your billing records, allowing you to attribute serverless usage to specific budgets. For more on creating budgets, see Create and monitor budgets.

Budget policy permissions

You must be a workspace admin to create budget policies. Non-admins can manage policies if they are assigned Budget policy: Manager permissions.

Workspace admins can manage and view budget policies they created or the ones they have explicit permissions on. To view and manage all policies for a given account, the workspace admin must additionally have the Billing admin account-level role. This role is assigned in the account console.

By default, new policies can only be accessed by the policy creator and users with the billing admin role.

Because budget policies are account-level resource objects, managers can manage policies from any workspace they have access to.

Create a budget policy

Budget policies are managed in your workspace admin settings. To create a new budget policy:

1. Click your username in the top bar of the Azure Databricks workspace and select Settings.
2. Click Compute.
3. Next to Budget policies, click Manage.
4. Click Create.
5. Add a name and your tags then click Create.

Manage budget policy permissions

There are two types of permissions you can grant on your budget policies:

• User: A budget policy user can select the policy when they create a notebook, job, pipeline, or serving endpoint.
• Manager: Managers can use the budget policy, but can also edit the policy’s definitions and permissions. Any user in your workspace can be given the manager permissions.

If a user is only assigned a single policy, then that policy is automatically applied to the user's newly created resources. If a user is assigned multiple policies, they must select the appropriate policy when they create a new serverless notebook, job, pipeline, or serving endpoint. If a user doesn’t select a policy, the setting defaults to whichever policy comes first alphabetically.

Assign permissions on a policy

To assign permissions on a budget policy:

1. On the policy’s page, click the Permissions tab.
2. Click Grant access.
3. Select the user, group, or service principal you want to assign the policy to.
4. Under Role, select the role(s) you want to give the user.
5. To continue adding identities to the policy, click Add another.
6. Click Save.

Update a budget policy

To update an existing budget policy’s name or tags:

1. Click your username in the top bar of the Azure Databricks workspace and select Settings.
2. Click Compute.
3. Next to Budget policies, click Manage.
4. Select the budget policy you want to update, then click Edit.
5. Make the updates then click Save.

Policy changes are only applied to usage initiated after the policy update. They are not applied to currently running serverless usage.

Analyze budget policy tags in the billing records

Once a policy is applied to a notebook, job, or DLT pipeline, any tags contained in the policy propagate to your system.billing.usage system table in the custom_tags column.

Note

Budget policy tags also propagate to the billing records emitted to Azure cost analysis.

If a notebook is run as part of a job, only the job’s budget policy is applied to the usage record.

Where to select the budget policy

Users assigned to multiple budget policies must select the appropriate policy whenever they create a new notebook, job, pipeline or model serving endpoint.

• Notebook: Users can select the budget policy in the Connect > More… drop-down or the Environment side panel. See Select a budget policy for your serverless usage.
• Job: Users select the budget policy in the Job details section. See Select a budget policy for your serverless usage.
• Pipeline: Users select the budget policy when they create or edit the pipeline. See Configure a serverless DLT pipeline.
• Serving endpoints: Users select the budget policy when they create or edit the model serving endpoint. See Add a budget policy for a model serving endpoint.

Known limitations

Budget policies have the following limitations:

• Existing notebooks, jobs, and DLT pipelines are not automatically assigned policies after their owners are granted access to a policy. To add a budget policy to an existing asset, you must manually update the asset’s budget policy setting in the UI.
• When creating a scheduled notebook job from the notebook page, the notebook’s budget policy defaults back to the first available policy by alphabetical order.
• Policy IDs stored with an asset remain even if the policy is deleted. These policies do not apply any tags.
• Updates to budget policy tags are reflected on any serverless usage initiated after the change to the policy. For example, if a job is running while a budget policy is updated, the update does not take effect for the existing job run. The next job run uses the updated tags.
• Git repo integration does not have repo-configurable budget policies, the budget policy defaults to the user’s last chosen policy.
• Budget policies do not apply tags to classic compute resources.
• Tags inherited from budget policies do not appear on the Jobs list UI.
• Updates to tags won’t be reflected in new pipeline updates if the pipeline is in Development mode. The changes take 24 hours to propagate.
• Pipelines triggered by jobs do not inherit the job’s budget policy. Users must set the pipeline’s budget policy.