Azure Databricks personal access token authentication

Azure Databricks personal access tokens (PATs) are used to authenticate access to resources and APIs at the Azure Databricks workspace level. Many storage mechanisms for credentials and related information, such as environment variables and Azure Databricks configuration profiles, provide support for Azure Databricks personal access tokens. Although users can have multiple personal access tokens in an Azure Databricks workspace, each personal access token works for only a single Azure Databricks workspace. The number of personal access tokens per user is limited to 600 per workspace.

Databricks automatically revokes personal access tokens that haven’t been used in 90 or more days.

Important

Databricks recommends using OAuth instead of PATs for user account client authentication and authorization due to the improved security OAuth has. To learn how to use OAuth to perform client authentication with a Databricks user account, see Authenticate access to Azure Databricks with a user account using OAuth (OAuth U2M) (for user account authentication).

Basic (not token-based) authentication using an Azure Databricks username and password reached end of life on July 10, 2024.

To automate Azure Databricks account-level functionality, you cannot use Azure Databricks personal access tokens. Instead, you must use the Microsoft Entra ID tokens of Azure Databricks account admins. Azure Databricks account admins can be users or service principals. For more information, see:

• Manage users

  • Manage service principals
  • Manage groups

  See also:

  • Azure managed identities authentication
  • MS Entra service principal authentication
  • Azure CLI authentication

Azure Databricks personal access tokens for workspace users

To create an Azure Databricks personal access token for your Azure Databricks workspace user, do the following:

1. In your Azure Databricks workspace, click your Azure Databricks username in the top bar, and then select Settings from the drop down.
2. Click Developer.
3. Next to Access tokens, click Manage.
4. Click Generate new token.
5. (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).
6. Click Generate.
7. Copy the displayed token to a secure location, and then click Done.

Note

Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token. If you lose the copied token, or you believe that the token has been compromised, Databricks strongly recommends that you immediately delete that token from your workspace by clicking the trash can (Revoke) icon next to the token on the Access tokens page.

If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following topics:

• Enable or disable personal access token authentication for the workspace
• Personal access token permissions

Azure Databricks personal access tokens for service principals

A service principal can create Databricks personal access tokens for itself, as follows:

This procedure assumes that you are using OAuth machine-to-machine (M2M) authentication or Microsoft Entra ID service principal authentication to set up the Databricks CLI for authenticating the service principal to generate Azure Databricks personal access tokens for itself. See OAuth machine-to-machine (M2M) authentication or Microsoft Entra ID service principal authentication.

1. Use the Databricks CLI to run the following command, which generates another access token for the service principal.

   Run the following command:

   databricks tokens create --comment <comment> --lifetime-seconds <lifetime-seconds> -p <profile-name>
   

   • --comment: Replace <comment> with a meaningful comment about the access token’s purpose. If the --comment option is not specified, then no comment is generated.
   • --lifetime-seconds: Replace <lifetime-seconds> with the number of seconds that the access token is valid for. For example, 1 day is 86400 seconds. If the --lifetime-seconds option is not specified, the access token is set to never expire (not recommended).
   • --profile-name: Replace <profile-name> with the name of an Azure Databricks configuration profile that contains authentication information for the service principal and the target workspace. If the -p option is not specified, the Databricks CLI will attempt to find and use a configuration profile named DEFAULT.

2. In the response, copy the value of token_value, which is the access token for the service principal.

   Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token.

   If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following:

   • Enable or disable personal access token authentication for the workspace
   • Personal access token permissions

Perform Azure Databricks personal access token authentication

To configure Azure Databricks personal access token authentication, you must set the following associated environment variables, .databrickscfg fields, Terraform fields, or Config fields:

• The Azure Databricks host, specified as the target Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
• The Azure Databricks personal access token for the Azure Databricks user account.

To perform Azure Databricks personal access token authentication, integrate the following within your code, based on the participating tool or SDK:

Environment

To use environment variables for a specific Azure Databricks authentication type with a tool or SDK, see Authenticate access to Azure Databricks resources or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default methods for client unified authentication.

Set the following environment variables:

• DATABRICKS_HOST, set to the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
• DATABRICKS_TOKEN, set to the token string.

Profile

Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values. To use the profile with a tool or SDK, see Authenticate access to Azure Databricks resources or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default methods for client unified authentication.

Set the following values in your .databrickscfg file. In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

[<some-unique-configuration-profile-name>]
host  = <workspace-url>
token = <token>

Instead of manually setting the preceding values in your .databrickscfg file, you can use the Databricks CLI to set these values instead, as follows:

Note

The following procedure uses the Databricks CLI to create an Azure Databricks configuration profile with the name DEFAULT. If you already have a DEFAULT configuration profile, this procedure overwrites your existing DEFAULT configuration profile.

To check whether you already have a DEFAULT configuration profile, and to view this profile’s settings if it exists, use the Databricks CLI to run the command databricks auth env --profile DEFAULT.

To create a configuration profile with a name other than DEFAULT, replace the DEFAULT part of --profile DEFAULT in the following databricks configure command with a different name for the configuration profile.

1. Use the Databricks CLI to create an Azure Databricks configuration profile named DEFAULT that uses Azure Databricks personal access token authentication. To do this, run the following command:

   databricks configure --profile DEFAULT
   

2. For the prompt Databricks Host, enter your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.

3. For the prompt Personal Access Token, enter the Azure Databricks personal access token for your workspace.

CLI

For the Databricks CLI, run the databricks configure command. At the prompts, enter the following settings:

• The Azure Databricks host, specified as the target Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
• The Azure Databricks personal access token for the Azure Databricks user account.

For more details, see Azure Databricks personal access token authentication.

Connect

Note

Azure Databricks personal access token authentication is supported on the following Databricks Connect versions:

• For Python, Databricks Connect for Databricks Runtime 13.3 LTS and above.
• For Scala, Databricks Connect for Databricks Runtime 13.3 LTS and above.

For Databricks Connect, you can use the Databricks CLI to set the values in your .databrickscfg file, for Azure Databricks workspace-level operations as specified in this article’s “Profile” section, as follows:

Note

The following procedure uses the Databricks CLI to create an Azure Databricks configuration profile with the name DEFAULT. If you already have a DEFAULT configuration profile, this procedure overwrites your existing DEFAULT configuration profile.

To check whether you already have a DEFAULT configuration profile, and to view this profile’s settings if it exists, use the Databricks CLI to run the command databricks auth env --profile DEFAULT.

To create a configuration profile with a name other than DEFAULT, replace the DEFAULT part of --profile DEFAULT in the databricks configure command as shown in the following step with a different name for the configuration profile.

1. Use the Databricks CLI to create an Azure Databricks configuration profile named DEFAULT that uses Azure Databricks personal access token authentication. To do this, run the following command:

   databricks configure --configure-cluster --profile DEFAULT
   

2. For the prompt Databricks Host, enter your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.

3. For the prompt Personal Access Token, enter the Azure Databricks personal access token for your workspace.

4. In the list of available clusters that appears, use your up arrow and down arrow keys to select the target Azure Databricks cluster in your workspace, and then press Enter. You can also type any part of the cluster’s display name to filter the list of available clusters.

Use the Azure Databricks REST API to issue personal access tokens

Azure Databricks provides a REST endpoint /api/2.0/token/create to issue PATs. See Create a user token for API details.

You must provide specific values to the REST API. In the following example, set these values:

• Replace <databricks-instance> with your Databricks workspace URL. For example, dbc-abcd1234-5678.cloud.databricks.com.
• Replace <your-existing-access-token> with an existing valid PAT (string) that has permissions to create new tokens.

Provide the values for these parameters:

• comment: A description for the new token.
• lifetime_seconds: The token’s lifetime in seconds.

curl -X POST https://<databricks-instance>/api/2.0/token/create \
-H "Authorization: Bearer <your-existing-access-token>" \
-H "Content-Type: application/json" \
-d '{
  "comment": "New PAT using DB API",
  "lifetime_seconds": <lifetime-of-pat-in-seconds>
}'

The -d flag provides the JSON payload for the request.

If successful, this results in a response payload similar to:

{
  "access_token": "<your-newly-issued-pat>",
  "token_type": "Bearer",
  "expires_in": <the-duration-of-the-new-pat>
}

Provide the new token from the response in the Authorization header of subsequent calls to Databricks REST APIs. For example:

# This example uses a simple GET. For POST or other REST verbs, you may need to provide additional parameters.
curl -X GET "https://<databricks-instance>/api/2.0/<path-to-endpoint>" \
     -H "Authorization: Bearer <your-new-pat>"

import requests

headers = {
    'Authorization': 'Bearer <your-new-pat>'
}
# This example is for an HTTP GET operation.
response = requests.get('https://<databricks-instance>/api/2.0/<path-to-endpoint>', headers=headers)