Share via

Authentication for the Databricks CLI

  • Article

Note

This information applies to Databricks CLI versions 0.205 and above. The Databricks CLI is in Public Preview.

Databricks CLI use is subject to the Databricks License and Databricks Privacy Notice, including any Usage Data provisions.

This article describes how to set up authentication between the Databricks CLI and your Azure Databricks accounts and workspaces. See What is the Databricks CLI?.

This article assumes that you have already installed the Databricks CLI. See Install or update the Databricks CLI.

Before you can run Databricks CLI commands, you must set up authentication between the Databricks CLI and your Azure Databricks accounts, workspaces, or a combination of these, depending on the types of CLI commands that you want to run.

You must authenticate the Databricks CLI to the relevant resources at run time in order to run Azure Databricks automation commands within an Azure Databricks account or workspace. Depending on whether you want to call Azure Databricks workspace-level commands, Azure Databricks account-level commands, or both, you must authenticate to the Azure Databricks workspace, account, or both. For a list of Azure Databricks workspace-level and account-level CLI command groups, run the command databricks -h. For a list of Azure Databricks workspace-level and account-level REST API operations that the Databricks CLI commands cover, see the Databricks REST API.

For information about Microsoft Entra authentication to Databricks with Azure DevOps specifically, see Authenticate with Azure DevOps on Databricks.

The following sections provide information about how to set up authentication between the Databricks CLI and Azure Databricks:

Azure Databricks personal access token authentication

Azure Databricks personal access token authentication uses an Azure Databricks personal access token to authenticate the target Azure Databricks entity, such as an Azure Databricks user account. See Azure Databricks personal access token authentication.

Note

You cannot use Azure Databricks personal access token authentication for authenticating with an Azure Databricks account, as Azure Databricks account-level commands do not use Azure Databricks personal access tokens for authentication. To authenticate with an Azure Databricks account, consider using one of the following authentication types instead:

To create a personal access token, do the following:

  1. In your Azure Databricks workspace, click your Azure Databricks username in the top bar, and then select Settings from the drop down.
  2. Click Developer.
  3. Next to Access tokens, click Manage.
  4. Click Generate new token.
  5. (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).
  6. Click Generate.
  7. Copy the displayed token to a secure location, and then click Done.

Note

Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token. If you lose the copied token, or you believe that the token has been compromised, Databricks strongly recommends that you immediately delete that token from your workspace by clicking the trash can (Revoke) icon next to the token on the Access tokens page.

If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following topics:

To configure and use Azure Databricks personal access token authentication, do the following:

Note

The following procedure creates an Azure Databricks configuration profile with the name DEFAULT. If you already have a DEFAULT configuration profile that you want to use, then skip this procedure. Otherwise, this procedure overwrites your existing DEFAULT configuration profile. To view the names and hosts of any existing configuration profiles, run the command databricks auth profiles.

To create a configuration profile with a name other than DEFAULT, add --profile <configuration-profile-name> or -p <configuration-profile-name> to the end of the following databricks configure command, replacing <configuration-profile-name> with the new configuration profile’s name.

  1. Use the Databricks CLI to run the following command:

    databricks configure

  2. For the prompt Databricks Host, enter your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.

  3. For the prompt Personal Access Token, enter the Azure Databricks personal access token for your workspace.

    After you enter your Azure Databricks personal access token, a corresponding configuration profile is added to your .databrickscfg file. If the Databricks CLI cannot find this file in its default location, it creates this file for you first and then adds this configuration profile to the new file. The default location for this file is in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows.

  4. You can now use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks clusters list -p <configuration-profile-name>.

OAuth machine-to-machine (M2M) authentication

Instead of authenticating with Azure Databricks by using Azure Databricks personal access token authentication, you can use OAuth authentication. OAuth provides tokens with faster expiration times than Azure Databricks personal access tokens, and offers better server-side session invalidation and scoping. Because OAuth access tokens expire in less than an hour, this reduces the risk associated with accidentally checking tokens into source control. See also Authenticate access to Azure Databricks with a service principal using OAuth (OAuth M2M).

To configure and use OAuth M2M authentication, do the following:

  1. Complete the OAuth M2M authentication setup instructions. See Authenticate access to Azure Databricks with a service principal using OAuth (OAuth M2M)

  2. Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host          = <account-console-url>
account_id    = <account-id>
client_id     = <service-principal-client-id>
client_secret = <service-principal-oauth-secret>

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host          = <workspace-url>
client_id     = <service-principal-client-id>
client_secret = <service-principal-oauth-secret>

    Note

    The default location for the .databrickscfg file is in the user’s home directory. This is ~ for Linux and macOS, and %USERPROFILE% for Windows.

  3. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile as part of the Databricks CLI command call, for example, databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    Press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

OAuth user-to-machine (U2M) authentication

Instead of authenticating with Azure Databricks by using token authentication, you can use OAuth authentication. OAuth provides tokens with faster expiration times than Azure Databricks personal access tokens, and offers better server-side session invalidation and scoping. Because OAuth access tokens expire in less than an hour, this reduces the risk associated with accidentally checking tokens into source control. See also Authenticate access to Azure Databricks with a user account using OAuth (OAuth U2M).

To configure and use OAuth U2M authentication, do the following:

  1. Before calling any Azure Databricks account-level commands, you must initiate OAuth token management locally by running the following command. This command must be run separately for each account that you want to run commands against. If you do not want to call any account-level operations, skip ahead to step 5.

    In the following command, replace the following placeholders:

    databricks auth login --host <account-console-url> --account-id <account-id>

  2. The Databricks CLI prompts you to save the account console URL and account ID locally as an Azure Databricks configuration profile. Press Enter to accept the suggested profile name, or enter the name of a new or existing profile. Any existing profile with the same name is overwritten with this account console URL and account ID.

    To get a list of any existing profiles, in a separate terminal or command prompt, run the command databricks auth profiles. To view a specific profile’s existing settings, run the command databricks auth env --profile <profile-name>.

  3. In your web browser, complete the on-screen instructions to log in to your Azure Databricks account.

  4. To view the current OAuth token value and upcoming expiration timestamp, run the command databricks auth token --host <account-console-url> --account-id <account-id>.

  5. Before calling any Azure Databricks workspace-level commands, you must initiate OAuth token management locally by running the following command. This command must be run separately for each workspace that you want to run commands against.

    In the following command, replace <workspace-url> with your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.

    databricks auth login --host <workspace-url>

  6. The Databricks CLI prompts you to save the workspace URL locally as an Azure Databricks configuration profile. Press Enter to accept the suggested profile name, or enter the name of a new or existing profile. Any existing profile with the same name is overwritten with this workspace URL.

    To get a list of any existing profiles, in a separate terminal or command prompt, run the command databricks auth profiles. To view a specific profile’s existing settings, run the command databricks auth env --profile <profile-name>.

  7. In your web browser, complete the on-screen instructions to log in to your Azure Databricks workspace.

  8. To view the current OAuth token value and upcoming expiration timestamp, run the command databricks auth token --host <workspace-url>.

  9. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Azure managed identities authentication

Azure managed identities authentication uses managed identities for Azure resources (formerly Managed Service Identities (MSI)) for authentication. See What are managed identities for Azure resources?. See also Azure managed identities authentication.

To create an Azure user-assigned managed identity, do the following:

  1. Create or identify an Azure VM and install the Databricks CLI on it, then assign your managed identity to your Azure VM and your target Azure Databricks accounts, workspaces, or both. See Set up and use Azure managed identities authentication for Azure Databricks automation.

  2. On the Azure VM, create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host            = <account-console-url>
account_id      = <account-id>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host            = <workspace-url>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

    For workspace-level commands, if the target identity has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID, instead of host along with the workspace URL. In this case, the target identity must have at least Contributor or Owner permissions on the Azure resource.

    Note

    The default location for the .databrickscfg file is in the user’s home directory. This is ~ for Linux and macOS, and %USERPROFILE% for Windows.

  3. On the Azure VM, use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile to set the profile for Databricks to use, for example, databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Microsoft Entra ID service principal authentication

Microsoft Entra ID service principal authentication uses the credentials of a Microsoft Entra ID service principal to authenticate. To create and manage service principals for Azure Databricks, see Manage service principals. See also MS Entra service principal authentication.

To configure and use Microsoft Entra ID service principal authentication, you must have the Azure CLI authentication installed locally. You must also do the following:

  1. Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host                = <account-console-url>
account_id          = <account-id>
azure_tenant_id     = <azure-service-principal-tenant-id>
azure_client_id     = <azure-service-principal-application-id>
azure_client_secret = <azure-service-principal-client-secret>

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host                = <workspace-url>
azure_tenant_id     = <azure-service-principal-tenant-id>
azure_client_id     = <azure-service-principal-application-id>
azure_client_secret = <azure-service-principal-client-secret>

    For workspace-level commands, if the target Microsoft Entra ID service principal has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID, instead of host along with the workspace URL. In this case, the target Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource.

    Note

    The default location for the .databrickscfg file is in the user’s home directory. This is ~ for Linux and macOS, and %USERPROFILE% for Windows.

  2. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Azure CLI authentication

Azure CLI authentication uses the Azure CLI to authenticate the signed-in entity. See also Azure CLI authentication.

To configure Azure CLI authentication, you must do the following:

  1. Have the Azure CLI installed locally.

  2. Use the Azure CLI to log in to Azure Databricks by running the az login command. See Azure CLI login with an Azure Databricks user account.

  3. Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host       = <account-console-url>
account_id = <account-id>

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host = <workspace-url>

    Note

    The default location for the .databrickscfg file is in the user’s home directory. This is ~ for Linux and macOS, and %USERPROFILE% for Windows.

  4. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Authentication order of evaluation

Whenever the Databricks CLI needs to gather the settings that are required to attempt to authenticate with an Azure Databricks workspace or account, it searches for these settings in the following locations, in the following order.

  1. For any command run from the bundle working directory (the bundle root and any nested path), the values of fields within a project’s bundle setting files. (Bundle setting files do not support the direct inclusion of access credential values.)
  2. The values of environment variables, as listed within this article and in Environment variables and fields for client unified authentication.
  3. Configuration profile field values within the .databrickscfg file, as listed previously within this article.

Whenever the Databricks CLI finds the required settings that it needs, it stops searching in other locations. For example:

  • The Databricks CLI needs the value of an Azure Databricks personal access token. A DATABRICKS_TOKEN environment variable is set, and the .databrickscfg file also contains multiple personal access tokens. In this example, the Databricks CLI uses the value of the DATABRICKS_TOKEN environment variable and does not search the .databrickscfg file.
  • The databricks bundle deploy -t dev command needs the value of an Azure Databricks personal access token. A DATABRICKS_TOKEN environment variable is not set, and the .databrickscfg file contains multiple personal access tokens. The project’s bundle settings file contains a dev environment declaration that references through its profile field a configuration profile named DEV. In this example, the Databricks CLI searches the .databrickscfg file for a profile named DEV and uses the value of that profile’s token field.
  • The databricks bundle run -t dev hello-job command needs the value of an Azure Databricks personal access token. A DATABRICKS_TOKEN environment variable is not set, and the .databrickscfg file contains multiple personal access tokens. The project’s bundle settings file contains a dev environment declaration that references through its host field a specific Azure Databricks workspace URL. In this example, the Databricks CLI searches through the configuration profiles within the .databrickscfg file for a profile that contains a host field with a matching workspace URL. The Databricks CLI finds a matching host field and then uses that profile’s token field value.