Share via

Configure Azure Databricks sign-on from dbt Core with Microsoft Entra ID

  • Article

Important

This feature is in Public Preview.

This article describes how to configure Azure Databricks sign-on from dbt Core with Microsoft Entra ID. After you complete this one-time configuration as an Azure Databricks account admin, users can connect Azure Databricks to dbt Core using single sign-on (SSO).

In addition to using Microsoft Entra ID, you can use Databricks M2M OAuth to integrate with dbt Core. See Enable or disable partner OAuth applications.

Before you begin

Before you complete the steps in this article, you must:

  • Have access to a local dbt Core environment
  • Get the Server Hostname and HTTP Path for a SQL warehouse in your Azure Databricks workspace
  • Be an Azure Databricks account administrator
  • Have permission to create an Azure app registration in the Azure tenant for your Azure Databricks account.

Configure machine-to-machine (M2M) authentication for dbt Core

This section describes how to configure machine-to-machine (M2M) authentication for Azure Databricks and dbt Core. OAuth’s M2M workflow empowers automated processes, like CI/CD integration, to securely access resources without human intervention. The dbt client runs as a service principal, acquires an OAuth token from Azure AD using the service principal, and utilizes this token to connect with the Databricks SQL API.

Create an Azure app registration

  1. Use the Azure portal to sign in to the Azure tenant for your Azure Databricks account, at https://portal.azure.com/<tenant-id>.
  2. Click App registrations. If App registrations is not visible, click More services and use the Filter services text box to search for App registrations.
  3. Click New registration.
  4. Enter a Name for the app.
  5. For Redirect URI, select Public client/native (mobile & desktop) from the drop-down menu and enter http://localhost:8020.
  6. Click Register.
  7. Copy the Application (client) ID. You’ll need this value later.

Generate a client secret for your Azure App registration

  1. In the Azure portal, select the app registration you created in the previous step.
  2. Click Certificates and secrets, and then click New client secret.
  3. Enter a description, and then click Add.
  4. Copy the secret value. You’ll need this value later, and you can only access it after secret creation.

Add the Microsoft Entra ID service principal to your workspace

  1. In your Azure Databricks workspace, click your username in the top right, and then click Settings.
  2. Click on the Identity and access tab.
  3. Next to Service principals, click Manage.
  4. Click Add service principal, and then click Client ID.
  5. For ApplicationId, enter the Application (client) ID you copied earlier.
  6. For Display Name, enter a logical name for your service principal, and then click Add.

Add the M2M profile to your dbt project

  1. Set the Application (client) ID and client secret you copied earlier as environment variables. Databricks doesn’t recommend storing sensitive information such as secrets in profiles.yml directly.

    ~ export DATABRICKS_CLIENT_ID=<client-id>
~ export DATABRICKS_CLIENT_SECRET=<client-secret>

  2. Add the M2M profile to the profiles.yml file in your dbt project.

    The following is an example profiles.yml file with the M2M profile azure-oauth-m2m specified. Specifying azure-oauth-m2m for target sets the M2M profile as the default run profile used by dbt.

    databricks_demo:
  outputs:
    ...
    azure-oauth-m2m:
      catalog: uc_demos
      host: "adb-xxx.azuredatabricks.net"
      http_path: "/sql/1.0/warehouses/9196548d010cf14d"
      schema: databricks_demo
      threads: 1
      type: databricks
      auth_type: oauth
      client_id: "{{ env_var('DATABRICKS_CLIENT_ID') }}"
      client_secret: "{{ env_var('DATABRICKS_CLIENT_SECRET') }}"
target: azure-oauth-m2m

Authorize your service principal to access a SQL warehouse

  1. In the sidebar, click SQL Warehouses, click your SQL warehouse, and then click Permissions.
  2. Next to the service principal you created earlier in this article, select CAN USE from the drop-down menu.

Verify OAuth application setup

Run the dbt debug command to verify that your OAuth application has been configured correctly. For example:

dbt debug --target azure-oauth-m2m

The following is an example output of a successful dbt debug run:

  ...
  Configuration:
    profiles.yml file [OK found and valid]
    dbt_project.yml file [OK found and valid]

  Required dependencies:
    - git [OK found]

  Connection:
    ...
    Connection test: OK connection ok

Configure user-to-machine (U2M) authentication for dbt Core

This section describes how to configure user-to-machine authentication with Azure Databricks and dbt Core. OAuth’s U2M workflow allows systems access on behalf of a user, like the dbt client, by securely obtaining an OAuth token through a browser popup from Azure AD. Dbt uses this token to connect to the Databricks SQL API, simplifying authentication and authorization.

Create an Azure app registration

  1. Use the Azure portal to sign in to the Azure tenant for your Azure Databricks account, at https://portal.azure.com/<tenant-id>.
  2. Click App registrations. If App registrations is not visible, click More services and use the Filter services text box to search for App registrations.
  3. Click New registration.
  4. Enter a Name for the app.
  5. For Redirect URI, select Public client/native (mobile & desktop) from the drop-down menu and enter http://localhost:8020.
  6. Click Register.
  7. Copy the Application (client) ID and the Tenant ID. You’ll need these values later.

Add the U2M profile to your dbt project

Add the U2M profile to the profiles.yml file in your dbt project.

The following is an example profiles.yml file with the U2M profile azure-oauth-u2m specified. Specifying azure-oauth-u2m for target sets the U2M profile as the default run profile used by dbt.

databricks_demo:
outputs:
  azure-oauth-u2m:
    catalog: uc_demos
    host: "adb-xxx.azuredatabricks.net"
    http_path: "/sql/1.0/warehouses/9196548d010cf14d"
    schema: databricks_demo
    threads: 1
    type: databricks
    auth_type: oauth
    client_id: "9729e77e-ba94-4d53-8cfb-bb609f43f881"
target: azure-oauth-u2m

Verify OAuth application setup

  1. Run the dbt debug command to verify that your OAuth application has been configured correctly. For example:

    dbt debug --target azure-oauth-u2m

    The Permissions requested page opens in your browser.

  2. Click Accept.

The following is an example output of a successful dbt debug run:

...
Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
  - git [OK found]

Connection:
  ...
  Connection test: OK connection ok

Additional resources

To create a new dbt Core project and connect to Azure Databricks using SSO authentication, see Connect to dbt Core.