Manage access to external cloud services using service credentials

Important

This feature is in Public Preview.

This article describes how to create a service credential object in Unity Catalog that lets you govern access from Azure Databricks to external cloud services. A service credential in Unity Catalog encapsulates a long-term cloud credential that grants access to such services.

Service credentials are not intended for governing access to cloud storage that is used as a Unity Catalog managed storage location or external storage location. For those use cases, use a storage credential. See Manage access to cloud storage using Unity Catalog.

To create a service credential for access to Azure services, you create an Azure Databricks access connector that references an Azure managed identity, assigning it permissions on a service or services. You then reference that access connector in the service credential definition.

Before you begin

Before you create a service credential, you must meet the following requirements:

In Azure Databricks:

• Azure Databricks workspace enabled for Unity Catalog.

• CREATE SERVICE CREDENTIAL privilege on the Unity Catalog metastore attached to the workspace. Account admins and metastore admins have this privilege by default. If your workspace was enabled for Unity Catalog automatically, workspace admins also have this privilege.

  Note

  Service principals must have the account admin role to create a service credential that uses a managed identity. You cannot delegate CREATE SERVICE CREDENTIAL to a service principal. This applies to both Azure Databricks service principals and Microsoft Entra ID service principals.

In your Azure tenant:

• An Azure service in the same region as the workspace you want to access the data from.
• The Contributor or Owner role on an Azure resource group.
• The Owner or User Access Administrator Azure RBAC role on the service account.

Create a service credential using a managed identity

To configure the identity that authorizes access to your service account, use an Azure Databricks access connector that connects an Azure managed identity to your Azure Databricks account. If you already have the access connector defined, you can skip ahead to step 2 in the following procedure.

Note

You can use a service principal instead of a managed identity, but managed identities are strongly recommended. Managed identities have the benefit of allowing Unity Catalog to access service accounts protected by network rules, which isn’t possible using service principals, and they remove the need to manage and rotate secrets. If you must use a service principal, see Create Unity Catalog managed storage using a service principal (legacy).

1. In the Azure portal, create an Azure Databricks access connector and assign it permissions on the service that you would like to access, using the instructions in Configure a managed identity for Unity Catalog.

   An Azure Databricks access connector is a first-party Azure resource that lets you connect managed identities to an Azure Databricks account. You must have the Contributor role or higher on the access connector resource in Azure to add the service credential.

   Instead of following the instructions in Step 2: Grant the managed identity access to the storage account, grant the managed identity access to your service account.

   Make a note of the access connector’s resource ID.

2. Log in to your Azure Databricks workspace as a user who meets the requirements listed in Before you begin.

3. Click Catalog.

4. On the Quick access page, click the External data > button, go to the Credentials tab, and select Create credential.

5. Select Service Credential.

6. Enter a Credential name, an optional comment, and the access connector’s resource ID in the format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>
   

7. (Optional) If you created the access connector using a user-assigned managed identity, enter the resource ID of the managed identity in the User-assigned managed identity ID field, in the format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<managed-identity-name>
   

8. Click Create.

9. In the Service credential created dialog, copy the External ID.

   You can also view the external ID at any time on the service credential details page.

10. Click Done.

(Optional) Assign a service credential to specific workspaces

Important

This feature is in Public Preview.

By default, a service credential is accessible from all of the workspaces in the metastore. This means that if a user has been granted a privilege on that service credential, they can exercise that privilege from any workspace attached to the metastore. If you use workspaces to isolate user data access, you may want to allow access to a service credential only from specific workspaces. This feature is known as workspace binding or service credential isolation.

A typical use case for binding a service credential to specific workspaces is the scenario in which a cloud admin configures a service credential using a production cloud account credential, and you want to ensure that Azure Databricks users use this credential to access an external cloud service only in the production workspace.

For more information about workspace binding, see (Optional) Assign a storage credential to specific workspaces and Limit catalog access to specific workspaces.

Bind a service credential to one or more workspaces

To assign a service credential to specific workspaces, use Catalog Explorer.

Permissions required: Metastore admin or service credential owner.

Note

Metastore admins can see all service credentials in a metastore using Catalog Explorer—and service credential owners can see all service credentials that they own in a metastore—regardless of whether the service credential is assigned to the current workspace. Service credentials that are not assigned to the workspace appear grayed out.

1. Log in to a workspace that is linked to the metastore.

2. In the sidebar, click Catalog.

3. On the Quick access page, click the External data > button and go to the Credentials tab.

4. Select the service credential and go to the Workspaces tab.

5. On the Workspaces tab, clear the All workspaces have access checkbox.

   If your service credential is already bound to one or more workspaces, this checkbox is already cleared.

6. Click Assign to workspaces and enter or find the workspaces you want to assign.

To revoke access, go to the Workspaces tab, select the workspace, and click Revoke. To allow access from all workspaces, select the All workspaces have access checkbox.

Next steps

• Learn how to view, update, delete, and grant other users permission to use service credentials. See Manage service credentials.
• Learn how to use service credentials in code. See Use Unity Catalog service credentials to connect to external cloud services.

Limitations

The following limitations apply:

• Databricks Runtime 15.4 LTS includes Python support only.
• SQL warehouses are not supported.
• Some audit events for actions performed on service credentials will not appear in the system.access.audit table. Audit information about who created, deleted, updated, read, listed, or used a service credential will be available. See Audit log system table reference.
• During the service credentials preview, INFORMATION_SCHEMA.STORAGE_CREDENTIALS (deprecated) displays both storage credentials and service credentials, and INFORMATION_SCHEMA.STORAGE_CREDENTIAL_PRIVILEGES (deprecated) displays privileges that apply both to storage credentials and service credentials. This is incorrect preview behavior that will be corrected, and you should not depend on it to continue. You should instead use INFORMATION_SCHEMA.CREDENTIALS and INFORMATION_SCHEMA.CREDENTIAL_PRIVILEGES for both storage and service credentials.