Share via

Identity best practices

  • Article

This article provides an opinionated perspective on how to best configure identity in Azure Databricks. It includes a guide on how to migrate to identity federation, which enables you to manage all of your users, groups, and service principals in the Azure Databricks account.

For an overview of the Azure Databricks identity model, see Azure Databricks identities.

For information on how to securely access Azure Databricks APIs, see Manage personal access token permissions.

Configure users, service principals, and groups

There are three types of Azure Databricks identity:

  • Users: User identities recognized by Azure Databricks and represented by email addresses.
  • Service principals: Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
  • Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.

Databricks recommends creating service principals to run production jobs or modify production data. If all processes that act on production data run using service principals, interactive users do not need any write, delete, or modify privileges in production. This eliminates the risk of a user overwriting production data by accident.

It is best practice to assign access to workspaces and access-control policies in Unity Catalog to groups, instead of to users individually. All Azure Databricks identities can be assigned as members of groups, and members inherit permissions that are assigned to their group.

The following are the administrative roles that can manage Azure Databricks identities:

  • Account admins can add users, service principals, and groups to the account and assign them admin roles. They can give users access to workspaces, as long as those workspaces use identity federation.
  • Workspace admins can add users, service principals to the Azure Databricks account. They can also add groups to the Azure Databricks account if their workspaces are enabled for identity federation. Workspace admins can grant users, service principals, and groups access to their workspaces.
  • Group managers can manage group membership. They can also assign other users the group manager role.
  • Service principal managers can manage roles on a service principal.

Databricks recommends that there be a limited number of account admins per account and workspace admins in each workspace.

Sync users and groups automatically from Microsoft Entra ID (Public Preview)

You can add users, service principals, and groups from Microsoft Entra ID into Azure Databricks without configuring an application in Microsoft Entra ID using automatic identity management. When automatic identity management is enabled, you can directly search in identity federated workspaces for Microsoft Entra ID users, service principals, and groups and add them to your workspace. Databricks uses Microsoft Entra ID as the source of truth, so any changes to users or group memberships are respected in Azure Databricks.

Users can also share dashboards with any user, service principal, or group in Microsoft Entra ID. Those users are added automatically to the Azure Databricks account upon login.

Automatic identity management is only supported in identity federated workspaces. For more information on identity federation, see Sync users and groups automatically from Microsoft Entra ID.

Enable identity federation

Identity federation enables you to configure users, service principals, and groups in the account console, and then assign those identities access to specific workspaces. This simplifies Azure Databricks administration and data governance.

Important

Databricks began to enable new workspaces for identity federation and Unity Catalog automatically on November 9, 2023, with a rollout proceeding gradually across accounts. If your workspace is enabled for identity federation by default, it cannot be disabled. For more information, see Automatic enablement of Unity Catalog.

With identity federation, you configure Azure Databricks users, service principals, and groups once in the account console, rather than repeating the configuration separately in each workspace. Once users, service principals, and groups are added to the account, you can assign them permissions on workspaces. You can only assign account-level identities access to workspaces that are enabled for identity federation.

Account-level identity diagram

To enable a workspace for identity federation, see Enable identity federation. When the assignment is complete, identity federation is marked as Enabled on the workspace’s Configuration tab in the account console.

Identity federation is enabled on the workspace-level, and you can have a combination of identity federated and non-identity federated workspaces. For those workspaces that are not enabled for identity federation, workspace admins manage their workspace users, service principals, and groups entirely within the scope of the workspace (the legacy model). They cannot use the account console or account-level APIs to assign users from the account to these workspaces, but they can use any of the workspace-level interfaces. Whenever a new user or service principal is added to a workspace using workspace-level interfaces, that user or service principal is synchronized to the account-level. This enables you to have one consistent set of users and service principals in your account.

However, when a group is added to a non-identity federated workspace using workspace-level interfaces, that group is a workspace-local group and is not added to the account. You should aim to use account groups rather than workspace-local groups. Workspace-local groups cannot be granted access-control policies in Unity Catalog or permissions to other workspaces.

Upgrade to identity federation

If you are enabling identity federation on an existing workspace, do the following:

  1. Migrate workspace-level SCIM provisioning to the account level

    If you have a workspace-level SCIM provisioning set up your workspace, you should set up account-level SCIM provisioning and turn off the workspace-level SCIM provisioner. Workspace-level SCIM will continue to create and update workspace-local groups. Databricks recommends using account groups instead of workspace-local groups to take advantage of centralized workspace assignment and data access management using Unity Catalog. Workspace-level SCIM also does not recognize account groups that are assigned to your identity federated workspace and workspace-level SCIM API calls will fail if they involve account groups. For more information about how to disable workspace-level SCIM, see Migrate workspace-level SCIM provisioning to the account level.

  2. Convert workspace-local groups to account groups

    Databricks recommends converting your existing workspace-local groups to account groups. See Migrate workspace-local groups to account groups for instructions.

Assign groups workspace permissions

Now that identity federation is enabled on your workspace, you can assign the users, service principals, and groups in your account permissions on that workspace. Databricks recommends that you assign groups permissions to workspaces instead of assigning workspace permissions to users individually. All Azure Databricks identities can be assigned as members of groups, and members inherit permissions that are assigned to their group.

Add workspace permissions

Learn more