Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview

This article outlines how to register Azure Databricks, and how to authenticate and interact with Azure Databricks Unity Catalog in Microsoft Purview. For more information about Microsoft Purview, read the introductory article.

Supported capabilities

Metadata Extraction	Full Scan	Incremental Scan	Scoped Scan	Classification	Labeling	Access Policy	Lineage	Data Sharing	Live view
Yes	Yes	Yes	Yes	Yes	No	No	Yes	No	No

When scanning Azure Databricks Unity Catalog, Microsoft Purview supports:

• Extracting technical metadata including:
  • Metastore
  • Catalogs
  • Schemas
  • Tables including the columns
  • Views including the columns
• Fetching lineage on assets relationships between tables, views, columns during notebook runs.

When setting up scan, you can choose to scan the entire Unity Catalog, or scope the scan to a subset of catalogs.

Note

This connector brings metadata from Azure Databricks Unity Catalog. To scan Azure Databricks workspace-scoped metadata, refer to Azure Databricks Hive Metastore connector.

Known limitations

• When object is deleted from the data source, currently the subsequent scan won't automatically remove the corresponding asset in Microsoft Purview.
• For more details on other limitations related to native Azure Databricks lineage, refer to Azure Databricks documentation.

Prerequisites

• You must have an Azure account with an active subscription. Create an account for free.

• You must have an active Microsoft Purview account.

• You need an Azure Key Vault, and to grant Microsoft Purview permissions to access secrets.

• You need Data Source Administrator and Data Reader permissions to register a source and manage it in the Microsoft Purview governance portal. For more information about permissions, see Access control in Microsoft Purview.

• To scan Azure Databricks Unity Catalog, Microsoft Purview connects to a SQL Warehouse in your workspace, and uses Personal Access Token for authentication. You need to have an Azure Databricks workspace that is Unity Catalog enabled and attached to the metastore you want to scan. In your Azure Databricks workspace:

  • Create a SQL Warehouse. You can use the autocreated Starter warehouse as well if applicable.

    • Note down the HTTP path. You can find it in Azure Databricks workspace -> SQL Warehouses -> your warehouse -> Connection details -> HTTP path.

    • Make sure the user has the Can Use permission so as to connect to the Azure Databricks SQL warehouse. Learn more from SQL warehouse access control.

• To fetch lineage from Azure Databricks using Microsoft Purview, the following prerequisites must be in place:

  • Enable the system schema: The system schema system.access must be enabled in your Unity Catalog. This is required because lineage information is stored in system tables, and enabling this schema allows access to those tables. Learn more about monitoring usage with system tables.

  • User privileges: The user account used for scanning needs to have SELECT privileges on the following system tables:

    • system.access.table_lineage

    • system.access.column_lineage

    These permissions are required because lineage data is read directly from the system tables, and without the necessary access, Microsoft Purview cannot retrieve the lineage information.

• If your Azure Databricks workspace doesn’t allow access from public network or if your Microsoft Purview account doesn’t enable access from all networks, you can use the Managed Virtual Network Integration Runtime or a kubernetes supported self-hosted integration runtime to scan. You can set up a managed private endpoint for Azure Databricks as needed to establish private connectivity.

Authentication for a scan

You can use personal access tokens, managed identities or service principal authentication methods for scanning Azure Databricks Unity Catalog.

System or user assigned managed identity

If using a system or user assigned managed identity

1. Select the system-assigned or user-assigned managed identity under Credential.

For using a user-assigned managed identity, you must set up Azure managed identities authentication for Azure Databricks.

Personal Access Token

If using Personal Access Token

1. Generate a personal access token, and store it as a secret in Azure Key Vault.

2. While setting up a scan, select the + New under Credential and register the credential.

3. Select the appropriate credential under Credential during scan setup.

Service Principal

If using Service Principal

1. Setup service principal authentication for Azure Databricks.

2. Select the + New under Credential.

3. Select the appropriate Key vault connection and the Secret name that was used while creating the Service Principal.

• For all the objects that you want to bring into Microsoft Purview, the user or service principal needs to have at least SELECT privilege on tables/views, USE CATALOG on the object’s catalog, and USE SCHEMA on the object’s schema.

  • In order to scan all the objects in a Unity Catalog metastore, use a user or service principal with metastore admin role. Learn more from Manage privileges in Unity Catalog and Unity Catalog privileges and securable objects.

  • For classification, user also needs to have SELECT privilege on the tables/views to retrieve sample data.

Register

This section describes how to register an Azure Databricks workspace in Microsoft Purview by using the Microsoft Purview governance portal.

1. Go to your Microsoft Purview account.

2. Select Data Map on the left pane.

3. Select Register.

4. In Register sources, select Azure Databricks Unity Catalog > Continue.

5. On the Register sources (Azure Databricks Unity Catalog) screen, do the following:

6. For Name, enter a name that Microsoft Purview will list as the data source.

   1. For Metastore ID, provide the metastore ID for the Azure Databricks Unity Catalog metastore that you want to scan.

   2. Select a collection from the list.

7. Select Finish.

Scan

Tip

To troubleshoot any issues with scanning:

1. Confirm you have followed all prerequisites.
2. Review our scan troubleshooting documentation.

Use the following steps to scan Azure Databricks to automatically identify assets. For more information about scanning in general, see Scans and ingestion in Microsoft Purview.

1. Go to Sources.

2. Select the registered Azure Databricks.

3. Select + New scan.

4. Provide the following details:

   1. Name: Enter a name for the scan.

   2. Connect via integration runtime: Choose the default Azure integration runtime, Managed VNet IR, or a Kubernetes supported self-hosted integration runtime you created.

   3. Credential: Select the credential to connect to your data source. Make sure to:

      • Select Access Token, Managed Identity or Service Principal.
      • You can create a new Access Token or Service Principal credential while registering a scan. For more information, see Credentials for source authentication in Microsoft Purview.

   4. Workspace URL: Provide the URL for the workspace that you want to scan.

   5. HTTP path: Specify the Databricks SQL Warehouse’s HTTP path that Microsoft Purview will connect to and perform the scan, e.g. /sql/1.0/endpoints/xxxxxxxxxxxxxxxx. You can find it in Azure Databricks workspace -> SQL Warehouses -> your warehouse -> Connection details -> HTTP path.

   6. Lineage extraction: Toggle lineage extraction to On to fetch lineage of the scanned assets.

5. Select Test connection to validate the settings.

   

6. Select Continue.

7. For Scan trigger, choose whether to set up a schedule or run the scan once.

8. Review your scan and select Save and Run.

Once the scan successfully completes, see how to browse and search assets.

View your scans and scan runs

To view existing scans:

1. Go to the Microsoft Purview portal. On the left pane, select Data map.
2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.
3. Select the scan that has results you want to view. The pane shows you all the previous scan runs, along with the status and metrics for each scan run.
4. Select the run ID to check the scan run details.

Manage your scans

To edit, cancel, or delete a scan:

1. Go to the Microsoft Purview portal. On the left pane, select Data Map.

2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.

3. Select the scan that you want to manage. You can then:

   • Edit the scan by selecting Edit scan.
   • Cancel an in-progress scan by selecting Cancel scan run.
   • Delete your scan by selecting Delete scan.

Note

• Deleting your scan does not delete catalog assets created from previous scans.

Browse and search assets

After scanning your Azure Databricks, you can browse Unified Catalog or search Unified Catalog to view the asset details and lineage.

When browsing by source types, you see two entries for Azure Databricks Unity Catalog and Azure Databricks respectively. The former contains the Unity Catalog artifacts including the metastore and its catalogs/schemas/tables/views, while the latter contains the workspace artifacts.

From the Azure Databricks workspace asset, you can find the associated Unity Catalog under Properties tab, reversed applies too.

Lineage

When browsing a particular Azure Databricks asset, you can see the notebooks that have captured lineage.

Go to the asset -> lineage tab, you can see the lineage on the Azure Databricks Notebook asset or table/view asset when applicable.

Refer to the supported capabilities section on the supported Databricks Unity Catalog lineage scenarios. For more information about lineage in general, see data lineage and lineage user guide.

Frequently asked questions (FAQ)

Is column level lineage from Unity Catalog captured by Microsoft Purview?

Microsoft Purview can capture lineage at both the Unity Catalog table/view level and the column level.

I don't see column level lineage, what's happening?

Column level lineage is generated when your notebook is run from a cluster and not generated through a SQL warehouse.

I am getting a timeout error, what do I do?

When there's a large volume of assets in your workspace, your scan could fail to complete. In this case, you can scope your scan to a few catalogs at a time which will reduce the volume of assets per scan and allow your scans to complete.

I just ran my notebook, but Microsoft Purview didn't fetch the lineage. What’s happening?

There might be a few minutes' delay for Databricks to update the lineage information in its system tables after your notebook execution. Microsoft Purview will be able to fetch the lineage once the system tables are updated.

Next steps

Now that your source is registered, use the following guides to learn more about Microsoft Purview and your data:

• Search the Unified Catalog
• Data Estate Insights in Microsoft Purview