Manage Unity Catalog metastores

This article shows how to update, delete, and manage the behavior of Unity Catalog metastores in your Azure Databricks account.

To learn about Unity Catalog metastores and how to create them, see Create a Unity Catalog metastore.

Enable a metastore to be automatically assigned to new workspaces

To assign an existing Unity Catalog metastore automatically to new workspaces in that metastore’s region, an account admin can enable workspace auto-assignment for the metastore. If this setting is not selected, the admin who creates a workspace in the same region as the metastore must manually enable the workspace for Unity Catalog and select the metastore from a drop-down.

Before an account admin enables this option, they should be sure to understand the following impacts on new workspaces:

• A workspace catalog will be created, and all workspace users will have the privileges required to create objects in it. See Automatic enablement of Unity Catalog.
• Workspace admins will have the permissions required to create metastore-level securable objects, like catalogs and external locations. See Workspace admin privileges when workspaces are enabled for Unity Catalog automatically.
• If metastore-level storage is already enabled for the metastore, the workspace will be able to use that storage. See Specify a managed storage location in Unity Catalog.
• If a metastore admin is defined for the metastore, they will be able to manage access to all securable objects in all workspaces attached to the metastore. See Metastore admins.
• The Delta Sharing setting (enabled or disabled) for the metastore will apply to all workspaces attached to the metastore. See Set up Delta Sharing for your account (for providers).

To enable automatic assignment:

1. As an account admin, go to the Azure Databricks account console.
2. Click Catalog.
3. Select your metastore.
4. On the Configuration tab, under Workspace assignment, select Automatically assign new workspaces in <region> to this metastore.
5. On the confirmation dialog, click Enable auto-assignment.

Add managed storage to an existing metastore

Metastore-level managed storage is optional, and it is not included for metastores that were created automatically. You might want to add metastore-level storage to your metastore if you prefer a data isolation model that stores data centrally for multiple workspaces. You need metastore-level storage if you are a Azure Databricks partner who uses personal staging locations.

See also Specify a managed storage location in Unity Catalog.

Requirements

• You must have at least one workspace attached to the Unity Catalog metastore.
• Azure Databricks permissions required:
  • To create an external location, you must be a metastore admin or user with the CREATE EXTERNAL LOCATION and CREATE STORAGE CREDENTIAL privileges.
  • To add the storage location to the metastore definition, you must be an account admin. For instructions on enabling the account admin role in your account, see Establish your first account admin.
• Azure tenant permissions required:
  • Permission to create a storage account to use with Azure Data Lake Storage Gen2. This storage account must have a hierarchical namespace. See Create a storage account to use with Azure Data Lake Storage Gen2.
  • Permission to create a new resource to hold a system-assigned managed identity. This requires that you be a Contributor or Owner of a resource group in any subscription in the tenant.

Step 1: Create the storage location

Follow the instructions in Step 1 (Optional): Create a storage container for metastore-level managed storage and Step 2 (Optional): Create a managed identity to access the managed storage location to create a storage container in Azure Data Lake Storage Gen2 and an Azure Databricks access connector that holds a managed identity that has access to the storage container.

Step 2: Create an external location in Unity Catalog

In this step, you create an external location in Unity Catalog that references the ADLS Gen 2 path that you just created.

1. Create a storage credential.

   The storage credential will represent the Azure managed identity that you created in Step 1: Create the storage location.

   Follow the instructions in Create a storage credential for connecting to Azure Data Lake Storage Gen2.

2. Create an external location that references the storage credential that you created in the previous step and the ADLS Gen 2 storage container that you created in Step 1: Create the storage location.

   Follow the instructions in Create an external location to connect cloud storage to Azure Databricks

3. Grant yourself the CREATE MANAGED STORAGE privilege on the external location.

   1. Click the external location name to open the details pane.
   2. On the Permissions tab, click Grant.
   3. On the Grant on <external location> dialog, select yourself in the Principals field and select CREATE MANAGED STORAGE.
   4. Click Grant.

Step 3: Add the storage location to the metastore

After you have created an external location that represents the metastore storage bucket, you can add it to the metastore.

1. As an account admin, log in to the account console.

2. Click Catalog.

3. Click the metastore name.

4. Confirm that you are the Metastore Admin.

   If you are not, click Edit and assign yourself as the metastore admin. You can unassign yourself when you are done with this procedure.

5. On the Configuration tab, next to ADLS Gen 2 path, click Set.

6. On the Set metastore root dialog, enter the ADLS Gen 2 path that you used to create the external location, and click Update.

   You cannot modify this path once you set it, but you can remove it and add a new path if necessary.

Remove metastore-level storage

If you have metastore-level storage for managed tables and volumes (also known as the metastore storage root), but you want to enforce data storage isolation at the catalog or schema level, you can remove the metastore-level storage option for the metastore. When you do, the following happens:

• Existing catalogs that have no storage root specified are given the metastore storage root’s cloud storage location as their catalog-level managed storage location. In other words, the metastore storage root is “pushed down” to these catalogs. Access to data in these catalogs continues to function without interruption.
• Depending on how your metastore was created, there might not be an external location securable defined in Unity Catalog for the metastore storage root. In that case, a new external location and associated storage credential are created for it. The new external location is named prior_metastore_root_location by default.
• Every time a user creates a catalog, they must provide a dedicated storage location that is registered in Unity Catalog as an external location.

Note

If you use Delta Sharing to share notebooks and you used the metastore root as shared notebook storage, you must do the following before you can remove the metastore root:

1. Remove your notebook from the share.
2. Re-add the notebook using a dedicated storage location.

See Add notebook files to a share.

To remove the metastore storage root:

1. As an account admin, log in to the account console.
2. Click Catalog.
3. Click the metastore name.
4. On the Configuration tab, under ADLS Gen 2 path, click the Remove button.
5. On the confirmation dialog, click Remove.

Add a metastore admin

Metastore admins are optional, but there are situations where you might want one for your metastore. See Assign a metastore admin.

Delete a metastore

If you are closing your Azure Databricks account or have another reason to delete access to data managed by your Unity Catalog metastore, you can delete the metastore.

Warning

All objects managed by the metastore will become inaccessible using Azure Databricks workspaces. This action cannot be undone.

Managed table data and metadata will be auto-deleted after 30 days. External table data in your cloud storage is not affected by metastore deletion.

To delete a metastore:

1. As a metastore admin, log in to the account console.
2. Click Catalog.
3. Click the metastore name.
4. On the Configuration tab, click the three-button menu at the far upper right and select Delete.
5. On the confirmation dialog, enter the name of the metastore and click Delete.