Edit

Share via

Mirroring Azure Databricks Unity Catalog (Preview)

  • Article

Many organizations today register their data in Unity Catalog within Azure Databricks. A mirrored Unity Catalog in Fabric enables customer to read data managed by Unity Catalog from Fabric workloads. Azure Databricks and Fabric are better together.

For a tutorial on configuring your Azure Databricks Workspace for mirroring the Unity Catalog into Fabric, see Tutorial: Configure Microsoft Fabric mirrored databases from Azure Databricks (Preview).

Mirrored databases in Fabric allow users to enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs. You can enjoy an easy-to-use product designed to simplify your analytics needs and built for openness and collaboration between Microsoft Fabric and Azure Databricks.

When you use Fabric to read data that is registered in Unity Catalog, there is no data movement or data replication. Only the Azure Databricks catalog structure is mirrored to Fabric and the underlying catalog data is accessed through shortcuts. Hence any changes in data are reflected immediately in Fabric.

What analytics experiences are built in

Mirrored catalogs are an item in Fabric Data Warehousing distinct from the Warehouse and SQL analytics endpoint.

When you mirror an Azure Databricks Unity Catalog, Fabric creates three items:

You can access your mirrored Azure Databricks data multiple ways:

  • Each mirrored Azure Databricks item has an autogenerated SQL analytics endpoint that provides a rich analytical experience created by the mirroring process. Use T-SQL commands to define and query data objects from the read-only SQL analytics endpoint.
  • Use Power BI with Direct Lake mode to create reports against the Azure Databricks item.

Metadata sync

When you create a new mirrored database from Azure Databricks in Fabric, by default, the Automatically sync future catalog changes for the selected schema is enabled. The following metadata changes are reflected from your Azure Databricks workspace to Fabric if automatic sync is enabled:

  • Addition of schemas to a catalog.
  • Deletion of schemas from a catalog.
  • Addition of tables to a schema.
  • Deletion of tables from a schema.

Schema/table selection:

  • By default, the entire catalog is selected when the user adds the catalog.
  • The user can exclude certain tables within the schema.
  • Unselecting a schema unselects all the tables within the schema.
  • If the user goes back and selects the schema, all tables within the schema are selected again.
  • Same selection behavior applies to schemas within a catalog.

There are other filtration conditions that are applied to catalogs/schemas/tables:

  • Materialized views and streaming tables will not be displayed.
  • External tables that don't support Delta format will not be displayed.

Related content