Lakehouse deployment pipelines and git integration (Preview)

The Lakehouse integrates with the lifecycle management capabilities in Microsoft Fabric, providing a standardized collaboration between all development team members throughout the product's life. Lifecycle management facilitates an effective product versioning and release process by continuously delivering features and bug fixes into multiple environments. To learn more, see What is lifecycle management in Microsoft Fabric?.

Important

This feature is in preview.

Lakehouse git integration

The Lakehouse is an item that contains both metadata and data that is referenced in multiple objects in the workspace. Lakehouse contains tables, folders, and shortcuts as primary manageable data container items. From a development workflow perspective, the following dependent objects might reference a Lakehouse:

The default semantic model and SQL analytics endpoint metadata are related to a Lakehouse and managed by the git update process by default. As a principle data is not tracked in git, only metadata is tracked.

Git representation

The following lakehouse information is serialized and tracked in a git connected workspace:

  • Display name
  • Description
  • Logical guid

Note

The tracked logical guid is an automatically generated cross-workspace identifier representing an item and its source control representation.

Important

Only the Lakehouse container artifact is tracked in git in the current experience. Tables (Delta and non-Delta) and Folders in the Files section aren't tracked and versioned in git.

Lakehouse git integration capabilities

The following capabilities are available:

  • Serialization of the Lakehouse object metadata to a git JSON representation.
  • Apply changes directly or use pull request to control changes to upstream or downstream workspaces and branches.
  • Renaming lakehouses are tracked in git. Updating a renamed lakehouse also renames the default semantic data model and SQL Analytics endpoint.
  • No action is applied to tables and folders metadata, and data of those items is always preserved.
  • OneLake Shortcuts metadata is preserved in git.

OneLake Shortcuts git integration capabilities

  • Shortcuts definitions in both the Tables and Files section are stored in a file named shortcuts.metadata.json under the lakehouse folder in git.
  • The following operations are supported and tracked automatically: addition, deletion and updates of Shortcuts.
  • The operations can be performed directly in the Fabric user interface or in the git repository by changing the shortcuts.metadata.json file.

Important

Use caution when changing OneLake Shortcut properties directly in the shortcuts.metadata.json file. Incorrect changes to the properties, specially GUIDs, can render the OneLake Shortcut invalid when updates are applied back to the workspace.

Lakehouse in deployment pipelines

The Lakehouse is supported in Microsoft Fabric lifecycle management deployment pipelines. It enables environment segmentation best-practices.

Lakehouse deployment pipelines integration capabilities:

  • Deployment across dev, test, and production workspaces.

  • Lakehouse can be removed as a dependent object upon deployment. Mapping different Lakehouses within the deployment pipeline context is also supported.

    • If nothing is specified during deployment pipeline configuration, a new empty Lakehouse object with same name is created in the target workspace. Notebook and Spark Job Definitions are remapped to reference the new Lakehouse object in the new workspace.

    • If the Lakehouse dependency is configured to reference a different Lakehouse during deployment pipeline configuration time, such as the upstream Lakehouse, a new empty Lakehouse object with same name still is created in the target workspace, but Notebooks and Spark Job Definitions references are preserved to a different Lakehouse as requested.

    • SQL Analytics endpoints and semantic models are provisioned as part of the Lakehouse deployment.

  • No object inside the Lakehouse is overwritten.

  • Updates to Lakehouse name can be synchronized across workspaces in a deployment pipeline context.

OneLake Shortcuts in deployment pipelines

  • Shortcuts definitions are synced across stages in the deployment pipelines.
  • Shortcuts with external targets (ADLS Gen2, S3, etc) will be the same across all stages.
  • Shortcuts with internal targets (OneLake Shortcuts) in the same workspace are automatically remapped across stages. Shortcuts targeting Data Warehouse and Semantic Models will not be remapped. Tables, Folders and Files will not be created in the target. In order for the Shortcut to be valid, those need to be created in the target workspace.
  • On the scenario that the same Shortcut needs to target different locations on different stages. For example, in Development point to a specific Folder in Amazon S3, and in Production a different folder in ADLS Gen2. After the deployment, update the OneLake Shortcut definition in Lakehouse or directly using OneLake APIs.

Important

A deployment will override the state of shortcuts in the target workspace. All the Shortcuts in the target lakehouse will be updated or deleted based on the state in the source lakehouse. New shortcuts will be created in the target lakehouse.