Data Engineering documentation in Microsoft Fabric

Data engineering in Microsoft Fabric enables users to design, build, and maintain infrastructures and systems that enable their organizations to collect, store, process, and analyze large volumes of data.

About Data Engineering

Overview

• What is Data engineering?
• What is a lakehouse?
• Workspace roles and permissions

Get started with lakehouse

Get started

• Create a lakehouse
• Navigate the lakehouse explorer

How-To Guide

• Get data into lakehouse using notebook

Get started with notebooks

Get started

• Create and use notebooks

How-To Guide

• Develop and run notebooks
• Explore lakehouse data with a notebook
• Visualize notebooks
• Notebooks source control and deployment
• Use NotebookUtils in notebook
• Author and run T-SQL Notebooks

API for GraphQL

Overview

• What is Fabric API for GraphQL?

Get started

• Create an API for GraphQL and add data

How-To Guide

• Manage relationships
• Connect applications
• Query multiple data sources

Delta Lake

Concept

• Lakehouse and Delta tables

How-To Guide

• Delta optimization and V-Order
• Load to tables

Lakehouse end-to-end tutorial

Concept

• Lakehouse scenario overview

Tutorial

• Step 1 - Create a workspace
• Step 2 - Create a lakehouse
• Step 3 - Ingest data into lakehouse
• Step 4 - Transform data into delta tables
• Step 5 - Build a report
• Step 6 - Clean up resources

Apache Spark

Concept

• Apache Spark job definition
• Spark monitoring overview
• Apache Spark advisor

Get started

• Create a Spark job definition
• Create a Spark job definition with a Fabric API
• Spark job definition Git integration

How-To Guide

• Schedule and run a Spark job definition
• Monitor a Spark application
• Manage Apache Spark libraries
• How to use Fabric MSSparkUtils
• Configure capacity settings

VS Code integration

How-To Guide

• Use notebooks from VS Code
• Use container image from VS Code
• Use notebooks from VS Code with resource
• Use Spark job definitions from VS Code
• Explore lakehouse from VS Code
• Develop notebook from VS Code web experience

Get started with environment

Get started

• Microsoft Fabric environments

How-To Guide

• Manage Spark compute in environments
• Manage libraries in environments
• Environment Git integration and deployment pipeline
• Manage the environment through public APIs
• Manage the resources in Fabric environment
• Migrate the existing workspace libraries and Spark properties to an environment

Livy API

Overview

• What is the Livy API?

Get started

• Create and run Spark jobs using the Livy API
• Submit Livy API session jobs
• Submit Livy API batch jobs