Data engineering with Databricks
Databricks provides a rich set of data engineering features that include a robust environment for collaboration among data scientists, engineers, and analysts. Through the center of it are the pipelines that ingest and transform the data. To enable easy engineering of the most efficient pipelines, Databricks offers Structured Streaming and Delta Live Tables. See Streaming on Azure Databricks and What is Delta Live Tables?.
For an overview of why incremental stream processing of data provided by Structured Streaming and Delta Live Tables is the best option for data pipelines, see Why incremental stream processing?.
Additional data engineering resources
Databricks provides features for data engineering tasks for many additional scenarios:
- Databricks notebooks are a popular tool for collaboration.
- Schedule and orchestrate your workflows using Databricks Jobs.
- See Databricks Mosaic AI if you are architecting machine learning solutions.
Note
If you primarily work with SQL queries and BI tools, see Databricks SQL.
For getting started tutorials and introductory informaton, see Get started with Azure Databricks and What is Azure Databricks?.
| Name | Use this when you want to… |
|---|---|
| Delta Live Tables | Learn how to build data pipelines for ingestion and transformation with Databricks Delta Live Tables. |
| Structured Streaming | Learn about streaming, incremental, and real-time workloads powered by Structured Streaming on Databricks. |