Data lake schema enforcement

Question

Hello, In Data Lake data is processed or ingested as schema on read and that is data is read in it format that it comes from the source. But I read an article that says schema enforcement makes data lakes high-performance and data readable. Please educate me on how to enforce schema on data lake and if it is possible, how many ways we can enforce schema in ADLS. Thank you.

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A.

In ADLS, enforcing schema can be a powerful way to ensure data consistency, optimize performance, and facilitate easier data processing and analytics. Even though data lakes traditionally use a "schema-on-read" approach, which is flexible, it also means that data is stored in its raw form without predefined structure. Schema enforcement helps ensure that data adheres to specific structural rules, which makes it more manageable and performant. Here’s how you can implement schema enforcement in ADLS and the methods available to do so:

1. Schema Enforcement Methods in ADLS

There are several approaches to enforce schema in ADLS. Each approach has its specific use cases, pros, and cons:

A. Data Lake Storage Format with Schema Support (Parquet, Delta Lake)

• Parquet: A columnar storage format that supports complex data types and enforces schema on data read and write operations. Schema is stored in the metadata, making it ideal for analytics and optimized queries.
• Delta Lake: A layer built on top of ADLS that adds ACID transaction support and schema enforcement. Delta Lake can manage schema at the file level and allows schema evolution, enforcing schema consistency across different versions.
• Usage: These formats make data querying faster and more reliable as they ensure a schema is always present when the data is read.

B. Using Synapse Analytics or Databricks for Schema Enforcement

• When ingesting data into ADLS via Synapse or Databricks, schema enforcement can be applied by defining the expected schema during ingestion.
• These platforms allow you to define and enforce the schema by reading and writing in formats like Parquet and Delta Lake. They provide schema validation to ensure that incoming data matches the predefined schema and can enforce schema evolution policies (e.g., allowing new columns but preventing changes to existing column types).

C. ADF Data Flows with Data Lake Schema Mapping

• ADF offers Data Flows that can define a schema for source and sink datasets, allowing transformations that include schema validation.
• Schema drift and projection options in ADF Data Flows allow you to enforce schema by checking for expected columns and data types, and handling mismatches before data lands in ADLS.

D. Azure Purview for Data Governance and Schema Definition

• Azure Purview is a data cataloging and governance tool that can catalog schema definitions for your data assets in ADLS.
• With Purview, you can define schemas, track lineage, and set data classifications that help enforce schema consistency across data sources. While Purview doesn’t enforce schema directly, it plays a role in schema governance and can flag data that doesn't conform to expected schemas.

E. Ingestion Policies with Azure Event Grid and Azure Functions

• Event-driven ingestion policies using Azure Event Grid and Azure Functions can enforce schema by validating the schema of data files upon arrival in ADLS.
• Functions can validate the incoming data schema against a pre-defined template, only allowing data that matches the schema to be ingested. Mismatched data can be routed for correction or rejected altogether.

2. How to Enforce Schema in ADLS

• Define Schema at Ingestion Time: Use Synapse Analytics or Databricks to define schemas and apply validation rules during ingestion. Use schema validation options in Data Flows for transformations.
• Use Schema-Aware File Formats: Parquet and Delta Lake file formats inherently store schema metadata. Writing to these formats ensures that the schema is enforced when the data is read.
• Schema Drift Handling: In ADF Data Flows, use schema projection to manage schema drift, either by allowing or disallowing changes based on your governance needs.
• Custom Scripts or Functions: For finer control, use Azure Functions to write scripts that validate schema upon ingestion. Functions can be triggered by file events in ADLS (e.g., file creation) and check schema compliance before the data is processed.

3. Best Practices for Schema Enforcement in ADLS

• Use Delta Lake for Data with Frequent Updates: For transactional data or data that requires upserts, Delta Lake provides schema enforcement with ACID guarantees.
• Optimize with Columnar Formats (Parquet): Use Parquet for high-performance analytics workloads. Ensure that schemas are consistent across Parquet files for efficient querying.
• Implement a Data Catalog: Use Azure Purview to manage schema definitions and data lineage, ensuring that all data adheres to expected schemas.

To summarise:

While ADLS typically uses a schema-on-read approach, schema enforcement can improve data management and performance. By using schema-aware formats (like Parquet and Delta Lake), leveraging Synapse and ADF Data Flows for validation, and utilizing governance tools like Purview, you can effectively enforce schema in your data lake. Each approach offers unique advantages based on data type, structure, and use case requirements.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.

Answer 2

Hi @azure_learner

In Azure Data Lake, schema enforcement helps improve data quality and performance by making sure the data follows a specific structure. This is especially important when using Delta Lake in Azure Databricks, which provides tools to enforce this structure.

Here’s how schema enforcement works in simple terms:

1. Schema on Write: When data is added to a Delta table, Azure Databricks makes sure that all the columns you’re inserting match the table’s structure. This means the columns must already exist in the table, and the data types must be correct. This ensures that only valid data gets added, keeping the data lake clean and consistent.
2. Schema Validation during MERGE Operations: When you use the MERGE operation to insert, update, or delete data, Azure Databricks checks if the data types match the target columns in the table. If there’s a mismatch, it tries to adjust the data types to fit the table’s schema. This ensures that any changes made to the data still follow the correct structure.
3. Automatic Schema Evolution: Delta Lake can automatically adjust the schema when new columns are added, or existing columns change. This means you don’t have to rewrite all your data to change the schema. It makes it easier to adapt to new data needs while keeping everything consistent.
4. Explicit ALTER TABLE Statements: If you need more control, you can manually change the schema of a table using ALTER TABLE commands. This lets you make specific updates to the table structure when needed.

By using these methods, Azure Databricks and Delta Lake help ensure your data in Azure Data Lake is accurate, high-quality, and easy to work with, while allowing for flexible changes to the schema as your data evolves.

For Additional information, please refer the below documents:

• Schema enforcement
• Clean and validate data with batch or stream processing

I hope this information helps. Please do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Thank you.