Share via

Microsoft Purview Data Quality overview

  • Article

Note

The Microsoft Purview Data Catalog is changing its name to Microsoft Purview Unified Catalog. All the features will stay the same. You'll see the name change when the new Microsoft Purview Data Governance experience is generally available in your region. Check the name in your region.

Microsoft Purview Data Quality is a comprehensive solution that empowers governance domain and data owners to assess and oversee the quality of their data ecosystem, facilitating targeted actions for improvement. In today's AI-driven landscape, the reliability of data directly impacts the accuracy of AI-driven insights and recommendations. Without trustworthy data, there's a risk of eroding trust in AI systems and hindering their adoption.

Poor data quality or incompatible data structures can hamper business processes and decision-making capabilities. Microsoft Purview Data Quality addresses these challenges by offering users the ability to evaluate data quality using no-code/low-code rules, including out-of-the-box (OOB) rules and AI-generated rules. These rules are applied at the column level and aggregated to provide scores at the levels of data assets, data products, and governance domains, ensuring end-to-end visibility of data quality within each domain.

Microsoft Purview Data Quality also incorporates AI-powered data profiling capabilities, recommending columns for profiling while allowing human intervention to refine these recommendations. This iterative process not only enhances the accuracy of data profiling but also contributes to the continuous improvement of the underlying AI models.

By applying Microsoft Purview Data Quality, organizations can effectively measure, monitor, and enhance the quality of their data assets, bolstering the reliability of AI-driven insights and fostering confidence in AI-based decision-making processes.

Data quality life cycle

  1. Assign users(s) data quality steward permissions in your data catalog to use all data quality features.
  2. Register and scan a data source in your Microsoft Purview Data Map.
  3. Add your data asset to a data product
  4. Set up a data source connection to prepare your source for data quality assessment.
  5. Configure and run data profiling for an asset in your data source.
    1. When profiling is complete, browse the results for each column in the data asset to understand your data's current structure and state.
  6. Set up data quality rules based on the profiling results, and apply them to your data asset.
  7. Configure and run a data quality scan on a data product to assess the quality of all supported assets in the data product.
  8. Review your scan results to evaluate your data product's current data quality.
  9. Repeat steps 5-8 periodically over your data asset's life cycle to ensure it's maintaining quality.
  10. Continually monitor your data quality
    1. Review data quality actions to identify and resolve problems.
    2. Set data quality notifications to alert you to quality issues.

Supported Data Quality Regions

Microsoft Purview Data Quality is currently supported in the following regions

Supported multi-cloud data sources

  • Azure Data Lake Storage (ADLS Gen2)
    • File Types: Delta Parquet and Parquet
  • Azure SQL Database
  • Fabric data estate in OneLake including shortcut and mirroring data estate. Data Quality scanning is supported only for Lakehouse delta tables and parquet files.
    • Mirroring data estate: CosmosDB, Snowflake, Azure SQL
    • Shortcut data estate: AWS S3, GCS, AdlsG2, and dataverse
  • Azure Synapse serverless and data warehouse
  • Azure Databricks Unity Catalog
  • Snowflake
  • Google Big Query (Private Preview)

Important

Data Quality for Parquet file is designed to support:

  1. A directory with Parquet Part File. For example: ./Sales/{Parquet Part Files}. The Fully Qualified Name must follow https://(storage account).dfs.core.windows.net/(container)/path/path2/{SparkPartitions}. Make sure we do not have {n} patterns in directory/sub-directory structure, must rather be a direct FQN leading to {SparkPartitions}.
  2. A directory with Partitioned Parquet Files, partitioned by Columns within the dataset like sales data partitioned by year and month. For example: ./Sales/{Year=2018}/{Month=Dec}/{Parquet Part Files}.

Both of these essential scenario which present a consistent parquet dataset schema are supported.
Limitation: It is not designed to or will not support N arbitrary Hierarchies of Directories with Parquet Files. We advise the customer to present data in (1) or (2) constructed structure.

Currently, Microsoft Purview can only run data quality scans using Managed Identity as authentication option. Data Quality services run on Apache Spark 3.4 and Delta Lake 2.4.

Data quality features

  • Data source connection configuration
    • Configure connection to allow Purview DQ SaaS application to have read access to data for Quality scanning and profiling.
    • MS Purview uses Managed Identity as an authentication option
  • Data profiling
    • AI enabled data profiling experience
    • Industry standard statistical snapshot (distribution, min, max, standard deviation, uniqueness, completeness, duplicate, …)
    • Drill down column level profiling measures.
  • Data quality rules
    • Out of box rules to measure six industry standards Data quality dimensions (completeness, consistency, conformity, accuracy, freshness, and uniqueness)
    • Custom rules creation features include number of out of the box functions and expression values.
    • Auto generated rules with AI integrated experience
  • Data quality scanning
    • Select and assign rules to columns for data quality scan.
    • Apply Data freshness rule in the entity / table level to measure the data freshness SLA.
    • Scheduling data quality scanning job for time period (hourly, daily, weekly, monthly, etc.)
  • Data quality job monitoring
    • Enable monitoring data quality job status (active, completed, failed, etc.)
    • Enable browsing the DQ scanning history.
  • Data quality scoring
    • Data Quality score in rule level (what is the quality score for a rule that applied to a column)
    • Data Quality score for Data assets, Data Products, and Governance Domains (one governance domain can have many data products, one data product can have many data assets, one data asset can have many data columns)
  • Data quality for critical data elements (CDEs)
    • This is one of the key features of Purview Data Quality, it's ability to apply data quality rules to the logical construct of CDEs, which then propagate down to the physical data elements that comprise them. By defining data quality rules at the level of CDEs, organizations can establish specific criteria and thresholds that CDEs must meet to maintain their quality
  • Data quality alerts
    • Configure alerts to notify data owners, data stewards if data quality threshold missed the expectation.
    • configure email alias or distribution group to send the notification about data quality issues.
  • Data quality actions
    • Actions center for DQ with actions to address DQ anomaly states including diagnostic queries for DQ steward to zero in on the specific data to fix for each anomaly state.
  • Data quality managed virtual network
    • A virtual network managed by data quality that connects with private endpoints to your Azure data sources.

Limitation

  • vNet is not supported for Google Big Query, Snowflake, and Azure Data bricks Unity catalog.

Next steps

  1. Assign users(s) data quality steward permissions in your data catalog to use all data quality features.
  2. Set up a data source connection to prepare your source for data quality assessment.
  3. Configure and run data profiling for an asset in your data source.