pre-data validation in Azure

Question

When the data is CSV or database we could do pre-data load validation such as number of rows, size of the tables and no of the table, etc but when the source is given as report as data and consumed by RaaS through API, how the pre-data load validation should be done to ensure the data loaded in ADLS is consistent to source data. Please suggest

Answer 1

Hello @azure_learner

Based on the information you provided, it seems like you are looking for a way to validate the data that is being loaded into ADLS from a report as data source, which is consumed by RaaS through API.

One way to ensure the data loaded in ADLS is consistent with the source data is to perform data validation checks after the data has been loaded into ADLS.

You can use tools like Azure Data Factory or Azure Databricks to perform data validation checks.

For example, you can use Azure Data Factory to create a pipeline that loads the data from the report as data source into ADLS, and then performs data validation checks using activities like Data Flow or Databricks Notebook. These activities can be used to compare the data in ADLS with the source data and check for any discrepancies.

Alternatively, you can also use Azure Databricks to perform data validation checks. You can write a script in Python or Scala that reads the data from ADLS and compares it with the source data.

You can also use Databricks Delta Lake to perform data validation checks, which provides features like schema enforcement and data integrity checks.

To summarize, you can perform data validation checks after the data has been loaded into ADLS using tools like Azure Data Factory or Azure Databricks.

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.

Answer 2

Hi azure_learner ,

Thankyou for posting your query in Microsoft Q&A platform.

I understand that you are trying to understand how the pre-data load validation should be done to ensure the data loaded in ADLS is consistent to source data.

When the source data is a report consumed by API , it can be challenging to perform pre-data load validation to ensure that the data loaded in ADLS is consistent with the source data. However, there are several approaches that you can take to perform some level of validation:

Before loading the data into ADLS, validate the API response to ensure that it contains the expected data. You can use tools like Postman or Fiddler to inspect the API response and verify that it contains the expected data .

Additionally, Validate the data schema of the API response to ensure that it matches the expected schema.

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou