ADF - FullandIncremental load with selections, sometimes working but not all the time

Naveen Nunna 0

Hello ADF Forum,

We are using ADF to get the data from SAP ECC extractors (CDC) to snowflake, we have built the data flow with conditions and without conditions.

ADF run mode -> "Full and Incremental", when I choose with selections sometimes it's extracts data based on my selection parameters but sometimes it's just getting the delta records when I ran in debug mode option.

Does anyone saw this kind of behavior, do we need to refresh any caching to update the selection parameters in ADF Flow?? weird part is day1 didn't work and day 2/3 it works for same selections.

Thanks

Ganesh Gurram 2,550 Reputation points Microsoft Vendor

2024-12-31T02:56:34.7666667+00:00

Hi @Naveen Nunna

Thanks for the question and using MS Q&A platform.

It sounds like you're experiencing inconsistent behavior with your Azure Data Factory (ADF) data flow when extracting data from SAP ECC using the Change Data Capture (CDC) method.

Here are a few things you might want to check or try:

Cache Refresh - Sometimes, ADF might be using cached data, which could cause inconsistencies. Try clearing any cache or ensuring that the cache is refreshed before each run.

Debug Mode - Running in debug mode can sometimes behave differently compared to a scheduled pipeline run. Ensure that your debug settings are consistent with your production settings.

Parameter Handling -Double-check how your parameters are being passed and used within the data flow. There might be an issue with how the parameters are being interpreted or applied.

Data Source Consistency - Ensure that the data source (SAP ECC extractors) is consistent and that there are no intermittent issues with data availability or connectivity.

Logs and Monitoring - Check the logs and monitoring data in ADF to see if there are any errors or warnings that could give you more insight into why the behavior is inconsistent.

Incremental Load Setup - Review the setup for your incremental load to ensure that it is correctly configured. Sometimes, issues can arise from how the delta records are identified and processed.

Reference:

Incrementally load data from a source data store to a destination data store

Incremental Data Loading using Azure Data Factory

How to Do Incremental Load in Azure Data Factory: Delta Load vs. Incremental Load

Hope this helps. Do let us know if you have any further queries.
Ganesh Gurram 2,550 Reputation points Microsoft Vendor

2025-01-02T07:28:50.66+00:00

@Naveen Nunna - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Naveen Nunna 0 Reputation points

2025-01-02T22:35:53.63+00:00

Hi Ganesh,

Cache - how do you want me to refresh in Production,??could you share the steps or process to clearing cache.

Debug Mode - Compared both and it has same values.

Parameter Handling - It worked sometime so nothing was changed.

Data Source Consistency -Same data source used for loading other years of data, no issues with data source.

Logs and Monitoring -No Errors and I see my selections are selected.

Incremental Load Setup - It's configured and running fine

Thanks

Naveen
Ganesh Gurram 2,550 Reputation points Microsoft Vendor

2025-01-03T18:17:12.2066667+00:00

@Naveen Nunna - Thank you for the additional context. Since you've confirmed that the other aspects of your ADF setup are functioning correctly, if you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?

Share via

ADF - FullandIncremental load with selections, sometimes working but not all the time

Your answer