Unable to write rows into sink from SAP CDC Connector

Question

I am working on extracting data from SAP BW and writing it to Azure Blob Storage using an ETL pipeline. Initially, I experimented with delta/incremental loads, but now I need to perform a full load. However, after switching back, I am unable to get any data written to the sink. Im using the SAP CDC connector. I have a solution running on another platform, where things are set up the exact same way, so my intitial guess is that there is a checkpoint hidden somewhere on the server etc. that needs to be removed.

What I Have Tried

• I have deleted all queues in SAP BW to ensure there are no remnants of delta configurations.
• I can see all rows in the staging area, which confirms that the data is successfully extracted from SAP BW. However, when writing to the Azure Blob Storage sink, the process completes but reports that 0 rows were copied.

Source:

Sink:

The issue seems to be on the sink side since the data is available in staging but doesn’t make it to Azure Blob Storage. There are no error messages, just an indication that no rows were written. Delta load configurations were previously in place, but I have removed them and expect a full load to work.

Answer 1

Try the following options

1. Write data to a different storage location, to isolate the issue.
2. Manually Reset the SAP CDC Connector State i.e If the SAP CDC Connector maintains internal tracking, try removing old configurations and reconfiguring it for a full load.

Thanks

Vijay Kuntla

Answer 2

Your SAP CDC Connector might still be maintaining checkpoint metadata, preventing a full load from being written to the sink.

Try the following options

1. Write data to a different storage location, to isolate the issue.
2. Manually Reset the SAP CDC Connector State i.e. If the SAP CDC Connector maintains internal tracking, try removing old configurations and reconfiguring it for a full load.