Tracing Stored Procedure Event in Azure Synapse pool & Handling Duplicates (No Composite Primary Key)

Question

Hi Team,

I am loading CSV file data from Azure Blob to Azure DW in external table.
Through ADF pipeline I am calling Stored Procedure which loads data from ext. to staging table.
I have created a Trace table where I wanted log info such as Date, SP name, No of records inserted each time when stored procedure is called
so that I can track easily daily process.

Other scenario is that, as we don't have Composite Primary Key concept in Azure DW,
How to deal with duplicates while loading data from Azure Blob to Azure external table? How to deal with any bad data getting loaded from Blob as we cant go with some constraints?

Your help is appreciated in this regard

Answer 1

Hello @SwapnilKalkar-4467 ,
Thanks for your patiance here .
On checking duplicate part of the question , i think you are refering to the use case where you do not want to have duplicate records on the table , if that the ase you can use WHERE NOT EXISTS ().

Since in your case you are using the SP , i think you can put the dedup logic in the SP itself .

Let me know if you have any further questions .

Thanks & stay safe

Himanshu

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members