Azure Data Factory Copy Data, Azure table storage to table storage only update if Timestamp of destination row is smaller

Question

Would like some help with the following scenario, I have setup a simple copy from one Azure table to another Azure table. What I want to do is only update the destination row if the timestamp of the source row is greater then the destination row. On the Sink tab i use the Insert Type, can see the Merge/Insert option and one that says dynamic content. Is there a way to achieve this?

Answer 1

Hello @Gavin Armitage

You can acheive this using copy acivity , but i think we will have to make some adjustments . Since the idea to get all the new records from the source which are not in destination . In SQL world this can be something get max(timestamp) from the destination table and then use that on the source query . So in the context of the issue in hand

we are looking for a query like

Timestamp gt datetime'2020-06-04T19:34:07.534Z'

To make it dynamic , I am using a variable activity and contruct the date clause

@concat('Timestamp gt datetime',''' , addMinutes(utcNow(), -10) ,''')   

, this clause will give me the all the records in the last 10 mins . You can run the trigger at every 10 mins and you should have the records in the destination .

I am adding the image for clarity , please so let me know as to how it goes .

Unfortunately tablke storage are not good for aggregation .