I have to load gatewaylog files from VM to Datalake through ADF. Some files will get locked while loading. How to load or skip them in Azure data factory? How can I load skipping files after lock get removed? Is there any option to check file is locked or not in ADF?

HI @PrashanthT-2849 , Sorry for the delayed response. Yes, you can use the getMetadata activity to get the lastModified dates for each file in a folder/container. You can then use a Filter activity to filter files based on the lastModified date. Alternatively you can use an If condition activity to do the filtering for you. Hope this helps.

How to load or skip locked files loading through datafactory?

Prashanth T 31 Microsoft Employee

I have to load gatewaylog files from VM to Datalake through ADF. Some files will get locked while loading.

How to load or skip them in Azure data factory?
How can I load skipping files after lock get removed?
Is there any option to check file is locked or not in ADF?

ChiragMishra-MSFT 956 Reputation points

2020-06-18T07:43:08.657+00:00

Hi @Prashanth T ,

Can you please give us some more details so that we have full context ? What gateway logs do you want to load? What's the source VM (is it an Azure VM)? What is the lock that's being placed on these files?
Prashanth T 31 Reputation points Microsoft Employee

2020-06-18T08:44:14.853+00:00

when power bi connects to different source systems it generates log files in server.

lets consider like this I have to skip the last modified file in that list.

How to get the last modified "file name" ?...I have used 'Getmetadata' Activity for 'last modified' but it's not giving the expected one.

1 answer

ChiragMishra-MSFT 956 Reputation points

2020-06-25T07:31:25.807+00:00

HI @PrashanthT-2849,

Sorry for the delayed response. Yes, you can use the getMetadata activity to get the lastModified dates for each file in a folder/container. You can then use a Filter activity to filter files based on the lastModified date. Alternatively you can use an If condition activity to do the filtering for you.

Hope this helps.
Please sign in to rate this answer.
ChiragMishra-MSFT 956 Reputation points

2020-07-14T06:25:48.977+00:00

Hi @PrashanthT-2849 ,

Just wanted to check - was the above suggestion helpful to you? If yes, please accept it as answer as it would help other community members going through this thread.

ChiragMishra-MSFT 956 Reputation points

2020-07-17T07:54:26.093+00:00

Hi @Prashanth T ,

We haven't heard from you in quite some time - was the above suggestion helpful to you? If yes, please accept it as answer as it would help other community members going through this thread.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to load or skip locked files loading through datafactory?

1 answer

Your answer