How to fetch the child items within a folder located on azure blob
I am trying to fetch child items of a folder for recursive deletion, so my expectation is to delete the folders within the folder, like if folder A has subfolder B with file b and another subfolder AB has subfolder D, then deleting all items from directory A with file a.txt and subfolder B and its contents b.txt and subfolder AB and its contents, on selection of deletion activity iteratively,
but after I filter the folder items that need to be deleted, which gives me the output
{
"ItemsCount": 1,
"FilteredItemsCount": 1,
"Value": [
{
"name": "Feb25_Test_del - Copy",
"type": "Folder"
}
]
}
I have provided a for each loop activity with items to loop on as
@activity('FilterFolders').output.value
within the foreach I gave a get metadata activity to fetch the child items of the folder
where the dataset correctly points to the folder on the container, which is parameterized to take the folder name as input
p / bp/ @dataset.folderName
and in the Dataset properties the foldername is provided as
@concat('/',item().name)
and field list as Child items which gave error
Field 'childItems' failed with error: 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot navigate through the provided data source. Make sure the provided data source is a directory that can do navigation.,Source=Microsoft.DataTransfer.ClientLibrary,'.
set variable activity, in for each loop, before getMetadata, which succeeded with output as
"name": "FolderName", "value": "/Feb25_Test_del - Copy" for pipeline expression builder
@concat('/',item().name)
Azure Data Factory
-
J N S S Kasyap • 860 Reputation points • Microsoft External Staff
2025-02-27T01:38:34.2833333+00:00 Thanks for posting your query!
Here are the potential issues:
The dynamic folder path being set in the
Set Variable
activity might not resolve correctly. The user is using the expression@concat('/', item().name)
to build the folder path. This might work for some cases, but it's possible that the path being passed to theGet Metadata activity
is incorrect or doesn't point to a valid folder in the Azure Blob Storage.Modify the dataset parameter folderName to @item().name instead of @concat('/', item().name) to ensure the correct folder structure for Get Metadata.
Confirm that the dataset is pointing to a directory in Azure Blob Storage that supports retrieving
ChildItems
. Also, check that thefield list
in theGet Metadata activity
is set correctly to includeChildItems
Sometimes, the error may be due to
insufficient permissions
to access the folder or its contents in Azure Blob Storage. This can cause the Get Metadata activity to fail.I hope this information helps. Please do let us know if you have any further queries.
Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Thank you.
-
Ajay Chowdary Kandula • 0 Reputation points
2025-02-27T12:22:15.2333333+00:00 Thanks for responding quickly. Really appreciate
your suggestion was the first thing I tried and then this posted expression was tried.
Both the times we had the same issues resurface. Upon further research, we understood, there is also a issue with fetch of items for deletion as we fetched only the root folder newly added items like a new folder and new files in the root folder but not the files or folders under the new folder added under the root directory.
My questions are as follows :
- The orginal questions I always had on how to retrieve the child items of a newly added folder at the root level
- How to obtain the child items of a sub-folder within the root folder indicated at the dataset level
- Why would ADF recursively not obtain the files within multiple subfolder structures to perform Delete
Would think it is a pretty basic requirement ? Let me know if you would need more information and Thanks yet again @J N S S Kasyap
-
J N S S Kasyap • 860 Reputation points • Microsoft External Staff
2025-02-28T10:33:57.3833333+00:00 @Ajay Chowdary Kandula
I would like to suggest working on the solution using the approach outlined in the thread below. This method effectively retrieves and deletes child items in a structured manner.
https://learn.microsoft.com/en-gb/answers/questions/1688082/recursive-delete-activity-in-azure-data-factory
However, if the issue persists or you encounter any roadblocks, I will be happy to provide an alternative solution. Let me know how it goes, and I am happy to assist further. -
Ajay Chowdary Kandula • 0 Reputation points
2025-02-28T20:15:16.1166667+00:00 @J N S S Kasyap Thanks a lot for the Information
I have been looking at a similar iterative approach indicated by Richard Swinbank
https://richardswinbank.net/adf/get_metadata_recursively_in_azure_data_factory#fnt__1and its corresponding Json for quick testing
https://github.com/richardswinbank/community/blob/main/adf-get-metadata/adf/GetMetadata.json
if the issue persists or in case of roadblocks, would let you know.
Thanks a lot for your suggestions @J N S S Kasyap
-
J N S S Kasyap • 860 Reputation points • Microsoft External Staff
2025-03-03T06:32:44.27+00:00 @Ajay Chowdary Kandula
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help. -
Ajay Chowdary Kandula • 0 Reputation points
2025-03-03T15:54:05.85+00:00 will post the solution once the resolution has been achieved, it will take some time for sure, to share the same with the community. I am positive it would be of help for others and I would post the json pipeline for quick turnaround. @J N S S Kasyap
-
Ajay Chowdary Kandula • 0 Reputation points
2025-03-03T18:49:27.2033333+00:00 Until Activity (Iterative Folder Processing):
- Add an Until Activity to process each folder in
foldersToProcess
recursively. - Condition:
@equals(length(variables('foldersToProcess')), 0)
- Inside Until:
- Add a Get Metadata Activity to get child items of the first folder in
foldersToProcess
. - Use the same ForEach and If Condition logic to handle the new child items.
- Add a Get Metadata Activity to get child items of the first folder in
- Remove the processed folder from
foldersToProcess
using a Set Variable Activity.
The until activity cannot have inner ForEach and so the instructions though theorotically work but practically they cannot for the instructions suggested at
https://learn.microsoft.com/en-gb/answers/questions/1688082/recursive-delete-activity-in-azure-data-factory@J N S S Kasyap the issue persists or encountered roadblocks, provide an alternative solution, please.
- Add an Until Activity to process each folder in
-
Ajay Chowdary Kandula • 0 Reputation points
2025-03-03T18:52:57.4066667+00:00 the issue persists or have encountered roadblocks, Please suggest/ provide an alternative solution. @J N S S Kasyap
https://learn.microsoft.com/en-gb/answers/questions/1688082/recursive-delete-activity-in-azure-data-factory
The solution suggestion has some drawbacks, such as no inner for each possible withing until interation at Until Activity (Iterative Folder Processing):- Add an Until Activity to process each folder in
foldersToProcess
recursively. - Condition:
@equals(length(variables('foldersToProcess')), 0)
- Inside Until:
- Add a Get Metadata Activity to get child items of the first folder in
foldersToProcess
. - Use the same ForEach and If Condition logic to handle the new child items.
- Remove the processed folder from
foldersToProcess
using a Set Variable Activity.
- Remove the processed folder from
- Add an Until Activity to process each folder in
-
J N S S Kasyap • 860 Reputation points • Microsoft External Staff
2025-03-04T23:52:55.58+00:00 @Ajay Chowdary Kandula
I am sharing useful videos to implement the task
https://www.youtube.com/watch?v=MZaH5FlCX3E
https://www.youtube.com/watch?v=yZYPaE05vI8Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.
However, if the issue persists or you encounter any roadblocks, I will be happy to reproduce the scenario from my end will provide details. Let me know how it goes, and I am happy to assist further. -
Ajay Chowdary Kandula • 0 Reputation points
2025-03-06T14:21:59.13+00:00 @J N S S Kasyap I have taken a look at both the videos and none match our scenarios where we are trying for a pure ADF based solution
one video uses sql table which would not work and the other video has look up activity which again is not suitable in our scenarios, Requesting to kindly suggest any other alternatives and also thank you as some of the ideas and pipeline expressions seem quite brilliant to evaluate certain conditions
-
J N S S Kasyap • 860 Reputation points • Microsoft External Staff
2025-03-06T23:45:20.0366667+00:00 @Ajay Chowdary Kandula
Can please confirm once these details, I will repro from end give you the details ADF Pipeline workflow
Here’s a structured breakdown of yourAdls Gen2
file hierarchy based on your description: Container: test├── A/
│ ├── a.txt
│ ├── B/
│ │ ├── b.txt
│ ├── AB/
│ │ ├── D/
│ │ │ ├── d.txt
Source and Sink both are Adls Gen2.You want to delete
everything inside A recursively
, including:- Files (a.txt, b.txt, d.txt)
- Subfolders (B, AB, D)
Here are the
Azure Data Factory (ADF) activities
you used in your pipeline:-
Filter Activity
– Filters folder items based on the deletion criteria. -
ForEach Activity
– Iterates over filtered folders for processing. -
Set Variable Activity
– Stores the folder path dynamically. -
Get Metadata Activity
– Fetches child items (subfolders and files). -
Delete Activity
(Not explicitly mentioned but likely needed) – Deletes files and folders iteratively.
-
Ajay Chowdary Kandula • 0 Reputation points
2025-03-07T14:51:01.4233333+00:00 I was able to perform by traversing through multiple and multiple subfolders at the cost of performance based on the above links shared.
https://richardswinbank.net/adf/get_metadata_recursively_in_azure_data_factory#fnt__1
https://github.com/richardswinbank/community/blob/main/adf-get-metadata/adf/GetMetadata.jsonbut since it is also time consuming, so based on the discussion above, I want to delete files/blobs on the azure container, where we do an incremental copy daily.
source > Network Folder
Sink > Azure Blob StorageSource and Sink are not Adls Gen2 and please help with Json file, if possible, at your earliest convenience sir.
Yes, I have used the ADF activities mentioned in your message.
-
J N S S Kasyap • 860 Reputation points • Microsoft External Staff
2025-03-08T00:21:25.14+00:00 @Ajay Chowdary Kandula
Optimized JSON-Based ADF Pipeline for Incremental Deletion in Azure Blob StorageThis pipeline will:
-
Identify Incremental Files
to delete (based on a date filter). -
Iterate Over Matching Files
using a ForEach loop. -
Delete Only the Incrementally Copied Files
instead of deleting everything recursively.
{ "name": "IncrementalBlobDeletionPipeline", "properties": { "activities": [ { "name": "LookupIncrementalFiles", "type": "Lookup", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30 }, "typeProperties": { "source": { "type": "AzureBlobStorage", "dataset": { "referenceName": "AzureBlobDataset", "type": "DatasetReference" }, "query": "SELECT name FROM blobs WHERE LastModified >= DATEADD(day, -1, GETDATE())" } } }, { "name": "ForEachBlob", "type": "ForEach", "dependsOn": [ { "activity": "LookupIncrementalFiles", "dependencyConditions": ["Succeeded"] } ], "typeProperties": { "items": "@activity('LookupIncrementalFiles').output.value", "activities": [ { "name": "DeleteBlob", "type": "Delete", "dependsOn": [], "typeProperties": { "dataset": { "referenceName": "AzureBlobDataset", "type": "DatasetReference" }, "recursive": false }, "userProperties": [] } ] } } ] } }
How This Works Efficiently:
1.Instead of fetching all files and processing them recursively, it only fetches incremental files (last 1 day, can be adjusted).
2.Uses a query-based filter (LastModified >= DATEADD(day, -1, GETDATE())) to target only recently copied blobs.
3.The ForEach loop processes only the necessary files, avoiding unnecessary operations.
Hope this info helpful
-
Sign in to comment