Get file names from different folders and combine them into one variable in ADF

Question

Get file names from different folders and combine them into one variable in ADF

Zhu, Yueli YZ [NC] 255

I have 100 folders: f1-f100, each folders have many different csv files. How to get all the csv files' names together,in a list, like results in one variable? Thanks a lot

source in a storage blob container like below:

 f1: test1.csv, test2.csv

 f2:test2.csv,

 f3: test3.csv,test4.csv, test5.csv

   ...

 f100: test1001:csv,test1002:csv,test1003:csv

expected result below:

{

"name": "file_names",

"value": [

	"test1.csv",

	"test2.csv",

	...

	"test1001.csv",

	"test1002.csv",

	"test1003.csv"

]

}

Chandra Boorla 9,910 Reputation points Microsoft External Staff

2025-03-10T20:46:11.5433333+00:00
@Zhu, Yueli YZ [NC]

To achieve the goal of getting all CSV file names from multiple folders in Azure Data Factory (ADF) and combining them into a single variable, you can use the following approach:

Use a ForEach Activity - Create a ForEach activity in ADF that iterates over the list of folders (f1 to f100). You can retrieve the folder names dynamically if they follow a consistent naming pattern.

Get Metadata Activity - Inside the ForEach loop, use a Get Metadata activity to retrieve the list of files in each folder. You can specify the field to retrieve as "Child Items" to get the names of the files.

Append Variable - Initialize an array variable outside the ForEach loop to store the file names. Inside the loop, use the Append Variable activity to add the names of the CSV files retrieved from each folder to this array.

Final Output - After the ForEach loop completes, the variable will contain all the CSV file names from the specified folders.

This way, you can consolidate all the file names into a single variable as you described.

For more details, please refer the similar threads:

Get all files names in subfolders Azure Data factory

Accessing file name from nested folder structure and passing it to Notebook activity as a parameter in ADF.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 9,910 Reputation points Microsoft External Staff

2025-03-11T23:20:14.8533333+00:00

@Zhu, Yueli YZ [NC]

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Shabarinath Chede 0 Reputation points

2025-03-12T06:22:58.1666667+00:00

Hi Chandra Boorla
I was followed you scenario so iam getting an issue like i use getmetadata activity to fetch the folders so it was working fine and i was connected to the for loop and inside forloop i was taken another getmetadata activity and i given directory path i was given @item.name so it was also working fine i can see the list of files in individual folders (my files are not only csv i have json and txt also.) and i was trying to connect the append variable activity so here iam getting the issue like i was given

@activity('Get Metadata2').output.childItems so iam not able to append the files. I want childtems.name right so i can't keep anothere forloop inside forloop.

--what is the solution for this? can you please tell me the solution for that.

1 answer

Your answer

Chandra Boorla 9,910 Reputation points Microsoft External Staff

2025-03-10T20:46:11.5433333+00:00

@Zhu, Yueli YZ [NC]

To achieve the goal of getting all CSV file names from multiple folders in Azure Data Factory (ADF) and combining them into a single variable, you can use the following approach:

Use a ForEach Activity - Create a ForEach activity in ADF that iterates over the list of folders (f1 to f100). You can retrieve the folder names dynamically if they follow a consistent naming pattern.

Get Metadata Activity - Inside the ForEach loop, use a Get Metadata activity to retrieve the list of files in each folder. You can specify the field to retrieve as "Child Items" to get the names of the files.

Append Variable - Initialize an array variable outside the ForEach loop to store the file names. Inside the loop, use the Append Variable activity to add the names of the CSV files retrieved from each folder to this array.

Final Output - After the ForEach loop completes, the variable will contain all the CSV file names from the specified folders.

This way, you can consolidate all the file names into a single variable as you described.

For more details, please refer the similar threads:

Get all files names in subfolders Azure Data factory

Accessing file name from nested folder structure and passing it to Notebook activity as a parameter in ADF.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 9,910 Reputation points Microsoft External Staff

2025-03-11T23:20:14.8533333+00:00

@Zhu, Yueli YZ [NC]

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Shabarinath Chede 0 Reputation points

2025-03-12T06:22:58.1666667+00:00

Hi Chandra Boorla
I was followed you scenario so iam getting an issue like i use getmetadata activity to fetch the folders so it was working fine and i was connected to the for loop and inside forloop i was taken another getmetadata activity and i given directory path i was given @item.name so it was also working fine i can see the list of files in individual folders (my files are not only csv i have json and txt also.) and i was trying to connect the append variable activity so here iam getting the issue like i was given

@activity('Get Metadata2').output.childItems so iam not able to append the files. I want childtems.name right so i can't keep anothere forloop inside forloop.

--what is the solution for this? can you please tell me the solution for that.

Answer 1

Please refer the below blog for step by step detail to achieve the goal :

https://datasharkx.wordpress.com/2023/02/06/calculate-folder-size-of-an-azure-blob-storage-data-lake-storage-via-synapse-data-factory-pipeline/

you need a combination of for each activity and get meta data activity to get list of all files within all folders recursively.

Note : rather than getting the file size and adding it up, you can use append variable activity and append the file names to an array variable .

Then take a union of the variable with itself at the end after all iterations to remove the duplicate values and get the unique file names

Share via

Get file names from different folders and combine them into one variable in ADF

1 answer

Your answer