Verify the existence of a folder in the metadata ChildItem list; if it does not exist, create the folder.

Question

I have a pipeline that searches for a folder formatted as "yyyymm". I need assistance in verifying whether the folder name is present in the Get Metadata ChildItem name list. If the folder name does not exist, I need guidance on creating it in the data lake.

Any help is appreciated. Thank you!

Answer 1

Hi libpekin,

Thanks for reaching out to Microsoft Q&A.

You are trying to verify if a folder with a specific "yyyymm" format exists in your data lake, and create it if it doesn't. Based on the pipeline image you have shared, I will try to provide some guidance on how to modify your existing pipeline to achieve this:

1. Lookup step: You are already getting the folder name from a database, which is good.
2. Get Metadata: This step is retrieving the list of folders in your data lake.
3. If Condition: You will need to modify this step to check if the folder exists.

To check if the folder exists and create it if not:

1. Modify the "If Condition" step:
   • Use an expression to check if the folder name exists in the metadata child items list.
   • You can use the contains() function in your expression.
2. True path (if folder doesn't exist):
   • Add an activity to create the folder in your data lake.
   • You might use a "Copy Data" activity with no source and the sink as your data lake, specifying the new folder path.
3. False path (if folder exists):
   • You can leave this as is or add any activities you want to perform when the folder already exists. @not(contains(activity('Get Metadata').output.childItems.name, pipeline().parameters.folderName))

The above code checks if the folderName parameter (which should be your "yyyymm" formatted string) is not in the list of child item names from the Get Metadata activity.

For creating the folder, you'd use a Copy Data activity with these settings:

• Source: None (or a dummy source)
• Sink: Your Data Lake storage
• Folder path: Combine your base path with the new folder name

Can you try the above and let me know?

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.