How to fix nested split issue in Azure data factory Pipeline expression builder?

Zohaib Sheraz Mughal 15

Hi there,

I'm trying to make use of hierarchical namespaces for Azure data lake storage gen2.

When I'm dynamically setting the file structure in the Pipeline expression builder in the ADF, the following error is returned:

Cannot fit unknown into the function parameter string.

The expression is:

@concat(split(item().name,'')[0],'/',split(split(item().name,'')[1],'-')[0],'/',split(split(item().name,'_')[1],'-')[1])

Exp file_name is "customers_2025-01-08.csv"

The expected result/folder structure is:

customers/2025/01

Should there be anything unclear to you, please ask here, I'll willing to clarify as much as possible.

Thanks for your help.

Regards.

Keshavulu Dasari 2,835 Reputation points Microsoft Vendor

2025-01-08T00:24:12.0433333+00:00
Hi Zohaib Sheraz Mughal,

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

I understand the issue might be with the way the split function is being used in your expression and see if we can adjust it to achieve the desired folder structure.

Given the file name customers_2025-01-08.csv, you want to extract the parts to form the structure customers/2025/01.

Please try the below revised expression:

@concat( split(item().name, '_')[0], '/', split(split(item().name, '_')[1], '-')[0], '/', split(split(item().name, '_')[1], '-')[1] )

split(item().name, '_')[0] extracts customers.

split(split(item().name, '_')[1], '-')[0] extracts 2025.

split(split(item().name, '_')[1], '-')[1] extracts 01.

This should give you the desired folder structure: customers/2025/01.

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be happy to help you.
Zohaib Sheraz Mughal 15 Reputation points

2025-01-08T00:51:45.99+00:00

Hi Keshavulu,

Thanks for the instant response.

Unfortunately, it didn't work. There is some issue with split or nested split idk.
Keshavulu Dasari 2,835 Reputation points Microsoft Vendor

2025-01-08T01:01:30.91+00:00
Hi Zohaib Sheraz Mughal,

Given the file name customers_2025-01-08.csv, we can simplify the expression by breaking it down into smaller parts and ensuring each part works correctly.

Extract the base name: Use the first split to get the base name (customers).

Extract the date part: Use another split to get the date part (2025-01-08).

Further split the date part: Split the date part to get the year and month.

@concat( split(item().name, '_')[0], '/', // Extract 'customers' split(split(item().name, '_')[1], '-')[0], '/', // Extract '2025' split(split(item().name, '_')[1], '-')[1] // Extract '01' )

If you are still facing issues, please provide some screenshots and details in the "comments" and I would be happy to help you.
Keshavulu Dasari 2,835 Reputation points Microsoft Vendor

2025-01-08T01:38:42.06+00:00
@ Zohaib Sheraz Mughal,

Extract the file name without the extension:

@split(item().name, '.')[0]

Extract the parts before and after the underscore:

@split(split(item().name, '.')[0], '_')

Extract the date part and split it further:

@split(split(split(item().name, '.')[0], '_')[1], '-')

This expression should:

Remove the file extension.

Split the remaining string by the underscore to separate customers and 2025-01-08.

Split the date part by the hyphen to get 2025 and 01.

Try this revised expression in your Pipeline expression builder. If it still doesn't work, please share the exact error message or any additional details in the "comments" and I would be happy to help you.
Zohaib Sheraz Mughal 15 Reputation points

2025-01-08T19:32:36.0433333+00:00

Hi @Keshavulu Dasari

Our expression is exactly the same, there is some other error or maybe with ADF expression builder. Please have a look at the screenshots attached.

Kind Regards.
Keshavulu Dasari 2,835 Reputation points Microsoft Vendor

2025-01-09T18:38:30.8733333+00:00
Hi Zohaib Sheraz Mughal,

I understand looks like the error is due to the nested split functions within the concat function.

I suggest few step-by-step method using variables in your pipeline:

Create a variable to store the file name without the extension:

@split(item().name, '.')[0]

Create another variable to store the part after the underscore:

@split(variables('fileNameWithoutExtension'), '_')[1]

Extract the year from the second variable:

@split(variables('partAfterUnderscore'), '-')[0]

Extract the month from the second variable:

@split(variables('partAfterUnderscore'), '-')[1]

Concatenate the final folder structure:

@concat(split(item().name, '_')[0], '/', variables('year'), '/', variables('month'))

This method uses transformations to break a complex expression into manageable It should help you avoid the nested partitioning problems you face.

If you still encounter issues, you might want to consider breaking down the expression into multiple steps using variables in your pipeline. This can help manage the complexity and make debugging easier.

Hope this information helps. Please do let us know if you have any further queries.
Zohaib Sheraz Mughal 15 Reputation points

2025-01-10T04:49:55.0266667+00:00

Hi @Keshavulu Dasari

Unfortunately, the variables didn't work as well, when I set the expression in a variable, for e,g.
file_name_year = split(split(item().name,'_')[1],'-')[0]

file_name_cust = split(item().name,'_')[0]

It copies the exact value i.e. "split(split(item().name,'')[1],'-')[0]" or "split(item().name,'')[0]" and makes the folder of the same value, weird.

But I managed to do it using substring, it worked as intended.

Thank you very much for your suggestions and help.

Best Regards.

1 answer

Keshavulu Dasari 2,835 Reputation points Microsoft Vendor

2025-01-10T19:04:22.8833333+00:00

Hi Zohaib Sheraz Mughal,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Issue: Trying to fix nested split issue in Azure data factory Pipeline expression builder

Solution: found a solution using substring, it resolved the problem.

Please let us know if you have any more questions and we will be glad to assist you further. Thank you!
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to fix nested split issue in Azure data factory Pipeline expression builder?

1 answer

Your answer