execute few tables in parallel and few tables in sequential based on the exec priority

yogeswara Reddy Chegireddy 0 Reputation points
2025-01-26T20:52:11.75+00:00

I have a ADF pipeline where it reads the control table to get the list of source and target tables that needs to copy from source to target. and pass the list to foreach where it executes copy activity. Enhancement needed: I have the exec_pprioroty in the control table, which will define the order that the table can execute the copy from source to target. Lit say Ex: if table A has exec_priority as 1 that means it needs to execute first after it completes only it can go to the next priority, if table B and table C has same ex_priority as 2, that means both B and C tables can exec in parallel after table A has completed successfully.

Thank you!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,165 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Nandan Hegde 33,561 Reputation points MVP
    2025-01-27T03:56:46.1833333+00:00

    Try implementing the below solution :

    1. Have an external for each activity with iterations equivalent to distinct ex_priority values and make it sequential
    2. within the foreach, have a lookup activity to get the number of rows for that ex_priority value
    3. Pass that output to an execute pipeline activity and that execute pipeline activity would contain another for each activity with settings kept parallel and the iterations equivalent to the lookup output.
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.