Delay in Azure data factory pipeline. Why it is taking so long like 15 minutes to complete execution?

Question

Azure data factory pipeline is calling several sub pipelines depending on the use case of the request.

It is observed that when using Script activity it is taking minutes to execute and slowing down the pipeline performance. For other activities it is taking maximum a minute.

We are NOT using Auto resolved Integration run time.

Answer 1

Hello @Chatti, Niharika [E]

One possible reason could be the use of the Script activity, which can be slower than other activities.

To improve the performance of your pipeline, you can try the following:

1. Use the Copy activity instead of the Script activity wherever possible. The Copy activity is optimized for data movement and can be faster than the Script activity.
2. If you must use the Script activity, try to optimize your script code to make it more efficient. For example, you can try to reduce the number of loops or use more efficient data structures.
3. Consider using a more powerful Integration Runtime (IR) if you are not using Auto resolved Integration Runtime. A more powerful IR can handle larger workloads and may improve the performance of your pipeline.
4. Check if there are any throttling errors on the sink or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
5. Gradually tune the parallel copies. Note that too many parallel copies may even hurt the performance.

If none of these solutions work, you may need to investigate further to identify the root cause of the delay. You can check the pipeline run details to see if there are any errors or warnings that could be causing the delay. You can also monitor the performance of your data store to see if there are any issues that could be impacting the pipeline performance.

I hope this helps.

Answer 2

Hi Chatti, Niharika [E] ,

Welcome to Microsoft Q&A platform and thanks for posting your query here .

As per my understanding you are trying to execute ADF pipeline having Script activity which is taking so long to run while other activities are working as expected.

I tried to execute a SELECT statement via

1. using Script activity and
2. using Look up activity

Lookup activity took 15 secs to complete while Script activity took 1mins 30 secs to complete the execution considering the Datasize of the table is very less and equal in both the cases, pointing to same table and using same query.

It is expected by design, however, in case you want to avoid it , you can use either lookup activity or else create a stored procedure with same query in DB and use Stored procedure activity , instead of Script activity. It should execute relatively very fast.

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou