There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'AutoResolveIntegrationRuntime'

Question

Hi,

I'm having the error above:

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'AutoResolveIntegrationRuntime'

I have checked other questions asking the same, and they all suggest to run my pipelines at different trigger times. The issue is that I'm online running one pipeline, and I'm even using a debug session, not a trigger.

Is there anything wrong on the service?

Regards,
Francisco.

Answer 1

Hello @Francisco Dominguez ,

Welcome to the MS Q&A platform.

Please correct me if my understanding is wrong. You are running a debug session on a pipeline with dataflow and getting a throttling error.

You will generally see the error when the parallel data flows go beyond the max value.

Could you please let me know if this is a new pipeline? Did it succeed at any time(intermittent or failing every time)? When did you notice the issue for the first time?

If you are seeing this error message consistently,

Can you please create a new integration runtime, and split your pipelines across multiple integration runtimes, and see if it resolves the issue

Here are the details regarding this error message from the documentation page.

Error message:

Type=Microsoft.DataTransfer.Execution.Core.ExecutionException,Message=There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'AutoResolveIntegrationRuntime'.

Cause

You've reached the integration runtime's capacity limit. You might be running a large amount of data flow by using the same integration runtime at the same time. See Azure subscription and service limits, quotas, and constraints for details.

Resolution

Run your pipelines at different trigger times.
Create a new integration runtime, and split your pipelines across multiple integration runtimes.

I hope this helps.

https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide#a-pipeline-run-fails-when-you-reach-the-capacity-limit-of-the-integration-runtime

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

I am having same issues. I tried all workaround above. Nothing fixed the issue.

Answer 3

Hi,

Recently I faced an issue In the dataflow

Internal Server Error in Synapse batch operation: WorkspaceType:<ADF> CCID:<>] MaxClusterCreationAttempts=[3] Attempt=[0] ComputeNodeSize=[Small] ClusterId=AdlaResourceId=[] [Creation] -> [Cleanup]. The cluster creation has failed more than the [3] of times. IsTimeout=[True] IsTerminal=[False] IsRetryable=[True] ErrorType=[None] ErrorMessage=[]'.

After a few days I got this error now:

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under the Integration Runtime.

It comes randomly, I change the dataflow runtime but it didn't work.