I run a Pipeline runs at the adf and keep reporting an error Job in progress

Question

I see that the error type prompted is user configuration. The problem occurred in the later link after it was run once before, but it will always be stuck in the second link after it is run again. How can I solve this problem?

Answer 1

Hi @lhwill

Thank you for posting your query!

The "Job in progress" error usually indicates that a previous pipeline run is still active, a resource is locked, or there's a misconfiguration preventing execution. Since your first run completes successfully but the second run gets stuck.

Here are some steps to troubleshoot and resolve the issue:

Check for Active or Stuck Pipeline Runs - Go to ADF Monitor → Pipeline Runs and check if any previous runs are still hanging or queued. Manually cancel stuck runs and retry execution. Ensure there are no overlapping triggers or dependencies causing conflicts.

Verify Linked Service Configurations - Ensure that linked services (Azure SQL, Blob Storage, etc.) have valid credentials (keys, service principals, or managed identities). Validate connection strings (e.g., correct URLs, database names, and endpoints). Use the Test Connection feature in ADF to confirm connectivity.

Check Permissions and Access Control - Ensure your Managed Identity or Service Principal has the necessary permissions, Storage Accounts → "Storage Blob Data Contributor" Azure SQL → "SQL DB Contributor" Azure Key Vault → "Get/List Secrets" permissions. If using Azure SQL, enable "Allow Azure services to access this server" (unless private endpoints are used).

Diagnose Integration Runtime (IR) Issues -

• For Self-hosted IR - Ensure the IR node is running and has network access to source/sink systems. Check firewall rules and NSGs for any blocked ports.
• For Azure IR - Verify that the data source and IR are in the same region. Restart the IR if it appears unresponsive.

Inspect Data Source/Sink Configurations -

• Azure SQL Database - Ensure "Allow Azure services to access this server" is enabled (if not using private endpoints). Check for T-SQL command timeouts.
• Blob Storage - Validate SAS tokens or access keys to ensure they have not expired. Confirm that containers/files are not locked by leases or other processes.

Monitor and Debug - Check Activity Run Details in ADF Monitor for error messages and logs. Enable Azure Monitor / Log Analytics to capture execution traces. Use the Correlation ID from failed activities to investigate further in Azure Support logs.

Common Fixes for User Errors -

• Token expiration - Re-authenticate OAuth-based connections (e.g., SharePoint, Dynamics).
• Parameter mismatches - Ensure pipeline/activity parameters align with dataset schemas.
• Quota limits - Check Azure subscription quotas for pipeline concurrency or data movement units.

By following these steps, you can identify and resolve the root cause of the "Job in progress" error in your ADF pipeline.

Please refer to the below thread links discussing a similar issue, as it may provide you with some useful insights.

• https://learn.microsoft.com/en-us/answers/questions/148791/adf-2011-pipeline-error?orderBy=Newest
• https://stackoverflow.com/questions/74379991/adf-error-code-2011-required-property-connectionstring-is-not-provided-in-con

I hope this information helps. Please do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.