Azure Data Factory random disconnections and pipeline experienced a "ghost run" with no trigger debug/history present.

Question

Hello,

I'm experiencing a few issues with Azure Data Factory.

1. Our team has Azure Data Factory integrated with GIT, but at random times it disconnects.
   1. It may show as connected for one team member but appear disconnected for another.
   2. This also has occurred while having two web browsers open on the same PC.
      1. Browser #1: Data Factory connected to GIT
      2. Browser#2: GIT disconnected
2. "Ghost Run" of Data Factory Pipeline; Does not show run/output history.
   1. On Jan 28th, it was noticed a Kusto table was empty. The table's associated pipeline, triggers at the end of each month. However, Data Factory does not show any run history except for last month's trigger (Dec 30, 2024), which successfully ran.
   2. The pipeline includes an If Condition that checks if a statement is True before clearing the table and moving the data via Dataflow. If the statement is False, an Error is returned.
      1. The run on Dec 30th, did not clear nor move data as the condition failed: [ "Container data has not been updated, no transfer for this month."

         *"Container's last updated:  ", "2024-05-22T10:41:17Z",* 
         
         *"Most Recent Table Ingestion: ", "2024-10-30T18:45:18Z"   ]*
         

      2. ***NOTE: ***Kusto table has a retention policy set to 90d.
      3. This also occurred last November.
3. At times, Data Factory either restores old datasets/pipelines/dataflows, or does not reflect recent modifications

Pipeline Last Run:

If Condition:

Answer 1

Hi @Kandice

Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.

Our team has Azure Data Factory integrated with GIT, but at random times it disconnects

It seems like there are random disconnections between Azure Data Factory and Git, especially with multiple team members or browsers involved. This might happen due to session handling or authentication issues. Possible solutions:

• Clear Cache - Try clearing the browser cache or use Incognito mode to see if that resolves the issue.
• Re-authenticate - Reauthenticate Azure Data Factory with Git in both browsers and ensure that all team members are properly logged in and authorized.
• Browser Compatibility - There could be conflicts between browsers. Try using one browser consistently or switch to Edge/Chrome and make sure the version of Azure Data Factory in the browser is up to date.

"Ghost Run" of Data Factory Pipeline; Does not show run/output history.

The pipeline appears to have run, but there is no corresponding history or output, and you're not seeing any trigger runs after December 30, 2024. This could be related to data retention settings or a trigger misconfiguration. Possible solutions:

• Check Trigger History - Check the triggers and ensure they are correctly configured. You may want to verify that your trigger has actually fired and that it’s not facing any issues on the trigger side.
• Retention Policy - The Kusto table has a 90-day retention policy. Since you mentioned that the most recent table ingestion was from October 2024, it’s possible that data was automatically purged before the pipeline executed, affecting the pipeline's operation.
• Logs & Debugging - Look into the Activity Runs in Azure Data Factory for any hidden logs or errors that may have occurred. If the pipeline was triggered by a schedule, ensure that the schedule is still active and has not been inadvertently disabled.

At times, Data Factory either restores old datasets/pipelines/dataflows, or does not reflect recent modifications

• Version Control - Ensure that the Git integration is functioning correctly and that all changes are being committed and pushed to the repository. Check for any merge conflicts that may have caused older versions to be restored.
• Refresh the ADF UI - Sometimes, the ADF UI may not reflect the latest changes immediately. Refreshing the browser or clearing the cache can help.
• Check for Branch Issues - If multiple branches are being used, ensure that the correct branch is checked out in ADF. Changes made in one branch will not reflect in another unless merged.

For more details refer:

• Troubleshoot pipeline orchestration and triggers in Azure Data Factory
• Troubleshoot CI-CD, Azure DevOps, and GitHub issues in Azure Data Factory and Synapse Analytics

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.