Data factory monitoring activity runs runtime consolidation report

Question

Hello team,

I have a master pipeline which has For Each and inside the loop, it has more than 10 activities. How/where to find the time taken of each activity run in consolidation.

For ex:-

Master
For Each
Iteration 1
Activity 1
Activity 2
Activity 3
Activity 4
Activty 5
Iteration 2
Activity 1
Activity 2
Activity 3
Activity 4
Activity 5

I want to generate consolidation runtime for each activity for all iterations. Currently, I can see the only option to go to activity manually, For Each has more than 500 iterations.

Please recommend if any better approach.

Answer 1

*anonymous user* Thanks for reaching out!! To monitor Azure Data Factory activity runs , I would suggest you to enable diagnostic logs to log the data in log analytics workspace related to ADFActivityRun, ADFPipelineRun and ADFTriggerRun. After enabling the logs, you can query the log analytics workspace to get the time taken for each activity run.

Sample Kusto example to time taken for activity runs if the destination table is selected as Resource-specific

 ADFPipelineRun
 | extend timetaken = End - Start
 | where Status in ('Succeeded','Failed')
 | project TimeGenerated , OperationName , Status , timetaken

For more information on how to monitor azure data factory with Azure Monitor , kindly refer this document. To learn more on how to write kusto queries, kindly refer to this documentation.

Hope this helps!!

Answer 2

once you have enabled the diagnostic settings and data is available in log analytics workspace you can create a workbook. You will find it in Azure Monitor or Log Analytics workspace to build your report - it is interactive.

See this https://cloudblogs.microsoft.com/industry-blog/en-gb/cross-industry/2020/06/18/log-analytics-kql-saved-queries-how-to-find-and-run-them-in-a-workbook/