Monitor AutoResolve Integration Runtime in Azure Data Factory

Question

Hi all

I need to monitor the Integration Runtime performance for a pipeline that uses stored procedure activity to load data from ADLS to Snowflake. The stored procedure in this case resides in snowflake itself and it’s called from Snowflake Tenant to read parquet files and load to Iceberg tables.

just like Mapping Data Flows and Copy Data Activities provide details of Data read, write and throughput respectively I need to see the performance in this case.

I have diagnostics settings enabled which at max captures the Queue time for activities in a pipeline.

Any sort of suggestions/workarounds appreciated

Thanks

Answer 1

To monitor the performance of the AutoResolve IR for your adf pipeline using a stored procedure activity, you can leverage Activity Run Metrics and Pipeline Logs for additional insights. Since diagnostics settings capture the queue time but not detailed performance metrics for stored procedure activities, you can use the following workarounds:

Custom Logging within the Stored Procedure: Add logging mechanisms inside the Snowflake stored procedure to capture execution time, file read/write details, and throughput. Store these logs in a Snowflake table or write them back to an external logging system like Azure Log Analytics or a blob storage.

1. Integration Runtime Metrics: Use the Monitor Hub in adf to view metrics such as the pipeline activity duration, integration runtime CPU/Memory utilization, and concurrency limits. This can help identify bottlenecks in the AutoResolve IR.

Azure Log Analytics Integration: Configure diagnostic settings to send logs to Azure Log Analytics. Although stored procedure activity details might be limited, you can gain insights into pipeline execution trends and potential queue bottlenecks.

Custom Performance Monitoring: Instrument the pipeline to log timestamps at critical stages, such as before calling the stored procedure and after execution. Use these timestamps to calculate execution time.

By combining stored procedure-level logging and adf monitoring features, you can approximate the performance details similar to what is available for mapping data flows and copy data activities.