Increased expenses 10 times

Question

My expenses increased 10 times from 3e per day to 35e per day. How to check which pipeline cost how much? The invoice only gives the total price. The pipeline itself uses Azure IR and uses a script, but it runs for 2 minutes.

The logic is as follows in ADF:

API request > load xml body to Azure DB > from Azure DB parsing xml body with SQL script and load data to Azure DB. Using Azure IR. 10 times per day 30s - 2min. Can SQL scripting in Azure db cost that much?

Answer 1

Here are some steps to analyze the cost:

1. Break Down ADF Costs: ADF costs can arise from:
   • Data Integration Runtime (IR): Azure Integration Runtime (IR) can cost based on the compute power used and the duration of execution.
   • Data Movement and Transformation: Costs for moving data between services (like API, Azure DB) and executing transformations or scripts.
   • Data Storage: Storing data in Azure DB and reading/writing operations can add to the cost.
   • API Calls: Depending on the number and type of API requests, this could also influence costs.
2. Use Azure Cost Management:
   • Navigate to Cost Management + Billing in the Azure Portal.
   • Set up cost analysis filters for Azure Data Factory and other services involved (Azure DB, API calls...).
   • Drill down to see a breakdown of costs by resource, resource group, and service. This will help identify which pipeline, data movement, or operation is consuming more resources.
3. Monitoring ADF Pipeline Performance:
   • Check Pipeline Run History and Activity Runs in ADF Monitoring to view resource consumption like Data Integration Units (DIUs) and execution time. Longer running activities or unexpected resource usage can contribute to higher costs.
4. SQL Script Performance:
   • Running SQL scripts in Azure DB should typically not result in high costs, especially for short execution times (30s–2min). However, consider:
     • Query optimization: If the SQL scripts are not optimized or are processing large amounts of data inefficiently, it can increase costs.
     • Azure SQL pricing tier: Higher pricing tiers of Azure SQL may increase costs depending on the resources allocated (DTUs or vCores).
5. Look into API Usage Costs:
   • If the API you're calling has costs associated with it (based on the number of requests or data size), check if this part of your pipeline is causing the expense increase.