What is the maximum throughput speed achievable on the Azure Data factory

Rahul 251

Hi All,

I am performing the data migration from oracle (on prem) to Azure sql database using the Self hosted integration runtime.

The throughput speed -2 MB/s on the Azure data factory.

135 GB takes approx. -10 hr.

What is the maximum achievable throughput speed on the Azure Data factory ?

How can increase the throughput speed on the azure data factory ?

JimmySalian-2011 42,241 Reputation points

2023-07-04T14:35:41.5766667+00:00

Hi Kashish,

Data Factory native integration runtime mode also allows parallelism at different levels. You can use parallelism to fully utilize your network bandwidth, storage IOPS, and bandwidth to maximize data movement throughput.

Check this detailed throughput and network speed for the Azure Data Factory - https://learn.microsoft.com/en-gb/azure/data-factory/copy-activity-performance

Also you can try out the optimisation and troubleshoot the issues via the link I have provided.

Hope this helps.

JS

==

Please Accept the answer if the information helped you. This will help us and others in the community as well.
Pratik Somaiya 4,206 Reputation points

2023-07-04T17:08:19.54+00:00

If you want to increase the speed of data copy from source then you need to check the current statistics of SHIR and when its peaking out. If your firm is using an Express Route for their network then its bandwidth will also play a role in deciding the speed of data transfer
Bhargava-MSFT 31,206 Reputation points Microsoft Employee

2023-07-10T20:11:36.1766667+00:00

Hi Kashish,

We are checking to see if you have any further questions here.
Rahul 251 Reputation points

2023-07-12T18:16:39.8166667+00:00

@Pratikyes, we are using the express route.

Connection-1

Oracle (on-prem)-->ADFSHIR-->F5-->Express Route Gateway-->Azurefirewall-->Private End Point-->Sql server database.

Connection-2

Oracle (on-prem)-->ADFSHIR-->F5-->Express Route Gateway-->Azurefirewall-->Private End Point-->data Factory instance.

90 % of the time is consumed while reading the data from oracle source while migrated using the azure data factory instance

I am getting throughput speed- 2 -5 Mbps due to which the 1.5 TB data migration is taking 100 Hr , for sequential each table migration .

how to improve the speed?

What's the maximum throughput speed achievable and we should target as we would be doing data migration in production database.

Accepted answer

Bhargava-MSFT 31,206 Reputation points Microsoft Employee

2023-07-14T17:03:32.4+00:00

Kashish

Did you check the network settings to see the line speed on the Oracle on-prem server?

As you mentioned, "90 % of the time is consumed while reading the data from Oracle source while migrated using the Azure data factory instance."

This indicates the bottleneck is on reading the data from the source. So the low throughput is due to SHIR retrieving data from Oracle.

Please check the Oracle status during the copy, any throttling happens on the Oracle side?

If SHIR is not in the same machine as the Oracle instance, check the network speed between these two.

You can test the upload speed between Oracle and SHIR machine from the below link

http://www.azurespeed.com/Azure/UploadLargeFile
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Rahul 251 Reputation points

2023-07-21T18:48:49.5733333+00:00

@Bhargava-MSFT

Thanks for response.During the Migration I am observing below throughput.

90% of the time taken is through reading the source data.

Following is the connection from oracle to ADF and Azure Sql database, we are using in our case.

Connection-1

Oracle (on-prem)-->ADFSHIR-->F5-->Express Route Gateway-->Azurefirewall-->Private End Point-->Azure Sql server database.

Connection-2

Oracle (on-prem)-->ADFSHIR-->F5-->Express Route Gateway-->Azurefirewall-->Private End Point-->data Factory instance.

Question-Please inform the 90% of the time it takes from the source as shown in the below ADF .

Reading from source -2:04:01 represented in ADF copy activity.

Is it the time taken between the Oracle and ADFSHIR virtual machine or Is it from taken taken Oracle (on-prem)-->ADFSHIR-->F5-->Express Route Gateway-->Azurefirewall-->Private End Point-->data Factory instance. ?
Please sign in to rate this answer.
Bhargava-MSFT 31,206 Reputation points Microsoft Employee

2023-07-24T20:19:53.31+00:00

Based on the details, it is hard to determine the exact location where 90% of the time is being taken. However, as I mentioned earlier, the time is likely between the Oracle database and the SHIR machine.

When copying data from an on-premises Oracle database to Azure SQL, the data must be read from the Oracle database and transferred over the network to the SHIR VM. The time taken for this transfer will depend on factors such as the size of the data, the network bandwidth, and the latency between the Oracle database and the SHIR machine.

Please check the below:

Check the network bandwidth/latency between the Oracle database and the SHIR VM

Check the performance of the SHIR virtual machine. If the SHIR VM is under-provisioned, it can impact the performance of the data copy operation. You can consider increasing the size of the virtual machine or using a more powerful virtual machine.

Monitor the Oracle database for any throttling or performance issues during the data copy operation.

You can test the upload speed between Oracle and SHIR machine from the below link

http://www.azurespeed.com/Azure/UploadLargeFile

Lu Zhang 0 Reputation points

2025-02-19T18:46:51.65+00:00

We had same issue. other tool run on the same machine with same data source can go over 2MB/S easily. This make me wonder, does SHIR limit itself to max 2 MB/s ?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

What is the maximum throughput speed achievable on the Azure Data factory

1 additional answer

Your answer