Copy Activity very slow for Azure data factory

Mayuri Shah 0 Reputation points
2024-09-13T03:50:38.0833333+00:00

Hi All,

we are having issue where copy activity from Azure Table storage to data factory is taking over 50mins on daily basis. it used to take 15-20mins until a week back.

What's the source of the slow copy activity?

Azure Table storage

What's the sink of the slow copy activity?

Azure data lake storage gen2

Which type of integration runtime are you using?

Azure IR

Azure Table Storage
Azure Table Storage
An Azure service that stores structured NoSQL data in the cloud.
171 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,497 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,918 questions
{count} votes

2 answers

Sort by: Most helpful
  1. AnnuKumari-MSFT 33,556 Reputation points Microsoft Employee
    2024-09-13T07:54:19.2266667+00:00

    Hi Mayuri Shah ,

    Welcome to Microsoft Q&A platform and thanks for posting your query here.

    I understand that you are facing performance issues while trying to copy data from Az table storage to ADLS gen2 using copy activity via Azure IR .

    As you mentioned, comparatively, earlier last week it was taking lesser time. Could you please check if there is significant increase in the volume of data getting copied as compared to last week? Kindly check the copy activity output log and see the datacopied size and compare .

    Also , is there any configuration related changes that were made in the recent times like changing the DIU, or DOP ? Kindly check.

    Additionally, what is the region in which ADF is provisioned, I would check if there is any ongoing outage or any service health impact in that region in recent times.

    Looking forward to your response. Thankyou

    0 comments No comments

  2. Amira Bedhiafi 26,961 Reputation points
    2024-09-13T08:03:00.1533333+00:00

    Verify if there are any performance bottlenecks on the Azure Table Storage side, such as increased data volume or throttling limits being hit. Check if the storage account is under stress or if there have been changes to the access patterns.

    • Assess if there has been an increase in the volume of data being copied recently. A larger dataset will naturally take longer to transfer. Compare the data volumes from previous days to identify any spikes in the size of the source data.

    Ensure that the integration runtime is scaling appropriately. If the IR is overloaded or under-provisioned, it might slow down the data transfer process. Check the performance metrics of the IR, such as CPU usage, memory consumption, and network bandwidth.

    You may need to upscale it to Azure Support if needed.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.