This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 31%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPD%3C/text%3E%3C/svg%3E)
Hi Team,
I have one use case where I need to copy 500GB single file from sFTP location to AWS S3 bucket using Azure iPaaS services.
I know ADF doesn't support S3 bucket as sink as of now but I'm planning to put that large file in AWS sFTP which can drop it in S3 bucket. I'm planning to use ADF copy activity. For that I need to know if ADF can handle 500 GB.
Please help,.
I'm open to other Azure Service which is fit for this purpose.
Regards,
Partha
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
In addition to @Nandan Hegde
Apologies for the delay in response.
How to send the data to AWS S3 bucket?
Yes, your approach to using Azure Data Factory for this use case is valid. While ADF does not natively support Amazon S3 as a sink, leveraging AWS Transfer Family (SFTP) to act as an intermediary is a good solution.
Additional Considerations:
Handling Large Files (500 GB) - Use a Self-hosted Integration Runtime (SHIR) in ADF for better performance and scalability. Optimize ADF Copy Activity settings such as parallelism and chunk size to handle the file efficiently.
Network Bandwidth - Ensure sufficient bandwidth between the source SFTP, ADF IR, and AWS SFTP to avoid slow transfers.
Fault Tolerance - Enable retry policies in ADF to handle potential interruptions during the transfer.
Cost - Review the pricing for AWS Transfer Family and Azure Data Factory operations for large file transfers.
I hope this information helps. Please do let us know if you have any further queries.
Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Thank you.
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Following up to see on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Hey,
Ideally yes it is possible via ADF to copy that much data :
https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance
You can even use SHIR based on the necessary CPU consumption.
How to send the data to AWS S3 bucket? there is no S3 sink for ADF. The reference link you shared doesn't talk about 500 GB single file copy
Regards,
Partha