AzCopy – Using Cross Account Copy Blob
You can always download latest version from aka.ms/AzCopy.
New features have been added in this release
Support of Cross-Account Copy Blob: AzCopy allows you to copy blobs within same storage accounts or between different storage accounts (visit this blog post for more details on cross account blob copy). This enables you move blobs from one account to another efficiently with respect to cost and time. The data transfer is done by the storage service, thus eliminating the need for you to download each blob from the source and then upload to the destination. You can also use /Z to execute the blob copy in re-startable mode.
Added /MOV: This option allows you move files and delete them from source after copying. Assuming you have delete permissions on the source, this option applies regardless of whether the source is Windows Azure Storage or the local file system.
Added /NC: This option allows you to specify the concurrent network calls. By default, when you upload files from local computer to Windows Azure Storage, AzCopy will initiate network calls up to eight times the number of cores this local computer had to execute concurrent tasks. For example, if your local computer has four cores, then AzCopy will initiate up to 32 (eight times of 4) network calls at one time. However, if you want to limit the concurrency to throttle local CPU and bandwidth usage, you can specify the maximum concurrent network calls by using /NC. The value specified here is the absolute count and will not be multiplied by the core count. So in the above example, to reduce the concurrent network calls by half, you would specify /NC:16
Added /SNAPSHOT: This option allows you transfer blob with snapshots. This is a semantic change, as in AzCopy CTP 1 (Released in October, 2012), it would transfer a blob’s snapshots by default. However starting from this version, by default AzCopy won’t transfer any snapshots while copying a blob. Only with /SNAPSHOT specified, AzCopy will actually transfer all the snapshots of a blob to destination, however these snapshots will became separate blobs instead of snapshots of original base blob in destination, so each of the blobs will be full charged for (not block reuse between them). The transferred blob snapshots will be renamed in this format: [blob-name] (snapshot-time)[extension].
For example if readme.txt is a source blob and there are 3 snapshots created for it in source container, then after using /SNAPSHOT there will be 3 more separate blobs created in destination container where their names will looks like
readme (2013-02-25 080757).txt
readme (2012-12-23 120657).txt
readme (2012-09-12 090521).txtFor the billing impact compare blob snapshots to separate blob, please refer to this blog post Understanding Windows Azure Storage Billing
Added /@:response-file: Thisallows you store parameters in a file and they will be processed by AzCopy just as if they had been specified on the command line. Parameters in the response file can be divided as several lines but each single parameter must be in one line (breaking 1 parameter into 2 lines is not supported). AzCopy parses each line as if it is a single command line into a list of parameters and concatenates all the parameter lists into one list, which is treated by AzCopy as from a single command line. Multiple response files can be specified, but nested response file is not supported, instead it will be parsed as a location parameter or file pattern. Escape characters are not supported, except that “” in a quoted string will be parsed as a single quotation mark. Note that /@: after /- does not means a response file, it is treated as a location parameter or file pattern like other parameters
Examples
Here are some examples that illustrate the new features in this release.
Copy all blobs from one container to another container under different storage account
AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/ /sourcekey:<key> /destkey:<key> /S
The above command will copy all blobs from the container named “sourcecontainer” in storage account “sourceaccount” to another container named “destcontainer” in storage account “destaccount”
If you have base blob with snapshots, please add/Snapshot to move all snapshots with base blob to destination, please be noted the blob snapshot will be renamed to this format in destination: [blob-name] (snapshot-time)[extension]
AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/ /sourcekey:<key> /destkey:<key> /S /SNAPSHOT
For example if you have readme.txt with 3 snapshots in source container, then it will be as below in destination container
readme.txt
readme (2013-02-25 080757).txt
readme (2012-12-23 120657).txt
readme (2012-09-12 090521).txt
If you’d like to delete those blobs from the source container when the copy is complete, then just added /MOV as below
AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/ /sourcekey:<key> /destkey:<key> /MOV /S
You can also create a response file to make it easier of running same command again and again. Create a txt file called “myAzCopy.txt” with content below
#URI of Source Container
https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/
#URI of Destination Container
https://<destaccount>.blob.core.windows.net/<destcontainer>/
Then you can run the command below to transfer files from source container to destination container
AzCopy </@:C:\myAzCopy.txt> /sourcekey:<key> /destkey:<key> /MOV /S
Comments
Anonymous
May 19, 2013
Hello, Thanks a lot for this amazing tool. Is there any plan to add a timeout switch to enable large file copy (by increasing it). Or, will you release the source code so we can submit contribution to this tool. Best regards, CyprienAnonymous
June 24, 2013
Hello, Does this copy also the Blob metadata? Thanks.Anonymous
June 25, 2013
@Ronen - Yes it copies the metadata. It internally uses (async) CopyBlob across storage accounts see msdn.microsoft.com/.../dd894037.aspx for details on other properties that the API sets. @Sandorfr - Timeout switch is a good request. We will add it to the list of feature requests.Anonymous
August 28, 2013
Is there a companion command to get the names of the files that are in the source container?Anonymous
August 28, 2013
The comment has been removedAnonymous
September 27, 2013
I am not sure what I am doing wrong but nothing I try seems to work. It doesn't throw any errors but no matter what I try it always prints the following: Transfer summary:
Total files transferred: 0 Transfer successfully: 0 Transfer failed: 0 I tried copying blobs between storage accounts in the same subscription. I tried copying blobs between storage accounts in different subscriptions. I tried making the blob containers public. I tried blob URLs with and without SSL. I tried specifying a single blob in the source and a container in the destination. I tried specifying containers in the source and destination. I tried with and without trailing slash in the container path. All to no avail. It resolutely produces the above given output. I tried running it with Fiddler on and all I see is 2 blob listing calls for the source and destination accounts with nothing after that. At this point I am ready to give up and write my own thing. Is the source for this tool up on github? I tried searching in the Windows Azure github account and I couldn't find it there!
Anonymous
September 29, 2013
@Raj - Source of this tool is not yet opened to public. Could you please try to add /S to see whether it resolves your issue? AzCopy sourceaccount.blob.core.windows.net/sourcecontainer destaccount.blob.core.windows.net/destcontainer /SourceKey:****** /DestKey:****** /SAnonymous
November 06, 2013
What is the Microsoft.WindowsAzure.Storage.DataMovement assembly that ships with AzCopy? There doesn't seem to be any information out there on it, other than it exists.Anonymous
November 11, 2013
@Reggie This Data Movement assembly is an internal library used by AzCopy today. We are considering to expose this library so that customer can introduce more data movement capabilities in their own applications, however we don’t have a date to share yet.Anonymous
March 26, 2014
Any word yet on sharing the code or assembly for this? Honestly, you are getting throughputs that are 3X-5X faster than we get with our code. Thanks, MarkAnonymous
March 27, 2014
Hi Mark, We will consider to open source AzCopy in the future, but do not have a timeline yet. Regards ZhimingAnonymous
April 22, 2014
Is there a way to know how many threads AzCopy is executing. I started to copy 2 small files (3GB each) from a directory as a test and I'm trying to get some statistics. I'm using /V switch but I don't believe it will provide the statistics I need.
- How many threads is AzCopy executing
- Time taken to copy the files
Anonymous
April 22, 2014
Hi Gus, Thanks for your feedback. AzCopy will by default start CORE*8 concurrent operations to copy the blobs, you can use option /nc to change the number. And AzCopy does not show ‘time taken to copy the files’, we will add it to the list of feature request. ZhimingAnonymous
June 12, 2014
Is it possible to use this tool to copy container content between two Storage Emulators on two machines (in the same LAN)?Anonymous
June 13, 2014
Hello Gabriel Access to emulator outside of localhost is not a supported scenario. SamarthAnonymous
June 15, 2014
Hi there, if I was to transfer some VHDs from one subscription to another WITHIN the same datacentre, what would the data charges be? ThanksAnonymous
June 15, 2014
Can I copy files from storage emulator to an azure storage account? Having problems connectting to storage emulator account.Anonymous
June 16, 2014
Hi J, There are no Bandwidth charges for ingress and egress within the same region, so the only charges would be transaction charges and the capacity used, as described here: blogs.msdn.com/.../understanding-windows-azure-storage-billing-bandwidth-transactions-and-capacity.aspx. ZhimingAnonymous
June 18, 2014
Hi Zach, It is not possible to transfer from storage emulator to cloud directly, but you could download all the files to local disk and then upload them to the cloud storage account. ZhimingAnonymous
July 17, 2014
can we initiate the download from SSMS or SQL agent jobAnonymous
July 19, 2014
The comment has been removedAnonymous
July 23, 2014
hi, i have many containers(100+ ) in a storage account and each of those containers have many blobs in them. I want to copy all these containers and the blobs to a different storage account. AzCopy https://<sourceaccount>.blob.core.windows.net/ https://<destaccount>.blob.core.windows.net/ /sourcekey:<key> /destkey:<key> /S will the above command copy all the containers and the blobs?Anonymous
July 24, 2014
Hi JAW, AzCopy does not support to copy whole storage account's data yet, you must explicitly specify the container in the 'source'. We will put this request into our feature list. ZhimingAnonymous
July 31, 2014
Hi Team, My customer has below query:- We are moving data from Azure VM to Azure storage account using AzCopy. Both VM and Storage account is in Same region. we have got ~60 MBps speed for 1GB of 50 files, and ~1Mbps speed for 200MB of 10000 files. What are the commands for AzCopy to parallelize this to allow more throughput? Any response would be appreciate.Anonymous
August 01, 2014
Hi Deva, AzCopy by default starts CORE * 8 concurrent operations to transfer data in parallel, you can specify option /nc:<number of concurrent operations> to change the default value, and the throughput is also limited by the size of the VM you are using. ZhimingAnonymous
September 07, 2014
Hi, can I move a VHD from one container to another, either within the same storage account or a different storage account? When I capture a VM, I don't believe I can specify the container so end up with many in the same container. I then want to copy a VHD to a colleague but all will be copied. Is there a way around this?Anonymous
September 08, 2014
The comment has been removedAnonymous
October 12, 2014
Hello, Is it possible to run this within Task sceduler. I can't seem to be able to open up AzCopy in a task.Anonymous
October 13, 2014
The comment has been removedAnonymous
October 13, 2014
The comment has been removedAnonymous
October 14, 2014
The comment has been removedAnonymous
November 03, 2014
It would be great if you could include all Azcopy parms on a single page somewhere and make it easily findableAnonymous
November 03, 2014
Hi, Thanks for your feedback, we have one single page to introduce all the details about AzCopy azure.microsoft.com/.../storage-use-azcopy (short link: http://aka.ms/azcopy). ZhimingAnonymous
December 09, 2014
Hello, Can i use AzCopy to copy/move whole storage account without mention container names. ThanksAnonymous
December 09, 2014
Hi Jalal, AzCopy does not support for copying whole storage account yet, we have put this requirement into our feature list, but no concrete timeline to share yet. You may want to use Azure Powershell to iterate all the containers within a storage account and call AzCopy to copy each container. Zhiming