Share via

Resuming timed out uploads/downloads to/from Azure blob storage with azCopy

  • Article

http://superwidgets.files.wordpress.com/2014/09/blob-storage.pngAzCopy is a command line tool by Microsoft that allows for easy uploads/downloads to and from Azure storage. In addition to offering a non-programmatic way of transferring files from/to Azure storage, it provides the flexibility of choice between page and block blobs in Azure blob storage. Page blobs have a maximum of 1TB size. Azure VMs' VHD files for example are implemented as page blobs and suffer from the same limitation. azCopy tool also offers an important feature which is the ability to resume timed-out or interrupted uploads and downloads. To get started, see this post to get Azure PowerShell module and this post to setup a certificate. Here's an example of uploading files to Azure blob storage using PowerShell:

$azPath = "C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy" Set-Location $azPath $StorageAccountName = "samscorner1" $StorageAccountKey = "Nk48Iljv0AHbbbbbbbbbbbbbbbbbbbbbbbbbbyEkRwbd+BTqew==" $ContainerName = "copiedvhds" $SourceFolder = "e:\vhd" $DestURL = "https://$StorageAccountName.blob.core.windows.net/$ContainerName " $Result = .\AzCopy.exe $SourceFolder $DestURL /BlobType:block /destkey:$StorageAccountKey /Y $Result

 This will upload all files in the "e:\vhd" folder to the "copiedvhds" container in the "samscorner1" Azure Storage account. The script output may look like:

http://superwidgets.files.wordpress.com/2014/09/az18.jpg

You can also see the files in your manage.windowsazure.com interface:

http://superwidgets.files.wordpress.com/2014/09/az19.jpg http://superwidgets.files.wordpress.com/2014/09/az20.jpg Sometimes, the upload or download my be interrupted or time-out, especially if transferring a large file that takes hours or days. A timed-out transfer may produce output like:

e:\vhd\v2012R2-AZ1-v2012R2-AZ1-0906-2b.vhd: The client could not finish the operation within specified timeout.

http://superwidgets.files.wordpress.com/2014/09/az21.jpg To resume the transfer, just run the same .\azCopy.exe command or use a script like:

[int]$Failed = $Result[5].Split(":")[1].Trim() $Journal = "$env:LocalAppData\Microsoft\Azure\AzCopy" $i=1 while ($Failed -gt 0) {     $i++     [int]$Failed = $Result[5].Split(":")[1].Trim()     $Result = .\AzCopy.exe /Z:$Journal     $Result     $i }

This script will continue to run the same .azCopy.exe command until "Transfer Failed" output is down to zero. azcopy tool stores two journal files under %\AppData%\Local\Microsoft\Azure\AzCopy like C:\Users\samb\AppData\Local\Microsoft\Azure\AzCopy folder. It looks them up upon starting to detect if this is a transfer that did not complete and resumes it. Output will look like: http://superwidgets.files.wordpress.com/2014/09/az22.jpg