Use PowerShell & azcopy to copy files from one Azure Storage account to another

I needed to build a script to copy files from one Azure storage account to another.  The storage accounts were in different subscriptions - but using the access keys meant this didn't matter.  All the files were in a single folder, and there were millions of files.  I only wanted to pick up and copy the -video- files.  In the script below I copied files from source > local folder > destination.  I also spat out a listing of files at the end to both screen and CSV.

(NB:  I could also have used azcopy to copy directly from source storage account to destination storage account - but there were a few 'page blob' type files for which that didn't work, so copying via local directory was the proposed solution.)

 

 # COPY BLOB FILES 
# FROM: AZURE SOURCE STORAGE ACCOUNT 
# VIA: LOCAL FOLDER
# TO: AZURE DESTINATION STORAGE ACCOUNT

#PAGING PARAMETERS
$LimitRowsForTesting = 150
$MaxReturn = 100
$token = $Null
$total = 0

$bloblist = $null

#source storage account (e.g. https://mysource.blob.core.windows.net)
$source = "mysourcestorage"
$sourcekey = "bT8KLwsa2gYSDsSdt6sl8XPaQLm0rSieW4tNZlvpSgzpXC7AANXuMxIqyaXXkoOrgeg9wQ=="

#destination storage account  (e.g. https://mydestination.blob.core.windows.net)
$destkey = "dOpX0PwYWjUf894WNWvd28ISGKxeplEbSiClqmwVrviOtmfIbqpZvNM0F8F4rtPI3PKIa52gSVQ=="
$dest = "mydestinationstorage"
 
$context = New-AzureStorageContext -StorageAccountName $source -StorageAccountKey $sourcekey
 
do
{    # Fetch first batch of files
    $blobs = Get-AzureStorageBlob -Context $context -Container "video" -ContinuationToken $token -MaxCount $MaxReturn
    $total += $blobs.Count
    if($blobs.Length -le 0) { break;}
    $token = $blobs[$blobs.Count -1].ContinuationToken;

    #break out early for testing
    if($total -ge $LimitRowsForTesting) { $token = $null } 
    Write-Host "Processed $total rows"

    # filter the returned list to only include the file types we're looking for. Also showing example of looking for BlobType
    #https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.blob.blobtype.aspx
    #$filteredBlobs = $Blobs | where {$_.BlobType -eq 'BlockBlob'}
    #$filteredBlobs = $Blobs | where {$_.BlobType -eq 'PageBlob'}
    $filteredBlobs = $Blobs | where {$_.ContentType -eq 'video/mp4'}


    # add matching files to our overall list
    if ($filteredBlobs.Count -ge 1) {
        $bloblist += $filteredBlobs
    }

    # loop through this batch of files and copy them
    foreach ($blob in $filteredBlobs){
        #this code will perform an azcopy to local, then to dest
        $sourcepath = "https://$source.blob.core.windows.net/video/"
        $localpath = "d:\videotemp\"
        $destpath = "https://$dest.blob.core.windows.net/video/"
        $filename = $blob.Name
        $localfile = $localpath + $filename
        $logfile = "c:\azcopylogs\" + $blob.name + ".txt"
        #copy to local temp folder
        azcopy /V:$logfile /Source:$sourcepath /SourceKey:$sourcekey /Dest:$localpath /Pattern:$filename /XO /z:C:\azcopylogs\copylocal
        
        #copy from local temp to remote storage        
        if (Test-Path $localfile){ 
            azcopy /V:$logfile /Source:$localpath /Dest:$destpath /DestKey:$destkey /Pattern:$filename /XO /z:C:\azcopylogs\copyremote 
        }
        
        #delete local copy
        if (Test-Path $localfile){ 
            Remove-Item $localfile 
        }
    }      
} while ($token -ne $null)
# when continuation/paging token is null there are no more batched rows to be returned

#output list of all files
$bloblist | Export-CSV c:\azcopylogs\file-listing.csv
$bloblist | Format-Table

Comments

• Anonymous
  December 14, 2016
  What is the problem with copying "page blobs?" Possibly the most common use of AzCopy is to copy VHDs which are page blobs.
  • Anonymous
    December 14, 2016
    whoops, it's not that azcopy doesn't support copying page blobs from storage account to storage account... it's that we were trying to copy from a general purpose storage account, to cool storage. Cool storage doesn't support page blobs - only block and block append blobs - and some of the video files we were transferring were created as page blobs.Cool storage: https://azure.microsoft.com/en-us/blog/introducing-azure-cool-storage/ Docs: https://docs.microsoft.com/en-us/azure/storage/storage-blob-storage-tiers
    • Anonymous
      March 10, 2017
      The comment has been removed
      • Anonymous
        March 28, 2018
        whoops sorry. blog transitioned and lost comment notifications. if anyone else has trouble, need to install azcopy first - see https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy
• Anonymous
  March 10, 2017
  The comment has been removed
• Anonymous
  July 06, 2017
  Hard-coded keys when the key can be retrieved via API? Makes the script only useful for one-off...
  • Anonymous
    March 28, 2018
    The comment has been removed