(Cross-Post) Introducing Azure Storage Data Movement Library Preview
Since AzCopy was first released, a large number of customers have requested programmatic access to AzCopy. We are pleased to announce a new open-sourced Azure Storage data movement library for .NET (DML for short). This library is based on the core data movement framework that powers AzCopy. The library is designed for high-performance, reliable and easy Azure Storage data transfer operations enabling scenarios such as:
• Uploading, downloading and copying data between Microsoft Azure Blob and File Storage
• Migrating data from other cloud providers such as AWS S3 to Azure Blob Storage
• Backing up Azure Storage data
Here is a sample demonstrating how to upload a blob, please find more samples at github.
using System;
using System.Threading;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
// Include the New Azure Storage Data Movement Library
using Microsoft.WindowsAzure.Storage.DataMovement;
// Setup the storage context and prepare the object you need to upload
string storageConnectionString = "myStorageConnectionString";
CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = account.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("mycontainer");
blobContainer.CreateIfNotExists();
string sourcePath = "path\\to\\test.txt";
CloudBlockBlob destBlob = blobContainer.GetBlockBlobReference("myblob");
// Use the interfaces from the new Azure Storage Data Movement Library to upload the blob
// Setup the number of the concurrent operations
TransferManager.Configurations.ParallelOperations = 64;
// Setup the transfer context and track the upoload progress
TransferContext context = new TransferContext();
context.ProgressHandler = new Progress<TransferProgress>((progress) =>
{
Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred);
});
// Upload a local blob
var task = TransferManager.UploadAsync(
sourcePath, destBlob, null, context, CancellationToken.None);
task.Wait();
Azure Storage Data Movement Library has the same performance as AzCopy and exposes the core functionalities of AzCopy. You can install the first preview of the library from Nuget or download the source code from Github. In the initial version (0.1.0) of this library, you can find the following abilities:
• Support data transfer for Azure Storage abstraction: Blob
• Support data transfer for Azure Storage abstraction: File
• Download / Upload / Copy single object
• Control the number of concurrent operations
• Synchronous and asynchronous copying
• Define the number of concurrent operations
• Define the suffix of the user agent
• Set the content type
• Set the Access Condition to conditionally copy objects, for example copy objects changed since certain date
• Validate content MD5
• Download specific blob snapshot
• Track transfer progress: bytes transferred, number of success/fail/skip files
• Recover (Set/Get transfer checkpoint)
• Transfer Error handling (transfer exception and error code)
• Client-Side Logging
DML is an open source project, we welcome contributions from the community. In particular we are interested in extensions to our samples to help make them more robust. Together with the release of version 0.1.0, we have created the following samples, for more details, please visit Github Readme.md.
• Upload/Download/Copy an Azure Storage Blob
• Migrate data from AWS S3 to Azure Blob Storage
Next Steps
We will continue the investment for both AzCopy and Data Movement Library, and in the next releases of Data Movement Library, we will add the support for more advanced features, which shall include:
• Download / Upload / Copy directory (Local file directory, blob virtual directory, File share directory)
• Transfer directory in recursive mode or flat mode
• Specify the file pattern when copying files and directories
• Download Snapshots under directories
As always, we look forward to your feedback.
Microsoft Azure Storage Team