Windows Azure Drive Demo at MIX 2010

With a Windows Azure Drive, applications running in the Windows Azure cloud can use existing NTFS APIs to access a network attached durable drive.  The durable drive is actually a Page Blob formatted as a single volume NTFS Virtual Hard Drive (VHD). 

In the MIX 2010 talk on Windows Azure Storage I gave a short 10 minute demo (starts at 19:20) on using Windows Azure Drives.  Credit goes to Andy Edwards, who put together the demo. The demo focused on showing how easy it is to use a VHD on Windows 7, then uploading a VHD to a Windows Azure Page Blob, and then mounting it for a Windows Azure application to use.

The following recaps what was shown in the demo and points out a few things glossed over.

Using VHDs in Windows 7

One can easily create and use VHDs on your local Windows 7 machine.   To do this run “diskmgmt.msc”.   This will bring up the Disk Management program in Windows 7.  You can then use the “Action” menu to create VHDs and attach (mount) VHDs.  Note, Windows Azure Drives only supports “Fixed Size” VHDs (we do not not support “Dynamic” VHDs).  So when creating a VHD in Disk Management, use the “Fixed Size” option.

Once the VHD is attached, you can then store data in it as any normal NTFS drive. 

You can also use Disk Management to detach a drive by right clicking on the mounted disk, and then choosing “Detach VHD”. This is shown in the below, where we right clicked on Disk 1, and it brought up the menu below to “Detach VHD” the X: drive we had mounted.

Uploading VHDs into a Windows Azure Page Blob

The demo then showed using a command line tool Andy wrote to upload the local VHD file to a Page Blob in a Windows Azure Storage account using the Storage Client Library.  The next blog posting will go into Page Blobs and provide the code for this tool.

Using a Windows Azure Drive in the Cloud Application

The demo then showed how to code up and use a Windows Azure Drive in a Windows Azure role.   To do this we used the Windows Azure SDK and the Windows Azure Storage Client Library provided with it, so make sure you have the Windows Azure SDK installed.

• Setting up the storage account

The first step in your application is to create a configuration settings to specify the storage account name and its secret key.  To do this you would add a configuration setting to your ServiceConfiguration.cscfg as shown below:

     <ConfigurationSettings>
      <Setting name="CloudStorageAccount" 
       value="DefaultEndpointsProtocol=http; 
       AccountName=xdrivedemo;AccountKey=vhz/HSBZzBwiNpx9399hnWAe3zJGX … f+VvVA4OQ==" />
    </ConfigurationSettings>
 
 

The storage account name we are using for the demo is “xdrivedemo” (the storage AccountKey isn’t fully listed for obvious reasons).

Now the same configuration setting can be specified by clicking on your web or worker role in Visual Studio to get the following settings screen.   To create the storage account configuration setting select “Settings” and then use “Add Setting”.

In the above, we added the CloudStorageAccount configuration setting with the storage account name and key.  When creating the setting we specified its type to be “Connection String”.   When you do this, you’ll see a “…” at the end in the value field.  When you click on that you’ll see the following dialog box, where you can enter your storage account name and account key.

Once the storage account configuration settings has been entered, we now need to write the code to allow us to access the storage account in our application.  To do this, in the application we create a CloudStorageAccount object or static member.  We can use this to then access our Windows Azure Storage account where the Page Blobs (drives) will be stored. This is done with the following code:

 public CloudStorageAccount account;
account = CloudStorageAccount.FromConfigurationSetting(
          "CloudStorageAccount"); 
 

At this point, we have an “account” object that we can use to perform actions against our storage account in the cloud.

 

• Initializing the Drive Cache

Now before the application can use a Windows Azure Drive it first needs to initialize the Drive Cache.  This initialization only needs to be done once per running VM instance.  Once this initialization is done successfully for the VM instance, then all processes/threads running under that instance can mount and manipulate drives.

To perform the drive cache initialization we first need to reserve part of the local disk resources for the VM instance to be used by the Drive Cache.  To do this, we first specify a Local Storage resource as part of the ServiceDefinition.csdef as follows:

     <LocalResources>
      <LocalStorage name="AzureDriveCache" 
       cleanOnRoleRecycle="false" sizeInMB="1000" />
    </LocalResources>
 
  

Alternatively, one can use the web/worker role interface specification provided as part of the SDK in Visual Studio to create the local storage resource.  To do this, click on your web or worker role in Visual Studio to get the following settings screen.   Then to create the local storage resource, select “Local Storage”. Then select “Add Local Storage”.  You then give it a name and specify the amount of local storage you want this resource to have in MBytes as shown here:

In the above, we’ve specified to reserve 1 GByte of the local disk space for this resource.  We also left “Clean on Role Recycle”  unchecked because we don’t want the cache flushed if/when the role restarts. 

The disk space for this local storage resource comes out the allocated disk space for the VM instance you are running your role on.  The maximum size you can specify for the local storage resource is the maximum local disk space your VM instance can have depending upon the VM size you have chosen (Small, Medium, Large or Extra Large).  For example, a small VM instance provides up to 250 GBytes of local disk space.

Now that we have the local storage resource defined, we now need to initialize the drive cache.  To do this, we add the following code to do this initialization when the role starts up:

 LocalResource azureDriveCache = 
           RoleEnvironment.GetLocalResource("AzureDriveCache");
CloudDrive.InitializeCache(azureDriveCache.RootPath + 
           "cache", azureDriveCache.MaximumSizeInMegabytes); 
 

For this, we pass into InitializeCache the RootPath of our local storage resource, which specifies the drive letter to use for the cache. Then the second parameter is the amount storage space from this local storage resource we want to use for the drive cache.  Note, this drive cache is to be used across all of the drives to be mounted on this VM instance.  You can mount up to 16 drives, and all of these mounted drives share this drive cache disk space that is set aside by this call to InitailizeCache.

One point here is that there is currently a bug in InitializeCache when its first parameter (the path) ends in “\”.   This is why we add the “cache” in the above, which puts the drive cache files under a cache directory in that RootPath. Another option to get around this bug would be to trim the “\” off as follows:

 CloudDrive.InitializeCache(azureDriveCache.RootPath.TrimEnd({'\\'}), 
           azureDriveCache.MaximumSizeInMegabytes); 
 

At this point we have initialized the drive cache, so we can start accessing and using Windows Azure Drives in our application.

• Creating a drive

Before we mount a drive we first need to have either uploaded a single volume NTFS VHD into a Page Blob (as we did in the demo), or create a drive with our application in the cloud.  Creating a drive can be done with the following:

 CloudDrive drive = account.CreateCloudDrive(containerName+"/"+blobName);
drive.Create(sizeOfDriveInMegabytes); 
 

We first create a CloudDrive object specifying the container name and blob name for the Page Blob we want to create.  The CloudDrive Create method call will then create the Page Blob in the storage account under “containerName/blobName”, and it will be created with the size in megabytes passed into the Create.   Note, the “containerName” Blob Container must already exist in the storage account for the call to succeed, so please make sure you have created that ahead of time.

In addition to creating the Page Blob, the CloudDrive Create method will also format the contents of the Page Blob as a single volume NTFS Fixed Size VHD.   Note, the smallest drive you can create is 16MBytes, and the maximum size is 1TByte.

One important point here is that for a Page Blob, all pages that do not have any data stored in them are treated as initialized to zero, and you are only charged for pages you write data into.  We take advantage of this when using Page Blobs as VHDs, since when you create and initialize a Fixed Size VHD most of it consists of empty blocks.   As we showed in the demo, we had a VHD of 100MBytes and we only had to upload 3MBs of pages, since the rest of the VHD was effectively still zeros.   Therefore, when you create a VHD using CloudDrive Create, the formatting only stores data in the pages that need to be updated in order to represent the VHD format and single volume NTFS initialization.   This means that your drive after creation will cost just a small fraction of the size you created, since most of the pages are empty.

• Mounting a drive

Once the drive cache has been initialized, the application can now start to use Windows Azure Drives.  To mount a drive one just needs to create a CloudDrive object using the “account” created earlier.  When doing this, we pass in the container name and blob name for the Page Blob we want to access using the CloudDrive object as follows:

 CloudDrive drive = account.CreateCloudDrive(containerName+"/"+blobName);
string path = drive.Mount(sizeOfDriveCacheToUse, 
                          DriveMountOptions.None); 
 

When calling mount, you pass in the amount of the drive cache space to use for this specific drive being mounted, and the drive mount options you want to use.  Most applications will want to use the default “DriveMountOptions.None”.  A successful mount will return a string specifying the drive letter the VHD was mounted to, so the application can start using the NTFS drive.

Remember that a given VM instance can mount up to 16 drives, and all of these mounted drives share the total amount of drive cache space when calling InitializeCache described earlier.  This means that the sum of all of the sizeOfDriveCacheToUse parameters for all of the currently active mounted drives for a VM instance has to be less than or equal to the amount of local disk space put aside for the Drive Cache when calling InitializeCache as described earlier.

• Unmounting a drive

Then to unmount the drive, it is as easy as creating a CloudDrive object and calling unmount as follows:

 CloudDrive drive = account.CreateCloudDrive(containerName+"/"+blobName);
drive.Unmount(); 
 

• Snapshotting a drive

Then to create a snapshot of the drive you just need to do the following:

 CloudDrive drive = account.CreateCloudDrive(containerName+"/"+blobName);
Uri snapshotUri = drive.Snapshot(); 
 

Snapshots are read-only.  Snapshots can be created while the drive is mounted or when it is unmounted.   Snapshotting a drive is useful for creating backups of your drives.   Under the covers this uses the Page Blob Snapshot command.  A nice advantage of this is that you only pay for the unique pages across all of the snapshots and the baseline version of the Page Blob.  So as you keep on updating the Page Blob you are only paying for the delta changes being made over the last stored snapshot.

Summary

Finally, there are a few areas that are worth summarizing and re-emphasizing about Windows Azure Drives:

• Only Fixed Size VHDs are supported.
• The minimum drive size supported is 16MBytes and the maximum is 1TByte.
• A VM instance can mount up to 16 drives, and all the active mounts share the drive disk cache space reserved by InitializeCache.
• A Page Blob Drive can only be mounted to a single VM instance at a time for read/write access.
• Snapshot Page Blob Drives are read-only and can be mounted as read-only drives by multiple different VM instances at the same time.
• The Page Blob Drives stored in the Cloud can only be mounted by Windows Azure applications running in the Cloud.   We do not provide a driver with the SDK that allows local mounting of the Page Blob Drives in the Cloud.
• When uploading a VHD to a Page Blob, make sure you do not upload the empty pages in the VHD, since the Page Blob in the Cloud will by default treat these as initialized to zero.  This will reduce your costs of uploading and storing the VHD Page Blob in the Cloud.  

Brad Calder
Windows Azure Storage