次の方法で共有

Writing a console app that access Azure Storage

  • [アーティクル]

Yes, it is possible to write a console app that accesses Azure Storage. I encountered several gotcha's in this process. My first attempt to write a console app that uploaded some files to azure resulted in something like this:

 using System;

using System.Configuration;

using System.IO;

using Microsoft.WindowsAzure;

using Microsoft.WindowsAzure.StorageClient;

...

var storageAccount = CloudStorageAccount.FromConfigurationSetting("myConnectionString");

CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

CloudBlobContainer blobContainer = blobClient.GetContainerReference(container);

blobContainer.CreateIfNotExist();                

DirectoryInfo di = new DirectoryInfo(directory);

foreach (FileInfo fi in di.GetFiles(searchPattern))

{

    string name = fi.Name;

    var blob = blobContainer.GetBlobReference(name);

    blob.UploadFile(fi.FullName);                 

    Console.WriteLine("uploaded " + fi.FullName);

}

where myConnectionString is defined in app.config like so:

 <?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <connectionStrings>
    <add name="myConnectionString" connectionString="DefaultEndpointsProtocol=http;AccountName=XXX;AccountKey=XXX"/>
  </connectionStrings>  
</configuration>

GOTCHA #1

Running this code gave me following exception:

System.Runtime.InteropServices.SEHException was caught Message=External component has thrown an exception. Source=msshrtmi ErrorCode=-2147467259
StackTrace:
at RoleEnvironmentGetConfigurationSettingValueW(UInt16* , UInt16* , UInt64 , UInt64* )
at Microsoft.WindowsAzure.ServiceRuntime.Internal.InteropRoleManager.GetConfigurationSetting(String name, String& ret)
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetConfigurationSettingValue(String configurationSettingName)
at AzureUpload.Program.b__2(String configName, Func`2 configSetter) in C:\Users\siddjain\Documents\Visual Studio 2010\Projects\ConsoleAplication1\Program.cs:line 51
at Microsoft.WindowsAzure.CloudStorageAccount.StorageAccountConfigurationSetting..ctor(String configurationSettingName)
at Microsoft.WindowsAzure.CloudStorageAccount.FromConfigurationSetting(String settingName)
at AzureUpload.Program.UploadBlob(String directory, String searchPattern, String

This post explains how to fix it. Azure calls into your code to get the configuration string. Hence, you need to define a method that will return the config string to azure, before you call

 CloudStorageAccount.FromConfigurationSetting("myConnectionString")

. The fix was to add following code:

 CloudStorageAccount.SetConfigurationSettingPublisher((configName, configSetter) =>
{
   // for a console app, reading from App.config
   configSetter(ConfigurationManager.ConnectionStrings[configName].ConnectionString);
   // OR, if running in the Windows Azure environment
   // configSetter(RoleEnvironment.GetConfigurationSettingValue(configName));
});

Alternatively, use the CloudStorageAccount.Parse method instead of CloudStorageAccount.FromConfigurationSetting.

GOTCHA #2

I was able to upload blobs now, but couldn't access them in IE. Kept getting 404s. To fix this, we need to mark the container as public as in:

 if (blobContainer.CreateIfNotExist())
{
    blobContainer.SetPermissions(new BlobContainerPermissions
        { PublicAccess = BlobContainerPublicAccessType.Blob }
    );
}

GOTCHA #3

the '/' character is OK to use in blob name (it gives the illusion of a hierarchy), but not OK to use in container name.

GOTCHA #4

I tried following container name "TestContainer" and got an exception at blobContainer.CreateIfNotExist() with following message: One of the request inputs is out of range. It turns out upper case characters are not permitted in container names. See this post: https://msdn.microsoft.com/en-us/library/dd135715.aspx

GOTCHA #5

You may find yourself in following scenario: you have a container that stores a number of blobs. This container's contents need to be periodically updated. When updating, you want to delete previous contents of the container. If you believe in writing least amount of code, you would probably delete the container, re-create it, and upload new blobs like this:

blobContainer.Delete();
if (blobContainer.CreateIfNotExist())
{
    blobContainer.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Container });
}
// start uploading new blobs

this is not going to work, and you will get a StorageClientException with the message "container does not exist" when you try to upload a blob. If you stick a breakpoint at
blobContainer.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Container });
you will find this breakpoint is not hit. Thus, it seems blobContainer.CreateIfNotExist() thinks that the container already exists even though we have deleted it in the very previous line. So what's the catch? The issue is documented here: https://social.msdn.microsoft.com/Forums/en-US/windowsazuredata/thread/f9ceaea3-be5b-40e0-a23d-bacea451d53d. Calling blobContainer.Delete(); just marks and queues up the container for deletion. When it returns, it is not guaranteed that the container has been deleted. To get around this issue, instead of calling blobContainer.Delete(), use following code:

public void EmptyContainer(string containerName)
{
var container = client.GetContainerReference(containerName);
if (ContainerExists(container))
{
foreach (var item in container.ListBlobs())
{
CloudBlob blob = new CloudBlob(item.Uri.AbsoluteUri);
blob.Delete();
}
}
}

 

private static bool ContainerExists(CloudBlobContainer container)
{
     bool exist = true;
     try
     {
             container.FetchAttributes();
}
     catch (StorageClientException e)
{
            if (e.ErrorCode != StorageErrorCode.ResourceNotFound)
{
                throw;
}
            exist = false;
}
     return exist;
}

Some more notes: while experimenting with high-throughput applications (applications that are uploading blobs at a high rate), I have found that the CreateIfNotExist() method is notoriously slow. How slow? Of the order of minutes. The ContainerExists method above is not faster either. It can take minutes for the FetchAttributes() call to return. Also read [this](https://social.msdn.microsoft.com/Forums/en/windowsazuredata/thread/8480223a-8cc7-4054-919c-a32ff02d5143) post that highlights concurrency issues with CreateIfNotExist(). Finally, if the container exists CreateIfNotExist() will cause a 409 in fiddler which appears highlighted in red and can lead you to think there is a bug in your code. 

Tips:

  • You don't need to add reference to Microsoft.WindowsAzure.ServiceRuntime.dll. The only reference I needed to add was to Microsoft.WindowsAzure.StorageClient.dll
  • If you do have a reference to Microsoft.WindowsAzure.ServiceRuntime.dll, RoleEnvironment.IsAvailable() tells whether you are running in an Azure environment or not. 

So we see its not that simple to write our first Hello World Azure app.

Comments

  • Anonymous
    February 26, 2011
    It's February 2011. I am just getting into Azure storage now, and was wondering if anything has changed since  you did this awesome blog post. Thanks!