Using BLOB Storage: Updated Sample Code
I made some changes to my sample code after receiving some feedback (thanks Aleks) on best practices and ways that I could better write the code that I had used as a sample for the upload piece in the post Using BLOB Storage: Azure Storage and GAE Blobstore. I thought that I might be useful to some if I posted the updated code for others to use as a reference.
The first block that was changed slightly contained two primary changes:
1) Using GetContainerReference is probably simpler/easier than using the constructor. That is the first change highlighted below.
2) Most of the blob options are set to default values so I only need to set the ones that I need to be different. Thus, you’ll see the removal of several assignments when comparing the before and after below
Original Code Sample |
Updated Code Sample |
//need to check if container exists //get the target container CloudBlobContainer BlobContainer = new CloudBlobContainer(StorageAccount.BlobEndpoint.ToString() + "/" + ContainerName, BlobClient); //set request options BlobRequestOptions options = new
options.UseFlatBlobListing = true;
BlobContainer.CreateIfNotExist(options);
|
CloudBlobContainer BlobContainer = BlobClient.GetContainerReference(StorageAccount.BlobEndpoint.ToString() + "/" + ContainerName); BlobRequestOptions options = new BlobRequestOptions(); options.UseFlatBlobListing = true;
BlobContainer.CreateIfNotExist(options);
|
In the next block is where there is a significant departure from the previous version. In the previous version I had purposefully grabbed the list of BLOBs for comparison as close to write time as possible to minimize the window for a race condition, but that also meant a call back into the cloud every time and didn't guarantee that I wouldn't hit a race condition nor did it help with BLOB updates. In the updated version I don’t even do the BLOB list comparison; instead I let the BLOB API handle it by using the AccessCondition.IfNotModifiedSince(). This shortens and simplifies the implementation while providing both the ability to avoid overwriting new files while at the same time creating new or updating older files. There is certainly a time to use the list mechanism, but in this case the conditional write serves the purpose very well.
Original Code Sample |
Updated Code Sample |
//iterate through files foreach (FileInfo file in FileList) { string NewFileName = PreAppendPath + "\\" + file.Name; NewFileName = NewFileName.Replace(@"\", "/").ToLower(); IListBlobItem foundItem = null; try {
}
if (foundItem == null) { //open file and read in
CloudBlob destBlob = TargetContainer.GetBlobReference(NewFileName);
} }
|
//iterate through files foreach (FileInfo file in FileList) { string NewFileName = PreAppendPath + "\\" + file.Name; NewFileName = NewFileName.Replace(@"\", "/").ToLower(); try { BlobRequestOptions options = new BlobRequestOptions(); options.AccessCondition = AccessCondition.IfNotModifiedSince(file.LastWriteTimeUtc); //write the file CloudBlob destBlob = TargetContainer.GetBlobReference(NewFileName);
destBlob.UploadFile(file.FullName); } catch (Exception ex){ Console.WriteLine("Exception: " + ex.Message); } } |
The last change that was made was to minimize the code to upload the file by not explicitly creating and using a stream in the code and instead I pass the file path to the BLOB API and let it handle the upload. At the end we have a cleaner implementation that has about 20 operational lines of code versus about 30. Here is the complete updated sample with comments:
static void MoveFromFolderToContainer(string FolderPath, string ContainerName, CloudStorageAccount StorageAccount)
{
//use the last folder name in path as the root and take it and any ancestor folders to the file name
string[] FolderNamesArray = FolderPath.Split(new char[] {'\\'});
string ResourceRootFolderName = FolderNamesArray[FolderNamesArray.Length -1];
CloudBlobClient BlobClient = StorageAccount.CreateCloudBlobClient();
CloudBlobContainer BlobContainer = BlobClient.GetContainerReference(StorageAccount.BlobEndpoint.ToString() + "/" + ContainerName);
BlobRequestOptions options = new BlobRequestOptions();
options.UseFlatBlobListing = true;
//In case this is the first time that we are accessing/loading this container we'll call CreateIfNotExist() to ensure the container exists
BlobContainer.CreateIfNotExist(options);
//open file system
System.IO.DirectoryInfo DirInfo = new DirectoryInfo(FolderPath);
//call recursive function to load all contents and children of a directory into the storage container
IterateFolders(DirInfo, BlobContainer, ResourceRootFolderName);
}
static void IterateFolders(DirectoryInfo CurrentDir, CloudBlobContainer TargetContainer, string RootFolderName)
{
DirectoryInfo[] ChildDirectories = CurrentDir.GetDirectories();
//recurively iterate through all descendants of the source folder
foreach (DirectoryInfo ChildDir in ChildDirectories)
{
IterateFolders(ChildDir, TargetContainer, RootFolderName);
}
//get the path name including only the rootfoldername and its decendants; it will be used as part of the filename
string PreAppendPath = CurrentDir.FullName.Remove(0, CurrentDir.FullName.IndexOf(RootFolderName));
//get file list
FileInfo[] FileList = CurrentDir.GetFiles();
//iterate through files
foreach (FileInfo file in FileList)
{
//filename + path and use as name in container; path + filename should be unique
string NewFileName = PreAppendPath + "\\" + file.Name;
//create a normal form for the filename: lcase + replace "\" with "/"
// Replacement of "\" is required to fix what seems to be a bug.
// When the filename "folder\childfolder\filename.jpg" is saved it is ends up saved in dev storage as "folder/childfolder/filename.jpg".
// When we check to see if there is a matching filename using First<> it fails.
// So, make the change to the filename ahead of time and it works fine for cloud deployment too.
NewFileName = NewFileName.Replace(@"\", "/").ToLower();
try
{
//In previous versions we iterated through the file list and did not upload if the file existed in the blob container
//In this version we will always attempt to upload the file and set the AccessCondition to only work if the blob is older than our source file
BlobRequestOptions options = new BlobRequestOptions();
options.AccessCondition = AccessCondition.IfNotModifiedSince(file.LastWriteTimeUtc);
//write the file
CloudBlob destBlob = TargetContainer.GetBlobReference(NewFileName);
destBlob.UploadFile(file.FullName, options);
}
catch (Exception ex)
{
Console.WriteLine("Exception: " + ex.Message);
}
}
}
HTH :)
Comments
Anonymous
September 18, 2010
I'm confused in your code I"m pasting below. It looks like you declare options but from what I can tell, it is never referenced. Am I missing something? try { BlobRequestOptions options = new BlobRequestOptions(); options.AccessCondition = AccessCondition.IfNotModifiedSince(file.LastWriteTimeUtc); //write the file CloudBlob destBlob = TargetContainer.GetBlobReference(NewFileName); destBlob.UploadFile(file.FullName); } catch (Exception ex) { Console.WriteLine("Exception: " + ex.Message); } }Anonymous
October 10, 2010
Sorry for the delayed response, comments are supposed to be emailed to me, but apparently haven't been. Anyway, you are correct, it was originally posted incorrectly without the options passed as a param to UploadFile([name], [options]). If you look at the bottom of the post you'll see the correct syntax.