Azure Blob now has append!!!
So just a couple of weeks ago we introduced the new "Append Blob" into the mix in the Windows.AzureStorage .Net client library in version 5. If you read the MSDN page on it, it states the following:
[
An append blob is comprised of blocks and is optimized for append operations. When you modify an append blob, blocks are added to the end of the blob only, via the Append Block operation. Updating or deleting of existing blocks is not supported. Unlike a block blob, an append blob does not expose its block IDs.
Each block in an append blob can be a different size, up to a maximum of 4 MB, and an append blob can include up to 50,000 blocks. The maximum size of an append blob is therefore slightly more than 195 GB (4 MB X 50,000 blocks).
]
So what does that mean to us as developers, what can we do with this new found awesomeness called an AppendBlob??? Let's say you had a blob that was a text file and you were writing events to it (logging). When you wanted to add something to that blob you had to download it, read it into a stream or something, add to it, and then upload it to overwrite the old blob. With AppendBlob all this work is now done for you and there is even some other checking and functionality added as well. Let's write some code and take a look.
// Using statements for our libraries.
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
using System;
using System.Collections.Generic;
using System.Configuration;
namespace AzureAppendBlobSample
{
class Program
{
static void Main(string[] args)
{
// Let's set up our connection for the account and store the name and key in app.config.
string accName = ConfigurationManager.AppSettings["StrAccName"];
string accKey = ConfigurationManager.AppSettings["StrAccKey"];
// Implement the accout, set true for https for SSL.
StorageCredentials creds = new StorageCredentials(accName, accKey);
CloudStorageAccount strAcc = new CloudStorageAccount(creds, true);
CloudBlobClient blobClient = strAcc.CreateCloudBlobClient();
//Setup our container we are going to use and create it.
CloudBlobContainer container = blobClient.GetContainerReference("logs");
container.CreateIfNotExistsAsync();
// Build my typical log file name.
DateTime date = DateTime.Today;
DateTime dateLogEntry = DateTime.Now;
// This creates a reference to the append blob we are going to use.
CloudAppendBlob appBlob = container.GetAppendBlobReference(
string.Format("{0}{1}", date.ToString("yyyyMMdd"), ".log"));
// Now we are going to check if todays file exists and if it doesn't we create it.
if (!appBlob.Exists())
{
appBlob.CreateOrReplace();
}
// Add the entry to our log.
appBlob.AppendText(
string.Format(
"{0} | Error: Something went wrong and we had to write to the log!!!\r\n",
dateLogEntry.ToString("o")));
// Now lets pull down our file and see what it looks like.
Console.WriteLine(appBlob.DownloadText());
// Hold the console open.
Console.WriteLine("Press any key to continue");
Console.ReadKey();
}
}
}
And here is the output of running the above code on the 6th run. As you can see there are entries time stamped over about 1 minute duration. This is great for logging of many kinds and tons of other applications to be able to just update the blob just like you would a text file on disk. However, if I were logging many events per/sec per/min or I wanted to search quick and easy, then I would look at using something else like Table storage instead (blog coming soon).
Keys to remember about an append blob:
You cannot edit, modify, change, update (whatever word you want to use) the existing blocks of the blob. You can ONLY append to the end of the blob.
Max entries to an append blob "from my testing", are about 5-6 per second. I have tested this several times with a max entries loop (50,0000) to a single append blob. The best ways to handle this limitation would be to queue up the messages and then append them to the blob in chunks instead of singular appends.
With that said, an append blob has a max of 50,000 blocks at a max of 4mb each.
Here are some of the append methods:
And last but not least, here is the referencing documentation:
CloudStorageAccount Class
StorageCredentials Class
CloudBlobClient Class
CloudBlobContainer Class
CloudAppendBlob Class
"Coming Soon" but the constructors should look like:
| Name | Description | |
|---|---|---|
 |
CloudAppendBlob(Uri) | Initializes a new instance of the CloudAppendBlob class using an absolute URI to the blob. |
|
CloudAppendBlob(Uri, StorageCredentials) | Initializes a new instance of the CloudAppendBlob class using an absolute URI to the blob. |
|
CloudAppendBlob(Uri, Nullable<DateTimeOffset>, StorageCredentials) | Initializes a new instance of the CloudAppendBlob class using an absolute URI to the blob. |
|
CloudAppendBlob(StorageUri, Nullable<DateTimeOffset>, StorageCredentials) | Initializes a new instance of the CloudAppendBlob class using an absolute URI to the blob. |
Happy Coding!
Comments
- Anonymous
February 06, 2017
Just what I was looking for. I'm gonna have to generate csv files from a mongodb databse specific aggregation query. I was worried that write all the lines to the the file at once would take too long. With this aproach, I can paginate the data. - Anonymous
May 12, 2017
Hi, I'm trying to append new text with separate methods calls. After the first file. creation it seems to be a lock on the blob/file that makes the addText method fail. From the Explorer I don't see any locks but still can't add new lines to the same file in the blob. Can you explain me if I'm doing something wrong?- Anonymous
May 13, 2017
@Adam - Without seeing your code and the exception that is being thrown I really cannot help much here. However, if you are doing it as I show above using the AppendText() method call then it should be working.
- Anonymous
- Anonymous
June 01, 2017
Thanks bro - Anonymous
January 18, 2018
Is there a way to Merge multiple .csv or .txt files in blob using powershell?- Anonymous
February 05, 2018
The comment has been removed
- Anonymous