Freigeben über

blambert/learnings – Don’t Base64 encode PartitionKey and RowKey values in Azure Table Storage.

  • Artikel

Azure Table Storage is a “typeless” entity storage system.

If you’re an RDBMS person, and learned about data storage by reading ISBN 0-201-14192-2, Azure Table Storage is a different way of thinking about your data.

In Azure Table Storage, an entity is simply a “bag of properties”; with a certain minimum “shape” or set of required properties.

These are:

  • Timestamp.
  • PartitionKey.
  • RowKey.

Timestamp
This is not the property you’re looking for; move along.  This property is managed by Azure.  Don’t put anything into it, don’t use it for anything, and don’t infer anything from its contents.  (Think of it as though it was named “NotYourProperty”.)

PartitionKey
The first part of an entity’s primary key.  Entities are organized (partitioned for load balancing, etc.) in Azure by their PartitionKey.

RowKey
The second part of an entity’s primary key, which serves to uniquely identify an entity within a given partition.

Read https://msdn.microsoft.com/en-us/library/dd179338.aspx to more fully grok the model.

As you design your storage needs in Azure Table Storage, you will notice that PartitionKey and RowKey are string values, which can be up to 32KB in length.

But, there are limitations on what can be stored in these properties!

They can’t contain the following characters:

  • The forward slash (/) character.
  • The backslash (\) character.
  • The number sign (#) character.
  • The question mark (?) character.

So you must be careful about what data you store in your PartitionKey and RowKey properties.

One of the simple things you need to know is this:

Do not Base64 encode binary data and use it as a PartitionKey or RowKey.

A lot of developers use Base64 encoding to turn binary data into strings.  It’s fast, reversible, well understood, and easy to find in the developer tool bag.  Especially in .NET, where you can simply call:

Convert.ToBase64String
Convert.FromBase64String

(If you have never read up on what Base64 is, I suggest reading the Wikipedia Base64 entry at this point.)

The Base64 “alphabet” (see https://www.rfc-editor.org/rfc/rfc4648.txt) contains these characters:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=

And forward slash “/” cannot be used in Azure Table Storage PartitionKey or RowKey values.

So what should you do?  More on this in a later post.  At the moment, I am simply turning binary data into HEX strings.  There are smarter, more efficient approaches…

Brian

Comments

  • Anonymous
    June 16, 2009
    PingBack from http://fixmycrediteasily.info/story.php?id=16098
  • Anonymous
    February 16, 2011
    You mentioned there was a "much better way"...did you mean your posting on the binary to HEX...or something else?
  • Anonymous
    December 23, 2014
    In like 4 lines of code can write a to string for any radix. A generic parse would just be a couple lines of code too...       static string ToString(char[] charset, int value)       {           char[] values = new char[value / charset.Length + 1];           int position = values.Length;           int radix = charset.Length;           while (value != 0)           {               position--;               values[position] = charset[value % radix];               value /= radix;           }           return new string(values, position, values.Length - position);       }        static void Main(string[] args)       {           char[] hexChars = "0123456789ABCDEF".ToArray();           char[] base61 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789".ToArray();           char[] base10 = "0123456789".ToArray();           char[] base2 = "01".ToArray();           Console.WriteLine(ToString(hexChars, 0xFFFF));           Console.WriteLine(ToString(base61, 0xFFFF));           Console.WriteLine(ToString(base10, 0xFFFF));           Console.WriteLine(ToString(base2, 0xFFFF));           Console.ReadLine();       }
  • Anonymous
    March 24, 2015
    This is probably not very good advice.  You can still use Base64 if you use a different alphabet.  After working with Base64 in a web environment for any length of time, most people will probably encounter the URL alphabet (avoids needing to url-encode): en.wikipedia.org/.../Base64