Binary Encoding, Part 1

This series on the .Net Binary Encoding protocol is going to be similar to the earlier series I did on .Net Message Framing. The two are also somewhat related as they’re used together frequently and the messaging framing protocol has direct knowledge of some options for binary encodings. The two are also used apart regularly though.

I’ll first go over some background of what the binary encoding protocol does and why we created it. Then, I’ll have a few parts covering the details of the protocol. The binary encoding protocol series is a bit shorter than the one on message framing because it has fewer concepts, but there are still a number of challenging sections and I’ll have a bit more background to get through.

One of the originally intended properties of XML was to have a human-readable and self-describing format. A substitute for human-readable might also be to allow for programs that consume data without using the originally intended decoding program or even perhaps long after all of the software originally written for the data has been lost. It’s debatable how well XML achieves those properties in practice, but it’s certainly true that well-formatted XML is quite a lot more readable than many forms of unstructured data or data whose content is obscured by its encoding.

In some cases though, an XML document is exchanged, processed, and then thrown away with no intention of ever being read by a person or preserved for the future. In these cases, the typical text-based encoding of XML becomes more of a liability than a benefit as the space and processing power required to encode the document greatly exceeds that of traditional binary or record-oriented protocols.

There’s no requirement that an XML document have a text-based encoding though, and a variety of binary XML protocols exist as a compromise between the two extremes. No one binary XML protocol appears to be winning in the market and there does not appear to be a convergence of the proposed options over the last ten years of discussion (the usually recognized standards bodies have each selected their own format).

Partially, this market fragmentation exists because it’s hard to design a binary XML protocol that optimizes all of the large variety of needs for such a protocol. For example, a bandwidth-constrained application might require the smallest encoded document size while a mobile device application might require low power processing. While the mobile application might also appreciate a small document size, the computational power required to compress a document increases greatly as you approach higher levels of compression.

We invented our own binary XML protocol with the intention of servicing a variety of needs in WCF. I doubt that everyone or even most people would want this to be the only binary XML protocol used but I do think some will find it useful even outside of WCF. Next time I’ll go over the basic idea of the protocol and what our needs were.

Comments

• Anonymous
  September 21, 2009
  Nicholas, many thanks for taking the time and effort to publish this series. It is of immense value to those of us work with this stuff day and day out.

• Anonymous
  September 22, 2009
  Thanks Peter.  The series has turned out to be more popular than I expected so I'm going to go back and add an extra part or two to cover the binary format more thoroughly.