Designed for the Cloud–it is all about State!
I just got back from a proof of concept (POC) review. A POC is what we call the investigation stage where a developer or architect has a great idea and wants to prove it out in code; the review is the chance to share what was learned with peers. Needless to say we’ve had a lot of Azure POC’s in the last year or so! Everyone has a great idea for using Azure to reduce cost or derive some other benefit.
I keep seeing two reoccurring themes. The first is that most POC’s are great ideas, capable of providing real business value. The second is that most are poorly architected from a cloud computing perspective!
The poor architecture slips past even our most experienced architects and developers. Why?
Simple, cloud computing is a paradigm shift and there is a lack of distributed system design knowledge. Cloud designs must address the effect of latency on data consistency and too often the architecture suffers by simply applying existing relational database knowledge.
I confess that I’ve presented a poorly architected Azure POC. I’m learning though and here is what I’ve figured out.
State management is a critical aspect of distributed system design. The cloud requires partition tolerance so according to Brewer’s CAP theorem you tradeoff either consistency or availability. Latency is driving Brewer’s CAP theorem. Latency is the result of physical laws, such as the speed of light, which we aren’t going to circumvent without a major change in mankind’s understanding of physics. Simply put, your state management must account for latency.
Relational databases favor consistency and hence provide reduced availability. Distributed transactions, for example, only succeed if all systems are available and respond. The system’s overall availability is a combination of each individual system’s availability; the combination is less available than the least available individual system. We usually only consider an outage as impacting a system’s availability but the network latency between servers has the same effect, it just completes before your transaction times out.
The distributed transaction example is easy to understand but the same thing applies to transactions within a single database; transactions are reducing data availability. Ever had to add “WITH (NOLOCK)” to your queries? If so you’ve experienced the tradeoff between consistency and availability. The tradeoff’s impact on data quality is easier to mitigate in a single database due to how little latency exists on a single server.
Key-value stores, such as Azure Storage, and other NOSQL stores are popular for cloud applications because they favor availability, but in doing so give up ACID transactions. After gaining an understanding of the effect of latency on distributed systems it is easy to see why the much requested Azure Storage transaction and multiple index features are challenging to implement.
Many smart people over in the SQL Azure and Azure Storage teams are working hard to simplify the issue but the reality is that good state management involves tradeoffs that are application dependent. Doug Terry, from Microsoft Research, explains it nicely using a Baseball analogy. Essentially there is a spectrum of consistency choices including: Strong Consistency, Eventual Consistency, Consistent Prefix, Bounded Inconsistency, Monotonic Reads, Read Your Writes, etc. If the application were baseball then the scorekeeper, umpire, sportswriter, and radio reporter all have different data consistency requirements to accomplish their jobs. Just think how an umpire needs Strong Consistency while the sportswriter can accept Eventual Consistency (BASE) as long as consistency is reached by the time he/she starts writing the article, i.e. Bounded Inconsistency.
Lessons learned by Azure first adopters, and their advice, is to use SQL Azure if its abilities satisfy your application’s needs. With SQL Azure you’ll enjoy the strong consistency of transactions and efficient query options of multiple indexes that you’re used to. Switch to Azure Storage when your application’s needs exceed SQL Azure’s capabilities; just understand how important your state management design becomes.
There is another approach though which offers you more state management control called a Replicated State Machine, also known as a Fault-tolerant State Machine. The approach hits a sweet spot in the consistency versus availability tradeoff and is designed for cloud computing. I suspect the approach will become very popular as more developers learn distributed system design. The paper titled “Replication Management using the State Machine Approach” gives a good introduction in the first 9 pages then becomes harder to understand.
You’ll notice the paper makes many references to Leslie Lamport, from Microsoft Research. He is the inventor of the Paxos consensus algorithm and several Paxos variations. He first described the Paxos algorithm in his paper “The Part-Time Parliament” then again in “Paxos Made Simple”. To get a sense of what you can do with Paxos take a look at Microsoft Research’s Farsite project and Service Migration and Replication Technique (SMART).
You can tell from this research that this is the way to manage state in large, self-healing cloud services which scale horizontally without single points of failure. You can also tell that developing a replicated state machine from scratch is exceptionally complex. Luckily several teams at Microsoft have already developed them; such implementations are at the heart of our fabric controllers, Azure Storage, SQL Azure, AppFabric, etc.
If you want your code to directly leverage this form of state management, develop for the AppFabric Container. From the overview page these are the AppFabric Container features you utilize:
- Scale-out and High Availability
The AppFabric Container provides scale-out by allowing application components to be cloned and automatically distributed; for stateful components, the container provides scale-out and high availability using partitioning and replication mechanisms. The AppFabric Container shares the partitioning and replication mechanisms of SQL Azure.
…
- State Management
- The AppFabric Container provides data and persistence management for application components hosted in the container.
You can get started at:
The programming model is surprisingly simple to use, especially the attribute based syntax, when you consider all that happens behind the scenes.
Anyhow, give the AppFabric team a warm round of applause for productizing this useful Microsoft research. I suspect you’ll really enjoy the simplified, architecturally sound, state management AppFabric surface as they build out their roadmap.