Hybrid Triangulation (with Cat Food and Bananas)

Once again I'm at one of those gloriously satisfying stages in my p&p working life when I'm trying to define the structure for a new guide. We know what technologies we want to cover, how we will present the guidance, and the kind of sample that we'll provide to demonstrate the all-encompassing wonderfulness of the technologies on offer. But after two weeks of watching videos, perusing technical documents, consulting experts, and RSI from repeated spells of vicious Visioing, I'm still floundering in a cloud of Azure confusion.

The target for the project is simple enough: explore the opportunities for building hybrid applications that run across the cloud/on-premises boundary, and provide good practice guidance on implementing such applications. It obviously centers on integration between the various remotely located bits, the customers and partners you interact with, and the stuff running in your own datacenter; and there is a veritable feast of technologies available in Azure targeted directly at this scenario.

So why is it so difficult to get started? Surely we can toss a few components such as Web and Worker roles, databases, applications, and services into a virtual food mixer and pour out a nice architectural schematic that shows how all the bits fit together. I wish. Even with bendy arrows and very small text I still can't fit the result onto a single Visio page.

Obviously you need a list of the technologies you want to use. In our case, the first things going into the plastic jug are ingredients such as Azure Service Bus (with its myriad and still growing set of capabilities), Azure Connect, Virtual Network Manager, Access Control Service, Data Sync, Business Intelligence, Data Market, and Azure Cache. Then add to that a pinch of frameworks such as Enterprise Library Extensions for Azure and Stream Insight.

Yet every connection between the parts begs different questions. Where do I put the databases (cloud or on-premises) to resemble real-world scenarios but still show technologies such as Connect and Data Sync in action? Do I use Service Bus Queues or Topics and Rules to communicate between the cloud application and the suppliers? If I use ACS for authentication, when and where do I match the unique customer ID with their data in the Customers database? What's the most realistic location for the stock database, and do I replicate it to SQL Azure or just cache the minimum required content in the cloud instances? Does SQL Federation fit my scenarios, or is that a whole different kettle of fish that deserves a separate recipe book?

And, most confusing of all, how do I cope with multiple geographical locations for the Azure datacenters and the warehouse partners who fulfill the orders? Do I allow customers to place orders that will be fulfilled from any warehouse (with the associated problem of delivery costs), or do I limit them to ordering only from their local warehouse? And if I take the second option (assuming I have a warehouse partner in both the East and West US), what happens if somebody in New York wants to place an order for delivery to California?

And after you decide that, look what happens when you factor in Azure Traffic Manager. If you use it to minimize response times and protect against failures, the customer in New York might end up being routed to the California datacenter. That's fine if they want the goods delivered to California, but most likely they'll want them delivered to New York and so the order needs to go to that warehouse. Unless, of course, the New York warehouse is out of stock but they have some in the California warehouse.

Of course, the whole concept of integrating applications and services is not new. Enterprise Application Integration (EAI) is a big money-spinner for many organizations, and everybody has their own favored theory accompanied by a different architectural layer model. And don't forget BPM (Business Process Management) and BPR (Business Process Reengineering). I read a dozen different reports and guides and none of them had the same layers, or recommended the same process.

And, in reality, building a hybrid application (or adapting an existing on-premises application into a hybrid application) is not EAI, BPM, BPR, or any of the myriad other TLAs. It's a communication and workflow thing. Surely the core questions focus on how you get service calls, messages, and management functions to work securely across the boundaries, and how you manage processes that require these service calls and messages to work in the correct order, and make decisions at each step of the process. Yes you can match these questions to layers in many of the EAI models, but that doesn't really help with the overall architecture.

What went wrong with the whole design process was that we started with a list of technologies rather than a business scenario that required a solution. We went down the route of trying to design an application that used all of the technologies in our list, but used each one only once (otherwise we'd be introducing unnecessary duplication). We'd effectively taken ingredients at random from the cupboard and expected the food mixer to turn them into a palatable, attractive, and satisfying beverage. It's obviously not going to work, especially if you keep the cat food in the same cupboard as the bananas.

In the real world people start out with a problem that the technologies can help to solve, not a predefined list of technologies chosen because they have tempting names and capabilities. If you want to build a public-facing website with an on-premises management and reporting application, you wouldn't start by buying 100 copies of Microsoft Train Simulator and a refrigerator. You'd design the application based on requirements analysis and recommended architectural styles, then order the stuff in the resulting Visio schematic. Somewhere along the line the choice of technologies would be based on the application requirements, rather than the other way round.

So at the moment we're tackling the issues from all three ends at once, and hoping for some central convergence. On our mental whiteboard there's a big circle containing the list of required technologies, another containing EAI and other TLA layer models, and a third containing the possible real-world scenarios. I'm just hoping that, like a Euler diagram, there will be a tiny triangle in the middle where they overlap.

But that's enough rambling. The pains in my fingers are starting to recede, so I need to get Visioing again. I reckon I've still got some bendy arrows left that I can squeeze in somewhere...

Comments

• Anonymous
  October 05, 2011
  Great post and I agree that hybrid solution is not mixing different concepts.
• Anonymous
  October 05, 2011
  Thanks. As our investigations and scenarios absorb more and more bendy arrows I'm fast coming to the conclusion that this type of integration is more complex and challenging than most other development areas!