Bottom-up SOA is harmful and should be discouraged

It's independence day week in the USA, so it is time for me to declare a little independence. 

There are three schools of thought around "how to build an Enterprise Service Oriented Architecture."  They are:

1. Top down - central group decides everything and the dev teams adopt them.
2. Bottom up - central group provides a directory and dev teams make whatever services they want.  Dev teams go to the directory to find services they can use.
3. Middle out - central group provides key elements of the interface, including numbering schemes, message exchange patterns, standard communication mechanisms, and monitoring infrastructure, and encourages the dev teams to use it to build services that can be shared.

Top down works only in some organizations.  I imagine smaller IT shops (with 300 or fewer total employees) and shops with a well established heirarchy or very strong central leadership can leverage this one.  (I'm conjecturing, of course, having not worked in SOA in that setting.  It's an educated guess).  For everyone else, it is slow and difficult to follow the Top Down path. 

Bottom up is an answer to top down seen as "the wisdom of crowds" or "economics in IT."  The idea being that many services can be developed without much central control, as long as they are in a directory.  Everyone would look in the directory to find the services they need and adopt the best or least expensive, and we'd get SOA adoption.  This one has the advantage of scale.  The larger the organization, the argument goes, the better the case for bottom up. 

Middle out is the newest method developed by folks who prefer a blended approach.  I won't go into a lot of details here.

The point I want to make today: Top down is slow, but functional.  You can end up with a SOA that is good for the enterprise.  Middle out has the same advantages and is a bit quicker. 

Bottom up is actively harmful to the enterprise as a whole and should be discouraged at all costs.

Of course, a lot of the literature in the SOA field says "build the catalog first," effectively supporting the bottom-up approach.  And that is a mistake.

As I've said before, I am not paid to solve a particular system problem.  I'm paid to look out for the needs of the enterprise, to create a "gravity" towards the center to counterbalance the natural tendency of IT to develop only for the individual needs, even when it works against the strategic goals of the enterprise.

Bottom up is a mess.  Let's look at the logical conclusion of bottom-up for a minute and compare what results we get with some very common enterprise strategic goals.

Let's say our goals look like this.  This is a very simple set.

1. Share a common concept of customer, order, sale, shipment, and inventory so that we can get a straight answer at the end of the day about the operational and financial health of the business.
2. Couple loosly so that we can keep the costs of maintenance as low as possible and hopefully, over time, divert some of those expensive maintenance resources over to developing new programs. 
3. Build your applications so that other folks can easily integrate with them so that we can add new capabilities to the IT ecosystem quickly. 

On first blush, SOA supports all three. It is clear that you can get common data, loose coupling, and ready integration from SOA, so a lot of folks have cried "SOA is the answer!" and then stopped.  They didn't pick the path to take, so the debate rages on: top-down, bottom-up, or middle-out.

The problem with bottom up is that you end up with systemic effects, not seen by individual contributors, that work against these same strategies.

There are three ways that bottom-up SOA can go in your organization.  One, it is ignored (no one decides to participate).  Two, it is wildly adopted with some folks contributing and others consuming, and Three, it is adopted only be people who have services they want to advertise, but no one consumes them out of fear of taking a dependency.  (There is a common word for both One and Three: "failure".)  Let's look at Two: some contributors and some adopters.

What are the folks contributing?  Well if we are creating a system, but we are not paid to think about the enterprise, then the system we are creating may, but probably won't, deliver capabilities that the enterprise needs.  The service will be 'local-specific'.  That means that the service will serve the needs of the paying customer (a business leader) but not the enterprise as a whole.

So the services that are in the catalog, most of them anyway, are not really 'enterprise ready.'  That means that when others consume them, they are using identifiers that are not enterprise-ready, or they are using data in a manner that may or may not be usable in the data warehouse. 

Remember that consuming a service is only demanded by a business process.  If it weren't for the need to change processes, we would never need flexibility in the first place.  If we do 'bottom-up SOA' without considering the business processes, that doesn't mean they aren't there.  It means we are ignoring them.  It is in some folks best interest that we don't look.  Consuming a service "automates and hardens" a business process, making it more difficult to change.  What if we are hardening really bad process?  Better, what if, in different parts of the company, we are hardening different processes? 

In the past, I had a run-in with very strong-willed and opinionated leaders, both with IT and the business, who strongly defended their absolute right to "do things their own way."  Sometimes that's healthy, but in the cases where it is not, it can become a shouting match.  Bottom-up SOA avoids the fight because if the service is attached in the "right way" you use it, but otherwise, you write your own.  Fight avoided.  However, when a business leader wants to take a basic, core, fundamental process, and go their own way, that is not a fight you should avoid. 

According to some fairly compelling research and literation in the field of business process automation, the most successful companies have automated their core business processes, which provides less flexibility, but gives them far greater agility when it comes to taking on new challenges.  The core stuff, the stuff they do every day, is routine, standard, and well thought out. It is a stable foundation, allowing the company to stretch in a dozen different ways. 

"Bottom up" is simply a mechanism to allow those strong-willed leaders to create multiple infrastructures in the company "because they want to" without any consideration for creating a common and stable foundation for the business.

So lets look at a bottom-up success story three years later.  A number of infrastructures have emerged.  You have one based on Division A's CRM system.  You have another that feeds Division B's ERP system.  You have a third in Division C to surround their mainframe Operations management system, and a fourth in Division D focused on their home-grown web-order management infrastructure.

You have four legacy systems.  Integrating them will require a slow, painful, expensive infrastructure, in the middle, with huge amounts of semantics and rules needed to translate the data from one legacy infrastructure to another.  That central system will be necessary to support Business Intelligence and Financial Reporting (so you cannot avoid it) and it will limit the ability of the company to respond to the needs of the marketplace.

You got SOA, but at a price. 

Of our three principles above:

1. We have our common understanding of customer, but it is so far away from the systems that create that understanding that it is useless for generating real value. 
2. We have our loose coupling, within each ecosystem, but each one is at war with one another, vieing for resources and investment.  The money we saved by developing the individual infrastructures has moved to the central integration hubs and BI.  No win there. 
3. We've got easy integration, as long as the system that needs to consume the data has the same semantic understanding of the data as the system that produces it.  Otherwise, consumer and producer cannot work together, and the cost of forcing an integration is high.

We will have won the battle, but lost the war.

This is not a scenario that I would wish on anyone (except perhaps my direct competition).  To be honest, I think my competition is smart enough to see this for themselves.

Bottom-up SOA may work if your measurement is "get services up an running" (or it may fail).  If it works, it produces a monster that is not worth producing.  That is not success either.  Measuring SOA success by the number of services, or encouraging the creation of services without any form of central understanding of what the services should contain, is doomed.

Avoid Bottom-up.  It is a pox on your house.

Comments

• Anonymous
  July 02, 2007
  Bottom-up SOA - Another Great Read

• Anonymous
  July 02, 2007
  Great article, Nick. Sometimes there is nothing to choose. In some (or many) organizations you just have to deal with the chaos around and take the approach that is feasible. Characteristics of the environment may vary strongly from one case to another, even within one organization. These characteristics may put strong constraints on your approach, that vary from project to project. I think currently in most cases it comes to common sense, passion, charisma and craftsmanship to get the right things done. Getting the things rightly done is step 2. http://soa-eda.blogspot.com/2007/06/there-is-no-one-size-fits-all-approach.html

• Anonymous
  July 02, 2007
  Yes, I agree 100%.  Bottom-up doesn't work (I'm in that environment right now).  In fact, your fictious scenario is unfolding right in front of my eyes.  Integrating two systems by putting a third in the middle is anti-agile.  Hopefully, I will be able to twist that proposal into something else. I do think the middle-out approach needs some more attention though.  I caught a good nugget of information at the bottom of a recent MS press release. http://www.infoworld.com/article/06/10/04/HNmsesbsoa_1.html "In the real world, the success stories we see are customers taking a pragmatic, middle-out approach," he said. "The only way to truly track to the needs of the business is to take small steps along the way. The mega projects are either big science projects that never get finished or they diverge from the needs of the business." [at the bottom]

• Anonymous
  July 07, 2007
  The comment has been removed

• Anonymous
  July 07, 2007
  @JJ Thank you for your kind words.  My daughter is gradually improving, but it is day-by-day.   I'm not saying that "top down" is great.  It is usually a mess.  What I am saying is that bottom up is worse. In your description, you describe a process where you choose to invest in improvement selectively, where the investment is needed and available.  That is neither top down nor bottom up.  That is simply the reality of IT.  We don't (and really shouldn't) decide the business priorities. However, when investment begins, you have to be ready, in the early planning process, with information about the services that are needed by the enterprise, and the ones that each major investment will provide.  It requires real committment by IT management to build these things, and a real committment by the IT architects to build a SOA in the first place.  (I do NOT have a common consensus even on that basic point). Being ready is difficult.  That requires planning before there is actually a project.   If you wait until the project begins, it is too late to build for the enterprise.  Reuse of that kind is harmful, because you will end up with competing visions of the enterprise business object model. The cost of NOT planning for SOA is the cost of creating a complex EAI infrastructure where one would not be needed if you did some planning in advance. So you do BPM where it is needed, and you do services where they are funded, but you don't do them blind and you don't expect that the cost of the project will pay for the work that the enterprise needs.  It won't.  

• Anonymous
  July 08, 2007
  SOA is about agile and expanding . . . To qoute Miko Matsumura (webMethods), "Think global, Act local" Begin implemntating the SOA from a part, keeping the organisation in perspective and "building governance right into it" and "not wasting your time in governing it's implemntation" :) SOA should start right of from the easiest/most difficult places, as per the companies perspective, so that we begin reaping on the benefits that it can offer at very early stages and also it alows it to get to the real world bottlenecks sooner than what you can in a design process.

• Anonymous
  July 08, 2007
  @RohitRai What you describe is middle out, not bottom up.   By the way, I find it interesting that you give credit to Matsumura for the quote. "Think Globally, Act Locally" is a motto that was commonly used about ten years ago in the US to encourage citizens to recycle and save energy.  Applying it to SOA is not particularly quote-worthy.  

• Anonymous
  July 09, 2007
  Hi! Nick, First of all thanks, for putting me in tune with the "middle out" approach, I was wondering what that thing might be. Can you give me a few pointers to that, have been searching around the details? I feel from this perspective, it is bottom up. Just keeping the ultimate goal in view and not wandering from it :) "Think gloabally! Act Locally!", Yes, it was the motto :) But also Matsumura, did put it to use in context to SOA, and that is not a research by me, it's mentioned by Joe McKendrik, in his blog about this post :) Check it here, http://blogs.zdnet.com/service-oriented/?p=912 And I do find it relevant in expressing the view of thinking about the organisational goal (global) while carrying on your work in small form (local). Correct, me if I am wrong. Am not as experienced as you poeple in the industry :)

• Anonymous
  July 10, 2007
  Thanks for the link, Rohit.  I wasn't aware I was being so extensively quoted.  I appreciate it.   I agree with nearly every word of Joe's blog.  He is using the term Bottom-up differently than I am.   I'll blog this in a seperate post.

• Anonymous
  November 07, 2007
  A few years ago I posted a short blog entry " The SOA Elevator Speech " to try to distill SOA into talking points that you might be able to cover on one elevator ride. With that posting in mind, here's my attempt at explaining BPM as concisely as I can..