Are we ready to prove the “Architecture hypothesis?”

Science progresses on the willingness to take widely held beliefs and test them.  It is one thing to say that a medicine will “cure all that ails ye” but it is another thing altogether to prove that a particular medicine will have a particular effect on health.  Proof is expensive, but science does not march forward without it.

For quite a while, our team has been working diligently to increase the use of architectural models (and architectural thinking) among our IT units.  There are thousands of employees engaged in developing software in MSIT, and it is not always easy to reach a wide audience and make a compelling case.  Our arguments have been based on appeals to logic (clearly, it works), metaphor (it works in other areas of engineering) and anecdotal evidence (it worked on project Foo and see the benefits they got). 

But at the end of the day, we have not been using science.  We never ran a true scientific study to demonstrate that the use of a particular architectural practice improved the outcome of software engineering in a specific and measurable way, compared to a project that did not use architecture.

I’d like to see us, as an industry, get to the point where we can perform a scientific experiment on the efficacy of using software architecture practices to improve software development outcomes.

We’d need a protocol for the experiment, and controlled variables, to reproduce the effects of using architecture.  We’d need a specific definition of what it means to “use architecture” and the specific practices that we believe will have an impact on outcomes.  We’d need a clear way to measure the outcome of the experiment so that, when repeated, anywhere in the world, the experiment could produce comparable results. 

And here’s the rub: I’d like to see it be something that YOU can help with.  The community of architects and developers that care about things like SOA, Architecture, and Planning… we are the people who can perform the experiment in 1,000 settings around the world.   We are the ones that can prove, or disprove, the hypothesis of software architecture.

If anyone has experience with this kind of experiment, I’d love to hear from you, just to see if this idea is too difficult for the community to do.  Personally, I’m not sure where to start.  If you have ideas, please reach out to me.

Comments

• Anonymous
  May 13, 2009
  PingBack from http://www.anith.com/?p=37455

• Anonymous
  May 13, 2009
  The comment has been removed

• Anonymous
  May 13, 2009
  Hello Robert, I hope all is well with you in NZ. I'm not sure that your experience, while instructive, is an example of a scientific experiment. Did you create a controlled experiment where two projects were launched, one with the use of architectural practices and the other without the use of architectural practices,  but with other variables controlled? Without a direct comparison of at least two situations where the variables are well understood and controlled, we have anecdotal evidence and not a scientific outcome. Did you create such an experiment?

• Anonymous
  May 14, 2009
  Not sure how you would launch two projects with the only variable being whether architectural practices were employed (or not).  One significant issue being that people are involved and they would be different between these two projects. Second, it would be quite an effort simply to identify all the variables to control unless the proejct were exceedingly simple. My "guess" is that a more reasonable approach would be multivariate analysis, rather than strict scientific method where only one aspect at a time is allowed to vary.

• Anonymous
  May 14, 2009
  I'm commenting on the design of your experiment, rather than putting the NZ experience as a similar experiment.  The point is a shared architecture hasn't got off the ground because a lack of practically applicable artifacts.

• Anonymous
  May 14, 2009
  What a surprise to see someone's trying to make a science of what I thought  is just a bureaucratic tool. Good, but, what medicine and health have to do with it? I'm a doctor and my experiences with IT technology are disastrous. Also, recently I found out that sofisticated IT program used in banks doesn't recognize a person, but only a number (of the account). It is possible to misuse that option in a very many ways. When we come to a point where human existence would be a mouse click away, we could then turn of the light and go to sleep for ever. Now, Nick Malik ask your self who you really are and what are you really up to, because IT is a PROJECTION OF IT'S CREATOR, his mirror, so if you are a thief you'll create a mischievous program and EA and SOA which would favor your sick goals...With all the respects to your person, I'm adressing to IT community and yes, your effort to make IT a science is worth attention and might help but also might not.. Who can tell?

• Anonymous
  May 15, 2009
  Hello Natalija, Your message is certainly one of the most provacative responses I've had.  I used an expression common in the USA and associated with false doctors.  (We call them quacks, or snake-oil salesmen, or more recently, suppliment manufacturers). These are people who make claims about the health effects of a medicine with no evidence that they work. Sounds like IT?  It is.  All the time, we have claims that adding a practice, or removing a practice, will produce positive results.  Neither side produces evidence. Your profession, medicine, is the opposite of mine.  There is a great deal of deference to science in your profession.  I'd like to see a bit of that scientific approach rub off onto my profession as well. I am sorry to hear that your experience with IT has not been good.  If you had been a person who wanted to buy a house in 1400 in Europe, you would have been buying a house from a craftsman, with little or no architecture involved.  Some of those houses remain today, but most do not.  What we cannot see, through the distant lens, is how different must have been the experience of each person in their own house.   Today, software engineering is similar to home construction in 1400 A.D. I believe that we need to make the leap forward that other engineering branches have made, and I believe that part of that leap is the embrace, as a profession, of science.   What am I really up to?  I am trying to fix a profession that creates software that you cannot stand to use, that you find to be an affront to your own humanity.  You, and the millions of professionals like you, in all walks of life, are our customers.   In business, customers are the king.  If we don't listen to customers, our business fails.  IT has, all too often, blundered around trying to understand what it means to listen to customers.   Science and Engineering can help.  We are a craft-hall trade.  I believe we can do better.  I believe that if we improve who we are, as a profession, we can build software that humanizes, that supports artistic endeavor, that encourages our own connections with one another.  As long as we are blundering around, we cannot. That is what I am up to.   I appreciate your support.

• Anonymous
  May 18, 2009
  Hi Nick, Away from your discussion with Natalija, I wanted to throw another point in the discussion. From my point of view one step towards the readiness to prove the "Architecture hypothesis" are architectural patterns when you define them as general and reusable solutions. Of course they are depending on context but with reusability you get quite near to our desired goal. Dont know if you are aware of this but that link seems to be very useful in that context: http://eampc-wiki.systemcartography.info Thanks again for this good discussion!

• Anonymous
  May 18, 2009
  Hello L, Thank you for the link to your site.  The site looks, on the surface, to be very valuable and I will be digging in as deeply as I can.   --- N

• Anonymous
  May 18, 2009
  The comment has been removed