Why Agile Development Requires Agile Architecture

The dark cloud of the economic downturn has produced a silver lining within Microsoft IT: an increased emphasis on Agile development techniques.  This does not mean that MS IT is new to using Agile.  Far from it.  Agile development practices have been used in various IT groups here for nearly a decade. 

What is new is the desire to address the basic development and governance processes themselves to remove the “Bias towards Waterfall” that exists in many of them.  That bias can create a situation where someone can choose to be Agile or choose to comply with corporate policy.  That’s a devil’s choice. 

For example, we have a governance point that is used throughout all levels of the IT organization called “Baseline” where all design and specification (and estimation) is done, and signed off.  This is supposed to occur before software is delivered.  Does that favor waterfall?  You betcha.  This governance point is mentioned in one of the metrics all the way up to the CIO scorecard!  BDUF is built in to the DNA itself.

But if you look at the reasons that people give for requesting BDUF (Big Design Up Front), it goes like this: you can reduce risks by requiring that everyone understands and agrees with the requirements, features, scope, timelines, responsibilities, deployment model, etc…

Right.  Pigs fly.

I do see the intent, and it is honorable.  Reducing risk is a good thing and we want to increase the number of systems that are delivered on time and on budget with fewer defects.  But I think that Agile methods have proven that there is another way to accomplish the goal of “improved delivery.” Unfortunately, they do not fit into the same neat little box. 

Where waterfall requires 600 pages of documentation to be written up front, Agile requires that most of those pages are never written.  It is not correct that Agile methods produce design documents “as you go.”  That is nonsense.  Agile methods produce 10 pages of diagrams, and nearly no accompanying text. 

So is architecture a BDUF process? 

Some of it is.  If you follow the RM-ODP model or 4+1 views, you are going to be tempted to over-specify and over-architect.  After all, the 4+1 views present a mechanism for creating a “complete” model of a software system.  That’s great, if you want to create a complete model of a software system.  But why would you want one?  Why would it be a goal to have a complete description of a software system outside the software itself?  Would a “partial” description serve the needs more effectively?

BDUF architecture is rarely a good thing.

Agile architecture, however, is often a good thing.  Agile architecture is the act of producing enough architecture to meet the needs of the project, and nothing more, and producing it in a timely fashion, with a minimum of effort.  Agile architecture RARELY produces a detailed class model in advance. 

Agile architecture often produces a high-level component model and context model to establish the existence of components, their names, and perhaps even the division of responsibilities in the dev team itself.  (think: Scrum of Scrums).  Agile Architecture nearly never produces diagrams of technical things, like the structure of a message.  Dev tools do that, from the code itself.

Agile architects will produce models using a dev environment tool, like Visual Studio or Sparx Enterprise Architect, not a diagramming tool like Visio that cannot easily be connected with the code. 

Agile architects will use diagrams to communicate between people and to express artifacts into code where developers have real freedom to make the magic happen.

Agile architects will not only leverage patterns, but also complete reference models.  They will consume a well-built reference model, inject the components that will be developed for the application, and get the team off on the right start.  They will not “craft a new architecture” for every need, but will build-from-pre-existing-designs 80-90% of the time using frameworks that produce maintainable, secure, reliable, and easily deployed systems.

Failure to architect a system is a failure to deliver.  On the other hand, hand-crafting every system is also a failure to deliver.  The difference between the two, and the middle-way that defines success, is agile architecture.

Comments

• Anonymous
  April 27, 2009
  The comment has been removed

• Anonymous
  April 28, 2009
  I agree with you Avinash, the problem is (at least from my point of view), the burocracy behind the software development companies. In order to get accountable ones usually think (by mistake) that more formal is more productive.

• Anonymous
  April 28, 2009
  The comment has been removed

• Anonymous
  April 28, 2009
  The comment has been removed

• Anonymous
  April 29, 2009
  I agree, Pericles.  Just as your "namesake" had to be a leader, orator, and politician in order to lead the golden age of Athens to build some of the greatest architecture in history, we sometimes have to lead in our IT departments in order to build great systems.   One more reason that architecture is necessary for Agile projects: if there will be more than a few sprints, and the project will last longer than a few months to deliver, then a good architecture protects the pace of delivery.  In other words, without a good architecture, "developer debt" begins to pile up quickly and later sprints spend half of their time refactoring from earlier ones.  Good architecture avoids <some> of that later-sprint refactoring. Thx.  -N-

• Anonymous
  May 03, 2009
  So this opportunity to go Agile was a generational issue, and the layoffs facilitated its resolution?

• Anonymous
  May 04, 2009
  Hello David, I would not make that characterization.  Layoffs had nothing to do with this shift.   We've been on this road for a long time. --- Nick

• Anonymous
  May 13, 2009
  Nice reading. Thanks! Really refreshing.

• Anonymous
  May 15, 2009
  In my opinion or what I have understood about agile is mostly interpreted wrongly. Agile is a methodology and XP, SCRUM are its implementation. Agiles itself says it works better for team size of upto 12-14 people. Other thing Agile assumes the people who going to develope are not fresh engineers, if they are they are paired with experienced ones. Documentation of 600 pages is not required as Buisness or clients are not sure of what they wants.... they chages happens very fast, may be everyday requirement change, can we re-write the whole big document or update it everytime--guess how costly that would be. Agile says be adaptive in nature rather than pridictive as real world is not pridictive. Agile works well when the team who uses it understands it and follows its values-principle, you can agile the practices depending on team environments. If you ask any true Agile team if they are happy with documentation you will hear yes... cause even if you write 600Pages no one has that much time to read it and that makes it painfull for new team members.... Diagrams of 10 pages are enough if code is written properly, good design and comments. I still feel Agile can be good for big teams or mroe than 12-14 people, properly implemented it should work on 100 people team as well. Agile development doesnt require Agile Architecturewhat but if you follow its principles - values you will have Agile Architecture automatically. --Akshya

• Anonymous
  May 15, 2009
  Hello Akshya, It sounds like you recited Agile practices well.  I disagree that "Agile" is a methodology.  It is more of a philosophy and set of principles.  XP, Scrum, Crystal, etc, are methodologies. That said, if you read those methodologies, you will percieve a distinct disdain for architecture.  This is probably because architecture was used as an objection, by Waterfall enthusiasts, to Agile's adoption.   I'm making the statement that Architecture is useful, even necessary, for Agile software development, and I'm making a case that the architectural document, delivered during an Agile project, will look quite different from an architectural document developed during a Waterfall project.   But not for free.   Architecture is not included in most Agile methodologies.  I'd like to see that change.

• Anonymous
  May 16, 2009
  The comment has been removed

• Anonymous
  May 16, 2009
  Also Nick. I am still big fan of Architecture Designs but Agile and its implementations in XP and Scrum. I was comparing both couple of weeks back and found XP is little better in the way it Implements Engineering Practices while in SCRUM one can easily fall in trap while implementing Agile principles. Even with agile in last couple of project we ended up with really nice design and architecture and that's why I feel its can work but I have also seen couple of other teams who failed to achieve that and my personal guess was the attitude in team which was lacking in other. Hopefully we will be able to agile the Rational approach in XP.

• Anonymous
  May 24, 2009
  On choosing architecture patterns, can you recommend any materials on how to categorize different styles of architecture? I know the P&P team started doing it, but wasn't sure if you are aware of any other significant efforts?

• Anonymous
  May 26, 2009
  Hello "Colin Jack" If anything, there are too many possible categorizations of architecture styles, leaving us, paradoxically, with too few well-thought-out taxonomies on which to build.   This is typical of human endeavor in general: we can call a particular building by any of the following monikers, and all could potentially be correct: house, post-modern, shared-common-area style, multi-storey structure, desert-climate structure, energy-efficient structure. Certainly you can view software from many viewpoints as well.  You can look at a particular concept of a stack, and then decide if the software "fits" within a particular part of the stack.  (This is how the P&P model approaches this problem).   You can also look at things like patterns of information flow, service orientation, security orientation, coding language, hosting environment, etc, as mechanisms to differentiate different "styles" of software architecture.  I'm not sure that any one taxonomy is 'better' than another.  They are simply mechanisms to navigate from "all possible designs" to "the design I want to use." If anything the internet has taught us, the notion of "search" is often not served by a taxonomy.  In the early days of internet search, we used taxonomies (like Alta Vista) to find web sites.  Now, we use full content search services (like Kumo.com and Google), and web.2.0 has given us tag clouds, RSS, and twitter.   Perhaps the best way to "find" the most appropriate architectural style, to suit a problem at hand, is through a tag cloud or a full-text search?  You tell me.   I'm not sure that a taxonomy will solve this problem.