Applications ARE the leaky abstractions

Recently, a message called my attention to a classic article from Joel on Software on the Law of Leaky Abstractions.  It's a fun article and I recommend it heartily.  In this article, Joel argues that most techologies are designed to abstract something more complex under them, but nearly all abstractions are "leaky" in that they fail to completely hide the complexity underneath. 

Therefore, you can teach a software developer student about the "higher level" stuff, but sooner or later, they will need to know more about the underlying technology to cope with some unexpected or misunderstood effect caused by a leaky abstraction.

As a result, we cannot forget the underlying technology and "move up" to the new abstraction every time a new library or language appears.  We have to continuously cram more knowledge into our brains.  This makes all software development increasingly complex. 

This little reminder of Joel's classic article came from a developer who complained that software can never be truly disposable, and can never be free, because software is the first, last, and only place where you can document the business process.  Every place else, including process diagrams and documents, are leaky abstractions.

I believe that notion is backwards.  I believe that software is a leaky abstraction of the business process.

Software, to our end users, is there to solve a problem.  The problem is caused by the business needed to implement a business process.  Software automates a portion of that business process.  In doing so, software must abstract the process details away from the users, hiding complexity.  But if the process is complex, or if the software is not able to abstract the process well, it is very likely that the complexity of the business process will "leak through" the abstraction.

In other words, using code as documentation of a business process is a leaky abstraction. 

This is just one of the reasons why I do not believe that code is a good place to document business processes.

1. As explained above, software is a leaky abstraction of the business process, which nearly always starts with, involves, and ends with human beings accomplishing a business task. 
    
2. Code is not accessable to most of the people who actually need the process to change to meet the needs of the business.  Especially Object Oriented code, where large portions of the functional bits are hidden in base classes or where a lot of code is spent dealing with Object Relational Impedence.  Therefore, the cost of changing a business process increases dramatically due to the need for specialized skills to understand the existing automation.
    
3. Most software modeling technologies focus their strengths on the static and structural elements of software (class diagrams and deployment diagrams) while the business folks want to deal with dynamic diagrams (state diagrams and transition diagrams).  As a result, we've refined a language that the business doesn't speak.  When you reverse engineer a diagram from code, you usually only get the unhelpful (to the business) static diagrams. 
    
4. Code is an indirect representation of the business process steps.  A software developer, or team of developers, designed a technical solution to a business problem.  There were probably many ways to perform the particular task in technology, yet they chose one.  This narrowing is fine when going from general (business process) to specific (implementation) but when going back to general, as we would need to do when using code to document a business process, we would inherently add value to the choices already made.  This is flawed thinking that constrains business flexibility needlessly.

You see, my notion of inexpensive software is not just to make it easy on developers.  My notion is to create an abstraction that is so good that it provides over 90% of all business problems a "minimally leaky" abstraction.  Of course, developers (not just product teams within MS, but IT developers in the field) will need to create this abstraction for our business users.

I believe those abstractions exist in the services spanning layers predicted by Middle Out SOA Architecture.

Of course, there will always be the need for "very deep" software development to expand the boundaries of technology or to create altogether new abstractions for the business to use.  We don't put developers out of work.  On the contrary, we allow them to lift themselves out of the details and into the language that the business speaks: business process.

Software is the leaky abstraction.  It makes poor documentation for a business process.

Comments

• Anonymous
  July 12, 2007
  Show me a business process not documented in the code and I'll show you an out of date, innacurate process documentation. Regarding #2, I hear this a lot from analysts and other "non-coders" who don't want to exert the effort to look at code to see documentation. Code isn't something magical that is impossible to access. It's usually just text files. And normally there's some tool like javadoc that goes through and makes the comments real pretty and linky. If you're an analyst or project manager and you can't look into a codebase to view comments then I don't want you on my team. What would you do if someone said "I can't look at that documentation, it's spread across 10 word files. Too complex, I'm not a MS Office Certified Word Operator." That's what I would do if someone said they couldn't look to a .java file to see what roles are involved in a service operation. Your other points are valid, and documenting the process in the code (which usually means building the documentation from the code itself) isn't perfect but it's better than the other means that get outdated or never even created properly. I think you're better off being realistic, adding some specific comment tags to your code and auto-generating documentation. This is the best way to get accurate documentation.

• Anonymous
  July 12, 2007
  @Brian "Show me a business process not documented in the code and I'll show you an out of date, innacurate process documentation." And I will show you an organization at Level 1 of process maturity. "If you're an analyst or project manager and you can't look into a codebase to view comments then I don't want you on my team." The feeling is mutual, I'm sure.  Anyone unwilling to document the business process, and maintain it, and drive application requirements FROM the process (and not vice versa) has no place on my team either. Your attitude is common, but indicative of poor practices.  You have learned the wrong way to develop business processes and you need to actually work in a progressive environment, where processes are considered valuable, before you will get the opportunity to learn a better way.   In the days when Object Oriented code was a new practice, many folks expressed real reticence about learning OO coding.  "There's nothing I can do in C++ that I can't do in C!"  There are a few of those folks left, and they don't like their jobs.   Get some training. Your career will benefit.

• Anonymous
  July 13, 2007
  While I agree that automated forward and backward engineering of processes, diagrams to code and back, is a lofty goal I think that we need an evolutionary path to get there. The problems I run into when getting handed some kind of process description is that it often leaves out critical details, specifically those around failure and partial failure conditions. Furthermore, I also experience variable resolution within each process - some parts are highly specific, others merely hand-waving. The methodology of implementing such processes involves requirements excavation, analysis, design, prototypes, etc. We iterate that a bunch of times. This is a two-way learning process; both business and IT come out of it with a deeper understanding of the what the process SHOULD be. Even if we could get a full and coherent mapping from the code to artifacts the business could understand, it would not mitigate the real problem. That of the request to change the process coming with similar fidelity as the original process description. The solution, if any exists, is a maturation of the whole organization - people first. Then we enter into the oh-so-problematic domain of organizational learning and memory. There we see our best efforts crumble under promotions, reorgs, and quality people leaving. Woe be us enterprise architects doomed to our promethean fate (http://en.wikipedia.org/wiki/Prometheus). But that's why we get the "big bucks" :)

• Anonymous
  July 13, 2007
  Nick, You speak from my heart. I was wishing to write the exact thing out of myself so long ago. Now you have done it:) However using SOA to provide that 90% real-life BP mapping sounds a little weird to me but time will tell you are right or not. I agree: at this time there is no other concise and commonly accepted technology (or maybe methodology???) out there than using SOA like (de-)composition. Thanks! Peter

• Anonymous
  July 14, 2007
  The comment has been removed

• Anonymous
  July 14, 2007
  Nice article & I love your first response to Brian.   Technologists need to remember they aren't patronised as artists.  They are hired as artisans to service their client's business needs. And on the topic of BP documetation; don't you see a determined effort by more mature organisations to develop business process competencies these days?  Things are getting better, gradually. Cheers

• Anonymous
  July 15, 2007
  @Craig, "Things are getting better, gradually." Yes.  They are.  I didn't drive this bandwagon, but I'm jumping on.  It's the right thing to do, and now is the right time to do it.   I do see major movements in standards, tools, and mindshare.  It will take some time to get the business and technology schools to teach the right material, but movement is happening.  I am hopeful.

• Anonymous
  July 18, 2007
  NickMalik,  I have been on the darkside of development for too long.  We could not trust external documentation, it was always out of date (And this is at a big insurance company).  We trusted the code.  PDP-11 Days.  But I have seen the light...  So I would like your opinion on the following...  Where do you see Using the Testing System to encapsulate the business requirements?  We are in the process of developing new software, and have decided to embrace a Test Driven Development attitude on top of an rather Agile Process.  BUT Only after very thorough analysis.  (I disagree that design should be done ad hoc, 22 years of development has taught me the best decisions are made with the best SOLID analysis).  So, our answer to the documentation delimma was that the Diagrams would be "STORED" in test modules that excericised the logic.  And as bugs are discovered they are put here and fixed, and the documentation updated.  Again, I am interested in your opinion on this... We believe it solves a couple key problems and helps to keep the developers and the business people working from the same pages.... Thanks, Kirk Out!

• Anonymous
  July 18, 2007
  The comment has been removed