test case reuse (in the future)

I’ve given my ‘future of testing’ talk four times (!) this week and by far the part that generates the most questions is when I prophesize about test case reuse. Given that I answered it differently all four times (sigh), I want to use this space to clarify my own thinking and to add some specifics.

Here’s the scenario: one tester writes a set of test cases and automates them so that she can run them over and over again. They are good test cases so you decide to run them as well. However, when you do run them, you find they won’t work on your machine. Your tester friend used automation APIs that you don’t have installed on your computer and scripting libraries that you don’t have either. The problem with porting test cases across machine boundaries is that they are too specific to their environment.

In the future we will solve this problem with a concept I call environment-carrying tests (nod to Brent Jensen). Test cases of the future will be written in such a way that they will encapsulate their environment needs within the test case using virtualization. Test cases will be written within virtual capsules that embed all the necessary environmental dependencies so that the test case can run on whatever machine you need it to run on.

The scope of technological advances we need for this to happen are fairly modest. However, the Achilles heel of reuse has never been technological so much as economic. The real work required to reuse software artifacts has always been on the consumer of the reused artifact and not on its producer. What we need is an incentive for testers to write reusable test cases. So, what if we create a “Testipedia” that stored test cases and paid the contributing tester, or their organization, for contributions? What is a test case worth? A dollar? Ten dollars? More? Clearly they have value and a database full of them would have enough value that a business could be created to host the database and resell or lease test cases on an as-needed basis. The more worthy a test case, the higher its value and testers would be incentivized to contribute.

Reusable test cases will have enough intrinsic value that a market for test case converters would likely emerge so that entire libraries of tests could be provided as a service or licensed as a product.

But this is only part of the solution. Having test cases that can be run in any environment is helpful, but we still need test cases that apply to the application we want to test. As it turns out, I have an opinion on this and I’ll blog about it next.

Comments

• Anonymous
  January 18, 2009
  The comment has been removed

• Anonymous
  January 18, 2009
  Have you ever though to sell you test cases? I am pretty sure that most of you have not. This is not

• Anonymous
  January 18, 2009
  I share your optimism to a certain degree but my take on testing reuse is more guarded. Just as software/code reuse can never be made to truly work, since every software project is necessarily different and specific, so I don't see how testing, automated or manual, could ever be portable either. You can certainly generalise testing techniques, strategies, etc, depending on the type of project under test: stand-alone application, public web site, corporate intranet, etc, but the devil is in the details. Testers could take direction from a generalised set of peer-reviewed test cases but the onus would still be on them to select the most appropriate ones, put them in the right order, and work out the details in order to get a decent test strategy together. Still, food for thought...

• Anonymous
  January 19, 2009
  I just blogged my thoughts on this post here: http://blog.browsermob.com/2009/01/the-future-of-test-case-reuse-using-amazon-ec2/ Basically, I see this vision of the future as very much possible, but it depends on it being easy to snapshot virtualized state (and not all state is easy to capture) and it being fast to re-created said states when running test suites. I think the speed issue is mostly solved, thanks to EC2 and other utility cloud computing projects. But the ease-of-use issue is a long ways off, and I wonder if it can ever be solved properly. A lot of taking proper application state snapshots involves application-specific knowledge that may never be able to be generalized. And even if taking snapshots is easy, some parts will always have to be mocked out (ie: dependency on real-time stock quotes, etc). Hopefully this is a hurdle that the industry can jump, because if it can this approach could drastically change how software is built.

• Anonymous
  January 20, 2009
  How many times we can see similarities in different solutions comes from different companies? The answer

• Anonymous
  January 21, 2009
  The comment has been removed

• Anonymous
  February 03, 2009
  [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

• Anonymous
  February 05, 2009
  [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez