Working with Permuations and Combinations

Something tests end up doing a lot of is working with permutations and combinations. For those of you who didn't have the benefit of a good statistics class in college (or some gambling experience after) doing the math can be a hassle. Generating all the possibilities can also be a bit of a puzzle (it's like counting in a mixed base number system sometimes).

I have found myself needing to generate lists of permutations (list1 x list2 x list3 in all combinations) and combinations (all the ways you can get dealt 5 cards from a 52 card deck) once in a while. While doing API testing I realized it would be handy to code up some generic generators that I could re-use anytime I wanted.

<www.codeplex.com/permutationTools> has the libraries I created for this purpose.

The math to calculate the number of permutations and combinations is built in. The logic to step through the lists and get a minimal set, or a full permutation list is encapsulate inside with an easy "foreach" handle.

Lately I have been using these to automatically generate thousands of test cases for methods I want to give a solid workout to. This kind of testing can be daunting to create by hand. Just a handful of parameters can put you into the thousands of test cases. You can create a central method that accepts various input and drive it at run time with the tools, or you can turn the process inside out and create a program to write tests.

I do the later because it fits into my test harness and reporting system better. If I run 10,000 tests and one permutation fails I want the report to say 99.99% pass. If you run all the permutations as one test you get 100% fail. That's not really an indication of the product quality.

You have to be careful not to write impenetrable test code when using techniques like this. Make sure your BVT and Basic acceptance test cases are hand written and clear before you generate 10,000 cases that rely on tricky logic.