Share via

Pairwise Testing for Dummies

  • Article

So, imagine that you're a tester, and your job is to test a pogo stick.

Yes, a pogo stick.

So, you sit down, and brainstorm (like I did just now) a list of things that you might try to do to break the pogo stick, while still within bounds of  “normal” use. (Running over the pogo stick with a cement truck would certainly break it, but this scenario is not covered by the warranty)

Let's imagine your list of ideas looks something like this:

  • Pogo in the rain
  • Pogo off a platform (1 or 2 feet)
  • Pogo in sand
  • Pogo under varying weights (50, 100, 150 lbs)

Now that we have our list ready, let's look at our testing approach

A relatively simple approach would be to simply go down the list, and write a test case for each variation.

  1. Pogo in the rain
  2. Pogo in the sun
  3. Pogo on level ground
  4. Pogo off a 1 foot platform
  5. Pogo off a 2 foot platform
  6. Pogo on pavement
  7. Pogo in the sand
  8. Pogo with 50 lbs
  9. Pogo with 100 lbs
  10. Pogo with 150 lbs

This would certainly do a fair job of testing things, but what about testing for certain problems that appear only when certain combinations are used (Say, pogo in the rain off a 1 foot platform, or pogo into sand under 150 lbs of stress)

In order to exhaustively test every possible combination, you'll need to design a test matrix like so:

Weather Height Surface Weight
Sunny 0 ft. Pavement 50 lbs
Sunny 0 ft. Pavement 100 lbs
Sunny 0 ft. Pavement 150 lbs
Sunny 0 ft. Sand 50 lbs
Sunny 0 ft. Sand 100 lbs
Sunny 0 ft. Sand 150 lbs
Sunny 1 ft. Pavement 50 lbs
Sunny 1 ft. Pavement 100 lbs
Sunny 1 ft. Pavement 150 lbs
Sunny 1 ft. Sand 50 lbs
Sunny 1 ft. Sand 100 lbs
Sunny 1 ft. Sand 150 lbs
Sunny 2 ft. Pavement 50 lbs
Sunny 2 ft. Pavement 100 lbs
Sunny 2 ft. Pavement 150 lbs
Sunny 2 ft. Sand 50 lbs
Sunny 2 ft. Sand 100 lbs
Sunny 2 ft. Sand 150 lbs
Rainy 0 ft. Pavement 50 lbs
Rainy 0 ft. Pavement 100 lbs
Rainy 0 ft. Pavement 150 lbs
Rainy 0 ft. Sand 50 lbs
Rainy 0 ft. Sand 100 lbs
Rainy 0 ft. Sand 150 lbs
Rainy 1 ft. Pavement 50 lbs
Rainy 1 ft. Pavement 100 lbs
Rainy 1 ft. Pavement 150 lbs
Rainy 1 ft. Sand 50 lbs
Rainy 1 ft. Sand 100 lbs
Rainy 1 ft. Sand 150 lbs
Rainy 2 ft. Pavement 50 lbs
Rainy 2 ft. Pavement 100 lbs
Rainy 2 ft. Pavement 150 lbs
Rainy 2 ft. Sand 50 lbs
Rainy 2 ft. Sand 100 lbs
Rainy 2 ft. Sand 150 lbs

36 possible combinations... OK, but here's the really bad news.  In order to make sure these tests are conducted professionally, you need to hire a professional Pogo Jumper (Boss's orders). Unfortunately, the Pogo Jumper's schedule is quite full (as one would expect for a world-renowned Pogo Jumper!), so he can only stop by on Tuesdays, and it takes him all day to go through one scenario from above (with all the stretching, pogo calibration, and self affirmation exercises involved).

Hmm, this means it will take you 36 weeks to test the pogo stick. 

Not good.

Not good at all.

Here's where the notion of pair-wise testing comes in... The basic idea is that most problems (bugs) manifest themselves with combinations of 2 factors. Once you start trying every combination of 3 or 4 factors, the test matrix quickly explodes (as we can see from the table above), and you don't find as many bugs as when you were testing combinations of 2 factors. Clearly a case of the law of diminishing returns.

So, let's take certain rows from out of the table, and make certain that each factor is matched with each other factor at least once. (Rainy should be tested with 0 ft, 1 ft, 2 ft, Sand, Pavement, 50 lbs, 100 lbs, 150 lbs at least once, and the same thing goes for Sunny. Then we must verify that each height is matched with each type of surface, etc. etc.)

Here's an example of a pairwise matrix derived from the exhaustive matrix above:

Weather

Height

Surface

Weight

Sunny

0 ft.

Pavement

50 lbs

Sunny

0 ft.

Sand

150 lbs

Sunny

1 ft.

Pavement

100 lbs

Sunny

2 ft.

Pavement

100 lbs

Sunny

2 ft.

Sand

150 lbs

Rainy

0 ft.

Sand

100 lbs

Rainy

1 ft.

Pavement

50 lbs

Rainy

1 ft.

Sand

150 lbs

Rainy

2 ft.

Pavement

50 lbs

So, now we only need to rent out the professional Pogo Jumper for 9 weeks. We've just cut our testing matrix to 1/4 of its original size! And we will still catch any problems that happen with any combination of 2 factors. This really is a great way to think about problems when there are just too many combinations to test exhaustively.

The best way to create a pairwise matrix is to use a tool, although it's possible to do it manually as well.

AllPairs is a free tool to do this.

Want a better explanation? Go Here: <www.developsense.com/testing/PairwiseTesting.html>

Still curious? blogs.msdn.com/micahel/archive/2004/04/28/122702.aspx

Want even more? blogs.msdn.com/nihitk/archive/2004/05/09/128821.aspx

And for those of you that are here for the Pogo Sticks: <www.post-gazette.com/healthscience/20010528bowgohealth2.asp>

Dave

Comments