The Tester's dilemma: proving you can count to a million

How would you prove to a skeptic that you can count to a million (and pronounce each of the numbers)? Proving you can count to 10 is easy since you can just rattle it off in a few seconds. This is the same sort of daily dilemma a tester faces: how do you prove something when you can't just churn every possible combo. Ultimately, you need to make some assumptions, and so you can view the real problem as "what assumptions are you willing to make".

A few approaches I've seen so far:

1. Spot checking:
   a) Have somebody count to 100 and then assume they can count to a million. The assumption here is that demonstrating basic proficiency is sufficient.
   b) Counting to 100 is quick but doesn't test the ability to hit large numbers. An alternative approach is to count in a range from (17 - 50 by ones), (1,019 - 1,102 by twelves), etc.
2. Solving a related problem. If there's a very high probability that solving problem A means you can solve problem B, and proving A is very cheap; then proving A may be the cheapest way to increase the confidence that you can solve B. 
   a) Ask somebody to pronounce several random numbers (1,768; 548,842; etc), and assume that they could fill in the gaps.
   b) Have somebody do some math problems like (457,233+3,917=?) . .
3. Decomposition. If problem B is really just a huge combination of a few small problems A1, A2, A3..., then prove you can solve all the small problems and that you can compose them. If you put great thought into decomposition, you can get probably get a small set that tests a very large area.
   a) Calculate a set of numbers that exercise every sub-word used in the words between one and a million, and then pronounce those.

Comments

• Anonymous
  November 03, 2006
  The obvious approach:  count to a million. But that'll take too long, you might respond.  So then tell the person asking you to test this that and then ask for a testable scenario that also meets their requirements. The key is to ask the customer to make the decision, not to make it yourself.  If that isn't feasible, document what you did instead and explain why it wasn't rational to perform the original request.

• Anonymous
  November 03, 2006
  It is now nearly two decades since Dijkstra pointed out that it "... is now two decades since it was pointed out that program testing may convincingly demonstrate the presence of bugs, but can never demonstrate their absence." [ http://www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/EWD1036.html] ;)

• Anonymous
  November 03, 2006
  The comment has been removed

• Anonymous
  January 30, 2008
  Anybody who says "I can write correct multi threaded code" probably should be saying "I