The Science of Software Testing

I was going through some old files today and came across a paper I had written February 7, 2002 shortly after becoming the director of test training at Microsoft. While a bit dated, it reflects many thoughts I maintain to this day on the role and challenges of software testing.

------------------------------------

Software testing was once an afterthought in the commercial software communities. Novice product managers and even developers looked upon software testing as an unfocused discipline that almost anyone could perform. Some popular books on the subject of software testing even included words such as art and craft in their titles, which may have led some people in the software industry to conclude that testing for quality was not a true software engineering discipline. To the contrary, effective testing is an software engineering discipline and a critical component of the sofltware development lifecycle to help improve software quality and reliability through defect identification, error prevention, observation and reporting. Essentially testing provides information for rational and analitical risk analysis.

The creativity in testing

Although some may consider testing software to be an art, its only similarity is that it involves a great deal of creativity. It could be argued that testing a large-scale computer program requires more creativity than designing that program. Glenford J. Myers backs this theory in his book The Art of Software Testing. "Testing is an extremely creative and intellectually challenging task." In fact, many people who could be designing and developing software admit that they find more aesthetic satisfaction in the creativity of testing.

A philosophical view of testing

Since the beginning of time, humans have participated in various forms of testing by questioning the designs of tools and processes. This constant re-evaluation led to improved quality and significant gains in effectiveness and efficiency. But testing based purely on trial and error is laborious and time consuming. In 1962, the philosopher Sir Karl Popper approached knowledge growth from a more scientific process. In his book Conjectures and Refutations, Popper hypothesized that science progresses by reducing falsity. Essentially, progress is achieved by formulating theories (conjecture - reasoning that involves the formation of conclusions from incomplete evidence) into testable hypotheses (refutations - any evidence that helps to establish the falsity of something).

Testing as a science

Computer software is similar to a scientific hypothesis; both are inherently fallible. The basic framework of the software debugging process is analogous to the trial and error practice that advances scientific hypotheses. Ccomputer software is simply technical conjecture. Test engineers attempt to refute the assumption of flawless software through rigorous tests mostly designed to prove that defects exist (falsification). The falsification process sharply contrasts with data-driven approaches such as induction and deduction, which attempt to justify validity based on the repetition of facts. Justificatory approaches mostly attempt to support claims through confirmation. Confirmation primarily endeavors to validate the error-free uncertainty by using specific data that demonstrates proper functionality. This approach clearly only proves that software functions correctly under certain conditions; it does not prove that the software is error free.

Establishing confidence in software

The best approach to establish confidence in software reliability is not only to demonstrate that a program works within the confines of the product specification, but also to expose potential defects in the design or functionality of the software. However, any conjecture can withstand countless hours of exhaustive testing and still contain errors. Refutation cannot guarantee flawless software because there are simply too many hypotheses to test in a reasonable amount of time. William Hetzel explains the futility of exhaustive testing and the limitations of refutation in The Complete Guide to Software Testing. He writes, "There are more test cases in a program with 70 branches than there are teaspoons of water in the Pacific Ocean." When a tester successfully refutes the software (finds a bug), he reports the defect. The developer then (hopefully) resolves the issue and assumes that the software works correctly and the refutation process repeats. Although testing can never prove that software is without errors, detecting errors highlights problems, which ultimately results in higher software reliability and quality, or at least provides a foundational perspective for risk analysis and mitigation.

What testers really do

When we ask test engineers to describe their job, most respond, "Breaking software." Perhaps this is why some people mistakenly view the role of the tester as a destructive process. However, testers do not really "break" software. Testers simply expose errors that already exist by refuting or falsifying the assumption that the software is flawless through the observed results of structured tests, logical deduction, and experiments. But, finding errors is just one aspect of software testing. Test engineers add tremendous value in the development process by validating the software, detecting errors in the software, providing developers with information to prevent bugs, presenting information to management to evaluate risk, and by acting as the advocate for the customers.

Comments

• Anonymous
  February 08, 2007
  The comment has been removed
• Anonymous
  February 10, 2007
  The comment has been removed