Oracles are hard...

In a previous post, I mentioned that when writing automated tests, the grey area between pass and fail can be confusing. A point I didn't mention is that often, just determining pass or fail can be hard.

Take the Win32 API CreateFile, for example. The CreateFile function in the Windows API creates a new file, or opens an existing file. If it succeeds, the function returns a handle (a unique integer value) to the file, and if the function fails, it returns an error code. You could test this function in a trivial manner by checking the return value to determine the test status.

TEST_RESULT TestCreateFile(void)
{
HANDLE hFile = CreateFile(...)
if (hFile == INVALID_HANDLE_VALUE)
{
return TEST_FAIL;
}
else
{
return TEST_PASS;
}
}

This “test” really only determines if the CreateFile function returns a value. A significant amount of additional testing is necessary to determine if the function actually worked to determine an accurate test result. A tester may create an oracle (or verification function) to aid in determining the test status.

TEST_RESULT TestCreateFile(void)
{
TEST_RESULT tr = TEST_FAIL;
HANDLE hFile = CreateFile(...)
if (IsValidFile(hFile, ...) == TRUE)
{
tr = TEST_PASS;
}
return tr;
}
BOOL IsValidFile(hFile, ...)
{
/* ORACLE:
check handle value for INVALID_HANDLE_VALUE,
determine if the file exists on disk,
confirm that the attributes assigned to the
file are correct,If file is writable, confirm that it
can be written to, and do any other applicable
verification.
Return true if the file appears valid, otherwise,
return false
*/
}

The difficulty with oracles is accurately predicting the result of the operations they are verifying. Accurate oracles require extensive knowledge of the functionality under test and clear documentation of the intent of the functionality. At a minimum, they must verify success, but they also must verify a variety of environment and program changes that occur in parallel, or as side effects of testing functionality.

Comments

• Anonymous
  November 23, 2007
  PingBack from http://msdnrss.thecoderblogs.com/2007/11/23/oracles-are-hard/
• Anonymous
  November 26, 2007
  Good post Alan ...Few point to note about oracles.I belive that u define the term in the same way as I do "Principle or mechanism to identify a problem".  So an oracle is a like a looking glass (or a microscope), using which results (outcomes) of a test can be observed to identify a problem.When thinking about oracles, we typically tend to think in terms of "deterministic" oracles - one that gives only two possible values "problem" or "no problem".  Consider a mechnical cylender-piston system where you can quickly check whether a newly manufactured peice is of correct dimension by matching or attempting to fit a piston in a "master" cylender. Deterministic oracles are like that.In current problem of file creation, a deterministic oracle would inspect the value of file handle and can identify whether file got created or not (plain venila check) and also it a binary decision "1" or "0".More than accuracy, in a oracle, I would be looking at range of situations it can handle and range of assessment that it can provide.With respect to testing, also in general, "variety" or "breadth" often appears to be valuable than "accuracy" and "precision".you might want to check following article from Doug Hoffmanon for detailed discussion oracles (if you have not already done) ...http://www.softwarequalitymethods.com/Papers/OracleTax.pdfAlternatively you can write a seperate post on desirable characterstics of an "useful" oracle.Shrini