Test Results and Conformance
Conformance tests and test results
Supporting conformance testing is Spec Explorer's primary goal. Spec Explorer supports such testing by allowing the user to create tests based on model exploration, or, in the case of on-the-fly (OTF) testing, by interacting with the model to create tests.
Each conformance test has one of three results: "Passed", "Failed", or "Inconclusive". A conformance test result depends not just on the agreement between the model, the generated or OTF test, and the responses from the system under test (SUT), but also on the test strategy specified during test generation and execution.
The following table gives a brief explanation of each possible result; a set of rules follows that show exactly how Spec Explorer determines the test result.
| Test Result | Summary Meaning |
|---|---|
Passed |
The test has reached an accepting end state in keeping with the test strategy and without finding a conformance error. |
Failed |
There are two basic ways in which a test fails: 1) The test detected a conformance error between the model and the implementation's corresponding action or lack of action; 2) The test ended in a non-accepting model end state. |
Inconclusive |
The test did not encounter a conformance error, but the test strategy was unable to determine what to use for the next model and matching SUT action. |
What is an end state during testing?
Model end states are automatically test end states. If the test reaches an accepting model end state, the test result is "Passed". If the test reaches a non-accepting end state, the test result is "Failed". A test which reaches a model end state will never return a result of "Inconclusive".
There are several cases where a test may end before a model end state is reached, usually when a specified bound is reached, or when attempting a test replay.
OTF testing in general, and the "shorttests" and "longtests" constructs, when used for test code generation or dynamic traversal, always attempt to end in accepting states. These strategies do not automatically stop at the first accepting state encountered during test execution.
Note
"shorttests" and "longtests" constructs actually create new machines (models) that can be used either to generate test code (including during dynamic traversal) or to be directly tested by OTF.
Determining a test strategy end state when the OTF "Exact" or "BestEffort" test strategies are specified is a bit more complex than the cases listed above. In general, a replay tries to do the same or almost the same thing as the initial test run.
During a replay, the same path depth as the original trace is usually used. However, if the replay attempt fails during action execution (that is, the replay is unable to transition to the next state), the replay will attempt to end the test with another action if one exists. This does not guarantee that the test ends in an accepting state.
How does Spec Explorer determine a Conformance Test Result?
"Passed". The test result is "Passed" if and only if the test ends in an accepting state. The test end, as detailed above, can occur because a model end state is reached, or the test strategy or the test bounds have been reached.
"Failed". The test result is "Failed" if:
The SUT provides an unexpected response. This includes throwing an exception instead of returning, returning the wrong value, or responding with an unexpected action or argument.
The model expected a response and the implementation does not provide one. This occurs even if the current state is an accepting state, as the test is attempting to move beyond the present state, and the SUT is not complying.
The model is in a non-accepting end state.
"Inconclusive". An "Inconclusive" means that the test cannot determine the test outcome. This can be due to:
A problem encountered in the model (e.g. an error state).
The test ends in a non-accepting non-end state. This usually occurs because of the test strategy selected, or because a bound is hit.
The test strategy cannot determine an acceptable next step. This occurs during Replay. There are two cases: 1) the replay ends on a non-accepting state, so that the next step is indeterminate, or 2) during BestEffort replay, the test strategy cannot choose between the model's next steps.
An "Inconclusive" condition is signaled to the test.
Events and test strategies
It may happen that at the end of a test, there are still unconsumed events on the event queue. These are ignored; they are considered to belong to a future part of the model that was not tested. Also, see Test Execution Switches for more details.