Test Execution Switches
The following switches apply during test execution. Note that some switches are for test code generation (TCG) only, some for on-the-fly (OTF) testing only, and some can be used during either but may have different impacts.
Test Execution Switches – TCG and OTF
These switches are valid in both TCG and OTF. Note that the TestClassBase switch has different behavior in TCG and OTF.
| Switch | Possible Values (Default Value Bold) | Description |
|---|---|---|
LogProbes |
none, string |
A comma-separated list of probe names. For each probe name in the list, the corresponding probe value of each state will be inserted into logging statements in the generated test code. Probe names may be short (local scope) method/property names or fully qualified names. The word "none" here is a keyword. |
ProceedControlTimeout |
100, number |
Time, in milliseconds, the test engine waits for observations before proceeding with a control action, if control actions are enabled. |
QuiescenceTimeout |
30000, number |
Time, in milliseconds, until the implementation is assumed to be unresponsive at which point the test case will be marked as Failed. |
TestClassBase |
vs, string |
This switch behaves differently in TCG and OTF. Class from which the generated test class or on-the-fly experiment will derive. TCG: When the value is "vs", the VsTestClassBase class in the Microsoft.SpecExplorer.Runtime.Testing namespace will be used as the base class, and all test-related attributes are VSTT test framework attributes unless users set other attribute switches to the desired value. Unlike this switch's operation under OTF, user-defined class bases need not be specified using a fully qualified name, although it is recommended that a fully qualified name be used. If a user-defined test class base is used, the test suite project must contain a reference to the assembly that contains the custom test class. OTF: When the value is "vs", the VsTestClassBase class in the Microsoft.SpecExplorer.Runtime.Testing namespace will be used. All test attributes for offline testing are ignored. If a user-defined class base is used, it must be specified using a fully qualified name. If a user-defined test class base is used, the modeling project must contain a reference to the assembly that contains the custom test class. |
Test Execution Switches – TCG only
These switches are ignored during OTF testing.
| Switch | Possible Values (Default Value Bolded) | Description |
|---|---|---|
ClassCleanupAttribute |
Microsoft.VisualStudio.TestTools.UnitTesting.ClassCleanupAttribute, string |
Attribute used to annotate generated test code. This attribute is applied to the generated test class cleanup method. A fully qualified class attribute name must be provided. |
ClassInitializeAttribute |
Microsoft.VisualStudio.TestTools.UnitTesting.ClassInitializeAttribute, string |
Attribute used to annotate generated test code. This attribute is applied to the generated test class initialize method. A fully qualified class attribute name must be provided. |
ReRuns |
1, number |
Number of times a testing experiment is repeated. This number is relevant for on-the-fly testing or for testing of non-deterministic implementations. |
SuppressGeneratedTestLogging |
true, false |
Whether to suppress the insertion of logging statements in generated test code. |
TestAssemblyAttribute |
none, string |
Attribute used to annotate the test code assembly's namespace declaration. |
TestClassAttribute |
Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute, string |
Attribute used to annotate program elements in generated test code. A fully qualified class attribute name must be provided. |
TestCleanupAttribute |
Microsoft.VisualStudio.TestTools.UnitTesting.TestCleanupAttribute, string |
Attribute used to annotate program elements in generated test code. A fully qualified class attribute name must be provided. |
TestInitializeAttribute |
Microsoft.VisualStudio.TestTools.UnitTesting.TestInitializeAttribute, string |
Attribute used to annotate the generated test initialization method. A fully qualified class attribute name must be provided. |
TestMethodAttribute |
Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute, string |
Attribute that will be used to annotate each generated test method. This switch overrides the TestIDAttribute, TestIDPropertyName, and TestDescriptionAttribute switches. A fully qualified class attribute name must be provided. |
Test Execution Switches – OTF only
These switches are ignored during TCG testing.
| Switch | Possible Values (Default Value Bolded) | Description |
|---|---|---|
OnTheFlyAllowEndingAtEventState |
true, false |
Controls whether Spec Explorer may end an OTF test in a state where one or more following events are expected. Setting this switch to true will allow tests to end at an accepting state without waiting for outgoing event steps. |
OnTheFlyExperimentInconclusiveException |
Microsoft.VisualStudio.TestTools.UnitTesting.AssertInconclusiveException, string |
Defines the exception type that a user-defined OTF test framework may throw to indicate an inconclusive test result. |
OnTheFlyMinimumPathDepth |
none, number |
Defines the attempted minimum path length before a nonterminal accepting state will be eligible to be considered as a candidate to terminate an experiment. Prior terminal accepting states will be allowed to end the experiment successfully. |
OnTheFlyMinimumRequirementCount |
none, number |
Defines the required minimum number of requirements covered by all successful experiments in one OTF test run, before it will be eligible to terminate. The value of "none" means that there is no minimum requirement count. |
OnTheFlyReplayStrategy |
ApproximateReplay, Exact |
Specifies the strategy to generate new experiments by OTF, when using a previous exploration result as guidance. ApproximateReplay indicates a heuristic to attempt to get close to the original with no guarantee of exactly reproducing the original experiment. Exact means to try to exactly match the original result. |
OnTheFlySaveExperimentTrace |
All, none, string |
Specifies whether to save an experiment's exploration result in a .seexpl file. This is required for the experiment to be eligible for replay, or to be viewable by the Spec Explorer Exploration Viewer. All saves the exploration result for all experiments regardless of outcome. The word none, which is a keyword, saves no exploration results. Any comma-separated combination of one or more of Passed, Failed, or Inconclusive will save the exploration result for the corresponding type of test result. |
OnTheFlySaveState |
true, false |
Defines whether to save state content to experiment's exploration result (.seexpl) during on-the-fly testing. Exploration result without state content can also be used for replay. |