Parameter Generation

This section illustrates the basic concepts of parameter generation in a running example.

Note

Parameter generation is not a feature of the Cord scripting language. Parameter generation can be achieved only by using the parameter generation API functions in code in an embedded block.

Thus, in the examples that follow, all parameter generation is surrounded by "{." and ".}".

The single demonstrated action adds jobs to a model of a scheduler. Several properties control periodic running of each job: the job name, the time of day, and how frequently to run the job. Here the interest focuses on defining parameter combinations for adding these jobs. The rest of the model does not bear directly on the examples. It is similar to the at command in Windows, or the ATSvc Protocol contained in the executable Spec Explorer samples.

The following basic Cord configuration declares one abstract action, AddJob, that adds a job but omits any other actions.

config Base 
{
  action abstract static void SUT.AddJob(
    string command, int time, Frequency frequency);
}

This configuration in the sample refers to the following enumeration type.

public enum Frequency
{
  Once, Daily, Weekly
}

Because the model behavior does not affect parameter generation, it can be treated as empty.

static class ModelProgram
{
  [Rule]
  static void AddJob(string name, int time, Frequency frequency)
  {
    // Does nothing.
  }
}

Assigning Domains and Generating a Product

Parameters can be assigned with the configuration on a per-action basis. Typically, this is done by a new configuration derived from a more basic one. The following code example adds parameter values to the AddJob action from the Base configuration.

config Base { /* as above */ }

config Product : Base
{
  action abstract static void SUT.AddJob(
    string name, int time, Frequency frequency)
  where
  {.
    Condition.In(name, "@$^", "t.cmd", "t.exe");
    Condition.In(time, -1, 60, 3600);
  .};
}
// Action AddJob is synchronized, or matched, with the rule program using the
// parameter values determined in the Product configuration.
machine Product() : Product
{
  AddJob || (construct model program) 
}

In the Product configuration, in addition to the declarations inherited from the Base configuration, parameter domains for AddJob are declared in the embedded C# code following the where keyword. In the Product machine of this example, no bind construct is required. Domains come from the configuration attached to the construction of the model program.

The use of Condition.In(name, "@$^", "t.cmd", "t.exe") assigns a parameter domain to the name parameter. This can be read as the In operation associating the name parameter with the set of values that follow. This set of values is known as the domain. The Condition class in the Microsoft.Modeling namespace introduces this method and various other tests. The Combination class, also in the Microsoft.Modeling namespace, provides methods for specifying how parameters interact with one another, enabling the configuration of parameter generation.

Exploring the Product machine with these definitions yields the following graph.

Note

A switch of the graph viewer has been used to show labels of all transitions between the same states with one single transition. Because the model code for AddJob is empty, all parameter combinations lead to the same state. In more realistic models, this would be different. The transitions do not lead back to state S0 because a single step was selected with a scenario in the Product machine but with multiple choices of parameter values.

By default, Spec Explorer generates the full product space for the parameters. Because the Frequency enumeration has three values and the time enumeration has three values, there are 27 different possible parameter combinations.

Defining Interactions Between Parameters

By default, parameter generation assumes that all parameter combinations are required. Pragmatically, it may not be necessary to generate all those combinations. Looking at the example, it might be possible to reduce the combinatorial space by assuming any of the following:

• The name of the program to schedule does not matter in combination with other parameters. However, the test suite should show all values provided for it.

• The relationship between frequency and time may be relevant, so the test suite should test all combinations of those.

This is expressed by declaring that an interaction between parameter frequency and time does exist. Also declare that name interacts with itself to get all values for this parameter in at least one combination. As soon as an interaction is declared, the default of full product space is deactivated, and only the explicitly declared interactions matter. This is shown in the following code example.

config Interaction : Base
{
  action abstract static void SUT.AddJob(
    string name, int time, Frequency frequency)
  where
  {.
    Condition.In(name, "@$^", "t.cmd", "t.exe");
    Condition.In(time, -1, 60, 3600);
    Combination.Interaction(name);
    Combination.Interaction(time, frequency);
  .};
}
machine Interaction() : Interaction
{
  AddJob || (construct model program) 
}

For each declared interaction, the generation algorithm guarantees that the full Cartesian product space is generated for the parameters in the interaction.

The graph produced from the Interaction machine is shown in the following illustration.

This reduces the number of combinations from 27 to 9, the product of the possible values of time and frequency. The possible values for name can be fitted into those combinations.

Note

The tool has made some arbitrary choice as to which parameter values of name to fit into which combinations of time and frequency. The way to control this is covered in the Isolating Values and Seeding Values sections.

Specifying a Combination Strategy

You can use the Microsoft.Modeling.Combination.Strategy(Microsoft.Modeling.CombinationStrategy) method to control the default way in which Spec Explorer generates parameter combinations. The following examples illustrate how the Strategy method affects parameter generation.

The following C# code defines a state variable a and a MyAction rule that are part of a model program. The rule takes three parameters and contains two possible branches.

static int a;

[Rule]
static void MyAction(int x, int y, int z)
{
    if (x > y)
    {
        a = x + z;
    }
    else
    {
        a = y + z;
    }
}

The following Cord script declares a MyAction action and a MySlice action machine that reference the model program. The action declaration provides parameter domains for each of the action parameters and also sets the combination strategy to use for parameter generation. In this case, each parameter domain contains three values, and the configuration uses the Microsoft.Modeling.CombinationStrategy.Product strategy.

config Main 
{
    action abstract static void Adapter.MyAction(int x, int y, int z)
        where
            x in {0..2},
            y in {0..2},
            z in {0..2},
            {. Combination.Strategy(CombinationStrategy.Product); .};

    // Set other switches here.
}

machine ModelProgram() : Main where ForExploration = true
{
    construct model program
    where scope = "CombinationStrategyModel.ModelProgram"
}

machine MySlice() : Main where ForExploration = true
{
    MyAction || ModelProgram
}

When the MySlice action machine is explored, Spec Explorer produces steps for all possible combinations of parameters for the MyAction action, and the exploration graph contains 27 steps. In this case, the statement Combination.Strategy(CombinationStrategy.Product); would be equivalent to the statement Combination.Product(x, y, z);.

If the action declaration is changed to use the Microsoft.Modeling.CombinationStrategy.Pairwise strategy and the MySlice machine is explored again, Spec Explorer generates combinations for all of the parameters in pairwise combination. In other words, Spec Explorer generates enough steps to cover all possible combinations of any two parameters. The exploration graph would contain nine steps, as this is sufficient to cover the pairwise combination of three parameters, each with three possible values. In this case, the statement Combination.Strategy(CombinationStrategy.Pairwise); would be equivalent to the statement Combination.Pairwise(x, y, z);.

If the action declaration is changed to use the Microsoft.Modeling.CombinationStrategy.Coverage strategy and the MySlice machine is explored again, Spec Explorer generates only enough combinations for the parameters to ensure that all paths are covered. The exploration graph would contain two steps, as this is sufficient to cover the two paths in the associate rule.

If the action declaration is changed to use the Microsoft.Modeling.CombinationStrategy.None strategy and the MySlice machine is explored again, Spec Explorer keeps all of the parameters symbolic. This is unlike the other strategies in that all steps produced contain symbolic action arguments instead of concrete action arguments. The exploration graph would contain two steps, as the action machine could have two different symbolic states depending on the relationship of the x and y parameters. In this case, the statement Combination.Strategy(CombinationStrategy.None); would be equivalent to the statement Combination.ExcludeFromExpansion(x, y, z);.

Isolating Values

The parameter combinations generated in the preceding section have flaws under the obvious assumption that the parameter assignments name = "@$^" and time = -1 represent some invalid parameter values that cause the implementation to produce an error. For example, if the implementation code begins by detecting the invalid command name @$^, all values for the other parameters that are combined with this value are not actually tested. The implementation code only returns an error, without doing anything with the other parameters.

Spec Explorer provides a way to isolate certain values to prevent this. If a value is isolated, a single combination is produced for it, and the value is excluded from all other combinations. Values for other parameters that appear in this combination are not covered. Isolation can be declared as shown in the following code example.

config Isolated : Base
{
  action abstract static void SUT.AddJob(
    string name, int time, Frequency frequency)
  where
  {.
    Condition.In(name, "@$^", "t.cmd", "t.exe");
    Condition.In(time, -1, 60, 3600);
    Combination.Interaction(name);
    Combination.Interaction(time, frequency);
    Combination.Isolated(name == "@$^");
    Combination.Isolated(time == -1);
  .};
}
machine Isolated() : Isolated
{
  AddJob || (construct model program) 
}

The Combination.Isolated method takes a list of parameter names and associated Boolean expressions that characterizes the values to be isolated. Here both the parameter assignments name = "@$^" and time = -1 are isolated. For every isolated combination, at least one isolation declaration must be used. As many can be given as desired. An isolated combination, however, must be allowed by the domain and other constraints. Otherwise, an error is raised.

When the Isolated machine is explored, the following graph is generated.

In this graph, the first two combinations represent the isolation of name == "@$^" and time == -1, respectively. Then six combinations follow that cover the interaction between time and frequency. The time value -1 and name value "@$^" have been excluded, so there are six remaining combinations. The values for name have been fitted into those combinations. However, the invalid value "@$^" does not appear again because it is isolated.

Seeding Values

It may be desirable to ensure that a certain combination of values is preferred over another one. This is called seeding. A seed is characterized by a Boolean expression, similar to isolated values. Without seeding, the parameter generation algorithm makes arbitrary choices as long as the coverage goal defined by interactions is met.

Besides the obvious application to guarantee that parameter combinations relevant to the problem domain are tested, a typical use of seeds is to ensure that parameter combinations are produced that actually cover all paths of the model program. If the model code contains a fragment such as the following

static class ModelProgram
{
   [Rule]
   static void AddJob(string name, int time, Frequency frequency)
   {
      ...
      if (name.EndsWith(".exe") & time > 60)
      {
         ...
         Requirement.Capture(...);
      }
   }
}

the parameter combinations must actually contain values that allow the conditional to be explored. This is expressed in a configuration, as in the following code example.

config Seeded : Base
{
  action abstract static void SUT.AddJob(
    string name, int time, Frequency frequency)
  where
  {.
    Condition.In(name, "@$^", "t.cmd", "t.exe");
    Condition.In(time, -1, 60, 3600);
    Combination.Interaction(name);
    Combination.Interaction(time, frequency);
    Combination.Isolated(name == "@$^");
    Combination.Isolated(time == -1);
    Combination.Seeded(name == "t.exe", time == 3600); 
  .};
}
machine Seeded() : Seeded
{
  AddJob || (construct model program) 
}

The graph resulting from exploration of the seeded machine is shown in the following illustration.

Now one combination is included that satisfies the seed goal, without changing the number of combinations compared with the same configuration without seeds.

Pairwise Combinations

The interactions discussed in the preceding sections can be used in pairwise testing. If parameters x, y, z are specified, pairwise combination is similar to Combination.Interaction(x, y); Combination.Interaction(y, z); Combination.Interaction(x, z);. The following code example declares a pairwise configuration.

config Pairwise : Base
{
  action abstract static void SUT.AddJob(
    string name, int time, Frequency frequency)
  where
  {.
    Condition.In(name, "@$^", "t.cmd", "t.exe");
    Condition.In(time, -1, 60, 3600);
    Combination.Pairwise(name, time, frequency);
  .};
}
machine Pairwise() : Pairwise
{
  AddJob || (construct model program) 
}

This pairwise declaration inserts exactly the interactions described. There are more interactions than those present when the parameter name was not in the Isolated interaction with any other parameter. Nevertheless, the combinations generated for the Pairwise machine have the same count as the combinations produced for the Interaction machine in the preceding section, as shown in the following illustration.

The additional requested combinations can be fitted into the nine combinations required to cover the product of time and frequency. In general, fewer interaction declarations do not necessarily mean fewer combinations. In the example, pairwise combination has a better distribution of values and may thus be preferable to declaring single interactions.

N-Wise Combinations

The interactions discussed in the preceding sections can be used in n-wise testing. If parameters w, x, y, z are specified and n is set to 3 in the call to the Microsoft.Modeling.Combination.NWise(System.Int32,System.Object[]) method, n-wise testing is equivalent to Combination.Interaction(w, x, y); Combination.Interaction(w, y, z); Combination.Interaction(x, y, z);. The following code example declares an Nwise configuration and an Nwise machine.

config Base2
{
  action abstract static void SUT.AddJob(
    string command, int time, Frequency frequency, Priority priority);
}

config Nwise : Base2
{
  action abstract static void SUT.AddJob(
    string name, int time, Frequency frequency, Priority priority)  
  where
  {.
    Condition.In(name, "@$^", "t.cmd", "t.exe");
    Condition.In(time, -1, 60, 3600);
    Condition.In(priority, "High", "Low");
    Combination.NWise(3, name, time, frequency, priority);
  .};
}

machine Nwise() : Nwise
{
  AddJob || (construct model program)
}

The call to the NWise method inserts exactly the interactions described. In this example, n-wise parameter selection generates a better distribution of values and may thus be preferable to declaring single interactions.

Constraints

In addition to declaring interactions, isolation, and seeding, constraints for the parameter combinations can be declared. The following code example ensures that in all combinations, the value "t.cmd" is not used with the value Frequency.Daily.

config Constraint : Base
{
  action abstract static void SUT.AddJob(
    string name, int time, Frequency frequency)
  where
  {.
    Condition.In(name, "@$^", "t.cmd", "t.exe");
    Condition.In(time, -1, 60, 3600);
    Combination.Interaction(name);
    Combination.Interaction(time, frequency);
    Combination.Isolated(name == "@$^");
    Combination.Isolated(time == -1);
    Condition.IsFalse(name == "t.cmd" & frequency == Frequency.Daily); 
  .};
}
machine Constraint() : Base
{
  AddJob || (construct model program from Constraint) 
}

The following illustration shows the graph generated from exploring the Constraint machine.

The number of combinations is unchanged compared with the graph for the Isolated machine. However, here there is no combination in which name = "t.cmd" appears with frequency = Frequency.Daily.

In general, Spec Explorer can find the minimum number of combinations that satisfy a declared constraint, such that maximum parameter interaction coverage is guaranteed in which there is virtually no restriction on the kind of constraints. For a discussion, see Parameter Expansion Points. However, constraints may nevertheless result in excluding some combinations because every combination produced must satisfy the constraints.

See Also

Concepts

Generating Parameters for Structured Values
Parameter Expansion Points
Using Combinations in a Let Statement
Probabilistic Parameter Selection

Other Resources

Modeling Toolbox