Jaa

TESTING WITH TUX: MULTI-THREADED TESTS

  • Artikkeli

Posted by: Vadim Yushprakh

After giving a talk about Tux at MEDC ’06 in Las Vegas, a lot of people asked me about multi-threaded testing scenarios. The basic problem is this: a tester has coded up a test function, which tests a particular component, such as a driver. In practice, this driver can be simultaneously accessed by many threads/processes, and it must work correctly (no crashes, deadlocks, etc). Wouldn’t it be convenient to use the existing test code to exercise the multi-threaded functionality of the driver? Could this be done without much extra coding? It’s actually a great idea, and it’s quite straightforward to implement! In this post I will describe how to make a Tux test case multi-threaded, how to figure out just how many threads you need, and finally, how to differentiate between instances of the same test case running in multiple threads.

Most of the functionality required to accomplish multi-threaded testing is already built into Tux. Let’s look at the following code. This is a default out-of-the-box Tux test function, generated by the Platform Builder Tux Wizard:

 

TESTPROCAPI TestProc(UINT uMsg,

                     TPPARAM tpParam,

         LPFUNCTION_TABLE_ENTRY lpFTE)

{

      if(uMsg != TMP_EXECUTE)

      {

            return TPR_NOT_HANDLED;

      }

// ... Test Code ...

}

The first thing we do is check the uMsg parameter. If it is anything other than TMP_EXECUTE, we end the test and return TPR_NOT_HANDLED. Let’s take a look at what else the value of uMsg could be, and how we could take advantage of it.

During a test run, Tux will call each test function twice. When Tux is ready to execute a test case, it will call the test function with the uMsg parameter set to TPM_EXECUTE. This lets your test code know that it’s time to run. However, Tux will call the test function once prior to execution, with the uMsg parameter set to TPM_QUERY_THREAD_COUNT. In the code snippet above, we will ignore anything other than TPM_EXECUTE. However, we could easily modify the test function to look like this:

 

TESTPROCAPI TestProc(UINT uMsg,

                     TPPARAM tpParam,

                     LPFUNCTION_TABLE_ENTRY lpFTE)

{

    if(uMsg == TPM_QUERY_THREAD_COUNT)

    {

  ((LPTPS_QUERY_THREAD_COUNT)tpParam)->dwThreadCount = 5;

        return SPR_HANDLED;

    }

    // ... Test Code ...

}

 

In the new code, instead of ignoring TPM_QUERY_THREAD_COUNT, we actually let Tux know how many threads this test function should run in (in this case, number is hard-coded to 5.). After querying the thread count, Tux will spawn 5 threads, each running our test function. The default tread count for a test case is 0, and Tux will run the test case in the same thread as the main Tux program. If the thread count is 1 or higher, Tux will spawn the specified number of threads to run the tests.

So far so good, but how do we figure how many threads we actually want? The previous example simply hard coded the value to 5, but what value actually makes sense? The answer depends on what we are testing. In certain situations we many not care about the precise number of threads, as long as that number if sufficient to provide a good stress test. In these situations, we could hardcode the value in advance. In other situations, we may want to calculate the number of threads based on the test environment. For example, the filesystem driver test (fsdtst.dll) tabulates the number of FAT-formatted partitions on the system, and returns that value as the number of threads for each of the test cases. This allows us to test every FAT partition in one test pass. The serial driver test (serdrvbvt.dll) uses a similar approach. It first calculates the number of COM ports available in the system, and then uses this as the number of test threads.

There is only once piece of the puzzle remaining. How can the test code figure out which thread it’s running in? If we have N instances of some component to test, how can we tell each thread to test a specific instance 1 … N? Let us again modify the test function from the previous example :

 

TESTPROCAPI TestProc(UINT uMsg,

                     TPPARAM tpParam,

                     LPFUNCTION_TABLE_ENTRY lpFTE)

{

    DWORD dwThreadIndex = 0;

    if(uMsg == TPM_QUERY_THREAD_COUNT)

    {

        ((LPTPS_QUERY_THREAD_COUNT)tpParam)->dwThreadCount = 5;

        return SPR_HANDLED;

    }

    // Grab the thread index of this instance

    dwThreadIndex = ((TPS_EXECUTE *)tpParam)->dwThreadNumber;

    // ... Test Code ...

}

 

Tux will set up the dwThreadNumber parameter for each thread. Since our requested thread count was 5, the value of dwThreadNumber will range from 1 to 5. This is really all the information we need for the test to take the appropriate actions, and the rest is only bookkeeping. For example the file system driver test (fsdtst.dll) builds up an internal table of all the testable partitions. The size of this table is precisely the number of threads that fsdtst requests from Tux. Each thread then uses it’s dwThreadNumber to index into the table of partitions. The information in the table tells the test case which file system path to run in. That way each thread runs in its own partition, and does not interfere with any of the other threads.

There is a couple of caveats I should mention. The dwThreadNumber will range between 1 and N, where N is the dwThreadCount that your code requested from Tux. If you use this value to index into an array, be sure to subtract 1 to make it 0-based. The other caveat is synchronization. Since all the test threads can access the same global data structures in your Tux test DLLs, you need to ensure that the test code either does not modify any shared data, or that any such modifications are synchronized via your favorite locking mechanism.

Tux Links:

https://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcepb40/html/ctcontuxclient.asp

Comments