The Microsoft Test Runner – innovation for the Generalist Tester

It’s a very busy phase for us at work. The entire team is busy stabilizing our products for the upcoming VS 2010 Beta release. This is exciting – the team is really proud of the new Test and Lab Management products that we have been designing and building for months, and can’t wait to get the Beta bits into your hands. I spent a significant amount of my time last week taking our bits for a test drive, and have been waiting to find the time to write to you more about them. It’s Saturday and I am just back from a round of golf. The weather here in Hyderabad, India is extremely hot already (95 degrees – and we are still in February) I am ensconced in my study now, and have the next post ready in the series of “re-thinking software testing”. I hope you enjoy learning about our new tools, and look forward to test driving them soon yourself!

In the last post, I talked about Camano, our new test case management solution. Once the test plans are in place, test execution is the next focus area for testers. In today’s post I will introduce MTR – the Microsoft Test Runner.

In an earlier post I had mentioned that our research shows that 70% of the testing done in the industry is manual testing done generalist testers. As we talk to the generalist testers though, we find that they today are frustrated by two aspects of their job:

1. They spend a lot of time in non-core tasks e.g. filing a detailed bug, reproducing the bug, reporting test status etc.

2. There is a lot of repetitiveness in manual testing e.g. they waste a lot of time navigating to a particular point in the application they are testing, and repeating same test with different test data and configurations.

MTR (Microsoft Test Runner) was designed to solve these types of problems. Below is a screen shot of MTR, and I then talk to about its key features.

1. Help manual testers by showing detailed steps & logging results: When one or more tests are launched from Camano’s testing activity center, they are loaded in the Test runner. You can look at the test case with its steps. The detailed steps provide information about what you are supposed to do next, any expected results and attachments etc. You can mark the results of the steps as passed/failed. You can also add any comments. When you are done with the testing, you can publish the results to the Test Case Management (TCM) server. Reports can be generated from the TCM server for any level of information about the tests. You don’t need to separately generate reports of what you have done.

2. Create actionable bugs: The problem of no-repro is a big issue between developers & testers. Testers spend enormous amount of time finding and reporting a bug, but many of the bugs are rejected by developers because the information in the bug report is not comprehensive enough, or they can’t reproduce the bugs on their systems. MTR goes a long way in solving this problem. While using Test runner, if you see an issue, you can, with one click of a button, get the screenshot of the area on the application and attach it to the step. To file a bug, simply click on the create-a-bug icon it populates the bug form. The bug form automatically includes all the information needed to reproduce the bug including what steps have been tested; what exact UI actions were performed on the application (yes each and every UI action), the video of the testing being done; detailed system information about the machine used to test; the comments & screenshots added during test etc. (Of course you control what to record in test settings). You may just type the title of the bug and fire away! The screen below shows how a part of the bug form looks like.

3. Fast forward for manual testing: When the developer fixes the bug , you will need to verify the fix. You may need to go through a lot of steps to reach the point where you want to pay more attention. With the fast forward feature, you can ask Test Runner to navigate the application to the right spot quickly. How does it do it? While testing the first time, the action recording engine in Test Runner captures your UI actions and can use these actions to fast forward quickly to the right spot. You can skip from any point in your test and then test manually and then skip again & so on. The two screen shots below show the tester being prompted to select the steps to fast forward, and the system playing back those actions to get the tester to specific test she wants to focus on.

4. Data driving tests: As mentioned earlier, it is a laborious process to test the same test case with different data sets. But it needs to be done to look for bugs that may come with different data (for example, try wrong data, no data …). Fast forward functionality of Test Runner not only works with the actions you performed, but is also intelligent enough to differentiate between normal UI actions and data entry. When you test the same test case with different data set, fast forward functionality will pick up the right data sets to be entered in the app while navigating. This helps you focus on the app and find bugs rather than typing all that data.

5. Shared steps: Many test cases share same steps to test various functionalities. Test Runner allows you to share these steps in sharable units. That is, you can create shared steps and then use then in other test cases. Writing manual tests just become easier. If application changes and you need to update the steps, just update it in the shared steps and all test cases benefit from the change. Similarly, if you had recorded the steps (for fast forward functionality), the recording is also shared among test cases. So if the application changes (which does so often) its impact can be contained to just one shared component.

Can you believe we are still talking about manual testing?

I hope you like what you have seen so far – I’d love to hear from you, so please do leave a comment with your opinions, comments, and questions. 

In a subsequent post I look under the covers of the MTR with you and talk about the innovative way it works. I will also discuss about its extensible architecture. But before I do that, I will introduce our support for automation in the next post. Talk to you soon!

Comments

• Anonymous
  February 27, 2009
  PingBack from http://www.clickandsolve.com/?p=16042
• Anonymous
  March 02, 2009
  Amit,As much as you are anxious to deliver the new beta bits to us, I am very anxious to see the progress on these new product areas which have a great deal of potential :)It is pretty cold yet in Wisconsin so we could try to ship some of our "coolness" to you :)Bob Hanson
• Anonymous
  April 27, 2009
  Amit... its definitely not 95 degrees :) it must be 95 Fahrenheit, good post by the way!
• Anonymous
  May 19, 2009
  I am very excited proud and proud to introduce the new testing features you’ll find in dev10. In addition
• Anonymous
  January 26, 2010
  The comment has been removed
• Anonymous
  December 09, 2013
  Poor Article: Where are the references? For example, what is the link to Microsoft Test Runner?