Freigeben über

Testing

  • Artikel

If you're interested in testing or would like to learn more about it, here are a couple of links:

  • MSDN recently announced Tester Center: https://msdn.com/testercenter - this is a centralized resource about testing from Microsoft. Quote:
    "The Microsoft Tester Center showcases the test discipline as an integral part of the application lifecycle, describes test roles and responsibilities, and promotes the test investments required to deliver high-quality software."
  • Our QA team has a blog about testing called CSharpening: https://blogs.msdn.com/csharpening/ Unfortunately, the content there is not being updated very frequently, because we're busy testing the Visual Studio 2008 SP1 and the first new features of Visual Studio "10". Yours truly intends to post testing-related articles there.

Some common facts about my job as a tester at Microsoft (to clarify some common misconceptions):

  • Officially my position is called SDET - Software Design Engineer in Test.
  • We develop all of our test code using most recent builds of Visual Studio "10" and C# 4.0.
  • We do almost no manual testing - almost all of our tests are automated, which means they can run on a lab machine unattended.
  • We run our tests on a variety of operating systems, languages and VS SKUs (e.g. Visual C# Express 2008 on Japanese Vista or VSTS on Windows Server 2008 with Visual Studio 2005 already installed side-by-side).
  • The Visual Studio Managed Languages Team has about 30000 functional C# and VB tests which run overnight for about 12 hours on 5 to 10 lab machines (paralleled).
  • C# tests are written in C#, VB tests are written in VB, F# tests are written in F#.
  • We have a lot of automated UI tests - where mouse moves automatically and keyboard "types" on its own - its fun to sit and watch a test creating a C# project, opening files, typing, using refactorings, navigating, etc.
  • We prefer testing on the component level, bypassing the UI.
  • Our developers write unit-tests, while we concentrate on the functional testing and end-to-end scenarios.
  • We reach more than 70% code coverage with our tests - this is a required minimum for our team.
  • We use TFS (Team Foundation Server) for source control and work item tracking.
  • Many of our tests use a new data-driven test framework where our test scripts are written in XML and are interpreted by a script-like engine.
  • Most of Visual Studio tests are out-of-process - our tests attach to a Visual Studio process and control it using Remoting.
  • We have a "bug-bash" day about once a month where the entire team just uses Visual Studio and logs bugs.
  • We also have AppWeeks, where we form groups and build applications using our product.

Please let me know if you'd like to know more. Thanks!

Comments

  • Anonymous
    May 31, 2008
    PingBack from http://www.arjansworld.com/2008/05/31/linkblog-for-may-31-2008/

  • Anonymous
    June 02, 2008
    Hi Kirill, I would be interested in how do you check for expected behaviors in the GUI testings. E.g. how do you test that the editor does the correct syntax highlighting or that the correct file opened after you selected it in the Solution Explorer. Do you use OCR or someting similar? Thx, Peter

  • Anonymous
    June 04, 2008
    hi kirill the backend and frontend team also uses unit-test? thanks hnh

  • Anonymous
    June 05, 2008
    Peter, GUI verifications are interesting. We don't do pixel-by-pixel baseline comparison, because this is too low-level - depending on a color scheme, font-size or dialog position on the screen the picture might be totally different. Instead, we use Active Accessibility and UI Automation a lot - Windows technologies that allow screen-readers and accessibility devices to enumerate windows and controls. For a given window, you have a list of all child windows that are contained in it and their names. You just search for a name, and once you've found a window, you issue windows messages to emulate keyboard and mouse. We have a pretty powerful framework with several abstraction layers that provides this functionality for us. For every UI control (like a listbox) it provides an automation peer - this way I can enumerate over all items present in a listbox and know what's the currently selected item. I can also programmatically manipulate UI controls - push buttons, clear lists, type text in textboxes, open and close windows, etc. Horacio, I suppose by the frontend and backend you mean the compiler team. C# compiler has a tremendous amount of test .cs files that is being run through the compiler to make sure it compiles them correctly. This huge set of tests is called the C# conformance tests. I'm not sure if the compiler devs use unit-tests - you better ask them yourself :) Almost all of the C# compiler devs are blogging: Ed Maurer is the dev lead on the C# compiler team: http://blogs.msdn.com/ed_maurer/ Other developers: Eric: http://blogs.msdn.com/ericlippert/ Sam: http://blogs.msdn.com/samng/ Sreekar: http://blogs.msdn.com/sreekarc/ Chris: http://blogs.msdn.com/cburrows/ Thanks, Kirill

  • Anonymous
    June 07, 2008
    Horacio, I just checked with the C# compiler team - they use unit-tests too.

  • Anonymous
    July 10, 2008
    You mentioned: Many of our tests use a new data-driven test framework where our test scripts are written in XML and are interpreted by a script-like engine. Is this available for people outside Microsoft to use?

  • Anonymous
    July 10, 2008
    Not yet. But we're definitely thinking about making this public. What are you more interested in: generic data-driven XML based test framework or its particular application to automated testing of Visual Studio?

  • Anonymous
    July 16, 2008
    Hello Kirill. I'm interested in generic data-driven XML based test framework.

  • Anonymous
    December 17, 2008
    I am interested in the generic data-driven XML based test framework itself.  I can see how that could serve in a lot of environments.  Can you go into more detail about the tools you use for the UI-automation with Active Accessibility? Thanks!

  • Anonymous
    December 26, 2008
    Thanks everyone for your interest. I've asked my management and it turns out that the test frameworks we use currently are not public yet, so I can't talk about them just now. As soon as I can tell you more, I'll post an article about this on my blog. Thanks again.