Partilhar via

Betas and Bedmates

  • Artigo

(Yawn)

It's Saturday morning and I'm sitting in the TV room watching my 2-year-old play with her puzzles while a cartoon show drones in the background. It seems like a perfect time to blog a bit about a recent thread on AvSim that starts off speculating whether third-party developers are 'in bed' with Microsoft by virtue of them participating in our beta program. I was wondering about that myself so after waking up this morning I rolled over and asked PMDG's Lefteris Kalamaras what he thought...

Okay. Sorry for that imagery. (Remember, you can destroy all the pictures except for those in your head. <g>) Without directly addressing any of the conspiracy theorists' concerns I thought you might like to know how the beta program works and what we (the Flight Simulator team) get out of it. (Much of this information was also prevented at the AvSim conference a few years back.)

Let's start with the intent and dispel one seemingly persistent myth, that the only testing done on the product is by the beta testers. In fact, the vast majority of testing, regardless of how you measure it (except perhaps for the number of testers), is done by teams internal to Microsoft. External beta testers may be the choice of some third-party developers simply because of resource issues but for Microsoft the testing starts here. Internal testing ensures a certain level of comprehensiveness that beta testing can't guarantee. Our dedicated test team has the time to plan a series of tests that cover all new and existing functionality. Plus we have shared resources for specialized testing such as configuration compatibility.

A way to restate this is that internal testing is structured. What beta testing is really useful for is unstructured testing. A good example of this is compatibility testing for video drivers. Our internal configuration testing lab usually tests video hardware using two varieties of drivers: reference (typically the driver version that shipped with the hardware) and latest (the version available at the time of the test). Often, though, interim versions have bugs that only show up when some beta tester in the field has it on his rig.

It should be obvious then that the beta group needs to be quite diverse. After all, if it were only made up of people who keep their drivers up to date we'd never find the types of bugs I just mentioned. Speaking specifically about the FS betas, we pull from two distinct populations. The first is a pool of testers maintained by Microsoft Game Studios and used on other games. This population is representative of the broad user base that buys games. The second population is a database we maintain of FS users and developers. This population tends to delve deeper into the product and given its depth is useful for finding issues that a general gaming population would miss. Looping back to the original thread, it's important to include third-party developers because they use and stress the product in a way unique among both general gamers and FS users. In the end we're looking for a population of beta testers that represent the greater population of FS consumers.

I'll close this post by saying how proud we are of our beta testers. When we compare the effectiveness of FS betas versus other game betas the FS betas consistent come out on top. How do we measure this? One way is to look at the percentage of issues reported by beta testers that get translated into actual bugs we track and (try to) fix. FS betas typically have a rate double that of other games. So, keep up the good work!

I hope that was interesting. Feel free to post a comment and tell me what else you'd like to know.

Comments

  • Anonymous
    November 05, 2005
    what else you'd like to know.???

    - tell us how far are you with fs 10?

    best
    Martin

  • Anonymous
    November 05, 2005
    I think Mike meant regarding Beta testing, Martin, so don't look for an answer on that.

    Great blog post, Mike. Draws a great picture of the whole testing scheme of things. I mean, testing the whole world has to be a huge challenge, internally or externally. :-)

    Owen

  • Anonymous
    November 05, 2005
    Yawn, and it's Sunday morning in Western Australia and I have just gone through my daily ritual of checking out the MSFS Developer Blogs.
    I must say that it's great that you guys are so visible. Being an avid simmer since the early Artwick days, and interacting closely with the Fly and Pro Pilot teams, I dreamt that one day we would be able to do the same with the MSFS team.
    Anyhow, do you take Beta Testers from outside of the US, and if so how does one get a gig?
    Cheers and all the best,
    Chris Porter
    Western Australia

  • Anonymous
    November 06, 2005
    How do we get on the FS Users beta list :P (email is davidvoogd@rogers.com ;) )

    I was wondering when one of you guys would see this, found the whole thing a bit amusing. BTW many of the betas for the third party addons are quite structured.

    One question - do you give developers (freeware and/or commercial) advanced copies of the sim so they can get a headstart on addons and compatibility patches?

  • Anonymous
    November 07, 2005
    So, I just bumped onto this post today ;-).

    "Honey, how do you want your eggs today?"...

    Oh jeez... If my wife found out about this thread, she'd have us both by the unnameds. :-)

  • Anonymous
    November 07, 2005
    Just seeing if you were paying attention...

  • Anonymous
    May 26, 2009
    PingBack from http://castironbakeware.info/story.php?title=the-little-wheel-goes-in-back-betas-and-bedmates

  • Anonymous
    June 09, 2009
    PingBack from http://insomniacuresite.info/story.php?id=11128