Playtesting
Last week we held another play test of missions, so I thought I would share some thoughts about play testing.
As team members we try to squeeze time into our day to play with Flight Simulator X, especially areas that we don't directly work on. This way we can have a better understanding of where we are failing, and have a chance to fix it. It also allows us to be inspired by the many things that go really well.
Of course we have our beta program where real end users agree to keep secrets and file bugs and suggestions in exchange for early play with FSX. This program hasn't yet kicked off, so all of the great feedback we get from the beta group is coming up.
We also take advantage of Flight Simulator enthusiasts who work at Microsoft from around the world and have them try specific things in the product and give us feedback (employees whom are not on the Flight Simulator team). On a weekly basis we have several employees come into one of our play testing labs and fly missions or particpate in multi-player sessions. It's fun, they enjoy it, and we get a lot of great feedback we use to make the product better. Most of these users are true enthusiasts, the kind of users that build full scale cockpits in their basement or garage (they are "hard-core" users). I've heard many of them admit they thought they would hate missions (or at least certain missions), but when they flew them, they actually really liked them. I can also say that many of the missions they have flown have challenged these advanced users and tapped skills they haven't yet fully developed. I think that's pretty cool as it gives them something new to strive for.
Lastly, and some would say most importantly, we bring in real users that essentially come in off the street and fly for a couple of hours or even for a full day and answer questions about what they thought of the product. It was one of these tests that we held last week.
We brought in complete novices, these are folks that have never used flight simulator before, or had very little experience with it. We had them fly three missions, all of which we consider to be beginner skill level. It's always kind of amazing to watch people play and it's always very painful when they fail, and at least one always does fail to do the simplest things. We are not allowed to interfere or answer questions during a test as that could skew the data (afterall in real-life we aren't in their living room waiting to answer questions), so we can only watch and try to telepathically send them the answers they seek. Sometimes I just have to turn away as it's just too frustrating to watch. Most existing Flight Simulator users take it for granted how to start the engine, maintain level flight, and understand what a flap is and what it's used for. Not so with novice users... But in reality, with this particular test only a small fraction of our participants actually had such frustrations and we were amazed at how many of them actually completed the missions we had them fly.
It gives me good feelings to know that even complete novices can be successful without us having to camp out in their living room to answer questions or teach them how to fly.
Comments
- Anonymous
March 01, 2006
I'm curioes what the build op aof the missions will be. Is it like "fly from A to B withing an X timelimit" or land there with this kind of weather. I know that you can't say much about it, just wondering what it could be. - Anonymous
March 01, 2006
Dennis, there is a lot of variety, so it would be difficult to say it's one kind of thing or another (even if I could right now). In general, it's a bit more involved than your examples. - Anonymous
March 01, 2006
So... how do we get on the beta team :P - Anonymous
March 01, 2006
Oh and as for the missions, if I can put in a "wish list" item for them, it would be for a mission editor, to let us create our own missions and easily share them with others, with a graphical interface and no need to learn various coding languages. - Anonymous
March 02, 2006
The comment has been removed - Anonymous
March 02, 2006
thanks for the heads up about the beta info.
I send Hall9000 an email :) - Anonymous
March 02, 2006
Thank you for the contact info.