Microsoft Forums Reputation System - Phase 2
A couple of posts ago, I covered my vision for the first phase of a reputation system for the Microsoft forums. Now it’s time to build a case for Phase 2 (I feel a bit like I’m building a subdivision here…) Phase 1 of the reputation was focused on incenting and rewarding top answerers. Phase 2 of the reputation system aims at helping another group of key users of the forums--the question-askers.
A few months back I did an informal survey of some of the top answerers on the MSDN Forums site. One of the questions I asked was "Why do you answer questions on the forums for free?" I was expecting answers such as "respect", "advertising my consulting business", etc. What I got was completely different. People said that they learned by answering interesting questions on the forums. The best way to get more answers was really to get more interesting questions posted.
Of course, some of the questions on the MSDN Forums are, admittedly, not very interesting. Many newbie posters will post something like "Help! I am using Visual Studio and can't run my program." Listen, I don't care what your third-grade teacher said--there are such things as stupid questions. :) Questions like that clog some of the forums, keeping our top answerers from wanting to answer questions there. In addition, new questioners are the ones most likely to post off-topic. It's not their fault--they aren't familiar with the structure of our site yet, but you'll often see databinding questions in the "Visual Studio General" forum, or questions about GridView in the "Visual Basic Express" forum. This makes it very, very difficult to find a good question to answer, and it keeps our more experienced questioners with the good questions from getting as many eyes looking at their on-topic question.
What do we do? We allow questioners to build reputation. If you're known for writing really good questions, or even known for good answers, you should have more flexibility than a newcomer to the site. So, let's give people some points for question-asking activities. Here's an idea:
Question Asking Points
- 1 Point for every question asked that was answered
- 1 Point if you marked a reply to your question as an answer (and it was never unmarked)
- 1 Additional Point for every question asked that was marked as helpful
- -1 Point for every post that was marked "off-topic" and moved to another forum
- -3 Points for every post that was so off-topic that it was deleted
- -10 Points for every reported post that was approved as abusive
Answering Points
- 5 Points for every answer given
- 2 Additional points every time your given answer was marked as helpful
So, "answer count" would be replaced by a synthetic "community score" in phase 2 of the reputation system that took into account more of the behaviors on the forums that we wanted to encourage--good question asking, helpful posting, answering...and discouraged the bad behavior--off-topic posting, abusive/spam language, etc. So, what would we use this new score to do? Remember--any reputation system needs to be linked back to something "tangible"--a number on a screen doesn't mean much to me, one way or another.
Points Based Rights
- 10 - able to post in a forum without going through forum selection steps
- Newbie posters would need to answer a few questions that helped select an appropriate forum before they posted, and possibly would force new users to search before posting. For example, I would be asked what version of Visual Studio I'm using, what language (C#, C++, VB), and what I'm trying to do (writing code, designing UI, debugging, deploying, etc.) The forums system would then suggest a forum that is on topic for me to post in.
- 25 - Answer Marking Rights (for others)
- 50 - Answer Marking Rights (for everybody)
- 100 - Can Move Posts
- etc...based upon the levels in Phase 1
- And, of course...
- -20 - Nomination for banning from the system, which would go to the site administrator
I toyed around a bit with the idea of having the top users questions actually appear higher on a forum thread list than others, but I really don't like the idea of the forums becoming an exclusive club. A new user should have just as much chance of having their question looked at as a veteran user--but I think it's okay to make them jump through a few hoops while they learn the site, ensuring quality for the older users of the forums.
This whole "rewarding question askers" idea is relatively new--it really stemmed from a conversation I had with Josh last week. As a result, I'd really welcome some feedback. What do you think?
Comments
- Anonymous
March 01, 2006
I have played with the thougth if it would help anyone to know what the technical level of a question was.
It would most likely help answeres with a higher knowledge level to find posts that are more challenging to answer than questions from beginners or users that do not bother to search for the answer.
If relating to my previous comment and this blog post about phase 2 I think it will help users to contribute to the community and learn from it more easily than if they have to see all questions ranging from novice to expert level.
As an example, there is 5 levels for a question. An answerer can then search for questions of level 5 that is the expert level. This will leave the lower levels to answerers that feel that it is within their range of knowledge.
What I have not come up with an answer to yet it how to determine the level of a question. It could be influenced by the user when posting but also by other answerers to increase its difficulty level if they can not answer it. - Anonymous
March 15, 2006
What about -1 for each post that is answered by linking to to another post that asked the same question and +3 for doing the above linking. - Anonymous
March 15, 2006
Not a bad idea...from a technical point of view, that would be a hard one to measure though. We'd have to give the moderator a button that let them say that this question was a duplicate--there are many unique questions that are partially answered with links to other questions...cool idea though. - Anonymous
April 06, 2006
I think adding the "helpful / not helpful" feature definitely takes it in the right direction. If the site is to become "for the community, by the community", the moderators (except participation as a community member) should seldom be seen.
I remember from my own experience developing a visual basic application for Excel that I used usenet to ask and get my problem solved. I was impressed with both the speed and quality of answers, of course I had no way of reflecting my appreciation. But I bring this up because it raises the question of how fleeting membership to dev forums can be. If someone just comes in, asks a question, then leaves, there's little community. However, in such a situation, you may want to consider another level of reputation and that is "expertise". If you consistently answer detailed questions that require fair depth of knowledge, some kind of community recognition system of their expertise would probably be appreciated by the answerer.
However, in situations where there's a strong steady community, my favorite that I've run across is omidyar.net's system. It gets away from "number of posts" (not a fan of) and more towards a scarcity of amount of reputation that can be given out (the belief that scarcity creates value, and that value causes people to put more thought behind their actions). That coupled with livinginstyle's system of allowing users to select their own reward for being good community members.
I'm looking forward to hearing the results of your various experiments. - Anonymous
April 06, 2006
Great thoughts. I checked out your site--interesting stuff on there. The idea of reputation scarcity is an interesting one--right now we've been heading in the other direction. The hard part about it is what happens to a dedicated person who has worked really hard to establish a great reputation in a community, and then has something happen where they can't participate for a few months--maybe they had a baby or went on vacation. When they come back...have we taken reputation away from them, even if they were one of the founding members of our community? I'm just playing Devil's Advocate here though, I'm really not sure what the final answer is, but you've given me some food for thought! Thanks! - Anonymous
April 07, 2006
I'm not sure what I'm proposing wouldn't work equally well even (perhaps better) if someone goes on vacation. Depending on how it's implemented, the system could also be used to reduce someone's credibility.
For example, user A and user B.
User A has a rep score of "7.5". They answer some questions, and it goes up to "11.4". User B has "12", but they get sarcastic over a series of posts, so they get dinged down to "6.1".
The above has nothing to do with a timeline for the most part. This is what I like most about it, it gets away from a "number of posts" form of reputation. So if someone goes on vacation or disappears for a year, they can come back with the same score they left. - Anonymous
April 07, 2006
I've been writing quite a bit about reputation and auto-moderation on the MSDN Forums on this blog. ...