Error messages: diagnostic is preferable to prescriptive

The new LINQ features are going to create new failure modes for the compiler, so we're going to need to create some new error messages. The compiler development team got together the other day to discuss what makes an error message good or bad. I thought I'd share with you guys what we came up with. We believe that good error messages are:

• Polite: making the user feel like an idiot is very, very bad.
• Readable: poor grammar and tortured sentence structure is bad.
• Accurate: error messages must accurately describe the problem.
• Precise: "Something is wrong" is an accurate error message but not a very precise one!
• Diagnostic but not prescriptive: describe the problem, not the solution.

The first four are obviously goodness. That last one is a little more controversial. Surely a good error message not only tells you what is wrong but helps you fix it, no?

The issue is that deducing what is wrong with bad code is hard enough. Trying to read the user's mind and figure out what they were thinking when they wrote the bad code, and then telling them how to correctly implement that thought is not something that we feel we can do with sufficiently high accuracy in most situations.

Look at it this way: suppose we pull off a miracle and manage to produce error messages which 90% of the time tell the user the correct way to fix their code so that it does what they want it to do. That means that 10% of the time we are telling people how to write a syntactically correct program that does something different than they intended! Pushing people towards writing buggy programs that still compile is very bad, and we do not want to go there.

It's instructive to look at a few places where we violated these guidelines in earlier versions of the compiler:

Static member 'Baz' cannot be marked as override, virtual or abstract

If the user wrote static virtual, then we don't know what the heck they meant to do. Assuming that they meant to say static and that the virtual is wrong is a little presumptuous. Maybe the static is the wrong part! Also, if the user said static virtual, then why is the error message mentioning override and abstract? That's accurate but not precise. A better error message in this case would be something like

Member 'Baz' cannot be both static and virtual

Here's another place where we get it wrong, but this one is more subtle:

A params parameter must be the last parameter in a formal parameter list

This is an example of an English sentence that can be interpreted different ways depending on the context. If I said "a punctuation mark must be the last symbol of a sentence" then I mean that every sentence must end in a punctuation mark, but I do not mean that punctuation marks are only legal at the end of a sentence. If I said "a period must be the last symbol of a statement" then I mean that every statement must end in a period, and furthermore that periods are forbidden anywhere else in the statement.

You and I know that what the error message is trying to say is that if there is a params then it must go at the end. But based solely on this error message, a user would be entirely logically justified in thinking that (int i) is an illegal parameter list because it doesn't end with a params parameter. Or, under another interpretation, they'd also be logically justified in concluding that (params int[] foo, params int[] bar) is legal, because it does end with a params parameter.

The portion of the specification which the error message is attempting to draw attention to is of course "If a formal parameter list includes a parameter array then it must be the last parameter in the list. There can only be one parameter array for a given method" which is nicely unambiguous. Why not simply use this quote from the specification for the error message? That's a reasonable idea, but it sounds a little stiff and doesn't call out where the problem is. I'd prefer:

Method 'Foo' has a parameter array parameter which is not the last parameter in the formal parameter list.

This tells you what is wrong without telling you how to fix it. Since we don't know how to fix it – whether the user should be removing the params modifier, or moving it to the end, or rewriting their method from scratch – we should just report the spec violation and let them sort it out.

There are times when we do want to tell the user what to do, but only when we are highly likely to be correct. For example:

User-defined operator 'Blah' must be declared static and public.

Here we are both diagnosing the problem and prescribing a solution. If they're trying to make a user-defined operator, this is what they absolutely must do to be successful. It is very unlikely that they wanted to make a private instance function and made a private instance operator by mistake!

This illustrates another principle of good error messages that I didn't call out before: good error messages use precise terminology from the standard rather than making up new jargon. Yes "formal parameter list" and "user-defined operator" are a little bit stiff, but they are also clearly defined in the standard.

Sometimes we get the error right but the wording could be improved:

Foo: static classes cannot be used as constraints

Why are they trying to use a static class as a constraint? Who knows? How should they fix it? Beats me! The best we can do is to tell them that it hurts when they try to do that. But the wording! Good heavens! Would you ever say "Pizza: delicious foods should be eaten while they're fresh!" ??? Clearly

Static class 'Foo' cannot be used as a constraint

is much better.

Anyone have additional suggestions for what makes a good error message? Or other examples of places where we got it wrong?

Comments

• Anonymous
  July 07, 2006
  The comment has been removed

• Anonymous
  July 07, 2006
  I used to hate it when the Pascal compiler would complain about a missing semicolon and love it when the Turbo Pascal compiler would put it in for me when I forgot.
  
  The other thing is to indicate exactly where the error is occuring.

• Anonymous
  July 07, 2006
  The comment has been removed

• Anonymous
  July 07, 2006
  The ever-interesting Eric Lippert has thoughtful post on how to create good error messages. His particular focus is on the error messages generated by a compiler, but there's very little in there, save perhaps the actual examples, that wouldn't apply

• Anonymous
  July 07, 2006
  The comment has been removed

• Anonymous
  July 07, 2006
  Unfortunately, if the user does not share the sense of humour of the developer, it can come across as arrogant, flippant or unprofessional, so we try to not go there.
  
  Another characteristic of good error messages that I forgot to mention above was "easily translated into foreign languages".  We provide many localized versions of the compiler and humour is hard to localize.
  
  

• Anonymous
  July 07, 2006
  The comment has been removed

• Anonymous
  July 09, 2006
  The comment has been removed

• Anonymous
  July 09, 2006
  The comment has been removed

• Anonymous
  July 09, 2006
  I am strongly against localizing error messages.
  
  Here’s why. When one sees an error message one doesn’t understand, one can copy-and-paste it into Google or a web discussion forum. If the message is in international English, one is very likely to find a solution or at least an explanation. If it is localized, then the search is only limited to that language. Asking for an explanation of a localized error message in an international forum is just plain wrong.

• Anonymous
  July 10, 2006
  The comment has been removed

• Anonymous
  July 10, 2006
  Re: localizing error messages:  That's why every error message has a unique error number which can be looked up on MSDN for more information.  
  
  You have to look at the most common usage case.  The common case is non-english-speaking programmer gets an error, if the error is in their language then they stay productive, and if it isn't, then they're stuck.  Googling it in English is hardly going to help them!
  
  Your remark displays a considerable bias towards English -- I would say that if the forum is international then one should expect discussions in any language.  
  

• Anonymous
  July 11, 2006
  I think prescriptive messages can be useful. There have certainly been times when I've gotten an error message and had to think really hard in order to fix my problem. There have also been times when I've gotten an error message that said something like "error blah...probably due to a whatever". Sure maybe the suggested solution is wrong but it at least gives you an initial way of attacking the issue.

• Anonymous
  July 12, 2006
  The comment has been removed

• Anonymous
  July 12, 2006
  The MSDN wiki is an attempt to leverage the power of the community to annotate the documentation. Is that the kind of thing you're talking about?
  

• Anonymous
  July 12, 2006
  Re: localizing error messages: I'm strongly against localized development tools. Most developers have to know English anyway, because all the programming languages are based on English, all library functions names are in English, and most documentation is in English anyway. In addition to this, the localization team usually has to invent a lot of terms on their own when there's no common translation of it in the foreign language at the moment of localization. These newborn terms are much less informative to a programmer, compared to the International English terms.
  

• Anonymous
  October 09, 2006
  PingBack from http://hasalotofnothingtosay.wordpress.com/2006/10/10/what-makes-a-good-error-message/

• Anonymous
  October 31, 2006
  If people aren't searching by error codes, then perhaps the error message needs a more specific coding.  If the code is unique, then perhaps the people in question need to be re-educated about error messages.

• Anonymous
  December 06, 2006
  oh men! error messages should be as unclear and messy as possible; only then someone who is facing problems will look into books and understand why is something an error and how to do it correctly instead of just playing with letters in source code, trying to satisfy short error report;

• Anonymous
  December 19, 2006
  The comment has been removed

• Anonymous
  December 19, 2006
  Localization "The common case is non-english-speaking programmer gets an error, if the error is in their language then they stay productive, and if it isn't, then they're stuck.  Googling it in English is hardly going to help them!" English language being not my mother tongue, I think I may say something here. Please localize KEYWORDS or forget the localization of programming languages altogether. No programmer (or at least a bit interested in programming) in Lithuania doesn't know English. I discourage my colleagues coming up with non-English variable names, because the code just gets more messy with that. You can't write "do{}while(teisybė)" what's "while" what's "do" in that? And if not, then what's "teisybė"? For professional programmers non-localized error messages is not a problem and it even is better (for Googling and stuff). And for non-professional ones... I don't really think error message localization is a solution here, but if it is and if variable names with international characters make sense, then keywords should follow that route too (although I would rather not use that kind of compiler:).

• Anonymous
  December 20, 2006
  The comment has been removed

• Anonymous
  December 20, 2006
  The comment has been removed

• Anonymous
  December 20, 2006
  The comment has been removed

• Anonymous
  December 21, 2006
  The comment has been removed

• Anonymous
  January 05, 2007
  The comment has been removed

• Anonymous
  January 30, 2007
  I agree in not localizing error messages. One example is in NullReferenceException. As a programmer you should/must learn in terms like Null, reference, object, exception. See is as names of something, not as explanations. What to do when you see a NullReferenceException while debugging, and not knowing this name/term? I think all terms should always be in english everywhere. It is part of the programming language/framework.

• Anonymous
  February 09, 2007
  The comment has been removed

• Anonymous
  March 01, 2007
  What would be very nice if someone rewrote the System Error Messages. Most are incomprehesible and for some the .h files are the only public documents containing the terminology.

• Anonymous
  May 07, 2007
  And the oscar goes to... "Catastrophic Failure"

• Anonymous
  May 08, 2007
  Reader Shaka comments on my post about error messages that "catastrophic failure" really does take the

• Anonymous
  March 01, 2011
  I totally agree with Adrian and the other "don't localize error messages" writers. From Norway.

• Anonymous
  March 01, 2011
  Example of bad  prescriptive message: (not from compiler) Contact your System administrator. Some times I am the System administrator - then it doesn't help me. Other times I don't know who the System administrator is, or how to reach hem.  - then it doesn't help me either.

• Anonymous
  June 05, 2012
  The comment has been removed

• Anonymous
  June 11, 2012
  The comment has been removed