Share via

File Handle Hell

  • Article

Raymond Chen has an interesting post about a program that tried to open a file, got back an error code, and used the error code as a file handle. In this particular scenario everything worked, but completely by accident.

Reading it, I remembered back to my own "bug from hell" related to file handle.

The scenario: I was working on NuMega's BoundsChecker in the very early days of Win32. BoundsChecker forces its DLL into the target process very early in the target process lifetime. The BoundsChecker DLL then writes various messages to its output file.

One particular program (SourceSafe) had some old dead DOS code that immediately closed the three standard handles (stdin, stdout, stderr) to free up as many handles as possible.

Imagine what happens in this scenario:

  • The BoundsChecker DLL loads, and opens an output file. The file handle it gets back is 1.
  • SourceSafe blindly closes stdin, stdout, & stderr (0,1,2, as I recall). Thus, it closes BoundsChecker file handle. BoundsChecker has no knowledge of this.
  • SourceSafe opens its open file handle, and gets back 1.

Imagine the havoc this creates. Both BoundsChecker and SourceSafe are writing to the same file handle (1). They're writing completely different data. The upshot in this case was that the SourceSafe database got completely corrupted.

Man, did I take a lot of flack for that one!

Comments

  • Anonymous
    January 18, 2005
    The comment has been removed
  • Anonymous
    January 18, 2005
    The comment has been removed
  • Anonymous
    January 18, 2005
    Ah, yes, the good old days. But you forgot to mention that after you did it, you went and did it again a few minutes later, which made it even worse! I remember spending a fair amount of the next two weeks after that trying to restore the SourceSafe data. :-)

    You should know that even today, we tell the tender young things that we hire out of college: "Don't run BoundsChecker on the version control system!"
  • Anonymous
    January 18, 2005
    I was just reminiscing with one of my co-workers and told him that story just a few hours before I saw your post.

    Ah, the good old days, we certainly had fun back then.
  • Anonymous
    January 18, 2005
    Maybe you can tell your readers about "Matt's Huge Irritating Banner" next...
  • Anonymous
    January 19, 2005
    Yes, tell us that one. I haven't heard that one yet.
  • Anonymous
    January 19, 2005
    The comment has been removed
  • Anonymous
    January 24, 2005
    NOW I UNDERSTAND! (Sorry for shouting)

    We had the same problem with VSS and the Novell Netware Client. Some Netware error information landed in some weirdly named aafaaaaa.b file and damaged the VSS repository...