The Case of the Mysterious Locked File
The other day I was intently editing code in Visual Studio and hit F7 to compile my latest batch of changes when I was interrupted with a dialog box that informed me that my file couldn’t be saved to disk:
I viewed the suggested cause with suspicion because I had been working with the file for about an hour, saving and recompiling modifications as I went without a problem. The first step in my investigation was to therefore run Filemon and determine the actual error underlying the dialog box:
The sharing violation error highlighted in the Filemon trace confirmed the dialog’s message that Visual Studio was unable to update the file. This error results when a process attempts to open a file that another process has previously opened in such a way as to prevent concurrent read, write, or delete access, or a combination of those access types.
What application would possibly want to open a source code file while I was working on it? To get the answer to that question I turned to Process Explorer. I opened Process Explorer, clicked on the Find Handle toolbar button, and entered “dfrgcmdr.cpp”, the name of the file in question, as the search text. Process Explorer populated the search results with one entry that identified the guilty process and when I double clicked on the entry I was taken to the specific handle representing the open of the file in Process Explorer’s handle view for that process:
The MSNGather.exe process, which describes itself as the MSN Desktop Search (MDS) executable, had opened two handles to the file I was working on. MDS was apparently indexing the file’s contents because it noticed that the file had recently changed. The file is relatively small so I expected MDS to be able to index it in a matter of seconds, but I had already spent about a minute looking into the error dialog’s cause and MDS still had not closed the file. In fact, it was not for another several minutes before MDS finally finished with the handle and let me get on with my work.
This experience uncovered what I believe are two MDS bugs. The first is that MDS is holding files open for indexing without allowing them to be modified. This can interfere with system and user file activity and, in any case, I don’t see it as necessary. MDS can’t keep its data current with the most recent modifications of all files without indexing changes as they occur, an approach that would introduce an unacceptable performance impact, so why should it care if a file is updated as its being indexed?
The second bug is that MDS holds files open for long or indefinite periods. The CPU usage for MSNGather.exe was zero for most of that time so my theory is that that MDS began indexing the file, but paused when it detected system activity in the form of my compiles in order to keep indexing a background operation. MDS should close handles to files that its scanning while paused so that it doesn’t interfere with foreground file activity such as deletions and renames.
Originally by Mark Russinovich on 4/24/2005 3:33:00 PM
Migrated from original Sysinternals.com/Blog
# re: The Case of the Mysterious Locked File
So, how much time and aggravation has MSN Desktop Search saved you so far?
4/24/2005 11:26:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
Mark,
I attended one of your talks at Windows Connections and it was excellent. I also thought you'd like to know that your free bug fixing for Microsoft is actually very, very useful to me. It seems to me that these products should never have been released with the bugs they have, but that's not the point I want to make. The point I'd like to make is that I wasn't born with any inherent knowledge of how to use the tools you've written. These troubleshooting exercises are outstanding in showing me how to make advanced use of your utlities. I didn't know that I should follow a handle in Process Explorer (though handle is a familiar term) to drill down to the process that has a file open. I don't doubt that some people would say that makes me a bad sysadmin, but I just want you to know that I find this information invaluable. Thank you.
4/25/2005 12:13:00 AM by Carl
# re: The Case of the Mysterious Locked File
I agree--I also get a lot from Mark's blog entries, and I hope he keeps it up. Now, if I could only convince him to stop by and give me some private tutoring...
4/25/2005 1:03:00 AM by Anonymous
# re: The Case of the Mysterious Locked File
You are right that the indexer will hold open a file. It happens when MDS starts indexing a file and then backoffs because of user activity. The file is kept open so you don't have to start from the begining again, but this is indeed mysterious behavior. I will make sure this gets fixed in the next version.
4/25/2005 1:29:00 PM by Chris McConnell (MSN Search)
# re: The Case of the Mysterious Locked File
There is another source of mysteriously locked files: an antivirus with enabled on-the-fly scan. Some particular brand has caused grief for some users of Visual Studio, because files were lost.
4/25/2005 7:10:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
If only the FILE_SHARE_READ was the default for CreateFile's 3rd arg. It would make life so much easier for the world.
4/26/2005 1:46:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
offtopic...
Why does the order of the items in the process explorer tray menu keep changing between versions, please put close on the bottom, or atleast below open. And how about shift+dbl click to toggle the system info dialog?
4/26/2005 2:34:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
I think I ran into this very same problem the other day.
I also notice I've been having problems with OTM file in Outlook 2003, I wonder if it is related.
Thanks for pushing it over to Microsoft.
4/26/2005 11:13:00 PM by Peter Ritchie
# re: The Case of the Mysterious Locked File
Mark - Thanks for posting this Blog. It was very helpful
4/27/2005 8:36:00 AM by Anonymous
# re: The Case of the Mysterious Locked File
Mark,
I think it's a bit unfair to blame MSN Desktop Search itself for all the wrong doing here. It isn't responsible for opening the files.
The opening (and eventual locking) is done inside the IFilter in charge of indexing of a particular type of file.
MSN DS is just loading the relevant IFilter DLL in memory and then call the functions to get the content.
Generally, when MSN DS decides to pause due to user activity, it will suspend for a while calls to the IFilter but will keep the DLL in memory. The IFilter doesn't know the indexing is paused or aborted. That's generally why IFilters only closes file handle when the DLL is unloaded. So, if MSN DS suspend the index right in the middle of a file locked by an IFilter and MSN DS doesn't decide to unload the DLL, the file will stay locked until at least the indexing resumes.
You would virtually have the same locking problem with SharePoint, SQL Server or indexing service. They use the same IFilters technology to index files, but generally they don't index files in real time, the files are stored in SQL tables or there isn't any pauses during the indexing process.
4/27/2005 10:04:00 AM by Ben
# re: The Case of the Mysterious Locked File
There are lots of problems with *bad* application interactions.
First comes to mind Word 2003 or Photoshop CS keeping an handle on the folder from where you opened the last file; even if the file is now closed, you can't delete this folder or you aren’t allowed to properly disconnect the media where the file sits.
I sincerely hope MS will eradicate these deficient behaviors one day. If they are engaged in pushing the “quality” bar higher, they should produce extensive "How to design for Windows guidelines?" -current guidelines are incompletes, sometimes not up to date, and scarce in some parts *cough*the registry*cough*- with a strict policy and a profound desire to enforce or to make these enforced -what about a strong marketing campaign for users and ISV-.
But this situation could also exist forever…
4/28/2005 9:36:00 AM by Sébastien Mouren
# re: The Case of the Mysterious Locked File
I lost work of nearly a week because I had hit F7, made a little modifications and hit CTRL+S while still compiling. I also had got this message, but didn't care and closed my DevStudio.
After reopening it, the source file was away and the neat algorithm I worked nearly a week for was away too. For 4 hours, I tried everything from undelete to unerase and back, couldn't find the file. Then I used a forensic tool and found about 10 instances of the source code in free clusters. Phew.
Daniel
5/9/2005 2:09:00 PM by Daniel
# re: The Case of the Mysterious Locked File
There's a little explorer utility that is perfect for these situations wher eyou know the file name in question here:
https://www.dr-hoiby.com/WhoLockMe/index.php
Joe
5/14/2005 10:45:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
Google Desktop Search does the same thing. It seems to hang onto a handle on every drive, making safe removal of flash disks impossible. Among other things.
5/23/2005 1:25:00 PM by Joe
# re: The Case of the Mysterious Locked File
Alternative for WhoLockMe:
https://ccollomb.free.fr/unlocker/
I wonder why any system expert would voluntarily install a tool from Microsoft for searching when the XP internal search has already proven to be buggy.
Have you ever tried to search inside all file types?
Even the
"FilterFilesWithUnknownExtensions"=dword:00000001
fix is not sufficient as I have learned recently. Just compare the search results inside files with decent applications like Total Commander, for example.
So, what is missing in this blog article is a warning to not use any search related stuff from Microsoft as it's simply nothing you can trust *in general*, no matter if file handles are correctly closed or not.
Icfu
5/24/2005 5:17:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
I've stumbled across the same error a year ago. It happened to be some antivirus (i think DrWeb), who kept file locked for ~15 seconds each time I hit Ctrl-S in visual studio.
6/19/2005 2:47:00 PM by Bolkin
# re: The Case of the Mysterious Locked File
I have my indexing services disabled, and occationally see the locked file error. In my case saving again will usually succeed.
I agree it could be anti-virus related. The next time it seems to happen consitently i will try changing AV settings.
8/22/2005 9:10:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
Well, I can obviously learn a lot here.
8/26/2005 6:00:00 PM by Anonymous
# re: The Case of the Mysterious Locked File
Has anyone used Effective File Search (EFS). Great app, small footprint, blazing fast searching of all drives, local and lan shares too. Try it, I think you'll agree.
8/28/2005 1:22:00 AM by Anonymous
# re: The Case of the Mysterious Locked File
...isn't this what opportunistic locks are for? Be notified when something else is trying to do something with the file, so you can do your own thing (like, say, close it)?
Jeeze. You'd think the MS developers don't read MSDN or something...
11/3/2005 1:43:00 AM by Kyle H
# re: The Case of the Mysterious Locked File
After being disappointed with Google Desktop Search (because of limits on the number of characters indexed in a file) I bought X1. Occasionally I have to shoot it in the head because it is tying up the CPU when I need it, but that is only once a week or so. Otherwise I am way more happy with that than I was GDS.
11/6/2005 5:14:00 PM by Student
# re: The Case of the Mysterious Locked File
I don't understand why it stops/slows down indexing if there is (any?)user activity. Shouldn't the program priority be set to below normal by default?
7/27/2006 10:16:00 AM by Anonymous
# re: The Case of the Mysterious Locked File
Good info. I've long been aware that while Microsoft hires very, very smart people in the public's view but I've found they hire very, very bad programmers to actually do a lot of the programming.
I don't know if these people have never used a computer before so are not used to common QA procedures or if they just aren't programmers and in fact just smart people that passed the interview by fakeing it.
All you need to do to see how inept they are is read any Microsoft documentation. How do you show the proper use of an object without showing the proper use of an object?! They have been notorious with their networking examples. They'll open a socket, receive data, AND NEVER CLOSE THE SOCKET!!!!! This of course will cause you resource errors when you try to actually use a socket for it's intended purpose of being a server port. So, any REAL programmer would instantly recognize this error. It's like not closing a database connection because you think the garbage collector will take care of it - which MS does in some of its examples!
7/30/2006 2:17:00 PM by Anonymous