Jaa

FileSystemWatcher Follies

  • Artikkeli

 

System.IO.FileSystemWatcher is a handy class that can be used to monitor directories for some types of changes with very little programming effort for the developer who uses it.  For some situations, it’s incredibly useful.  However, it often gets used in program designs with poor assumptions about how things are going to play out in a real world enterprise scenario, and that can lead to problems.   I’ll be going over several examples below and using them to explain limitations of the FileSystemWatcher class, but before that, let’s discuss what the FileSystemWatcher actually does.

FileSystemWatcher: ReadDirectoryChangesW wrapped for .NET

System.IO.FileSystemWatcher is basically a wrapper class for the native API ReadDirectoryChangesW.  There are some differences, mainly added parsing capabilities by FileSystemWatcher after it receives results from ReadDirectoryChangesW that it can use to decide whether or not to report the changes to its subscribers based on its settable properties.  It’s the similarities that are going to matter in most cases though.  This is the function prototype for ReadDirectoryChangesW from WinBase.h in the Windows 8.1 SDK:

WINBASEAPI BOOL WINAPI ReadDirectoryChangesW( _In_ HANDLE hDirectory, _Out_writes_bytes_to_(nBufferLength, *lpBytesReturned) LPVOID lpBuffer, _In_ DWORD nBufferLength, _In_ BOOL bWatchSubtree, _In_ DWORD dwNotifyFilter, _Out_opt_ LPDWORD lpBytesReturned, _Inout_opt_ LPOVERLAPPED lpOverlapped, _In_opt_ LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine );

There is a restriction here that one probably wouldn’t notice when just looking at FileSystemWatcher: Only directories can be monitored, and while monitoring of sub-directories is optional, at least the entirety of the directory is going to monitored.  So if FileSystemWatcher is set to watch a directory with a lot of activity but filtered to only watch one file, it is going to consume a lot of resources processing things that never make it back to its subscribers.

There’s also some guidance in the documentation that is often relevant to subscribers of FileSystemWatcher.  When the buffer is overflown, ReadDirectoryChangesW returns ERROR_NOTIFY_ENUM_DIR, and FileSystemWatcher will report that with its OnError event as an InternalBufferOverflowException.  When that happens, the documentation states that “you should compute the changes by enumerating the directory or subtree”.  That effectively means that if an application using FileSystemWatcher wants to not miss changes that happen when a buffer overflow occurs, it needs to keep track of the state of things as they change, and whenever an InterBufferOverflowException is reported, manually enumerate things and compare it against its last known state to compute the changes.

With that basic overview complete, let’s move on to our examples.

Example 1: Attempting to detect when a file has been created and its contents finished writing to disk

This often happens when someone creates a design where data needs to be moved from one server to another, and when the data arrived at the destination, the destination needs to react to the data.  In an effort to “simplify” things and not use networking code, some developers use the file system to pass data to another machine, either through a file sharing technology or another local service like FTP or SFTP.  For this example, let’s say that Server A has an application that is creating files and then using a script to FTP them to Server B.  Server B then has a service that is watching the directory where the FTP service is storing the files.  When data through the form of files come into the directory, Server B attempts to open them and act on the data.

Obviously, there are a lot of holes in the strategy above, so we won’t cover all of them, but here are some highlights:

  1. There is no FileSystemWatcher event for when a file is closed.
  2. Monitoring for size changes and/or last_write is going to be affected by the disk cache if present.
  3. Suppose that the developers of the application realize the above two things and decide to send a second file over to indicate that the first file has finished transferring through FTP.  This also will not work as there is no guarantee that the second file doesn’t get written before the first.
  4. Repeatedly trying to open a file when its creation is reported with no file sharing may eventually let you know when it is closed, but it will use a lot of resources and could become problematic if many started happening at once or if the directory being watched is accessed over the network.

Example 2: Attempting to watch for changes on a high traffic network share

This happens when the directory being watched has many files changing in it and/or subdirectories are being watched too and those have many changes happening.  The buffer size for network paths is capped due to limits in the underlying network technologies used by file sharing protocols.  If many changes are happening, the buffer will often overflow.  In order to get the changes, the directory or directories will have to be enumerated.  The more files that there are in a directory, the longer it takes to enumerate, so this can quickly add up to a lot of resources for the server actually hosting the files and delays for the application trying to access the information.

Watching a single directory without subdirectories and minimizing the number of files in that directory will improve performance and reduce the number of potential buffer overflows.

Example 3: Assuming that all directories behave the same way

Often times a developer will design an application that uses the FileSystemWatcher to monitor a modifiable path through some sort of configuration.  If that developer assumes that the location will be a local hard disk on a physical machine, many unexpected problems could arise in the application.  For instance, let’s go over some of the things that could happen if the user enters a path that ultimately ends up on a network file share:

  1. The remote file server may not be a Windows Server and may not properly support ReadDirectoryChangesW; though many other operating systems will support at least one protocol that does support ReadDirectoryChangesW, not all of them do, and that protocol may not be in use for some reason in the environment.  
  2. There could be latency in the connection that causes performance problems in the application especially if the application performs synchronous I/O operations on GUI interacting threads.  This could be actual network latency or it could just be the additional resources it takes to perform the same functionality against a remote system.
  3. The remote file server may be shut down incorrectly and fail to send a packet back to the machine to inform it that the handle is now invalid.  If that happens, the FileSystemWatcher won’t report an error or any changes from that point on.  It won’t know that there’s a problem, and there’s no built-in timeout mechanism for the class which is about the best one can do to cover this situation.

Wrapping up

In conclusion, FileSystemWatcher is a handy class, but its use needs to be thought about carefully when considering implementing it especially in enterprise scenarios in order to avoid unexpected scenarios within an application and be a good member of the computing ecosystem while achieving desirable outcomes.

 

Follow us on Twitter, www.twitter.com/WindowsSDK.

Comments

  • Anonymous
    May 21, 2015
    Thanks for the article

  • Anonymous
    May 27, 2015
    any solution to example 1?

  • Anonymous
    May 27, 2015
    Yes, there are several, which depending on some other factors may or may not be appropriate: Change Journaling (if the file system is NTFS), Filter Drivers, and Event Tracing Consumption of Kernel-File events to name a few.  I may address one or more of these in a follow up blog post at some point.

  • Anonymous
    May 27, 2015
    @Chris Lewis, thanks