Debugging a Text Service (and Visual Studio problems)

I've received a few emails asking about debugging Text Services.  I figured if two people actually went to the trouble of sending an email, that there's enough demand for a post.

But first, a little diversion:  It turns out that Visual Studio 2005 SP1 ships with an updated version of the C Runtime library, and, by default, adds an entry to the application manifest to use that new library.  Unfortunately, this can cause serious problems for text services - in particular, the text service DLL often won't load.  I work around this problem by turning off manifest generation in the linker - either on the command line, with /manifest:no, or by going to the project property page, then selecting Configuration Properties/Linker/Manifest File in the tree view, and setting 'Generate Manifest' to 'No'.   In general, you should minimize your use of the C Runtime library anyway, as it's more stuff that gets loaded into every process.

Now, on to debugging.  There are two options:  tracing, and real debugging.  You can use OutputDebugString and DBMon in your text service to trace the flow of execution without any problems, but for serious problems, like hangs, or crashes, you'll need to use a debugger.  I've never been able to debug a text service running on the same machine as the debugger, mostly because the TSF Manager needs to use your (stopped) text service in order to give keyboard focus back to the debugger.  Instead, you will either need a second machine, or a virtual machine like Virtual PC.  (Microsoft gives this away for free, and also publishes trial virtual disks that you can use for evaluation.)

Once you have installed your new text service on your other machine, you can use remote debugging to debug problems in your text service.  I've used the visual studio debugger with great success, but it can have some problems, particularly when you need to debug an application that needs to run elevated on Windows Vista.  In those situations, I use cdb and windbg (cdb on the test machine, and windbg on the local machine).  

To be honest, I've gotten used to the power of windbg (and cdb), and I use that almost exclusively.  But windbg isn't the friendliest debugger in the world, so I want people to know that visual studio works just fine.

Once you have your debugger set up, you can launch a process on your other machine (e.g., wordpad, notepad, Microsoft Word, etc.), and set a breakpoint at your dll entry point (or at the entry point of your choice, for example, CMyTextService::Activate).   Visual studio's syntax for setting a breakpoint in a specific DLL is a little baroque (e.g., {,,mydll}MyFunction) , and I find windbg's syntax (e.g., mydll!MyFunction) a little easier to use.