Old school debugging - VB6 middleware applications

VB6 has fallen off the supported list here at Microsoft. It had a good run since it was released in 1998. We no longer offer free support and limited support even for ready cash. The only hotfixes that we will entertain are security fixes and we haven’t ever been asked to do one of those. However, there are a lot of VB6 apps still out in the field doing good solid work.

I should say at the outset that I liked classic VB a great deal. I was never blind to its weaknesses and there were some things that it was very bad at doing. However, it was an excellent tool for rapid applications development and over time it was pressed in to many other roles. So, when I say that VB wasn’t good at something, I bear it no malice. The VB dev team (especially Matt Curland) did remarkable things.

Although I will refer to VB6 throughout, pretty much anything that you can say about it also applies to VB5 SP2 and later. They were very much the same language. Ok, VB6 had data environments and the data report. I am sure that someone used them to great effect. I just never met that person.

So, most VB developers liked to debug in the IDE. There were excellent reasons for doing so and the greatest of these was edit and continue. The debugging features were pretty good by the standards of the time with the watch window, the immediate window and the conditional breakpoints all being useful features. For debugging thick client apps, the IDE was and is still the best tool. There was a time when it was useful for debugging middleware but times and operating systems changed. I will need to explain a bit out how debugging worked in the IDE.

Any code that you ran in the IDE ran as Pcode and used VBA6.dll rather than msvbvm60.dll. This wasn’t the same as MSVBVM60 in quite a few ways but the important one here is that it had a lot more hooks for debugging. The Pcode engine generally did a good job of pretending to be the proper runtime system and contained a great deal of the same code. When you ran in the IDE, your app wasn’t really its own process but that hardly ever showed. When your component was not intrinsically runable in isolation (because it was a DLL or OCX), the VB6 IDE hosted it for you. That isn’t to say that VB6 was the client. Another application would be the client and would be pretty much unaware that VB6.exe was sitting in the middle. To do this, the IDE played fast and loose with the registry and fixed it up so that a component was instantiating an out of process server (VB6 itself) while the component being debugged thought that it had an inprocess client. In NT4, this allowed you to debug a component nominally hosted in MTS within the IDE although it sometimes required a few small changes.

If you are familiar with COM (another fine legacy technology), you will know that there is quite a difference between an in-process and out of process components. Some things you can share within your process but you can’t share them outside of your process. You don’t need to marshal data with in-process calls and you do out of process. There isn’t normally a change of context (for example, user account) in an in-process call but there often is with an out-of-process call. Also, a client would reasonably expect to see the interfaces supplied by your DLL and so VB6.exe suddenly grew a lot of new interfaces when it was hosting your component.

This was all very clever and a little bit fragile. As MTS grew up and became COM+, it stopped working because the interrelationships  between the component and the host became much more complex. You couldn’t always use the IDE to debug hosted components any more. If you want to know how it was done under IIS4 then there is still a KB - https://support.microsoft.com/default.aspx?scid=kb;en-us;299633

If you are using something more modern that IIS4 (and I really, really hope that you are) then you have to go hardcore. While you can run Pcode compiled components under DLLHOST, I really wouldn’t recommend it for many reasons. You certainly can’t debug them as (a) you will have no symbols and (b) you would be trying to debug the Pcode interpreter. I have done that more than once and it is interesting and challenging but not otherwise rewarding. So, compile to native code with symbols. If you can reproduce the problem without optimisation (and you generally can) then turn off optimisation. By the way, compiling for small code often gives better performance than compiling for fast code anyway.

Because the component really will be hosted in DLLHOST.EXE (or MTX.EXE or INETINFO.EXE), you will have to debug the whole process. You can use the VC debugger for this if you like but I always prefer WinDbg or CDB for this. Please do NOT do this on a live server if you have any options at all as it will effectively take the server offline. In that case, you are probably better off debugging it with tracing (OutputDebugString for example) and post mortem debugging techniques. For information on how to get dumps for later analysis, check out https://support.microsoft.com/default.aspx?scid=kb;en-us;286350

If you can debug at the machine console while logged in as administrator then that will make your life easier. If you don’t or can’t do that then maybe you can use a remoting tool such as Netmeeting. It does work surprisingly well for this. Alternatively, if your platform is Windows Server 2003 then check out the console option of Mstsc (https://technet2.microsoft.com/WindowsServer/en/Library/f47ce263-f72e-469d-bf14-6605b7f4cce51033.mspx)

If you have limited access to the server machine then you can use the remote debugging facilities of WinDbg. Attach a copy of WinDbg to the process in the usual way and then turn it in to a debugging server (check out .server in the WinDbg help). You can then connect to it remotely from the File menu of WinDbg. It will be just like being there except for the lack of noise from the server room fans. When debugging a remote, your copy of WinDbg is just a very smart terminal so all extensions, symbols and so on have to be on the remote server. You set this up the exact same way for any DLL, VB6 or .NET.

The symbols for your component will not load until your component does and so you have to let the server run at least that long. You can put a break in early in your VB code if you want to stop the debugger at that point but if you do, remember that it will stop there every time through the code. Let’s assume that you let it run and then break in. If you list the loaded symbols for your module with "x MyModule!*" then you will see all of your functions together with a lot of symbols bundled in there for you. VB adds interfaces and symbols quite unashamedly but you don’t generally need to worry about those. One thing that will probably look strange is that all class/method syntax with the C++ double colon convention instead of the friendly little dot. WinDbg doesn’t understand that VB is different and it is treated just like any DLL with symbols.

From here, you can set breakpoints in the usual way (bp etc) and step through code. You can also open up VB source code modules and set breakpoints in them with F9 although the VB file extensions are not in the source file type dropdown. Stepping through the code is revealing but might be a little alarming if you have not seen the code that VB generates for you before. You will be stepping through the assembler and there is a lot of COM goo in there. Hresults get checked a lot. You will probably need to refer to the source often to work out where you are since it takes a bit of practice to be able to know what the source code looked like. Variants are especially challenging because VB does a lot of work for you there and what looks like a simple equation can result in a great deal of code. Optimised code is even harder because the order of execution is often very different from what you might expect and it is harder than usual to see the data.

Data is not easy to get at this way. When you look at local variables (dv is the command) then you may see that variables are simply listed as eclipsed which means that the memory is being used for something else as well within the function lifetime or that the name is not unique in this context. Enums just show as integers or longs and objects show as pointers. In fact, they always were exactly that but the VB IDE hides that from you. VB strings are COM BSTRs (and accordingly Unicode) under the covers and byte arrays are really char arrays. You might be surprised to discover that VB strings are Unicode as VB appears to have no support for anything but ANSI. That is because the Ruby forms engine was ANSI only. The runtime converts the Unicode strings to ANSI for Ruby and API calls although there are ways to pass Unicode if you want.

You are not going to be able to get at the Err, App or Printer objects since you would need to go through a lot of internal and completely undocumented structures to get at them. Even if you could get there, they would just be raw data without the accessor functions that you use in VB. If you need to look at any of those fields, your best bet is to embed debug code in the source code to copy their values to somewhere that you can get at.

You can step in to the VB runtime if you want but it probably won’t be very revealing if you are trying to debug your application. If you do, you will notice that VB’s internals are very COM influenced. The influence was actually two way since some COM ideas came from VB originally.

You may see exceptions when running your code. Null reference exceptions (i.e dereferencing a null pointer) are not uncommon or anything to worry about. They will show up as first chance C000005 exceptions with a 0 or almost 0 address. The runtime will sometimes do that if there are objects set to nothing but that is safe because the only possible values are null or a valid value. You will also see exceptions if your code does lookups in collections and the value is not there. Because exceptions are now so expensive, you probably want to avoid doing that if you can. Another exception that you will commonly see is c000008f. If you look the number up then you will find that it is a floating point inexact result exception. It is used in a different meaning here – since we don’t generate real floating point inexact result exceptions, they can safely be thrown to indicate VB errors of the normal trappable type.

Debugging hangs and crashes in VB components is done very much in the same way as with any other unmanaged component but it is just a little harder because of the compilations described above. If you have to try debugging VB code this way, I would strongly recommend that you start on a "Hello world" application and work your way up. All the things that may VB an easy language to code in make it a terrible language to debug.

I hope that this has been helpful and not too daunting. One of my readers (Mark Capaldi) suggested that it would be an interesting topic and I hope that he found it so.

Signing off

Mark

P.S. Please forgive any typos. It is a shade before 2 A.M. as I write this.

Comments

• Anonymous
  January 29, 2006
  Thanks Mark. I certainly did find this an interesting post and will be rattling off some "Hello World" COM apps first thing tomorrow morning.

• Anonymous
  February 15, 2006
  ' By the way, compiling for small code often gives better performance than compiling for fast code anyway.' This statement did give me some food for thought as most of the time we usually compile for fast code assuming it gives better performance. Would you mind elaborating about this issue.
  

• Anonymous
  February 16, 2006
  I touch on it in my blog on fast code. The big secret to fast code is to less stuff. With modern architectures, CPU cycles are much cheaper than memory access so the more of your code that is in cache, the quicker it runs. The small code options give some very decent optimisations even without the cache effect but cache makes a huge difference these days. Pcode can be faster than native code because it is smaller and the bits of the runtime needed for it are pretty small - each Pcode instruction comes to about 8 lines of assembler. The mechanism that links them all together is not that complex and pretty much stack based (so, very different to IL? Nope)
  
  As always, your mileage may vary but give it a try and you may be suprised by the results.

• Anonymous
  March 31, 2007
  But the fact the VB program is not running in a separate process really shows when debugging subclassed window procedures, hooks, or Declares. That is why I recommend using native debuggers to debug these.

• Anonymous
  March 31, 2007
  The VB.NET debugger, however, runs the debuggee in a separate process

• Anonymous
  March 31, 2007
  The comment has been removed

• Anonymous
  April 01, 2007
  Also keep in mind that according to the VB6 help topic "Passing Function Pointers to DLL Procedures and Type Libraries": "If your application fires a callback function while in break mode, the code will be executed, but any breaks or steps will be ignored. If the callback function generates an exception, you can catch it and return the current value. Resets are prohibited in break mode when a callback function is on the stack." If you delete a function, it is modified so that if any external caller calls the function though the pointer given to it by using AddressOf, it returns 0. I wonder how errors are passed to the procedure that called it if the caller is written in a language other than Basic and the caller called the procedure by using the pointer given to it by using AddressOf.

• Anonymous
  May 05, 2007
  "If you delete a function, it is modified so that if any external caller calls the function though the pointer given to it by using AddressOf, it returns 0." I have never tried this but I wouldn't want to rely on it. The reason that I say that is because I don't see how it would work with multiple instances and threading. Consider:

1. How do you delete a function? You should only use AddressOf with functions in .BAS modules unless there is some magic way of knowing that you will call in to the right instance of a class. Even then, what wll be the thread context? I could have 100 threads, each with their own TLS and accordingly state. If you don't care which one you get then all well and good but that is a pretty unusual state, I think.
2. When X happens then Y happens is dangerous in multi-threaded application unless there is thread synchronisation. I don't recall seeing any in that area of the code though I admit it has been a while since I looked. Unless the code that handled the deletion of a function removed the function from the jump table prior to removing the function. I am not the final authority here but I suspect that is a doc error. Exceptions in VB are just plain old exceptions. To be precise, VB normally throws "floating point inexact result" Hope that helps Mark