WOW64 on HTTP.sys
Back when Windows Server 2003 SP1 shipped, HTTP.sys hit an important milestone, WOW64 support.
What is WOW64 support and why is it important?
I’m glad you asked.
WOW64 lets 32 bit applications run on top of 64 bit windows. WOW64 support for HTTP.sys is an important piece of letting 32 bit IIS worker processes run on top of 64 bit windows. You want this because of all the existing applications that run on top of and plug in to IIS, like ISAPIs are still distributed as 32 bit binaries. You want the 64 bit kernel because big sites are getting squeezed from two directions; the number of connections and memory address space use in user mode. The number of connections that windows and HTTP.sys can support is dependent on non-paged pool memory (kernel memory that can’t be paged out to disk), since every connection in TCP/IP and HTTP.sys require a non paged pool structure. Some people try to trade off between these two limits via the /3gb PAE switch which constrains kernel mode addressing to 1 GB (down from 2 GB) which causes there to be less room for non-paged pool and hence constrains the number of simultaneous connections. 64-bit addressing gives us a lot of room to address non-paged pool in the kernel and hence can greatly expanded the connection limit. On the user mode side each process now gets much of the full 4 GB address space for 32 bit processes running under WOW64. The people who run Microsoft.com have a nice article about the trade off and how much more memory they get in both spaces when they moved to 64 bit.
So what was the work needed to make this happen (and why didn’t we ship support for this in win2k3 original)?
For us it is nontrivial work that didn’t fit in the win2k3 ship schedule. HTTP.sys is a driver that communicates with user mode via an IOCTL interface and it turns out that the vast majority of HTTPAPI.dll is just mapping the API to an IOCTL (Part of testing HTTP.sys we have to test the IOCTL interface as well as the API). This interface DeviceIOControl, much like IDispatch::Invoke, turns all possible function calls in to a single call. It takes a handle, the IOCTL code, an input buffer, an output buffer, the number of bytes returned and an OVERLAPPED structure. Depending on what you stick in the buffer the IO manager takes care of almost all the 64/32 bit book-keeping, like the change in HANDLE size and work with the top level buffers. However HTTP.sys structures tend to be a bit more complex. Looking at HTTP_RESPONSE we see at least two pointers who can’t point to other structures. Since the IO manager isn’t psychic we have to do the translation of those pointers our selves. A lot of the HTTP.sys IOCTL code has bits like this (much like the guidelines under IOCTL Support on WHDC 64-bit checklist):
#if defined(_WIN64)
if (IoIs32bitProcess(Irp)) {
GET_OUTPUT_BUFFER(Irp,
STRUCTURE32,
&OutputBufferLength);
}
else
#endif //_WIN64
{
GET_OUTPUT_BUFFER (Irp,
STRUCTURE,
&OutputBufferLength);
}
It turns out we were real gluttons for punishment because our structures tend to have pointers to other structures who also contain pointers. Each buffer and pointer has to translated both in the api call as well in the structures that we return back to user mode like HTTP_REQUEST.. In the send path we capture all the 32 bit information up front and in one place, but in the receive path we fill out the user data buffer in a number of places and we do the translation as we go, avoiding a copy step later to translate from the native 64 bit structure to a 32 bit one.
When we tested this feature we found a number of alignment issues that the devs didn’t expect. The alignment issues were caused by 32 bit aligned structures not being 64 bit aligned and it can be tough figuring out all the places this could occur. In the cases where this happens we had to write our own byte by byte copy routine since the system one expects byte alignment.
I remember that we got some flak from David Cutler for being one of the few components that didn’t have WOW64 support at Win2k3 ship.
-- Ari Pernick