Share via

IIS6 and HTTP Server API, Part 1

  • Article

Question:

Hi,

I've found these articles:
https://support.microsoft.com/default.aspx?scid=kb;en-us;q311852
https://support.microsoft.com/default.aspx?scid=kb;en-us;q327611

First of all I've got a question. READ_RAW_DATA can be used to modify the request body not only the headers. Why is it gone? If you don't want to modify its behavior you should add a new notification type (input and output as well) only for modifying the entity body.

But my main question is that this article says that IIS 6 operates quite different in IIS 5 mode.

I section "SSL in IIS 6.0" I've read that in IIS 5 mode IIS 6 hosts SSL and filters can use SF_NOTIFY_READ_RAW_DATA. This can mean one of the two tings:
1. IIS 6 is not using HTTP API at all. In this case a lot of HTTP API based things can be broken, including HTTP API based Windows system services and SQL Server Yukon's web services. If it's the truth is a very strange thing.

2. IIS 6 is using undocumented HTTP API calls that make it possible to process the raw data. In this case you should document these APIs as they could be used by others.

httpapi.dll exports some undocumented functions (without header declarations) that are used only by IIS. But as HTTP API is an operating system service and not part of IIS I think you should document all the functionality of HTTP API.

Sincerely,

Answer:

Thanks for the query.  I've actually not seen either of these KBs until now, but I can tell you that the KB is wrong to say that SSL in IIS6 uses the SF_NOTIFY_READ_RAW_DATA filter notification to function. It does not. SSL in IIS6 is not an ISAPI Filter.  I'm also going to try to clarify the info in 311852 because I think the usage of the words "READ_RAW_DATA" and "SF_NOTIFY_READ_RAW_DATA" are quite erratic/confusing, and I'm also going to work to get these KBs fixed.

HTTP.SYS, Conceptually...

The crux of what is going on is this (rough conceptual details):

HTTP.SYS basically:

  1. Reads data from the networking layer
  2. Parses it
  3. Does some basic HTTP-level validations according to HTTP/1.1 spec
  4. Routes it to a user mode process for request execution

All of these steps happen in kernel mode.

SSL and ReadRawData, conceptually...

Two concepts muck with this architecture in IIS -- SSL and SF_NOTIFY_READ_RAW_DATA filters.  Both concepts require access to the raw data stream prior to it being parsed by HTTP.SYS, meaning that it has to be somehow interjected between steps #1 and #2 above.  Since we cannot just execute existing SF_NOTIFY_READ_RAW_DATA filters (which are user mode DLLs) in kernel mode, this means that HTTP.SYS will need to have some "mechanism" to stream raw data from #1 into a user mode process, let it munge the data however it wants, and then return the data to HTTP.SYS in kernel mode for #2 and continue request handling.  Needless to say, this is quite a privileged operation.

Thus, both SSL and SF_NOTIFY_READ_RAW_DATA rely on this "mechanism" to stream raw data from kernel mode into user mode and back, prior to step #2. Now, HTTP.SYS happens to only support this "mechanism" for a SINGLE user mode process for various technical reasons (including the fact that host headers are not known [SF_NOTIFY_READ_RAW_DATA could change it as well]).

Therefore, all requests that require SSL and/or SF_NOTIFY_READ_RAW_DATA go through this mechanism -- first, the data is read in kernel mode, then transitioned into user mode to munge/decrypt, then transitioned back into kernel mode to parse, and later shuttled back into user mode to execute. Not very pretty, but it works and is not the default case.

IIS6 Process Model and SSL/ReadRawData

Now, what does this have anything to do with IIS6 modes?  Well, in IIS5 Compatibility Mode, the "mechanism" can use inetinfo.exe as the lone user mode process to handle the raw data since both ISAPI Filters and SSL needs to run in that process anyway for compat reasons, and so SF_NOTIFY_READ_RAW_DATA is cleanly supported.

This all breaks down in IIS6 Worker Process Isolation Mode because individual w3wp.exe load their own ISAPI Filters -- so multiple instances of the same Filter DLL can be in memory in separate w3wp.exe -- and what happens if this Filter uses SF_NOTIFY_READ_RAW_DATA ?  We now have >1 process wanting the raw data routed to it, but HTTP.SYS can only route to a single process.  Now, you may argue that we should have spun up one single surrogate process to host all ISAPI Filters and SSL, but this design destroys Worker Process Isolation Mode -- this surrogate process is once again a single point of failure for all w3wp.exe and is no better than the IIS5 process model (substitute inetinfo.exe for the single surrogate process and you get the same looking picture).  All of the other designs at trying to enable SF_NOTIFY_READ_RAW_DATA in IIS6 Worker Process Isolation Mode have other similar but fatal flaws.

The User-Oriented Compromise

So, the whole SF_NOTIFY_READ_RAW_DATA concept just does not work with Worker Process Isolation Mode, and hence we did the next best thing -- disable the broken concept, but give a better alternative that allows 99.9% of people to do what they actually wanted to do and live with breaking that 0.1%:

  1. Let SSL have the lone raw data "mechanism" and host it in lsass.exe to
    have only one process involved in decrypting/encrypting data
  2. Disable SF_NOTIFY_READ_RAW_DATA in IIS6 Worker Process Isolation Mode
  3. Make sure that the new IIS6 feature, HSE_REQ_EXEC_URL, can take care of the 99.9% usage case of SF_NOTIFY_READ_RAW_DATA (which is the munge the incoming request, in particular POST entity body from FORMs)

From my perspective, the main users that are left in the cold by this decision are the people who wrote proprietary non-opaque stream filters, such as custom encryption/compression, and want to run in IIS6 Worker Process Isolation Mode.  There are no solutions in this case other than IIS5 Compatibility Mode.

Your suggestion of "...If you don't want to modify its behavior you should add a new notification type (input and output as well) only for modifying the entity body." was considered at one point, but we rejected it.  We already introduced HSE_REQ_EXEC_URL in ISAPI Extension in IIS6 to handle the 99.9% usage case of modifying the entity body, and ISAPI Filter is not the preferred IIS extension mechanism (ISAPI Extension is far richer and deterministic), so we chose not to add any new filter notifications.

So, if your whole reason for SF_NOTIFY_READ_RAW_DATA is to get access and control of the incoming request, there are better and supported ways of doing this in IIS6.  Configure a ISAPI Extension as a Wildcard Application Mapping and use HSE_REQ_EXEC_URL.  You will notice that HSE_REQ_EXEC_URL is able to modify the entire request that is processed by IIS, including URL, headers, entity body, IIS impersonation token, and some ServerVariable values (like AUTH_TYPE, AUTH_USER, LOGON_USER, etc), and when configured as a Wildcard Application Mapping, it gets first crack at all the requests and can filter/redirect as it pleases. HSE_REQ_EXEC_URL is even able to consume and otherwise munge the entire entity body without the child URL realizing it, and it is fully compatible with SSL.

This is basically what most people want to do on the server, and IIS6 gives it to you in that manner.  We didn't have this in prior versions of IIS, so people have had to hack together Filters using SF_NOTIFY_READ_RAW_DATA to buffer and modify incoming requests (including entity body) -- which doesn't work with SSL, SF_NOTIFY_PREPROC_HEADERS filter and combo ISAPI Extension if you just want to change URL, headers, and fudge some ServerVariable values if you coordinate with SF_NOTIFY_AUTHENTICATION, and so on. HSE_REQ_EXEC_URL has none of those hacky limitations -- you simple change the entire request that gets executed by IIS and then tell IIS to execute it.

You can still do all those hacky things in IIS5 Compatibility Mode (that's what compatibility is for), but we highly discourage it in IIS6 Worker Process Isolation Mode because there are better alternatives.  If we didn't have better alternatives, we would have made sure that SF_NOTIFY_READ_RAW_DATA remained the same in all modes.

If you want to treat HTTP logically, such as URL, headers, entity-body, you can either do the hard work of parsing through data stream as with the APIs you're asking about, or you can use the logical API of HSE_REQ_EXEC_URL to modify the exact same thing prior to IIS executing the request.  I leave it up to you to decide which is easier and makes more sense.

Conclusion

Personally, I suggest that you don't get hung up on SF_NOTIFY_READ_RAW_DATA nor HTTPAPI -- you're really not "losing" anything important other than a big hairball mess.

As for your statement in #2 about IIS6 using undocumented HTTP API to process the raw data -- I am not a lawyer nor am I authoritative on the subject, but I can tell you that:

  1. The IIS and HTTP.SYS teams spent a lot of time with the lawyers, who grilled us over all of this
  2. Legally, IIS6 is considered a part of Windows Server 2003 OS; thus this is merely a private API between two OS components -- which is perfectly acceptable by the consent decree

//David

Comments

  • Anonymous
    April 28, 2006
    I finally have enough blog entries about various portions of IIS6 request processing that I can stitch...
  • Anonymous
    May 10, 2006
    This is a frequently asked question about IIS6 extensibility - how to access the request entity body...
  • Anonymous
    June 14, 2006
    You state that SF_NOTIFY_READ_RAW_DATA is cleanly supported in IIS5 Compatibility Mode. Is this true for encrypted data as well, because I cannot get my filter to work in this mode unless I disable the notification for SF_NOTIFY_READ_RAW_DATA.
  • Anonymous
    June 14, 2006
    Steven - Whether SF_NOTIFY_READ_RAW_DATA and SSL works will depend on what your filter is doing.

    Prior to IIS6, SSL was implemented as a ReadRawData filter, which has its own implementation-specific and messy interaction with other ReadRawData filters, and not everything is possible.

    As publicly documented on MSDN, the easiest way to support SSL and access/modify request headers/entity-body is to use HSE_REQ_EXEC_URL and ISAPI Extension configured as wildcard application mapping. It works in all cases where you use SF_NOTIFY_READ_RAW_DATA for opaque filtering and even others where SF_NOTIFY_READ_RAW_DATA and SSL fail.

    //David
  • Anonymous
    June 15, 2006
    Thanks for the response.  In addition to writing the new ISAPI extension, I need to get the current  filter to work in IIS 5.0 isolation mode. Curiously it only works if I either disable or enable both notifications for SF_NOTIFY_NONSECURE_PORT and SF_NOTIFY_SECURE_PORT. In which case I get both encrypted and non-encyrpted pages. If I just enable just SF_NOTIFY_SECURE_PORT, my HttpFilterProc is never called.

    My filter needs to see, but not modify, all of the request and response data for just encrypted data.
  • Anonymous
    June 15, 2006
    The comment has been removed
  • Anonymous
    June 16, 2006
    Thank you. We don't intend to maintain two code bases but currently we only have the filter right now. The extension is a prototype that is actually an extension and a filter. The filter portion gets notifications for send_raw_data.  

    I am assuming this was done this way because you can't actually get the response from the application in the extension. If this is not the case, please let me know and I will research how to do this.
  • Anonymous
    June 16, 2006
    Steven - yeah, nothing really "owns" a response the same way that a handler "owns" the request, so you just have a synchronous serial event notification style of interaction like SF_NOTIFY_SEND_RAW_DATA.

    We considered adding response buffering in IIS6, in which case it would be possible for the handler to say "I am interested in seeing the entire response when the child is done, as a big buffer", but we did not end up adding it for various other reasons.

    //David
  • Anonymous
    June 20, 2006
    Question:
    In IIS6, is there a way to have a top-level filter run in a process space separate from each...