Why some ISAPI Filter events trigger multiple times per request

Article
03/12/2006

Question:

Hi everyone,

I wrote a ISAPI filter DLL to process text/html content. Whenever there's a request for a .php file, the OnUrlMap gets called 3 times whereas .html or .asp files causes the right behaviour in Filter (i.e. - OnUrlMap gets called once only). How do I get rid off this problem? Below is the excerpt of the code -

DWORD OnSendRawData(...) {
//Process pvRawData....
........
//Writes HTTP HEADER to client
pfc->WriteClient(....);
//Writes BODY
//pfc->WriteClient(...);
}

When the code executes for HTTP HEADER, the function gets suspended and the second request of the .php file calls OnSendRawData. When any Win32 API gets called in the OnSendRawData code, the second request is suspended and the first one is resumed (i.e.- starts writing the BODY).

As both are same actually request, it messed up the private (or content specific) data.

So, How to stop multiple request in OnUrlMap?

Answer:

Actually, the behaviors you describe are all by-design.

There is no guarantee that every event fires at least once per request. There is also no guarantee that every event fires at most once. In fact, the following events can fire multiple times per request:

SF_NOTIFY_READ_RAW_DATA
SF_NOTIFY_URL_MAP (MFC Wizard: OnUrlMap)
SF_NOTIFY_SEND_RESPONSE (MFC Wizard: OnSendResponse)
SF_NOTIFY_SEND_RAW_DATA (MFC Wizard: OnSendRawData)

Thus, your assumption that the "right" behavior for OnUrlMap is getting called once per request... is incorrect. You also assume that OnSendRawData gets called once per request and that no user code can trigger multiple calls to it... and that is incorrect.

Details on the Events

This is how the events actually work:

SF_NOTIFY_READ_RAW_DATA happens each time a data packet is read from the client (assuming either IIS6 running in IIS5 Compatibility Mode or prior IIS versions). This happens at least once per request (to read in the headers), and depending on how the headers are sent and the size of the request entity body, additional events can be triggered by IIS buffering up to W3SVC/UploadReadAheadSize or by ISAPIs making ReadClient() calls. Only the first trigger of this event is guaranteed to happen before any response has been sent to the client; any other trigger of this event can obviously happen at any time during request processing.

SF_NOTIFY_URL_MAP happens each time IIS or user-code requests a URL-to-physical mapping of a URL. This happens at least once per request because IIS has to do the URL-to-physical mapping to obtain the physical resource to process. It can also happen for other reasons, such as:

Wildcard Application mappings calling HSE_REQ_EXEC_URL - explicit re-execution of the URL from parent to child, so you have one resolution for parent URL and another for child URL.
Default Document resolution, which you can treat like "internal HSE_REQ_EXEC_URL" which re-executes from the parent URL of "/" to the child URL of "/resolved.default.document".
Retrieval of server variables like PATH_TRANSLATED by an ISAPI (which includes ASP, ASP.Net, PHP, etc).

SF_NOTIFY_SEND_RESPONSE happens each time IIS or user-code sends structured response headers.

This normally happens once per request when IIS sends out the response.
However, an ISAPI Filter or ISAPI Extension can also trigger it each time it calls something like SF_REQ_SEND_RESPONSE_HEADER or HSE_REQ_SEND_RESPONSE_HEADER.
An ISAPI Filter can prevent this event from triggering if it uses WriteClient to generate a response and then finishes the request with SF_STATUS_REQ_FINISHED before anything else triggers structured response headers to be sent.

SF_NOTIFY_SEND_RAW_DATA happens each time IIS or user-code sends data to the client.

This happens at least once per request if a response has been generated and sent.
It also triggers each time an ISAPI Filter or ISAPI Extension calls something like WriteClient() as well as other ServerSupportFunctions which write structured data back to the client, like HSE_REQ_SEND_RESPONSE_HEADER, HSE_REQ_TRANSMIT_FILE, etc.

Conclusion

I am not certain whether your Filter has bugs in OnUrlMap or OnSendRawData, but the code snippet you show is definitely wrong. In general, it is a bad idea to call WriteClient() from OnSendRawData because WriteClient() triggers OnSendRawData... so your code has to somehow WriteClient HTTP_HEADER, and when that WriteClient() triggers OnSendRawData, remember to NOT WriteClient HTTP_HEADER... and so on. Yeah, complicated. Good luck.

Since you are trying to process response content, I assume you are trying to buffer the response with OnSendRawData, make the appropriate modifications, and then flush the modifications to the client. There are a couple of approaches:

Buffer every OnSendRawData event into a huge buffer and suppress the data from being sent by OnSendRawData. Then, in either OnLog or OnEndOfRequest, perform the modification and then flush the entire response buffer with WriteClient(), taking care to NOT buffer the OnSendRawData event triggered by this final WriteClient().
Buffer consecutive OnSendRawData events to perform in-flight modifications as necessary, and then immediately flush the changes with the pvRawData of the consecutive OnSendRawData. Final flush is made with WriteClient from OnLog or OnEndOfRequest, taking care to not buffer the OnSendRawData event triggered by this final WriteClient.

The first approach is easy to implement but buffers the entire response, which can be resource-intensive on large responses/downloads. The second approach is more complicated to implement but only buffers the necessary amount of data to perform necessary modifications.

//David

Comments

Anonymous
June 14, 2006
Hi,

you mention that some requests can be executed more than one time. Do you know if the notification OnLog also can be executed more than once per request?

You state that 'There is no guarantee that every event fires at least once per request'. Does that mean that there are events that sometimes not will be executed/fired? Which events might not be executed?

Thanks in advance!
Anonymous
June 14, 2006
The comment has been removed
Anonymous
June 20, 2006
The comment has been removed
Anonymous
June 20, 2006
JC - Which IIS version are you running, and are you running more than 1 ISAPI Filter at either global or site level.

In general, you want to designate your data as either per-request or per-connection and then manage its lifetime appropriately.
- Per-connection memory can be handled with pfc->AllocMem()
- Per-request memory can be handled with PreprocHeaders and Log/EndOfRequest

Also, you need to be aware that pfc->pFilterContext is PER-CONNECTION and NOT PER-REQUEST, which has special implications when servicing keep-alive or pipelined requests. For example:
- keep-alive requests mean that one connection and pFilterContext could be in use for 2 or more requests
- pipelined requests mean that one connection and pFilterContext could be used for 2 or more consecutive requests

Finally, you probably want to minimize the amount of memory creation/deletion done by an ISAPI Filter because it can fragment memory. Pre-allocating memory and then re-using is the preferred approach.

//David
Anonymous
May 06, 2008
Hi David, I am a little confused with the l differences in PER-CONNECTION and PER_REQUEST. As mentioned, it is a matter of scope, so I am thinking that is I am writing a ISAPI Filter to change elements ( special tags ) in the reponse back to the client, and they are all sliced up into a series of includes ( SSI includes ) , then it would be wise to take the PER-REQUEST Route???? Here is some code, very simple, but it helps to try to show what I am doing. Thoughts on how to make the whole body of the response available, in an effiecent manner ? Also in a prior blog entry you mentioned content-size is important to note. Does this mean that I am replacing these tags with lets say a paragraph or a few lines of text, I now have to change the response header ( content-length: ) as well? I would assume so. #include <string> #include <windows.h> #include <stdio.h> #include <stdlib.h> #include <httpfilt.h> BOOL WINAPI __stdcall GetFilterVersion(HTTP_FILTER_VERSION pVer) { / Specify the types and order of notification */ pVer->dwFlags = (SF_NOTIFY_SEND_RAW_DATA); pVer->dwFilterVersion = HTTP_FILTER_REVISION; strcpy_s(pVer->lpszFilterDesc, "PreProcessor filter, Version 1.0"); return TRUE; } DWORD WINAPI __stdcall HttpFilterProc(HTTP_FILTER_CONTEXT *pfc, DWORD NotificationType, VOID *pvData) { CHAR in; CHAR out[10000]; DWORD cbBuffer, cbtemp; signed int newCharCount, contentStartPos; switch (NotificationType) { case SF_NOTIFY_SEND_RAW_DATA : PHTTP_FILTER_RAW_DATA pResponse; pResponse = (PHTTP_FILTER_RAW_DATA)pvData; in = (CHAR)pResponse->pvInData; cbBuffer = 0; //Skips the header. for( ;cbBuffer<pResponse->cbInData-2;cbBuffer++){ if (in[cbBuffer] == 'n' && in[cbBuffer + 2] == 'n'){ cbBuffer +=3; break; } } //Verifies that the requested file is an html file. for (cbtemp = 0; cbtemp < (cbBuffer - 3); cbtemp++) { if (in[cbtemp] == '/' && in[cbtemp + 1] == 'h' && in[cbtemp + 2] == 't' && in[cbtemp + 3] == 'm') { pfc->pFilterContext = (VOID )2; break; } } if (cbtemp == cbBuffer) pfc->pFilterContext = 0; / not an html file */ //Process the file if(pfc->pFilterContext){ newCharCount = 0; contentStartPos = cbBuffer; for ( ;cbBuffer < pResponse->cbInData-2;cbBuffer++){ if (in[cbBuffer] == '<' && in[cbBuffer + 1] == '&'){ for (cbtemp = cbBuffer+2; cbtemp < pResponse->cbInData-2; cbtemp++){ if (in[cbtemp] == '&' && in[cbtemp + 1] == '>'){ break; } } cbBuffer = cbtemp + 2; continue; }else{ out[newCharCount] = in[cbBuffer]; newCharCount++; } } pResponse->cbInData = contentStartPos + newCharCount+1; for (cbtemp=0 ;cbtemp < newCharCount; cbtemp++){ in[cbtemp + contentStartPos] = out[cbtemp]; } } break; default: break; } return SF_STATUS_REQ_HANDLED_NOTIFICATION; } Thank you again David. And if it helps, I think you blog and your reccomendations are both clear, helpful, and insightful.
Anonymous
May 06, 2008
Mario - yes, if you want to modify the outbound response, you have to buffer sufficient amount of entity body to make your modification, and depending on the response transfer method (either Content-Length, or Transfer-Encoding: chunked), you will have to make the appropriate modifications. For example, your code will fail right now if the double rn is sent over two HTTP packets. What you want to do is:

Buffer enough raw data over possibly multiple triggers of SF_NOTIFY_SEND_RAW_DATA to determine if the response header has "Content-Length:", "Transfer-Encoding: chunked", or neither header 2a. If Content-Length, then you must continue to buffer everything, perform your modifications, recalculate the new entity body length, and modify "Content-Length" to reflect the new length 2b. If Transfer-Encoding: chunked, then you can easily insert chunks if you're just inserting data. If modifying existing data, then you have to buffer and modify as in 2a, this time having to update the chunk's length header. In all cases, your buffering has to be a sliding one that crosses two SF_NOTIFY_SEND_RAW_DATA because the thing you are looking to replace may come across two different event notifications -- for example, </ht comes on the first event and ml> comes on the second event, so if you want to replace </html>, you have to do this across the two events. //David

Anonymous
May 06, 2008
Thank you David, That was concise and to the piont, and as mentioned in other posts, this is a duanting task. Would it say to assume the following as I move in this task.

Regardless of how many event triggers, I will always have access to the Response header to calculate and modify the appropiate header ( "Transfer-type", "Content-length" )?
As per this entry, what is the wisest way to use to ensure safe and efficient ways of storing the entire body , across multiple child exec requests ( SSI Includes ). I would take a guess that as per you above mentioned cautionary notes, PER-CONNECTION would be the preffered method
Is there a Event handler that I could use to find out when the EndOfRequest has been reached? Thank you again Mario

Anonymous
May 08, 2008
Mario - modifying the raw response data to perform logical HTTP operations is very difficult to do correctly and impossible to do performantly. SendRawData filter turns off just about all the IIS output caches and optimizations and turns async IO paths synchronous. For example, TransmitFile used to do optimal user-kernel transitions but NOT when SendRawData Filter is involved, even if the filter does nothing.

You must buffer the response header at the very beginning SF_NOTIFY_SEND_RAW_DATA and prevent it from being sent -- since you likely have to modify it.
Using pfc->pFilterContext to store the buffered response header AND entity body is required for efficiency. If you do not use it, you would have to use global read/write variables, and that introduces read/write locks and thus contention into the critical data output pathway across the entire server... which would be horrendous.
SF_NOTIFY_END_OF_REQUEST tells you that the request would not generate any more SF_NOTIFY_SEND_RAW_DATA. However, you would have to WriteClient() your buffered response in that event to flush the response back to the client, which itself would generate SF_NOTIFY_SEND_RAW_DATA... so if you are not careful, you would buffer everything and emit nothing. //David

Share via