Freigeben über

HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net

  • Artikel

Developers frequently confuse Request Headers, Response Headers, and Server Variables as well as the appropriate syntax to retrieve/manipulate each of them, depending on the API (ISAPI, ASP, and ASP.Net). I am going to clarify all of this right now. :-)

Question:

So I've managed (thanks to your samples) to create a filter that adds a header. I call it SM_USER. When I use ASP to enumerate all the headers I can see it as HTTP_SM_USER in the ALL_HTTP and ALL_RAW header (with a value set).

Unfortunately though when I list out the HTTP_SM_USER variable there is no value for it.

I am setting it in the onPreProcessHeaders with a call sort of like this

fRet = pPPH->SetHeader(pfc, szMyHeader, szMyHeaderValue);

The question is: What's happening here? Why can I see it in the ALL but not when I list out the particular header?

Cheers,

Answer:

Request Headers, Response Headers, and Server Variables represent three totally different logical concepts available to server-side applications (though not all APIs expose these logical concepts), and you manipulate them in different ways.

The main syntactic sugar which exists for legacy compat reasons with CGI is that Server Variables ALSO allow retrieval of certain Request Headers when using the special HTTP_ prefix.

The following is how I treat the logical concepts...

Request Headers

Request Headers represent the header name/value pairs that the HTTP client, such as a web browser, sent to the server. A typical request looks like the following, and request headers are highlighted:

 GET / HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n
\r\n

Response Headers

Response Headers represent the header name/value pairs that the HTTP server, such as a web server, sends to the client. A typical response looks like the following, and response headers are highlighted:

 HTTP/1.1 200 OK\r\n
Content-Type: text/html\r\n
Server: Microsoft-IIS/6.0\r\n
Date: Thu, 20 Apr 2006 09:00:00 GMT\r\n
My-Response-Header: Dash\r\n
My_Response_Header: Underscore\r\n
Content-Length: 11\r\n
\r\n
Hello World

Server Variables

Server Variables represent runtime state available to the server-side application for request processing. The list of variables and syntax available on IIS is described here. They include state like:

  • SCRIPT_NAME (client-requested URL, such as "/")
  • AUTH_TYPE (type of authentication performed)
  • etc

As stated earlier, it also supports syntactic sugar to access Request Headers via the HTTP_ prefix, in accordance to the CGI 1.1 Specification.

The CGI Specification for Accessing Request Headers

The CGI 1.1 Specification defines the HTTP_ prefix to retrieve Request Headers. Dashes in Header Name transform into Underscores with HTTP_ prefix prepended and make them available as Server Variables (in the CGI context, available as Environment Variables).

In other words, the HTTP_ACCEPT_ENCODING Server Variable name retrieves the Accept-Encoding: Request Header.

The astute reader should notice a quirk between the HTTP and CGI specifications - in particular, how does one retrieve the Accept_Encoding: Request Header in CGI???

Remember, HTTP Request Header names are message-headers, which by definition are (I have collected all the relevant BNF definitions for reference):

        message-header = field-name ":" [ field-value ]
       field-name     = token
       field-value    = *( field-content | LWS )
       field-content  = <the OCTETs making up the field-value
                        and consisting of either *TEXT or combinations
                        of token, separators, and quoted-string>

       token          = 1*<any CHAR except CTLs or separators>
       separators     = "(" | ")" | "<" | ">" | "@"
                      | "," | ";" | ":" | "\" | <">
                      | "/" | "[" | "]" | "?" | "="
                      | "{" | "}" | SP | HT

       OCTET          = <any 8-bit sequence of data>
       CHAR           = <any US-ASCII character (octets 0 - 127)>
       CTL            = <any US-ASCII control character
                        (octets 0 - 31) and DEL (127)>
       SP             = <US-ASCII SP, space (32)>
       HT             = <US-ASCII HT, horizontal-tab (9)>

       LWS            = [CRLF] 1*( SP | HT )
       TEXT           = <any OCTET except CTLs,
                        but including LWS>

According to HTTP specifications, header names can include any CHAR except CTLs or separators. In particular, both "-" (dash) and "_" (underscore) are valid characters in header name.

The problem should be clear now. The CGI 1.1 Specification defines a substitution of "-" in Request Header to "_" in Server Varible names, but what is the substitution of "_" in Request Headers in Server Variable??? It cannot be "_" in Server Variable names, and it is conveniently undefined...

Oops! Did we just spot a flaw in a sacred specification underlying major Internet Applications? You bet. Oh no, heavens forbid that there is a flaw in a publicly implemented specification! The sky is falling!

And you can also bet that it is not the only one in existence... so the next time you think that a peculiar software behavior is due to a software bug and not a specification bug, think again... because believe it or not, even the W3C recommendations are not flawless. Remember, to err is human, and humans write those documents and software.

When troubleshooting, I always recommend to think and evaluate the situation instead of assuming one or another is magically sacred. For example, developers tend to assume that their code is perfect and that the flaws must be from the system.

The Solution in IIS 6.0

Yes, we decided to do something unspecified in IIS 6.0 to resolve this situation. Otherwise, how can you possibly retrieve the SM_USER request header within ASP?

We introduced the HEADER_ prefix for Server Variable names which uses the name as-is to retrieve request headers. In other words:

  • ServerVariable( "HEADER_SM_USER" ) retrieves SM_USER: Request Header
  • ServerVariable( "HEADER_SM-USER" ) retrieves SM-USER: Request Header

It is a neat solution to the problem. Yes, it is unspecified, but I think users are better off with than without it.

Table of Operations

Whew... soapbox aside, the following table categorizes the available syntax to retrieve the three logical concepts of Request Header, Response Header, and Server Variable. Without parsing, of course...

API / Concept Request Header Response Header Server Variable
ISAPI Filter PreprocHeaders:GetHeader("Header_As-is:")AuthComplete:GetHeader("Header_As-is:")Any event after PreprocHeaders:GetServerVariable:HTTP_ prefixHEADER_ prefix SendResponse:GetHeader("Header_As-is:")

GetServerVariable:HTTP_ prefixHEADER_ prefix
ISAPI Extension

GetServerVariable:HTTP_ prefixHEADER_ prefix

 Not Possible

GetServerVariable:HTTP_ prefixHEADER_ prefix
ASP

GetServerVariable:HTTP_ prefixHEADER_ prefix

 Not Possible

GetServerVariable:HTTP_ prefixHEADER_ prefix
ASP.Net Request.Headers("Header_As-is")GetServerVariable:HTTP_ prefixHEADER_ prefix Not Possible

GetServerVariable:HTTP_ prefixHEADER_ prefix

The key take-aways:

  • ISAPI Filters can retrieve everything as-is, assuming you are in the right filter event. You just need to remember that header names MUST have the ":" appended, while GetServerVariable() calls using HTTP_ or HEADER_ prefix do NOT have ":"
  • GetServerVariable() calls using HTTP_ or HEADER_ prefix work everywhere (after PreprocHeaders event for ISAPI Filters) to retrieve request headers, especially when using HEADER_ prefix
  • ASP.Net offers the Request.Headers collection to give you parsed access to the Request Headers
  • Notice that ASP and ASP.Net behavior mirror that of ISAPI Extension... because they are implemented as ISAPI Extension DLLs on IIS

ASP Example:

For example, using WFetch, try making the following raw request:

 GET /GetServerVariable.asp HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n
\r\n

And have the following source code for /GetServerVariable.asp

 <%
headerDash = "My-Request-Header"
headerUnder = "My_Request_Header"
HTTP_headerDash = "HTTP_" & headerDash
HTTP_headerUnder = "HTTP_" & headerUnder
HEADER_headerDash = "HEADER_" & headerDash
HEADER_headerUnder = "HEADER_" & headerUnder

Response.Write( HTTP_headerDash & " = " & Request.ServerVariables( HTTP_headerDash ) & VbCrLf )
Response.Write( HTTP_headerUnder & " = " & Request.ServerVariables( HTTP_headerUnder ) & VbCrLf )
Response.Write( HEADER_headerDash & " = " & Request.ServerVariables( HEADER_headerDash ) & VbCrLf )
Response.Write( HEADER_headerUnder & " = " & Request.ServerVariables( HEADER_headerUnder ) & VbCrLf )
%>

You get the following output:

 HTTP_My-Request-Header = Dash\r\n
HTTP_My_Request_Header = Dash\r\n
HEADER_My-Request-Header = Dash\r\n
HEADER_My_Request_Header = Underscore\r\n

Notice that HTTP_ syntax only retrieves the header name with dashes, while HEADER_ syntax retrieves both headers with dashes and underscores individually.

ASP.Net Example

For example, using WFetch, try making the following raw request:

 GET /GetServerVariable.aspx HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n
\r\n

Have the following source code for /GetServerVariable.aspx

 <%
dim headerDash,headerUnder, HTTP_headerDash, HTTP_headerUnder, HEADER_headerDash, HEADER_headerUnder

headerDash = "My-Request-Header"
headerUnder = "My_Request_Header"
HTTP_headerDash = "HTTP_" & headerDash
HTTP_headerUnder = "HTTP_" & headerUnder
HEADER_headerDash = "HEADER_" & headerDash
HEADER_headerUnder = "HEADER_" & headerUnder

Response.Write( HTTP_headerDash & " = " & Request.ServerVariables( HTTP_headerDash ) & VbCrLf )
Response.Write( HTTP_headerUnder & " = " & Request.ServerVariables( HTTP_headerUnder ) & VbCrLf )
Response.Write( HEADER_headerDash & " = " & Request.ServerVariables( HEADER_headerDash ) & VbCrLf )
Response.Write( HEADER_headerUnder & " = " & Request.ServerVariables( HEADER_headerUnder ) & VbCrLf )
Response.Write( "Header(" & headerDash & ") = " & Request.Headers( headerDash ) & VbCrLf )
Response.Write( "Header(" & headerUnder & ") = " & Request.Headers( headerUnder ) & VbCrLf )
%>

You get the following output:

 HTTP_My-Request-Header = \r\n
HTTP_My_Request_Header = Dash\r\n
HEADER_My-Request-Header = \r\n
HEADER_My_Request_Header = \r\n
Header(My-Request-Header) = Dash\r\n
Header(My_Request_Header) = Underscore\r\n

With ASP.Net, the Headers collection is the only way to get all the header values. HTTP_ prefix only retrieves the header name with dashes, as expected.

Conclusion

Whew... a very long winded entry. :-)

But... I wanted to be thorough on the subject of Request Headers, Response Headers, and Server Variables when it comes to ISAPI, ASP, and ASP.Net. So, now you have everything...

//David

Comments

  • Anonymous
    April 20, 2006
    The comment has been removed

  • Anonymous
    April 20, 2006
    <trivia>
    The HTTP_SM_USER header is used by Netegrity SiteMinder, for holding a single sign-on user ID.
    </trivia>

    In a previous project, we ran into problems where the - to _ replacement was causing a bit of a pain, in our ASP code that needed to parse the user's SSO.

    However, our ever-friendly SiteMinder team changed the header to HTTP_SMUSER, thus alleviating our woes.

  • Anonymous
    April 21, 2006
    Jeff - hehe. My job is done, then. ;-)

    I actually started writing more code snippets in ISAPI, ASP, and ASP.Net before I decided to tone it down and just focus on the basics. I'll leave those details to another time.

    //David

  • Anonymous
    April 21, 2006
    The comment has been removed

  • Anonymous
    April 21, 2006
    ... I don't see the ambiguity.

    OK, Some-Header: and Some_Header: both map to the same environment variable.  Fine.  That just means the header-to-environment mapping isn't one-to-one.  We knew that anyway... environment variables aren't case-sensitive, for example.

    So as far as CGI is concerned, the following are all the same:

    Some-Header: a
    Some_Header: a
    sOmE-HeAdEr: a

    Isn't this only a problem when you have two headers, both important to your app, that differ only in case or "- vs. _"?

  • Anonymous
    April 21, 2006
    Maurits - The classic problem here is when you are an ISAPI, like ASP or ASP.Net, and want to retrieve a Request Header with "_" (underscore) using Server Variable... because you cannot.

    For example, SM_USER used by SiteMinder. You will have to parse the ALL_RAW Server Variable to get it in ASP. On IIS6, you can finally use ServerVariable( "HEADER_SM_USER" ) and get it declaratively, without parsing.

    //David

  • Anonymous
    April 22, 2006
    This much is clear:

    * On IIS, HTTP_SM_USER does not catch the value of SM_User:

    You imply that * is a weakness in the CGI spec. In particular, you claim that there is no CGI means to access the value of SM_User: via server/environment variables.

    I disagree.  I think there is a CGI means, and HTTP_SM_USER is precisely it.

    Therefore I consider * a bug in IIS.
    See this post:
    http://channel9.msdn.com/ShowPost.aspx?PostID=184522
    And this repro/analysis:
    http://www.geocities.com/mvaneerde/sm_user.txt

    Note at the end of the analysis that there is at least one third-party CGI ISAPI that correctly pulls SM_User: from HTTP_SM_USER.

  • Anonymous
    April 22, 2006
    It seems this is a known issue:
    http://support.microsoft.com/default.aspx?kbid=294217

  • Anonymous
    April 23, 2006
    The comment has been removed

  • Anonymous
    April 24, 2006
    The comment has been removed

  • Anonymous
    April 24, 2006
    There's a bug in the KB article, too.  It says "Retrieve and parse the HTTP_ALL header variable" instead of "Retrieve and parse the ALL_HTTP header variable" -- I'll report that through the appropriate method.

  • Anonymous
    April 24, 2006
    Maurits - I think we are merely arguing whether the glass is half empty or half full.

    One can argue that the CGI specification says that HTTP_SM_USER should retrieve SM_USER or SM-USER, whichever one exists, and if both exist, the behavior is unspecified.

    Or HTTP_SM_USER only retrieves SM-User, there is no way to retrieve SM_USER, and there is no unspecified behavior.

    Yes, the specification did not say "you cannot retrieve headers containing '_'", so you may be inclined to say that IIS interpretation is too narrow. However, it also did not say "you cannot retrieve headers containing CTRL characters", either... so it is a matter of whether one wants unspecified or underspecified behavior.

    Neither option is more "correct" than other, so until the CGI specification clarifies the confusion, there is no way for one to say "this is a bug in X". This is the problem with an underspecified specification - things get too open for interpretation. ;-)

    //David

  • Anonymous
    April 25, 2006
    FWIW, here's the CGI/1.1 RFC:
    http://www.ietf.org/rfc/rfc3875.txt
    ...
    4.1.18.  Protocol-Specific Meta-Variables
    ...
    The HTTP header field name is converted to upper case, has all occurrences of "-" replaced with "" and has "HTTP" prepended to give the meta-variable name

  • Anonymous
    April 25, 2006
    Maurits - Good information...

    But, this is not a specification... it's more like an "oh, we did it this way; we think it should be codified".

    And it still does not deal with what happens with retrieving header values for requests containing both SM_USER and SM-USER headers. tsk tsk.

    And it is dated after all versions of IIS under discussion, so kinda moot point except for maybe IIS7, but it's not an endorsed specification, so it is hard to rally behind it

    //David

  • Anonymous
    April 26, 2006
    ... and both of the authors work for Apache ;)

  • Anonymous
    April 26, 2006
    There's also this Internet-Draft from 1999, by IBM and E*Trade...
    http://cgi-spec.golux.com/draft-coar-cgi-v11-03-clean.html#6.1.5
    Each HTTP header field name is converted to upper case, has all occurrences of "-" replaced with "", and has "HTTP" prepended to form the metavariable name.
    ...
    If multiple header fields with the same field-name are received then the server MUST rewrite them as though they had been received as a single header field having the same semantics before being represented in a metavariable.
    </quote>

    That last sentence is wonderfully vague :)

    This draft hasn't had a great deal of work done on it recently, though :(

  • Anonymous
    April 26, 2006
    Maurits - hehe... yeah. It's like patting yourself on the back and writing the spec after the implementation. ;-)

    The vague sentence is dealing with what happens when the browser sends multiple SM_USER headers on the same request (i.e. HTTP spec says to merge their values into one value, comma-delimited)... which is also wonderful information to have in a specification... but sigh, it still ignores the SM_USER and SM-USER ambiguity.

    I wonder if people just copy/paste the original sentences of the spec and then just augment/embelish the parts they are interested in...

    //David

  • Anonymous
    April 28, 2006
    I finally have enough blog entries about various portions of IIS6 request processing that I can stitch...

  • Anonymous
    May 08, 2006
    Sorry, but I think that the spec being vague and broken is a very poor excuse in this case.

    As has been stated before, the spec misses the case of two different http headers that will result in the same converted value.

    It has no ambiguity whatsoever oh the handling of 'SM_USER': zero, nil, zilch. Anybody doing his best for implementing it, should treat it as valid. And even if the spec had been twice as shady and broken, the main goal for everybody is interoperability, isn't it?

  • Anonymous
    May 11, 2006
    gaetano - Good point, but I hope you agree that vague/broken spec is the very bane against interoperability.

    As a general rule, when you have communication between two parties, miscommunication and misinterpretations are bound to happen, and having vague/broken specifications (which basically set the foundation of common understanding) does NOT help.

    We can argue about relative correctness of miscommunication all day long, but at the end of the day, I think it is the root vagueness that matters and NOT the various instance-manifestations of that vagueness. If you solve the root problem (mapping of headers to server variables), the instance-problems naturally solve themselves. Those are sound design principles.

    Now, I caution you against mistaking miscommunication for "not doing his/her best for implementing it". Who says that we are not doing our best?

    I mean, you should agree that EVEN IF everyone is doing their best-effort for supporting interoperability, any vagueness in the specification that causes miscommunication destroys interoperability... and you somehow want to imply that someone did NOT do their best effort? Give me a break. That logic is down-right arrogant and insulting.

    My statement is that mistakes/ambiguity in specifications happen more than you think, and they introduce mis-communication which can affect interoperability while implying NOTHING about the effort put forth by the implementors.

    //David

  • Anonymous
    May 22, 2006
    David, I've found your efforts to explain the situation to be very useful. Our company has struggled to support customers doing single sign-on between our product and Netegrity SiteMinder because of the fact that the header containing the SiteMinder Id (by default, without customizations) is either sm_user, or sm-user and we see them (I think correctly) as different.

    Thus our customers get frustrated and I had to put a debug page into our product just to show our customers that its not our fault they spelled the header wrong.

    I've also done some research to educate myself why Tomcat shows headers in one case on WebSphere on Unix shows them in another case (only to later find in the spec that case is not relevant). Your thread here was helpful  as you explain why the ASP sample code I downloaded that uses HTTP_ might be contributing to our confusion.

  • Anonymous
    August 07, 2006
    Hi,
         I am very new in ISAPI but i have my very urgent requiremnet that i have to set a cutsom varibale and then again need to fetch it on next page or any other page like

    suppose first i wll create a custom server valriable like "username" and on in other page i will just check that it have any value or not
    like
    if request.servervariable("http_username")<>"" then
    some processing
    end if

    as im new so please if you explain me step by step how i can create ISAPI as also i dont know much more abour ISAPI please give me full step that how i can craete it and then again how i can access it in my asp page.

    Thanks :
    Arun Chaudhary

  • Anonymous
    August 07, 2006
    Arun - I suggest using Session Variables to accomplish what you want.

    Using ISAPI to set custom variables will not do what you describe because such variables do not "carry over" between pages.

    I suggest searching the web for basic documents on how web technologies work.

    //David

  • Anonymous
    September 12, 2006
    I am trying to retrieve an header value and empty it before it is written to the IIS log. Based on the code example given I am able to access the header. But how to modify it. Primarily we are having issue with Site mind assertions being logged in the IIS and we do not want to have it logged.

  • Anonymous
    May 29, 2007
    How to Retreive Request Headers using VB.net.can any one knows the synatx for that.if yes please provide with a sample code for that

  • Anonymous
    September 24, 2007
    How to forward the request headers when we want redirect the request.

  • Anonymous
    April 25, 2008
    Duuuuuuuude....  this was really helpful!  Thanks, man!

  • Anonymous
    September 01, 2008
    This is an excellent post, I am surprised to see the IIS behavior and understood now. Parsed all_http to solve my problem. Thanks a lot David.  Thanks!

  • Anonymous
    May 20, 2009
    Great information David!  Quick question for you.  I see that you can set a value for a header in an ISAPI filter using "SetHeader".  How can you do this using an ISPI Extension?  I've got an existing extension that we're using to block SQL injection.  We want to modify it to prevent XSS using header variables like the HTTP_ACCEPT_LANGUAGE header.

  • Anonymous
    May 20, 2009
    Great information David!  Quick question for you.  I see that you can set a value for a header in an ISAPI filter using "SetHeader".  How can you do this using an ISPI Extension?  I've got an existing extension that we're using to block SQL injection.  We want to modify it to prevent XSS using header variables like the HTTP_ACCEPT_LANGUAGE header.

  • Anonymous
    September 15, 2013
    My iis7.5 is give me permision denied error..