When Troubleshooting Use All the Tools at Your Disposal (Part 2)

Just this week I had another very interesting case that allowed me to really dig into some data and it took the use of multiple tools to figure it out. This blog is targeted at folks that are either new to the IT field, thinking about getting in to the field, or are always looking to improve their troubleshooting skills.

Problem

Web proxy clients using TMG 2010 are not able to browse a certain website.  Clients using Internet Explorer would see “This page cannot be displayed” error.

Clients not using TMG 2010 as their proxy can browse the site normally.

Data Gathering and Analysis

As with most TMG issue the first place I looked is Live Logging. It’s very easy to narrow it down by querying for IP address that the client is using. Live Logging will allow me to see in real time what is being returned to TMG by the remote server and what TMG is returning to the client. It will also tell me right off the bat if TMG is Denying the request for some reason. Figure 1 below shows you what Live Logging reveals. (Note: I removed the Destination IP and the Domain name from the Request field to protect the guilty)

Fig. 1

The first thing we notice is that TMG is allowing the request but is getting back an HTTP Status Code of 301 Moved Permanently when proxying the request from the client.

I wanted to get additional information that Live Logging was not giving me so I set up Netmon 3.4 to get a trace on the wire.

Request from TMG to misbehaving website (Figure 2)

Fig. 2

Response from misbehaving website to TMG (Figure 3)

Fig. 3

Notice that we get a 301 Moved Permanently that tells us to go right back to the same place we already requested. Again, I changed the website name to protect the guilty. So the reason our web proxy client is getting the error is that the remote website throws TMG into an infinite loop of 301 redirects.

I wanted to make sure that I knew exactly how a 301 should be behaving so I consulted the Request For Comments (RFC) which in this case is RFC 2616. These are extremely dry and not a lot of fun to read but they tell you what you should expect.

https://tools.ietf.org/html/rfc2616 (Figure 4)

Fig. 4

So there is nothing in the RFC that says the new permanent URI cannot be the same as what you already requested. Obviously we get a very undesired behavior when this is the case.

Since a client that is not using TMG as the proxy obviously behaves differently I wanted to compare the two to see what is different. Figure 5 shows the GET request from a client that is not using TMG as a proxy.

Fig. 5

In this case we get an immediate Status Code of 200 OK which is what we would expect (Figure 6).

Fig. 6

Based on what I was seeing in the Netmon tracing I suspected that the misbehaving remote website was responding with a 301 Moved Permanently in response to something it was or wasn’t getting in the initial GET request. Comparing the requests from both client traces I determined that there were 2 major differences.

1. ) TMG sends a VIA header when making the request for a web proxy client. Clients not using a proxy do not send this header when making a request.

2.) Clients not using TMG as their proxy send an Accept-Encoding header with a value of gzip, deflate. When TMG sends a GET request on behalf of a web proxy client it is not including the Accept-Encoding header.

Using a client in my test lab that could go directly to the Internet without having to use a proxy I wanted to test 2 things. I wanted to add a VIA header although the client was not using a proxy and I also wanted to strip the Accept-Encoding header out of the request.  Luckily there is a tool called Fiddler by Telerik that will allow you to do just that.

Fiddler is a great debugging tool that we use pretty frequently to troubleshoot web related issue.

https://www.telerik.com/fiddler

Adding a VIA header to the request made no difference whatsoever, the web page loaded just fine. Stripping the Accept-Encoding: gzip, deflate header from the request, however, caused the remote website to behave with the 301 Moved Permanently infinite looping.

Customers usually want to know whose product is at fault in cases like this. For that I always go back to the RFC. If your product is following the RFC to the letter then it is pretty clear that the remote website is at fault. In this case there is nothing in the RFC that states that an Accept-Encoding header MUST be sent.

I was never able to get a response from the web admins at the remote offending website to find out exactly why their Apache server is behaving this way. The RFC is not specific and does not state that they CANNOT do what they are doing but it is just very odd and pretty unexpected.

In the end I was able to find a setting in TMG that would force it to send the Accept-Encoding: gzip header which apparently makes the remote site happy. This is accomplished in the TMG MMC, under Web Access Policy, then click HTTP Compression. If you add the remote site under the “Request Compressed Data” tab then it will actually send that header.

Conclusion

As in so many of the cases that we work here at Microsoft the answer is not always readily apparent. Sometimes you have to dig pretty deep and employ multiple tools to arrive at your answer. In this case I had to use no less than 3 tools (Live Logging, Netmon 3.4, Fiddler) and the RFC as a resource to find an acceptable solution for our customer.  I hope that you enjoyed this article and it helps you to realize that troubleshooting can be a fun process especially when you use all of the tools at your disposal.

Cheers!

Note: If the information contained here was useful please let me know in the comments below. Also, if there are any corrections needed or you would like to see future content on a particular subject please let me know that as well. Thanks!