Access Denied error when running app on IIS 6 and IIS 5.1. Runs fine on Windows 2000.
First off, there are many different things that can cause an Access Denied error in an application hosted in IIS. This post will cover a little known and "under documented" cause for this error called Loopback Check. IIS 5.1 and 6.0 added an additional security measure call Loopback Checking to help prevent reflection attacks on the server. I am not a hacker so I do not fully understand how one would implement such an attack on a server without Loopback Checking but the general idea is this: Mr Hacker opens a connection to my IIS server to a site that is secured with Windows Integrated Authentication and is challenged for credentials. At this point Mr. Hacker opens a second connection to the server and sends the challenge that he received on Connection 1 back to the server in an effort to get a security token based on the servers credentials. Again, I only have a really high level understanding of this so my explanation may not be technically precise. On a server without Loopback checking (i.e. IIS 5), you could theoretically gain access to the server using this type of attack.
Enter Loopback Checking. In an effort to thwart reflection attacks, IIS 5.1 (XP SP2) and IIS 6 (W2K3 SP1) check any challenges made on the server for credentials to see where they originated from. I am not privy to the actual implementation on this but it basically checks the source IP address of the challenge and if it is an IP address from the local machine it checks to see if the host name for the challenge matches the machines host name. If the host name doesn't match the machine name, then the challenge fails which gives us the access denied errors we see in our apps.
"Ok so how does this affect my application and why am I getting access denied", you say. Most likely your application is doing some type of activity that requires it to authenticate (doing a CreateObjectEx to instantiate a COM component is where I have see this quite a bit). You would also have to be using a host-name other than the machine name to access this in order for the problem to occur. For example, let's say you have an ASP application on a server named WEBSERVER1. Let's also say that you have created a host-name for the site called KillerApp.CompanyA.com so that users can access the site using this URL. If your ASP application tries to instantiate a COM component on this machine using CreateObjectEx("MyProg.MyClass", KillerApp.CompanyA.com) and you are using Windows Integrated Authentication, you would get the access denied error. Why? Good Question. IIS sees a challenge for credentials coming from one of the local loopback ip's but the host-name is KillerApp.CompanyA.com. IIS/Windows Security thinks this challenge is being reflected because it's not from WEBSERVER1 so it doesn't grant the token.
The good news is that this is fairly easy to spot and even easier to fix. To find out if this is what is causing your access denied. Turn on Audit Logon Events and Audit account logon events for failures in the Local Security Settings - Local Policies - Audit Policy. Reproduce the access denied error and then check your security event log on the server. Notice the "strange" characters for the "Logon Process in the screen shot below. This is a dead giveaway that you are failing due to the loopback check feature.
There are two methods listed in the KB listed below that you can use to get around this problem. I recommend method two, since it will keep your attack surface area much smaller. Method one, completely disables the loopback check. You could also get around this by using the machine name instead of the FQDN.
896861 You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6
https://support.microsoft.com/default.aspx?scid=kb;EN-US;896861