Exchange 2013 Clients not able to connect using Outlook Anywhere
I want to share with the readers about an issue that I see happening more and more often when Exchange 2013 is deployed in the environment. What I'm seen is that nobody has made any changes to the system but next day nobody cant connect nor any of the web services work. What changed? The first place that you need to look at is that Exchange back end web site certificate bindings on port 444. Normally, if you check the IIS logs and the HTTP proxy logs you can see that you get Status code 500 when proxying to the Exchange 2013 back end website. Another symptom that you notice is that the Exchange PowerShell console won't open correctly. The reason that this issue happens is due to the way connection flow happens in Exchange 2013. Below you can see the data flow when outlook client connects to exchange 2013 mailbox, and the explanation of the connection. I will cover the normal flow first and then we go over why this breaks when missing or having a third party certificate bound to 444 on the Exchange Back End Web Site.
Exchange 2013 Mailbox
- Outlook >>>>> 2013 CAS DWS:443
- 2013 CAS >>>> 2013 MBX:444
- 2013 CAS <<<< 2013 MBX:444
- Outlook <<<<< 2013 CAS DWS:443
Step by Step Connection Flow
- IIS authenticates the request and Microsoft.Exchange.FrontEndHttpProxy.dll running in the MSExchangeRpcProxyAppPool. HTTP Proxy determines the Mailbox Anchor, accessing Active Directory to look up the user, then maps to the user object that contains the mailbox GUID. As soon as we obtain the GUID, HTTP Proxy refers to Active Manager to locate the database where the mailbox resides. At this point, HTTP Proxy then builds the URL to the mailbox server that contains the user's mailbox GUID@Domain.com.
- HTTP Proxy takes the authenticated request and creates a serialized blob (that contains SID, user groups, and more. Then a new header is attached that contains the Client Access server name, in order to use CSC (serialized access token authentication, also known as server to server authentication) and sends this new http request URL to the Mailbox server over SSL on port 444. The new URL contains the Mailbox server name so that the Mailbox server knows the request belongs to this server.
- The request is sent to the Mailbox server's RPC Virtual Directory that is running in the MSExchangeRpcProxyAppPool by using by default, integrated authentication.
- RPCProxy.dll on the Mailbox server is responsible for extracting the RPC packet from the tunnel and forwarding it to the local RPC Client Access Service on Port 6001.
Now that we went over the connection flow when a mailbox is located in an Exchange 2013 mailbox server, we can cover why this breaks and why is so important of not changing or having a third party certificate bound to port 444 on the Exchange Back End Web Site. When the client connects to the CAS, and HTTP Proxy determines where the mailbox is located, it builds an URL to the destination mailbox server on port 444. The URL is built using the hostname. Once we have the URL we try to connect on secure port 444 using SSL. In the process of the SSL handshake there is several validation task that have to take place before the connection is allowed. If the name being used on the URL is not in the certificate the request will fail and your client cant connect.
But I see 200 status code in the IIS logsā¦
Yes, the reason you see 200 status codes on IIS is due to the initial communication is happening on port 443 and in the Default website. Where the process breaks is in the second part of connecting to the mailbox server on port 444. If you look closely at the HTTP proxy logs, you have two columns one for HTTP status and one for backend status. In that log you can see the 200 on the first hop but when passing it to the mailbox server on port 444 it fails with the 500 error. Another log that you can check is the IIS log for W3SVC2. In this folder you have the logs for all traffic going thru the Exchange Back End web site.
Conclusion
The main point to show with this article is that when you have the incorrect certificate binding on the Exchange Back End web site can cause the web services on Exchange 2013 server not to work properly. This incorrect certificate binding will break the connection flow, causing clients to have a bad experience.