HTTPS Inspection incompatible with DIRSYNC tool for Office 365
So for my first blog post, I thought I would talk about an issue I ran across while working to setup an on-premises Exchange 2010/2013 DIRSYNC to Office 365 through a forward proxy server. (In my case proxy was TMG 2010, but the same concepts should apply for third party forward proxy solutions.) We have a fairly typical layout that you would see in a majority of corporate environments. See High Level diagram below:
Once the DIRSYNC tool was installed (which is downloaded from your admin account for O365) I ran the configuration wizard to setup directory synchronization.
After clicking the next button the tool failed, and give me a very informative error. :)
So I took its advice and went and checked the Windows application log. Low and behold, the problem stuck out like a sore thumb. The forward proxy/TMG 2010 server, or as it’s referred to in the description “Remote Server”, I was connecting through has HTTPS inspection turned on. I will talk a bit more later why this is an issue.
Log Name: Application
Source: Directory Synchronization
Date: 12/20/2013 11:44:55 AM
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: DIRSYNC.xxxxx-xxxxxxxx.com
Description:
The remote server returned an unexpected response: (502) Proxy Error ( The Web site requires a client certificate, but a client certificate cannot be supplied when HTTPS inspection is applied to the request. ).
Event Xml:
<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Directory Synchronization" />
<EventID Qualifiers="0">0</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2013-12-20T17:44:55.000000000Z" />
<EventRecordID>1248</EventRecordID>
<Channel>Application</Channel>
<Computer>DIRSYNC.xxxxx-xxxxxxxx.com</Computer>
<Security />
</System>
<EventData>
<Data>The remote server returned an unexpected response: (502) Proxy Error ( The Web site requires a client certificate, but a client certificate cannot be supplied when HTTPS inspection is applied to the request. ).</Data>
</EventData>
</Event>
Below you can also see the error in the TMG log as well
Denied Connection
12/20/2013 11:44:55 AM
Log type: Web Proxy (Forward)
Status: 12229 The Web site requires a client certificate, but a client certificate cannot be supplied when HTTPS inspection is applied to the request.
Rule: Anonymouse Access To Web Source: Internal (10.10.10.9:51269)
Destination: External (157.56.55.72:443)
Request: POST adminwebservice.microsoftonline.com
Filter information: Req ID: 0a18f9a4; Compression: client=No, server=Yes, compress rate=0% decompress rate=0%
Protocol: https-inspect
User: anonymous
Object source: Internet (Source is the Internet. Object was added to the cache.)
Cache info: 0x0
Processing time: 63 MIME type: -
This is more like it. As you can see the error clearly states this is a proxy configuration issue, and HTTPS inspection needs to be turned off. HTTPS inspection is not compatible with servers that require a client certificate. So be armed with this error (and this blog post) when you go request that your proxy admin remove HTTPS inspection.
Removing HTTPS inspection on TMG 2010
So while this talks specifically about TMG, most 3rd party proxy solutions will allow you to make HTTPS inspection exceptions.
Log into TMG and open the management console and click "configure HTTPS" inspection" in the right pane.
You have two ways to tackle this, by source host or by destination address. I chose by source exceptions because my many years of supporting Exchange and TMG have emphasized the cliché of an “ounce of prevention is worth a pound of cure.” Keep in mind either method will work, but if you choose or your company’s policy is to make exceptions by remote domain, should any of those remote domain names change in the future (which is outside of your control) your DIRSYNC may break if you do not actively go in once you get notice said change may occur and update your remote exceptions.
Add your source host in the list. Click “Ok” and then click the “Apply” button to commit the change to the TMG configuration database.
One more thing before you swing back to your DIRSYNC server is to check to make sure the TMG CSS has been updated. Select the monitoring in the left pane, and then select configuration. Make sure the status is “Synced” and the description is “Server configuration matches the stored configuration.” If you have an array make sure all array members have sync’d with the CSS.
Now jump back to your DIRSYNC server, re-run the wizard and bam! It works!!
So why does HTTPS inspection on a Forward Proxy not allow the DIRSYNC tool to work?
Long story short here, the certificate on the remote web server you are (DIRSYNC tool in this case) connecting to states “Client Authentication (1.3.6.1.5.5.7.3.2)” in the Enhanced Key Usage field of its certificate. So HTTPS inspection breaks this because of the way it works.
In order for TMG to inspect the clients HTTPS request it must be able to decrypt it. So in order to do this TMG establishes a new HTTPS connection between itself and the client using a certificate signed by its own CA. (See picture below). Once this happens it is then able to read the contents of the clients request and apply filtering rules based on the policy setup by proxy administrator. If TMG determines everything looks good it will then establish a second HTTPS session with the remote web server which will allow it to inspect the return traffic. Because the remote web server requires client authentication to prove the identity of the client the web server is talking with TMG will not continue the SSL session. Therefore disabling HTTP inspection creates an SSL tunnel directly between the Server and client allowing the two way authentication and establishing a highly secure link between the two systems to synchronize your directory with your Azure/O365 directory.
Further Reading