Troubleshooting TLS / SSL communication problems for ASP.NET applications making HTTP Web Request or WCF queries to SSL endpoints – Scenario 3
In our first and second posts about troubleshooting the TLS / SSL problems, we worked to fix a "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel" error message. We made some basic tests, such as a "browser test" and found that the certificate used was not valid and then in the second scenario, we compared a network trace for successful and failing traffic and saw that the TLS negotiation between the client and the server was failing.
We fixed that issue as well and we know that the TLS versions match for the client and the server now and the certificate is obtained for correct name but we are still getting an error message.
If you recall the outcome of the scenario 2, we were seeing a FIN package coming from the client and the server was resetting the TCP connection:
(Please note that the TMG is just the name of the server, and not the "Threat Management Gateway", sorry for the confusion)
This time the error we get is different:
Server Error in '/' Application. -------------------------------------------------------------------------------- The remote certificate is invalid according to the validation procedure. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
The exception message tells us that the remote certificate is not valid. We know that we passed the TLS and Cipher negotiation so there should be some other validation failing. What could it be and how can we troubleshoot this?
Here the SChannel ETL logging and System.Net tracing come in the picture. We will not focus on SChannel ETL logging here as it could be another post's topic. Instead we will use System.Net tracing to troubleshoot this problem.
A quick word about System.Net tracing
A System.Net tracing is a diagnostic option availabile in .NET Framework since the version 2.0. This is a part of System.Diagnostic namespace and it could be used to capture the details of System.Net classes including the System.Net.Socket operations. To enable System.Net tracing you can use the configuration sample from the following article:
Collect network trace data within your ASP.NET application
https://blogs.msdn.microsoft.com/amb/2011/02/16/collect-network-trace-data-within-your-asp-net-application/
To enable the tracing we copy the configuration from the article and paste it in our ASP.NET application's web.config file. Pay attention to the folder the traces will be created:
<add name="System.Net" type="System.Diagnostics.TextWriterTraceListener" initializeData="c:\tracelogs\network_trace.log" />
Here in the example above, the trace will be written in the c:\tracelogs\ directory and the application pool's identity should have the write permission on that folder.
How to read System.Net trace logs
If you are not familiar with System.Net trace logs, I suggest you to take some minutes to read the following post:
Use System.Net Trace and SSL Alert Protocol to troubleshoot SSL connection issue.
https://blogs.msdn.microsoft.com/webapps/2012/11/05/use-system-net-trace-and-ssl-alert-protocol-to-troubleshoot-ssl-connection-issue/
Once we reproduce the same issue a network_trace.log file should be written in the trace folder we specified. Let's open the trace log in our favorite text editor. If you look closely you would see the following reported at the end of the trace:
[Public Key] Algorithm: RSA Length: 2048 ... ... ... System.Net Information: 0 : [2504] SecureChannel#26517107 - Remote certificate has errors: System.Net Information: 0 : [2504] SecureChannel#26517107 - A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider. System.Net Information: 0 : [2504] SecureChannel#26517107 - Remote certificate was verified as invalid by the user. System.Net.Sockets Verbose: 0 : [2504] Socket#16754362::Dispose() System.Net Error: 0 : [2504] Exception in HttpWebRequest#46228029:: - The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.. System.Net Error: 0 : [2504] Exception in HttpWebRequest#46228029::GetResponse - The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel..
This error message "A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider" tells us that there is a problem with the CA's (certification authority) root certificate and it is not trusted, so most probably the certificate store for trusted root CAs does not contain the root CA for the remote certificate.
But, remember the browser test, it was working so the user's trusted root CA certificate store should have the root CA for the remote certificate. Let's go and check the certificate once more with "browser test": open a browser on the server where your ASP.NET application works, browse the endpoint URL (which is https://iis85.buggybits.com/ in our case) and click the details of the certificate and move to the "Certification Path" tab:
As seen the certificate is from an internal CA called amborp-AMBDC-CA.
Let's look at the user's certificate store. We confirm that the root certificate is already there:
But this is user's certificate store. ASP.NET client will use the "local computer" account's "Trusted Root Certification Authorities" store. If we open and check that store we see that the CA's root certificate is not installed there. Once we import the CA's root certificate in computer account's trusted root CA store, the application starts to work fine.
Summary:
- In our first post, we learnt that the certificate must be a valid one. Unlike "human clients", an ASP.NET client cannot click the "Ok, I trust this certificate / web site" if a certificate error occurs, unless you explicitly tell the application to ignore the certificate errors, which is actually not a recommended approach.
- In our second post, we learnt the TLS / Cipher negotiation is a requirement for a successful SSL handshake.
- In this third and final post, we learnt how System.Net tracing could be helpful when troubleshooting TLS / SSL related issues.
If you have any comment or suggestion, please use the comments section below.
Happy troubleshooting...
Comments
- Anonymous
October 30, 2017
Great, this is what I was looking for in yahoo- Anonymous
November 06, 2017
Happy to be of service and help out.
- Anonymous
- Anonymous
January 15, 2019
Great series of articles. Thanks! Any idea what would need to be changed to use this for asp.net core (that targets the full 4.6.1 framework)? I am seeing some socket trace information, but none of the handshake info.