WCF/WS: SSL Mutual Client Cert Authentication 403.16 or 403.7

Problem
When attempting to use a certificate to authenticate to an IIS website or self hosted WCF service over SSL/TLS channel, we receive a 403.16 error code.

Troubleshooting
We can collect server side System.Net Traces or WCF Activity Traces
https://blogs.msdn.microsoft.com/saurabs/2017/06/12/system-net-tracing-collection-steps/
https://blogs.msdn.microsoft.com/saurabs/2017/06/12/wcf-tracing/

Observation from System.Net Traces:
You might observe the GetClientCertificate API was successful and Client Cert was received at server side.
From WCF traces you might see a native error (0X109) being sent with below stack.

msdn.microsoft.com/en-US/library/System.ServiceModel.Channels.HttpsClientCertificateInvalid.aspx
Client certificate is invalid with native error code 0x109 (see go.microsoft.com/fwlink/?LinkId=187517 for details).
Process.exe
System.ServiceModel.Channels.HttpsChannelListener`1[System.ServiceModel.Channels.IReplyChannel]/11268815
Keep-Alive
text/html, application/xhtml+xml, */*
gzip, deflate
en-US
X.X.X.X:YY
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
https://Service/Method
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at System.Diagnostics.TraceEventCache.get_Callstack()
at System.Diagnostics.XmlWriterTraceListener.WriteFooter(TraceEventCache eventCache)
at System.Diagnostics.XmlWriterTraceListener.TraceData(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, Object data)
at System.Diagnostics.TraceSource.TraceData(TraceEventType eventType, Int32 id, Object data)
at System.ServiceModel.Diagnostics.LegacyDiagnosticTrace.TraceEvent(TraceEventType type, Int32 code, String msdnTraceCode, String description, TraceRecord trace, Exception exception, Object source)
at System.ServiceModel.Diagnostics.TraceUtility.TraceEvent(TraceEventType severity, Int32 traceCode, String traceDescription, TraceRecord extendedData, Object source, Exception exception)
at System.ServiceModel.Channels.HttpsChannelListener`1.ValidateAuthentication(HttpListenerContext listenerContext)
at System.ServiceModel.Channels.HttpRequestContext.ListenerHttpContext.ValidateAuthentication()
at System.ServiceModel.Channels.HttpRequestContext.ProcessAuthentication()
at System.ServiceModel.Channels.HttpChannelListener`1.HttpContextReceivedAsyncResult`1.Authenticate()
at System.ServiceModel.Channels.HttpChannelListener`1.HttpContextReceivedAsyncResult`1.ProcessHttpContextAsync()
at System.ServiceModel.Channels.HttpChannelListener`1.BeginHttpContextReceived(HttpRequestContext context, Action acceptorCallback, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.SharedHttpTransportManager.EnqueueContext(IAsyncResult listenerContextResult)
at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContextCore(IAsyncResult listenerContextResult)
at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContext(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.ListenerAsyncResult.IOCompleted(ListenerAsyncResult asyncResult, UInt32 errorCode, UInt32 numBytes)
at System.Net.ListenerAsyncResult.WaitCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Cause
The cause of this was due to having non-self-signed certs in the Trusted Root Certificate Authority store on the IIS server.
https://support.microsoft.com/en-us/kb/2802568

If we have a non-self-signed certificates in the Trusted Root certificate store. We can run this command to check:
Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Format-List * | Out-File "c:\computer_filtered.txt"

Only self-signed certs should be located in this store. If there are any non self-signed, then they should be exported for backup and deleted from the trusted roost certificate authority store. Computer_filtered.txt should not have any certs listed in it.

ClientAuthTrustMode controls how client cert are validated at S-Channel level.

Value Trust Mode Description
0 Machine Trust (default) Requires that the client certificate is issued by a certificate in the trusted issuers list.
1 Exclusive Root Trust Requires that a client certificate is chained to a root certificate that is contained in the caller-specified trusted issuer store. The certificate must also be issued from the trusted issuers list.
2 Exclusive CA Trust Requires that a client certificate is chained to an intermediate CA certificate or to a root certificate in the caller-specified trusted issuer store.

Resolution
If there are non-self signed certs in the trusted root store but we do not want to delete them, the work around would be setting this reg key and then rebooting the server.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL ClientAuthTrustMode =dword:00000002

Changing it to a value of 2 means the client cert has to chain to a intermediate CA cert or Root CA cert in the caller-specified trusted issuer store. Even if we have not specified a specifc cert trust list store (CTL store). In that case, we will still use the Trusted Root CA store. However, because value 2 states it can chain to a “intermediate CA cert” or “root certificate”, then we do not run into the issue of having subCA certs in the root CA store.

The preferred method would be to keep the value to 0 and remove the unneeded certs from the Root CA store. However, if we feel that is not possible, then changing this to a value of 2 should fix this issue.
From my research the only effect this has is, it allows the application / http.sys binding to specify a different trusted issuer store other then the Root CA store. In case we do not specify it, s-channel just bypasses the check to look for non-self-signed certs in the Root CA store.

Other Information
Client Authentication mapping
https://msdn.microsoft.com/en-us/library/aa292114(v=vs.71).aspx
https://www.iis.net/configreference/system.webserver/security/authentication/iisclientcertificatemappingauthentication
https://support.microsoft.com/en-us/kb/907274
https://technet.microsoft.com/en-us/library/cc770480(v=ws.10).aspx

I hope this helps!

Thanks
Saurabh Somani