Direct access client fails to connect to UAG DA server through IPHTTPS with various errors (0x800b0109 and 0x274c)
In this post, I want to talk about how I troubleshooted a direct access connectivity problem hoping that it might also help you in troubleshooting similar problems.
The issue was that the IPHTTPS client cannot connect to UAG DA server. We first checked static logs collected from a problem client (GPO output, firewall outputs, etc) and all seemed to be fine:
- Required services were running on the client (Windows Firewall/IKE and AuthIP/IP Helper)
- Firewall was chosing the correct firewall (Public profile):
-------------------------------------
netsh advfirewall show currentprofile
-------------------------------------
Public Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Enable
RemoteManagement Disable
UnicastResponseToMulticast Enable
Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName C:\Windows\System32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096
Ok.
- All categories including connection security was owned by Windows Firewall:
Categories:
BootTimeRuleCategory Windows Firewall
FirewallRuleCategory Windows Firewall
StealthRuleCategory Windows Firewall
ConSecRuleRuleCategory Windows Firewall
- gpresult output collected confirmed that the client was getting the required policy which is UAG DirectAccess: Clients (UAGServer-FQDN)
Then I asked our customer to implement the below action plan to collect dynamic data while reproducing the problem
Action plan for log collection:
=========================
The outline of the action plan is below:
- Start WFP/DirectAccess/network traces on the DA client
- Start WFP/DirectAccess/network traces on the UAG DA server
- Stop and restart the relevant services (like IKE/ip helper) to trigger the tunnel establishment and see things from scratch
- Once we observe that the client cannot access any internal resources we’ll stop logs on the DA client and UAG DA server
1) Please install Network Monitor 3.4 on a problem DA client and on the UAG DA server. You can download the tool at the following link:
2) Please disable and stop “IKE and AuthIP IPSec Keying Modules” service with the below commands on the DA client:
Note: All commands need to be run from an elevated command prompt.
a) We first disable the service:
sc config ikeext start= disabled
(Note: There must be a SPACE between “=” and “disabled”, otherwise the command doesn’t work)
b) We stop the service:
net stop ikeext
3) Now we start the network capture on the DA client and on the UAG server: (from an elevated command prompt)
=> On the DA client:
nmcap /network * /capture /file DAClient.chn:100M
=> On the UAG DA server:
nmcap /network * /capture /file DAServer.chn:100M
Note: You have to run the nmcap command from an elevated command prompt on Windows Vista/Windows 7/Windows 2008/Windows 2008 R2
Note: nmcap will create a new capture file once the first one if full (200 MB) and so on. So please make sure that you have enough free disk space on the related drive.
Note: If you get the error message "'nmcap' is not recognized as an internal or external command, operable program or batch file", you may need to run the nmcap command inside "%ProgramFiles%\Microsoft Network Monitor 3" folder
Note: Network trace could be stopped any time by pressing Ctrl + C
4) Let’s start tracing for various components at a different elevated command prompt on the DA client and on the UAG DA server:
=> On the DA client:
netsh wfp capture start
netsh trace start scenario=directaccess provider=microsoft-windows-iphlpsvc level=5 capture=yes tracefile=darepro.etl maxsize=350
=> On the UAG DA server:
netsh wfp capture start
netsh trace start provider=Microsoft-Windows-DNS-Client provider=Microsoft-Windows-Iphlpsvc-Trace provider=Microsoft-Windows-WFP provider=Microsoft-Windows-NCSI provider=Microsoft-Windows-NlaSvc provider=Microsoft-Windows-NetworkProfile provider=Microsoft-Windows-WinINet provider=Microsoft-Windows-WinHttp provider=Microsoft-Windows-WebIO provider="Microsoft-Windows-Windows Firewall With Advanced Security" tracefile=c:\logs\UAGDAServer.etl maxsize=400
5) Now we’re going to reproduce the problem from the DA client side.We’re going to stop iphlpsvc on the DA client: (the service which implements the IPv6 transition technologies)
net stop iphlpsvc
6) Just before we start re-enabling the relevant services on the DA client, we run the following ping on the DA client: (with 22 bytes ping)
ping -l 22 -n 2 IP-address-of-default-gateway-configured-on-the-client
Note: Please replace IP-address-of-default-gateway-configured-on-the-client with the real default gateway address configured on the client
7) Now please run the following commands in the given order on the DA client:
net start iphlpsvc
sc config ikeext start= auto
net start ikeext
(Note: There must be a SPACE between “=” and “disabled” in "sc config" command, otherwise the command doesn’t work)
8) Let’s please wait for 30 seconds or so and then try the following commands on the DA client:
ping -l 33 an-internal-server1
ping -l 44 an-internal-server2
net view \\an-internal-server1
net view \\an-internal-server2
As an additional test, you can try to reach an internal web server/sharepoint portal etc if any exists from Internet explorer on the DA client (like accessing https://internalserver3/portal)
NOTE: Please replace the “an-internal-server” with real ones
NOTE: Please write down the full commands that you run
NOTE: Please take a screenshot of each access attempts
9) After you get errors for the above commands, please run the following commands (all from an elevated command prompt) on the DA client and on the UAG DA server to stop the logs running:
=> On the UAG DA server:
a) To stop WFP tracing:
netsh wfp capture stop
b) To stop Directaccess tracing:
netsh trace stop
c) To stop Network trace on the UAG DA server:
Please hit CTRL+C at the command prompt where nmcap tool runs
=> On the DA client:
a) To mark the end of problem reproduce (with 55 bytes ping)
ping -l 55 -n 2 IP-address-of-default-gateway-configured-on-the-client
b) To stop WFP tracing:
netsh wfp capture stop
c) To stop Directaccess tracing:
netsh trace stop
d) To stop Network trace on the DA client:
Please hit CTRL+C at the command prompt where nmcap tool runs
ANALYSIS 1:
==========
After analyzing the logs collected as per the above action plan, I got the following findings:
=> IPHTTPS interafce was not connected with the error number 0x800b0109
Interface IPHTTPSInterface (Group Policy) Parameters
------------------------------------------------------------
Role : client
URL : https://UAGDAServer-FQDN:443/IPHTTPS
Last Error Code : 0x800b0109
Interface Status : failed to connect to the IPHTTPS server. Waiting to reconnect
err 0x800b0109
# for hex 0x800b0109 / decimal -2146762487 :
CERT_E_UNTRUSTEDROOT winerror.h
# A certificate chain processed, but terminated in a root
# certificate which is not trusted by the trust provider.
# 1 matches found for "0x800b0109"
=> IPHTTPS client doesn’t accept the certificate provided because it doesn’t trust the issuer
=> When I checked the network activity, I saw the following communications:
2084 15:13:43.441664 463.087187 0.002359 192.168.99.5 192.168.99.100 DNS Standard query A UAGDAServer-FQDN
2085 15:13:43.442251 463.087774 0.000587 192.168.99.100 192.168.99.5 DNS Standard query response A 1.1.1.2
Now client tries to establish the TLS tunnel:
No. Time Time since Delta Source Destination Protocol Identification S-Port D-Port Sequence TCP Len TCP Ack TCP Win Info
2086 15:13:43.443150 463.088673 0.000000 192.168.99.5 1.1.1.2 TCP 0x58ba (22714) 57134 443 1683802426 0 8192 57134 > 443 [SYN] Seq=1683802426 Win=8192 [TCP CHECKSUM INCORRECT] Len=0 MSS=1460 WS=256 SACK_PERM=1
2087 15:13:43.479192 463.124715 0.036042 1.1.1.2 192.168.99.5 TCP 0x61f4 (25076) 443 57134 536292141 0 1683802427 8192 443 > 57134 [SYN, ACK] Seq=536292141 Ack=1683802427 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2088 15:13:43.479356 463.124879 0.000164 192.168.99.5 1.1.1.2 TCP 0x58bb (22715) 57134 443 1683802427 0 536292142 65536 57134 > 443 [ACK] Seq=1683802427 Ack=536292142 Win=65536 [TCP CHECKSUM INCORRECT] Len=0
2089 15:13:43.491000 463.136523 0.011644 192.168.99.5 1.1.1.2 TLSv1 0x58bc (22716) 57134 443 1683802427 119 536292142 65536 Client Hello
2092 15:13:43.530437 463.175960 0.039437 1.1.1.2 192.168.99.5 TLSv1 0x61f5 (25077) 443 57134 536292142 853 1683802546 65536 Server Hello, Certificate, Server Key Exchange, Server Hello Done
2093 15:13:43.569422 463.214945 0.038985 192.168.99.5 1.1.1.2 TLSv1 0x58be (22718) 57134 443 1683802546 134 536292995 64768 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
Certificate (id-at-commonName=Default Web Site) => Returned certificate is not the one expected by the client
2094 15:13:43.608227 463.253750 0.038805 1.1.1.2 192.168.99.5 TLSv1 0x61f6 (25078) 443 57134 536292995 59 1683802680 65536 Change Cipher Spec, Encrypted Handshake Message
2095 15:13:43.608869 463.254392 0.000642 192.168.99.5 1.1.1.2 TCP 0x58bf (22719) 57134 443 1683802680 0 536293054 64768 57134 > 443 [FIN, ACK] Seq=1683802680 Ack=536293054 Win=64768 [TCP CHECKSUM INCORRECT] Len=0
2096 15:13:43.645152 463.290675 0.036283 1.1.1.2 192.168.99.5 TCP 0x61f7 (25079) 443 57134 536293054 0 1683802681 0 443 > 57134 [RST, ACK] Seq=536293054 Ack=1683802681 Win=0 Len=0
=> On the other hand when checking the certificate bindings on the UAG server side, I saw that UAG DA server IPHTTPS server was bound to 1.1.1.1:443 with the appropriate certificate
c:\> netsh http show sslcert
...
IP:port : 1.1.1.1:443
Certificate Hash : 8691087835614a5f09b5f11c403d595c461ff603
Application ID : {5d8e2743-ef20-4d38-8751-7e400f200e65}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier :
Ctl Store Name :
DS Mapper Usage : Enabled
Negotiate Client Certificate : Disabled
...
c:\> certutil -store -v my
...
================ Certificate 2 ================
Serial Number: 5485200900020000015c
Issuer: CN=CA-name, DC=company, DC=com
NotBefore: 18/04/2012 16:49
NotAfter: 18/04/2014 16:49
Subject: CN=UAGDAServer-FQDN
Non-root Certificate
Template: WebServer3, Web Server 3
Cert Hash(sha1): 86 91 08 78 35 61 4a 5f 09 b5 f1 1c 40 3d 59 5c 46 1f f6 03
Key Container = d53d6468acfdb333ca5698ae67f3549e_6dd37665-5fa6-46ec-89de-8857879f2500
Simple container name: le-WebServer3-afc38199-0bb5-4cb9-b043-50851171183d
Provider = Microsoft Base Cryptographic Provider v1.0
Encryption test passed
...
The certificate seems to be the correct one but since UAGDAServer-FQDN is not mapped to 1.1.1.1 (it’s mapped to 1.1.1.2), another certificate is returned to the client in the TLS sessions and IPHTTPS connection fails as well. That was also evident in the network trace:
2084 15:13:43.441664 463.087187 0.002359 192.168.99.5 192.168.99.100 DNS Standard query A UAGDAServer-FQDN
2085 15:13:43.442251 463.087774 0.000587 192.168.99.100 192.168.99.5 DNS Standard query response A 1.1.1.2
I informed my customer about the problem and they changed the DNS settings. But afterwards my customer informed me that IPHTTPS connectivity problem was still in place but this time with a different error number. We re-run the previous action plan and collected logs one more time:
ANALYSIS 2:
=========
This time the problem was a bit different. As per the log outputs from the client and UAG DA server, there was a packet drop issue between the client and server, client’s IPHTTPS traffic wasn’t arriving at UAG DA server so the IPHTTPS tunnel establishment was failing:
c:\> netsh interface httpstunnel show interface
Interface IPHTTPSInterface (Group Policy) Parameters
------------------------------------------------------------
Role : client
URL : https://UAGDAServer-FQDN:443/IPHTTPS
Last Error Code : 0x274c
Interface Status : failed to connect to the IPHTTPS server. Waiting to reconnect
err 0x274c
# for hex 0x274c / decimal 10060 :
WSAETIMEDOUT winerror.h
# A connection attempt failed because the connected party did
# not properly respond after a period of time, or established
# connection failed because connected host has failed to
# respond.
WSAETIMEDOUT winsock2.h
# 2 matches found for "0x274c"
=> And we can really see that the DA client could not connect to TCP 443:
No. Time Time since Delta Source Destination Protocol Info
2657 14:43:33.170638 199.176755 197.498302 192.168.99.7 1.1.1.1 TCP 54007 > 443 [SYN] Seq=2446300399 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2693 14:43:33.563827 199.569944 0.393189 192.168.99.7 1.1.1.1 TCP 54008 > 443 [SYN] Seq=4024014335 Win=8192 Len=0 MSS=1460 SACK_PERM=1
2736 14:43:36.583165 202.589282 0.258635 192.168.99.7 1.1.1.1 TCP 54008 > 443 [SYN] Seq=4024014335 Win=8192 Len=0 MSS=1460 SACK_PERM=1
2839 14:43:42.577128 208.583245 0.259486 192.168.99.7 1.1.1.1 TCP 54008 > 443 [SYN] Seq=4024014335 Win=8192 Len=0 MSS=1460 SACK_PERM=1
3482 14:44:24.615646 250.621763 18.289976 192.168.99.7 1.1.1.1 TCP 54023 > 443 [SYN] Seq=3215921282 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
3516 14:44:27.621705 253.627822 3.006059 192.168.99.7 1.1.1.1 TCP 54023 > 443 [SYN] Seq=3215921282 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
3585 14:44:33.627033 259.633150 6.005328 192.168.99.7 1.1.1.1 TCP 54023 > 443 [SYN] Seq=3215921282 Win=8192 Len=0 MSS=1460 SACK_PERM=1
4432 14:45:36.642904 322.649021 4.472205 192.168.99.7 1.1.1.1 TCP 54035 > 443 [SYN] Seq=55761659 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
4472 14:45:39.638130 325.644247 2.995226 192.168.99.7 1.1.1.1 TCP 54035 > 443 [SYN] Seq=55761659 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
4539 14:45:45.653327 331.659444 6.015197 192.168.99.7 1.1.1.1 TCP 54035 > 443 [SYN] Seq=55761659 Win=8192 Len=0 MSS=1460 SACK_PERM=1
6202 14:48:00.662720 466.668837 41.395992 192.168.99.7 1.1.1.1 TCP 54064 > 443 [SYN] Seq=2998968172 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
6239 14:48:03.672009 469.678126 3.009289 192.168.99.7 1.1.1.1 TCP 54064 > 443 [SYN] Seq=2998968172 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
6309 14:48:09.673882 475.679999 6.001873 192.168.99.7 1.1.1.1 TCP 54064 > 443 [SYN] Seq=2998968172 Win=8192 Len=0 MSS=1460 SACK_PERM=1
=> Also none of the above requests seen on the client side trace were present on the UAG DA server side trace which meant that UAG DA server never saw them
My customer further checked the connectivity with their service provider and it was found out that due to an incorrect routing configuration traffic sent to 1.1.1.1:443 was really dropped/routed somewhere else and hence the IPHTTPS tunnel wasn’t established. After fixing the issue with the router configuration, the problem was resolved and IPHTTPS tunnel was successfully established.
Hope this helps
Thanks,
Murat