Debugging sSL handshake failure using network monitor – a scenario

In one of my earlier post I explained how to use Microsoft Network Monitor to debug a networking problem. Network trace tools aren’t very useful in debugging problems when the channel is secured (HTTPS) and you need to view the data to make your conclusions. However you can still debug SSL handshake failures using network monitor. Here’s a scenario.

A client (Not browser) is trying to connect to a IIS web server by sending its client certificate to post some data. However, the IIS machine always rejects the authentication.

The first step was to take a network trace as usual. For instructions on how to capture simultaneous traces, see this post

Analyzing the network trace:

I filtered the traffic by the keyword SSL. In the display filter tab, type SSL and click on Apply button.

Here are some of the frames the we picked from the capture:

Frame	Src	Dst	Protocol	Description
7	Client	Server	SSL	SSL:SSLv2RecordLayer, ClientHello(0x01)
9	Server	Client	SSL	SSL: Server Hello. Certificate.
12	Server	Client	SSL	SSL:Continued Data: 1460 Bytes
13	Server	Client	SSL	SSL:Continued Data: 1460 Bytes
18	Server	Client	SSL	SSL:Continued Data: 1460 Bytes
22	Server	Client	SSL	SSL:Continued Data: 1460 Bytes
27	Server	Client	SSL	SSL: Encrypted Handshake Message.
30	Client	Server	SSL	SSL: Certificate. Client Key Exchange. Change Cipher Spec. Encrypted Handshake Message.
32	Server	Client	SSL	SSL: Encrypted Alert.

During SSL connection negotiation process, the client and server can mutually exchange certificates for authentication. The client authentication can be optional. To understand more about how the SSL negotiation takes place, please see these Microsoft KBs

Description of the Secure Sockets Layer (SSL) Handshake

Description of the Server Authentication Process During the SSL Handshake

Essentially, what is going on here can be summarized as:

1. The client makes a hello request in Frame 7
2. The server responds with its certificate and then continued bytes from the server certificate.
3. The client then sends an "Encrypted handshake message"
4. The client then sends its certificate with Client Key exchange and also indicates a change of cipher spec.
5. In frame 32, we can see an encrypted alert!

While performing any type of debugging, we need to follow the data; we need to look for something that is “interesting”. In this case an “alert” from the server is being sent. So we looked at what this alert is. 

Since our “interesting” frame is 32, we looked more at the headers and the details in the frame. Here is the frame in detail

  Frame: Number = 32, Captured Frame Length = 61, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[MAC Address],SourceAddress:[Source MAC Address]
+ Ipv4: Src = ServerIP, Dest = ClientIP, Next Protocol = TCP, Packet ID = 5022, Total IP Length = 47
+ Tcp: Flags=...AP..., SrcPort=5443, DstPort=1100, PayloadLen=7, Seq=2764379044 - 2764379051, Ack=3896915131, Win=64240 (scale factor 0x0) = 64240
- Ssl: Encrypted Alert.
- TlsRecordLayer:
ContentType: Encrypted Alert
- Version: TLS 1.0
Major: 3 (0x3)
Minor: 1 (0x1)
Length: 2 (0x2)
EncryptedData: Binary Large Object (2 Bytes)

Here is the handshake data in Hex (This data isn't encrypted yet because the handshake is still in progress)

00 1E 68 0F 3E 80 00 30 48 7E 1B 90 08 00 45 00 00 2F 13 9E 40 00 3D 06 D2 0E 0A F4 40 0F 0A 33 02 E7 15 43 04 4C A4 C5 13 A4 E8 46 34 BB 50 18 FA F0 2B B8 00 00 15 03 01 00 02 02 2A

We can conclude a lot from this frame. We know that this is an ethernet packet. The TCP flags set are AP, which means transfer data to the end application on the client without buffering at the TCP level, the communication port on the server is 5443, the client port is 1100, the window size for data transmission is 64240 bytes and so forth. The most interesting set of data is the TLS record layer… It shows the major and minor versions in use after negotiation and the length of the data, which is 2 bytes. This data is encrypted.

So here’s the trick now. This is an encrypted alert and RFC defines the alert descriptions. You can find it here for TLS 1.0 (Transport layer security)

Take the hex value of the 2 bytes of this message, which is 2A (The last 2 values, all values before this are headers in hex and we can ignore that because network monitor already gave us that information)

Using Windows calculator, convert hex 2A into decimal. We get: 42.

Now, let’s go look at the TLS 1.0 RFC to find out what what decimal value 42 represents for Alert Description. You can search for “AlertDescription” on the page. You should get:

decompression_failure(30),
handshake_failure(40),
bad_certificate(42),
unsupported_certificate(43),
certificate_revoked(44),
certificate_expired(45),
certificate_unknown(46),
illegal_parameter(47),
unknown_ca(48),
access_denied(49),
decode_error(50),
decrypt_error(51),
export_restriction(60),
protocol_version(70),
insufficient_security(71),
internal_error(80),
user_canceled(90),
no_renegotiation(100),
(255)
} AlertDescription;

Thus, we can see that 42 translates to bad_certificate. So we can say that the client sent a bad certificate to the server and therefore the server rejected the connection request. Armed with this information, we checked the client certificate in the certificate store and indeed, we had a bad certificate. Replacing it with a good certificate fixed the problem!

Software has some a long way and enabled people around the world to do many things. It has also made our lives so much easier. However these new capabilities have also created new problems and the answers to those problems are software tools which are excellent utilities for debugging these problems. I want to once again thank all those developers who made these tools, continue to improve and build new tools. It makes the job of a support professional much easier and helps a lot of customers resolve complex problems. Thank you!

Comments

• Anonymous
  February 24, 2009
  Dude, this is pretty impressive of you...our Sherlock Holmes :-)

• Anonymous
  June 05, 2009
  If the data is encrypted then what did you do to decrypt to get the hex?  It says "Here is the encrypted data in hex" so it needs to be decrypted doesn't it? Thanks

• Anonymous
  June 15, 2009
  Actually the wording "encrypted data in hex" is a little misleading. I'll correct that. Thank you for letting me know. What I am looking at is the information during the SSL handshake. In the alert, it is only 2 bytes which basically means the reason for the failure. So nothing needs to be done to "convert" the data.

• Anonymous
  April 04, 2010
  Nice job. ;)  I have always used network sniffers to diagnose web app problems.  I stumbled across your article because I had a similar issue, although my alert code is 40.  Simple "handshake failure" without much else to go on.  Lovely. creeping up on a year after the last comment to this post...not sure I'll get a response. :/