How to Configure Data Captures for Intermittent/Sporadic SChannel Events

 

Recently I have seen a large uptick in the number of SSL/TLS cases where Schannel Event IDs are logged. In some cases the event is not easily reproducible and sporadic. These are without a doubt the most challenging types of issues to diagnose and resolve. Data is needed at the time of the event to adequately capture exactly what is happening not only on the wire but also with our Security Support Provider (Schannel). The Schannel Event IDs that are observed in the System Event Log vary but the two most common are 36888 and 36887. These events signal a fatal alert in the SSL/TLS communication between clients and servers. The alerts are generally encrypted and a network trace alone is very rarely sufficient enough to determine the exact nature of the problem.

For a full list of SChannel Events please see the below link.

https://technet.microsoft.com/en-us/library/dn786445.aspx?f=255&MSPPError=-2147217396

Sometimes the fatal alert is easily reproducible and sometimes it is intermittent and sporadic. So how can you gather the data that Microsoft Support will need to thoroughly diagnose your issue?

At a minimum I always ask for the below sources of data.

a.) Network capture from both the client and server while the issue is happening. I realize that sometimes this is not always possible because you may not own both sides or only one side may be running Windows. Netmon 3.4 is preferred and you can download it here. Wireshark can also be used but the method I detail later will require netcap which is installed with Netmon.

b.) Schannel tracing from both the client and server if you own them and they are running Windows.

c.) System Event log from both the client and the server. Again, this assumes you own both sides and both are running Windows. I prefer that the log be saved as a .CSV file so that I can easily sort and filter in Excel.

 

So you might be asking, “how can I capture this event and send Microsoft Support the data that they need if it is not easy for me to predict when the event will occur?”  Good question. I put together a way to do this and tested it in my lab to verify that it does indeed work as expected. Again, if you own both the client and the server and they are running Windows please do this on both machines. In these scenarios there are typically network devices in the middle such as load balancers and having both sides is extremely helpful in ruling those in or out.

 

Setting Up a Scheduled Task to Stop NMCap and Schannel Tracing After an Schannel Event is Logged

1.) Download and install Netmon 3.4 from the link I supplied above. The method I use to start and stop the capture relies on NMCap which is installed with Netmon.

2.) Open an Administrative command prompt and enter the below command as one continuous line:

logman create trace "ds_security" -ow -o c:\ds_security.etl -p {37D2C3CD-C5D4-4587-8531-4696C44244C8} 0xffffffffffffffff 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 –ets

3.) In the same administrative command prompt start NMCap with the following command:

start nmcap /network * /capture /file c:\nmcap.cap:200m /stopwhen /frame (ipv4.address == ipconfig.localipv4address) AND (Ipv4.DestinationAddress == 1.2.3.4)

4.) Create a batch file on the machine with the below commands and call it something like StopCap.bat

@echo

Ping 1.2.3.4

logman stop "ds_security" –ets

4.) Now you need to set up a task in Task Scheduler to run when whatever the particular Schannel Event ID is logged in the System Event log.

Note: In the example below I use the Event ID 36888 because it is the most common but you can alter it to whatever event you are seeing.

Add the criteria that will be used to trigger the task to run. In my example below it will run when Schannel Event ID 36888 is logged.

Under the Actions Tab choose to start a program and put in the location and name of the batch file you created in Step 4 above. In my example is is c:\stopcap.bat

 

 

Click OK and you are done.

So what happens now you ask? Well, both Schannel tracing and the NMCap will run until the Event ID that you are looking for is logged (it uses a 200MB circular buffer). As soon as it is logged, the Scheduled Task that you set up will run the batch file which sends a ping to IP Address 1.2.3.4 which is the trigger for NMCap to stop. Additionally it runs the command to stop the SChannel tracing.

Gather the following files that were created and zip them up.

c:\NMCap.cap

c:\ds_security.etl

System Event log saved as .CSV file

When you call to open up the support incident let the engineer know that you are smarter than the average bear and have already collected the information that would have been requested. They will be impressed and it will save you time and money in the long run. In the majority of cases we can figure out your issue with the above collected data. In a minority of cases we may have to get much deeper level tracing which your support engineer can readily assist you in gathering.

As always, if this post was helpful to you please let me know in the comments below.

Cheers!

Comments

• Anonymous
  October 30, 2016
  As a side note you can narrow the NMCap to capture only the traffic from the TCP port you are concerned with. Most of the cases involve SSL/TLS on port 443 but this is NOT always the case. If you know which TCP port you are concerned with you can add tcp.port == to the nmcap command. So if I only wanted to capture traffic with a source or destination of tcp port 443 the command would be modified to look like this: start nmcap /network * /capture tcp.port == 443 /file c:\nmcap.cap:200m /stopwhen /frame (ipv4.address == ipconfig.localipv4address) AND (Ipv4.DestinationAddress == 1.2.3.4)
• Anonymous
  October 30, 2016
  The comment has been removed
  • Anonymous
    November 03, 2016
    The second step is only for SChannel tracing and would not be of value for SMTP traffic. You can modify Step 4 and put whatever event it is that you may be looking for. This way you could still get the Netmon capture stopped when the event happens.