This is how we troubleshoot Windows interoperability issues in the Open Specifications support team

Hi y’all, Sebastian from Texas here!

I’ve been at the File Sharing Plugfest last week.

This time around I’ve had the chance to give a presentation on how we use the different sets of tools that we have at hand (both internal and external ones) when working on support cases related to the open specifications world.

Attendees showed good interest on the presentation, so I decided to write a blog about it.

Even though the original presentation was more focused on SMB, this adaptation applies for any of the protocols we support.

In this post I will talk about the “What”. I will later work on one or more posts about the “How”.

Here we go!

 

Introduction

When working with wire protocols, a network capture is almost always involved in the troubleshooting process; however, sometimes we need more tools to look at the problem from a different perspective.

Memory dumps, debugging, logging and tracing are often key tools used in finding the root cause of a problem.

 

Network Monitor 3.X

Network captures are often enough to find protocol issues.

For those that are not familiar with the tool, Network Monitor has full parsers that are based on the MCPP and WSPP documents as well as RFCs and other normative documentation.

For certain scenarios, there are “Experts” that expand the tool’s functionality:

-          Nmdecrypt (SSL decryption)

-          Top Users (shows the heavy talkers in a trace)

-          Others… (Experts and parsers can be downloaded from www.codeplex.com and are open GPL, Experts SDK available at: https://nmexperts.codeplex.com/ )

 

Debugging Tools for Windows

There are several tools to debug in Windows but, Windbg is the most common debugger in use for customer issues.

Windbg is a GUI debugging tool that counts with many features including:

–      Ability to do live kernel/user level and local/remote debugging

–      Ability to load post mortem memory dumps

–      Ability to work with Time Travel Tracing dumps

–      Comes in 32 & 64-bit versions

–      Can make use of the Microsoft web symbols server: SRV*DownstreamStore*https://msdl.microsoft.com/download/symbols

For an overview of the tool and to download the installable packet, click here.

 

Time Travel Tracing

What is Time Travel Tracing?

Time Travel Tracing (also known as iDNA tracing) consists of two elements:

-          an utility to capture TTT trace (available to everyone)

-          an extension to Windbg that allows to load the trace for analysis (available only internally)

The novelty of this tool is that when the trace is loaded into WinDbg, it allows the user to move back and forth in the process execution time.

Thanks to this capability, it helps us debug the Windows interoperability behaviors without the need to reproduce the problem or remote live debugging.

As of now, the tool only allows to capture user mode processes.

 

ETW Tracing

What is ETW Tracing?

It’s a tracing facility widely used by Windows OS that is available in user and kernel mode components.

The event provider is registered with ETW using unique GUID and can be dynamically enabled / disabled.

These traces can even be captured with Perfmon however the results have to be sent to us for analysis since the decoding of a log file requires private TMF files.

 

Windows Resource Kit

The Resource Kit is a set of tools and documentation available for download.

These tools can provide lots of insight on what the OS is doing and how the environment is configured.

These tools are standalone (no installation needed) and are mostly command window based.

We generally request logs and outputs from these tools in order to have the most information possible out of the Windows environment we are working with.

As these tools are public, anyone troubleshooting a Windows issue can make use of them.

The ones we use the most are:

-          Klist: Lists the active Kerberos tickets in the cache

-          SrvInfo: Provides loads of information about the OS and its running processes + configuration.

-          Oh: Shows open handles.

-          SrvCheck: Shows information about the shares and their permissions.

 

Sysinternals Utilities

These tools were created (at least most of them) by Mark Russinovich.

They can be downloaded for free from https://technet.microsoft.com/en-us/sysinternals.

What makes these tools great is that they provided a humongous amount of information about IOs, networks, Processes, Services, Registry, etc.

In their vast majority they are standalone tools so they do not need installation.

The most common tools we use are:

-          Process Monitor: It allows users to track IO and Registry activity from the processes perspective. Can also track the activity that happens at the boot process

-          Process Explorer: It’s a task manager replacement tool that provides loads of information about each process. It can even drill down to the stack level and provide insight about each running thread.

-          DiskMon: Provides information about disk activity in great detail.

-          AccessChk: Quick way to understand what types of access a user or group has over a specific resource

 

 I truly hope this introduction enables you to have a better set of tools to work both on your own and with us when troubleshooting interoperability issues.

In upcoming posts, I will provide more information on how to use these tools to get the best out of them.

Until next time!

Comments

• Anonymous
  September 29, 2011
  Hi Sebastian, this is great artical and very help full for me. Looking forward the detail informations. Thanks Sven