How to debug LSASS.exe process
Hi, welcome back,
I've been dealing these days with an issue about a Custom Authentication Package which was crashing LSASS.exe process even before we had the opportunity to log on the machine. So, how can I debug the package/LSASS process with my favorite debugger, WinDbg (Debugging Tools for Windows), to know what's going on there?
To make things easier I use two machines: a Windows XP machine running on Virtual PC and my Windows Vista machine. Virtual PC makes my life much easier as I can recover the machine very easily if I "break" it. It also makes kernel debugging easier as I don't need any cables to connect the machines involved. The target package/LSASS process will run on WinXP, and WinDbg will run on Vista.
1) Let's configure LSASS on WinXP to load and use the problematic package (i.e. MyPackage.dll):
1.1) Copy MyPackage.dll to %SystemRoot%\system32.
1.2) Add "MyPackage" to the Authentication Packages list in the \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa registry key.
2) Let's configure WinXP to run LSASS under a user mode debugger like NTSD.exe (Debugging Tools for Windows). We won't be able to use this debugger directly, but we will be able to use it through WinDbg working as kernel mode debugger on Vista.
2.a) Set Debugger value to REG_SZ "ntsd -d" in HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\lsass.exe.
2.b) Instead of manipulating the registry directly, we may use GFlags.exe (Debugging Tools for Windows) to make these changes: go to Image File tab, enter "LSASS.exe" in Image field, press TAB to refresh, click on Debugger checkbox and enter "ntsd -d" as the debugger.
After these changes NTSD will break in LSASS execution just when LSASS is about to start (thx to -d parameter), so we can debug it from the very beginning of its execution. Now we'll need a kernel debugger (WinDbg in our case) to control NTSD remotely.
3) Let's configure WinXP to start on debug mode and be able to attach a kernel debugger to it:
3.1) Run msconfig.exe (System Configuration utility), go to BOOT.INI tab, click on Advanced Options... button, and select /DEBUG, select & set /DEBUGPORT= to COM1: , and select & set /BAUDRATE= to 115200.
3.2) On Virtual PC go to Edit > Settings menu option, select COM1 and set it to a Named pipe called i.e. \\.\pipe\machinename.
Now our WinXP is prepared to be debugged. We can restart it now. We will see that WinXP hangs before showing the logon screen. NTSD has broken in LSASS, and because LSASS is not executing, WinXP won't start either. NTSD is now waiting for our debugging commands, but we will need WinDbg to send those commands to it.
4) Let's configure WinDbg on Vista to do kernel debugging on WinXP:
On WinDbg go to File > Kernel Debug... menu option, select COM tab, check Pipe checkbox, set Baud Rate to 115200 and Port to \\.\pipe\machinename(the same we used on Virtual PC). When we click OK WinDbg will connect to WinXP as its kernel debugger.
When WinDbg connects to WinXP and WinLogon is about to start, WinDbg shows the prompt of NTSD ("Input> "). We will now be able to send commands to NTSD, like bp to set a breakpoint, etc.
Please take a look to "Controlling the User-Mode Debugger from the Kernel Debugger" topic on WinDbg's help for more info on how to control NTSD from WinDbg.
We will now be able to do user mode (thx to NTSD) and kernel mode debugging at the same time with WinDbg.
Note: I had some issues to load the right symbols (.pdb files), because I couldn't set .sympath on NTSD to c:\symbols, for instance. So I did the following: I took a dump of LSASS process with .dump command, I opened the dump and loaded all the symbols I needed (i.e. lsasrv.pdb, lsass.pdb, kernel32.pdb, ntdll.pdb and MyPackage.pdb). I copied all those symbols to System32 folder on XP, and after that NTSD was able to find them without setting its .sympath.
Note: We may debug CSRSS and WinLogon processes following the same principles. Take a look to "Debugging CSRSS with NTSD" and "Debugging WinLogon with NTSD" topics on WinDbg's help.
I hope this helps.
Alex
Comments
- Anonymous
October 30, 2008
Good information, thanks Alex. I ran through this because I am debugging LSASS for unexpected access violations. The interesting thing I found was that when I run the .dump command, the dumps are saved to the local machine being debugged. Is there a way to save those to the debugger machine instead? - Anonymous
October 30, 2008
That makes sense. The debugger in the problematic machine is acting as a debugging server. The debugger that is controlling the session from a distance is called the debugging client. The client just transmits commands to the server and shows the resulting output. The server is actually doing the debugging. So a .dump command will take effect in the server. - Anonymous
February 05, 2009
The comment has been removed - Anonymous
February 25, 2009
Never mind! I realized that ntsd -d has nothing to do with remote debugging. I was confusing the remote/server concept since I was debugging across machines. But the virtual serial port is considered local. Doh! All is good now. - Anonymous
October 09, 2009
Hey Alex-I really enjoy reading your posts.This one is helpful for lsass, but could you perhaps expand on debugging csrss? Windbg help doesn't do the trick. Can't pop input> with NTSD instructions. :( - Anonymous
October 19, 2009
Hi Kj, did you follow "Debugging CSRSS" topic on WinDbg then? Did you enable CSRSS Debugging as the article says? If that doesn't work, I don't know what could be going on...