Capturing memory dumps for 32-bit processes on an x64 machine
This is an issue that I often get questions around and we often have cases where we have to re-capture memory dumps because the memory dumps were captured the “wrong” way.
The short story is: If you are executing a 32-bit process on a 64-bit machine, (which is the default case for IIS on x64 machines for example) you need to capture the dump with a tool that allows you to create 32-bit dumps.
How do you know if your process is 32-bit?
If you are on a 64-bit machine, you can check task manager to see what architecture your process is using.
Processes with *32 are 32-bit and the rest are 64-bit so in the example above we can see that for example w3wp.exe is executing 32-bit code.
Why is it important to capture them with the right tools?
If you capture a dump with a tool that captures 64-bit dumps you will still get a memory dump, but you will get a memory dump of the syswow64 which means that a lot of extensions like sos and psscor2 won’t be able to read the data. Some things may still work but it is very limited and you might get errors or erroneous call stacks etc.
Typical stuff that happens when you try to read a 64-bit memory dump of a 32-bit process is:
1. You may see warnings like
WARNING: wkscli overlaps srvcli
..............WARNING: wship6 overlaps dnsapi
.WARNING: IPHLPAPI overlaps dsrole
...WARNING: FWPUCLNT overlaps rasadhlp
WARNING: FWPUCLNT overlaps dnsapi
.....WARNING: compstat overlaps iisres
2. The stacks show wow64cpu methods
0:000> kp
Child-SP RetAddr Call Site
00000000`000ce728 00000000`73a22bcd wow64cpu!CpupSyscallStub+0x9
00000000`000ce730 00000000`73a9d07e wow64cpu!Thunk0ArgReloadState+0x1a
00000000`000ce7f0 00000000`73a9c549 wow64!RunCpuSimulation+0xa
00000000`000ce840 00000000`76d684c8 wow64!Wow64LdrpInitialize+0x429
00000000`000ced90 00000000`76d67623 ntdll!LdrpInitializeProcess+0x17e2
00000000`000cf290 00000000`76d5308e ntdll! ?? ::FNODOBFM::`string'+0x2bea0
00000000`000cf300 00000000`00000000 ntdll!LdrInitializeThunk+0xe
3. You see that all addresses are 64 bit addresses, (i.e. 00000000`76d5308e rather than 76d5308e), even though it is a 32-bit process.
4. You get errors like the below when trying to run sos commands.
(Note: You can get these errors if you don’t have symbols properly set up as well, so this is not the only reason for getting these errors)
0:000> !eeheap -gc
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer) 2) the file mscordacwks.dll that matches your version of clr.dll is in the version directory 3) or, if you are debugging a dump file, verify that the file mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path. 4) you are debugging on the same architecture as the dump file. For example, an IA64 dump file must be debugged on an IA64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
What tools should you use?
The best way to capture a process dump in this scenario is to use 32-bit versions of tools like Debug Diag, or adplus+cdb (32 bit debugging tools for windows). You can install the 32-bit versions of these tools on a 64-bit system. In the next version of Debug Diag you will be able to capture 32-bit dumps with the 64-bit version of Debug Diag.
I have mentioned before that you can capture memory dumps with task manager on Vista+, but if you do this on a 64-bit machine you will get a 64-bit dump. You can however use the 32-bit task manager, located in C:\Windows\SysWOW64\taskmgr.exe to get 32-bit dumps. If you want to verify that you are running the 32-bit version of task manager, you can check that taskmgr.exe is listed with *32 in task manager itself.
Edit: Sorry Mark and David, I of course didn't mean to leave out the nice and very lightweight procdump tool from SysInternals which will capture 32-bit dumps of 32-bit processes even on 64-bit Windows. Thanks for reminding me:)
For some reason I thought that I had written about this tool in an earlier post but a quick search reminded me that this is still in my blog-todo bin.
For all other dump capturing tools, I would suggest that you read the documentation to see if there is a 32-bit version (like for procmon where you can pass in the /run32 parameter) or if they will capture 32-bit dumps of 32-bit processes.
Is it important to read the dump with the right “bit” debugger?
Yes, to read 32-bit dumps you should use 32-bit windbg (either on a 32-bit or 64-bit OS), and for 64-bit dumps, the machine that you debug the dumps on, needs to be a 64-bit machine, running 64-bit windbg.
Remember that you should also use the 64-bit versions of sos, psscor2, sosex and any other extensions that you might use, if you are debugging a 64-bit dump.
Have a good one,
/Tess
Comments
Anonymous
September 28, 2010
The comment has been removedAnonymous
September 28, 2010
Tess, In my experience, w3wp defaults to 64 bit on 64 bit windows. 32 bit is enabled when I change the app pool to 'enable 32 bit applications' JohnAnonymous
September 29, 2010
The comment has been removedAnonymous
September 29, 2010
And procdump has a significant number of other features that make it much more flexible and applicable to more scenarios.... -Mark (with a heavy heart)Anonymous
September 29, 2010
Well if we're going to be technical, there's also no such thing as a "running" process, as Windows schedules threads, not processes. So you're really "executing a Wow64 process" JAnonymous
September 29, 2010
Mark, David, I was actually thinking of adding it to the list before along with a few other dump tools, but the list got a bit long, so I just included debug diag and adplus and left the others under (other dump creation tools). The reason I singled debug diag and adplus out was because of the types of issues i normally deal with where you often need to do leaktracking or dump on specific exceptions (handled). For gathering dumps on "hangs", or straight snapshots for memory analysis we usually use taskmgr (because it is already there). Having said that, I personally like procdump a lot and use it very often, and I do agree that it does have a lot of neat features like gathering dumps on memory/CPU threasholds and based on perfcounter info. As I wrote in the post (in the edit) I thought I had written a post about it already but just realized it is still in my todo bin, but I think it does merit it's own post. On a separate note, since I love the nitpicking:) perhaps you should change the help then for procdump technet.microsoft.com/.../dd996900.aspx from "By default Procdump will capture a 32-bit dump of a 32-bit process when running on 64-bit Windows." to ... a 32-bit dump of a 64-bit process running 32-bit code:) ... sorry couldn't resist, thanks for the clarification btw... aionescu, thanks:) changed that, i'll blame it on my Swenglish (Swedish - English) :)Anonymous
September 30, 2010
It's fairly easy to see the 32-bit view of the world if you start seeing wow64! show up in the call trace, just run: !wow64exts.sw I'm 85% sure that this will also let SOS work correctly - no need to worry about being able to decode an "Incorrect Arch" dump!Anonymous
September 30, 2010
Paul, Thanks for pointing out the !wow64exts.sw command, it's definitely very useful in this case. Unfortunately it doesn't fix the whole situation though... it does let you use some commands for sos like !clrstack and other stack related stuff, but for anything that requires sos to look at the heaps, like !dumpheap -stat, !threads, etc. etc. it doesnt work unfortunately. I am not sure what the sw command actually does internally but I have a feeling that it does something similar to changing process contexts with .process in kernel dumps which has similar effects on sos. Perhaps someone else can shed some more light, but either way, to get the full sos functionality you need to get 32-bit dumps...Anonymous
October 04, 2010
Hi Tess, Is there way to get the information, which calls into the dbgHelp.dll have to be done to get a 'correct' 32 bit process dump out of a 64 bit executable. The current tools are pretty nice, but we can't use them. We are having a lot of processes to monitor automatically during development. Therefore we have an own tool, which attaches to every process of interrest within a single thread and creates .dmp files depending on the type/address of an exception. For getting the correct information out of the single processes, the tool has to be run in 32 bit on 32 bit OSes and as a 64 bit one on 64 bit OSes, respetively. But, as you mentioned above, how to call MiniDumpWriteDump out of a 64 bit process for a 32 bit target? thanks in advance andyAnonymous
April 03, 2011
Hi Tess, The above didn't work for me, got "Operation could not be completed. Only part of the memory was written". We have a x64 server and 32 bit development machines. Our memory of the process (ASP.NET) is hitting 2.2gb! I'm sure you get asked this lots but could have you have a 10min glance over it and see where you think we should be looking.Anonymous
May 05, 2011
Procdump is the nice and very lightweight tool to capture 32-bit dumps of 32-bit processes on 64-bit Windows. However, based on the license of ProcDump, we could not use it within our product. Any suggestion of capturing 32-bit dumps on 64-bit Windows in the program.Anonymous
August 16, 2011
@Leo: have a look at channel9.msdn.com/.../Devdays084 esp. 1hr 09 minAnonymous
August 16, 2011
@Mark Russinovich I have a feature request for Process Explorer. Version 15 makes a 64-bit dump of 32-bit processes... Please help our testers.Anonymous
February 12, 2012
You are my hero... I wanna cry :( thanksAnonymous
March 20, 2012
I second the request of Gregory Chernis. The latest Process Explorer 64 bit, V 15.13 still dumps 64 bit of 32 bit processes.Anonymous
August 11, 2013
Thanks for this article ,I launched 32 bit task manager and analyzed the dump.Anonymous
October 29, 2013
I would love to know what MS is thinking when they publish, as the only download I've found, winsdk_web.exe as the installer for windbg which ONLY installs 64b debuggers. Maybe a decade from now that makes sense. Until then, the (&)& installer should install BOTH 32 and 64 bit on a 64b system but I only got the 64. (If the installer sees low storage space it could ASK.) MS should be embarrassed on the entire 64b debacle. I'm off to download the entire ISO just to get the 32b. Wheeee. :-PAnonymous
March 07, 2016
stupid question: what can I do when I have this kind of dumps from Windows Error Reporting?Anonymous
July 29, 2016
From the post:Yes, to read 32-bit dumps you should use 32-bit windbg (either on a 32-bit or 64-bit OS)From MSDN page:https://msdn.microsoft.com/en-us/library/windows/hardware/ff539099(v=vs.85).aspxIf you are analyzing a dump file, you can use either the 32-bit debugging tools or the 64-bit debugging tools. (It is not important whether the dump file is a user-mode dump file or a kernel-mode dump file, and it is not important whether the dump file was made on an x86-based or an x64-based platform.)Which is correct?