Extracting Pictures from a Byte Array in a Memory Dump
I ran into a case recently where there was a large amount of memory being used in a 32 bit IIS process leading to an out of memory exception. The steps to getting the correct data are documented all over the place. Here’s the link to Tess’ site that I learned on: https://blogs.msdn.com/b/tess/archive/2008/02/15/net-debugging-demos-lab-3-memory.aspx
I’ll skip the setup and go straight to the debug of the high memory usage dump. To start with, as always when I’m debugging, we start with what we know:
- Managed application running in IIS
- 32 bit
- Higher than expected memory usage
Our next step is to list what we need to know:
- How much memory is being used?
- Manged or Native?
- Leak or heavy usage?
- What can we do about it?
The first thing to do is figure out where the memory usage is, and whether it’s managed or Native.
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 280 355ac000 ( 853.672 Mb) 41.68%
<unknown> 830 34808000 ( 840.031 Mb) 70.34% 41.02%
Image 1013 12479000 ( 292.473 Mb) 24.49% 14.28%
Heap 218 3189000 ( 49.535 Mb) 4.15% 2.42%
Stack 129 ac0000 ( 10.750 Mb) 0.90% 0.52%
Other 26 14e000 ( 1.305 Mb) 0.11% 0.06%
TEB 43 2b000 ( 172.000 kb) 0.01% 0.01%
PEB 1 1000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 892 36629000 ( 870.160 Mb) 72.86% 42.49%
MEM_IMAGE 1314 13df7000 ( 317.965 Mb) 26.62% 15.53%
MEM_MAPPED 54 624000 ( 6.141 Mb) 0.51% 0.30%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_COMMIT 1823 369e6000 ( 873.898 Mb) 73.17% 42.67%
MEM_FREE 280 355ac000 ( 853.672 Mb) 41.68%
MEM_RESERVE 437 1405e000 ( 320.367 Mb) 26.83% 15.64%
The memory usage we’re concerned about is any committed memory as that will usually be close to the Private Bytes that value in PerfMon. This is shown in the State Summary section as MEM_COMMIT and is currently at approximately 874 MB. We can tell a few other things from the results above by comparing State, Type, and Usage. For one, generally the majority of Image memory will be of state MEM_COMMIT. Right off the bat, we can extract about 292 MB from MEM_COMMIT leaving us with about 582 MB to place still. If we look at the Usage summary, we can see that 840 MB is from <unknown>. That memory usage most often comes from Virtual Allocations. It’s also where any managed memory will be. Going back to our MEM_COMMIT memory, it seems likely that the majority of the 58 2MB will be handled fall under this bucket. We don’t see much memory at all in Heap, so we can conclude that the memory is either allocated by managed memory, or by a native component making use of a managed allocator.
The next step is to look at managed memory and see if we have around 500 MB of memory usage. We can do that by loading psscor and running !clrusage.
0:000> .load psscor2
0:000> !clrusage
Native Heap for aspnet_filter: 0x01220000
Native Heap for .NET: 0x01fd0000
Number of GC Heaps: 8
Heap Size 0x62fb198(103,788,952)
Heap Size 0x303c5b8(50,578,872)
Heap Size 0x41e9e58(69,115,480)
Heap Size 0x2ca3010(46,805,008)
Heap Size 0x3f8209c(66,592,924)
Heap Size 0x25b3b68(39,533,416)
Heap Size 0x20b5a18(34,298,392)
Heap Size 0x17a65a8(24,798,632)
May contain Reserved segments, run with -v switch to see them.
------------------------------
GC Heap Size 0x19f5617c(435,511,676)
Total Commit Size 1f9f8000 (505 MB)
Total Reserved Size 11608000 (278 MB)
Initial reservation type: All at once
Initial Allocation Size: 18000000 (402,653,184) (384 MB)
Reserved Memory Size: 31000000 (822,083,584) (784 MB)
Reserved Memory Limit Size: 31000000 (822,083,584) (784 MB)
The managed committed memory is 505MB. This lines up with what we were expecting to see if the memory pressure came from managed memory. That means we’re investigating a managed memory issue. We can now move on to seeing what is taking up all of the managed memory by using !dumpheap -stat
0:000> !dumpheap -stat
Using our cache to search the heap.
0x7aa50c98 1 12 System.Diagnostics.CounterCreationDataCollection
0x7aa40c38 1 12 System.ComponentModel.Design.RuntimeLicenseContext
0x7a5eed4c 1 12 System.ComponentModel.WeakHashtable+WeakKeyComparer
0x7a5eeb38 1 12 System.ComponentModel.BooleanConverter
<snipped for brevity>
0x000e8600 521 24,483,528 Free
0x02af0c6c 119,732 74,368,372 System.String
0x02af36dc 11,634 313,028,160 System.Byte[]
Total 672,044 objects, Total size: 435,460,280
Right away, those Byte arrays stand out. It should be stated, that we usually expect to see Byte arrays at the bottom of the list, but as there are 313 MB of them here, it’s definitely something we want to look into.
There are a couple ways to proceed. We could dump all byte arrays and look for a pattern in the sizes, or we could look for large byte arrays. We’ll start with large byte arrays. By large, we mean anything that would have ended up in the large object heap. Since we know LoH objects are all greater than or equal to 85,000 bytes, we can specify a minimum size when we view the heap to limit to these large objects.
0:000> !dumpheap -mt 0x02af36dc -min 85000 -stat
Using our cache to search the heap.
Address MT Size Gen
Statistics:
MT Count TotalSize Class Name
0x02af36dc 139 305,074,736 System.Byte[]
Total 139 objects, Total size: 305,074,736
This shows us we have 139 Byte Arrays in the large object heap that account for 290.94 MB of our managed space. Since there were over 11 thousand byte arrays total and all together they accounted for 298 MB, it’s fair to say the LoH Byte Arrays should be the focus of our investigation. Our next step then is to dump out the Byte Arrays and begin looking to see what they are and if they’re still rooted.
0:000> !dumpheap -mt 0x02af36dc -min 85000
Using our cache to search the heap.
Address MT Size Gen
0x202fb838 0x02af36dc 2,077,524 3 System.Byte[]
0x205dc320 0x02af36dc 3,704,228 3 System.Byte[]
0x209648d8 0x02af36dc 4,990,036 3 System.Byte[]
0x20e26d40 0x02af36dc 125,592 3 System.Byte[]
0x3b1f0038 0x02af36dc 3,449,224 3 System.Byte[]
0x3b53a1d0 0x02af36dc 109,140 3 System.Byte[]
0x3b554c38 0x02af36dc 2,625,400 3 System.Byte[]
<snipped for brevity>
I ran !gcroot on several of the address listed, but did not find any roots. As it turns out, the dump I was looking at was taken on a terminate process breakpoint and many of the managed structures were already in the process of being torn down. Since we couldn’t find any roots explaining why the Byte Arrays were still around, we have to look at the Arrays themselves to figure out what they are and hopefully tie them back to some source code.
A quick db on one of the Byte Arrays shows
0:000> db 0x202fb838
202fb838 dc 36 af 02 46 b3 1f 00-ff d8 ff e1 63 1d 45 78 .6..F.......c.Ex
202fb848 69 66 00 00 4d 4d 00 2a-00 00 00 08 00 0a 01 0f if..MM.*........
202fb858 00 02 00 00 00 16 00 00-01 b2 01 10 00 02 00 00 ................
202fb868 00 25 00 00 01 c8 01 12-00 03 00 00 00 01 00 01 .%..............
202fb878 00 00 01 1a 00 05 00 00-00 01 00 00 01 ee 01 1b ................
202fb888 00 05 00 00 00 01 00 00-01 f6 01 28 00 03 00 00 ...........(....
202fb898 00 01 00 02 00 00 01 32-00 02 00 00 00 14 00 00 .......2........
202fb8a8 01 fe 02 13 00 03 00 00-00 01 00 01 00 00 47 46 ..............GF
Most of what we see is unhelpful, but we see the string “Exif” suggesting this might be a digital photo. Dumping further into the array we see another interesting string:
202fb9f8 00 00 00 00 00 00 45 41 53 54 4d 41 4e 20 4b 4f ......EASTMAN KO
202fba08 44 41 4b 20 43 4f 4d 50 41 4e 59 00 4b 4f 44 41 DAK COMPANY.KODA
202fba18 4b 20 45 41 53 59 53 48 41 52 45 20 56 31 30 37 K EASYSHARE V107
202fba28 33 20 44 49 47 49 54 41 4c 20 43 41 4d 45 52 41 3 DIGITAL CAMERA
That certainly supports the idea that we have a digital photo taken with a digital camera (a Kodak Easyshare to be precise). Checking a few of the other byte arrays, we can see the same pattern.
0:000> db 0x202fb838
202fb838 dc 36 af 02 46 b3 1f 00-ff d8 ff e1 63 1d 45 78 .6..F.......c.Ex
202fb848 69 66 00 00 4d 4d 00 2a-00 00 00 08 00 0a 01 0f if..MM.*........
0:000> db 0x3c478e80
3c478e80 dc 36 af 02 1c c6 49 00-ff d8 ff e0 00 10 4a 46 .6....I.......JF
3c478e90 49 46 00 01 01 01 00 60-00 60 00 00 ff e1 19 78 IF.....`.`.....x
3c478ea0 45 78 69 66 00 00 49 49-2a 00 08 00 00 00 11 00 Exif..II*.......
0:000> db 0x3a1f0038
3a1f0038 dc 36 af 02 0a ff 32 00-ff d8 ff e0 00 10 4a 46 .6....2.......JF
3a1f0048 49 46 00 01 01 01 00 48-00 48 00 00 ff e1 23 27 IF.....H.H....#'
3a1f0058 45 78 69 66 00 00 49 49-2a 00 08 00 00 00 10 00 Exif..II*.......
At this point, we’ll assume they were all digital photos. We can revise that assumption if source review doesn’t agree with us. At this point, we really need to look at the source to see why the pictures are staying around. I don’t have the source available as this was given to me as a dump taken during shutdown, so exporting the modules didn’t work. The only thing left to really get from the dump is the pictures themselves. All that’s left to do is export a few of the byte arrays to disk so we can open them and maybe narrow down what we need to look at in the source (are these employee photos? a family outing? product photos?)
We’ll use the writemem command in windbg and simply write the byte array from the address of the array plus 8 to avoid the metadata.
0:000> !do 0x202fb838
Name: System.Byte[]
MethodTable: 02af36dc
EEClass: 028aeb7c
Size: 2077522(0x1fb352) bytes
GC Generation: 3
Array: Rank 1, Number of elements 2077510, Type Byte
Element Type: System.Byte
Fields:
None
0:000> .writemem f:\output.jpg 0x202fb838+8 L?0x1fb352
Writing 1fb352 bytes........................................
Then we can open it up and view our result. I dumped several out, all of which were pictures of kids on what looked to be a field trip. That helped narrow down the user activity and focus the source review for the resolution. If the source code didn’t provide a resolution, our next step would have been to take another dump when the memory was high but the process wasn’t shutting down. This should have shown us how the Byte Arrays were rooted at which point we would have been able to handle them.
In this case, I used the following command to export all of the pictures at once:
.foreach (var {!dumpheap -mt 0x02af36dc -min 85000 -short}) {.writemem f:\Pictures\${var}.jpg ${var}+8 L?poi(${var}+4)}
-Jarrod