The Case of the Frozen Clock Gadget
Besides Aero Glass, one of the most visible features of Windows Vista is the Sidebar with its set of default Gadgets, like the clock, RSS feed, and photo viewer. The convenience of having frequently-accessed information on the desktop and the ease of their development has led to the availability of literally thousands of third-party Gadgets through sites like the Windows Vista Gadget Gallery. I’ve downloaded and installed a few out of curiosity, and in some cases kept them in my Sidebar’s standard configuration, and never experienced a problem. A few days after installing a batch of new Gadgets, however, I noticed that a third-party clock Gadget had stopped updating, and so I set out to investigate.
My system was otherwise functioning normally, so my first step was to see if something was amiss with the Sidebar’s configuration. I right-clicked on the Sidebar screen area and selected the Properties menu item, but instead of displaying the Sidebar configuration dialog, the Sidebar crashed:
Gadgets run inside of shared Sidebar processes, so my first thought was that memory corruption in the Sidebar process had caused the clock to stop and subsequent crash, and verifying that theory required that I analyze the crash. The Windows Error Reporting (WER) service creates a crash-dump file, which is the saved state of a faulting process, in case you agree to send information to Microsoft about a problem. I clicked open the View Details area to see where Windows had saved the dump:
The last path displayed by the dialog, WERD8EE.tmp.mdmp, is a dump file, so I launched the Microsoft Debugging Tools for Windows Windbg utility and opened the file. When you open a dump file, Windbg automatically shows you the instruction that ultimately lead to the crash. In this case, it was a memory copy operation in Msvcrt, the Microsoft C Runtime:
The right side of the line showing the instruction indicates that the target address of the copy is 0. When a memory resource is exhausted, memory-allocation functions typically return address 0, also known as a NULL pointer, because that’s an illegal address by default for a Windows process (an application can manually create read/write memory at address zero, but in general it’s not done). The fact that Sidebar referenced address 0 didn’t conclusively mean the crash was due to low-memory instead of corruption, but it appeared likely.
I next looked at the code that led to the crash, which would tell me if it was a Gadget or the Sidebar itself that had passed a NULL pointer to the C Runtime. To do so, I opened Windbg’s stack dialog:
I had previously configured Windbg’s symbol path to point at the Microsoft symbol server so that Windbg reports names of internal functions in Windows images, because knowing function names can often make understanding a dump file easier. The functions listed in the stack trace implied that Sidebar was querying the version of a “package” when it crashed. I’m not sure what the Sidebar calls a package, but the trace did seem to show that Sidebar was the culprit, not a Gadget.
So had Sidebar run out of memory? There are several types of resource exhaustion that can cause a memory allocation to fail. For example, the system could have run out of committable memory, the process could have consumed all the memory in its own address space, or an internal heap could have reached its maximum size.
I started by checking the committed memory, since that was quick. Total commitable memory, also known as the commit limit, is the sum of the paging file(s) and most of physical memory. When commitable memory runs low, Windows Vista’s low-resource watchdog warns you by presenting a list of processes consuming the most memory and gives you the option of terminating them to relieve the memory pressure. I hadn’t seen a warning, so I doubted that this was the cause, but opened Process Explorer’s System Information dialog to check anyway:
As I suspected, there was plenty of available Committable memory. I next looked at Sidebar’s virtual memory usage. Memory leaks are caused when a process allocates virtual memory, stores some data in it, uses the data, but doesn’t free the memory when it’s done with the data. Virtual memory that processes allocate to store their own data is called Private Bytes, so I opened Process Explorer and added the Private Bytes column:
On a 32-bit Windows system, processes have 2 GB of address space available to them by default, so the highest possible Private Bytes value is close to 2 GB, which is exactly what the Sidebar process with process ID 4680 had consumed. That confirmed it: a memory leak in Sidebar caused it to run out of address space, which in turned caused a memory allocation to fail, which finally caused a NULL-pointer reference and a crash. I suspect that the clock stopped when Sidebar’s address space was exhausted and the clock Gadget couldn’t allocate resources to update its graphic.
Next I had to determine which Gadget was causing the leak, which may or may not have been the frozen clock Gadget. The Sidebar consists of two processes, one Sidebar.exe process that hosts the Windows Gadgets and a child Sidebar.exe process for third-party Gadgets. At this point I knew that a third-party Gadget had leaked memory or caused the Sidebar to leak, but I had several third-party Gadgets running and I didn’t know which one to blame. Unfortunately, the Sidebar offers no way to track memory usage by Gadget (or any other resource usage for that matter), so I had to apply manual steps to isolate the leak.
After restarting the Sidebar, I removed the third-party Gadgets and added them back one at a time, leaving each to run for a minute or two while I monitored Sidebar’s Private Bytes usage. I added the Private Bytes Delta column to Process Explorer’s display to make it easy to spot increases, and after adding one of the Gadgets I started to see periodic positive Private Bytes Delta values, implicating it as the leaker:
Now that I knew the guilty Gadget, I could have simply uninstalled it and considered the case closed. But I was curious to know how the Gadget had managed to cause a leak in the Sidebar – a leak that persisted even after I removed the Gadget.
I navigated to the Gadget’s install directory and opened its HTML file to see what it was doing. The Gadget consists of around 3-dozen lines of pretty simple Javascript and I didn’t spot anything amiss. To narrow in on the problematic code, I began commenting out pieces and re-adding the Gadget to the Sidebar until the leak disappeared. The code I was left with was a function the Gadget configured to execute every 10 seconds to update its graphics. It called the Sidebar background object’s RemoveObjects method and then added back graphics and text by calling the background’s AddImageObject method. Here’s a simplified version of the relevant code:
The fact that it was using these APIs correctly meant that the leak was in the Sidebar’s code, but a quick Internet search didn’t turn up any mentions of a leak in the background object. If Sidebar APIs had a memory leak, why wasn’t it well known? I scanned the source code to the other Gadget’s on my system and discovered that none of them used the APIs, which explained why the leak isn't commonly encountered. However, comments in the Windows Gadget Gallery for the Gadget that inadvertently caused the leak revealed that other users had noticed it.
Having tracked the original unresponsive Gadget problem down to a leaky Sidebar API, I filed a bug in the Windows bug database and closed the case.
Comments
Anonymous
January 01, 2003
Mark, thanks for the reply. Yes, I know the problem is with a recursive call to SetInterval. The point I was trying to make is actually how badly the operating system behaves under a timer leak (even a non-exponential leak). Try writing a simple C++ application which simply creates lots of timers with a NULL handle. If this was done over a sufficiently long period, you will see that the only visble symptom is a general slowness of all UI threads with excessive kernel mode cpu usage. There will be very few clues that point to what's wrong or which process is causing the slowness. I was wondering what debugging techniques you would employ to track this down.Anonymous
January 01, 2003
Amazing post! really gets in depth to the "digging to the bug" process in a very educative and entertaining way. M.-Anonymous
January 01, 2003
RSS Feed 1.1.0.0 Gadget with Vista and also Win7 32bit seems to cause continuous increasing small memory leaks too.... :(Anonymous
January 01, 2003
In my experience - and probably not just my own - the quality of Windows Sidebar is far, far below par for such a prominent feature in such a major release of the world's most popular operating system. Like I wrote in March: "I don't know who signed off on a Sidebar of this quality going into the Vista release, but whoever it was, it must have been the kind of B category employee that Bill Gates supposedly said Microsoft ought not to employ. "The whole Sidebar thing looks like it's been polished by Microsoft, but developed by students. It's just not good enough. Not by a wide margin." http://denisbider.blogspot.com/2007/03/doghouse-windows-sidebar.htmlAnonymous
January 01, 2003
The comment has been removedAnonymous
January 01, 2003
Con is a reserved name in Win32 that represents serial ports. Thanks for the feedback.Anonymous
January 01, 2003
The comment has been removedAnonymous
January 01, 2003
A big thanks for taking the time to debug and file a bug report. Otherwise I'm sure all the blame is falling on the gadget's author, and worse, soon there will be a lot of folklore about "don't call RemoveObjects because there's a leak in sidebar.exe ..." Two observations:
- somebody isn't checking for NULL after calling malloc(). sidebar.exe should never crash.
- Microsoft is isolating their gadgets from everyone elses by running two sidebar.exe? Are their gadets more trustworthy than others? Why not isolate every gadget on its own?
Anonymous
January 01, 2003
The Windbg debugger and debug symbols are available to anybody. See the link to the symbol server in the blog post.Anonymous
January 01, 2003
Yes, the package-version functionality was a victim of the leak. The way to read the symbol is: <image>!<class>::<method> Thanks for the feedback.Anonymous
October 15, 2007
Was the function checking a version just a victim of the memory leak then? And when looking at stack traces with lines like: sidebar!Package::EnsurePackageMinVersion Is the bit after the ! and before the :: a c++ object? Are there any good resources to read to help understand stack traces? Great post!!!Anonymous
October 15, 2007
Great in-depth explanation, as usual. Could the repeated use of window.SetTimeout within the same function the source of the problem? Would it be easy to convert the code to use window.setInterval instead, and compare? I could try it myself if I knew which gadget it is! q;-)Anonymous
October 16, 2007
I expected GetFileVersionInfoExW to return an error instead of using a null-pointer. Why not file a bug for that?Anonymous
October 16, 2007
Your post rocks, as always !Anonymous
October 16, 2007
Do you think that reporting the bug to Microsoft through the Error reporting service would have lead to the same ? Since this error is due to something local (a gadget) that expose a memory leak in a windows API.Anonymous
October 16, 2007
I love these thrilling investigations of yours Mark! Very educative and entertaining! Thanks a lot! /TommyAnonymous
October 16, 2007
May I ask what did you write when you filed the issue in the Windows Bug database? I'm just curious on how would you file a bug like this.Anonymous
October 16, 2007
I wish someone would notice the celsius bug in built-in Weather gadget.If you have it set for showing degrees in celsius, it will only check the weather rougly once a day. You will notice this as your work day progresses and the weather changes outside, but not in the gadget.
Change back to fahrenheit anytime, and the weather changes to the correct status.
Set it to degrees again and the wrong weather status (from morning time) returns. I have tried to locate the source of the bug but seems to come from the dll "wlsrvc.dll" in "c:Program FilesWindows Sidebar", which the gadget uses for weather services. Maybe this could be a future case story?
Anonymous
October 16, 2007
As always great post.Anonymous
October 16, 2007
Man I love this stuff! Thanks, Mark.Anonymous
October 16, 2007
cannot understand one thing, how come such critical APIs passthrough testing with this kind of Bug !!.Anonymous
October 16, 2007
I know -exactly- which gadget ran into this memory leak, heh. Or, at least, I know of -a- gadget that, if left alone for a few days (say, over the weekend) causes sidebar.exe to start nomming up RAM. And then, when you close it, sidebar ramps up. What I find most amusing, though, is that the ending of sidebar.exe doesn't identify the correct gadget that may have caused the problem...Anonymous
October 16, 2007
Hi Mark, I have seen it on my machine also, and the symbols drive me to same leaking scenario. quite interesting stuff. btw is any public release of the 6.8.0.0.1 windbg ? Thanks, cosminAnonymous
October 16, 2007
The comment has been removedAnonymous
October 16, 2007
The comment has been removedAnonymous
October 17, 2007
"Two observations:
- somebody isn't checking for NULL after calling malloc(). sidebar.exe should never crash. ... " This is false. If after a malloc, you get a NULL pointer, what can you do? Trying to prevent that makes only the program crash elsewhere and its really tough track back the error in that case, because the stack trace is irrelevent. I suggest only adding debug asserts. May I had that powerful memory leak revealer tools exist for both Native and Managed code. A trivial leak like this might have been easily found.
Anonymous
October 17, 2007
Strange. It's not only a bug. Looks also like a bad design. This canvas API's should not make any File-Version-Check ? Or do I miss something ?Anonymous
October 18, 2007
The comment has been removedAnonymous
October 19, 2007
Great, Thanks Terry.Anonymous
October 19, 2007
The comment has been removedAnonymous
October 19, 2007
By the way, "con" represents the console (keyboard) I think.Anonymous
October 20, 2007
inform the user how? bearing in mind anything you do can't involve allocating memory... and if you do inform the user, what then? it's not like you can do anything about it, so you'll have to close anyway. closing your application and informing the user is, not coincidentally, what happens when an application crashes, with the added advantage that a bug report is generated that might lead to the issue being fixed.Anonymous
October 20, 2007
I've submitted those automatic crash reports for years, without seeing any change after many updates. I've concluded that if I'm very lucky and the crash is resolved, it will be in the next version of Windows, not this one.Anonymous
October 21, 2007
Mark, are you seen the bug in the Calendar Gadget? http://www.fayerwayer.com/up/2007/10/vista-calendario.png (13th August is twice) I tried to fix it, but is very tricky because I discovered Javascript was the origin of the error, especially with Date.getDate() and Date.getDay() functions.Anonymous
October 22, 2007
The comment has been removedAnonymous
October 25, 2007
"(after coming all this way to a protected address space with NT [...]) that we are seeing a problem where one gadget's misfortune is affecting all the others." No, we are seeing a problem where the OS's misfortune is affecting all of the applets. In some ways it's like getting a BSOD, though not as serious. In Windows 2000 if you used Windows Media Player and got a BSOD, then the misfortune affected all running applications (and affected disk files if any had been open for writing at the time). But Windows Media Player wasn't the reason, the video driver was the reason. Microsoft did better testing on video drivers that they put into XP. Now maybe they'll do better testing on user mode libraries that they put into the successor of Vista.Anonymous
October 28, 2007
The comment has been removedAnonymous
October 30, 2007
the "n/a" you are seeing is because you don't have rights to access that information for the LSM process. Try running Process Explorer elevated and you'll see that information.Anonymous
November 04, 2007
The comment has been removedAnonymous
November 06, 2007
The comment has been removedAnonymous
November 12, 2007
> I've submitted those automatic crash reports for years, without seeing any change after > many updates. I've concluded that if I'm very lucky and the crash is resolved, >it will be in the next version of Windows, not this one. The only answer really is "it depends" - an annoying occasional bluescreen I got every now and then when using nmap (winpcap was the actual crashing program) was solved by an update to the windows kernel about 3 months after I started sending error reports with my computer model, network card, and mentioning the word "nmap". Maybe that was coincidence, maybe not.Anonymous
November 21, 2007
I think the reason for this memory leak is simply the recursive call setInterval. All you have to do is log the D/T stamp and you'll see that the # of leakTimer() call will grow exponenially. So each call to leakTimer it in itself will trigger more calls to itself after the set delay (in addition to the original "timed" called). function leakTimer() { window.setInterval("leakTimer();", 1000); }Anonymous
December 06, 2007
> If malloc() returns NULL then your program is dead in the water. Let's try to use malloc(), LocalAlloc() or HeapAlloc() to allocate 2GB memory at a time, you will definitely receive a NULL pointer. But this does not mean that you program have crashed. The NULL value only means that the API could not complete your request and memory is not allocated in the user space. You could still do a complete cleanup and graceful exit. Even though your memory allocation will not be as large as 2GB, if the system do not have a large enough continuous memory space, you could still receive a NULL pointer. What does 'dead in the water' mean? To me, it means a memory chip failed and flips one or two bits, or the heap table is corrupted. In these cases, your program probably hangs in the kernel or issue an exception, and the alloc() functions simply do not return, and you hardly need to think of any way to handle it. I still remember that several years ago when I read some old programming books, there still had some footnotes saying like "for clarity, the example does not illustrate the handling code, and the readers are responsible for doing this." However, some days later, I saw that most of the people just knew how to copy code, using just the piece of example code without any error handling. What a bad habit......Anonymous
December 13, 2007
I think Joseph and Norman are right: unless it was a very, very small memory allocation which failed (i.e. you've actually used up 99.999% of your available RAM, usually through a huge number of tiny allocations, as in the memory leak case here) you can - and should - try to report the error. It's much more likely for a larger allocation to fail, which can easily be reported and recovered from sensibly. Often the failure isn't a true shortage of resources, but erroneous usage of them: in Norman's case, a DLL which interferes with very large allocations later, in other cases a corrupt or misinterpreted file. (Rename almost any data file to .com/.exe and try running it, you'll get a memory allocation error from Windows: it tries to interpret the data as a program header, resulting in a 'program' which has nonsensical memory requirements like requesting 800k of the 640k conventional memory area.)Anonymous
December 22, 2007
ok, it's been a while since I have programmed. But couldn't you pre-allocate some space on the heap in order to gracefully handle an error event like that? Maybe you'd need a GOTO to get to it, but isn't that ok if your code is in a 'oh heck, I have to quit NOW' situation? Just a thought for graceful exiting in a memory allocation failure situation.Anonymous
December 30, 2007
2 CJ and ShawnNa there was window.setTimeout and not window.setIntervalAnonymous
February 27, 2008
This seemed a fitting place to make this wonderful announcement. See the list w/ babyfaced Mr. Gates on it here: antiaging4geeks.com Just scroll down the page a bit. AND, just for the record, I noticed this exact problem with my sidebar gadgets: it is especially interesting when an afterimage remains and I have to left click and attempt to select the area in order to remove it. Most interesting. I had been the most rabid Microsoft/Windows/PC supporter in the multiverse until recently. I am still a fan, but endless glitches and the iPhone have led me somewhat astray. Still love Vista, though, for all its issues. Peter J. Lupo Esq. antiaging4geeks.comAnonymous
March 16, 2008
Unbelievably, this is not fixed in SP1. I'm curious if your "report" to Microsoft was even looked at. Something tells me it sits in the same status as it did when reported in October 07. A shame really, because though I can turn the sidebar off, or use an alternative, you'd think Microsoft would give attention to something most casual PC users would play with. For these are the people who will be affected most, and not have a clue about the cause. Thanks.Anonymous
August 13, 2008
The comment has been removedAnonymous
August 19, 2008
Hi, My gadgets have completely frozen!!!! I downlaoded about 30 new ones today & they all kept going on the sidebar which is now frozen. I shouldn't have put them all there but the side bar just kept making new columns & I've only had my laptop for about a month. It's a great computer, i have not had one single issue. Now the whole side bar is black, even after trying to restart the computer about 10 times!!! Please help! What should I do? You can email me at blondebabyfashionista@yahoo.com.au Thankyou so muchAnonymous
October 29, 2008
So if I am experiencing the problem right now, what should I do? Contact PSS and get a hotfix? PS: I think you need to do a blog article on this.
