The Case of the Periodic System Hangs
A few months ago I began experiencing periodic system freezes of about a second where even my mouse would pause during a movement. Needless to say, this became very annoying very quickly. A few minutes with Process Explorer, however, and I not only determined the cause, but came up with a fix.
One apparent clue as to the cause of the hangs was that I only experienced the freezes when I had the beta release of VMWare 5 running. That fact alone wasn’t enough to blame VMware for the spikes and in any case my reliance on VMware prevented a workaround of simply not using it. I therefore wanted to determine if VMware was really the cause and so the first step of my investigation was to look at the system’s overall CPU history in Process Explorer’s System Information dialog. I clicked on the mini-CPU history chart in the Process Explorer toolbar and in the larger graph confirmed a CPU spike every 10 seconds. I moved the mouse over a spike and the graph’s tooltip reported the System process as the major contributor to CPU usage at the time of the spike:
The System process serves as the host process to operating system worker threads, such as the modified and mapped page writer threads, as well as dedicated device driver threads, so my investigation wasn’t complete: I needed to look inside the System process to see what thread or threads were responsible for the spikes. To do that I double-clicked on the System process to open its Process Properties dialog and selected the threads tab. I pressed the space bar to pause Process Explorer’s refresh updates and at the next CPU spike (it might have take a two or three tries to get the timing right) I pressed F5 to cause a manual refresh:
Two threads contributed the majority of CPU usage. One had a start address in Http.sys and the other in Ftser2k.sys. Http.sys was introduced in Windows XP to serve as an in-kernel Web server accelerator that can serve cached content directly from kernel-mode. I didn’t know what Ftser2k.sys was so I clicked on the Module button to view the file properties for the driver’s file and saw that it describes itself as the “FTDIBUS Serial Device Driver”. This wasn’t very helpful so I investigated some more and found that it’s a driver that provides a virtual serial port interface for USB devices so that applications that aren’t USB-aware can interact with USB devices. I had recently installed XM Satellite Radio’s XM PC Player (which XM has since discontinued) and suspected that it was the application that required the Ftserv2k driver. Closing the XM Player resulted in Ftser2k’s CPU usage dropping to 0, confirming that it was the application using the driver’s services. However, I continued to experience CPU spikes in Http.sys.
My attention therefore turned to Http.sys. Since Http.sys simply implements a cache I theorized that IIS, which I run on my Windows XP system to host the Sysinternals staging site, would function even when Http.sys wasn’t present. I opened a command-prompt and typed “Net stop http” to stop the driver and was informed that several dependent services would also stop if I stopped the driver, but answered affirmatively anway:
Within a few seconds I verified that that stopping the driver ended the CPU spikes. I spent a few more minutes testing IIS and VMware to make sure that the driver’s absence had no adverse effect and came to the conclusion that the system was functioning fine without it. My next step was therefore to disable the driver permanently. I opened the Device Manager, selected “Show Hidden Devices” in the View menu, navigated to HTTP in the “Non-Plug and Play Drivers” node and double-clicked to open its properties dialog. On the Driver tab I changed its startup type from “Demand” to “Disabled”, which would prevent the driver from starting the next time the system booted (note that I could have just as easily navigated in Regedit to HKLM\System\CurrentControlSet\Services\Http and changed the Start value from 3 to 4).
My CPU spiking mystery was solved. I might have spent time trying to determine why Http.sys was causing the spikes in the first place, but since they only occurred when the beta of VMware 5 was running it obviously had something to do with VMware’s networking subsystem. Since I wasn’t using any applications that required Http’s services I turned my attention back to the work I had interrupted with the investigation. I recently re-enabled Http.sys with the final release of VMware 5 running and the spikes no longer occur.
So why do I use VMware instead of Virtual PC, even for my presentations at Microsoft and Microsoft conferences like TechEd? I can answer with one word: snapshots. Snapshots are a VMware feature that allows you to save the state of a VM, make modifications to the VM and later return its state to that of the snapshot. I create a baseline “clean” operating system installation snapshot and perform experiments, including installing software, and can always restart with the clean installation.
VMware 5 takes snapshots a step further by introducing snapshot “trees”. Using snapshot trees I can start with my clean OS snapshot, get the VM into a state that demonstrates an interesting behavior, and then create another snapshot. I make more modifications and snapshot again, or go back to the original clean snapshot and take the VM in a different direction to make another branch of the snapshot tree.
In my malware talk, for example, I use two snapshots to demonstrate how RootkitRevealer can detect the HackerDefender rootkit. I start with the clean OS snapshot, copy HackerDefender to the system, open an Explorer window to the files and create a new snapshot. Then I activate HackerDefender and run RootkitRevealer through its scan where it reports the presence of the cloaked HackerDefender files and Registry keys. Then I snapshot again. During the presentation I resume the first snapshot and show how HackerDefender’s files disappear when I activate the HackerDefender executable. A little later I resume the second snapshot and display RootkitRevealer’s ability to detect the cloaked files, but without making the audience wait through a scan, something which takes several minutes.
I’ve grown so dependent on snapshots that only when Virtual PC includes snapshot trees will I consider switching.
Originally by Mark Russinovich on 7/17/2005 4:48:00 PM
Migrated from original Sysinternals.com/Blog
# re: The Case of the Periodic System Hangs
I feel the same way about VMware 5 versus Virtual PC. I own a license for each, yet I switched to VMware simply because of the snapshots feature. It's too bad, because I sort of prefer VPC--I find it generally easier to use, and lighter on the system.
7/17/2005 6:32:00 PM by Irreligious
# re: The Case of the Periodic System Hangs
Virtual PC? What's that?
:)
7/18/2005 9:55:00 AM by bigjon440
# re: The Case of the Periodic System Hangs
What's your point? VPC is a good product, it just lacks this important feature.
7/18/2005 1:48:00 PM by Irreligious
# re: The Case of the Periodic System Hangs
OK, the final VMWare 5 has been released; have you upgraded and does it fix the problem?
If haven't upgraded, when?
Need to know!!!!
7/19/2005 6:36:00 AM by Shawn
# re: The Case of the Periodic System Hangs
Yes, I mention in the post that the spikes don't occur with VMware 5 final release.
7/19/2005 6:56:00 AM by Mark Russinovich
# re: The Case of the Periodic System Hangs
VPC does have the "Undo disks" feature which by the sounds of it provides the same option as the snapshots.
7/19/2005 2:53:00 PM by dave
# re: The Case of the Periodic System Hangs
I don't think solved is the correct word. You're looking for workaround. It would have been interesting to find out what was causing the spike in http.sys as it seems to be the victim in this case. Did you contact VMWare? One of the problems today is that customers expect ISVs will find all the bugs in their products and provide fixes. During a beta program it's extremely important to provide feedback to the right development teams so these issues don't appear in released code. Luckily, what seems to have been a VMWare bug was fixed before RTM.
7/19/2005 6:37:00 PM by Dan
# re: The Case of the Periodic System Hangs
You're technically correct, but it was solved from my perspective in that it no longer interfered with my productivity and I didn't lose any required functionality with the workaround.
7/19/2005 7:44:00 PM by Mark Russinovich
# re: The Case of the Periodic System Hangs
To dave: VPC differencing "undo" disks really don't provide the same level of flexibility that VMWare snapshots do.
7/19/2005 11:10:00 PM by Wes
# re: The Case of the Periodic System Hangs
Anyone now way multiple threads in the System process is using the HTTP.sys driver on my system when I do _not_ have IIS installed/running?
(WinXP PRO SP2)
--
Christer
7/20/2005 3:45:00 AM by Christer
# re: The Case of the Periodic System Hangs
HTTP.sys is included with Service Pack 2 but isn't used by IIS 5.1. You’ll need to look at what applications and services you have running and check their dependencies. The Universal Plug and Play Device Host is one service in Windows XP Service Pack 2 that has a dependency on HTTP.
7/20/2005 6:13:00 AM by Dan
# re: The Case of the Periodic System Hangs
Thank you for the article. I also experinced a similar issue with the beta of VMWare 5, but ignored it due top project time constraints. Looking back, it could have been the exact same issue. If not, you have shown me a strategy to use next time. Thank you.
VMWare v. MS VPC
I have used both extensively, and consistently find that VMWare performs more effeciently, specifically with Linux installations. A VPC installation of a Linux distribution can take two or more hours, while a VMWare installation can take as little as 30 to 40 minutes. Not willing to take the time to determine root cause, since VMWare works so well.
7/20/2005 10:57:00 AM by Richard
# re: The Case of the Periodic System Hangs
Mark, thanks for this excellent article were the root cause is CPU consumption. I have a similar problem were the root cause seems to be excessive I/O activities on disc. Unfortunately I didn't find a way to trace back the relating hardware interupts to an application. Any hints?
Regards
Michael
7/20/2005 2:58:00 PM by miho
# re: The Case of the Periodic System Hangs
Performance Monitor contains several I/O counters that can show per process statistics. That should work.
7/23/2005 5:25:00 AM by Dan
# re: The Case of the Periodic System Hangs
@Richard: I'm willing to bet that the reason Linux installs take longer under VPC has to do with the CPU time you allow VPC to take. If you select the "Give processes on the host operating system priority" option, then it's no surprise that VPC slows to a crawl.
The solution--and I realize that you don't care for one, but someone else may--is to use the "Run Virtual PC at maximum speed" option, even if only during the guest OS installation.
It also helps a great deal to run a dual-CPU system. :)
8/20/2005 11:31:00 PM by Irreligious
# re: The Case of the Periodic System Hangs
a few days ago I saw this problem of every second spiking, but on the network, this was a machine that had been working fine, we did everything we could think of, up to resetting the tcp protocol, nothing worked.
I wrote a little application to demostrate the behaviour, it just unzips files from a server to the local machine, the application unzips a few files, then takes exactly a second (1000 ms) to unzip the next file, and keeps going like that, the only "solution" was to image one machine that works fine to that machine, then it works ok, so that throws any cable/server/network therories out... any ideas?
I still have the exact same problem in other machines
8/24/2005 12:54:00 PM by BlackTigerX
# re: The Case of the Periodic System Hangs
Thanks Mark, I'm having a similar problem of mouse that's freezing, etc. Except, on my system, I can see peaks in the DPC "process", but not really in any of the other processes. How can I find out more about those peak DPCs, is this normal (I hope not :-))? I've understood that DPCs are really low priority interrupts. Is that correct and how can I see which component is sending them then?
10/4/2005 2:56:00 PM by Anonymous
# re: The Case of the Periodic System Hangs
I'll have a future blog posting that describes how to track down the source of interrupt and DPC activity.
10/4/2005 2:57:00 PM by Mark Russinovich
# re: The Case of the Periodic System Hangs
What's your point? VPC is a good product, it just lacks this important feature
Exactly the point ;)
12/17/2005 2:15:00 PM by Tarry
# re: The Case of the Periodic System Hangs
What tool did you use to double click the "system" process. I tried on Win2K and XP...doesnt' work.
1/25/2006 5:28:00 PM by Jamie
# re: The Case of the Periodic System Hangs
Sounds like he's using Process Explorer, which can replace your Task Manager, and it's easy to forget you did. I'd do that, but it doesn't track network usage, which I find myself watching more than the CPU for some reason I can't explain even to myself. I just find it comforting.
3/5/2006 10:01:00 PM by Jim Hill
# re: The Case of the Periodic System Hangs
Hello,
I experienced the same problem recently and at first, I was so glad that I found your post, but unfortunately, after following the steps you provided I realized that this wasn't it. In the list of threads I don't have http.sys, but I do get spikes every 10 seconds in Kernel32.MulDiv+0x120 ?!? Actually 3 of them with approximately 7% of the CPU each. In the services tab it says there are 2 services registered in that process:
EventLog
PlugPlay
I can't disable any of them unfortunately and to tell you the truth, I wouldn't want to disable EventLog logging.
I tried to stop http.sys as you said but it didn't make any difference. I did use Vmware 5, but already uninstalled it before I tried this. Also in my NIC connection's properties, I can still see "Virtual Machine Network Services" protocol (which I think was installed by vmware).
My system is a Windows Server 2003 with (unfortunately) a lot of software installed. :)
What can I do next?
Thank you in advance for any pointers you might have.
4/6/2006 3:43:00 AM by Florin Sabau
# re: The Case of the Periodic System Hangs
Mark (and other vmware 5 users),
I realize this is an old post by now, but it seems to take a while to create and then to restore from a snapshot. I realize it is a shorter amount of time than what is needed to "roll-your-own snapshot" that would require making multiple copies of the VM at the various points. Could you comment on the performance of the snapshot feature in vmware 5, and how that affects your presentations? Do you simply use filler or elaborate on explanations while the snapshots are created and restored? Or do you use a high powered machine where the delay is not significant?
-Tim
9/11/2006 12:13:00 PM by Tim Lewis