High CPU Utilization by Wspsrv.exe Process

项目
05/19/2009

1. Introduction

The Firewall service is ISA Server's main user mode service; it runs with the Network Account credentials and under the context of the process wspsrv.exe. The CPU utilization of this process might vary according to the server load and other external elements, for example: antivirus, third party plug ins, etc.

Note: For more information on ISA Server architecture read http://download.microsoft.com/download/e/7/6/e76fdda3-5c2c-4fbb-9c6f-3bcd0ed4b8ef/Firewall_Corewp.doc

The goal of this post is to help you to understand that in some scenarios where the wspsrv.exe process is consuming too much CPU the issue might be an external component that is not necessarily related to ISA Server itself. To exemplify that this post describes a real scenario where the wspsrv.exe process was using almost 100% CPU for approximately 3 to 4 minutes and then backs it up to normal utilization. During this time, since ISA Server was too busy it couldn’t answer in a timely manner to the requests coming from the internal network. As result of that clients were receiving "time- out" errors when trying to browsing Internet. This 3 to 4 minutes interval was enough to cause help desk calls and chaos to the end users.

2. Data Gathering

For this type of scenario you can user Performance Monitor to capture the main set of objects. Use the guidelines created by Jim Harrison at: http://blogs.isaserver.org/shinder/2008/10/21/tips-for-isa-firewall-performance-assessment. Also, install DebugDiag on ISA and get a full user mode dump for the process wspsrv.exe when the issue is happening.

Note: For more information on Debug Tools you can use the article http://blogs.msdn.com/tess/archive/2008/05/21/debugdiag-1-1-or-windbg-which-one-should-i-use-and-how-do-i-gather-memory-dumps.aspx

3. Analyzing the Data

One of the most valuable elements for this type of scenario is the dump file of the wspsrv.exe process and even without having the Microsoft Internal symbols you will still be able to review some key data using the public symbols. Download the debugging tools and configure the symbols according to the Debugging Tools and Symbols: Getting Started Page in Windows Hardware Developer Central.

After you get the dump you can use windbg to analyze this dump file. Here are some conclusions from this dump file:

1. By using the command !runaway 7 we were able to find out the top threads that were using resources:

0:000> !runaway 7

User Mode Time

Thread Time

253:98c 0 days 0:00:59.296 << Thread using more CPU

263:1104 0 days 0:00:56.828

70:1ad8 0 days 0:00:51.156

…

Kernel Mode Time

Thread Time

253:98c 0 days 0:00:35.093

263:1104 0 days 0:00:33.265

13:13dc 0 days 0:00:32.218

…

Elapsed Time

Thread Time

0:243c 0 days 20:41:59.956

1:23e4 0 days 20:41:59.643

2:1ec0 0 days 20:41:59.456

...removed the complete list since it was very long.

Note: The parameter 7 comes from 1 (user time) + 2 (kernel time) + 4 (time elapsed since thread start)

2. We switched to the thread 253 by using the command below:

0:000> ~253s

eax=00002000 ebx=46d66810 ecx=00000193 edx=00003800 esi=7fffffff edi=ffffffff

eip=7c82860c esp=4b90cea0 ebp=4b90cedc iopl=0 nv up ei ng nz ac pe cy

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000297

ntdll!KiFastSystemCallRet:

3. From there we got the call stack for this thread (253), which was the one that was consuming more user mode space within wspsrv.exe process, causing high CPU consumption:

0:253> kbnL

# ChildEBP RetAddr Args to Child

00 4b90ce9c 7c827d29 71b21af5 0001f0ac 00000001 ntdll!KiFastSystemCallRet

01 4b90cea0 71b21af5 0001f0ac 00000001 4b90cec8 ntdll!NtWaitForSingleObject+0xc

02 4b90cedc 71b2c507 0001f0ac 000268f4 00000000 mswsock!SockWaitForSingleObject+0x19d

03 4b90cf54 71c02fee 000268f4 4b90cf8c 00000001 mswsock!WSPRecv+0x203

04 4b90cf9c 37cb47da 000268f4 4b90d018 00000001 ws2_32!recv+0x83

WARNING: Stack unwind information not available. Following frames may be wrong.

05 4b90cfc0 37cb4880 4b90d018 00000001 00000001 IcapClientApi!TmDbgger::write+0xaba

06 00000000 00000000 00000000 00000000 00000000 IcapClientApi!TmDbgger::write+0xb60

3. We start to analyze patterns and the first 6 threads had a similar stack where this third party component was showing up there. We got more information about this component using the command lmvm:

0:253> lmvm IcapClientApi

start end module name

37cb0000 37cca000 IcapClientApi C (export symbols) IcapClientApi.dll

Loaded symbol image file: IcapClientApi.dll

Image path: D:\Program Files\Microsoft ISA Server\IcapClientApi.dll

Image name: IcapClientApi.dll

Timestamp: Fri May 26 17:58:55 2006 (4476D15F)

CheckSum: 00000000

ImageSize: 0001A000

File version: 5.0.0.1164

Product version: 5.0.0.1164

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 1.0 App

File date: 00000000.00000000

Translations: 0409.04b0

CompanyName: XXXXXXXXXX

ProductName: XXXXXXXXXX

InternalName: XXXXXXXXXX

OriginalFilename: XXXXXXXXXX

ProductVersion: 5.0.0.1164

FileVersion: 5.0.0.1164

PrivateBuild: Build 1164

SpecialBuild: 1164

FileDescription: XXXXXXXXXX

LegalCopyright: XXXXXXXXXX.

LegalTrademarks: XXXXXXXXXX

Comments: XXXXXXXXXX

Note: As you can see this is a very old version of this component.

4. Conclusion

In this case wspsrv.exe was externalizing the problem since it was the process that was consuming more CPU, however within the process we have many threads running and not always those threads are exclusively using calls from the ISA Server components itself. This is a classic example of external components that can make wspsrv.exe use more CPU than what it should. In this particular scenario the problem was fixed by the vendor of the third party application by updating the software to the latest version at that time.

5. References

Here are some good articles for troubleshooting high CPU utilization scenario (regardless of having ISA Server):

Troubleshooting High CPU Utilization

Troubleshooting High CPU Hangs with a Debugger

Isolating the Threads Consuming High CPU

How to track down High CPU in User Mode Applications - A live debug!

Authors

Yuri Diogenes

Sr Security Support Escalation Engineer

Microsoft CSS Forefront Edge Team

Congyong Su

Sr Support Engineer