Connectivity verifier memory issues caused by optional update KB971737

Recently Microsoft Customer Service and Support have seen cases with the firewall process (wspsrv.exe) in ISA Server 2006 “leaking” memory until it reaches the 32-bit process limits or face too much memory fragmentation to continue working properly.

On an ISA Server with high loads, memory fragmentation can always cause issues and that’s why we recommend setting the HeapDeCommitFreeBlockThreshold registrykey on high load systems. This is to improve how the heap manager decommits the memory and to minimize the heap fragmentation. This is described in detail in http://support.microsoft.com/kb/315407.

However even though setting this key, can help to improve the overall stability and performance of your server, it will only be a temporary workaround if you’re facing a leak inside the process. In a nutshell, a memory leak is memory allocated in a process that is not freed at a later point in time. If enough memory allocations like this happen, the process may run out of memory.

For more details about memory leaks I suggest that you have a look at some of Mark Russinovich’s blog posts like “Pushing the Limits of Windows: Virtual Memory” or recorded sessions around “Pushing the Limits of Windows” which can be found here: http://technet.microsoft.com/en-us/sysinternals/bb963887

A leak inside the ISA firewall process (wspsrv.exe) can be caused by many things. E.g. we’ve seen issues with 3^rd party Add-Ins, drivers, and unexpected behavior in our application. Identifying the cause of a leak can be a very complex troubleshooting task and in most cases you’ll need to collect Performance Monitor logs to see how the allocated memory is behaving. E.g. how much does it increase by per day, does it ever decrease etc. If the memory allocations keep increasing over time, then this may be a sign of a leak and it may be necessary to collect LeakTrack memory dumps using DebugDiag and analyze the dumps for leaks.

In order to avoid you having to do this, we’re doing our best to identify and fix the leaks we find and to include these fixes in regular software updates.

As mentioned at the beginning, I want to let you know about one issue we identified recently. The issue we identified is unfortunately introduced in an optional update of the winhttp.dll for Windows 2003 x86 (http://support.microsoft.com/kb/971737). This update implements EAP Authentication in Windows HTTP services. There’s no need to install this update on ISA Server 2006, because we don’t use this feature of the winhttp.dll.

The leak occurs when you configure HTTP connectivity verifiers in ISA that verify HTTP connections to web servers that use Windows Integrated (NTLM/Kerberos) authentication.

To prevent running into this issue, please either uninstall the optional update from KB971737 or review your connectivity verifiers, and check whether any of them verifies against an authenticating HTTP server. If so, disable or move to an anonymous server. Please note that HTTP connectivity verifiers may also be used in Web Farms.

If you’re not sure if the update is installed on your ISA server, then please open your Control Panel -> Add Remove Programs -> Show Updates and review the list of installed updates

Or open Accessories -> System Tools -> System Information -> Loaded Modules

Verify the version number of winhttp. The version number of the leaking dll is 5.2.3790.4584.

Summary

Please don’t install KB971737 on computers running ISA Server. If it’s installed, please uninstall the fix if you’re using Connectivity verifier on Webserver with Integrated Authentication.

Author

Philipp Sand
Microsoft CSS Forefront Security Edge Team

Technical Reviewer

Lars Bentzen

Microsoft CSS Forefront Security Edge Team

Comments

• Anonymous
  January 01, 2003
  Thanks for this post. I think this might be responsible for killing out ISA system every few weeks ever since the beginning of last year! Any ideas how we could remove this from a server or servers that have had the uninstall files deleted? I tried reinstalling the update in the hopes it would let me uninstall cleanly again - but that was a dead end. Also is there any chance of a 'real' fix for this such as an ISA hotfix to get around the problem? I mean just saying "don't install it" isn't too helpful. There are bound to be loads of admins out there that just install all updates on their ISA servers in an attempt to keep them patched to the hilt - all of them assuming ISA is tested against all patches as extensively as we 'thought' Microsoft were testing it...

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  Further to my earlier comment, I managed to uninstall the update by copying the hidden $NTUninstallKB971737 folder from another 2003 server into my ISA servers' windows folder. I could then use add/remove to lose the update. After a reboot the module had reverted to an earlier version. I also found I couldn't run system information as we had uninstalled pchealth (and thus disabled Help and Support) as part of our hardening process, so if you find you're in the same situation, run the following from cmd and it'll reinstall itself: cd /d %windir%pchealthhelpctrbinaries start /w helpsvc /svchost netsvcs /regserver /install net start helpsvc Thanks again for the article - now I just have to cross my fingers for the next 30 days and see if my ISA boxes stop falling over!

• Anonymous
  March 17, 2011
  Hi "NeedsCoffee" unfortunately chances are very low that we will see any hotfix for this. As stated the issue is not caused by ISA, but by an cmponent in Windows 2003. As Windows 2003 is in extended support already + the workaround is quite easy...though I agree not really "nice"... + it's not security related, the weight of arguments for a hotfix vs. the cost of a hotfix will end up with a reject of a hotfix request. :-S

• Anonymous
  March 21, 2013
  The comment has been removed