Intermittent Performance Problem while Accessing Internet through ISA Server 2006 – Round 2
1. Introduction
Well, I was not planning to have a round two on this subject, but it is just amazing how ISA is blamed by sys admin even before the deep troubleshoot being done. I hear every other week someone saying: my ISA is slow to answer, takes forever to open a page when we are in a rush hour, etc.
We need to demystify that, any affirmation that ISA is causing slow down in the Internet access should be carefully analyzed before this statement being said. This post is about one more complain about Internet being slow and of course ISA was the one that the admin was blaming.
2. Start from the Basics – Yeah, I know
Review the start from the basics session explained in the last post.
3. As usual, everything looks fine when we first look
When everything seems to be okay, the challenge is to concentrate in the details. This particular case all the items were covered fine, including the logging (which was done locally in the TXT file). The footprint of something that could cause the issue was discovered while the issue was happening.
4. Reviewing the Data
Here the first set of data that was capture during the time that the issue was happening:
· Perfmon and Netmon from ISA and DC/DNS
· ISA Data Packager
After look to the netmon trace we could clearly see that during the time that the issue was happening the DNS Server was not answering the queries sent by ISA. You might think: but you should review DNS in the basic review, right? Correct, but when we reviewed during that time it was answering in timely manner. This leads to a conclusion that the DNS Server works and then stops to work for some reason.
Another interesting point was that the DNS.exe process was having a leaking behavior, very similar to the behavior explained in KB830381. The problem was: customer’s DNS.exe process was much newer than that one since he was using Windows Server 2003 SP2.
At least at that point we knew that the issue was not caused by ISA, it was the DNS that was not answering for some reason.
5. Looking closer to the DNS Server
All right, since ISA was kind of out of the game we start to concentrate more in the DNS Server itself and we found out that the server had the following event:
Event ID : 7502
Category : None
Source : DNS
Type : Error
Generated : 8/13/2008 8:52:34 PM
Machine : SRVDNS
Message : The DNS server was unable to service a client request due a shortage of available memory. Close any applications not in use or reboot the computer
to free memory.
Clearly we have an issue here; looking to the perfmon data we found the following suspicious counters (data for the DNS.exe process):
Minimum: 25MB
Maximum: 2,564MB
Average: 996MB
Minimum: 95MB
Maximum: 3,049MB
Average: 1,586MB
Processor Time
Minimum: 0%
Maximum: 180%
Average: 93%
The user mode dump from the DNS.exe process also showed us the remaining data that helped us to categorize this as a memory leak. Here the relevant data extracted from the DebugDiag Report:
Top 1 functions by allocation count
dns!allocMemory+8f
58 638 allocation(s)
Function details
Function dns!allocMemory+8f
Allocation type Heap allocation(s)
Heap handle 0x007c0000
Allocation Count 58638 allocation(s)
Allocation Size 106,91 MBytes
Leak Probability 100%
6. Resolution
The problem was really caused by the DNS.exe process that was leaking and the issue was fixed applying the post SP2 update for DNS described in KB946565. If your DNS Server is running Windows Server 2003 SP2, make sure to update your DNS.exe process and remember: don’t blame ISA before you are 100% sure that it is ISA that is causing the issue J.