Another Case of the Missing PTEs
I did an IIS Health Check (IISHC) this week and found another 32-bit Windows 2003 web server that was out of System PTEs. This blog entry talks about how we resolved this issue.
I detected this issue by looking at the boot.ini file which is one of the validations we do in the IISHC. The boot.ini file had the /3GB switch, the /PAE switch, and a few others. My concern is that the /USERVA switch was not in the boot.ini file. I immediately brought up Perfmon and looked at the “\Memory\Free System Page Table Entries” performance counter. It had a value of about 2300 which puts the server in a critical condition. When a server has less than 5,000 free PTEs, then it is considered critical because the server will begin failing to allocate memory to processes.
Per the Vital Signs Workshop (created by Shane Creamer) offered by my organization:
“\Memory\Free System Page Table Entries”
Warning: If less than 10,000
Critical: If less than 5,000
Without going into too much detail, the /3GB switch steals 1GB of virtual memory from the kernel and gives it to processes. This allows processes that are compiled with LARGE_ADDRESS_AWARE to use 3GBs of memory versus the standard 2GBs. While this sounds like a good thing, the kernel needs memory too and you can cut into it too much very easily and the first casualty is System Page Table Entries (PTEs) which are needed to map virtual memory to physical memory.
In a nutshell, if you use the /3GB switch in the boot.ini, then you should use /USERVA=3030. The /3GB switch puts the server at 3072MBs (3GBs), so the /USERVA switch moves that line between the process memory and the kernel memory at a new location. For example, a value of 3030 will give back 42MBs to the kernel going directly toward System PTE’s.
For detailed troubleshooting on System PTEs, see my (and other contributors) blog entry at: https://blogs.technet.com/clint_huffman/archive/2008/04/07/free-system-page-table-entries-ptes.aspx
In any case, this web server never needed the /3GB switch in the first place. I determined this by looking at the “\Process(*)\Virtual Bytes” performance counters. All of the processes were using less then 1.1 GBs of virtual memory out of 3GBs. Since the default (non-/3GB system) is 2GBs of virtual address space per process, there is no need to have /3GB on this server.
For good measure, I fired up WinDBg (a kernel debugger) and did a “!vm” to see what the virtual memory of the kernel looked look. It had an alert saying there had been about 1 million failed PTE allocations, so this confirmed the critical condition.
After discussing this with the customer, we agreed to remove the /3GB switch. After the reboot, the virtual memory of the server looked *much* better. We showed about 153,000 free PTEs and the Pool Paged and Pool Non-Paged kernel memory maximum sizes doubled (good thing), but of course, this reverted the maximum addressable memory for processes back to 2GBs.
In conclusion, if you use the /3GB switch, then also use the /USERVA switch. Ultimately, all of this would not be problem on 64-bit Windows on the same hardware because 64-bit Windows can address up to 16TBs per process (8TB for each process and 8TB for the kernel) by default.
A quick note about power savings (Green IT) through performance analysis
This web server had 16GBs of RAM installed. The “\Memory\Available MBytes” performance counter shows that the server always has over 14GBs of RAM available. This is great for performance, but wasteful on power consumption.
Bruce Worthington mentioned to me a few months ago that one of the biggest power consumers in PC hardware is the RAM modules. Therefore, if your server constantly has free RAM available even during peak load, then you can save some money on power by removing some of the unused RAM from the server. You can measure this using power meter in the AC outlet measuring the kilowatts used before and after the RAM has been removed. Bring these numbers back to your manager on how your company can save money.
Alternatively, you could run virtual computers on the same hardware to make use of that extra RAM completely saving on the AC power needed to power another physical computer.
I hope you find this blog entry useful. Thank you!
Comments
Anonymous
January 01, 2003
PingBack from http://www.keyongtech.com/5229622-boot-ini-w-paeAnonymous
July 07, 2009
I wonder how long it will be until we have servers that can dynamically shut off Ram modules when they're not needed and turn them back on again when they are.