The Case of the Enormous Page File

Introduction

It seems like every time I deliver the Vital Signs workshop (a Windows architecture course focused on performance analysis patterns) lately, we always have a great discussion about proper sizing of the page file. There is a lot of bad information floating around on the internet about how to size the page file on servers with large amounts of RAM, so this seemed like a great topic to talk about in my blog.

First of all, don’t take my word on this subject – go to the source – David Solomon. David Solomon is one of the people closest to the Windows source code and he specializes in operating systems. In addition he and Mark Russinovich are the authors of the Windows Internals book 5th edition which just released last month which many of us consider to be *the* book on Windows architecture. This edition of the book is focused on Windows Server 2008/Vista and Windows 7.

David Solomon has a very good webcast on Troubleshooting Windows Memory. It’s a 90 minute webcast, but it took me 8 hours to go through it because I had to pause every few seconds and think about what he just said. ;-) This was a very enlightening experience for me because it really brought everything together for me. One of the topics he discusses is proper sizing of the page file. I will try to summarize his teaching in this blog entry and supplement it with a real case.

In addition, Mark Russinovich has a great post on Pushing the Limits of Windows: Virtual Memory which goes deep into the relationship of virtual and physical memory.

Memory States

Memory can be in one of three states: Free, Reserved, or Committed. Free memory is memory that is free for reuse by the operating system. Reserved memory is memory that an application intends on using, but is not using it yet – similar to reserving a table at a restaurant. Committed memory is memory that has data in it and is being used similar to sitting at the table you reserved earlier.

Reserved and Committed Memory

Let’s say you reserved a table for 10 at your favorite restaurant for tomorrow night. Do you think that the restaurant will cordon off the table until your get there? Of course not. They will allow their customers who are there right now to use that table until your party actually arrives.

The memory manager in Windows works the same way with reserved memory where it will not actually allocate any real memory until an application actually needs it. When memory is actually being used (Committed memory) the operating system must do something with that memory, therefore it will either put it in RAM or on the page file.

What is in the Page File?

The page file only contains data that cannot be found elsewhere on disk. For example, if I open up Notepad and do nothing else, the memory being used by Notepad will never be paged to the page file because the EXE and DLL’s needed to load Notepad can be retrieved from the disk at anytime. If you typed the sentence, “The quick brown fox jumps over the lazy dog.” into Notepad, then that text is in RAM right now, but could eventually be paged to the page file because it cannot be found anywhere on the disk. This is why you will never see EXE’s or DLL’s in the page file – only data that cannot be found elsewhere on disk could be put in the page file.

The Commit Limit

The Commit Limit of your computer is the total amount of RAM and the size of all of the page files on the system. In the case of my laptop, I have 4GBs of RAM and a 4GB page file giving my system a Commit Limit of 8GBs. This means that my laptop can handle up to 8GBs of committed memory.

The Commit Charge or Committed Bytes is the total amount of memory used/committed by the operating system and all processes running on the computer.

Shown here in my Windows 7 x64 Task Manager. Highlighted in red is my laptop’s Commit Charge (2074MBs) and it’s Commit Limit (8058MBs). Since the amount of RAM on my system is 4GBs, my system can handle the 2GBs of committed memory with no problem even without a page file.

If the amount of Commit Charge (also known as Committed Bytes) is more than the amount of physical RAM on the computer, then some of that memory has to be in the page file and frequent hard page faults to the page file can cause the system to be slow. The page file is really just an extension of physical RAM allowing the system to handle more committed memory if needed, but just *much* slower than RAM.

Running with No Page File

A common question is, “Can I run my server without a page file and will I gain a performance boost?”. Yes, you can run a server without a page file. Windows has supported the ability to not have a page file for a very long time. Since the operating system only pages out memory that is hasn’t been accessed in a long time, the performance impact should be minimal unless the Commit Charge is greater than the amount of RAM on the system. If the Commit Charge is less than the amount of RAM on the system, then the only performance gain that you *might* get is snappier responses from applications or services that have been idle for a long time and even then it might trivial.

Without going into too much detail about working set trimming, the operating system will page out “old” memory used by idle processes - “old” meaning the pages of memory that haven’t been used/accessed the longest. For example, if you opened up a large Word document and you are now on page 100, then page 1 has likely been removed from RAM (paged out). If you have not modified page 1, then page 1 would not go to the page file because the operating system can always just read the document file off of the disk if it needs to. This will cause a hard page fault, but not against the page file. See my blog post on, “The Case of the Phantom Hard Page Faults”. This behavior will happen regardless of the amount of RAM on the server because the operating system is optimizing the server’s RAM for memory pages that are frequently used.

If you run with no page file, then your Commit Limit (RAM + page file size) is now equal to the amount of RAM on your server. In this case, all it means is that your server cannot exceed a Commit Charge greater than the amount of RAM your server has installed. If the Commit Limit is reached on the system, then the system will be out of memory and will likely hang for long periods of time until the committed memory is released. This can sometimes be as easy as killing the process that has the most committed memory measured by the “\Process(*)\Private Bytes” counter in Performance Monitor.

The Problem/Symptoms

Okay, so now that we have the technical stuff out of the way, let’s put this into a real world case.

Last month, one of my customers had a 64-bit (x64) Windows Server 2003 Server with 32GBs of RAM. They went with the standard page file size of 1.5 times the size of RAM for their page file which gave them a 48GB page file which gave them a Commit Limit (RAM + page file size) of 80GBs! In addition, the page file was taking up an enormous amount of disk space.

Troubleshooting

I fired up Task Manager to see how much Committed memory (Commit Charge) is being used by the system and it was only using about 2GBs. In Task Manager on Windows Server 2003, the Commit Charge is shown as PF Usage which is not the proper name for it. PF Usage should read as “Committed Memory”, but at least it reads as just “Memory” now in Windows 7.

Anyway, the customer’s server was only using 2GBs (Commit Charge) out of 80GBs (Commit Limit) at peak. On Windows Server 2003’s Task Manager there is a Committed Memory Peak value which shows you how much committed memory the system used at peak from the time the server was booted up until now. If the peak ever gets close to the Commit Limit, then you need to increase the size of the page file or identify and kill the leaking (memory hog) process. In any case, my customer’s server was barely using the RAM on the server let alone needing to use the page file.

The Solution

Since the Commit Charge Peak was only at 2GBs and since the amount of RAM on the server was at 32GBs, the customer could run the server without a page file and not have any problems. In fact, they could remove some of the RAM modules from the server and save electricity costs. Ultimately, the customer stuck with the 1.5 times RAM setting just in case. This is a waste of disk space, but is a very safe choice.

With all of that said, we (Microsoft) still *generally* recommends a page file that is at least greater than the amount of RAM installed + 1MB (for the header). This is because by default Windows Servers are set to “Complete Memory Dump” which means to dump all of the physical memory and the page file to a memory.dmp file when a kernel dump (blue screen) occurs. Since a page file will be enormous based on this settings requirements, you can certainly choose a smaller kernel dump setting such as “Kernel memory dump” or “Small memory dump (64K)”. The smaller dump sizes still contain valuable data at a more reasonable disk space cost.

According to the “Windows Internals 4th edition” book, “The disadvantage of minidumps is that to analyze them, you must have access to the exact images used on the system that generated the dump at the time you analyze the dump” . This would certainly be difficult for a Microsoft Support Professional to assist you unless they are physical onsite with you. For a full explanation of these settings and their impact on the system, read the “Crash Dump Analysis” chapter in the Windows Internals book.

In a nutshell, use the smaller “Kernel memory dump” or “Small memory dump” settings if your server does not have a history of crashes (blue screens).

Sizing of the Page File

The reason page files have been difficult to size is because to truly size them properly, the server must be put into production and the Commit Charge peak must be closely monitored. Once you have a baseline of what the system’s commit charge peak is, then you simply need to make the Commit Limit larger than it with a bit of room to grow. Ironically, IT professionals need to set the page file to something *before* going into production, so this is a bit of the chicken and the egg problem. Since we have no idea of how much committed memory your server will use, it made sense to use a 1.5 times RAM as a *very general* guideline.

The Checklist

To *properly* size your server’s page file, follow this checklist:

1. Kernel Memory Dump Setting: First check your kernel dump setting to see if your server is set to “Complete Memory Dump” (default setting on Windows Servers). If so, your page file must be greater than RAM plus 1MB. This kind of memory dump is extremely large and sometimes wasteful, so consider a smaller memory dump size unless your server has a history of kernel crashes (blue screens). Changing this setting will allow you to set the page file size to a size smaller than the amount of RAM installed which can free up some disk space.
2. Monitor the Commit Charge Peak: Monitor your server’s Commit Charge peak to determine how much committed memory the system is using at peak. This can be monitored from the Performance tab in Task Manager or by monitoring the “\Memory\Committed Bytes” counter in Performance Monitor.
3. Adjust the Commit Limit: After monitoring the Commit Charge peak, set the system’s Commit Limit to be larger than the highest monitored Commit Charge peak plus a little extra for room to grow just in case. I recommend it to be at least 10% greater than the maximum observed Commit Charge peak. Remember, the Commit Limit is the total of RAM + the total size of all of the page files on the system, therefore you can adjust it by adding/removing RAM or changing the sizes of the page file(s). The Commit Limit on the server can be monitored either from the Performance tab in Task Manager or by monitoring the “\Memory\Commit Limit” counter in Performance monitor.

Tools Used in this Case

Task Manager: This tool is native to the Windows operating system and can be opened with the key combination of Ctrl+Shift+Esc.

Performance Monitor: This tool is native to the Windows operating system and can be opened by clicking, Start, Run, type “perfmon”, then Enter or click OK.

Process Explorer: While I didn’t use Process Explorer in this case, it is a great tool for monitoring server memory. Process Explorer is a free download from Microsoft Technet Windows SysInternals at https://technet.microsoft.com/en-us/sysinternals/default.aspx or you can run it direct from https://live.sysinterals.com/procexp.exe

Conclusion

In conclusion, the page file is difficult to size because you must monitor the server’s Commit Charge peak over a long period of time while in production. Always make sure that your Commit Limit (RAM + page file size) is at least 10% greater than the Commit Charge peak. And please check out David Solomon’s Troubleshooting Windows Memory webcast. It’s great and free if you have a Windows Live account! Also, check out Mark Russinovich’s blog on Pushing the Limits of Windows: Virtual Memory.