Tuning Free System Page Table Entries when using /3GB and /USERVA=wxyz

 

If you enable /3GB in the boot.ini of a Windows Server 2003 x86 server, you risk running out of address space for the kernel.

You can tweak this by adding the switch /USERVA=wxyz where wxyz is the number of megabytes that should be allocated to the user mode processes. This will give more address space back to the kernel.

But how should you choose the correct value for /USERVA?

Here's the easiest way. This doesn't involve pool monitoring applications, debugging tools or enabling Free System Page Table Entries (FSPTEs) tracking registry keys (trackPTEs).

1. Logon directly to the server. This will cache your credentials on the server should the next step cause the server to fail

2. Add the switch /3GB to boot.ini

3. Add an additional entry in the boot.ini which is the same as the default entry, but doesn't have /3GB in the boot.ini (this is useful if your system reboots in an unusable state after adding the /3GB switch). Your boot.ini should look like this:

    

[Boot Loader] Timeout=5 Default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS [Operating Systems] multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows Server 2003, Enterprise" /fastdetect /NoExecute=OptOut /3GB multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows Server 2003, Enterprise without 3GB" /fastdetect /NoExecute=OptOut

 

1. Reboot the server
2. Login, and open perfmon.msc. Add the counter Memory\Free System Page Table Entries and record the number. This should be a fairly static number.
   1. If the number is less that 10,000 and the server is behaving erratically, remove the line with /3GB to return the server to normal behaviour
3. If the FSPTEs is greater than 10,000 then the server is ready to go back into production
4. If the FSPTEs is less than 10,000 then we need to add the /USERVA=wxyz  switch to boot.ini
5. Calculate the number using the method below, reboot and login again, and check that FSPTEs is indeed at the level you expected.
6. If user load lowers the value to below the value you expected, recalculate your value for USERVA using the method below and reboot.

 

How to calculate the right value for /USERVA

 

Take your value for FSPTEs you got when you rebooted with only /3GB defined in the boot.ini. In our example we'll use 6,400 for the number of FSPTEs.

6,400 is a counter of the number of 4KB memory "blocks" are available to the kernel. So the kernel has 6,400 * 4KB = 25,600 KB = 25MB of free address space to use.

(We need to reboot the system, as we don't know how many PTEs are used by the system load all the required kernel resources when /3GB is specified as this changes the sizes of NPP and PP memory)

FSPTEs must be greater than 10,000 at all times. So let's just make that 15,000 to ensure that future changes to required kernel resources (e.g. new video or network drivers) will not fail.

15,000 * 4KB = 59MB

 

In this example we have 25MB of free kernel memory and need 59MB, so we must add 59-25 = 34MB

 

So we need to set USERVA to be 3072MB – 34MB = 3038

So we add the switch /USERVA=3038 to boot.ini

 

This will give every user-mode application 3038MB of address space, and give the kernel the 59MB of free address space it needs to be happy.

 

This means that USERVA and FSPTEs are directly related to each other by a factor of 4KB/4*1024 = 256

 

But will there be any other negative impact to any other critical kernel resource (like Pool of Non-Paged memory or Pool of Paged Memory)?

In my tests, there will be no change in these resources, and this whitepaper confirms that adding or tuning /USERVA will only have an impact on the count of FSPTEs:

 

https://www.microsoft.com/downloads/details.aspx?FamilyID=ed0e8084-abf7-4c00-ba6a-7d658cdb052a&DisplayLang=en

 

With the " /USERVA" boot.ini switch, you can customize how the memory is allocated when you use the /3GB switch. The number following /Userva= is the amount of virtual memory address space in megabytes (MB) that will be allocated to each user process. If you set /3gb /Userva=3030 in Boot.ini, 3,030 MB of memory is reserved to the process space, as compared to 3,072 MB when you use the /3GB switch alone. The 42 MB that is saved when you set /Userva=3030 is used to increase the kernel memory space and free system page table entries (PTEs). The PTE memory pool is increased by the difference between 3 GB (specified by the /3GB switch) and the value that is assigned to the /Userva switch. There is no reduction in any other kernel resource as a result of this switch.

 

Here's the results I found when changing the USERVA on a Windows Server 2003 SP2 x86 server with 3582MB RAM (4GB inserted, but the video card claimed 400MB at power-on):

/3GB State	/USERVA Value	Free System Page Table Entries	Free Kernel Memory
OFF	OFF	175,000	684MB
ON	OFF	19,900 (1)	78MB
ON	3030	30,700	120MB
ON	2900	64,000	250MB
ON	2800	89,800 (2)	351MB
ON	2650	128,000	500MB
ON	2500	166,500	650MB
ON	2466	175,000	684MB

⁽¹⁾ = This is the lowest value for FSPTEs on this system, which is higher than 15,000, so no action is needed for this server.

⁽²⁾ = Exchange mailbox servers should never have a /USERVA lower than 2800 as this will cause problems for store.exe

 Below: Vertical Axis = Free System Page Table Entries, Horizontal Axis = Value for USERVA