IIS and PAE
I recently got a question by one of my customers about PAE and IIS that I thought I’d share the answer to.
Their environment looked something like this:
- 32bit OS (Windows 2003)
- IIS 6 with multiple application pools, where each app pool hosts a number of applications
- ~20 GB RAM
They were having problems with out of memory exceptions and also with paging under high load. They knew a lot about their application and memory usage in general, so they knew what they were using the memory for and why they got the OOMs, in this case, memory usage was relatively high per application (intentionally), but it remained stable so no leak. With this in mind, we discussed solutions to the OOMs like splitting up the applications further etc. to avoid this. Now their question was:
Is IIS 6 PAE aware, i.e. will we be using the full 20 GB of RAM or only 4 GB?
I didn’t really know much about this so I consulted with one of our IIS escalation engineers Andreas to get the scoop.
The answer (which Joe Stagner points out in this forum thread) is that IIS 6 is not PAE aware, and since they are running on W2k3 x86 which will not manage memory above 4GB they will effectively only use 4 GB, leaving 16 GB unused. (Unless of course they run other PAE aware apps on it, SQL server for example I believe is PAE aware).
Update: As per Dave's comments below the statement above seems to be incorrect, if you have multiple w3wp.exe processes they can have a total workingset bigger than 4GB (combined for all w3wp.exe processes) even on 32-bit if /PAE is on... So in retrospect, I suspect that the above statement is talking about RAM used for HTTP.SYS etc. This should teach me to not blog about things that I'm only hearing 2nd hand:) even still, it has been a good learning experience for me... and thanks to blogging about it, I now know more... Thanks Dave for commenting on it.
On the other hand, a solution that would solve both the RAM question and the OOM issue is to move to a 64bit OS (preferably 2008 R2 because of improvements in the kernel, memory management and IO management).
Since it is a 64bit OS it can take advantage of all of the RAM, and even if you run 32bit IIS, you can then address 4 GB of Virtual Memory rather than 2 GB which means that there is more room for memory usage before an OOM (in this case since memory usage was stable but relatively high, that is exactly what we want). In fact, as Bernard Cheah also mentions in the same thread, there is no reason to move to 64bit IIS unless you need to, i.e. unless you need to use even more memory than the 4GB of virtual bytes that the 32bit process can address.
Have a good one,
Tess
Comments
Anonymous
April 26, 2010
It's funny that you just wrote about this. I was just discussing the OutOfMemoryExceptions we've been seeing lately on one of our apps with one of my coworkers yesterday. Specifically, we were discussing the memoryLimit setting on the processModel element in the machine.config and wondering about what the limit should be set at for a machine with 4GB of RAM. On this page: http://msdn.microsoft.com/en-us/library/ms998583.aspx, under the "Configuring the Memory Limit" section, it says that the default setting is 60%. But on a 32bit OS (which we are using), that means 2.4 GB is the limit. Since IIS can only address 2.0 GB, I think this may be the culprit. What we see on our server is OOM's being occasionally thrown, but the process does not recycle, which is unexpected. Assuming we are OK with the application recycling, should we set the memoryLimit to be 50% so that the process will recycle instead of just throwing OOM's?Anonymous
April 26, 2010
Tess, I believe you're interpreting Joe's post incorrectly when you state: "IIS 6 is not PAE aware ... effectively only use 4 GB, leaving 16 GB unused" When you run multiple app pools in IIS, each app pool runs in it's own worker process, recieving 2GB user space each. With multiple app pools, you can effectively use much more than 4gb of total RAM for IIS - we do this all the time, and I can definitely show that my total IIS memory (IIS + AppPools) is much larger than 4GB. Each process is limited to 2GB on x86, (4gb on x64) but when you're hosting multiple applications and the OS can address more than 4GB (i.e. x86 standard vs enterprise) separate app pools allow you to more fully utilize your server's available memory.Anonymous
April 26, 2010
Thanks Cristopher, That is very intersting information. That is what I thought before as well but then my colleague seconded the opinion that Joe stated in the forum, that IIS can only map in to the first 4 GB of RAM. Please note that this is very different from saying that the total memory usage for all your app pools can't be over 4 GB since memory can also be paged to disk. So, in your case, what is it that is greater than 4 GB? Is it the total virtual memory usage for your app pools, the total private bytes usage or is it the total workingset that is greater than 4 GB? I'm not saying that you are not correct, just want to verify that we are talking about the same thing...Anonymous
April 27, 2010
This is actually really good advice for any application, not just IIS. We run into similar situations with our large multimedia app that uses a large amount of memory. Started that we were using the /3gb switch to get a bit more address space, but eventually we had many of our customers move to 64bit to at least get the 4gb of address space (for a 32bit process). Hopefully, this will be good enough till we fully convert our app to a 64bit process.Anonymous
April 27, 2010
Tess - think of it this way. If you have a process in x86, it can address 2GB of user space ram. If it spawns child processes, they each can address 2gb, regardless of what the parent can address. When IIS is running in worker process isolation mode, IIS's relationship to it's worker processes (w3wp.exe) work this way - each WP is an isolate process, with no memory overlap. It's a little funky since IIS is actually several processes (InetInfo, w3svc (under SvcHost) and W3wp in IIS6, InetInfo, w3svc & WAS (under SvcHost) and W3wp in IIS7) . Of course, I didn't write the code, so could be wrong, but this is empirical based off of lots of web servers running 10-20 web sites each. We obviously are mostly x64 now, so the 2GB user space becomes 4GB, but we're constantly running into AppPools that are leaking memory. You certainly can test this by running 4 or 5 of your BuggyBits web sites and the links.aspx page.Anonymous
April 27, 2010
Cristopher, What you are saying is true, but still I think we are not talking about the same thing. Each process can address 2GB of virtual memory, this is different from user space RAM. The pages used in virtual memory can reside either in RAM or be paged out to disk. I agree that you could have 50 processes on a system, each allowed to use 2GB of RAM, and perhaps together using 50 GB of private bytes if the page file + RAM is big enough. The question posed here is "how much of all the virtual memory they use can be paged in to RAM at the same time?" In other words, their problem here was not that they couldnt use a total of x GB in their processes, the problem was that they were paging way too much, causing them perf issues. Unfortunately I dont have a machine with more than 4 GB of RAM at the moment so I can't test, but I would be curious to see if the total workingset (pages in RAM) of all the W3WP.exe processes exceeds 4GB.Anonymous
April 28, 2010
So also Web Garden is not useful?Anonymous
May 03, 2010
Set objWMIService = GetObject("winmgmts:rootCIMV2") Set colItems = objWMIService.ExecQuery("SELECT name,pagefileusage,workingsetsize FROM Win32_Process where name='w3wp.exe'") ws = 0 pf = 0 wscript.echo "Count=" & colItems.count For Each objItem In colItems ws = ws + objitem.workingsetsize pf = pf + objitem.pagefileusage Next wscript.echo "Ws=" & ws wscript.echo "Pf=" & pf Produces the following on my W2K3 32bit server with /PAE and 8 gig of ram. Note pagefileusage is in Kilobytes. Count=51 Ws=6462443520 Pf=6803752 That says that we're using 6.4 gig of memory for IIS worker processes split across 51 w3wp.exe's.Anonymous
May 25, 2010
Christopher/Tess: Why would there be a 4GB per-process limit on 64-bit Windows as mentioned above? Are you just talking about running apps in 32-bit mode on 64-bit Windows?Anonymous
July 27, 2010
Tess, So what are we conculding here?Anonymous
August 13, 2010
Environment.WorkingSet returns the working set incorrectly for my asp.net application which is the only application in its application pool. On a Windows 2003 Server SP2 with 3GBs of Ram which is a VMWare Virtual Machine, it reports working set as 2.047.468.061 bytes(1952MBs) and Process.WorkingSet value is 75.563.008 bytes(72MBs). • Memory Status values returned by GlobalMemoryStatusEx: AvailExtendedVirtual : 0 AvailPageFile: 4.674.134.016 AvailPhys: 2.140.078.080 AvailVirtual: 1.347.272.704 TotalPageFile: 6.319.915.008 TotalPhys: 3.245.568.000 TotalVirtual: 2.147.352.576 • GetProcessMemoryInfo() Working Set : 55.140.352 Peak Working Set: 75.571.200 PageFile : 94.560.256 QuotaPagedPoolUsage : 376.012 QuotaNonPagedPoolUsage : 33.261 • GetProcessWorkingSetSize() - min : 204.800 - max : 1.413.120 • GetPerformanceInfo() CommitLimit : 1.542.948 pages 6.319.915.008 bytes CommitPeak : 484.677 pages 1.985.236.992 bytes CommitTotal : 417.514 pages 1.710.137.344 bytes HandleCount : 57.012 KernelNonpaged : 8.671 pages 35.516.416 bytes KernelPaged : 27.302 pages 111.828.992 bytes KernelTotal : 35.973 pages 147.345.408 bytes PageSize : 4.096 bytes PhysicalAvailable : 508.083 pages 2.081.107.968 bytes PhysicalTotal : 792.375 pages 3.245.568.000 bytes ProcessCount : 43 SystemCache : 263.734 pages 1.080.254.464 bytes ThreadCount : 1.038 After loding the new patch, support.microsoft.com/.../en-us, .NET Version changes to 2.0.50727.3615 and Environment.WorkingSet now returns the value: 2.047.468.141.(which is 80 bytes bigger than previous one) On a Vista machine with 3GBs of Ram, Environment.WorkingSet and Process.WorkingSet values are similar and around 37 MBs. So, why Environment.WorkingSet returns a fixed value? Restarting the application pool does not change anything, it always return the same magic value, 2.047.468.061. I have also setup an .NET 1.1.4322.2443 application, and it weirdly WorkingSet is returned a number from a random set of unrelated numbers(193.654.824, 214.101.416, 57.207.080, 287.635.496) each time page refreshed while GetProcessMemoryInfo() returns the expected number. I have also found that when the application run by impersonating "NT AUTHORITYNetworkService" account this problem does not occur, Environment.WorkingSet returns the expected number both .net v1.1 and v2.0. I have checked CodeAccessPermissions like EnvironmentPermission for windows user and NetworkService but could not find anything that restricts reading the WorkingSet value. So, what could cause this? Is it a bug, some incorrect configuration or corrupt file etc.?Anonymous
November 06, 2010
This article lacks some serious technical background/concepts... No application is PAE aware!!!!. Since this is about managing more than 4 GB pysical Memory. So its about how much physical memory the kernel can allocate. Its a feature of 32 Bit Windows Server OS's and a kernel construct. Applications access memory through virtual addresses and are bound to their 2 GB virtual address space (3 if you use a switch in the boot.ini and the exe is aware of it) . There are application out there like the mentioned SQL Server that can allocate more than 2 GB virtual memory with the so called AWE feature. This allows an application to stick more than 2 GB's in physical Ram and access it through "windows" in its still 2 GB big virtual address space.Through this it can effectivly handle more than 2 GB's of data. But AWE has no connection with PAE and vice versa. They just happend to be used together which is understandable on on 32 bit OS.Anonymous
February 06, 2013
I too running multiple worker processes in Win 2003 Server SP2 x32 bit with 7 GB RAM. When I look at process(_total)private bytes it is more than 5 GB during peak time and process(_total)working set is more than 4 GB at the same time. All the IIS applications are facing some memory leak issues and throwing OOM. I am trying to fix the memory leaks now. Meanwhile, I would like to know the increase in memory beyond the current limit of 7 GB would be useful based on the actual need of each w3wp.exe process together?