Thoughts on Application Pool Recycling and Application Availability
Question:
I am running IIS 6.0. We are currently trying to incorporate our application pools to recycle every 2 hours. The problem is everytime the the pool is recycled and the process is killed everyone connected to that webpage loses all connectivity and they have to log back into their applications that they were running. I have read about overlapping but i do not know where to go and check to see if it is still set to the default (overlapping) or if it is set to non-overlapping.
Does anyone have any suggestions on how to keep our recycle process going without our clients getting kicked out of their applications
thank you for any help
Answer:
When you recycle an application pool, HTTP.SYS holds onto the client connection in kernel mode while the user mode worker process recycles. After the process recycle, HTTP.SYS transparently routes the new requests to the new worker process. Thus, the client never "loses all connectivity" to the server - the TCP connection is never lost - and never notices the process recycle. Also, your issue is not related to overlapped recycling (enabled by default) because that does not involve connectivity nor state.
I believe your problem is with the applications running in your application pool that store state within the process, such as whether a user is logged in or not. Everytime the process recycles, that state is automatically lost... which is by-design since that is what a process recycle accomplishes. As a result, your users "lose all connectivity" and "have to log back into their applications" to re-establish that lost state. The only way to fix this is for your applications to store its state outside of the IIS6 worker process such that it is friendly to being recycled.
The following blog entry talks more about what is going on:
Now, lots of people have wondered why IIS does not provide facilities to somehow "save" the in-process session state prior to recycling and then "restore" it after the recycle, so that all applications run by IIS are automatically friendly to being recycled. The astute reader should realize that this request has several problems, most prominently:
One reason to recycle a process is to proactively get rid of potentially bad/stale state that leads to crashes, hangs, or memory leaks. If IIS simply saves and restores all state across the process recycle, both good and bad, then it defeats the purpose of the recycle.
Assuming that IIS has some magical way to determine the bad state from the good state and only toss away the bad state, what happens to the user application that depend on both the good and what WAS the bad state? How can IIS determine a "good" value for that bad state to prevent introducing logical errors into the user application?
One can argue that IF IIS can determine the good value, then the application should never get the bad value to begin with... and recycling is not necessary so this discussion is moot. ;-)
Thus, in order for you to keep your process recycling AND not have your clients getting kicked out of their applications, you must make sure they are architected to work with recycling by storing its state outside of the IIS worker process. Depending on the application, this may not be possible, but this is really a problem for you to solve...
Personally, I think it is strange that you are trying to establish a process that recycles the application pool every 2 hours. This sort of aggressive action usually indicates the presence of some misbehaving user code, and you usually get better bang-for-the-buck by fixing that bug (see this blog entry for the diagnostic approach). The ability to health-monitor and application pool recycle is a crutch to help you hobble along in the face of adversity, to buy you time to fix the underlying issue... and not as a primary means of running a broken application which will ultimately cause your demise.
//David
Comments
Anonymous
January 26, 2006
This is the reason why i will give you an example.
EX: We have a company that has 10 users connect to a web-based application. And when we look in task manager at the W3WP.exe it shows a memory usage of around 160 MB. We had all of the users log off all within 1/2 an hr of each other but the
memory usage does not drop. Is this memory really used up or is just waiting for another app that needs it.
The memory does not clear up within task manager until we run a recycle.
This is why we want to run recycling every few hours.Anonymous
January 26, 2006
The comment has been removedAnonymous
February 07, 2006
The comment has been removedAnonymous
February 07, 2006
The comment has been removedAnonymous
March 15, 2006
Really nice thoughts!
Do you have any ideas about intermittent performance problems? Computing resource are fine, and when slowness is found, recycling will "solve " the problem.
Someone suggested "perhasp app pool is running out of threads", where can I find this setting or measure it then?
CheersAnonymous
April 09, 2006
The comment has been removedAnonymous
April 09, 2006
SP - How does losing the NAS connection trigger the recycling of the worker process? That does not sound like a built-in IIS6 behavior.
In particular, it sounds like your triggering of the "recycling" is not known to IIS nor HTTP.SYS, so further requests still get routed to the old worker process.
When IIS triggers the recycling of a worker process, an atomic "switch" of the Application Pool Queue happens such that new requests get routed to new worker processes while the old worker process finishes up on the existing requests. This prevents what you claim from happening.
Right now, it sounds like you are running custom recycling-related code on the server and you are observing bugs in it, not IIS6.
//DavidAnonymous
April 09, 2006
hmm. I have a doubt with the process recycling then. I think my understanding about the process recycling is wrong. In our case when the connection with the NAS is lost, we get an Application_End event at global.asax.cs. Once the connection is restored an Application_start is received to the same worker process. Is this behaior correct? My understanding was when we receive Application_End event, worker process (w3wp.exe) will be killed by IIS and a new w3wp.exe will be spawned at the time of next request. Is my understanding correct? I am a novice with the IIS process recycling concepts and hence not able to understand the behavior that I am seeing. Any help will be much appreciated.
Thanks,
SPAnonymous
April 09, 2006
SoP - What you describe sounds like Application Domain Unloading, which is an ASP.Net specific concept that has no correlation to IIS worker process recycling.
IIS process recycling concepts are pretty simple - worker process starts up only when servicing requests, and you configure a whole bunch of metrics on when the worker process goes away - such as by time, by # requests, by CPU, on-demand, etc. These metrics are generic and know NOTHING about Application Framework specific concepts like Application_End/Application_Start, etc.
What is happening in your case is this - When ASP.Net detects that an assembly is missing or "refreshed", it has to restart the AppDomain to reload/init that assembly - so it unloads the Application Domain (thus you get an Application_End event), and on next new request to an unloaded AppDomain, you get an Application_Start event. This is pretty much by-design behavior, and your application should work well with it. If it doesn't, it is usually a problem in your application.
In other words, Application_End event (an ASP.Net concept) has no correlation to worker process recycling (an IIS concept). Can you point me to the documentation that gave you this understanding?
//DavidAnonymous
April 09, 2006
Thanks a lot for this information. Not any documentation in particular. I think I got confused going through the worker process recycling and application domain concepts. Thanks a again for the information.
SoPAnonymous
April 19, 2006
We've just had heaps of trouble with what we thought was either the worker process being recycled or application_end being called and it now looks like neither were the cause of our cache issues. for whatever reason, the cache just kept losing entries randomly - although the higher the load, the more frequent the losses. We've completely given up trying to figure out why and stopped using the cache altogether in favour of a statis class initialised in global asax that exposes a hashtable...it never fails.Anonymous
April 19, 2006
The comment has been removedAnonymous
May 09, 2007
Last year we deployed a new underwriting system. Since the Fall of 2006 we have been steadily releasingAnonymous
June 20, 2007
David, assuming you are receiving a lot of concurrency or that you have a memory leak what is the highest you can safely set your application pool w3wp.exe process to recycle and still prevent out of memory errors? I know this sounds like a dangerous question, but assuming I can't fix the poor code, I want to know how far I can push the limit. Let's assume I have 32GB of physical memory. Is the limit around 900MB physical or can I go higher?Anonymous
June 23, 2007
Topcat - You can just set any arbitrary private-bytes-consumed limit, determined through experimentation. 900MB is just as arbitrary as 2000MB. Otherwise, you will have to figure out:
- What memory pool is the leak in and what is the limit of that memory pool
- What is the rate of leakage (either in terms of time or #-requests) And you calculate: #1 ---- = units to hit limit #2 And you make sure to either recycle by time or by #-requests at about 80% of the limit. //David
Anonymous
June 25, 2007
Hi Great posts and helping me to understand a current problem we have. I'm somewhat of a newbie to this btw. One of our Sharepoint sites (just a webapp in IIS) stops functioning after an overnight app pool recycle ie. cannot find resource or locate page. We can resolve this by renaming web.config or moving/replacing a dll file, hitting site getting an error, renaming web.config and on next hit all seems well. I'm guessing the rename of web.config is causing something in IIS to recompile the webapp hence why it works. What I'm not sure about is how to properly diagnose what may be causing this problem? Any help gratefully received.Anonymous
July 01, 2007
Jim - what Application Pool are you recycling, and do the applications in that Application Pool support recycling. Changing web.config or changing a DLL file in /bin causes ASP.Net to recompile that web application on next request. My guess is that your problem happens because of the overnight Application Pool recycling of an application that does not support it, and that causing ASP.Net to recompile the web application resets whatever internal linkages that are broken by the unexpected Application Pool recycling. FYI: This really has very little to do with IIS and everything to do with diagnosing the behaviors of an arbitrary application, such as Sharepoint. If you truly want to get to the bottom of things, then you should hire a professional to help - like Microsoft PSS or appropriately skilled consultant. //DavidAnonymous
July 03, 2007
Hi Thanks David - I'll take a look at the application support for recycling app pools that you mention. The particular site which is having problems includes some workflow/infopath functionality. We have several other MOSS sites on the same server which are fine after the overnight app pool recycle and don't use this functionality. We haven't done any separation of apps across different app pools yet but this seems a sensible next test to try. I'm sure you are right about Sharepoint causing the problem. I'll take your advice and engage a pro to help. many thanks JimAnonymous
July 12, 2007
The comment has been removedAnonymous
September 24, 2007
We're experiencing a symptom in one of our ASP .NET applications that could use some direction before analysis. The symptom is a user will start a session with the application and begin working, shortly, the performance degrades to such a slow pace that work in the session becomes impossible-or the user leaves a working session to take a break, etc. -- then tries to create a new session, but the session will not connect. We've been restarting the application pool to address both conditons, however we know this isn't the solution. From an administrative perspective, all I'm seeing in the logs are idle timouts, etc. Any advice on what else to look for to help resolve this issue over here would be very helpful.Anonymous
June 05, 2008
Thanks for very useful article. My application pool stop sometimes. I wish to restart the application pool automatically if it stop. How can I do? Thanks.Anonymous
July 31, 2008
Just a comment about the freeze in the apps, if I understood, your application is connected to webservices. If you restart the appPool, you propably will require the web app to recompile on the first access to it, that is how .net framework works on websites, and a webservice is a kind of a website. I think it is normal... You application is just waiting the ready state of your webservices. Regards, André Luiz Sobreiro MCPDAnonymous
August 02, 2008
Okisan - if the application pool stops, it means that some catestrophic error happened in the application running in the application pool. You need to follow the instructions in my "AppPool Crash Diagnosis" link from all my blog entries to diagnose the cause. Until you determine and resolve the crash in your application, your application pool will continue to periodically stop. //DavidAnonymous
September 05, 2008
IIS6 and 7 I believe come with a default recycle value set, and you should always consider increasing or extending those values based on your available memory, expected load, number of processes and processors on your server, etc. Microsoft has an algorythm for that increase, I believe, to max that out. If your app is poorly written...not closing objects and releasing threads, using too much memory per session, heavy user load event design, etc, you can crash the AppPool more than expected. Your only work around is to get a faster or bigger server, and bump up the app pool defaults, or look at your processes and see if threads or objects are not getting collected and disposed in a timely fashion.Anonymous
September 14, 2008
Hi everybody I have a little question... In our environment we have a Web services based on XML files, when we edit an xml file for a specific web site, we apply an iisreset command to take the new configuration. What happens if we apply an application pool recycle on the web site? Does the web service take the new configuration? ThanksAnonymous
September 17, 2008
The comment has been removedAnonymous
September 19, 2008
Ok. Thank you David. I'll check that.Anonymous
February 03, 2009
I am new to IIS management, and could use some help please. I have a web server which is running several ASP applications which have memory leaks. I am trying to set up the application pool recycling to free the memory without having our site go down. We established a maximum of 450K of memory before we have a problem, so I set the application pool to recycle at that point. Generally, this works, I can look at the event logs and see where the application pool recycled without incident (it occurs roughly 3 times per day), however, sometimes I end up having the site go down (our customers receive out of memory errors) and have to sign in and recycle it myself. Here is my question (although any thoughts on the above are also welcome), if I reduce the amount of memory it can use, do you think that would solve the problem? Or would it just occur more often? Is this normal behavior for a recycle? Is it just that there are so many processes running at that time that it takes too long to startup and shutdown (the out of memory errors always seem to occur during the day). I was also wondering if I could leverage both the virtual and physical memory limits… is there a rule of thumb about that, or would it not help? Any assistance is greatly appreciated. terriAnonymous
February 26, 2009
The comment has been removedAnonymous
February 26, 2009
Dave, Sorry, did'nt know this comment section wasn't moderated, I retract my earilier staement.Anonymous
March 02, 2009
The comment has been removedAnonymous
March 23, 2009
I have a web application that deployed in windows 200 server IIS6.0, running in its own application pool. We are using some page level caching also, while accessing the site with more than 50 users, abrupt session expiry is happening and we could understand that the application pool recycling is happening and the cache value is also cleared. The memory usage at that time is 40%. Could you please give your thoughts on what may be the problem?Anonymous
June 02, 2009
Hi, I have the following doubt. Is there any way that the worker process can determine at what time it is going to be recycled or can it calculate how many minutes are there before the app pool recycle the worker process? Any method that gets called or any indication that the worker process can listen to so that it come to know before hand that recylce process is to begin shortly. Regards VishalAnonymous
June 19, 2009
Hi David, I am running an Insurance web application on .NET 2.0 this application has 1000 s of user every day and huge volume. In the application we show all the data for insurance policies on a grid (paged properly) Now the problem is .. once in a weeks time we receive errors navigating the page containing grid : Errors e.g. 1.) System. System.Web.HttpException: Multiple controls with the same ID 'PnlState' were found. FindControl requires that controls have unique IDs. at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at System.Web.UI.Control.FillNamedControlsTable(Control namingContainer, ControlCollection controls) at 2.) System. System.Web.HttpException: Multiple controls with the same ID 'PnlPolicyNumber' were found. FindControl requires that controls have unique IDs. at System.Web.UI.Control.FillNamedControlsTable(Cont Now when we recyle the application pool the site is back available and user can browse through grid. Please suggest some solution so that where we can fix in application or some setting in IIS so that we do not get this errors.Anonymous
June 24, 2009
Hi Kirtesh, could be a cache problem Take a look at http://support.microsoft.com/kb/834608 Cheers.Anonymous
June 24, 2009
Hi Cristian, I checked the url posted by you but it seems to be applicable for .NET framework 1.1 . I am using Framework 2.0 . Can I still use the hotfix ?Anonymous
August 19, 2009
The comment has been removedAnonymous
December 16, 2009
Hi David, We have a website, which has global.asax file, which contains Seesion_End as well as Application_End events. These are firing in normal scenarios. But, if IISreset happens, it doesnt... Is it the behavior or anything wrong in our side? If its the behaviour, is there any other alternative to know when IIS is reset? Ultimately, we want to log some data (in the backend) when iisreset happens. Thanx in advance. AthiAnonymous
June 03, 2010
Hi David, Please excuse this post resurrection but we are having some issues with one of our implementations and I thought you might be able to offer some insight. We are running 2 web servers on a load balanced setup. We are getting an intermittent issue whereby the site seems to go down (on one of the web servers) and traffic is not redirected to the other. Thats a separate issue. The main issue is why the site is going down and its somethign i cannot fix :( We do know that simply recycling the application pool brings the site back online. The site is configured to recycle its application pool overnight but this issue seems to arise through the day. Its odd as the solution is not exposed to all users YET. Its only a handful. Can you offer any advice or next steps that you think i should be looking at, i would greatly appreciate that. Thanks AndyAnonymous
June 03, 2010
Hi David, Please excuse this post resurrection but we are having some issues with one of our implementations and I thought you might be able to offer some insight. We are running 2 web servers on a load balanced setup. We are getting an intermittent issue whereby the site seems to go down (on one of the web servers) and traffic is not redirected to the other. Thats a separate issue. The main issue is why the site is going down and its somethign i cannot fix :( We do know that simply recycling the application pool brings the site back online. The site is configured to recycle its application pool overnight but this issue seems to arise through the day. Its odd as the solution is not exposed to all users YET. Its only a handful. Can you offer any advice or next steps that you think i should be looking at, i would greatly appreciate that. Thanks AndyAnonymous
August 10, 2010
We have a Exchange server 2003 SP2 that is running on a Windows server 2003 R2. Recently it reported the default application pool was stopped because unexpected error and the mobile users who use rcp over https cannot sync with Exchange server. Therefore I tried to set recycle the process but it make my users always disconnect from Outlook (also RPC over Https). Is any better solution like patch can solve my difficulty?Anonymous
August 10, 2010
The comment has been removedAnonymous
September 09, 2010
We have a web application and there are about 300 concurrent users and we have 4 GB RAM on the server , sometimes we feel the application slowing and it crashes and immediately it comes up and then after a few minutes it crashed again. IS it a memory constraint. Did the max numbe rof people access the poolAnonymous
March 22, 2012
First of all thanks for all the userful information in this blog. Not sure if someone is monitoring this blog, but im hoping someone might have an answer for my question. For one of the applications pools in IIS, we have the maximum used memory set to 800 MB. We have enabled webgarden with 12 processes. But, An individual worker process does not always consistently recycle when the 800 MB threshold is crossed. What could be the problem? SrinathAnonymous
March 12, 2015
I have deployed 4 asp.net web applications in windows server 2012 and IIS 8, i am facing a peculiar issue of webpage not opening till i manually recycle the application pool. please help me...