ASP.NET Case Study: Lost session variables and appdomain recycles

Artikkeli
08/02/2006

Last night I got a question from one of the readers of the blog that went like this:

“We are facing a problem that i cannot understand, every now and than i see that my app domain is recycled (i have a log in the application_end), I check the IIS logs and i don't see a restart of IIS and i know that no one is changing configuration (web.config).

I wanted to know if you know of any way that i can pinpoint the reason for that app domain to die?

The application pool that i am using only have the recycle every 20 minutes of idle time enabled“

..and I thought that since I haven’t written for a while (due to a really nice and long vacationJ) and this is a pretty common scenario I would write a post on it…

Before we go into the details of how to figure out why it is recycling I want to bring up two things

What happens when an application domain is recycled
What are the reasons an application domain recycles

What happens when an application domain is recycled?

In ASP.NET each individual asp.net application resides in its own application domain, so for example if you have the following website structure

WebSite root

/HrWeb

/EmployeeServices

/FinanceWeb

/SalesWeb

…where each of the subwebs HrWeb, EmployeeServices etc. are set up as an application in the internet service manager, you will have the following application domains (appdomains) in your asp.net process

System Domain

Shared Domain

Default Domain

Root

HrWeb

EmployeeServices

FinanceWeb

SalesWeb

Apart from the first three domains (in italic) which are a bit special, each of the other ones contain the data pertinent to that application (Note: this is a bit simplified for readability), specifically they contain these things worth noting…

All the assemblies specific to that particular application
A HttpRuntime object
A Cache object

When the application domain is unloaded all of this goes away, which means that on the next request that comes in all assemblies need to be reloaded, the code has to be re-jitted and the cache including any in-proc session variables etc. are empty. This can be a pretty big perf-hit for the application so as you can imagine it is important to not have the application domain recycle too often.

Why does an application domain recycle?

An application domain will unload when any one of the following occurs:

Machine.Config, Web.Config or Global.asax are modified
The bin directory or its contents is modified
The number of re-compilations (aspx, ascx or asax) exceeds the limit specified by the <compilation numRecompilesBeforeAppRestart=/> setting in machine.config or web.config (by default this is set to 15)
The physical path of the virtual directory is modified
The CAS policy is modified
The web service is restarted
(2.0 only) Application Sub-Directories are deleted (see Todd’s blog https://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx for more info)

There may be some reasons in 2.0 that I have missed but hopefully this should cover most scenarios.

Specific issues

I want to pay a bit more attention to a few of these, which seem to be especially popularJ

Unexpected config or bin directory changes

You swear on all that is holy that no-one is touching these, but still when we start logging (as I’ll show later) the reason for the app domain recycle is a config change… how the heck can that be?

Elementary, Dr. Watson… something else is touching them… and that something else is usually a virus scanning software or backup software or an indexing service. They don’t actually modify the contents of the files, but many virus scanners etc. will modify attributes of files which is enough for the file changes monitor to jump in and say “aha !, something changed, better recycle the appdomain to update the changes”.

If you have a virus scanner that does this, you should probably consider removing the content directories from the real-time scan, of course after carefully making sure that no-one can access and add any virus software to these directories.

Web site updates while the web server is under moderate to heavy load

Picture this scenario: You have an application with 10 assemblies in the bin directory a.dll, b.dll, c.dll etc. (all with the version number 1.00.00). Now you need to update some of the assemblies to your new and improved version 1.00.12, and you do so while the application is still under heavy load because we have this great feature allowing you to update assemblies on the go… well, think again...

Say you update 7 of the 10 assemblies and for simplicity lets say this takes about 7 seconds, and in those 7 seconds you have 3 requests come in… then you may have a situation that looks something like this…

Sec 1. a.dll and b.dll are update to v 1.00.12 - appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 2. Request1 comes in and loads a new appdomain with 2 out of 7 of the dlls updated

Sec 3. c.dll is updated - appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 4. d.dll is updated

Sec 5. Request2 comes in and loads a new appdomain, now with 4 out of 7 dlls updated

Sec 6. e.dll and f.dll is updated - appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 7. f.dll is updated

Sec 8. Request3 comes in and loads a new appdomain with all 7 dlls updated

So, many bad things happened here…

First off you had 3 application domain restarts while you probably thought you would only have one, because asp.net has no way of knowing when you are done. Secondly we got a situation where Request1 and Request2 were executing with partially updated dlls, which may generate a whole new set of exceptions if the dlls depend on updates in the other new dlls, I think you get the picture… And thirdly you may get exceptions like “Cannot access file AssemblyName because it is being used by another process” because the dlls are locked during shadow copying. https://support.microsoft.com/kb/810281

In other words, don’t batch update during load…

So, is this feature completely worthless? No… if you want to update one dll, none of the problems above occur… and if you update under low or no load you are not likely to run into any of the above issues, so in that case you save yourself an IIS restart… but if you want to update in bulk you should first take the application offline.

There is a way to get around it, if you absolutely, positively need to update under load, and it is outlined in the kb article mentioned above…

In 1.1 we introduced two new config settings called <httpRuntime waitChangeNotification= /> and <httpRuntime maxWaitChangeNotification= />.

The waitChangeNotification indicates how many seconds we should wait for a new change notification before the next request triggers an appdomain restart. I.e. if we have a dll updated at second 1, and then a new one at second 3, and our waitChangeNotification is set to 5… we would wait until second 8 (first 1+5, and then changed to 3+5) before a new request would get a new domain, so a request at second 2 would simply continue using the old domain. (The time is sliding so it is always 5 seconds from the last change)

The maxWaitChangeNotification indicates the maximum number of seconds to wait from the first request. If we set this to 10 in the case where we update at second 1 and 3, we would still get a new domain if a request came in at second 8 since the waitChangeNotification expired. If we set this to 6 however, we would get a new domain already if a request came in at second 7, since the maxWaitChangeNotification had then expired. So this is an absolute expiration rather than a sliding… and we will recycle at the earliest of the maxWaitChangeNotification and waitChangeNotification.

In the scenario at the beginning of this section we could have set the waitChangeNotification to 3 seconds and the maxWaitChangeNotification to 10 seconds for example to avoid the problems.

(I know this explanation might have been a bit confusion but I hope you catch the drift)

A few things are important if you fiddle with these settings

They default to 0 if not set
maxWaitChangeNotification should always be >= waitChangeNotification
If these settings are higher than 0 you will not see any changes until the changeNotifications expire. i.e. web.config changes and dll changes etc. will appear cached.

Re-compilations

A common scenario here is that you have a set of aspx pages (containing some news items and what not) and you have a content editor that goes in periodically and updates the news with some new articles or other new content. Every time you update an aspx page it has to be recompiled, because again, asp.net has no way of knowing if it was a code update or just update of some static text… all it knows is that someone updated the files.

If you have followed some of my previous posts you know that assemblies can not be unloaded unless the application domain is unloaded, and since each recompile would generate a new assembly there is a limit to how many recompiles you can do, to avoid generation of too many assemblies (and thus limiting the memory usage for these). By default this limit is 15.

If the contents of the page is constantly updated I would recommend to dynamically get the content from a database or file rather than actually modifying the aspx pages. Or alternatively using frames with HTML pages for this content.

How do you determine that you have application recycles?

If you experience cache or session loss, it is probably a good bet, but to make sure you can look at the perfmon counter ASP.NET v…/Application Restarts.

How do you determine what caused an appdomain restart?

In ASP.NET 2.0 you can use the built in Health Monitoring Events to log application restarts along with the reason for the restart. To do this you change the master web.config file in the C:WINDOWSMicrosoft.NETFrameworkv2.0.50727CONFIG directory and add the following to the <healthMonitoring><rules> section

<add name="Application Lifetime Events Default" eventName="Application Lifetime Events"

provider="EventLogProvider" profile="Default" minInstances="1"

Comments

Anonymous
August 02, 2006
Great post!
Anonymous
August 02, 2006
PingBack from http://soci.hu/blog/index.php/2006/08/02/mikor-es-miert-indul-ujra-egy-aspnet-app/
Anonymous
August 02, 2006
Great post, Tess. Thanks so much for sharing this info!
Anonymous
August 02, 2006
Well, if you have ever wondered, Tess has a great post listing out the various reasons why. Well worth
Anonymous
August 02, 2006

My
ASP.NET 2.0 Tips, Tricks, Recipes and Gotchas "Highlights Page" [Via:
ScottGu ]
Sending...
Anonymous
August 02, 2006
Great post indeed, Thank you.
Anonymous
August 02, 2006
Great post it helped my a lot, we were having many timeout's and we didn't have a clue on why they were happening. Thank's.
Anonymous
August 03, 2006
Great! Very useful post. Hope that will help us to solve a problem. Thanks!
Anonymous
August 08, 2006
Below are some nice articles and links I&rsquo;ve found on the web over the last week that I enjoyed,
Anonymous
August 08, 2006
Tess,

This is a really useful post and has given me several pointers.

However, I am having a problem where my users are loosing their session and the folowing messag eis appearing in th event log:
Event code: 4005
Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

Doing a bit of googling reveals that a few other people are having this problem but nobody has really come up with a good explanation why. Perhaps this could be the subject of a future blog? /emote Fails miserably at charming smile

Also of possible relevance is that my web application is organised with each of my ciustomers having their own virtual folder which is pointing at a particular version of the product. So I have folders v1.1, v1.2 etc with virtual folders CompA..CompS pointing to v1.1 and CompT..CompZ pointing to v1.2. When I make a new release I create a new release folder and then repoint each company virtual folder as and when they are ready for the new release.

What are the pros and cons for this? Am I wasting loads of memory by having an AppDomain for each company when they could all be using a single AppDomain?

Thanks for your time,
Jason
Anonymous
August 09, 2006
A very nice post. keep them coming.
Anonymous
August 10, 2006
I want to know the internals of how the ASP.NET request is processed from the begining ( the user typing the url) to the end ( Appdomain unload). Is there any material or links. This post is really useful and if there are many posts like this then that will be great :)
Anonymous
August 12, 2006
I'm having a Session clearing problem with a project I'm running.
I'll try yo investigate on you leads (web.config change by some service) but I think that it's due to another thing. The website allows a lot of file uploads, through simple http requests, so my guess is that it is caused by inflation in the memory used by the AppDomain. Do you think that this can cause AppDomain restarts?
Anonymous
August 13, 2006
If it is due to memory usage, it would probably be caused by an out of memory exception, but then you wouldnt only see an appdomain restart, you would see the whole process restart (likely with a stopped unexpectedly eventlog entry). Could it be that as part of the file upload you copy the results into the bin directory or similar? I would try to log the restart reason and go from there.
Anonymous
August 13, 2006
Ramachandran,

I am not sure if this is what you are looking for, but here is an article on the page lifecycle, from when the request comes in to the ASP.NET layer to the unload of the page object

http://msdn2.microsoft.com/en-us/library/ms178472.aspx

It doesn't cover what happens before ASP.NET gets a hold of it, and it doesnt cover the appdomain unload since that is not really part of a request, but a part of the whole application, so again, I am not sure if it is what you are looking for, but it is pretty good...
Anonymous
August 13, 2006
JasonBSteele,

I hadn't seen the issue you describe before but a quick internal search revealed that you might be running into this issue which has to do with URL rewrites:

http://support.microsoft.com/kb/911300

The NullReferenceException mentioned in there might be what causes your tickets to seem expired.

If it was my own app, and I could reproduce, i would probably log exceptions to see what I could find.

I am not sure if you have any alternative ways of setting it up (without getting too deep into your particular architecture) but each appdomain has a certain overhead, and depending on how things are linked it might be quite large if you keep loading copies of the same dlls.
Anonymous
August 23, 2006
In my case, I was using the Enterprise Library Logging Block to write to the web application root directory (locally, on my dev machine only!). Once I reconfigured to stop this logging, and reset the app domain manually (resaving web.config) the session reset went away. THANKS!!! This page gave me the clues to move on.
Anonymous
August 23, 2006
Awesome, glad it helped:)
Anonymous
September 14, 2006
I enjoyed the article, and was hopeful when I started tracking application restarts. I found that my application isn't restarting when the session is unexpectedly abandoned.

Can you recommend a course of debugging when it is only the session that seems to be intermittently and [seemingly] inconsistently restarting?

TIA,

Rich
Anonymous
September 19, 2006
The comment has been removed
Anonymous
October 03, 2006
JasonBSteele & Tess,I can confirm the same behavior here. I am seeing app domain restarts no apparent cause under the label of "Application is shutting down. Reason: Configuration changed." according to the ASP.NET 2.0 health monitoring API. I also ran Filemon and confirmed there were no writes affecting of config file around the time of the restart. The only file activity affecting the web.config file is the aspnet_wp itself, and only reads.
Anonymous
October 12, 2006
JDelmerico, In ASP.NET 2.0, any change to the contents of the application folder can cause an appdomain restarts. Check out Toddca's site: https://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx Does filemon show any files from the application folder being touched ?
Anonymous
October 20, 2006
How can I get a dump file when my AppDomain restarts? I am logging asp.net app restarts via the method described by ScottGu and am getting some weird results and want to dig deeper. _shutDownMessage=Overwhelming Change Notification in C:WINDOWSMicrosoft.NETFrameworkv2.0.50727Config nothing is changing in that directory (watched it with filemon) and even turned off NTFS last access time to see if that would help and it didn't.
Anonymous
October 24, 2006
I can confirm the same behavior here. I am seeing app domain restarts no apparent cause under the label of "Application is shutting down. Reason: Configuration changed." according to the ASP.NET 2.0 health monitoring API. I also ran Filemon and confirmed there were no writes affecting of config file around the time of the restart. The only file activity affecting the web.config file is the aspnet_wp itself, and only reads.
Anonymous
October 24, 2006
If you receive the Overwhelming Change Notifications and noone is touching the files I would recommend that you contact Microsoft Support. It might be a problem in the framework that you are running into where ASP.NET reports an error during filechangesmonitoring as "overwhelming change notifications", normally the way to troubleshoot is to set breakpoints on filechanges notification functions in the webengine to see what error occurred.
Anonymous
October 25, 2006
JasonBSteele - Did you ever figure out the cause of your issue?
Anonymous
November 10, 2006
We're having frequent recycles with event message "Reason: Configuration Changed". Like several other folks, we've confirmed with filemon that nothing but w3wp.exe is touching the application folder. Wondering if anyone found the problem??
Anonymous
January 31, 2007
Great article..and helped me a lot..but i have one other issue related to session variable..My application is in asp.net 1.1..when i login in to the application i can see my details properly..but after some time in between i m able to see other person's details instead of mine..and this problem is not happening frequently.. It is happening some times..can you please tell me reason and the solution. I have stored user information in one session variable..depending on that i have shown user details. This problem we are getting on live server only..not on other servers.
Anonymous
January 31, 2007
Hi Payal, Pretty much every time I have seen this it has been one of two things.

The thing that is being put in session scope is comming from a cached or static object of some sort without proper locking mechanisms so while you are reading, some other thread/user has already written new data to it.
A router or load balancer is used and the users come from the same ISP getting the same IP address so the router gets confused and returns the session cookie to the wrong user The second scenario is a misconfiguration in a router, and is by far much less frequent than the first scenario.

Anonymous
January 31, 2007
Hi Tess, thanks a lot.. i think i m getting this problem because of second issue because i have already checked first option but session scope is not comming from any cached or static object. and if that is the case, i should get the same issue on my other servers. Previously for this same site..we are getting above issue i.e. lost of session variable due to appdomain recycles.. that time ur article helped me lot..as we found on live server there are two anti-viruses are running and causing the problem.
Anonymous
March 27, 2007
Hi Tess, We are getting the same problem as JasonBSteele: Event code: 4005 Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired. I have a question about this: I have FormsAuthentication enabled, so shouldn't the ticket expiration (the cookie) be handled regularly by FormsAuthentication subsystem? We have this problem only on IIS5, it can not be reproduced on IIS6 :-( Any suggestions would be greatly appreciated... Boris
Anonymous
March 29, 2007
Hi, there are several Apps in IIS 6.0 on Windows 2003, the config is: <authentication mode="Forms"> <forms name="QuotationCollector" loginUrl="Authentication.asmx" protection="All" timeout="20" slidingExpiration="true"> <credentials passwordFormat="SHA1"> <user name="System" password="7C222FB2927D828AF22F592134E8932480637C0D"/> <user name="Feader1" password="7C222FB2927D828AF22F592134E8932480637C0D"/> </credentials> </forms> </authentication> the others are : authentication mode="Forms"> <forms loginUrl="Login.aspx" name="TradingConsole" protection="All" timeout="15" slidingExpiration="true"></forms> </authentication> Now, only the first app log event every minute as: Event code: 4005 Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.
Anonymous
March 29, 2007
I think the most common reason for this error is that the appdomain restarted, and since you are posting these questsions in connection to a post about appdomain recycling does that mean that you do see in perfmon that the appdomain has recycled? If it has, add logging to see why it recycled
Anonymous
April 06, 2007
Hi Tess I solved the "Forms authentication failed for the request. Reason: The ticket supplied has expired. " problem by providing permanent valudation and decription keys in machine.config. The problem was caused by generation of new random validation/decription keys when appdomain restarted... Here the article about how to create keys... http://support.microsoft.com/default.aspx?scid=kb;en-us;Q312906 Boris
Anonymous
April 09, 2007
The comment has been removed
Anonymous
April 09, 2007
Hi Glenn, I'm happy that the post helped:) The optimal setup would be if the files that were updated weren't in the application path at all, but i suspect that they are supposed to be accessible from the web so in that case if you set the subdir up as an application, only that application would restart when the files are updated and a new request comes in to that application which would save the main app from restarting.
Anonymous
April 10, 2007
Tess, thanks for your reply. Unfortunately, my idea does not work; creating an application for the subdirectory does not exclude it from the parent application's FCN (file change notification). There are not many other options. It may be possible to create a directory junction (as discussed by Toddca) but from my reading it seems like they are somewhat delicate. What does seem to work--and is not delicate enough--is setting <httpRuntime waitChangeNotification="2147483647" maxWaitChangeNotification="2147483647" /> in the site's web.config, but then we lose the whole auto-deployment feature, and it becomes difficult to re-JIT the site without a full-blown manual 'iisreset'. With this option, to force the AppDomain to reinitialize I tried having a button on the site which causes it to call HttpRuntime.UnloadAppDomain() or AppDomain.Unload(AppDomain.CurrentDomain) on itself (even from a QueueUserWorkItem), but had little success. I can't see a way to do it from the IIS Manager snap-in, either. Glenn
Anonymous
April 10, 2007
The comment has been removed
Anonymous
April 10, 2007
The comment has been removed
Anonymous
April 12, 2007
Just for closure: We had to contact Microsoft. They gave 3 or 4 scenarios, but it turned out that one of our aspx pages had the directive <%@ OutputCache %>. This caused the page to be cached, hence the second user will get the first users page (and details taken from the first session). What was weird (and a bit of a red fish) was the 2nd user getting the 1st users session ID. After removing this directive - it all started to work correctly. Colin.
Anonymous
April 18, 2007
And for closure on mine: It looks like the directory junction approach, while bizarre, will work for my application and I will eventually have to use it. For now though, I have been testing <httpRuntime waitChangeNotification="2147483647" maxWaitChangeNotification="2147483647" /> on the production site for a week and this works as well, with the caveat that this disables auto-deploy and you must do a manual iisreset to force updating or re-JIT. Glenn
Anonymous
April 22, 2007
Hi There, I am having a similar problem, what other possible causes did MS give you.
Anonymous
April 24, 2007
MS response was as follows: Everything we have checked so far results to be fine, so there is only one possibility left which can lead to this problem: caching. We are probably sending to the second client a page (or data) being cached by the first client. There are 3 way this could happen:

using OutputCache on those pages: please, open the page directive section of all the pages in relation with the problem and ensure you are not using OutputCache.
caching sensitive data using the Cache[] object. Please, make sure you aren’t
having a proxy/firewall between server and clients which caches pages: please make sure this does not happen. Could you try to reproduce the problem using 2 different boxes as client, both directly connected to the server (no proxy between)? Hope that helps, Colin.

Anonymous
April 27, 2007
I too am having mysterious "Reason: Configuration Changed" application restarts. Filemon confirms that there are no writes taking place either in the application tree or in the Framework folder tree. The first mention of anything relevant in Filemon is a CLOSE of web.config, which I suspect is just the beginning of the restart, not the cause. Has anyone made progress on deducing why this happens? ...Mike
Anonymous
May 07, 2007
The comment has been removed
Anonymous
May 29, 2007
In ASP.NET Case Study: Lost session variables and appdomain recycles beschreibt Tess was passiert wenn
Anonymous
June 15, 2007
I have a question. How can we capture session timeout in IIS/ASP.NET?
Anonymous
June 25, 2007
Great article, really helped. My antivirus was casuing my appdomain to recycle. Thanks man.
Anonymous
August 26, 2007
A local radio show has started giving out cookie points to their interviewees for personal development,
Anonymous
October 04, 2007
Using ASP.NET 2, we're seeing Appdomain recycles pretty frequently with a low load - just a few users on our test system. The reason given in the eventlog is 'HostingEnvironment caused shutdown', which looks to me like a memory issue. Looking at memory, we see big chunks being taken every time another user logs in and the wp process gets to 2.5G, at which point I guess it decides to recycle itself. I can't find any reason for that amount of memory being used and on my development laptop, that doesn't happen. We are using Ajax, and WCF to communicate with the data layer. Does anyone have any experience of that combination and whether there are known performance/memory issues. Cheers, Rob
Anonymous
October 08, 2007
Since you have a pretty well defined scenario where you see memory going up everytime someone logs in, an idea would be to attach windbg or visual studio for that matter, load up sos, do !dumpheap -stat just before the log in hit g to go !dumpheap -stat after the log in compare the outputs to see what types of objects have increased the most and investigate those. Alternatively you could hook up CLR Profiler and look there at what objects have been instantiated while you were logging in. Thanks Tess
Anonymous
October 10, 2007
The comment has been removed
Anonymous
October 14, 2007
Hi George, Not sure why the config changes are being reported, but to answer your questions

The workerprocess doesnt restart only the application domain inside it so the PID should stay the same. It will only change if the whole process restarts.
No, only .config files are considered configuration files. I do seem to recall some issues reported in the early stages of 2.0 with false config restarts because of exceptions under certain conditions but i dont recall the exact details. You might just want to check that you are on the latest rollups/service packs for the framework if you haven't checked already.

Anonymous
October 23, 2007
Hi Tess, Thanks for the clarification. It looks like the framework is indeed up to date. The only other thing I can think of is that we recently began writing data to a directory inside of the application's directory tree, although Filemon confirms there are no writes in the moments leading up to the recycle. Maybe this is tangentially related to the overwhelming changes (http://support.microsoft.com/?id=913297) issue? I'll keep after it and post back here if I discover anything new. Thanks again, George
Anonymous
December 06, 2007
the perfmon counter ASP.NET v…/Application Restarts always ==0 for 2 hours, BUT the eventLog is full of this event, see Event occurrence: 36401 ! Help ！！！ Event code: 4005 Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired. Event time: 12/7/2007 11:17:06 AM Event time (UTC): 12/7/2007 4:17:06 AM Event ID: 57a0de50fe674a3fb5ab61c36938c1a7 Event sequence: 85310 Event occurrence: 36401 Event detail code: 50202
Anonymous
March 14, 2008
The comment has been removed
Anonymous
March 14, 2008
Have you gone through and verified if your appdomain is recycling (in performance monitor) that is the most common case for lost session variables. Did you follow hte article and log application restarts in the eventlog? The solution will depend on why the application/appdomain is restarting...
Anonymous
March 14, 2008
If you mean, copying 'Application Lifetime Events ' in web.config file. then yes i have copied it to webconfig but didnt got any of known reason as yet, like it is being happened becos of 'Configuration changed.' do i have to restart webserver in order to make below mention changes come in to effect. <add name="Application Lifetime Events Default" eventName="Application Lifetime Events" provider="EventLogProvider" profile="Default" minInstances="1" maxLimit="Infinite" minInterval="00:01:00" custom="" />
Anonymous
March 14, 2008
No you shouldnt have to, the best thing is probably to contact support and have someone look at it in more detail
Anonymous
March 14, 2008
<add name="Application Lifetime Events Default" eventName="Application Lifetime Events" provider="EventLogProvider" profile="Default" minInstances="1" maxLimit="Infinite" minInterval="00:01:00" custom="" /> Even after adding this , still i m getting this in event log. Event code: 4005 Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired. Event time: 14/03/2008 13:19:26 Event time (UTC): 14/03/2008 13:19:26 Event ID: d9f0333c2eed46e0b0207da69d2ea70e Event sequence: 154 Event occurrence: 5 Event detail code: 50202
Anonymous
March 17, 2008
hi Tess, it is now confirmed that in my case appdomain is not recycling. beoc since one day i have received even single event of appdomain recycle. so problem is somewhere else. i have also update machine.cofig file with custom generated keys but still having same errors in evenlog. please suggest some solution thanks
Anonymous
March 17, 2008
hi Tess, it is now confirmed that in my case appdomain is not recycling. beoc since one day i have received even single event of appdomain recycle. so problem is somewhere else. i have also update machine.cofig file with custom generated keys but still having same errors in evenlog. please suggest some solution thanks
Anonymous
March 17, 2008
I would have to look at it in a lot more detail to see what is going on and unfortunately I can't really do that in this blog setting so the best is probably to contact support and have someone look at it for you. sorry:(
Anonymous
April 10, 2008
The purpose of my presentation was to show some common pitfalls and of course to show off windbg and
Anonymous
May 08, 2008
Hi guys, I was working on the issue where event viewer showing the log "Forms authentication failed for the request. Reason: The ticket supplied has expired. ". Finally able to get to the roots of this message. here what it was happening to me. I was working on web application which is using Form Authentication and Sessions to store the user specific profiles. Here is my setting in web.config file. <sessionState timeout="60" mode="InProc" cookieless="false"/> <authentication mode="Forms"> <forms loginUrl="Pages/Login.aspx" defaultUrl=".PagesDefault.aspx" timeout ="60" slidingExpiration ="true"> </forms> </authentication> Now problem was before i didn't had form authentication timeout, which had default of 30 mins. So when request is made after being idle for 30 mins, Form authentication was throwing the user back to login page. beacuse after 30 mins the ticket supplied with the request gone expired. and this also logged warrning that user was thrown out beacuse of expire ticket passed. other problem i was having that Form Authentication was unable to redirect the user back to login page and end up with page no found error. reason for that was, missing <authorization> tags. ForAuthentication requires deny anonymous access in the Authorization tags in order to redirect the user back to login page in case if authorization failed or timeout. So in order to fix these issues i have stick in the Authorization tags. and along with that, i have also set the Form Authentication timeout same as session timeout. But this still give the warraning in Event viewer beacuse if session and Form Autentication Cookie is expiring at same time. and user makes any request after the timout first things which triggered is Form Authentication and it will throw the user to login page and log the event in Event viewer. To do quick fix i have increased Form Autentication timeout to 240 mins by leaving the session timout 60 just to make session timout before Form Authentication timeout. So when session timeout after 60 mins Code is already there to take the user back to login page so user can re loggin. If any one has batter solution to this problem, please do share it thanks
Anonymous
May 12, 2008
Hi I just thought I would share another scenario I came across, not that its because of application pool recycling but because it looked like it at first. I had an application that kept losing its session state and the actual reason was I had a maximum number of worker processes allowed set to 5. Because the application used inproc session state the session state was lost each time it jumped a worker process. The setting is in the properties of the Application Pool. Thanks
Anonymous
May 12, 2008
Absolutely, the same is of course true for web farms.
Anonymous
May 13, 2008
The comment has been removed
Anonymous
May 13, 2008
I honestly had no clue that that was the case, and it surprises me, because that is a framework setting, not an IIS setting.
Anonymous
May 13, 2008
Tess, look here, point 25. http://mvolo.com/blogs/serverside/archive/2007/12/08/IIS-7.0-Breaking-Changes-ASP.NET-2.0-applications-Integrated-mode.aspx
Anonymous
May 13, 2008
Oh, and obviously, this occurs in the Integrated Pipeline mode only.
Anonymous
May 13, 2008
The comment has been removed
Anonymous
May 13, 2008
Issues caused by frequent application restarts are pretty common. If you are interested in what causes
Anonymous
May 13, 2008
The comment has been removed
Anonymous
May 13, 2008
Still that would only mean one appdomain recycle, which is what you would see anyways with the maxWaitChangeNotification, so I am not sure I understand the issue...
Anonymous
May 13, 2008
The comment has been removed
Anonymous
May 13, 2008
ok, then you would either have to do the robocopy first, and then copy the web.config or do the update when the site is not live...
Anonymous
May 13, 2008
Yes, that's the workaround I came up with, but that's not as optimal as it was before, and mostly I can't see the reason why it should have been changed in IIS7, it worked so well in the previous versions..
Anonymous
May 21, 2008
I started this blog 2.5 years ago today, mostly because I felt that the same types of issues came up
Anonymous
May 24, 2008
I just read a post on If broken it is, fix it you should , where he's debugging a performance problem
Anonymous
August 19, 2008
The comment has been removed
Anonymous
August 19, 2008
I got an email with some questions around application domains, application pools and unhandled exceptions
Anonymous
September 18, 2008
Thanks for this post, the recycling issue has become quite an issue for our DotNetNuke implemenations of recent times (Sept 2008) and this post will help me to investigate further - great stuff!
Anonymous
September 29, 2008
I have handled a lot of session related issues till now; I felt the need to key in a few checkpoints
Anonymous
October 22, 2008
I stumbled across a very interesting find yesterday morning and Todd Carter confirmed it by sending a
Anonymous
November 05, 2008
You learn something new every day… I was working on an issue with appdomain restarts caused by page re-compilations
Anonymous
February 01, 2009
Application restart is a common issue in ASP.NET web application. Here is a very good blog entry providing
Anonymous
February 16, 2009
Nice post! But, I have a problem with app domain restarts that does not seem to be covered in your post. My web application runs smoothly on a win2003 server in iis 6 but when I deployed it on a win2008 server with iis 7 it started hickupping. After about 60 seconds the appdomain restarts due to "_shutDownMessage=IIS configuration change". No configuration files have been changed! What is happening though is that the web application is downloading files using ftp on a background thread and placing them in a directory in the app_data folder. I seems unreasonable to me that adding files in the app_data folder should cause the application to restart. Calling an "IIS configuration change" doesn't make any sense to me either! Any ideas? /Måns
Anonymous
February 17, 2009
Hi Måns, Just adding or removing files from there should not restart the app, but if something creates or deletes subfolders, even in the app_data folder then you will see app restarts with the iis config change message
Anonymous
April 28, 2009
Just fixed my session loss problem (deleting folders in App_Data). Thanks Tess (& Måns :), useful blog.
Anonymous
May 19, 2009
Nice article Tess. Got a really good understanding about application domain recycling. I have been working on a content management website since the past few weeks and my part involves the file handling functionalities. Our website actually has to create files, folders etc. in a 3-level hierarchy and also manipulate them based on what the user does. While everything was working fine, I kept getting puzzled as to why, everytime a directory was renamed (moved) successfully, I used to lose everything in the session... Debugging only revealed that the session was getting invalidated. Only after I read this and Scott Forsyth's article, did the problem become clear to me. But I have to admit that this is a sticky problem. And there is no easy way to work around this one. Keep writing!
Anonymous
July 06, 2009
Thanks for sharing knowledge. It's really good article.
Anonymous
August 16, 2009
Hi! I have a question, maybe you could help me. For example, if I try to submit a job (request to the server) after a recycle, my job is lost - no trace of it, no errors are logged, nothing but the information in event viewer that the application has recycled. I understand why this happens, but I would like to know if there's by any chance a way of showing a message to the user that something happened and he will be logged out when submitting the job? Or maybe after he logged back in? I cannot see a way of doing this, since all the dlls need to be reloaded, but I thought it's worth a shot to ask :) Thanks for the article, it's really good and revealing :)
Anonymous
August 16, 2009
i dont really see a way that you could do that... unless perhaps you keep some session variables and detect that they are lost for example
Anonymous
August 21, 2009
The comment has been removed
Anonymous
September 17, 2009
The comment has been removed
Anonymous
September 17, 2009
The comment has been removed
Anonymous
December 06, 2009
The comment has been removed
Anonymous
December 06, 2009
unfortunately i dont think there is, but even if there was, that would mean that the updates wouldnt go through until you manually recycle. So given that, perhaps it is better to have the content editors edit to a staging location and copy them all in one go at some point.
Anonymous
December 12, 2009
''Sec 1. a.dll and b.dll are update to v 1.00.12 - appdomain unload started (any pending requests will finish before it is completely unloaded) Sec 2. Request1 comes in and loads a new appdomain with 2 out of 7 of the dlls updated Sec 3. c.dll is updated - appdomain unload started (any pending requests will finish before it is completely unloaded) Sec 4. d.dll is updated Sec 5. Request2 comes in and loads a new appdomain, now with 4 out of 7 dlls updated Sec 6. e.dll and f.dll is updated - appdomain unload started (any pending requests will finish before it is completely unloaded) Sec 7. f.dll is updated Sec 8. Request3 comes in and loads a new appdomain with all 7 dlls updated So, many bad things happened here… First off you had 3 application domain restarts while you probably thought you would only have one, because asp.net has no way of knowing when you are done.''

In the above scenario, don't you feel there are more than 3 appdomain restarts, but you've stated there are 3. Am I getting this wrong ?

Anonymous
February 04, 2010
Then how about the web.config <system.web><hostingEnvironment idleTimeout="Infinite" approach? The Microsoft docs suggest that this can be set in the application level web.config and will prevent application shutdown. Problem is, I tried this on a Server2003/IIS 6.0 hosting service setup and it had no effect. The result is that every few minutes the website page requests take 20 seconds or so for a page to load up as the whole site recompiles or whatever it needs to do. The config setting seems to have no effect. Would this be something that would be overridden higher up the chain?
Anonymous
December 07, 2010
Hi Tess Came across your blog by chance and found it amazing! Was hoping you could help us with a weird problem we're encountering on our production server which has IIS6 installed. What we deal with is a simple WCF application which is hosted on default app pool in IIS. This is the only application hosted btw. Once in a while the server requests just hang up for a while, a blank period of total inactivity and at the end of the lull period activity suddenly shoots up and request completes successfully. But why this delay? There's nothing in the logs in event viewer or in the logs of the application which suggests anything. The time in between is just LOST. Nothing in the dump suggested deadlocks. Can you please help me figure out the root cause? Thanks a bunch!
Anonymous
May 11, 2011
Hi, I am updating style.css dynamically which is in App_Themes, whenever style.css is updated user gets logged out frequently. I think application restarts on style.css changes since it is in App_Themes folder. Is there any workaround for this issue?
Anonymous
July 08, 2011
Very Bad to the user.It can't give clear view to the user.
Anonymous
March 21, 2013
The comment has been removed
Anonymous
May 20, 2013
Thank you, Tess! The explanation of when an application domain is unloaded helped me solve a performance problem for a customer.
Anonymous
August 20, 2013
Thank you very much Tessa Fernandez. You've helped me solve an issue of mine. I've blogged about it here: dannyvanderkraan.wordpress.com/.../losing-session-variables-with-asp-net The world of web development can be a treacherous place! Pphheeww...
Anonymous
January 21, 2014
so great, the bin directory change will restart the domain,how awful
Anonymous
July 31, 2014
Hi Tess, thanks for this great article, help me a lot. The next step for me is to understand where the FileChangesMonitor gets its configuration from? How does FileChangesMonitor know which files and directories to monitor? Any clues? Have a nice day!
Anonymous
June 26, 2015
I'm using Microsoft routing service with Net Tcp binding. This routing service has a heavy load, 200 - 300 calls per second. The problem is, the service stop processing request from time to time. But I can see the application pool is still "started" state, and the process is still running. But just doesn't process anything. I have to manually restart application pool. Any idea for that?
Anonymous
April 18, 2016
Great Post ... thank you so much for sharing this article.

Jaa

ASP.NET Case Study: Lost session variables and appdomain recycles

Comments

Lisäresursseja