Jaa

BizTalk 2013 or 2010 Hosts Crashing

  • Artikkeli

I’ve seen some posts on some discussion boards and blogs and support cases opened on this BTS2013 issue so I just wanted to provide some general information about the problem, what your workaround options are, and details on the fix. 

Problem

Periodically, your BizTalk host process crashes with the following errors in the eventlogs.  Note the error code is 80131544.  If you see the same error with a different code, you’re likely running into a different issue.

Also, notice none of the errors come from event source BizTalk Server.  The ones in the Application event logs come from .NET Runtime and Application Error and the one in the System event logs comes from Service Control Manager.

Log Name: Application
Source: .NET Runtime
Date: 9/20/2013 3:47:42 PM
Event ID: 1023
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: servername
Description:
Application: BTSNTSvc64.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 000007FDED170BC1 (000007FDECE00000) with exit code 80131544.

 

Log Name: Application
Source: Application Error
Date: 9/20/2013 3:47:42 PM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: servername
Description:
Faulting application name: BTSNTSvc64.exe, version: 3.10.229.0, time stamp: 0x50fe567a
Faulting module name: clr.dll, version: 4.0.30319.19106, time stamp: 0x51a512d4
Exception code: 0x80131544
Fault offset: 0x0000000000370bc1
Faulting process id: 0xca8
Faulting application start time: 0x01ceb6394f1dd32a
Faulting application path: C:\Program Files (x86)\Microsoft BizTalk Server 2013\BTSNTSvc64.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 830374f6-222d-11e3-93f8-00155d4683a2
Faulting package full name:
Faulting package-relative application ID:  

 

Log Name: System
Source: Service Control Manager
Date: 9/20/2013 3:47:43 PM
Event ID: 7031
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: servername
Description:
The BizTalk Service BizTalk Group : BTSOrchHost service terminated unexpectedly. It has done this 2 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.

 

Root Cause

There is a change in the .NET 4.5 CLR that results in the BizTalk process crashing during XLANG AppDomain shutdown.  XLANG AppDomain Shutdown is when the .NET AppDomain that contains the Orchestration Engine tears itself down during periods of inactivity or idleness. 

The Orchestration engine’s AppDomain shuts down by default after 20 minutes of idleness (all orchestration instances are dehydrateable) or 30 minutes of inactivity (no orchestration instances exist).

BTW, if you see this same issue in BizTalk 2010, it’s because you installed .NET 4.5 in your BizTalk 2010 environment – which is not tested or supported.  You need to be on .NET 4.0 if you’re running BTS2010.

Update:   BizTalk 2010 CU7 added support for .NET 4.5 so you should not see this issue once you are on CU7 - see Permanent Solution section for details.

Workarounds

The first and easiest workaround is to do nothing.  Note that this crash happens during periods of idleness/inactivity so your orchestrations aren’t doing anything anyway.  Also, Service Control Manager is configured to bring a BTS process up a minute after a crash so the process will come back up just fine and continue to process fine.  The crash won’t happen again until after there is orchestration activity and another period of idleness/inactivity occurs.

Note that if there are other things going on in the same host (like receive or send ports) then I wouldn’t be comfortable letting the host crash since non-orchestration work within the same host could be impacted.  In that case, you can separate out non-orchestration work into other hosts (that’s generally recommended anyway) or you can go with the below workaround. 

Another thing to consider is that, depending on the Windows Error Reporting (Dr. Watson) settings on the machine, these crashes can build up dump files on the drive.  The default location for these would be C:\ProgramData\Microsoft\Windows\WER\ReportQueue.  Check for any subfolders starting with "AppCrash_BTSNTSvc" - you can delete them if you don't need them.  So if you have limited disk space on the system drive, that might also be a reason to go with the below workaround. 

If allowing the host to crash during periods of orchestration idleness/inactivity is not an option, the workaround is to turn off XLANG AppDomain shutdown.  This is generally safe to do.  The only concern is if you have an orchestration that calls custom code that pins objects in memory to the appdomain (not good in general), then not tearing it down periodically could lead to excessive memory usage.  Still, any 24x7 environment will never have appdomain shutdown happening anyway and I’ve seen a number of environments that have very low latency requirements turn it off (since reloading XLANG engine after periods of inactivity is a perf hit to that first request). 

So, to prevent the crash from happening altogether, here’s what you do:

  1. Go to your BTS folder (default is C:\Program Files (x86)\Microsoft BizTalk Server 2013)
  2. First, save a copy of the BTSNTSvc64.exe.config file with a new name since we need to modify the original.  (BTSNTSvc.exe.config if it’s a 32 bit host that is crashing – you can check the error message to see if the crash is happening to BTSNTSvc.exe or BTSNTSvc64.exe)
  3. Open the original file in notepad and directly below the <configuration> node, add the following:
     
        <configSections>
            <section name="xlangs" type="Microsoft.XLANGs.BizTalk.CrossProcess.XmlSerializationConfigurationSectionHandler, Microsoft.XLANGs.BizTalk.CrossProcess" />
        </configSections>
     
  4. Then, directly below the </runtime> node, add the following:
     
        <xlangs>
            <Configuration>
                <AppDomains AssembliesPerDomain="50">
                                    <DefaultSpec SecondsIdleBeforeShutdown="-1" SecondsEmptyBeforeShutdown="-1"/>
                </AppDomains>
            </Configuration>
        </xlangs>
     
  5. Recycle the host

 

Permanent Solution

Microsoft has created fixes for this issue in BizTalk 2013 and 2010.

BizTalk 2013:

BizTalk 2010: