Performance Tuning for Email router
Issues frequently reported
- Microsoft.Crm.Tools.EmailAgent.SystemState.xml file gets corrupted very often which stops email router from processing the emails.
- The e-mail router configuration manager was unable to read configuration information. Make sure no files in the current directory are locked, missing or inaccessible.
Cause/Explanation
When the email agent is trying to service the emails, and updates the ThresholdDate and LastError tags in the large XML file, I/O issues can happen causing the file to grow very large and start becoming unresponsive. When this happens, the writing to the file is incomplete and some of the XML tags are not completely closed. When they are not closed the file is not recognized as a valid XML file.
<hzdzpo6bojytfabbohbc41r4rjho>
<emailaddress>Administrator@cts.lab</emailaddress>
<lasterror>
</lasterror>
<thresholddate>2010-06-19T02:37:42.0000000Z<thresholddate>
</hzdzpo6bojytfabbohbc41r4rjho>
.
.
.
<hc7huh096asdikakwriugmr3p1bk>
<emailaddress>userno1509@cts.lab</emailaddress>
<lasterror>The request failed with HTTP status 503: Service Unavailable.</lasterror>
<thresholddate>
TAG HERE IS NOT CLOSED
Now at this state when email router tries to process emails, it looks at this systemstate.xml file in order to write information into it, however it finds that this file is invalid as the Tags haven’t been closed due to huge size of the file, I/O issues.
This validation is done when the service is started, and since it’s not a valid XML file the service fails to start. There isn't a sure fire way to prevent this from ever happening, but we should be able to back up Microsoft.Crm.Tools.EmailAgent.systemstate.xml and restore it on the fly is xml validation does not work. This would save the time it takes to re-build the file, and re update the threshold dates which typically happens when users tend to uninstall and reinstall the email router. If you ask me, uninstalling and reinstalling is not a best way to work around this as it typically takes longer times.
Post update roll up 10, we have come up with an automated generation of the Microsoft.Crm.Tools.EmailAgent.systemstate.xml.bak (backup file in a valid state) which stores the same set of information as Microsoft.Crm.Tools.EmailAgent.systemstate.bad.xml.
Typical Event viewer Errors you would see when your email router is either corrupted or not working as expected
#14494 - A provider work item using assembly: Microsoft.Crm.Tools.EmailProviders.dll
and class: Microsoft.Crm.Tools.Email.Providers.SmtpPollingSendEmailProvider has
been aborted because it has been running for longer than the allowed execution time.
This behavior may occur when there is a problem with the network or because the
e-mail server is busy. Verify that you can log on to the e-mail server by using
an e-mail application that uses the account specified for the E-mail Router.
#61346 - An error occurred while checking for outgoing e-mail messages to process for SMTP: https://mia33530app001/Insphere.
System.Threading.ThreadAbortException: Thread was being aborted.
at Microsoft.Win32.Win32Native.CreateFile(String lpFileName,
Int32 dwDesiredAccess, FileShare dwShareMode, SECURITY_ATTRIBUTES
securityAttrs, FileMode dwCreationDisposition, Int32 dwFlagsAndAttributes, IntPtr hTemplateFile)
at Microsoft.Win32.Win32Native.SafeCreateFile(String lpFileName, Int32 dwDesiredAccess,
FileShare dwShareMode, SECURITY_ATTRIBUTES securityAttrs, FileMode dwCreationDisposition,
Int32 dwFlagsAndAttributes, IntPtr hTemplateFile)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access,
Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options,
SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access,
FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode,
FileAccess access, FileShare share)
at System.Xml.XmlDOMTextWriter..ctor(String filename, Encoding encoding)
at System.Xml.XmlDocument.Save(String filename)
at Microsoft.Crm.Tools.Email.Providers.ConfigFileReader.Save()
at Microsoft.Crm.Tools.Email.Providers.ConfigNodeReader.SetIntValue(String elementName, Int32 setValue)
at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.RetrieveNextMessageInternal()
at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.RetrieveNextMessage()
at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.Run()
#26234 - The E-mail Router service could not process a provider work item using assembly:
Microsoft.Crm.Tools.EmailProviders.dll and class: Microsoft.Crm.Tools.Email.Providers.SmtpPollingSendEmailProvider.
System.IO.IOException: The process cannot access the
file 'D:\Program Files\Microsoft CRM Email\Service\Microsoft.Crm.Tools.EmailAgent.SystemState.xml'
because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access,
Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options,
SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share,
Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Xml.XmlDOMTextWriter..ctor(String filename, Encoding encoding)
at System.Xml.XmlDocument.Save(String filename)
at Microsoft.Crm.Tools.Email.Providers.ConfigFileReader.Save()
at Microsoft.Crm.Tools.Email.Providers.ConfigNodeReader.
SetStringValue(String elementName, String setValue)
at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.
LogErrorToState(Exception exception)
at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.
HandleRetrieveNextMessageError(Exception exception)
at Microsoft.Crm.Tools.Email.Providers.
CrmPollingSendEmailProvider.Run()
at Microsoft.Crm.Tools.Email.Agent.ServiceCore.
ExecuteProviderWork(Object providerQueueRequestObject)
#16192 - The E-mail Router service could not run the service main background thread.
The E-mail Router service cannot continue and will now shut down. System.Configuration.ConfigurationErrorsException:
The E-mail router service cannot access system state file Microsoft.Crm.Tools.EmailAgent.SystemState.xml.
The file may be missing or may not be accessible.
The E-mail Router service cannot continue and will now shut down. --->
System.Xml.XmlException: Unexpected end of file has occurred.
The following elements are not closed: ThresholdDate, Hc7huh096aSDIKakwrIUgMr3p1Bk,
State. Line 9878, position 20.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.Throw(Int32 pos, String res, String arg)
at System.Xml.XmlTextReaderImpl.ThrowUnclosedElements()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.Load(String filename)
at Microsoft.Crm.Tools.Email.Providers.
ConfigFileReader..ctor(String filePath, ServiceLogger serviceLogger)
at Microsoft.Crm.Tools.Email.Providers.
SystemState.Initialize(ServiceLogger serviceLogger)
at Microsoft.Crm.Tools.Email.Providers.SystemState..ctor(ServiceLogger serviceLogger)
at Microsoft.Crm.Tools.Email.Agent.ServiceCore.InitializeSystemConfiguration()
--- End of inner exception stack trace ---
at Microsoft.Crm.Tools.Email.Agent.ServiceCore.InitializeSystemConfiguration()
at Microsoft.Crm.Tools.Email.Agent.ServiceCore.ExecuteService()
#16192 - The E-mail Router service could not run the service main background thread.
The E-mail Router service cannot continue and will now shut down.
System.Threading.ThreadAbortException: Thread was being aborted.
at Microsoft.Crm.Tools.Email.Agent.ServiceCore.UpdateConfiguration()
at Microsoft.Crm.Tools.Email.Agent.ServiceCore.ScheduleProviderWork()
at Microsoft.Crm.Tools.Email.Agent.ServiceCore.ExecuteService()
Recommendations
- Rename the Microsoft.Crm.Tools.EmailAgent.systemstate.xml to Microsoft.Crm.Tools.EmailAgent.systemstate.bad.xml. This file is located at X:\Program Files\Microsoft CRM Email\Service, where X is the email router install directory.
- Rename the Microsoft.Crm.Tools.EmailAgent.systemstate.xml.bak to Microsoft.Crm.Tools.EmailAgent.systemstate.xml
- Lower the max messages to 500 on Incoming / Outgoing Profile – Advanced Tab. This value is set to 1000 by default. Reducing the count to 500 will reduce the number of emails to be processed by the email router in 1 minute.
- I would recommend to implement the registry keys mentioned in the KB articles - https://support.microsoft.com/kb/981053 and https://support.microsoft.com/kb/2403296 .Please do not go by the Title of the article though. I want to focus on the real time errors, events that we hit with Email router in such scenarios which are explained in the symtopms OR the cause sections.
- Improve E-mail Router performance within large deployments:-
For Windows Server 2003: https://msdn.microsoft.com/en-us/library/aa560610(BTS.20).aspx
For Windows Server 2008: https://www.microsoft.com/downloads/details.aspx?familyid=12AC9780-17B5-480C-AEF7-5C0BDE9060B0
FAQ’s on Email router: https://support.microsoft.com/kb/2023229
CRM 4.0 E-mail Router deployment guidelines - https://rc.crm.dynamics.com/rc/regcont/en_us/op/articles/configrouter.aspx#o30952
These would be the best practices I would recommend in such scenarios where there is a huge list of users using the email router and when the email router stops responding / EmailAgent.xml file gets corrupted.
Cheers,
Solomon Einstein