Share via

Exchange 2010 Correlation Engine Not Generating Alerts

  • Article

While helping a customer migrate from SCOM 2007 to SCOM 2012 R2, we noticed a large reduction in Alerts for Exchange 2010.  After investigating further, we noticed that we were not getting any Alerts that come from the Exchange 2010 Correlation Engine.

After verifying with my customer that the Exchange 2010 Monitors were generating state changes, but no alerts were coming in, we reviewed the event log for the Correlation Engine (it logs to the Application Event log on the server it is installed on) and saw event 717 being logged every 5 minutes (more details below).

After doing some research, I found the following information which resolved the problem (provided by Microsoft Escalation Engineer Manoj Parvathaneni).

Quick Summary:
We had to install the Correlation Engine on a server without any SCOM 2012 server components installed, and include some SCOM 2007 dlls in the Correlation Engine installation directory.

Symptoms

When the Exchange Correlation Engine is installed on a System Center 2012 Operations Manager Management Server, the Automatic Alert Resolution feature in the Exchange Correlation Engine ceases to function and the following event is logged in the Application event log on the Management Server:

Log Name: Application
Source: MSExchangeMonitoringCorrelation
Event ID: 717
Task Category: General
Level: Warning
Keywords: Classic
User: N/A
Description:
Connection with the Operations Manager Root Management Server failed.

Error: System.InvalidCastException: [A]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo] cannot be cast to [B]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo]. Type A originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'. Type B originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'.
at Microsoft.EnterpriseManagement.Utilities.GetDeserializeObject[T](Byte[] serializedObject)
at Microsoft.EnterpriseManagement.DataAbstractionLayer.InstanceSpaceOperations.UpdateAlerts(Byte[] alerts, String comments, Nullable`1 modifyingConnectorId)
at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateAlertsInternal[T](IList`1 alerts, String comments, Nullable`1 modifyingConnectorId, ManagementGroup managementGroup)
at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateInternal(String comments, Nullable`1 modifyingConnectorId)
at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.Update(String comments)
at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlert(Entity entity, MonitoringAlert monAlert)
at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlerts(Entity entity, String alertCriteriaString)
at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateWithinEntity(Node entity)
at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateBatchTask(Object batchData)

Number of occurrence: 3

You'll also notice the following error in the Exchange Correlation Engine logs:

2013-11-18T08:16:15.673Z,4,Information,"CorrelateBatchTask: caught exception [System.InvalidCastException: [A]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo] cannot be cast to [B]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo]. Type A originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'. Type B originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'. at Microsoft.EnterpriseManagement.Utilities.GetDeserializeObject[T](Byte[] serializedObject) at Microsoft.EnterpriseManagement.DataAbstractionLayer.InstanceSpaceOperations.UpdateAlerts(Byte[] alerts, String comments, Nullable`1 modifyingConnectorId) at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateAlertsInternal[T](IList`1 alerts, String comments, Nullable`1 modifyingConnectorId, ManagementGroup managementGroup) at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateInternal(String comments, Nullable`1 modifyingConnectorId) at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.Update(String comments) at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlert(Entity entity, MonitoringAlert monAlert) at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlerts(Entity entity, String alertCriteriaString) at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateWithinEntity(Node entity) at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateBatchTask(Object batchData)].

In addition, the Exchange Correlation Engine will not report any new Alerts or report the alerts to SCOM 2012 inconsistently.

Resolution

The exception above can be resolved by using one of the following methods:

Method 1: Install Exchange Correlation Engine on a dedicated system without the SCOM 2012 Management Server component

  1. Install Exchange Correlation engine on this machine.
  2. Stop the Microsoft Exchange Monitoring Correlation service.
  3. Copy SCOM 2007 R2 SDK binaries into the CE install directory (Program Files\Microsoft\Exchange Server\v14\Bin).
    • Microsoft.EnterpriseManagement.OperationsManager.Common.dll
    • Microsoft.EnterpriseManagement.OperationsManager.dll
    • MomBidLdr.dll

**UPDATE: Files attached to this blog

Where can you find the above binary files?

  • The SCOM 2007 R2 SDK Binaries Microsoft.EnterpriseManagement.OperationsManager.Common.dll and Microsoft.EnterpriseManagement.OperationsManager.dll can be found on a system that has SCOM 2007 R2 Console in the following directory Program Files\System Center Operations Manager 2007\SDK Binaries.
  • The MOmBidLdr.dll can be found can be found on a system that has SCOM 2007 R2 Console in the following directory Program Files\System Center Operations Manager 2007

Method 2: Disable the Automatic Alert Resolution Feature in the Correlation Engine

By following this approach the customer will lose the ability to resolve the alerts automatically via the Exchange Correlation Engine.

To disable Automatic Alert Resolution, perform the following steps:

  1. Log on to the server that's hosting the Microsoft Exchange Monitoring Correlation Engine service.
  2. Locate the Correlation Engine configuration file named Microsoft.Exchange.Monitoring.CorrelationEngine.exe.config. By default, the file is located in C:\Program Files\Microsoft\Exchange Server\V14\Bin\, where C:\ is the Exchange installation directory.
  3. Open Microsoft.Exchange.Monitoring.CorrelationEngine.exe.config in a text editor such as Notepad
  4. Locate the following line in the configuration file:
    <add key="AutoResolveAlerts" value="true" />
  5. Change <add key="AutoResolveAlerts" value="true" /> to <add key="AutoResolveAlerts" value="false" />
  6. Restart the Microsoft Exchange Monitoring Correlation Engine service.

More Information

Exchange Correlation Engine:
The Correlation Engine(CE) is a stand-alone Windows service that uses the Operations Manager SDK interface to first retrieve the health model (or instance space) and then process state change events. By maintaining the health model in memory, and processing state change events, the Correlation Engine is able to determine when to raise an alert based on the state of the system. The Correlation Engine solely relies on the Operations Manager SDK. It does not use the agent to perform any of the correlation and other aspects of its operations. For further details, you can also reference the Exchange 2010 Management Pack guide.

Automatic Alert Resolution feature in Exchange Correlation Engine:
The Automatic Alert Resolution feature automatically closes related alerts when the Exchange 2010 Management Pack determines that the underlying issue is no longer a problem. This feature is provided by the Correlation Engine, and is enabled by default. Using Automatic Alert Resolution can cause multiple alerts to be logged if the same alert is logged again for another instance of the problem before the associated ticket has been resolved by support teams.

You may also want to disable this feature under the following or other conditions:

  • If you're using ticketing or another support system that wouldn't work correctly if alerts are automatically resolved.

If you're using a connector with Operations Manager 2007. A connector is a custom service or program that allows Operations Manager to communicate with external systems. For example, you may want to disable this feature if you're using a connector that allows an external application to track Exchange 2010 Management Pack alerts.

SCOM2007FilesForEXCHCorrelationEngine.zip

Comments

  • Anonymous
    January 01, 2003
    Each of the Rules should have a corresponding monitor which is targeted at one of the Exchange 2010 classes...these monitors generate state changes which the Correlation Engine uses to generate alerts. If you want to disable the monitoring for a specific server, set an override on the monitor (target the OR at the server you want to disable).
  • Anonymous
    January 01, 2003
    As far as I know, the Correlation Engine has never required being installed on a Management Server (although I always have installed it on one). It does need to connect to an SDK service, which can be local (if on a MS) or remote. In SCOM 2007 it had to connect to the RMS (since that is the only role with SDK service), but in SCOM 2012 it can connect to any Management Server, since they all run the SDK service (Data Access Service). The only role of the RMS Emulator, as it relates to the Correlation Engine, is to generate the Alert data.
  • Anonymous
    January 01, 2003
    This happens if some Management Pack has a Rule where Category=None. You can run the following query on the OperationsManager database to find it:

    select * from Rules where RuleCategory = 'None'
  • Anonymous
    April 17, 2015
    The comment has been removed
  • Anonymous
    April 17, 2015
    Hmm, looks like you're right. I double-checked the MP guide and it says:

    "Determine which server will host the Correlation Engine. While not strictly required, it's strongly recommended that the Correlation Engine be installed on the root management server."

    I guess when I originally read that I interpreted it to mean that it had to be installed on the server hosting the RMS emulator role....
  • Anonymous
    April 21, 2015
    The comment has been removed
  • Anonymous
    April 21, 2015
    Do you happen to know if there is any way to target an override against a specific system for an alert generated by the Correlation Engine? For example I have one server that I want to disable alerts for. However when creating the override I have to target it against the MS server hosting the correlation engine (in my case my MS holding the RMS emulator roll). Since I am already targeting "all objects of another class", I don't see any way to target it against just a specific server as opposed to having the override apply to an entire object class. Am I missing something here?

  • Anonymous
    April 29, 2015
    I've found the Rule with Category "none" its from Advisor Change Tracking Intelligence Pack...but i cannot change/delete/remove category on the rule.
  • Anonymous
    December 16, 2015
    Where else can we get these needed 2007 DLLs if we don't have a 2007 instance? We only have 2012 R2 instances. Using method 2 to disable the auto resolve is not ideal for us.
  • Anonymous
    December 16, 2015
    I just zipped the files and attached to the blog.
  • Anonymous
    December 16, 2015
    Thanks, hopefully they can be added to the MP MSI with the other dlls that are needed at some point.