Configuration update fails with event IDs 29106, 29206, 29100 in OM 2007 R2
IMPORTANT: Allways perform a FULL Backup of the databases before doing anything to it !!!
ALSO: It is advised that you open a case at Microsoft before doing this - directly editing the database is not supported and you may find yourself in an unsupported state if anything goes wrong.
You may notice that the configuration fails to get updated on the RMS or a MS and if you see these 3 warning events in the Operations Manager event log, then you are impacted by this issue:
Log Name: Operations Manager<br> Source: OpsMgr Config Service<br> Date: TIMESTAMP<br> Event ID: 29106<br> Task Category: None<br> Level: Warning<br> Keywords: Classic<br> User: N/A<br> Computer: FQDN<br> Description:<br> The request to synchronize state for OpsMgr Health Service identified by "SOME_GUID" failed due to the following<br> exception "Microsoft.Mom.ConfigService.Public.ConfigServiceException: Config service failed to service the request because it is waiting to refresh in memory copy of the database.<br> at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.Managers.Synchronize(OnDoSynchronizedWork onDoSynchronizedWork)<br> at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.Execute(Managers managers)<br> at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.Run(Guid source, String cookie, Managers managers, IConfigurationDataAccessor dataAccessor, Stream stream, IConnection connection)".
followed by this event:
Log Name: Operations Manager<br> Source: OpsMgr Config Service<br> Date: TIMESTAMP<br> Event ID: 29206<br> Task Category: None<br> Level: Error<br> Keywords: Classic<br> User: N/A<br> Computer: FQDN<br> Description:<br> OpsMgr Config Service failed to build in-memory state (snapshot).<br> Reason:<br> Microsoft.Mom.ConfigService.Public.ConfigServiceCanNotApplyConfigurationChangesException: Failed to apply configuration changes. ---> System.ApplicationException: Failed to apply configuration change of type 'Addition' for object of type 'Relation' with id 'GUID' timestamped 'TIMESTAMP'. ---> System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.<br> at System.ThrowHelper.ThrowKeyNotFoundException()<br> at System.Collections.Generic.Dictionary`2.get_Item(TKey key)<br> at Microsoft.Mom.ConfigService.Common.KeyedDictionary`2.get_Item(K identity)<br> at Microsoft.Mom.ConfigService.Common.Graph`3.AddEdge(I added, E edgeContext, I fromVertex, I toVertex, OnAttachIEdge onAttachIEdge, Boolean addToVertices)<br> at Microsoft.Mom.ConfigService.Engine.InstanceSpaceManager.OnRelationChange(ConfigurationChange change)<br> at Microsoft.Mom.ConfigService.Engine.InstanceSpaceManager.OnChange(ConfigurationChange change)<br> at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.DatabaseHelper.OnChange(ConfigurationChange change) at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.DatabaseHelper.ApplyConfigurationChanges(Boolean isIncremental, ReadOnlyCollection`1 configurationChanges)<br> --- End of inner exception stack trace ---<br> at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.DatabaseHelper.ApplyConfigurationChanges(Boolean isIncremental, ReadOnlyCollection`1 configurationChanges)<br> --- End of inner exception stack trace ---<br> at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.DatabaseHelper.ApplyConfigurationChanges(Boolean isIncremental, ReadOnlyCollection`1 configurationChanges)<br> at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.DatabaseHelper.OnConfigurationChange(Boolean isIncremental, ReadOnlyCollection`1 configurationChanges)<br> at Microsoft.Mom.ConfigService.DataAccess.DatabaseAccessor.NotifyInitialization()<br> at Microsoft.Mom.ConfigService.DataAccess.DatabaseAccessor.DoNotification(Boolean isInitial, Boolean isIncremental)<br> at Microsoft.Mom.ConfigService.DataAccess.DatabaseAccessor.PollingThreadMethod()
and then this event:
Log Name: Operations Manager<br> Source: OpsMgr Config Service<br> Date: TIMESTAMP<br> Event ID: 29100<br> Task Category: None<br> Level: Error<br> Keywords: Classic<br> User: N/A<br> Computer: FQDN<br> Description:<br> OpsMgr Config Service configuration state has become invalid. This maybe a temporary issue that may be recovered from automatically. If problem persists, it usually indicates a problem with the OpsMgr database.
So wait ... what's the "actual" error?... Well the actual error is this:
Failed to apply configuration change of type 'Addition' for object of type 'Relation' with id 'GUID' timestamped 'TIMESTAMP'
So what you have to do is take a note of all the GUIDs that appear in all 29206 events and using these GUIDs, add them to the @RelIds variable separated by a comma and execute this SQL Query on the operational database:
DECLARE @RelIds NVARCHAR(MAX)<br> SET @RelIds = N' REL_GUID_1,REL_GUID_2,REL_GUID_X
' SELECT<br> BME.BaseManagedEntityId AS [BME Id],<br> BME.FullName AS [BME FullName],<br> BME.Name AS [BME Name],<br> BME.DisplayName AS [BME DisplayName],<br> BME.IsDeleted AS [BME isDeleted],<br> BME.LastModified AS [BME LastModified],<br> MT.TypeName AS [MT TypeName],<br> MT.IsDeleted AS [MT isDeleted],<br> MT.LastModified AS [MT LastModified],<br> MP.MPName AS [MP Name],<br> MP.MPVersionDependentId AS [MP VersionDependentId],<br> MP.MPVersion AS [MP Version],<br> MP.MPLastModified AS [MP LastModified],<br> R.RelationshipId AS [RelationshipId],<br> R.SourceEntityId AS [R SourceEntityId],<br> R.TargetEntityId AS [R TargetEntityId],<br> R.LastModified AS [R LastModified],<br> R.IsDeleted AS [R isDeleted]<br> FROM BaseManagedEntity AS BME WITH(NOLOCK)<br> JOIN ManagedType AS MT WITH(NOLOCK)<br> ON BME.BaseManagedTypeId = MT.ManagedTypeId<br> JOIN ManagementPack AS MP WITH(NOLOCK)<br> ON MT.ManagementPackId = MP.ManagementPackId<br> JOIN Relationship AS R WITH(NOLOCK)<br> ON (<br> R.SourceEntityId = BME.BaseManagedEntityId OR<br> R.SourceEntityId = BME.BaseManagedEntityId<br> )<br> WHERE R.RelationshipId IN (<br> SELECT value<br> FROM dbo.fn_ParseIdListInOrder(@RelIds)<br> )
Using the results out of the list - uninstall and delete out of the console all agents that will appear in the list of results.Then perform a HealthService Cache-Flush on the affected servers:
- stop the System Center Management service
- rename the \Health Service State\ folder (default in: %ProgramFiles%\System Center Operations Manager 2007\ or on the Cluster Shared Disk on a clustered RMS)
- now start the System Center Management service
If any new 29206 events appear, repeat the procedure until we get the 1210 event stating that the configuration has been received, processed and activated.
NOTE: Iff there are any 29106 events from different agents remaining, perform a HealthService Cache-Flush for all of these as well.
You should be set to go! :D