Share via

System Center Operations Manager 2012 - Monitoring the Monitor

  • Article

Introduction

Get notified when System Center Operations is down! System Center Operations Manager can’t monitor itself. When (not if) it goes off-line it cannot generate an Alert notifying you it’s dead because, well…it’s dead. There are so many conditions that result in a healthy management group going off-line. I am only going to focus on the most common scenario that results in management group service disruption - when management servers lose connectivity to the OperationsManager database.

SQL Connectivity - Single Point of Failure

System Center Operations Manager is essentially a SQL application and all functionality depends on healthy access to the OperationsManager database. If access is disrupted  Operations Manager monitoring capabilities will be disabled or at the very least become unreliable. Generally speaking, access is most commonly disrupted due to a Network Connectivity Disruption resulting is a complete Loss. When  Operations Manager can’t connect to the SQL server hosting the OperationsManager database it can’t function.

Of course there are many other reasons but SQL connectivity is the most common scenario I encounter.  The key point is regardless the reason, SCOM can’t function properly if the management servers can’t communicate / connect with the OperationsManager database. So back to the original challenge: when  Operations Manager is off-line it can’t generate an Alerts or send Notifications.

Strategy

The System Center Operations Manager log (Operations Manager) contains many events pertaining to the health of  Operations Manager. Buried deep inside are two very useful events to aid in determining SCOM / SQL connectivity health:

  1. DataAccessLayer - Event ID 26308 - This event is generated the very instant  Operations Manager loses connectivity. Often connectivity is restored quickly and SCOM will continue to operate normally – consider this a warning that it could get worse.
  2. OpsMgr SDK Service – 26330 - This event is generated after  Operations Manager has been unable to connect to SQL for approximately 5 minutes. Once 26330 is logged is safe to say  Operations Manager is off-line and will unlikely not recover without some assistance.

So all we need to do is attach an e-mail notification to this event and we are all set. Well that was the case if you were running Windows Server 2008 however with Windows Server 2012 that particular piece of functionality is deprecated – See Figure 1:

Figure 1

http://mobieus.files.wordpress.com/2013/05/1.gif

No big deal because we have PowerShell. The strategy is to assign a PowerShell script that sends an e-mail to whomever when event 26308 or 26330 is generated or any event for that matter.

Procedure

  1. Download the PowerShell script here.
  2. Save it to c:\scripts on t all management servers.
  3. Edit it with you favorite tool, I use Windows PowerShell ISE

As you can see the script is simple enough – see Figure 2:

 

 

Figure 2

 

http://mobieus.files.wordpress.com/2013/05/2.gif

Figure 3 is a sample of what I put together for Event ID 26308:

 

 

Figure 3

 

http://mobieus.files.wordpress.com/2013/05/3.gif

You can get really creative. Once I verified the script works I saved it to c:\scripts as 26308.ps1 and then I repeated for Event ID 26330.

4. Open the Operations Manager log on a management server and comb through it for event 26308. Of course there is a good chance it is not there because you may have never had an outage or it’s been overwritten. So I will leave it to you on how to force the event…stopping the SQL Service certainly will accomplish this.

5. Select the event and click on ‘Attach Task To This Event…’:

 

 

Figure 4

 

http://mobieus.files.wordpress.com/2013/05/4.gif 

  1. Add a suitable description:

 

 

Figure 5

 

http://mobieus.files.wordpress.com/2013/05/5.gif

6. Verify the Log, Source and Event ID:

 

 

Figure 6

 

http://mobieus.files.wordpress.com/2013/05/6.gif

7. Select ‘Start a program’:

 

 

Figure 7

 

http://mobieus.files.wordpress.com/2013/05/7.gif

8. Insert the path:

 

 

Figure 8

 

http://mobieus.files.wordpress.com/2013/05/8.gif

9. Select Yes when you receive the pop-up in Figure 9:

 

 

Figure 9

 

http://mobieus.files.wordpress.com/2013/05/9.gif

* *10. Be sure to place a check mark in the ‘Open the properties dialog…’:

 

 

Figure 10

 

*http://mobieus.files.wordpress.com/2013/05/10.gif 

*11. The task must execute when you are not logged in so select ‘Run whether user is logged on or not’ and also be sure to select Windows Server 2012 in the ‘Configure for:’ field:

 

 

Figure 11

 

http://mobieus.files.wordpress.com/2013/05/11.gif

12. When you click ‘OK’ you will be prompted for credentials to run the task. Be sure to specify an account with the appropriate permissions:

 

 

Figure 12

 

http://mobieus.files.wordpress.com/2013/05/12.gif

The end result is an e-mail resembling the following:

http://mobieus.files.wordpress.com/2013/05/13.gif