Using OpsMgr Notifications in the real world - Part 1
Using notifications such as email, has proven to be somewhat difficult in OpsMgr. I get questions from each and every customer I work with on this topic.... as most people never get these to work as they want, or expect, without fully understanding the object oriented class model of OpsMgr and what that means to Notifications on alerts. Even once this is understood, there are often challenges.
First.... lets make sure we use the correct terms. Alerts are alerts.... a message that is sent to the OpsMgr database with details on something that happened. A Notification is simply a message in email, IM, SMS, etc... that is triggered by an alert.
Ok - now... on to actually using these things. :-)
First off.... notifications are configured via subscriptions. We can create a subscription and get a notification for every single alert that gets generated.... or we can filter them for specific alerts based on specific criteria.
For instance... we can subscribe to "all critical alerts" by creating a subscription that includes all classes, all groups, and just looks at "Critical" alert severity on the criteria selection.
We can subscribe to "all critical alerts coming from the exchange management pack" by creating a subscription that includes ONLY Exchange classes, All groups, and just looks at "Critical" alert severity on the criteria selection.
Further filters can be created... using groups.... such as "all critical alerts from the exchange MP for servers in Texas".
Sound good? At this point.... then - it is time to read my previous blog post on configuring notifications for specific groups and classes. It will walk you through the steps to filter based on a group of servers, or filter based on specific alerts from MP's you are interested in... and some of the issues created when you do that.
Moving on.....
Now - you have set up your notification subscriptions that you like:
You have created groups that contain the Windows Computer objects and Health Service Watcher objects for each region or team.
You have created multiple subscriptions based on specific management packs... by filtering the subscription based on class, and group.
You have also specified specific criteria for these subscriptions... like only getting critical alerts in email.
Everything seems to be going well.... when someone throws a big wrench in your plan. You get major complaints with your system because:
Users are complaining that they are getting WAY too many emails.... because they are constantly getting flooded with noise from critical alerts.
Even with all the work you did.... filtering by specific class, and specific groups... you still get a LOT of email notifications. The FIRST focus should be on tuning the management packs and fixing problems. However, it is possible that some MP's are just noisy, or problematic, etc. You can adjust some of the alert severity to Warning... for alerts you don't want notifications.... but ultimately... you are getting too much in email.
Possibly - you want alerts in the console... but only want VERY SPECIFIC notifications on the "most critical" items. Your users have decided to set up a rule in Outlook and send all these noisy critical notifications to a folder.... to keep them out of their Inbox. This essentially makes sending these notifications useless.... because they will be ignored and annoy the end user. If we were sending them to a phone/pager... the phone/pager is constantly going off. What they REALLY want is a way to subscribe to only the alerts they deem REALLY critical. This may be from 10 rules, or 300 rules.
Using the GUI.... there is no easy way to do this. We cannot subscribe to a single rule, or monitor, that creates an alert. We can only subscribe to a CLASS... and therefore we will get email notifications for any critical alert that uses that class as a TARGET.
So.... what we need to develop is a STRATEGY for notifications... so we only get emails/pages on alerts that WE DEEM to be actionable. There are many ways to accomplish this... I will talk about a few that I have implemented with customers and seem to work pretty well.
1. Using Alert Priority as a notification filter strategy. (this blog post)
2. Using Powershell to create individual notification subscriptions for individual rules/monitors. This is blogged here:
3. Using the SDK and a product connector to modify alerts after they are generated, modifying some property on the alert such as Resolution State to a custom state, which we can then use as a criteria for a subscription.
4. Creating a custom class for a specific set of computers using the authoring console, and then subscribing only to that class.... therefore, we will only get alerts from rules/monitors that use that class as a target.
This is blogged here: https://technet.microsoft.com/en-us/magazine/cc983816.aspx
Ok.... let's get started.
#1. Using Alert Priority as a notification filter strategy.
This is my most preferred method right now for most scenarios. In MOM 2005... we had 7 alert severities to choose from....
This was great.... because we could make our notification criteria something like "Critical Error or higher" and then tune the alerts down or up.
Now, in OpsMgr - this won't work.... because with sealed MP's, and only three alert severities... it isn't as simple. There are just too many alerts in our default MP's that are tagged as critical... and perhaps we really don't want to bump them down to warning... we just want to pick which critical alerts to notify on. In comes "Priority".
There is a new attribute of an alert in SCOM, called Priority. High, Medium, or Low. Essentially, when you think about this - we now have 9 possible states for an alert "severity": Critical High, Critical Med, Critical Low, Warning High, Warning Med.... etc...etc...
In almost all of the default management packs you download... the alert priority is set to medium. We have also updated most of the MP's so that alert severity and priority are exposed as adjustable via override. This way - we can create a new notification subscription, and ONLY select "Critical" severity, and "High" priority:
What you will find.... is there you will get very few notifications by default with this criteria. This is good! The next step is to simply examine the rules and monitors that create alerts that you are interested in, and override those to change the alert priority to HIGH.
Note - when creating this override - sometimes we show the word "High" and sometimes we show the numeric number which relates to "High". In the database, this is translated numerically, just like Alert Severity is. See the table below to understand these:
Severity: Critical 2 Warning 1 Information 0 Priority: High 2 Medium 1 Low 0
Here is an example - I am overriding my Exchange Services Monitor alert to High priority... so I will get email notifications for it based on my subscription above:
Here is an example, for an alert for very low disk space, from a rule, where we have to use the numeric:
From that point forward, you will continue to get all the alerts in the console, but only email notifications on the alerts you want, which you set to have a HIGH priority.
So now we have a strategy! Now we can have alerts coming in the console as normal... and we will only get email notifications on specific alerts that we have deemed to be a high priority.
You could use this same design for a product connector as well... to limit what is forwarded up the chain to Tivoli, OpenView, Remedy, etc.
For part two.... see the following blog post on using powershell to subscribe to individual notifications:
Creating granular alert notifications - rule by rule, monitor by monitor
Comments
Anonymous
January 01, 2003
Great article Kevin, Thank you!Anonymous
January 01, 2003
I have had a few requests now for this, so I thought I would take the time to write up the process.  Anonymous
June 28, 2008
To be completely honest, the creation of custom classes is really the only acceptable solution in my mind. Not only does it allow us to have the highest level of control, it lets us setup much more specific notification lists. If you have a class called "Widget Application Error" and write events to it, then you can subscribe people just to those events. At the current site they do do things a lot .. ahem.. different then is the norm, but it seems to be the only way I can make them happy. Using powershell to create alerts is nice in theory, but the second anyone opens that notification inside the GUI it converts it to an 'all alerts' rule since the GUI can't handle it. But alas, it seems like once again we hit upon the same thing at the same time - but you were faster on the blogging front :P Nice work though, looking forward to the rest of the series. Jeremy Pavleck Pavleck.Net - SCOM & PowerShellAnonymous
August 08, 2008
It is unfortunate that you have not had time to do part 4 of your post. The section that will cover the custom classes. I have created a custom management pack along with custom classes. I populate these classes using my developed Connector. My next step would be to monitor these classes and generate alerts. And then create reports based on the data inserted over time into the classes. I have no idea how to do this. Would be nice if you could help me out.Anonymous
April 23, 2010
This is great I have all my monitors set up so far but where is the first screen, did the screen changed in the latest version? Thanks, DomAnonymous
August 29, 2014
С принятием Федерального закона Российской Федерации от 21.07.2014г. № 248-ФЗ «Об исчислении времени»Anonymous
August 29, 2014
С принятием Федерального закона Российской Федерации от 21.07.2014г. № 248-ФЗ «Об исчислении времени»Anonymous
October 08, 2014
Обновление от 09.10.2014: В данную статью внесены дополнения, в связи с выпуском пакета обновлений KB2998527Anonymous
August 02, 2015
sdAnonymous
August 02, 2015
The comment has been removedAnonymous
July 28, 2016
They problem from our environments, perspective is there is no flood control to any of this. If, for instance, a network switch goes out and all the downstream computers start throwing alerts... out Monitoring team is flooded with SMS pages. I'm think I need to build a "Big Red Button" to enable us turn off notifications from a limited time.