Simple look at OMS Alert Remediation with Runbooks: Part 3

Alerting in OMS was recently updated with the GA release. One of the great features this offers is the ability to automatically trigger an Azure Automation Runbook.

There are already some great blogs out there, but in this series I wanted to take a simple look how this works, now it's generally available.

Part 1 link

Part 2 link

 

Triggering child Runbooks

In the previous blogs, we saw how to trigger a Runbook from an OMS Alert & read out the alert details.

Over the years, working with Automation tools, like System Center Orchestrator, I've become a fan of keeping my Runbooks simple!

Having big, monolithic Runbooks might look cool, but they can be a pain to troubleshoot & maintain. Splitting bigger tasks into small Runbooks also brings benefits, such as the ability to reuse!

 

In the spirit of this, the model I suggest is:

• Create a Trigger Runbook, which is triggered from the OMS Alert
• The trigger Runbook can parse the Alert RequestBody & then go and start another Runbook to complete a task

Why?

• If you've already got a Runbook which does something, you don't have to modify it to handle WebhookData input
• If you've got an existing PowerShell script, you can put it into Azure Automation, without having to alter the parameters

 

I also think it's cleaner to separate out the parsing of the alert data & the actual Remediation Runbook.

Step 1: Create a child Runbook

This is the Runbook which I want to be triggered by my existing OMSAlertTrigger Runbook.

I created a new Runbook called OMSAlertTriggerChild

 

 

I've just created a very basic Runbook, which takes 3 parameters.

Each parameter is simply written out. I saved & Published this Runbook.

 

Step 2: Call the child Runbook

Next, I've gone back to OMSAlertTrigger. I want to modify this, so it triggers the child Runbook instead of writing out data.

You can see below, I simply needed to call the Runbook OMSAlertTriggerChild by name & pass in the 3 parameters it accepts.

You can add it easy from selecting 'Runbooks' > Right click > Add to canvas:

 

You may have noticed I also added in -parallel so it can parse the alerts in parallel!

I then saved & published this Runbook

 

 

Step 3: Trigger the updated Runbook

As before, I'll trigger the Runbook by manually raising event 123 on one of my OMS managed machines. I ran this several times, to rest out multiple alerts:

Write-EventLog -LogName Application -Source MsiInstaller -EventId 123 -Message "hello"

I'll wait for a moment to get the Alert notification, then check the Runbook Job in the portal.

First thing you'll notice – you only see the Job for the OMSAlertTrigger, we don't see a separate job for the child Runbook:

 

 

However, if I look in the output for OMSAlertTrigger, I can see it shows me the output from the OMSAlertTriggerChild Runbook.

This is a lot simpler – I don't have to worry about linking job outputs together. I can simply see the output, although it called my child Runbook:

 

 

What about Hybrid Runbooks?!

In case you've not seen it, we can use the Hybrid Worker to run the Runbook in your datacenter. This is easy to setup, more info is at https://azure.microsoft.com/en-us/documentation/articles/automation-hybrid-runbook-worker/.

This is also exposed when we setup an Alert in OMS. In the screenshot you can see this time, I've selected to run on Hybrid Worker and selected my hybrid workers.

 

 

That's OK, but what about the Child Runbook? Although I've clearly indicated I wanted OMSAlertTrigger to run on my hybrid worker, where will OMSAlertTriggerChild be run? Obviously, I want this to be run on the hybrid worker also.

To test this out, I added a line to my OMSAlertTriggerChild Runbook, to write out the name of the machine it's running on:

 

Great News! It triggered my child Runbook on my Hybrid Worker too! J

As we can see in the screen below, the Parent Runbook Ran on prem. (see 'Ran on').

The Child runbook, which wrote the output said it's running on DC01, which is my on-prem hybrid worker.

 

To conclude

It sometimes makes sense to split up your Runbooks into bitesize pieces. It's often easier to manage, you may end up re-using parts!

If you trigger a Runbook on a Hybrid Worker & it calls another Runbook – that child Runbook is Run on the Hybrid worker also.

One final note – You can run an Azure Automation Runbook from PowerShell, using Start-AzureRmAutomationRunbook. This accepts the RunOn argument, which allows you to select it to Run on a Hybrid Worker. Let's say you wanted to trigger one Runbook from another, but have that run on a different Hybrid Worker, then that's how it could be done.