Use Azure Automation for OMS Log Search using REST API

發行項
03/26/2016

OMS is the new player in management space and its gaining momentum every day. In this blog post I want to demonstrate the capabilities around Log Search API using Azure Automation and how it can help us in daily challenges.

Scenario : While managing our datacenters we face many challenges. When we face a high utilization on a particular server or slowness in an application , we need to log into the server and collects/analyze many information to find the root cause. Operations Manager is already tracking these and reporting to us but without further analysis operator does not have any insight what's is causing the behavior on the target system. In our scenario we will use Azure Automation to do the investigation using log search API and send us the results.

General Requirements

First of all you will need an OMS subscription ( you can create one with free tier) and an Azure Automation account . Free tiers will work perfect for testing.

Runbook Requirements

Authentication

There are different ways to authenticate with Azure Resource Manager (ARM) . In this sample I’ll be mainly using the ARM REST Api for almost everything so I will authenticate using a Service Principal. You can also use native PowerShell cdmlets to perform the same . For details you can check Tiander’s blog;

https://blogs.technet.microsoft.com/privatecloud/2016/04/05/using-the-oms-search-api-with-native-powershell-cmdlets/

To create a Service principal for unattended authentication to ARM ;

Go to Active Directory tab in Azure Portal

Select Applications

Add a new application suing the following option ;

Select Web Application and give the application an URL. this URL will be used only for identifying the application so any url will work. For this demo I will use https://myomsclient

On Active Directory Applications page select the application you created in last step.

Copy ClientID – This is the username for our application

Generate a new key and select validity 1 or 2 years , this is the password for the service principal.

Copy both information as we will be storing this as variables in Azure automation.

Role Assignment in Azure Resource Manager

After service principal is created it needs to be given permissions to read the information about the resources in Azure Resource Manager. To give Read right on all resources in your subscription run the following command in Azure Powershell.

New-AzureRmRoleAssignment -RoleDefinitionName Reader –ServicePrincipalName <yourclientid>

To give permission to only OMS Workspace ;

New-AzureRmRoleAssignment -RoleDefinitionName Reader -ServicePrincipalName <yourclientid> -ResourceGroupName oi-default-east-us

oi-default-east-us is the default name for OMS accounts created in East US. You can run

Get-AzureRmResourceGroup to get the resource group name for your OMS resource group.

P.S. Replace ServicePrincipalID and SubscriptionID with your own values

Collecting Data

Now we can authenticate to Azure Resource Manager and start querying OMS Log Search API.

For unattended authentication you will need to provide the service principal ID and access key as well as you AD tenant . Easiest way to provide this to the runbook is using Runbook assets . Runbook requires the following assets;

$SPAppID = Get-AutomationVariable -Name 'SPCertAppID'    # Variable Asset   - ClientID of the AD Application
$subscriptionid= Get-AutomationVariable -Name 'MsIntSubsID' # Variable Asset   - Azure subscriptionID
$CLIENTKEY=Get-AutomationVariable -Name 'ArmclientKey' # Variable Asset   - Key to authenticate with the AS application
$resgrp=Get-AutomationVariable -Name 'OIEUSRG'     #Variable Asset   - OMS resource group   oi-default-east-us in my case
$oiwssclab=Get-AutomationVariable -Name 'OIWS_SCLAB'     #Variable Asset   - OMS workspace Name
$hotmailcreds=Get-AutomationPSCredential -Name "myhotmail" # Credential Asset : my Hotmail credential
$mailto=Get-AutomationVariable -Name "Notificationrecp" #Variable Asset   - Receipent email address
$tenant = Get-AutomationVariable -Name "ADTenant" #Variable Asset   - My Ad tenant name
$ARMUrl = "https://management.azure.com"
$ARMGraphUrl = "https://graph.windows.net"
$apiver=Get-AutomationVariable -Name 'ArmApiVersion'   # Variable asset    . I’m using 2015-03-20    as API Version

Complete runbook can be downloaded from ;

OMSSearchbyComputerWebHook.txt

Performing Queries

Runbook utilizes search-oi function to perform the OMS search. To make your own search first define the query

$query="Type=SecurityEvent AccountType=user (EventID=4624) AND TimeGenerated>NOW-8HOUR (Computer=""$server"") LogonTypeName=""2 - Interactive"" | top 5|select Account, Process,timegenerated"

and supply it as a parameter to search-oi function

$interactivelogons=search-oi -query $query

Query result :

By using these technique you can modify/extend the searches performed in this runbook.

Email Findings

Runbook will get all returned search queries and construct a html email body . We will send this mail to the recipient using the Hotmail credentials . Here after using the code Hotmail will mpst probably will block it as suspicious activity. You need to go the settings and allow this activity. If you use other email provider you might need to enable some security so Azure Automation can authenticate and send email on your behalf.

After collecting the data I’ll send the findings using the following code

$SMTPServer = "smtp.live.com"
$SMTPPort = "25"
$Username = "yourliveid@hotmail.com"
$Password = ""

$to = "<yourrecipientaddress>"

$subject = "OMS Search Demo"

$message = New-Object System.Net.Mail.MailMessage
$message.subject = $subject
$message.body = $body
$message.to.add($to)

$message.from = $username

$message.IsBodyHTML=$true

$message.AlternateViews
$smtp = New-Object System.Net.Mail.SmtpClient($SMTPServer, $SMTPPort);
$smtp.EnableSSL = $true
$smtp.Credentials = $hotmailcreds

$smtp.send($message)
write-output "Mail Sent"

Webhook

Since we will be triggering this runbook from OMS/OpsMgr easiest way to implement it is to use Webhooks.

Webhooks are URLs which we can utilize to trigger the runbook. All security is based on the URL so when you create the webhook keep the URL safe.

Navigate to your automation Account / runbooks select the runbook you have created and then select Webhooks

Create a new webhook

Give webhook a name, enable it and set an end date. Here you need to copy and save the URL , after saving the webhook you wont be able to fetch it again!!!

From webhook run settings select Run on Azure and save .

Now you will be able to trigger this runbook by Http POST.

Operations Manager Notification Channel

Now we will utilize this webhook to sstart log search automation by using the following script

Param ($server,$alert,$data )

$uri = “https://s1events.azure-automation.net/webhooks?token=replacewithyourownWebhookURI”
$headers = @{"From"=Scom@contoso.local;"Date"="$((Get-date).ToString())"}

$vms = @(
            @{ Server="$server";Alertname="$alert";Alertdata="$data";}

        )
$body = ConvertTo-Json -InputObject $vms

$response = Invoke-RestMethod -Method Post -Uri $uri -Headers $headers -Body $body

IN Operations Manager you can trigger this webhook using the powershell script above in a command notification channel ;

"G:\MPauthoring\OMSNotification.ps1" -server '$Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$' -alert '$Data[Default='Not Present']/Context/DataItem/AlertName$' -data '$Data[Default='Not Present']/Context/DataItem/AlertDescription$'

So all you need is to create a notification subscription using this new channel for the selected alerts and Operations Manager will trigger the runbook Please keep in mind, for this scenario to work your Management server needs to be able to access the webhook url.

Result

Here is the email generated after the automated search. I have queried almost all the data I have for that specific server in this demo. In real life you can decide which details to fetch from OMS based on your own needs.