Using the HTTP OMS Data Collector API for real-world scenario’s–Part 2
In the first part of this blog post series I started with a PowerShell function to retrieve stock information and converting this to the correct format so the Log Analytics HTTP Data Collector API is able process the record with stock information.
For more information about how you would post JSON data to the Log Analytics repository from any client able to call the REST API I refer to the Azure documentation here: https://azure.microsoft.com/en-us/documentation/articles/log-analytics-data-collector-api/
On that page you can find a PowerShell example which we are going to use as basis for our Azure Automation Runbook which will do the following:
- Retrieve stock information
- Format retrieved stock data so the Log Analytics HTTP Data Collector API is able process the record
- Create PowerShell Functions to create a authorization signature and finally send the data to OMS.
StockPriceOMS Runbook:
If you are new to creating Azure Automation Runbooks I would recommend that you have a look at some my earlier blog post about Azure Automation and Runbooks. Another great start is the Azure Automation overview from Matt Goedtel .
I created the following PowerShell script which we are going to use for the Azure Automation Runbook.
[sourcecode language='bash' padlinenumbers='true']
#requires -version 3.0
# ---------------------------------------------------
# Script: C:\Users\Stefan\Documents\GitHub\AzureAutomationDemo\StockPriceOMS.ps1
# Tags: OMS, PowerShell, Azure, Runbook, Automation
# Version: 0.1
# Author: Stefan Stranger
# Date: 09/05/2016 17:07:24
# Description: Demo Runbook which retrieve stock information and uses the Log Analytics HTTP Data Collector API
# to send this data to OMS.
# Comments:
# Changes:
# Disclaimer:
# This example is provided “AS IS” with no warranty expressed or implied. Run at your own risk.
# **Always test in your lab first** Do this at your own risk!!
# The author will not be held responsible for any damage you incur when making these changes!
# ---------------------------------------------------
#region variables
# Replace with your Workspace ID
$CustomerID = Get-AutomationVariable -Name "CustomerID"
# Replace with your Primary Key
$SharedKey = Get-AutomationVariable -Name "SharedKey"
#Specify the name of the record type that we'll be creating.
$LogType = Get-AutomationVariable -Name "Logtype"
#Specify a time in the format YYYY-MM-DDThh:mm:ssZ to specify a created time for the records.
$TimeStampField = Get-AutomationVariable -Name "Logtype"
#Specify the name of the Stock we want to retrieve.
$StockName = Get-AutomationVariable -Name "StockName"
#Create Array
$StockName = $StockName.Split(",")
#endregion
#region functions
# Function to retrieve Stock value information
function Get-StockPrice
{
[CmdletBinding()]
Param
(
# Stockname(s)
[Parameter(Mandatory = $true,
ValueFromPipelineByPropertyName = $true,
Position = 0)]
$StockName
)
Process
{
$HttpRequestUrl = 'https://finance.google.com/finance/info?client=ig&q={0}'-f $StockName
$stockvalue = Invoke-WebRequest -Uri $HttpRequestUrl -UseBasicParsing
#Fixing output. ONLY works with one STOCK PRICE!
$stockprice = ((($stockvalue.content).Replace('//','')).Replace('[','')).Replace(']','') | ConvertFrom-Json
#Convert to correct types
$stockprice.id = [int]$stockprice.id
$stockprice.l = [double]$stockprice.l
$stockprice.l_fix = [double]$stockprice.l_fix
$stockprice.l_cur = [double]$stockprice.l_cur
$stockprice.c = [double]$stockprice.c
$stockprice.c_fix = [double]$stockprice.c_fix
$stockprice.cp = [double]$stockprice.cp
$stockprice.cp_fix = [double]$stockprice.cp_fix
$stockprice.pcls_fix = [double]$stockprice.pclx_fix
ConvertTo-Json -InputObject @($stockprice)
}
}
# Function to create the authorization signature.
Function New-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
$xHeaders = 'x-ms-date:' + $date
$stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource
$bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
$keyBytes = [Convert]::FromBase64String($sharedKey)
$sha256 = New-Object -TypeName System.Security.Cryptography.HMACSHA256
$sha256.Key = $keyBytes
$calculatedHash = $sha256.ComputeHash($bytesToHash)
$encodedHash = [Convert]::ToBase64String($calculatedHash)
$authorization = 'SharedKey {0}:{1}' -f $customerId, $encodedHash
return $authorization
}
# Function to create and post the request
Function Send-OMSData($customerId, $sharedKey, $body, $logType)
{
$method = 'POST'
$contentType = 'application/json'
$resource = '/api/logs'
$rfc1123date = [DateTime]::UtcNow.ToString('r')
$contentLength = $body.Length
$signature = New-Signature `
-customerId $customerId `
-sharedKey $sharedKey `
-date $rfc1123date `
-contentLength $contentLength `
-fileName $fileName `
-method $method `
-contentType $contentType `
-resource $resource
$uri = 'https://' + $customerId + '.ods.opinsights.azure.com' + $resource + '?api-version=2016-04-01'
$headers = @{
'Authorization' = $signature
'Log-Type' = $logType
'x-ms-date' = $rfc1123date
'time-generated-field' = $TimeStampField
}
$response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
return $response.StatusCode
}
#endregion
# Main
# Submit the data to the API endpoint
# Iterate through the configured StockNames.
Foreach($Stock in $StockName) {
$json = Get-StockPrice -StockName $Stock
Write-Output $json
Send-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $logType
}
Explanation of Runbook PowerShell script:
This PowerShell Script Runbook has the following sections:
- Azure Automation Assets
- Three Functions
- Get-StockPrice
- New-Signature
- Send-OMSData
If you look at the Functions you see they have some parameters that need to be configured using Azure Automation Assets. The following Azure Automation Assets need to be configured:
- CustomerID (OMS Workspace ID, see picture below)
- SharedKey (Primary Key, see picture below)
- LogType (this is name of the Type of data entries you want to use for the Stock data entries. This could be anything, like “StockPrice”. Keep in mind OMS will ad “_CL” to your LogType Name)
- TimeStampField (Specify a time in the format YYYY-MM-DDThh:mm:ssZ to specify a created time for the records.) See here for more info). Not configured in the example.
- StockName (this is string with one of more stock names separated by a comma. Example: “MSFT,GOOGL” or “MSFT”)
You can find the CustomerID aand SharedKey information within the OMS Portal under Overview –> Settings –> Connected Sources
Configure Azure Automation Assets in the Azure Portal
Go to Azure Portal, select your Automation Account and select Assets.
Create the following Variables Assets with your own values. All the Variable Assets are of the type String but make sure you select Encrypted for the CustomerID and SharedKey Variable Assets because this is confident information you would keep kind of secured.
Create StockPrice Runbook
Go to the Azure Portal, select your Automation Account and select Runbooks.
Click on Add Runbook –> Create a new runbook.
Enter a Name and select Runbook type PowerShell and click on Create.
Copy and paste PowerShell script from above to editor in the Azure Portal and click on Save.
Before publishing the Runbook we are first going to test the Runbook. Select the Test Pane button and click on Start.
If the test is successful you should see something similar to below example.
We can now publish the Runbook. We need to Publish the Runbook if we want to add a Schedule to have the Runbook running at a configured interval, like every hour.
Publish the Runbook and let’s run the Runbook manually a couple of times to get some more data collected and send to OMS.
Check StockPrice data in OMS
Open your OMS workspace
Select OMS Portal to open the portal.
Go to the Log Search Solution to search for the StockPrice_CL Type entries.
Depending on the name you configured earlier for the LogType in the Assets you can search for StockPrice_CL. (OMS adds _CL to the name of your configure LogType variable)
For more info on the field name of the StockPrice data being send to OMS please check the documentation.
The final step in this part of the blog post series is configuring a Schedule for the StockPrice Runbook.
Runbook Schedule
Now we have the Runbook tested and published we want to run this runbook regarly, say every hour. For this to happen we need to add a Schedule to the StockPriceOMS Runbook in the Azure Automation Account Portal.
Go to Automation Accounts -> AutomationAccount -> Runbooks -> Runbook –> Settings and click on Schedules.
Select Add Schedule and create a new Schedule.
Enter the values to have the Schedule run every hour.
If you have configured the Schedule correctly the StockPrice Runbook will run every hours and StockPrice data will be send to OMS.
Watch this blog for the next blog post in this series of blog posts. In the next blog post we are going to create an OMS alert to get notified when a certain threshold is reached.
Comments
- Anonymous
December 13, 2016
when i run this runbook, the Runbook Output will be like above. But on OMS i will see only two fields: TimeGenerated and SourceSystem. In Field SourcSystem thee is the Value "RestAPI"- Anonymous
March 15, 2017
Did you manage to find the problem to this? I'm having the same issue with another script and can't figure out why it's not showing me all of the fields
- Anonymous