Monitoring your home IP security cameras with OMS Log Analytics

Summary: Monitor devices, like home IP security cameras, with OMS Log Analytics without installing an agent!

Hi folks, In this blog post I would like to share with you how you can monitor devices with Log Analytics without the need to install the OMS (MMA) agent.

We’ve recently announced the Log Analytics HTTP Data Collector API. This enables a number of scenarios for which you may not have considered OMS Log Analytics previously. Especially in an environment where you cannot install the OMS agent or when the device does not run a supported OS version or distro like Windows or Linux. Well, let me show you the following.

A solution typically starts with a problem, so let’s dive into my problem first.

I have a home security system and part of that system are a couple of Foscam IP security camera’s and a Foscam NVR (Network Video Recorder) which records the camera feeds. The outdoor camera’s are at my front door and my carport, which leads to my backdoor. The camera’s are connected to the NVR and are configured in such a way that upon motion detection, they start a recording and optionally I can configure alert actions which typically would result in sending an email with a snapshot taken from the camera feed.

All goodness at this point. The problem or challenge if you will, is that I want to know if someone is doing some reconnaissance around my house in such a way that a person is detected by my front door camera AND by my carport camera WITHIN a specific interval of minutes, let’s say 5 minutes. I can visually see this correlation in my NVR when I play recorded streams, like this:

 

These systems (cameras and NVR) were never designed though to be able to perform any form of correlation and certainly not to do any data analytics. Did I say analytics?  Yes I did. So whenever I hear analytics, OMS Log Analytics obviously comes to mind. But to use Log Analytics in this scenario, I need to have the data available in Log Analytics first. Now with the ingestion API you can! Let’s break this project up for a second.

Step 1 – Does my camera log events and puts it in some kind of log file?

Well it does, but it’s mainly focused at the web UI and there’s no export capability to be found in the UI , but hey, I have logged events!

Step 2 – Is there any form of automation possible to retrieve the logs?

After thorough research (through Bing that is) I’ve discovered that there’s a limited SDK available which supports CGI requests in the form of POST and GET commands. Well that’s a start. It turned out that I can get the logs, but they are limited in the number of rows returned (10 max) and they come in this format:

Fast forward in time and more coffee….it appears that the datetime field and source IP address are notated in UNIX time and in decimal notation. Nothing that PowerShell can’t handle , let’s move on to step 3.

Step 3 – Create PowerShell automation scripts – am I hearing Azure Automation here?

So the fun part begins:

1. Creating PowerShell snippets to get the logs through CGI requests, using Invoke-WebRequest,  and put the result in an array – done

2. Utilize existing PowerShell functions to convert Unix time and the IP address – done

3. Update the array so I end up with DateTime, UserName, Source IP address and Camera EventType – done

4. Create a table with custom log field names, based on the array from step 1, and send it to the Log Analytics ingestion API – done

5. Test-drive the ingestion process – done

 

Step 4 – Using Log Search to query my camera data

Now that I have the log data from my outdoor camera’s sent to OMS Log Analytics, we can explore the data through Log Search:

Nice!

Step 5 – Correlating the camera log data

Time to take an outside walk to get some sun and wave at my two outdoors cameras. The cameras have done their job, they’ve detected me and have streamed the video feed to my NVR which has recorded my movements.  Let’s see if I can correlate this in Log Search:

Great! Let’s turn that into an alert with a schedule:

 

And while we’re there, let’s add an Azure Automation (webhook enabled) runbook which will send a text message (leveraging the Twilio text service):

Let’s test the webhook…. Ok, that works:

And we’re done!

Let’s test drive the solution. Again getting some sun, waving at my camera’s, sending the data to Log Analytics and….here we go:

Awesome! Peace of mind accomplished.

Now I can add my PowerShell script to Azure Automation, assign variables through assets, add it to a schedule and execute it on a Hybrid Runbook Worker, which has connectivity to my camera’s on my internal network.

 

So what’s next?

Besides creating visualization with the View Designer…

For the next project I’ve noticed that in some rare occasion I did not have the complete camera recording, what is going on with that? Exploring the logs of the NVR I saw this:

That sounds like a good use case to send the NVR logs to Log Analytics too for analyzing and correlating the data. I can leverage the approach followed previously which allows me to search through the NVR data as well:

Since the camera’s, but also the NVR are IP based, I wanted to be able to troubleshoot if there are some kind of connectivity issues going on between my camera’s, NVR and my home router. So I’ve decided to leverage and enable Syslog forwarding on my router. That was easy, since on my Asus Router I luckily have this:

The Remote Log Server destination IP address is an Azure VM running Ubuntu, which has an OMS agent running, which forwards the data to Log Analytics:

 

With the router data in Log Analytics as well, I can now go ahead and start troubleshooting and correlating my “video lost” errors and hopefully find the root cause by searching for keywords like NVR or drop (for potentially dropped packets, etc.):

 

If you want to explore Syslog forwarding, you can go here: Configuring syslog collection from the OMS portal

I hope that you’ve enjoyed this blog post and have seen the power and possibilities of the Log Analytics HTTP Data Collector API.

Thanks,

Tiander.