Flying with IoT & the Cognitive Services
The past May 18th we did one of the biggest IoT events in Spain: the IoT Connect 2017. It happened in our Madrid offices. We did it in collaboration with Intel and other important partners. We could see how many companies are innovating with this new technology wave we call IoT.
We talked about cloud, blockchain, artificial intelligence, security and all what you can do when you connect your things, focusing in automating anything you can automate. To show what you can do, we built in a few days a complete solution to demonstrate how far we can go. This time we had a pipe, a drone and all the power of the Azure Services to make our project alive.
Remote pipe maintenance and something more...
The scenario we planned was a remote maintenance system for pipes equipped with a sensor system. With the gathered information from the sensors, we can create a remote monitoring system for the pipes, but we step into a set of additional problems that we would like to answer:
- When a pipe has an anomaly, a technician must physically go to see what's actually happening
- Sometimes we will get false positives, mainly because of sensor failures
- The sensor information is not enough to know the scope of the problem, this difficulties the work of the technicians
- Work orders have to be manually created
- The digital tools we have for the technicians are not natural for them
So, what we thought was a bolder and funnier solution proposal: we wanted to create a system that would detect and evaluate a pipe failure, and once the anomaly is detected, it sends a drone that will fly automatically to the pipe, takes a picture of it and sends it to Azure. Then awork order will be generated automatically, and the technician will use a bot to ask for what to do and how to do it.
Pipe simulator
To test all the system we didn't have a real pipe system, so our colleague Daniel Tabuenca built a pipe simulation system with Node-RED to send all the telemetry to Azure IoT Hub. Node-RED is a flow based programming system, that specializes in IoT. You can install nodes to connect to our Azure IoT Hub services.
You can install Node-RED in any device, and you can also deploy in an Azure Web App with just one click. Using this system we also built a user interface with the Dashboard model, so we could show the simulated pipes' state:
For the live demo, we also used an ESP8266 module with a distance sensor, so we had a physical device that gave us the water level of the pipe and send it to IoT Hub through a WiFi connection.
Drone controller
Once all the info arrives to IoT Hub, we analyze it with Stream Analytics and an anomaly detection function inside the Azure Machine Learning Studio. This generates an alert that is retrieved by an Azure Function to send a message to the drone controller software. The drone is also registered as an IoT Hub device, so we can easily send remote messages to it. Furthermore, the drone can also send its telemetry and pictures of the broken pipes in a secure and reliable way.
For controlling the drone indoor we cannot use a global positioning system, so we built the auto-pilot with an artificial vision library to recognize patterns and drive the drone towards a predefined form and color, in this case an orange pipe. To develop the controller application we used the OpenCV libraries so we can analyze the drone image stream in real time. OpenCV can be used in C# thanks to the EmguCV wrapper. The drone approaches the pipe and when arrives to a determined distance, it takes a picture and sends it to IoT Hub, then we can process the picture with Cognitive Services to better identify the breakdown.
- AR.Drone, we have a C# library to remotely control the drone. You can compile it in .NET and Mono.
- OpenCV is an open source library, initially developed by Intel. You have several wrappers to use it in .NET, I used EmguCV.
- Microsoft Cognitive Services is a set of machine learning models that are already prepared to be consumed with a REST API.
With the information generated by the pipe and the picture taken by the drone, a work order is generated in Dynamics 365. Then, the technician will have a conversation with a bot, built with the Microsoft Bot Framework, to get the list of pending orders. Besides getting the pending issues and review the generated information, the technician can ask the bot for other matters, like finding similar repairs so he can prepare a list of materials before getting out of the workshop:
As I commented before, to make all this work we need to connect the dots. The IoT Hub real time information must be processed by the AzureML model, send a notification to the drone, create a work order and so on. For automating these tasks we will use some of our serverless services we find in Azure:
- Azure Stream Analytics does the real time analytics task as easy as writing a SQL sentence. Allows us to link with an Azure Machine Learning function, so we can apply a trained model to the gathered data coming from IoT Hub, and send the results back to an Event Hubs queue so we can continue managing the data in real time.
- Azure Functions, using the Event Hubs as a data source, we deploy our code in pay for what you use mode. It is only executed when a new event arrives, and depending on the content we send a message to IoT Hub for the drone.
- Logic Apps, connected to Event Hubs as well, whenever we have gathered all the information, we send the data to Dynamics 365 without having to develop anything nor maintaining a VM.
Technologies
You can find the demo source code in GitHub. We have used these technologies:
- Simulation:
- Artificial vision:
- Local: EmguCV
- Cloud: Cognitive Services
- Bi-directional communication and telemetry:
- Integration
- Anomaly detection
Have a happy flight