From the MVPs: Introduction to the Internet of Things – From the Device to Microsoft Azure Cloud
This is the 52nd in our series of guest posts by Microsoft Most Valued Professionals (MVPs). You can click the “MVPs” tag in the right column of our blog to see all the articles.
Since the early 1990s, Microsoft has recognized technology champions around the world with the MVP Award . MVPs freely share their knowledge, real-world experience, and impartial and objective feedback to help people enhance the way they use technology. Of the millions of individuals who participate in technology communities, around 4,000 are recognized as Microsoft MVPs. You can read more original MVP-authored content on the Microsoft MVP Award Program Blog .
This post is by Visual C++ MVP Alon Fliess . Thanks, Alon!
Introduction to the Internet of Things – From the Device to Microsoft Azure Cloud
Technology advances in “Buzzwords” steps. At the beginning, there is the basic technology. It slowly evolves, and then after a few years, sometimes even many years, everything becomes connected and the world, not the early bird world, but everybody is ready to embrace that technology. This is where the big buzz begins and everybody predicts that in five to ten years the technology will generate incremental revenue exceeding hundreds of billions. With this buzz, all major companies invest in the technology and we begin seeing TV news reports and economy magazine articles about the technology, telling that the everyday life of every human being on the planet is going to change because of that technology!
Of course, I am a bit cynical, but this is exactly what is happening now with regards to the IoT – the Internet of things. The basic technology is already here for almost a decade so far. Amazon Web Services (ASW) started in 2006. Microsoft Azure is 5 years old. Devices such as those based on the Amtel AVR controller are more than 20 years old, and the affordable Arduino family of devices used by IoT hobbyists are 10 years old. It is not (just) the technology that makes IoT what it is, but the concepts, the perception, the commitment and the challenges that the entire industry is dealing with nowadays. IoT is about the machine-to-machine (M2M) communication at scale. Vast numbers of devices using different hardware and software technologies are connected between them and to the cloud. The cloud provides many services, which can handle huge streams of data, analyze and can extract vital information about the current state of the system and can even predict future state.
So what exactly is IoT?
In one simple form, IoT is about a device that can monitor a physical character of the environment and transfer this data over the Internet to a cloud service. In a more complex form, IoT is the combination of many smart devices that can “feel” the environment, read the data, and transfer this information to a collector service in the cloud. This service has to deal with large amounts of devices and huge streams of data. Such services take in information and can extract a vital information from a live stream, or run algorithms such as those based on big-data map-reduce patterns. Services can act on historical and new data, or can provide future insight about the collected data. Services like Azure Machine Learning can use the collected data to predict future behaviors. The cloud can send commands to devices. Take for an example a system that starts the water sprinklers according to an algorithm that reads information from a group of soil moisture level sensors. Based on information that it gets from a forecast service that predicts that no rain is coming, it decides to send a command to an actuator that starts the sprinklers.
Sometimes the end device has the capabilities to communicate directly with the cloud service, and sometimes the device is cheap, weak, and has to conserve power, or has no encryption capabilities. In the latter case, a group of such weak devices is connected to a local gateway – a software that runs on a stronger local device that serves as a mediator between the local device sub-network and the cloud.
To illustrate, let us take a simple personal and fun project – which I have only recently turned into an IoT projectJ). With this project, I can control my home electrical devices such as lights, shutters, hot water boiler, and the air condition. I can use a web browser, Windows Phone or Windows 8.1 Modern application to send a command or read the current state of a device. I have other services that turn on the garden lights on sunset, or raise the shutters automatically every morning except on weekend days. I even have a service that sets the boiler according to the forecast.
My own electrical network is based on a well-known protocol named Instabus. This protocol is based on transmitting messages over a two-wire low-voltage command network. To communicate with the phone, or other devices and applications, I have built my own local gateway service that translates my REST based protocol to Instabus commands and vice versa. Whenever I use my Windows Phone to turn on the light, the phone application calls the REST based local gateway that extracts the command and transfers it to the local Instabus. The bus in turn publishes the command in its own protocol making an actuator that is the end receiver of the command to close the electrical circuit and turn on the light.
An IoT System
Every device that has a way to communicate can take part in an IoT system. A modern IoT system is built from at least one end device and one service that runs over the internet, however most large IoT systems include one or more of the following elements:
1. Many different devices with sensors and actuators
2. Local gateways
3. A collection of cloud services that enables:
a. Registration of devices
b. Management of devices
c. Different communication protocols that provides reliability and security
d. The ability to collect a vast amount of data at a very high input rate
e. The ability to analyze the stream of information in close to real-time manner
f. The ability to analyze the current and historical collected information
g. The ability to show the resulting conclusion and the collected data
In the rest of the article, we will see the two sides of an IoT system. We will start with the device and we will continue with the cloud services focusing on the rich possibilities that Microsoft Azure provides such a system.
The Devices:
Any device that can transmit and receive information is a candidate for an IoT system. Usually these devices are based on a System on a Chip (SoC) – an integrated circuit that has all the components of a computer on a single chip. Many hardware companies build new versions of these chips to be the heart of an IoT device. Some of the devices have an 8 bit based processors, others are based on 32-bit architecture and have the power of a small PC. For example the Intel Galileo 2 that is based on the Quark™ - a Pentium based architecture.
Hobbyists, or professionals that need a prototype device, in most cases, use a device from one of these families:
1. Arduino
2. Netduino
3. Raspberry Pi
4. Intel Galileo & Edison
5. Tessel
6. Spark Photon, Core and Electron
All of these device families have the ability to get information from sensors using digital or analog inputs, have the power to run custom algorithms and they can transmit and receive data using one or more means of communications.
The Arduino family is one of the older existing device families. It serves as a simple and cheap hardware to develop simple electronic devices. The Arduino is an open source hardware and software; anyone can build their own Arduino device. You can find Arduino compatible boards that cost about $1.50. These small devices have no internet capabilities; however, they can communicate using RS232 (Serial COM Port) to a local gateway, or even use a very inexpensive Wi-Fi device such the ESP8266. In case that there is a need for a more powerful and expensive device, which can also serve as a local gateway; the Arduino family has the Arduino Yun board. The Yun is a hybrid of two systems on one board. The first system is the Arduino. The second system is based on OpenWRT – a Linux distribution for embedded devices. The two systems communicate using a bridge library. The Yun has a Wi-Fi and Ethernet capabilities and it can serve as a gateway or as a client and communicate directly with the cloud. Usually, to develop for Arduino, you use the simple dedicated Arduino Editor that lets you write simple C/C++ code, however there is a Visual Studio Add-on that lets you develop and debug Arduino programs from Visual Studio. For the Linux side of Arduino Yun you can use C/C++ with GCC, Python, JavaScript with Node.JS or any other language that has a Linux compiler or interpreter for OpenWRT. The Yun is an IoT ready Arduino device.
Arduino Boards
The Netduino family is another example of open source hardware and software. This family is hardware compatible with the Arduino sensors and shields – Arduino add-on hardware boards. The Netduino software is based on the .NET Micro framework, a tiny .NET version that lets you develop for the device using C# in Visual Studio!
To use the latest versions of Visual Studio and .NET Micro framework, you might need to flash a new firmware. The Netduino plus 2 has an Ethernet on board and it can use a Wi-Fi shield such as the SparkFun WiFly to become an even better IoT device.
The Raspberry Pi is more like a small form factor PC than a hardware device; it can be connected to a monitor or TV using analog output or a HDMI cable. It has USB connectors that can be used to connect mouse, keyboard and even a Wi-Fi dongle. Raspberry Pi runs a special Linux distribution called Raspbian. Microsoft has announced that a special version of Windows 10 for IoT devices will support the Raspberry Pi 2.
Developing for the Pi is quite easy. You can use the built-in Python or Scratch, or you can install any other language compiler including .NET Mono. If you like to use C/C++ and port Arduino code to the Pi, you can use the WiringPi library with GCC.
Like the Arduino and Netduino, the Raspberry Pi can be connected to devices through its GPIO pins. The raspberry uses 3.3 Volt sensors and connecting it poorly or using the wrong voltage may render it inoperable. As opposed to Arduino, the Raspberry Pi has no analog inputs, which in many cases are important for reading a level value from a simple sensor, such as temperature or ambient light level. You can connect an Analog to Digital chip that has a Serial Peripheral Interface Bus (SPI) channel to overcome this problem, or use a combination of inexpensive 3.3V Arduino with the Pi. If you are concerned about your Pi becoming inoperable, you can use boards like the Pibrella that makes it safe to play with it.
Raspberry Pi 2 – Will run Windows 10
There are other Raspberry Pi compatible but more powerful devices such as the Banana-Pi and the Orange-Pi. The Orange Pi has a built-in Gigabit Ethernet and Wi-Fi, which make it a good IoT (gateway) device.
An Orange Pi Intel supplies both hardware and software for IoT. The Intel Galileo 2 board is a relatively robust machine that runs Linux or the new Windows IoT (still in alpha version). The Galileo is compatible with Arduino shield and software and you can use Visual Studio C++ and sub-set of the Win 32 API. With these C++ and Win 32 capabilities, you can use the C++ REST SDK and Azure C++ Storage API with the Galileo to communicate with Microsoft cloud.
Intel Galileo Gen 2 on Clay’s Breadboard kit – Windows IoT is installed on the micro-sd card.
There is also an Arduino Development Environment for the Galileo. Intel has another smaller IoT device – the Intel Edison. Though I haven’t had the chance to play with it yet, according to specification it appears to be a great IoT capable device with dual-core CPU, integrated Wi-Fi, Bluetooth, 40 GPIO interfaces and libraries to develop using C, C++, Python and JavaScript.
The Tessel is a nice device for beginners and for people that have no electronics knowledge or as the Tessel people describe it – “Hardware development for software developers”. A tiny board that has four connecters on the side for connecting a dedicated set of modules - sensor boards. The Tessel has built-in Wi-Fi and it runs JavaScript. The Tessel is a very easy to use IoT device.
The Tessel
Last but not least in the devices part is a new and very promising device family – the Spark. The family currently has two devices and one in development: the Core – the first Wi-Fi enabled device, the Photon – a relatively inexpensive tiny Wi-Fi capable device and the Electron – A Kickstarter project, an IoT device that connects the Internet via cellular network.
Spark Photon
The common dominator of all above devices is that they have the capability to bring the physical environmental information to the Internet or execute commands. They do that via a set of input and output pins. In many cases, each pin can serve as an input or an output based on its settings. Some of the pins serve as communication protocol connectors such as RS-232, I2C and SPI that were mentioned earlier. Other pins can be used to control motors and servos with Pulse-Width-Modulation (PWM). Sensors are connected to these input/output pins. Sensors provide information such as temperature, moisture, water level, rain, wind speed, weight, motion detection, sonar – range detector, infra-red transmitter and receiver, microphone, camera, Serial over Wi-Fi, Bluetooth, RFID reader/writer and so forth.
In the following example, each time a motion sensor connected to input pin 12 detects a movement, it lights a led:
Smart Refrigerator Scenario
We would like to have a set of devices and a backend system that transforms our refrigerator to an IoT system. The system can tell us that we are running out of milk, it can tell our retail store that they need to deliver a new milk bottle, and with the power of the cloud it can tell the retail chain headquarter the trends of using their goods and predict the near future supply needs. This knowledge provides real business value. There are many different ways to make a smart-refrigerator – for example, each product can come in a smart package that knows the product details such as the expiration date, the price, the nutrition and the current amount that is in the box. For our humble example, we are going to use a device that can weigh the milk bottle; we will convert the result to the current amount that is left in the box. We can set a collection of such devices in the refrigerator for different products.
A prototype setup looks like this:
Once everything works using the breadboard, we can use a prototype shield:
This circuit can operate using a battery or a strong enough power supply. Using the ESP 8266 with the simple Arduino requires going through a local gateway, since most of the cloud services that deal with a stream of events require the use of standard protocols and the use of security layer which only powerful devices such as the Intel Galileo or the Raspberry PI. The gateway runs a simple software that bridges the local device network with the cloud. The gateway can handle many local client connections.
To read the weight, we use the Weight sensor through a HX711 ADC device. The HX711 reads the very small voltage changes that occur when we put a weight on the scale and transfer this information as a stream of bits. A special library lets you read the stream of bits coming from the HX711. The library parses the raw bits data and converts it to a usable floating-point number value. You can download the code for the Arduino from here, or if you like to play with Netduino – I have ported the library to C#.
The Arduino code:
1 #include "HX711.h"
2 #include <SoftwareSerial.h>
3
4 #define SSID "NOKIA Lumia Alon"
5 #define PASS "AF123456"
6 #define DST_IP "192.168.137.36"
7
8 int ledPin = 2; // select the pin for the LED
9 HX711 scale(9, 8); // parameter "gain" is ommited; the default value 128 is used by the library
10
11 SoftwareSerial mySerial(7, 6); // RX, TX
12
13 void blink(int times, bool isGood = true)
14 {
15 int d = isGood ? 100 : 300;
16 for (int i = 0; i < times; ++i)
17 {
18 digitalWrite(ledPin, HIGH);
19 delay(d);
20 digitalWrite(ledPin, LOW);
21 delay(d);
22 }
23 }
24
25 void setup()
26 {
27 // Open serial communications and wait for port to open:
28 pinMode(ledPin, OUTPUT);
29 //scale.set_scale(2280.f); // this value is obtained by calibrating the scale with known weights; see the README for details
30 scale.set_scale(160.f); // this value is obtained by calibrating the scale with known weights; see the README for details
31 scale.tare(); // reset the scale to 0
32
33 mySerial.begin(9600);
34 mySerial.println("Ready");
35 blink(5);
36
37 Serial.begin(115200);
38
39 Serial.setTimeout(5000);
40
41 //test if the module is ready
42 Serial.println("AT+RST");
43 delay(3000);
44
45 //connect to the wifi
46 boolean connected=false;
47 for(int i=0;i<50;i++)
48 {
49 if(connectWiFi())
50 {
51 connected = true;
52 break;
53 }
54 blink(2, false);
55 }
56 if (!connected)
57 {
58 mySerial.println("Not connected");
59 blink(100, false);
60 while(1)
61 ;
62 }
63 delay(5000);
64 blink(5);
65 Serial.println("AT+CIPMUX=0");
66 mySerial.println("Connected");
67 }
68
69 void dumpSerial()
70 {
71 while (Serial.available())
72 {
73 char c = Serial.read();
74 mySerial.write(c);
75 if(c=='\r') mySerial.print('\n');
76 }
77 }
78
79 void loop()
80 {
81 scale.power_up();
82 double sensorValue = scale.get_units(10);
83 scale.power_down(); // put the ADC in sleep mode
84
85 String cmd = "AT+CIPSTART=\"TCP\",\"";
86 cmd += DST_IP;
87 cmd += "\",5001";
88 Serial.println(cmd);
89 //mySerial.println(cmd);
90 delay(250);
91 if(Serial.find("Error"))
92 {
93 blink(4, false);
94 mySerial.println("Error");
95 Serial.println("AT+RST");
96 delay(3000);
97 connectWiFi();
98 delay(3000);
99 return;
100 }
101
102 cmd = String("Weight:") + String(sensorValue) + ",Scale.Arduino\r\n";
103 //mySerial.println("");
104 Serial.print("AT+CIPSEND=");
105 Serial.println(cmd.length());
106 delay(300);
107 Serial.print(cmd);
108 delay(2000);
109 //mySerial.println(cmd);
110 dumpSerial();
111 delay(1000);
112 }
113
114 boolean connectWiFi()
115 {
116 Serial.println("AT+CWMODE=1");
117 String cmd="AT+CWJAP=\"";
118 cmd+=SSID;
119 cmd+="\",\"";
120 cmd+=PASS;
121 cmd+="\"";
122 Serial.println(cmd);
123 delay(3000);
124 if(!Serial.find("OK"))
125 {
126 blink(4);
127 return false;
128 }
129 //else, OK, read assigned IP for debugging
130 Serial.println("AT+CIPSTATUS");
131 delay(250);
132 dumpSerial();
133
134 blink(2, false);
135 delay(250);
136 return true;
137 }
The Netduino code:
1 using System;
2 using System.IO.Ports;
3 using System.Net;
4 using System.Net.Sockets;
5 using System.Threading;
6 using Microsoft.SPOT;
7 using Microsoft.SPOT.Hardware;
8 using SecretLabs.NETMF.Hardware;
9 using SecretLabs.NETMF.Hardware.Netduino;
10
11
12 namespace WiFiScale
13 {
14 public class Program
15 {
16 private const string SSID = "NOKIA Lumia Alon";
17 private const string Password = "AF123456";
18 private const string DestinationIp = "192.168.137.36";
19 private static readonly OutputPort LedPin = new OutputPort(Pins.GPIO_PIN_D2, false);
20 private static readonly SerialPort DebugSerial = new SerialPort(SerialPorts.COM4, 9600, Parity.None, 8,
21 StopBits.One);
22 private static readonly HX711 Scale = new HX711(Pins.GPIO_PIN_D9, Pins.GPIO_PIN_D8);
23 private static readonly SerialPort WiFiSerial = new SerialPort(SerialPorts.COM1, 115200, Parity.None, 8, StopBits.One);
24
25 public static void Main()
26 {
27 Scale.Scale = 160;
28 WiFiSerial.Open();
29 DebugSerial.Open();
30
31
32 // this value is obtained by calibrating the scale with known weights; see the README for details
33 Scale.Tare(); // reset the scale to 0
34 DebugSerial.Write("Ready");
35
36 Blink(5);
37 // write your code here
38
39 WiFiSerial.WriteLine("AT+RST");
40 Thread.Sleep(3000);
41
42 //connect to the WiFi
43 bool connected = false;
44
45 for (int i = 0; i < 50; i++)
46 {
47 if (ConnectWiFi())
48 {
49 connected = true;
50 break;
51 }
52 Blink(2, false);
53 }
54 if (!connected)
55 {
56 DebugSerial.WriteLine("Not connected");
57 while (true)
58 Blink(100, false);
59 }
60
61 Thread.Sleep(3000);
62 Blink(5);
63 WiFiSerial.WriteLine("AT+CIPMUX=0");
64 DebugSerial.WriteLine("Connected");
65 Thread.Sleep(2000);
66 while (true)
67 {
68 Loop();
69 }
70 }
71
72 private static void Blink(int times, bool isGood = true)
73 {
74 int d = isGood ? 100 : 300;
75 for (int i = 0; i < times; ++i)
76 {
77 LedPin.Write(true);
78 Thread.Sleep(d);
79 LedPin.Write(false);
80 Thread.Sleep(d);
81 }
82 }
83
84 private static void Loop()
85 {
86 Scale.PowerUp();
87 double sensorValue = Scale.GetUnits(10);
88 Scale.PowerDown(); // put the ADC in sleep mode
89
90 String cmd = "AT+CIPSTART=\"TCP\",\"";
91 cmd += DestinationIp;
92 cmd += "\",5001";
93 WiFiSerial.WriteLine(cmd);
94
95 Thread.Sleep(250);
96 if (WiFiSerial.Find("Error"))
97 {
98 Blink(4, false);
99 DebugSerial.WriteLine("Error");
100 WiFiSerial.WriteLine("AT+RST");
101 Thread.Sleep(3000);
102 ConnectWiFi();
103 Thread.Sleep(3000);
104 return;
105 }
106
107 cmd = "Weight:" + sensorValue + ",Scale.Net\r\n";
108
109 WiFiSerial.Write("AT+CIPSEND=");
110 WiFiSerial.WriteLine(cmd.Length);
111 Thread.Sleep(300);
112 WiFiSerial.Write(cmd);
113 Thread.Sleep(2000);
114 //DebugSerial.WriteLine(cmd);
115 DumpSerial();
116 Thread.Sleep(1000);
117 }
118
119 static void DumpSerial()
120 {
121 while (WiFiSerial.BytesToRead > 0)
122 {
123 var line = WiFiSerial.ReadLine();
124 DebugSerial.WriteLine(line);
125 }
126 }
127
128 private static bool ConnectWiFi()
129 {
130 WiFiSerial.WriteLine("AT+CWMODE=1");
131 String cmd = "AT+CWJAP=\"";
132 cmd += SSID;
133 cmd += "\",\"";
134 cmd += Password;
135 cmd += "\"";
136 WiFiSerial.WriteLine(cmd);
137 Thread.Sleep(3000);
138 if (!WiFiSerial.Find("OK"))
139 {
140 Blink(4);
141 return false;
142 }
143 //else, OK, read assigned IP for debugging
144 WiFiSerial.WriteLine("AT+CIPSTATUS");
145 Thread.Sleep(250);
146 DumpSerial();
147
148 Blink(2, false);
149 Thread.Sleep(250);
150 return true;
151 }
152 }
153 }
154
In this demo, the Server IP address and the Access Point SSID and password are all hardcoded. In a real world scenario, you can have a setting switch on the board that can command the device to move to setup mode. In setup mode, the ESP 8266 is set to an Access Point mode where a simple client running on a laptop, tablet or even Raspberry Pi can connect to it and set these parameters. You will also need a SD card as a drive and a card to store the information – The Netduino Plus 2 has a built-in one, and it is very easy to add such a device to the Arduino. Whenever the controller is turned on, it tests the setting-switch and decides whether to go to setup mode or read the IP address and other parameters from the SD card and set the ESP 8266 as a client. This is similar to the way you initialize an Arduino Yun board for the first time.
The Gateway and the Cloud:
Up until now, we have seen all about the device, but IoT is mainly about connecting devices to the cloud and handling the stream of data.
Microsoft Azure provides the required framework to build a large-scale IoT backend:
Devices and applications collect the data and create a stream of events. These devices may directly call the Event Hub or use a local gateway to do that. Azure Stream Analytics is a Complex Event Processing (CEP) service that can handle millions of events per second. It can transform the data to other representations, can filter, aggregate or search for a specific information. Using Azure Stream Analytics, we can extract vital information from the current data. We can also store the data and use Azure big data - HDInsight and Azure Machine Learning - to get a deeper analysis and even get a projection and decisions that can be used to send a command back to the device.
In a case that we need a system that can answer queries from the old and the new data, we can implement the Lambda Architecture (LA) – an architecture that aims to provide scalable and fault-tolerant data processing system for low latency queries. The basic idea behind this architecture is to use scalable services such as Azure Event Hubs or Apache Kafka, for handling the huge data stream. The data event stream is dispatched into two layers – a batch layer – for old data and a speed layer for recent, short, near real time data. In Azure, the batch layer is based on storing the information in one of the Azure storage options (Blob, Table, DocumentDB, Azure SQL DB) and the speed layer can be built with Stream Analytics or with Azure HDInsight Apache Storm. To implement the Lambda Architecture, you need to create an Azure service (Web-Site REST service for example) that for any incoming query, it distributes the query to both layers and then merges the query result.
I would like to show you a simple IoT system, from the device through a gateway to the cloud. Of course, I will not touch many IoT aspects such as device registration and management, but I will handle secure communication and build a scalable solution.
I already shared the code that runs on the Arduino and Netduino, that sends the weight information to the gateway. For the example, I choose to use the Raspberry PI as a gateway. I use Raspberry PI Model 2 with a Wi-Fi dongle, for which I developed the gateway code with Python. I chose Python since I found it very easy to use and talk with Azure Event Hub Service. There is a great blog post of Kloud that shows how to use Azure event Hub directly from the Arduino Yun – a device that can run C++ and Python. There are also many other options such as using Java Script with Node.JS or Using C++ with Apache QPid that communicate with the Azure Event Hub using the standard AMQP protocol.
The Python code is a multi-threaded TCP server that accepts connections from the local network. Once it has a new connection it spawns a thread that reads the stream of events coming from a simple device and sends the raw information to Azure Event Hub.
This is the Gateway code:
1 #!/user/bin/python
2 import sys
3 import azure
4 import socket
5 import SocketServer
6 import threading
7
8 from azure.servicebus import (
9 _service_bus_error_handler
10 )
11
12 from azure.servicebus.servicebusservice import (
13 ServiceBusService,
14 ServiceBusSASAuthentication
15 )
16
17 from azure.http import (
18 HTTPRequest,
19 HTTPError
20 )
21
22 from azure.http.httpclient import _HTTPClient
23
24 import time
25
26 class EventHubClient(object):
27
28 def sendMessage(self,body,partition):
29 eventHubHost = "aloniot.servicebus.windows.net"
30
31 httpclient = _HTTPClient(service_instance=self)
32
33 sasKeyName = "iotpolicy"
34 sasKeyValue = "l6TGnew90yy0Pw1u1VWTt0awHOezwEB315+BCeNfH8c="
35
36 authentication = ServiceBusSASAuthentication(sasKeyName,sasKeyValue)
37
38 request = HTTPRequest()
39 request.method = "POST"
40 request.host = eventHubHost
41 request.protocol_override = "https"
42 request.path = "/iot/publishers/" + partition + "/messages"
43 request.body = body
44
45 print(body)
46
47 request.headers.append(('Content-Type', 'application/atom+xml;type=entry;charset=utf-8'))
48
49 authentication.sign_request(request, httpclient)
50
51 request.headers.append(('Content-Length', str(len(request.body))))
52
53 status = 0
54
55 try:
56 resp = httpclient.perform_request(request)
57 status = resp.status
58 except HTTPError as ex:
59 status = ex.status
60
61 return status
62
63 class EventDataParser(object):
64
65 def getMessage(self,payload,sensorId):
66 host = socket.gethostname()
67 body = "{ \"Time\" : \"" + time.asctime(time.localtime(time.time())) + "\", \"GatewayId\" : \"" + sys.argv[1] + "\", \"DeviceId\" : \"" + host + "\","
68
69 sensorType = payload.split(":")[0]
70 sensorValue = payload.split(":")[1]
71 body += "\"SensorId\" : \"" + sensorId + "\", \"SensorType\" : \"" + sensorType + "\", \"SensorValue\" : \"" + sensorValue + "\"}"
72 return body
73
74
75 class IoTAzureGatewayTCPHandler(SocketServer.BaseRequestHandler):
76 """
77 The RequestHandler class for our server.
78
79 It is instantiated once per connection to the server, and must
80 override the handle() method to implement communication to the
81 client.
82 """
83
84 def handle(self):
85 hubClient = EventHubClient()
86 parser = EventDataParser()
87 hostname = socket.gethostname()
88
89 while True:
90 # self.request is the TCP socket connected to the client
91 self.data = self.request.recv(1024).strip()
92 if not self.data:
93 print("Nothing to read. Closing conncetion")
94 break
95 print("{} wrote:".format(self.client_address[0]))
96 print(self.data)
97
98 splitData = self.data.split(",")
99 sensor = splitData[1]
100 body = parser.getMessage(splitData[0],sensor)
101 hubStatus = hubClient.sendMessage(body,hostname)
102 # return the HTTP status to the caller
103 print(hubStatus)
104 # just send back
105 self.request.sendall(str(hubStatus))
106
107
108 class IoTAzureGatewayTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
109 pass
110
111 if __name__ == "__main__":
112 HOST, PORT = "0.0.0.0", 5001
113
114 server = IoTAzureGatewayTCPServer((HOST, PORT), IoTAzureGatewayTCPHandler)
115 ip, port = server.server_address
116
117 # Start a thread with the server -- that thread will then start one
118 # more thread for each request
119 server_thread = threading.Thread(target=server.serve_forever)
120 # Exit the server thread when the main thread terminates
121 server_thread.daemon = True
122 server_thread.start()
123 print("Server loop running in thread:", server_thread.name)
124
125 # this will keep running until you
126 # interrupt the program with Ctrl-C
127 while True:
128 pass
129 server.shutdown()
130
The result: 
This sums the device part of the story; we are sending telemetry information to Azure Event Hub. Each gateway can connect many local devices and we can deploy many gateway devices, for example, one gateway per refrigerator and one end device per product box or a place on the refrigerator shelf.
Azure Event Hub
Azure Event Hub is a highly scalable service that enables ingesting and processing millions of events per second. Queues and topics are great for building reliable and scalable enterprise solutions but event hubs are particularly suitable to the IoT pattern of scale. In any scenario that involves scalability – you must tackle the potential bottlenecks of the system. Queues, topics and event hubs handle message throughput differently. Queues synchronize producers and consumers – for example, queues provide a mechanism that hides a message once a consumer starts to handle it, a mechanism to delete it when the consumer finishes handling it, or the consumer can return the message to the queue if it could not process it to completion. Topics are similar, with the exception that multiple consumers can process a message. To enable improved scalability, low latency and performance, event hubs have a different structure and approach. Event hubs have many data structures similar to queues called partitions. When you create an event hub, you provide the number of partitions – currently up to 32, but you can request additional partitions if this is insufficient for your solution. Each message sent to the event hub can include a target partition but you can also use a key such as a device id and allow the event hub infrastructure to co-located messages from the same device id to the same destination partition. In case of the latter – the device id goes through a hash function that chooses the partition for its stream of data. This enables scalability and message order from a device source. To further increase the performance, event hubs do not delete events immediately on consumption. In fact they don’t track the consumer state at all. Instead, event hubs use the notion of consumer groups. There is at least one group – the default consumer group, however in many cases you would create more than one consumer group to scale out or add alternate processing behaviors. Each consumer group tracks the location of the reading head – to track the location of the next event. There is one tracking cursor per partition. Event hubs delete messages based on the time-to-live setting on the message. You can supply a time-to-live for a message from one to seven days. Since messages are not automatically removed upon consumption, you can create multiple consumer groups to read and process past messages – a handy feature that lets you analyze a range of messages.
Event Hub Schema
Azure Event Hub has a built-in throttling mechanism that influences the scalability of your system. To prevent throttling bottlenecks, the user can raise the Event Hub throughput units. Throughput units apply to all Event Hubs in a Service Bus namespace. Each throughput unit enables the Event Hub Azure service to handle up to 1 MB per second of events coming (ingress) to the service and 2 MB of events to be consumed by consumer groups (egress). Each throughput unit supports storage of up to 84 GB of event data. Throughput units are billed hourly, based on the maximum number of units selected during the given hour. Event hubs are associated with a Service Bus namespace, and you can set up to twenty throughput units per namespace. As with partitions, you can request additional units if needed. Throughput units are used to throttle message volumes which means if the quota is exceeded message senders and receivers (consumers) will receive errors.
Event Hub Settings
You can read event hub messages from cloud services such as Web Jobs or Worker Roles. You can also use Azure Stream Analytics (still in preview) to analyze the data near real-time. You may also send the event data to storage, or back to the event hub for further analysis.
Azure Stream Analytics
Azure Stream Analytics is a new service that can serve as a mediator between event hubs and other services. You can create a new stream analytics job to read raw data from a consumer group and store it as is to Azure Blob Storage for example. However, the main feature of Stream Analytics is to serve as a complex event processing service by reading the stream of events and run queries on the data. To run a SQL like query on an endless stream of data, you provide a window in time that splits the data stream into group of events; hence, Stream Analytics can execute the query on that group.
In your stream analytics query, you take a group of events and query for an aggregated result using one of the predefined operators such as AVERAGE, COUNT, MAX, MIN, SUM, etc. This is something like real-time map-reduce algorithm.
Stream analytics can deduce the type of the fields in the event message even if it comes as schema-less record such as in the case of JSON, however for some queries it is better to have a schema. For that, you can create a table that describes each field.
CREATE TABLE TollTagEntry (
TollId nvarchar(max),
EntryTime datetime,
LicensePlate nvarchar(max),
State nvarchar(max),
Make nvarchar(max),
Model nvarchar(max),
VehicleType nvarchar(max),
VehicleWeight nvarchar(max),
Toll float,
Tag nvarchar(max)
);
SELECT System.TimeStamp AS OutTime, TollId, SUM (Toll)
FROM Input TIMESTAMP BY EntryTime
GROUP BY TollId, TumblingWindow(minute,3)
In the example I’ve been describing, I have created two consumer groups: one that reads the raw data and stores it in blob storage and another one that takes the average weight of group of events. It might be even a good idea to take the minimum value of the weight in a time window of 1 hour if all we need to know is when we need to deliver a new milk bottle.
To read the table storage data or the blob storage raw data, you can create a cloud service such as a Web Job, or you can use the new Microsoft Power BI tool (still in preview)
Getting Data from Azure – Microsoft Power BI Tool
You can watch the complete demo video.
Summary
Internet of Things is much more than hype. You can leverage the technology to get vital information to drive your business. Companies are investing in IoT in a variety of ways. Some are building better IoT frameworks, protocols, services and systems. Others are looking at providing IoT services or looking at how they can use IoT systems to improve their current solution. A Security related companies will provide security for IoT devices and services, billing companies may evolve to provide solutions for using IoT devices and services, and retail companies – they will know when to deliver a new bottle of milk!
Thanks
I would like to thank Microsoft Azure MVP Michele Leroux Bustamante for technical reviewing and editing the article. I also would like to thank Aaron Etchin for spending the time discussing with me my ideas and the way to present them and for the patience of reviewing this and previous articles. I would like to thank the MVP team – Melissa Travers, Hande Kayadeniz Torkan and Lina Magdy for encouraging me to write and publish articles and last but not least to my wife Liat and my kids: Yarden, Saar and Adva.
About Alon Fliess
Alon Fliess is the Chief Architect & Founder of CodeValue. CodeValue is the home of software experts. We build software tools, foundations and products for the software industry. We Build OzCode – Your road to magical debugging!
We offer mentoring, consulting and project development services.
Alon got his BSc degree in electrical & computer engineering from the Technion, the Israel Institute of Technology. Alon is also recognized as a Microsoft Regional Director (MRD) an as a VC++ MVP. Alon is an expert in many Microsoft technologies, be it Windows internals, C++ Windows programming (Win32/WinRT), .Net with C# and Windows Azure Cloud Computing. Alon spends his time doing many interesting tasks such as software architecting, designing, mentoring and programming. Alon is the co-author of the "Introducing Windows 7 for Developers" book and technical reviewer of other books. Alon is an active member of several Patterns & Practices councils, among them Project Hilo – A Windows Store Application in C++/CX and XAML.
Alon is one of the experts in Microsoft Israel community; as such, he helps Microsoft clients in many technological aspects. Alon gives lectures in Microsoft Israeli & international conferences such as Tech-Ed.
More than all Alon loves the nights. This is the time that he enjoys sitting on the computer, learning and testing new technologies. Follow Alon on Twitter @ALON_FLIESS
Comments
Anonymous
April 27, 2015
Best Article on IOT, I ever read.Anonymous
December 13, 2015
thanks for sharing. Good summary. I assume you'd introduce some sort of queuing (e.g. to handle internet outages etc.) on your gateway as without it it would seem better to just write the local sensor data directly from the ardunino to a simple REST web service hosted in Azure.Anonymous
June 14, 2016
good idia