“Latent Data” – the secret sauce of the Internet of Things

This is part 2 of a series of articles exploring the Internet of Things, starting with the first post, What is this “Internet of Things” thing, anyway?

The secret sauce, the Holy Grail, the raison d’ être for Internet of Things is data. That much is pretty obvious to anyone with more than a passing interest in the field – why would you go to the bother of deploying a load of sensing devices and the infrastructure to manage and communicate with them, unless the data they provide is particularly interesting?

At Microsoft, we work with lots of partner companies who use our technology to build their own products and solutions. This often puts us into contact with people and organisations who are doing things we’d never expected or even imagined they’d do, and that is one of the reasons why it’s such a great place to work and an amazing ecosystem to be part of.

As part of this working with companies that are beginning to inhabit this growing Internet of Things niche, a special interest group sitting in Microsoft UK has drawn a few interesting, and sometime controversial, observations:

• No one technology or technology provider will own the IoT, and a lot of systems will use a smorgasbord of standards and components
• Scaling a system that manages a few hundred gadgets to one dealing with hundreds of thousands of sensors is very hard, as is managing and analysing a massive quantity of data
• There are “stacks” within IoT

• • Sensors: the “things” in the IoT, massive in number but small in compute power
  • Hubs: the concentrators which harvest data from sensors, provide some degree of control, logic and processing and ultimately pass the information up the chain
  • Comms:many incompatible but functionally similar wireless standards will connect sensors to hubs, and hubs to the…
  • Cloud: the place where the data is brought back to, where analysis can take place on it and where insights can be passed on to other systems or even back to the devices

• The real value will come from “latent data”

What is “latent data”?

To a large degree, the IoT is an emerging set of technologies, protocols and patterns for the collection, aggregation, analysis and actioning of intrinsic, latent data, and the management of this process.

Data is ubiquitous and inherent is all environments, be it an outside space, an ecosystem, a manufacturing complex, a supply chain or a city. This data can be regarded as “latent data” or “potential data” in the physical world – the data exists but is not accessible, or if it is accessible then it is of limited use since it is not combined with other, relevant data (such as historical readings, or data from complementary systems) . Maybe the data is being accessed by some silo’ed system which uses that data for its own purposes but was never designed to provide any wider access to it.

Every physical thing has properties and attributes which may be discernable but are probably not being measured. A mechanical thermostat has intrinsic data on the temperature of a room and its own state, but this data remains in the physical world. A light bulb could be measured to see if it’s on or off, but this only becomes truly interesting when we could measure all the bulbs in a building, or a facility, or a city. If we can sense when all the bulbs need replacing, or alter their individual brightness depending on other conditions, that’s even more interesting.

For the avoidance of doubt, “Latent Data” is also a legal term applied to deleted files that need special forensic tools to extract… we’re talking about a more ethereal concept here, that there is data all around us in everything, but it’s untapped – and therefore, latent – unless we specifically decide to measure it and do something with it.

We believe that IoT is fundamentally about bringing this latent intrinsic data into the digital world in a way that allows the creation of value. This value is due to the aggregation of collected data, its analysis, and the use of that insight to drive decision-making and actions. The Cloud is the place where this data will be collected, where the data is likely to be stored in the long term, and where data aggregation and analysis will (mostly) occur.

The IoT patterns, the technologies and protocols that allow for this aggregation of latent data, are similar in a way to the OSI 7-layer networking model – the stacks which encompass devices, communications and Cloud. There are differing degrees of abstraction between these stacks and their constituent layers which means the IoT is inherently (and to the benefit of everyone within it and using it) a heterogeneous world.

Microsoft’s role in the Internet of Things

Microsoft has developed embedded systems that run in billions of devices already, and some of these could be considered part of an “intelligent system” that forms part of the IoT. Microsoft also has a hugely scalable and low cost cloud computing system in the Microsoft Azure cloud platform, where IoT applications can be quickly deployed and where the data that results from them can be securely kept and worked on.

Almost all IoT applications are likely to generate large volumes – petabytes, even more – of data, which will only become valuable when it is cost effective to keep it for a period of time and to perform large-scale computational analysis, both of which are difficult to do or economically unviable without the availability of public cloud computing.

If you’re an IoT developer building devices with Arduino or using systems like Raspberry Pi, and you’re writing your code in Python or Java and storing your data in some form of NoSQL database… that’s just fine by us. We think we have just the cloud service you need to let you concentrate on doing the stuff you started in this business for in the first place – writing your applications, building your devices and consuming your data.

With Microsoft Azure providing the backplane for these billions of devices to communicate – whether they are running Microsoft software or not – and to store and analyse their data, there is an opportunity for us and our partners to enable and monetize far-reaching change.

Some further reading:

• If you’ve not seen much around Azure before, see an overview here.
• For developer resources, including Node.JS, PHP, Python, Java and iOS & Android resources, see here.
• For the more technically minded - The Microsoft Azure Service Bus and the Internet of Things (part 1 and part 2).

Comments

• Anonymous
  June 23, 2014
  Working Lunch series - includes forthcoming sessions and previous sessions on demandhttp://www.microsoft
• Anonymous
  December 31, 2014
  As buzzwords go, 2014 has been a big year for the “ Internet of Things ”, or IoT . A term coined 15 years