Using Cortana Intelligence in HoloLens Applications
This post is authored by Scott Haynie, Senior Software Engineer, and Senja Filipi, Software Engineer, at Microsoft.
Telemetry plays an important role when you operationalize new experiences/apps that span the web, mobile and IoT, including new gadgets such as the Microsoft HoloLens. The abundance of data that is made available can help developers monitor and track system health and usage patterns, and provide important new insights into how users interact with your application. Tapping into this wealth of information can really help you align your customers' experiences with their needs and expectations.
At the Ignite 2016 Innovation Keynote, we showed the future of home improvement as envisioned by Lowe's and Microsoft. As part of this experience, we showed how to use telemetry emitted by HoloLens to understand the [virtual] places that customers are gazing at, as they experience an immersive home remodeling experience in 3D, using the augmented reality capabilities of HoloLens.
Kitchen remodeling experience with HoloLens.
Heat map based on HoloLens telemetry data.
In this post, we show how you can use the Cortana Intelligence Suite to ingest data from the HoloLens application, analyze it in real-time using Azure Stream Analytics, and visualize it with Power BI.
Empowering HoloLens Developers with Cortana Intelligence
In HoloLens applications, users can interact through one of these methods:
- By gazing at an object.
- By using hand gestures.
- By using voice commands.
When wearing, and interacting with, the HoloLens, a frontal camera tracks head movements, including the "head ray" and focus. The point that is being looked at is reflected using a cursor as a visual cue. These interactions are handled by the Update method (part of the MonoBehavior class). This method gets called at the refresh rate frequency, usually 60 frames per second. It's crucial to not slow down the update process with any side operations.
We set out with a simple goal that HoloLens developers should be able to use the Cortana Intelligence Suite and any related Azure services with just a few lines of code. In this example, you can see the code in one of the Update methods that tracks the gaze of the HoloLens user. To use Cortana Intelligence, the HoloLens application needs to just add this one line:
logger.AddGazeEvent(FocusedObject.gameObject.name)
The HoloLens is now able to send telemetry data to Event Hub, and you can further analyze the data using various Azure Services, for instance, Azure Stream Analytics.
Calling the telemetry AddGazeEvent from the HoloLens app.
Building the End to End Solution
A canonical telemetry processing pipeline consists of the following:
- An Event Hub that enables the ingestion of data from the HoloLens client application.
- A Stream Analytics job that consumes the telemetry data, analyzes it real time, and writes the insights derived into Power BI, as an output.
- A PowerBI dashboard.
If you don't have an Azure account yet, you can obtain a free account which will help you follow along in this post. Azure provides a rich set of libraries that developers can use to interact its many services. When using these libraries with a HoloLens application (which is a Universal Windows App), you will not be able to use NuGet packages directly if they rely on the .NET full framework implementation. Additionally, Azure services provide REST interfaces for client communication.
In our case, to send telemetry data from the HoloLens to the Event Hub, we implemented a .NET library, using the core .NET framework to meet the UWP apps requirements. The DLL handles the batching of events and composing of the request payload, and takes care of network retries.
Initializing the Telemetry Library
Step-by-step instructions on how to set up the Azure services and send data to the Azure Event Hub with retries from the UWP app can be found in GitHub here. Additional resources that you may find useful are included below. We would love to hear from you, so do let us know what you think – you can send your feedback via the comments section below.
Scott & Senja