Natural Interaction and Microsoft BI

Today, Microsoft released the official Kinect for Windows SDK. Here is something we have been experimenting with here in the SQL Server BI product group that you might find interesting.

A while ago, at SQL PASS Summit 2011 in Seattle, WA, I demoed turning any random surface or wall into a touch screen using a single Microsoft Kinect device. I also demoed controlling Microsoft’s latest BI reporting tool which provides easy data exploration, visualization and presentation capabilities, Power View, through natural language and simple gestures.

Let’s start with a video of the above scenarios (You hear me say the word "Crescent" in the video. Crescent is the code-name for Power View):
[View:https://www.youtube.com/watch?v=ShyNV9hHPpo&feature=youtu.be]

Using Kinect to turn any wall into a touch screen

Using the Skeletal Tracking feature of the official Kinect for Windows SDK, it is possible to track the position of “joints” such as arms and hands. The data points provided for each joint include depth. It is, therefore possible to gather enough calibration data to correctly map the position of the hand to screen coordinates. It is also possible to calculate how far the projection surface area is from the Kinect allowing for the detection of the touch gesture: 

 

The part that is not demoed in the above video is the calibration stage. Currently it takes about a couple of minutes for one to calibrate the system by touching the wall at different locations. One could imagine advancing the algorithm to remove the need for that level of calibration.

Using Kinect and Microsoft Speech to control Power View

In the second section of the video, you saw some really interesting ways of controlling Power View through voice. As you saw, we can navigate between the views, sort different data regions, and filter the selected view based on the attributes in the semantic model.

The voice is picked up by the microphone array built into Kinect, and then passed onto Microsoft Speech for recognition. The rest is pretty straight forward: voice commands are converted to commands within Power View.

Pedram Rezaei