Share via

Natural Interaction and Microsoft BI

  • Article

Today, Microsoft released the official Kinect for Windows SDK. Here is something we have been experimenting with here in the SQL Server BI product group that you might find interesting.

A while ago, at SQL PASS Summit 2011 in Seattle, WA, I demoed turning any random surface or wall into a touch screen using a single Microsoft Kinect device. I also demoed controlling Microsoft’s latest BI reporting tool which provides easy data exploration, visualization and presentation capabilities, Power View, through natural language and simple gestures.

Let’s start with a video of the above scenarios (You hear me say the word "Crescent" in the video. Crescent is the code-name for Power View):
[View:https://www.youtube.com/watch?v=ShyNV9hHPpo&feature=youtu.be]

Using Kinect to turn any wall into a touch screen

Using the Skeletal Tracking feature of the official Kinect for Windows SDK, it is possible to track the position of “joints” such as arms and hands. The data points provided for each joint include depth. It is, therefore possible to gather enough calibration data to correctly map the position of the hand to screen coordinates. It is also possible to calculate how far the projection surface area is from the Kinect allowing for the detection of the touch gesture: 

 

The part that is not demoed in the above video is the calibration stage. Currently it takes about a couple of minutes for one to calibrate the system by touching the wall at different locations. One could imagine advancing the algorithm to remove the need for that level of calibration.

Using Kinect and Microsoft Speech to control Power View

In the second section of the video, you saw some really interesting ways of controlling Power View through voice. As you saw, we can navigate between the views, sort different data regions, and filter the selected view based on the attributes in the semantic model.

The voice is picked up by the microphone array built into Kinect, and then passed onto Microsoft Speech for recognition. The rest is pretty straight forward: voice commands are converted to commands within Power View.

Pedram Rezaei

Comments

  • Anonymous
    February 01, 2012
    This is awesome!!

  • Anonymous
    February 01, 2012
    The comment has been removed

  • Anonymous
    February 01, 2012
    Sweeeeet :)

  • Anonymous
    February 01, 2012
    Reposting from Pedram Rezaei’s blog: “Today, Microsoft released the official Kinect for Windows SDK

  • Anonymous
    February 01, 2012
    That's a great demo providing the strong integration of MS products and technologies

  • Anonymous
    February 02, 2012
    Hey Pedram, looks fantastic. You accent seems to have travelled :) (Or does the Kinect require you to speak American?)

  • Anonymous
    February 03, 2012
    Rabib, this is very much a prototype. We however tested this in front a large audiance and it worked just fine. James, Interesting, I should listen to it again.

  • Anonymous
    February 04, 2012
    You could use kinect lasers to determ where the wall is. You could also use the projector to project a QR code and the Kinects camera. Can you put up your project on CodePlex and share it with the rest of us?

  • Anonymous
    February 05, 2012
    @Rabih, as a "human", I struggle in those kind of environments!  If Kinect can do it, that would be a huge improvement :)

  • Anonymous
    February 06, 2012
    Ok, we shipped ! Kinect for Windows SDK 1.0 is now available to all the Windows developers to change

  • Anonymous
    February 10, 2012
    @Fredrik: Unfortunately the code for this cannot be shared at this time.

  • Anonymous
    February 14, 2012
    People have been using Open Kinect to do variety of things including playing instruments. .this probably too late :) Youtube has the proof..

  • Anonymous
    February 16, 2013
    Any plans to release this to CodePlex?

  • Anonymous
    February 17, 2013
    @Essam, unfortunately the code for this cannot be shared at this time.

  • Anonymous
    May 07, 2014
    Hi Pedram, over a year later, the source code would be very helpful now that PowerBI is officially released. Any thoughts on posting the code to kinectforwindows.codeplex.com ?

  • Anonymous
    December 10, 2014
    Hi, any possible for sharing the code? looking forward. Jasonren2005@hotmail.com