Condividi tramite

Using Kinect Interactions to Create a Slider Control

  • Articolo

In the 1.7 release, the Kinect for Windows Toolkit added the "Interactions Framework" which makes it easy to create Kinect-enabled applications in WPF that use buttons and grip scrolling.  What may not be obvious from the Toolkit samples is creating new controls for this framework is easy and straightforward.  To demonstrate this, I’m going to introduce a slider control that can be used with Kinect for Windows to “scrub” video or for other things like turning the volume up to eleven.

A solution containing the control code and a sample app is in the .zip file below.

Look Before You Leap

Before jumping right in and writing a brand new WPF control, it's good to see if other solutions will meet your needs.  Most WPF controls are designed to be look-less.  That is, everything about the visual appearance of the control is defined in XAML, as opposed to using C# code.  So if it's just the layout of things in the control, transitions, or animations you need to be different, changing the control template will likely suit your needs.  If you want the behavior of multiple controls combined into a reusable component then a UserControl may do what you want.

Kinect HandPointers

HandPointers are the abstraction that the Interactions Framework provides to tell the UI where the user's hands are and what state they are in.  In the WPF layer the API for HandPointers resembles the API for the mouse where possible.  Unlike the mouse, there is typically more than one hand pointer active at a time since more than one hand is visible by the Kinect sensor at a time.  In the controls that are in the toolkit (KinectCursorVisualizer, KinectTileButton, KinectScrollViewer, etc.) only the primary hand pointer of the primary user is used.  However, your control will still get events for all the other hand pointers.  As a result there is code in the event handlers to only respond to the primary user's primary hand.

KinectRegion Events

KinectRegion is the main component to look to when adding Kinect Interactions functionality to a WPF control.  All the WPF controls that are descendants of the KinectRegion will receive HandPointer* events as the HandPointers are used.  For example, when a hand pointer moves into the control's boundaries, the control will receive a KinectRegion.HandPointerEnter event.  If you've handled mouse events before, many of the KinectRegion events will feel familiar. 

KinectRegion events - https://msdn.microsoft.com/en-us/library/microsoft.kinect.toolkit.controls.kinectregion_events.aspx

Handling KinectRegion Events in the Slider

The slider control handles KinectRegion events to allow the user to grip and drag the thumb of the slider.  When a control "captures" a hand pointer it means that all the events of the captured hand pointer will be sent to that control until capture is released.  A general guideline for implementing control interactions is that a control should always capture hand pointer input events while the user is interacting with it otherwise it will miss many of the events it needs to function properly

The state diagram below gives the basic states of the control and what causes the state transitions.  The key thing to note is that the transitions in and out of dragging are caused by capture changing.  So that leads to the question, what causes capture to change?

The control takes capture when it gets a grip event.  That will put the control into the dragging state until capture is released.  Capture can be released for a number of reasons.  Most commonly it is released when the control receives a GripRelease event indicating the user opened their hand.  It can also be released if we lose track of the hand.  This can happen when the hand moves too far outside the bounds of the KinectRegion.

Expanding the Hit Area of the Control 

This control was originally designed to control video playback.  The design of the UI was such that we wanted to put the control at the bottom of the UI but allow the user to grab anywhere to move the playback position.  The way the slider does this is to allow the app to specify a different WPF UIElement that will attach hover and grip handlers.  See the KinectSlider.GripEventTarget property.  This uses WPFs ability to register event handlers on controls other than yourself.

Things Missing

While this control works and could actually be used in a real application, it is far from complete in a WPF sense.  It does not implement an automation peer so accessibility is limited.  While touch and keyboard usage may work a little, it is not fully supported.  Focus visuals, visuals for all the Slider permutations, and support for multiple themes are missing.

Resources for Building WPF Controls

Books and other resources we use to build controls include:

WPF 4 Unleashed https://www.informit.com/store/wpf-4-unleashed-9780672331190

WPF Control Development Unleashed - https://www.informit.com/store/wpf-control-development-unleashed-building-advanced-9780672330339

WPF source code - https://referencesource.microsoft.com/

Retemplating WPF controls - https://msdn.microsoft.com/en-us/magazine/cc163497.aspx

Comments

  • Anonymous
    July 08, 2013
    In the absence of better documentation on the new SDK 1.7, your KinectSlider code is arguably the best resource I have found that explains how to use the new gestures support in applications other than those built in controls.  One question:  you have defined this internal class KinectRegionBinder to handle changes in KinectRegion and KinectSensor, it is however not clear at all where this is actually used or called.  My apologies if I am missing something obvious here.  Many thanks again for your enlightening example.

  • Anonymous
    July 16, 2013
    it's best sample for beginners to learn slider control

  • Anonymous
    October 07, 2013
    @Super_lee:  thanks! @Drew:  i'm so sorry for such a long delay in replying to you.  thanks for the encouragement and glad you liked the sample.  to answer your question:  KinectRegionBinder should have been removed.  we started this project based on another sample and we forgot to delete this artifact.  sorry about the confusion!

  • Anonymous
    December 10, 2013
    The comment has been removed

  • Anonymous
    December 10, 2013
    It breaks right after executing Line 56 -60:        [SuppressMessage("Microsoft.Performance", "CA1810:InitializeReferenceTypeStaticFieldsInline", Justification = "Need to OverrideMetadata in the static constructor")]        static KinectSlider()        {            DefaultStyleKeyProperty.OverrideMetadata(typeof(KinectSlider), new FrameworkPropertyMetadata(typeof(KinectSlider)));        }

  • Anonymous
    March 11, 2014
    i am not getting "control code and a sample app" please send a link..thank you

  • Anonymous
    March 11, 2014
    I want "control code and a sample app" for slider interaction in real time please share it with me thank you or you may send to "sanghviharmesh@yahoo.com"

  • Anonymous
    April 06, 2014
    Hello, I cannot find the download link on the page. Can someone help me with that.

  • Anonymous
    August 06, 2014
    I want "control code"for slider interaction. Please share it with me. You may send to "engin_10@naver.com" Thank you

  • Anonymous
    August 07, 2014
    Hello, Kindly visit the public forum and post your questions. Engineers and developers are actively monitoring and will respond to your questions. http://aka.ms/k4wv2forum Thank you!

  • Anonymous
    August 07, 2014
    Gav, The download link for the public preview SDK 2.0 is www.microsoft.com/.../details.aspx Please let us know if that is not what you are looking for. Thank you!