Tracking Users with Kinect Skeletal Tracking

Kinect for Windows 1.5, 1.6, 1.7, 1.8

• Enabling Skeletal Tracking
• Accessing Skeletal Tracking Information
• Skeleton Position and Tracking State
• Joint Positions and Tracking States
• Clipped Edges
• Active User Tracking
• Player ID in depth map

Enabling Skeletal Tracking

To track users, an application needs to enable skeletal tracking.

How to enable skeletal tracking in C#

To enable skeletal tracking, call the SkeletonStream.Enable method; you can access the KinectSensor.SkeletonStream property from the KinectSensor class. To receive information about the recognized users, subscribe to the KinectSensor.SkeletonFrameReady event or the KinectSensor.AllFramesReady event managed by the KinectSensor class.

  KinectSensor kinect = null;

  void StartKinectST()
  {
    kinect = KinectSensor.KinectSensors.FirstOrDefault(s => s.Status == KinectStatus.Connected); // Get first Kinect Sensor
    kinect.SkeletonStream.Enable(); // Enable skeletal tracking

    skeletonData = new Skeleton[kinect.SkeletonStream.FrameSkeletonArrayLength]; // Allocate ST data

    kinect.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(kinect_SkeletonFrameReady); // Get Ready for Skeleton Ready Events

    kinect.Start(); // Start Kinect sensor
  }
      

How to enable skeletal tracking in C++

To enable skeletal tracking, call the INuiSensor::NuiSkeletonTrackingEnable method. To receive information about the recognized users, call the INuiSensor::NuiSkeletonGetNextFrame method.

    // Call StartKinectST once at application start.
    HRESULT MyApplication::StartKinectST()
    {
      m_hNextSkeletonEvent = NULL;

      // Initialize m_pNuiSensor
      HRESULT hr = FindKinectSensor();
      if (SUCCEEDED(hr))
      {
        // Initialize the Kinect and specify that we'll be using skeleton
        hr = m_pNuiSensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_SKELETON);
        if (SUCCEEDED(hr))
        {
          // Create an event that will be signaled when skeleton data is available
          m_hNextSkeletonEvent = CreateEventW(NULL, TRUE, FALSE, NULL);

          // Open a skeleton stream to receive skeleton data
          hr = m_pNuiSensor->NuiSkeletonTrackingEnable(m_hNextSkeletonEvent, 0);
        }
      }
      return hr;
    }

    // Call UpdateKinectST on each iteration of the application's update loop.
    void MyApplication::UpdateKinectST()
    {
      // Wait for 0ms, just quickly test if it is time to process a skeleton
      if ( WAIT_OBJECT_0 == WaitForSingleObject(m_hNextSkeletonEvent, 0) )
      {
        NUI_SKELETON_FRAME skeletonFrame = {0};

        // Get the skeleton frame that is ready
        if (SUCCEEDED(m_pNuiSensor->NuiSkeletonGetNextFrame(0, &skeletonFrame)))
        {
          // Process the skeleton frame
          SkeletonFrameReady(&skeletonFrame);
        }
      }
    }
      

Accessing Skeletal Tracking Information

Information about the recognized users is provided in the form of an array of Skeleton objects present in the frame.

How to access skeletal tracking information in C#

At every KinectSensor.SkeletonFrameReady or KinectSensor.AllFramesReady event, open the frame using the SkeletonFrameReadyEventArgs.OpenSkeletonFrame method and call the SkeletonFrame.CopySkeletonDataTo method to copy the data to an array of Skeleton objects. It is advisable to reuse the same array and not allocate a new one for every frame.

    private void kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
    {
      using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame()) // Open the Skeleton frame
      {
        if (skeletonFrame != null && this.skeletonData != null) // check that a frame is available
        {
          skeletonFrame.CopySkeletonDataTo(this.skeletonData); // get the skeletal information in this frame
        }
      }
    }
        

How to access skeletal tracking information in C++

Every time INuiSensor::NuiSkeletonGetNextFrame returns a valid frame, access the members of the NUI_SKELETON_FRAME structure.

  void MyApplication::SkeletonFrameReady(NUI_SKELETON_FRAME* pSkeletonFrame)
  {
    // Access members of pSkeletonFrame...
  }
        

Skeleton Position and Tracking State

The skeletons in a frame have can have a tracking state of "tracked" or "position only". A tracked skeleton provides detailed information about the position in the camera's field of view of twenty joints of the user's body.

A skeleton with a tracking state of "position only" has information about the position of the user, but no details about the joints. An application can decide which skeletons to track, using the tracking ID as shown in the Active User Tracking section.

Tracked skeleton information can also be retrieved in the depth map as shown in the PlayerID in depth map section.

How to access tracked skeleton data in C#

Loop in the array of Skeleton objects returned in each frame and verify the TrackingState: this can be a value of the SkeletonTrackingState enumeration (Tracked, PositionOnly, or NonTracked). Ignore the non-tracked skeletons; they are placeholders in the array. For tracked skeletons, access the Joints collection to get the position of the joints; for position-only skeletons, access the Position property.

  private void DrawSkeletons()
  {
      foreach (Skeleton skeleton in this.skeletonData)
      {
          if (skeleton.TrackingState == SkeletonTrackingState.Tracked)
          {
              DrawTrackedSkeletonJoints(skeleton.Joints);
          }
          else if (skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
          {
              DrawSkeletonPosition(skeleton.Position);
          }
      }
  }

How to access tracked skeleton data in C++

Loop in the array of NUI_SKELETON_DATA structures returned in each frame and verify the TrackingState field; this can be a value of the NUI_SKELETON_TRACKING_STATE enumeration (NUI_SKELETON_NOT_TRACKED, NUI_SKELETON_POSITION_ONLY, or NUI_SKELETON_TRACKED). Ignore the untracked skeletons, they are placeholders in the array; for tracked skeletons, access the SkeletonPositions array to get the position of the joints; for position-only skeletons, access the Position field.

      
  void MyApplication::SkeletonFrameReady(NUI_SKELETON_FRAME* pSkeletonFrame)
  {
  for (int i = 0; i < NUI_SKELETON_COUNT; i++)
    {
        const NUI_SKELETON_DATA & skeleton = pSkeletonFrame->SkeletonData[i];
 
        switch (skeleton.eTrackingState)
        {
        case NUI_SKELETON_TRACKED:
            DrawTrackedSkeletonJoints(skeleton);
            break;
 
        case NUI_SKELETON_POSITION_ONLY:
            DrawSkeletonPosition(skeleton.Position);
            break;
        }
    }
}
    

Joint Positions and Tracking States

For tracked skeletons, an array of joints provides the position of the twenty recognized human joints in space over time. For example, applications can use the hand joint to guide a cursor on the screen or simply draw the position of user’s body on the screen.

A joint also has a tracking state: it can be "tracked" for a clearly visible joint, "inferred" when a joint is not clearly visible and Kinect is inferring its position, or "non-tracked", for example, for a lower joint in seated-mode tracking.

How to access joint information and draw a skeleton in C#

The Joint class has a TrackingState that can accessed for each joint in the skeleton and used to verify the value of the JointTrackingState enumeration (Tracked, NotTracked, or Inferred) that is set for the joint in question.

    private void DrawTrackedSkeletonJoints(JointCollection jointCollection)
    {
      // Render Head and Shoulders
      DrawBone(jointCollection[JointType.Head], jointCollection[JointType.ShoulderCenter]);
      DrawBone(jointCollection[JointType.ShoulderCenter], jointCollection[JointType.ShoulderLeft]);
      DrawBone(jointCollection[JointType.ShoulderCenter], jointCollection[JointType.ShoulderRight]);

      // Render Left Arm
      DrawBone(jointCollection[JointType.ShoulderLeft], jointCollection[JointType.ElbowLeft]);
      DrawBone(jointCollection[JointType.ElbowLeft], jointCollection[JointType.WristLeft]);
      DrawBone(jointCollection[JointType.WristLeft], jointCollection[JointType.HandLeft]);

      // Render Right Arm
      DrawBone(jointCollection[JointType.ShoulderRight], jointCollection[JointType.ElbowRight]);
      DrawBone(jointCollection[JointType.ElbowRight], jointCollection[JointType.WristRight]);
      DrawBone(jointCollection[JointType.WristRight], jointCollection[JointType.HandRight]);

      // Render other bones...
    }

    private void DrawBone(Joint jointFrom, Joint jointTo)
    {
      if (jointFrom.TrackingState == JointTrackingState.NotTracked ||
      jointTo.TrackingState == JointTrackingState.NotTracked)
      {
        return; // nothing to draw, one of the joints is not tracked
      }

      if (jointFrom.TrackingState == JointTrackingState.Inferred ||
      jointTo.TrackingState == JointTrackingState.Inferred)
      {
        DrawNonTrackedBoneLine(jointFrom.Position, jointTo.Position);  // Draw thin lines if either one of the joints is inferred
      }

      if (jointFrom.TrackingState == JointTrackingState.Tracked &&
      jointTo.TrackingState == JointTrackingState.Tracked)
      {
        DrawTrackedBoneLine(jointFrom.Position, jointTo.Position);  // Draw bold lines if the joints are both tracked
      }
    }
          

How to access joint information and draw a skeleton in C++

The NUI_SKELETON_DATA structure has a SkeletonPositionTrackingState array, which contains the tracking state for each joint, and a SkeletonPositions array, which contains the position of each joint.

    void MyApplication::DrawSkeleton(const NUI_SKELETON_DATA & skeleton)
    {
      // Render Head and Shoulders
      DrawBone(skeleton, NUI_SKELETON_POSITION_HEAD, NUI_SKELETON_POSITION_SHOULDER_CENTER);
      DrawBone(skeleton, NUI_SKELETON_POSITION_SHOULDER_CENTER, NUI_SKELETON_POSITION_SHOULDER_LEFT);
      DrawBone(skeleton, NUI_SKELETON_POSITION_SHOULDER_CENTER, NUI_SKELETON_POSITION_SHOULDER_RIGHT);

      // Render Left Arm
      DrawBone(skeleton, NUI_SKELETON_POSITION_SHOULDER_LEFT, NUI_SKELETON_POSITION_ELBOW_LEFT);
      DrawBone(skeleton, NUI_SKELETON_POSITION_ELBOW_LEFT, NUI_SKELETON_POSITION_WRIST_LEFT);
      DrawBone(skeleton, NUI_SKELETON_POSITION_WRIST_LEFT, NUI_SKELETON_POSITION_HAND_LEFT);

      // Render Right Arm
      DrawBone(skeleton, NUI_SKELETON_POSITION_SHOULDER_RIGHT, NUI_SKELETON_POSITION_ELBOW_RIGHT);
      DrawBone(skeleton, NUI_SKELETON_POSITION_ELBOW_RIGHT, NUI_SKELETON_POSITION_WRIST_RIGHT);
      DrawBone(skeleton, NUI_SKELETON_POSITION_WRIST_RIGHT, NUI_SKELETON_POSITION_HAND_RIGHT);

      // Render other bones...
    }

    void MyApplication::DrawBone(
      const NUI_SKELETON_DATA & skeleton,
      NUI_SKELETON_POSITION_INDEX jointFrom,
      NUI_SKELETON_POSITION_INDEX jointTo)
      {
        NUI_SKELETON_POSITION_TRACKING_STATE jointFromState = skeleton.eSkeletonPositionTrackingState[jointFrom];
        NUI_SKELETON_POSITION_TRACKING_STATE jointToState = skeleton.eSkeletonPositionTrackingState[jointTo];

        if (jointFromState == NUI_SKELETON_POSITION_NOT_TRACKED || jointToState == NUI_SKELETON_POSITION_NOT_TRACKED)
        {
          return; // nothing to draw, one of the joints is not tracked
        }

        const Vector4& jointFromPosition = skeleton.SkeletonPositions[jointFrom];
        const Vector4& jointToPosition = skeleton.SkeletonPositions[jointTo];

        // Don't draw if both points are inferred
        if (jointFromState == NUI_SKELETON_POSITION_INFERRED || jointToState == NUI_SKELETON_POSITION_INFERRED)
        {
          DrawNonTrackedBoneLine(jointFromPosition, jointToPosition); // Draw thin lines if either one of the joints is inferred
        }

        // We assume all drawn bones are inferred unless BOTH joints are tracked
        if (jointFromState == NUI_SKELETON_POSITION_TRACKED && jointToState == NUI_SKELETON_POSITION_TRACKED)
        {
          DrawTrackedBoneLine(jointFromPosition, jointToPosition); // Draw bold lines if the joints are both tracked
        }
      }
          

Clipped Edges

When a user moves to the edge of the field of view, the skeletal tracking system warns the application that the user skeleton is clipped. This information can be used by the application to tell the user to move to the center of the scene to allow for better tracking.

How to access clipped edges in C#

Use the ClippedEdges property to verify if the value of the FrameEdges enumeration has any of the Bottom, Top, Left, or Right flags set.

  private void RenderClippedEdges(Skeleton skeleton)
  {
    if (skeleton.ClippedEdges.HasFlag(FrameEdges.Bottom))
    {
      DrawClippedEdges(FrameEdges.Bottom); // Make the border red to show the user is reaching the border
    }

    if (skeleton.ClippedEdges.HasFlag(FrameEdges.Top))
    {
      DrawClippedEdges(FrameEdges.Top);
    }

    if (skeleton.ClippedEdges.HasFlag(FrameEdges.Left))
    {
      DrawClippedEdges(FrameEdges.Left);
    }

    if (skeleton.ClippedEdges.HasFlag(FrameEdges.Right))
    {
      DrawClippedEdges(FrameEdges.Right);
    }
  }
          

How to access clipped edges in C++

Use the dwQualityFlags field of the NUI_SKELETON_DATA structure, verifying if any flags are set.

    void MyApplication::RenderClippedEdges(const NUI_SKELETON_DATA & skeleton)
    {
      if (skeleton.dwQualityFlags & NUI_SKELETON_QUALITY_CLIPPED_BOTTOM)
      {
        DrawClippedEdges(NUI_SKELETON_QUALITY_CLIPPED_BOTTOM); // Make the border red to show the user is reaching the border
      }

      if (skeleton.dwQualityFlags & NUI_SKELETON_QUALITY_CLIPPED_TOP)
      {
        DrawClippedEdges(NUI_SKELETON_QUALITY_CLIPPED_TOP);
      }

      if (skeleton.dwQualityFlags & NUI_SKELETON_QUALITY_CLIPPED_LEFT)
      {
        DrawClippedEdges(NUI_SKELETON_QUALITY_CLIPPED_LEFT);
      }

      if (skeleton.dwQualityFlags & NUI_SKELETON_QUALITY_CLIPPED_RIGHT)
      {
        DrawClippedEdges(NUI_SKELETON_QUALITY_CLIPPED_RIGHT);
      }
    }
          

Active User Tracking

Kinect skeletal tracking allows the application to choose the skeleton to track among the six users recognized in the field of view.

By default, skeletal tracking will select the first two recognized users in the field of view. This behavior is consistent (not random), but not driven by specific criteria. If the application wants, it can override the default behavior and define custom logic for selecting the users to track. For example, you could track the user closest to the camera or the user that is raising his or her hand.

To do so, applications can cycle through the proposed skeletons, chose those that fit the needs, and pass their tracking IDs to the skeletal tracking APIs for full tracking.

After the application has control over the users to track, the skeletal tracking system will not take control back: if the user goes out of the screen, it is up to the application to select a new user to track.

Note that if a user exits the scene and comes back, he or she will receive a new tracking ID chosen randomly. The new ID will not be related to the one he or she had when exiting the scene.

Applications can also select to track only one skeleton or no skeletons at all by passing a null tracking ID to the skeletal tracking APIs.

How to select the users to track in C#

The application can take control of the users to track by setting AppChoosesSkeletons to true. To specify the user or users to track, call the SkeletonStream.ChooseSkeletons method and pass the tracking ID of one or two skeletons you want to track (or no parameters if no skeletons are to be tracked).

  private void TrackClosestSkeleton()
  {
    if (this.kinect != null && this.kinect.SkeletonStream != null)
        {
        if (!this.kinect.SkeletonStream.AppChoosesSkeletons)
        {
        this.kinect.SkeletonStream.AppChoosesSkeletons = true; // Ensure AppChoosesSkeletons is set
        }

        float closestDistance = 10000f; // Start with a far enough distance
        int closestID = 0;

        foreach (Skeleton skeleton in this.skeletonData.Where(s => s.TrackingState != SkeletonTrackingState.NotTracked))
        {
        if (skeleton.Position.Z < closestDistance)
        {
          closestID = skeleton.TrackingId;
          closestDistance = skeleton.Position.Z;
        }
      }

      if (closestID > 0)
      {
        this.kinect.SkeletonStream.ChooseSkeletons(closestID); // Track this skeleton
      }
    }
  }
      

How to select the users to track in C++

The application can take control of the users to track by setting the NUI_SKELETON_TRACKING_FLAG_TITLE_SETS_TRACKED_SKELETONS flag when calling INuiSensor::NuiSkeletonTrackingEnable. To specify the user or users to track, call INuiSensor::SetTrackedSkeletons, passing the tracking ID of the skeleton or skeletons to be tracked.

  void MyApplication::TrackClosestSkeleton(NUI_SKELETON_FRAME* pSkeletonFrame)
  {
    m_pNuiSensor->NuiSkeletonTrackingEnable(
    m_hNextSkeletonEvent,
    NUI_SKELETON_TRACKING_FLAG_TITLE_SETS_TRACKED_SKELETONS);

    float closestDistance = 10000.0f; // Start with a far enough distance
    DWORD closestIDs[NUI_SKELETON_MAX_TRACKED_COUNT] = {0, 0};

    for (int i = 0; i < NUI_SKELETON_COUNT; i++)
    {
      const NUI_SKELETON_DATA & skeleton = pSkeletonFrame->SkeletonData[i];
      if (skeleton.eTrackingState != NUI_SKELETON_NOT_TRACKED)
      {
        if (skeleton.Position.z < closestDistance)
        {
          closestIDs[0] = skeleton.dwTrackingID;
          closestDistance = skeleton.Position.z;
        }
      }
 
      if (closestIDs[0] > 0)
      {
        m_pNuiSensor->SetTrackedSkeletons(closestIDs); // Track this skeleton
      }
   }
      

Player ID in depth map

When skeletal tracking identifies a user in the field of view, the location of the user is also reported in the depth map.

For every pixel in the depth map, the three lowest-order bits contain player index information. An index of "0", indicates no player at that given pixel; an index of "1" through "6" indicates the presence of a player.

To look up the skeleton that corresponds to the player index for a specific depth pixel, subtract "1" from the player index, and use the resulting value as an index into the array of skeletons.

How to find a player ID in a depth pixel in C#

    private void FindPlayerInDepthPixel(short[] depthFrame)
    {
      foreach(short depthPixel in depthFrame)
      {
        int player = depthPixel & DepthImageFrame.PlayerIndexBitmask;

        if (player > 0 && this.skeletonData != null)
        {
          Skeleton skeletonAtPixel = this.skeletonData[player - 1];   // Found the player at this pixel
          // ...
        }
      }
    }
        

How to find a player ID in a depth pixel in C++

  void MyApplication::FindPlayerInDepthPixel(NUI_IMAGE_FRAME* pDepthFrame, NUI_SKELETON_FRAME* pSkeletonFrame)
  { 
      INuiFrameTexture* pTexture = pDepthFrame->pFrameTexture;
      NUI_LOCKED_RECT LockedRect;
      pTexture->LockRect(0, &LockedRect, NULL, 0);
      if (LockedRect.Pitch != 0)
      {
          const USHORT * pBufferRun = reinterpret_cast<const USHORT *>(LockedRect.pBits);
          const USHORT * pBufferEnd = pBufferRun + (LockedRect.size / sizeof(USHORT));
          for (; pBufferRun < pBufferEnd; pBufferRun++)
          {
              int player = (*pBufferRun & NUI_IMAGE_PLAYER_INDEX_MASK);
 
              if (player > 0 && pSkeletonFrame != NULL)
              {
                  const NUI_SKELETON_DATA & skeletonAtPixel = pSkeletonFrame->SkeletonData[player - 1]; // Found the player at this pixel
                  // ...
              }
          }
      }
      pTexture->UnlockRect(0);
  }