Abstract
This article describes a code walkthrough for creating a virtual joystick app (see Figure 1) that incorporates the new Hand Cursor Module in the Intel® RealSense™ SDK. This project is developed in C#/XAML and can be built using Microsoft Visual Studio* 2015.
Figure 1: RS Joystick app controlling Google Earth* flight simulator.
Introduction
Support for the new Intel® RealSense™ camera, model SR300, was introduced in R5 of the Intel RealSense SDK. The SR300 is the successor to the F200 model and provides a set of improvements along with a new feature known as the Hand Cursor Module.
As described in the SDK documentation , the Hand Cursor Module returns a single point on the hand that allows accurate and responsive tracking. Its purpose is to facilitate the hand-based UI control use case, along with supporting a limited set of gestures.
RS Joystick, the joystick emulator app described in this article, maps 3D hand data provided by the SDK to virtual joystick controls, resulting in a hands-free way to interact with software applications that work with joystick controllers.
The RS Joystick app leverages the following Hand Cursor Module features:
- Body Side Type– The app notifies the user which hand is controlling the virtual joystick, based on a near-to-far access order.
- Cursor-Click Gesture– The user can toggle the ON-OFF state of button 1 on the virtual joystick controller by making a finger-click gesture.
- Adaptive Point Tracking– The app displays the normalized 3D point inside the imaginary “bounding box” defined by the Hand Cursor Module and uses this data to control the x-, y-, and z-axes of the virtual joystick.
- Alert Data– The app uses Cursor Not Detected, Cursor Disengaged, and Cursor Out Of Border alerts to change the joystick border from green to red when the user’s hand is out of range of the SR300 camera.
(For more information on the Hand Cursor Module check out “What could you do with Intel RealSense Cursor Mode?”)
Prerequisites
You should have some knowledge of C# and understand basic operations in Visual Studio like building an executable. Previous experience with adding third-party libraries to a custom software project is helpful, but this walkthrough provides detailed steps, if this is new to you. Your system needs a front-facing SR300 camera, the latest versions of the SDK and Intel® RealSense™ Depth Camera Manager (DCM) installed, and must meet the hardware requirements listed here. Finally, your system must be running Microsoft Windows* 10 Threshold 2.
Third-Party Software
In addition to the Intel RealSense SDK, this project incorporates a third-party virtual joystick device driver called vJoy* along with some dynamic-link libraries (DLLs). These software components are not part of any distributed code associated with this custom project, so details on downloading and installing the device driver are provided below.
Install the Intel RealSense SDK
Download and install the required DCM and SDK at https://software.intel.com/en-us/intel-realsense-sdk/download. At the time of this writing the current versions of these components are:
- Intel RealSense Depth Camera Manager (SR300) v3.1.25.1077
- Intel RealSense SDK v8.0.24.6528
Install the vJoy Device Driver and SDK
Download and install the vJoy device driver: http://vjoystick.sourceforge.net/site/index.php/download-a-install/72-download. Reboot the computer when instructed to complete the installation.
Once installed, the vJoy device driver appears under Human Interface Devices in Device Manager (see Figure 2).
Figure 2: Device Manager.
Next, open the Windows 10 Start menu and select All apps. You will find several installed vJoy components, as shown in Figure 3.
Figure 3: Windows Start menu.
To open your default browser and go to the download page, click the vJoy SDK button.
Once downloaded, copy the .zip file to a temporary folder, unzip it, and then locate the C# DLLs in \SDK\c#\x86.
We will be adding these DLLs to our Visual Studio project once it is created, as described in the next step.
Create a New Visual Studio Project
- Launch Visual Studio 2015.
- From the menu bar, select File, New, Project….
- In the New Project screen, expand Templates and select Visual C#, Windows.
- Select WPF Application.
- Specify the location for the new project and its name. For this project, our location is C:\ and the name of the application is RsJoystick.
Figure 4 show the New Project settings used for this project.
Figure 4: Visual Studio* New Project settings.
Click OK to create the project.
Copy Libraries into the Project
Two DLLs are required for creating Intel® RealSense™ apps in C#:
- libpxcclr.cs.dll – the managed C# interface DLL
- libpxccpp2c.dll – the unmanaged C++ P/Invoke DLL
Similarly, there are two DLLs required to allow the app to communicate with the vJoy device driver:
- vJoyInterface.dll – the C-language API library
- vJoyInterfaceWrap.dll – the C# wrapper around the C-language API library
To simplify the overall structure of our project, we’re going to copy all four of these files directly into the project folder:
- Right-click the RsJoystick project and select Add, Existing Item…
- Navigate to the location of the vJoy DLLs (that is, \SDK\c#\x86) and select both vJoyInterface.dll and vJoyInterfaceWrap.dll. Note: you may need to specify All Files (*.*) for the file type in order for the DLLs to become visible.
- Click the Add button.
Similarly, copy the Intel RealSense SDK DLLs into the project:
- Right-click the RsJoystick project and then select Add, Existing Item…
- Navigate to the location where the x86 libraries reside, which is C:\Program Files (x86)\Intel\RSSDK\bin\win32 in a default SDK installation.
- Select both libpxcclr.cs.dll and libpxccpp2c.dll.
- Click the Add button.
All four files should now be visible in Solution Explorer under the RsJoystick project.
Create References to the Libraries
Now that the required library files have been physically copied to the Visual Studio project, you must create references to the managed (.NET) DLLs so they can be used by your app. Right-click References (which is located under the RsJoystick project) and select Add Reference… In the Reference Manager window, click the Browse button and navigate to the project folder (c:\RsJoystick\RsJoystick). Select both the libpxcclr.cs.dll and vJoyInterfaceWrap.dll files, and then click the Add button. Click the OK button in Reference Manager.
In order for the managed wrapper DLLs to work properly, you need to ensure the unmanaged DLLs get copied into the project’s output folder before the app runs. In Solution Explorer, click libpxccpp2c.dll to select it. The Properties screen shows the file properties for libpxccpp2c.dll. Locate the Copy to Output Directory field and use the drop-down list to select Copy Always. Repeat this step for vJoyInterface.dll. This ensures that the unmanaged DLLs get copied to the project output folder when you build the application.
At this point you may see a warning about a mismatch between the processor architecture of the project being built and the processor architecture of the referenced libraries. Clear this warning by doing the following:
- Locate the link to Configuration Manager in the drop-down list in the menu bar (see Figure 5).
- Select Configuration Manager.
- In the Configuration Manager screen, expand the drop-down list in the Platform column, and then select New.
- Select x86 as the new platform and then click OK.
- Close the Configuration Manager screen.
Figure 5: Configuration Manager.
At this point the project should build and run without any errors or warnings. Also, if you examine the contents of the output folder (c:\RsJoystick\RsJoystick\bin\x86\Debug) you should find that all four of the DLLs got copied there as well.
The User Interface
The user interface (see Figure 6) displays the following information:
- The user’s hand that is controlling the virtual joystick, based on a near-to-far access order (that is, the hand that is closest to the camera is the controlling hand).
- The ON-OFF state of Button 1 on the virtual joystick controller, which is controlled by making a finger-click gesture.
- An ellipse that tracks the relative position of the user’s hand in the x- and y-axes, and changes diameter based on the z-axis to indicate the hand’s distance from the camera.
- The x-, y-, and z-axis Adaptive Point data from the SDK, which is presented as normalized values in the range of zero to one.
- A colored border that changes from green to red when the user’s hand is out of range of the SR300 camera.
- Slider controls that allow the sensitivity to be adjusted for each axis.
Figure 6: User Interface.
The complete XAML source listing is presented in Table 1. This can be copied and pasted directly over the MainWindow.xaml code that was automatically generated when the project was created.
Table 1:XAML Source Code Listing: MainWindow.xaml.
<Window x:Class="RsJoystick.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:local="clr-namespace:RsJoystick"
mc:Ignorable="d"
Title="RSJoystick" Height="420" Width="420" Background="#FF222222" Closing="Window_Closing"><Window.Resources><Style x:Key="TextStyle" TargetType="TextBlock"><Setter Property="Foreground" Value="White"/><Setter Property="FontSize" Value="14"/><Setter Property="Text" Value="-"/><Setter Property="Margin" Value="4"/><Setter Property="HorizontalAlignment" Value="Center"/></Style></Window.Resources><StackPanel VerticalAlignment="Center" HorizontalAlignment="Center" Width="320"><TextBlock x:Name="uiBodySide" Style="{StaticResource TextStyle}"/><TextBlock x:Name="uiButtonState" Style="{StaticResource TextStyle}"/><Border x:Name="uiBorder" BorderThickness="2" Width="200" Height="200" BorderBrush="Red" Margin="4"><Canvas x:Name="uiCanvas" ClipToBounds="True"><Ellipse x:Name="uiCursor" Height="10" Width="10" Fill="Yellow"/><Ellipse Height="50" Width="50" Stroke="Gray" Canvas.Top="75" Canvas.Left="75"/><Rectangle Height="1" Width="196" Stroke="Gray" Canvas.Top="100"/><Rectangle Height="196" Width="1" Stroke="Gray" Canvas.Left="100"/></Canvas></Border><StackPanel Orientation="Horizontal" HorizontalAlignment="Center"><TextBlock x:Name="uiX" Style="{StaticResource TextStyle}" Width="80"/><Slider x:Name="uiSliderX" Width="150" ValueChanged="sldSensitivity_ValueChanged" Margin="4"/></StackPanel><StackPanel Orientation="Horizontal" HorizontalAlignment="Center"><TextBlock x:Name="uiY" Style="{StaticResource TextStyle}" Width="80"/><Slider x:Name="uiSliderY" Width="150" ValueChanged="sldSensitivity_ValueChanged" Margin="4"/></StackPanel><StackPanel Orientation="Horizontal" HorizontalAlignment="Center"><TextBlock x:Name="uiZ" Style="{StaticResource TextStyle}" Width="80"/><Slider x:Name="uiSliderZ" Width="150" ValueChanged="sldSensitivity_ValueChanged" Margin="4"/></StackPanel></StackPanel></Window>Program Source Code
The complete C# source listing for the RSJoystick app is presented in Table 2. This can be copied and pasted directly over the MainWindow.xaml.cs code that was automatically generated when the project was created.
Table 2.C# Source Code Listing: MainWindow.xaml.cs
//--------------------------------------------------------------------------------------
// Copyright 2016 Intel Corporation
// All Rights Reserved
//
// Permission is granted to use, copy, distribute and prepare derivative works of this
// software for any purpose and without fee, provided, that the above copyright notice
// and this statement appear in all copies. Intel makes no representations about the
// suitability of this software for any purpose. THIS SOFTWARE IS PROVIDED "AS IS."
// INTEL SPECIFICALLY DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, AND ALL LIABILITY,
// INCLUDING CONSEQUENTIAL AND OTHER INDIRECT DAMAGES, FOR THE USE OF THIS SOFTWARE,
// INCLUDING LIABILITY FOR INFRINGEMENT OF ANY PROPRIETARY RIGHTS, AND INCLUDING THE
// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Intel does not
// assume any responsibility for any errors which may appear in this software nor any
// responsibility to update it.
//--------------------------------------------------------------------------------------
using System;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Media;
using vJoyInterfaceWrap;
using System.Threading;
using System.Windows.Shapes;
namespace RsJoystick
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
private PXCMSenseManager sm;
private PXCMHandCursorModule cursorModule;
private PXCMCursorConfiguration cursorConfig;
private vJoy joystick;
private Thread update;
private double joySensitivityX;
private double joySensitivityY;
private double joySensitivityZ;
private const uint joyID = 1;
private const uint MaxSensitivity = 16384;
public MainWindow()
{
InitializeComponent();
// Configure the sensitivity controls
uiSliderX.Maximum = MaxSensitivity;
uiSliderY.Maximum = MaxSensitivity;
uiSliderZ.Maximum = MaxSensitivity;
joySensitivityX = uiSliderX.Value = MaxSensitivity / 2;
joySensitivityY = uiSliderY.Value = MaxSensitivity / 2;
joySensitivityZ = uiSliderZ.Value = MaxSensitivity / 2;
// Create an instance of the joystick
joystick = new vJoy();
joystick.AcquireVJD(joyID);
// Configure the cursor mode module
ConfigureRealSense();
// Start the Update thread
update = new Thread(new ThreadStart(Update));
update.Start();
}
public void ConfigureRealSense()
{
// Create an instance of the SenseManager
sm = PXCMSenseManager.CreateInstance();
// Enable cursor tracking
sm.EnableHandCursor();
// Get an instance of the hand cursor module
cursorModule = sm.QueryHandCursor();
// Get an instance of the cursor configuration
cursorConfig = cursorModule.CreateActiveConfiguration();
// Make configuration changes and apply them
cursorConfig.EnableEngagement(true);
cursorConfig.EnableAllGestures();
cursorConfig.EnableAllAlerts();
cursorConfig.ApplyChanges();
// Initialize the SenseManager pipeline
sm.Init();
}
private void Update()
{
bool handInRange = false;
bool joyButton = false;
// Start AcquireFrame-ReleaseFrame loop
while (sm.AcquireFrame(true).IsSuccessful())
{
PXCMCursorData cursorData = cursorModule.CreateOutput();
PXCMPoint3DF32 adaptivePoints = new PXCMPoint3DF32();
PXCMCursorData.BodySideType bodySide;
// Retrieve the current cursor data
cursorData.Update();
// Check if alert data has fired
for (int i = 0; i < cursorData.QueryFiredAlertsNumber(); i++)
{
PXCMCursorData.AlertData alertData;
cursorData.QueryFiredAlertData(i, out alertData);
if ((alertData.label == PXCMCursorData.AlertType.CURSOR_NOT_DETECTED) ||
(alertData.label == PXCMCursorData.AlertType.CURSOR_DISENGAGED) ||
(alertData.label == PXCMCursorData.AlertType.CURSOR_OUT_OF_BORDERS))
{
handInRange = false;
}
else
{
handInRange = true;
}
}
// Check if click gesture has fired
PXCMCursorData.GestureData gestureData;
if (cursorData.IsGestureFired(PXCMCursorData.GestureType.CURSOR_CLICK, out gestureData))
{
joyButton = !joyButton;
}
// Track hand cursor if it's within range
int detectedHands = cursorData.QueryNumberOfCursors();
if (detectedHands > 0)
{
// Retrieve the cursor data by order-based index
PXCMCursorData.ICursor iCursor;
cursorData.QueryCursorData(PXCMCursorData.AccessOrderType.ACCESS_ORDER_NEAR_TO_FAR,
0,
out iCursor);
adaptivePoints = iCursor.QueryAdaptivePoint();
// Retrieve controlling body side (i.e., left or right hand)
bodySide = iCursor.QueryBodySide();
// Control the virtual joystick
ControlJoystick(adaptivePoints, joyButton);
}
else
{
bodySide = PXCMCursorData.BodySideType.BODY_SIDE_UNKNOWN;
}
// Update the user interface
Render(adaptivePoints, bodySide, handInRange, joyButton);
// Resume next frame processing
cursorData.Dispose();
sm.ReleaseFrame();
}
}
private void ControlJoystick(PXCMPoint3DF32 points, bool buttonState)
{
double joyMin;
double joyMax;
// Scale x-axis data
joyMin = MaxSensitivity - joySensitivityX;
joyMax = MaxSensitivity + joySensitivityX;
int xScaled = Convert.ToInt32((joyMax - joyMin) * points.x + joyMin);
// Scale y-axis data
joyMin = MaxSensitivity - joySensitivityY;
joyMax = MaxSensitivity + joySensitivityY;
int yScaled = Convert.ToInt32((joyMax - joyMin) * points.y + joyMin);
// Scale z-axis data
joyMin = MaxSensitivity - joySensitivityZ;
joyMax = MaxSensitivity + joySensitivityZ;
int zScaled = Convert.ToInt32((joyMax - joyMin) * points.z + joyMin);
// Update joystick settings
joystick.SetAxis(xScaled, joyID, HID_USAGES.HID_USAGE_X);
joystick.SetAxis(yScaled, joyID, HID_USAGES.HID_USAGE_Y);
joystick.SetAxis(zScaled, joyID, HID_USAGES.HID_USAGE_Z);
joystick.SetBtn(buttonState, joyID, 1);
}
private void Render(PXCMPoint3DF32 points,
PXCMCursorData.BodySideType bodySide,
bool handInRange,
bool buttonState)
{
Dispatcher.Invoke(delegate
{
// Change drawing border to indicate if the hand is within range
uiBorder.BorderBrush = (handInRange) ? Brushes.Green : Brushes.Red;
// Scale cursor data for drawing
double xScaled = uiCanvas.ActualWidth * points.x;
double yScaled = uiCanvas.ActualHeight * points.y;
uiCursor.Height = uiCursor.Width = points.z * 100;
// Move the screen cursor
Canvas.SetRight(uiCursor, (xScaled - uiCursor.Width / 2));
Canvas.SetTop(uiCursor, (yScaled - uiCursor.Height / 2));
// Update displayed data values
uiX.Text = string.Format("X Axis: {0:0.###}", points.x);
uiY.Text = string.Format("Y Axis: {0:0.###}", points.y);
uiZ.Text = string.Format("Z Axis: {0:0.###}", points.z);
uiBodySide.Text = string.Format("Controlling Hand: {0}", bodySide);
uiButtonState.Text = string.Format("Button State (use 'Click' gesture to toggle): {0}",
buttonState);
});
}
private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
{
update.Abort();
cursorConfig.Dispose();
cursorModule.Dispose();
sm.Dispose();
joystick.ResetVJD(joyID);
joystick.RelinquishVJD(joyID);
}
private void sldSensitivity_ValueChanged(object sender,
RoutedPropertyChangedEventArgs<double> e)
{
var sliderControl = sender as Slider;
switch (sliderControl.Name)
{
case "uiSliderX":
joySensitivityX = sliderControl.Value;
break;
case "uiSliderY":
joySensitivityY = sliderControl.Value;
break;
case "uiSliderZ":
joySensitivityZ = sliderControl.Value;
break;
}
}
}
}Code Details
To keep this code sample as simple as possible, all methods are contained in a single class. As shown in the source code presented in Table 2, the MainWindow class is composed of the following methods:
- MainWindow() – Several private objects and member variables are declared at the beginning of the MainWindow class. These objects are instantiated and variables initialized in the MainWindow constructor.
- ConfigureRealSense()– This method handles the details of creating the SenseManager object and hand cursor module, and configuring the cursor module.
- Update()– As described in the Intel RealSense SDK Reference Manual, the SenseManager interface can be used either by procedural calls or by event callbacks. In the RSJoystick app we are using procedural calls as the chosen interfacing technique. The acquire/release frame loop runs in the Update() thread, independent of the main UI thread. This thread runs continuously and is where hand cursor data, gestures, and alert data is acquired.
- ControlJoystick()– This method is called from the Update() thread when the user’s hand is detected. Adaptive Point data is passed to this method, along with the state of the virtual joystick button (toggled by the CURSOR_CLICK gesture). The Adaptive Point data is scaled using values from the sensitivity slider controls. The slider controls and scaling calculations allow the user to select the full-scale range of values that are sent to the vJoy SetAxis() method, which expects values in the range of 0 to 32768. With a sensitivity slider set to its maximum setting, the corresponding cursor data point will be converted to a value in the range of 0 to 32768. Lower sensitivity settings will narrow this range for the same hand trajectory. For example: 8192 to 24576.
- Render()– This method is called from the Update() thread and uses the Dispatcher.Invoke() method to perform operations that will be executed on the UI thread. This includes updating the position of the ellipse on the canvas control and data values shown in the TextBlock controls.
- sldSensitivity_ValueChanged() – This event handler is raised whenever any of the slider controls are adjusted.
Using the Application
You can test the app by running vJoy Monitor from the Windows 10 Start menu (see Figure 3). As shown in Figure 7, you can monitor the effects of moving your hand in three axes and performing the click gesture to toggle button 1.
Figure 7: Testing the app with vJoy Monitor.
For a more fun and practical usage, you can run the flight simulator featured in Google Earth* (see Figure 1). According to their website, “Google Earth lets you fly anywhere on Earth to view satellite imagery, maps, terrain, 3D buildings, from galaxies in outer space to the canyons of the ocean.” (https://www.google.com/earth).
After downloading and installing Google Earth, refer to the instructions located here to run the flight simulator. Start by reducing the x- and y-axis sensitivity controls in RSJoystick to minimize the effects of hand motions on the airplane, and set the z-axis slider to its maximum position. After some experimentation you should be able to control the airplane using subtle hand motions.
Summary
This article provided a simple walkthrough describing how to create an Intel RealSense SDK-enabled joystick emulator app from scratch, and how to use the Hand Cursor Module supported by the SR300 camera.
About Intel RealSense Technology
To learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/intel-realsense-sdk.
About the Author
Bryan Brown is a software applications engineer at Intel Corporation in the Software and Services Group.