How to Migrate Intel® RealSense™ SDK R4 (v6.0) Hand Mode Functionality to Intel RealSense SDK 2016 R1 Cursor Mode

Abstract

With the dual arrivals of the Intel® RealSense™ camera (SR300) and the Intel® RealSense™ SDK 2016 R1, comes a new mode of gesture interaction called Cursor Mode, for use with the SR300 only. This tutorial sets out to highlight code changes that developers will have to make in order to exploit the new capability.

Introduction

Prior to the release of Intel RealSense SDK 2016 R1, applications wanting to effect cursor movement and detect click-actions were reliant on using the Hand Mode and detecting the “click action” through gesture recognition. That functionality existing in the Hand Mode has now been decoupled into a new feature called Cursor Mode.  As such, applications that relied on the previous functionality can now change their code to take advantage of the refinements and upgrades to cursor control with the new Cursor Mode.

Please note that Cursor Mode is available only to devices and peripherals that use the Intel RealSense camera (SR300). As a developer of Intel® RealSense™ applications looking to use the SR300, you must upgrade to Windows* 10, and we require that you use version 2016 R1 of the Intel RealSense SDK.

Tutorial

More than likely you already have an application that is written for the F200 camera with the Intel RealSense SDK R4 (v6.0) and would like to know how to move forward and use the new Cursor Mode functionality. This tutorial presents the following:

Part 1

Initialization of the processing pipeline must occur in a manner similar to the previous version of the Intel RealSense SDK.  Thus, you must instantiate the Sense Manager and check for no errors in the process.

PXCSenseManager *pSenseMgr = new PXCSenseManager::CreateInstance();
if( !pSenseMgr ) {< continue on to creating the modes >
}

Part 2

Previously for the F200 Hand Mode, to get anything resembling cursor actions you had to rely on the Hand Module and track the hand set to various configurations. Your code might have looked like this (note that the following code is for reference purposes and will not compile directly as written below):

PXCHandModule *pHandModule;
PXCHandData *pHandData;
int confidence;
. . . <additional library and variables setup> . . .
pxcStatus status;
if( !pSenseMgr ) {
	status = pSenseMgr->EnableHand()
	if(status == pxcStatus::PXC_STATUS_NO_ERROR) {
	// Get an instance of PXCHandModule
handModule = pSenseMgr->QueryHand();
// Get an instance of PXCHandConfiguration
PXCHandConfiguration handConfig = handModule
handConfig->EnableGesture("cursor_click");
handConfig->ApplyChanges();
	. . . <additional configuration options> . . .
}
}

Part 3

Beginning with the Intel RealSense SDK 2016 R1 a new Cursor Mode has been implemented, and cursor actions have been decoupled from the Hand Mode. This means that previous code paths that queried the Hand Mode in the Sense Manager must change. The new code will take the following form:

PXCHandCursorModule *pCursorModule;
PXCCursorData::BodySideType bodySide;
// please note that the Confidence values no longer exist
. . . <additional library and variables setup> . . .
pxcStatus status;
if( !pSenseMgr ) {
// Enable handcursor tracking
status = pSenseMgr::EnableHandCursor();
	if(status == pxcStatus.PXC_STATUS_NO_ERROR) {
	// Get an instance of PXCCursorModule
pCursorModule = pSenseMgr->QueryHandCursor();
// Get an instance of the cursor configuration
PXCCursorConfiguration *pCursorConfig = CursorModule::CreateActiveConfiguration();

// Make configuration changes and apply them
pCursorConfig.EnableEngagement(true);
pCursorConfig.EnableAllGestures();
pCursorConfig.ApplyChanges();
	. . . <additional configuration options> . . .

}
}

Part 4

Implementation examples of the main processing loops for synchronous and asynchronous functions can be found in the Intel RealSense™ SDK 2016 R1 Documentation in the Implementing the Main Processing Loop subsection of the Cursor Module [SR300] section.

A summary of the asynchronous—and preferred—approach is as follows:

class MyHandler: public PXCSenseManager::Handler {
public:
    virtual pxcStatus PXCAPI OnModuleProcessedFrame(pxcUID mid, PXCBase *module, PXCCapture::Sample *sample) {
       // check if the callback is from the hand cursor tracking module
       if (mid==PXCHandCursorModule::CUID) {
           PXCHandCursorModule *cursorModule=module->QueryInstance<PXCHandCursorModule>();
               PXCCursorData *cursorData = cursorModule->CreateOutput();
           // process cursor tracking data
       }

       // return NO_ERROR to continue, or any error to abort
       return PXC_STATUS_NO_ERROR;
    }
};
. . . <SenseManager declaration> . . .
// Initialize and stream data
MyHandler handler; // Instantiate the handler object

// Register the handler object
pSenseMgr->Init(&handler);

// Initiate SenseManager’s processing loop in blocking mode
// (function exits only when processing ends)
pSenseMgr ->StreamFrames(true);

// Release SenseManager resources
pSenseMgr ->Release()

Conclusion

Though the Intel RealSense SDK 2016 R1 has changed the implementation and access to the hand cursor, it is worth noting that the changes have a consistency that allow for an easy migration of your code. The sample code above demonstrate that ease by showing that your general program structure during initialization, setup, and per-frame execution can remain unchanged while still harnessing the improved capabilities of the new Cursor Mode.

It is worth repeating that the new Cursor Mode is only available to systems that are enabled with the SR300 camera, either integrated or as a peripheral, and using RealSense™ SDK 2016 R1. The ability to detect and branch your code to support dual F200 and SR300 cameras, with either one as peripherals, during development will be discussed in other tutorials.