Introduction
The Intel® RealSense™ SDK Gold R2 (v4.0) release is now available! This brief overview walks you through product improvements from Gold R1 (v3.1) and some of the new natural interaction modalities that can be used to create compelling applications using the Intel RealSense SDK.
Changes-At-A-Glance
- The SDK and Depth Camera Manager (DCM)
- New modalities
- 3D scanning
- Pulse Estimation
- Blob Tracking Module
- New utilities
- New capabilities for the Intel® RealSense™ rear facing camera (R200)
- Improvements in Gold R2 SDK
- Hand Tracking module
- Face Tracking module
- Unity* Support
- Java*, Processing* support
- JavaScript* support
- Touchless Controller
- Deprecated functions
- Additional Resources
The SDK and DCM
Two major changes in the Gold R2 release demand attention.
First, the SDK now comes with improved and additional capabilities for the front facing Intel® Realsense™ camera (F200). Details on these capabilities are listed in subsequent sections. Additionally, this version of the SDK also includes APIs for the rear facing camera (R200). Developers can read the API documentation to understand the capabilities specific to each model of the camera. Most of the capabilities listed in the article focusses on F200 camera. Features specific to R200 are explicitly called out.
DCM packaging has changed. Essentially, the camera driver is called the Depth Camera Manager. In R1, the DCM was a part of the SDK install package. From the R2 SDK forward, the DCM will only be available as a separate download. Note: Uninstalling Gold R1 SDK will also uninstall the DCM, requiring that you separately install DCM 1.2 after the R2 SDK installation. End users with an integrated camera OEM device will notice that the DCM is already installed on the system. In case you need to update the DCM, please use the DCM install link below for updates.
DCM version required for F200: 1.2.14.24922
DCM version required for R200: 2.0.0.47980
Download link: https://software.intel.com/en-us/intel-realsense-sdk/download
Installation instructions for Gold R2 SDK
- Uninstall previous versions of the SDK completely, including cleaning up the destination directory if needed. Reboot for a complete uninstall.
- Run the DCM installer: intel_rs_dcm_f200_1.2.14.24922 for the F200 camera
- Run the SDK offline installer: intel_rs_sdk_offline_package_r_4.0.0.112526.exe and reboot the computer.
New modalities and utilities in Gold R2 SDK
3D Scanning
The long awaited 3D scanning feature is available for preview with the Gold R2 SDK. The module generates 3D mesh data from the camera images. With this feature, you can now perform static object capture, optional object on planar surface detection/trimming and optional hole filling and solidification. The module works in targeting and scanning modes and allows for standard OBJ, PLY and STL output formats.
Current limitations
- The scanning volume size is limited by the camera’s field of view. The best practice is to place the object 15-30 inches in front of the camera for effective targeting.
- The algorithm works with rigid objects. If some portions of the object move in relation to the other portions, scanning may not be effective.
- The tracking algorithm works best when the object movement is slow.
Pulse estimator
In addition to identifying up to 4 faces, you can now track subtle changes in face skin color over time and estimate the person’s pulse rate. These functions are included as a part of the face detection modality. For more information, refer to the PulseConfiguration interface, QueryPulse and PulseData functions.
Blob tracking
For improved performance, where joint level data is not essential, blob tracking is a good alternative. It allows tracking anything in front of the camera, supports up to 4 blobs, without hand classification and supports different access orders. Note, however, that this feature is in preview mode.
New utilities
The Gold R2 SDK comes with some utilities that help improve the robustness and performance of your application.
- Rotation utility: This utility provides for a conversion between different rotation representations of points in 3D space. Several commonly used rotation representations like the Quaternion, rotation matrix, Euler angles and Angle + Axis are supported. For more information, refer to the PXC[M]Rotation class.
- PointConverter: This utility converts a point in 2D or 3D camera space to a different value range with the same dimensions. For example, convert a 2D point from a sensor with Height/Width to a point on the screen with resolution (1280X 960).
- PCXSmoother: This utility replaces the PXCDataSmoother utility in R1, includes online smoothing algorithms and is easier to use compared to the PXCDataSmoother utility.
New R200 Capabilities
The APIs required to program for the R200 camera is introduced for the first time in the Gold R2 SDK. Some of the key features are highlighted in the table below. Please note that the feature maturity varies between available capabilities. Developers are instructed to read the SDK documentation for a detailed list of APIs supporting each of the below modalities.
| Maturity | R200 |
|---|---|
| Gold | Speech Recognition and Synthesis |
| Alpha | SDK essential interfaces and color/depth/IR data streaming. Scene Perception Face Tracking |
| Preview | 3D Capture Enhanced Photography |
Improvements in Gold R2 SDK
Hand Tracking Module
The hand tracking module has been enhanced for performance in Gold R2 together with the inclusion of several new capabilities.
- New gestures added to the set of standard gestures:
- Swipe gesture is enhanced to now allow swipe_up and swipe_down
- There is also a new “click” gesture
- Confidence values and alerts: Scenarios such as hands out of the field of view (FOV), speed of motion, distance from the camera and hands pointing towards camera can now be tracked better using the low confidence values and alerts. Refer to the documentation for QueryTrackingStatus and PXCHandData::TrackingStatusType for more information.
- Kid sized hands calibration files: In Gold R1, tracking of hand sizes for ages <8 years was difficult. The Gold R2 SDK not only supports smaller hands but also provides capabilities to set the user age that allows you to load calibration files specifically for children. For more information, refer to the documentation for SetDefaultAge and QueryDefaultAge.
- Palm radius: An additional capability for detecting hand sizes more effectively is the inclusion of palm radius detection. For more information, refer to the documentation for QueryPalmRadiusImage and QueryPalmRadiusWorld.
Performance improvements
- Gesture detection improvements: Users of Gold R2 will see an improvement in the detection of thumbs_up and thumbs_down gestures that no longer require hand recognition and calibration before use. There is also a significant reduction in false positives for all gestures.
- Improvements to left and right hand switches allows for better hand detection and tracking.
- Improved performance for two hand scenarios and contour mode: In Gold R1, using two hand tracking and gestures significantly reduced the performance of the app. This feature is now enhanced to provide better FPS.
Face Tracking Module
Improved capabilities
- More robust landmarks: Includes support for wider angles and different skin tones. Improved landmark sensitivity around the mouth allows for improved recognition of expressions like open mouth and smile.
- Improved facial expressions now supports “kiss” based on 3D data.
- Stable and robust face detection along yaw, pitch and roll with improved head center tracking
Unity* Support
New capabilities
- Intel RealSense SDK now works with the free version of Unity 4.6. In order to get this to work, you must download the patch 4.6.3p1 from http://unity3d.com/unity/qa/patch-releases.
Improved Capabilities
- Additional scripts and prefabs for face tracking, hand tracking, object tracking and speech recognition are provided with Gold R2 which makes game development easier.
- New samples also demonstrate how to use these scripts and prefabs in app development.
Web programming capabilities
Web programmers are able to code either using JavaScript or the Unity web player. Browser support is still a Beta feature, and support is only available for hand, blob, face tracking and speech recognition. In R2, Support is now available for HTTPS websites.
The Unity web player works through a web server and allows for the use of any C# function except those marked not supported (For example, File I/O).
Java*/ Processing* Capabilities
The Intel RealSense SDK architecture allows for all of the SDK capabilities to be uniformly accessible from Java/Processing as with any other C++ or C# interfaces. However, Java/Processing support in the Intel RealSense SDK R2 release is still in preview and capabilities are currently limited. Please watch future releases for enhanced support.
Touchless Controller
The touchless controller, which provides the ability to control Windows UI functions with hand gestures now provides more gestures, improved recognition and performance, and methods to control point and scroll sensitivity. Additional capabilities include mapping user behavior to UI events, scrolling, zooming, navigating to the start menu, backing up and edge scrolling. For more information, refer to the PXC[M]TouchlessController function.
Deprecated functions in Gold R2
Some of the utilities that were available in Gold R1 are now deprecated in R2. These include:
- PXC[M]BlobExtractor – replaced by PXC[M]BlobModule
- PXC[M]ContourExtractor – replaced by PXC[M]BlobModule
- PXC[M]DataSmoothing – replaced by PXC[M]Smoother
To ensure that applications run on end-user systems without any problems, package the application with the runtime specific to the SDK version used for development.
Summary
Intel® RealSense™ Gold R2 SDK provides significant feature and performance improvements compared to Gold R1 SDK. Developers are encouraged to move to the Gold R2 SDK to take advantage of the improved and new capabilities to develop robust natural interactive applications.