Abstract
This code sample demonstrates how to create a real-time “green screen” effect for capturing augmented selfies using the Intel® RealSense™ SDK for Windows*. The SDK includes a User Segmentation module that generates a segmented image on each frame that can be used to remove or change the background behind the user’s body. This sample app demonstrates a method for replacing the background with a static image selected by the user.
Figure 1. User Segmentation, Before and After
Introduction
On startup the app replaces the live background behind the user with a solid green image. The selection of green for the initial bitmap background color is completely arbitrary. It is simply a nod to the traditional chroma key compositing (or “green screen”) technique for removing the background from the subject of a photograph or video.
In contrast to chroma keying and 2D camera background subtraction techniques, the User Segmentation module works on synchronized color and depth images streaming from the 3D camera. Unlike 2D methods, there is no possibility of colors on the user’s clothing being interpreted as part of the background, or of a moving background being interpreted as part of the user.
The module outputs a 32-bit BGRA formatted image consisting of the input color data and a synthesized alpha channel (referred to as a mask). Pixels corresponding to the background have an alpha channel of zero, and pixels corresponding to the user will have an alpha value greater than zero. (Refer to the installed SDK manual located in RSSDK_DIR\doc\PDF\sdkmanuals.pdf for more details.)
The app includes a Checkbox control that allows the user to remove the background image if desired (see the “before” image in Figure 1). Background images are selected by clicking the “Open File” button, which opens a dialog box allowing the user to navigate to image files (*.bmp, *.jpg, *.png, *.gif) on the hard drive (Figure 2).
Figure 2. Open File Dialog Box
The selected background image is automatically scaled by the app to the pixel dimensions of the color and depth streams (640x480). On each frame the background and segmented images are stitched together and rendered in a WPF Image control.
When the “Capture” button is clicked, a snapshot is taken and saved to the output folder as “MyPic.jpg” (Figure 3).
Figure 3. Captured Image
The Depth Advantage
The Intel® RealSense™ 3D Camera (F200) houses both a 640x480 resolution IR camera and a 1080p RGB color camera. Combining data from both the IR depth and color cameras has the advantage of overcoming many of the issues associated with 2D camera background subtraction, such as a dynamic backgrounds (e.g., people or objects moving in the background), shadows, and varying lighting conditions.
Development Environment
The sample app can be built using Microsoft Visual Studio* Express 2013 for Windows Desktop or the professional versions of Visual Studio 2013.
About the Code
The code sample is a Windows Desktop app coded in C# using Extensible Application Markup Language (XAML) to define the user interface. It uses the unsafe keyword to define a section of code for working with pointers to facilitate faster image processing (refer to the C# Examples website for other fast image processing techniques).
As stated in the MSDN* Unsafe Code and Pointers C# Programming Guide, “unsafe code in C# is not necessarily dangerous; it is just code whose safety cannot be verified by the CLR. The CLR will therefore only execute unsafe code if it is in a fully trusted assembly.” To use the unsafe keyword in your code you must check the “Allow unsafe code” setting under Project→ Properties→ Build.
This project uses an explicit path to libpxcclr.cs.dll (the managed DLL): C:\Program Files (x86)\Intel\RSSDK\bin\x64. This reference will need to be changed if your SDK installation path is different. Since we are referencing the 64-bit version of the DLL, you must also ensure that “x64” is specified under the Project→ Properties→ Platform target setting.
For more information on using the User Segmentation module please refer to the SDK’s installed documentation (RSSDK_DIR\doc\PDF\sdkmanuals.pdf) and the online Background Segmentation Tutorial.
About Intel® RealSense™ Technology
To get started and learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/intel-realsense-sdk
About the Author
Bryan Brown is a software applications engineer in the Developer Relations Division at Intel.