Native Intel® RealSense™ SDK Image Copying in Unity*

This sample uses a native Unity plug-in to increase performance of displaying Intel® RealSense™ SDK image data by bypassing the C# layers of the SDK. Image data is uploaded to the GPU directly through the graphics API.

The sample is available at https://github.com/GameTechDev/UnityRealSenseNativeTexture.

The sample supports Direct3D* 9, Direct3D* 11, and OpenGL*, but it could be modified to support other APIs. This plug-in is based on the native plug-in example provided by Unity. That sample, and more information on native Unity plug-ins, can be found here.

Figure 1. Screenshot from the sample running in Unity*.

Sample Contents

This sample includes:

• The plug-in source code
• A sample Unity project that uses the plug-in

How It Works

The process of this native texture copy in Unity is as follows:

1. The application obtains a PXCMImage object from the Intel® RealSense™ SDK and uses the QueryNativePointer function to obtain the native PXCImage pointer.
2. The PXCImage pointer and the destination Unity Texture2D pointer are passed to the native plug-in.
3. When the native plug-in’s callback is invoked on Unity’s render thread, the data from the PXCImage image is copied to the native texture using the appropriate graphics API.

Requirements

• Intel® RealSense™ SDK R5 or higher is recommended. Previous SDKs can be used but the developer will need to modify the PXCCLR project to add support for the QueryNativePointer function.
• Unity 5.2 or higher is recommended. The native plug-in was not tested on previous versions.
• Visual Studio* 2013 or higher. Previous versions can be used, but haven’t been configured.

Running the Sample

The sample includes a Unity project called SampleProject which demonstrates using the plug-in to display the color stream of an Intel® RealSense™ camera.

Follow these steps to run the sample:

1. Download the Intel® RealSense™ SDK version – You will need Intel RealSense SDK version R5 or greater.
2. Update the SampleProject’s SDK DLLs - Run the UpdateSDKDlls.bat in the SampleProject folder to copy the required managed DLLs from the SDK into the SampleProject.
3. Build the native plug-in – Open and build NativeRSTextureCopy.sln which can be found in within the src folder. A post-build step will copy the plug-in into the SampleProject folder.
4. Open and Run the SampleProject in Unity.

Integrating the Plug-in

Follow these steps to integrate the native plug-in into an existing Unity application that uses Intel® RealSense™ SDK:

1. Build the native plug-in – The plug-in must be rebuilt for the SDK the application is using. Open the Visual Studio solution and build the Release configuration for the desired target architecture.
2. Add UseTexturePlugin.cs to the project – Copy the UseTexturePlugin.cs script from the SampleProject folder into your project, and attach it to a persistent game object in the scene. This script is responsible for pumping the native render callback event as well as providing an interface to the native plug-in functionality.
3. Pass PXCMImages to the plug-in – Replace calls in your application to PXCMImage.ToTexture2D with calls to UseTexturePlugin.CopyPXCImageToTexture.
4. Handle API specific formatting – Depending on the graphics API, the texture may be displayed flipped vertically, or with the colors inverted. The NativeRGBImage.shader file demonstrates how to correct this formatting issue.
5. Notify on shutdown - Call the UseTexturePlugin.RealSenseShutdown function before shutting down the Intel® RealSense™ SDK Sense Manager.

Important Notes

• Because this is a native plug-in linked against a specific version of the Intel® RealSense™ SDK, you must recompile the plug-in when switching to different SDK versions. Also, be sure to compile it for all CPU architectures you plan on supporting.
• The QueryNativePointer function was added to the C# API in the R5 release to allow access to managed SDK objects’ underlying native objects. This sample will still work on previous SDKs, but you will need to add the QueryNativePointer functionality to the PXCCLR project in the installed SDK.