Tutorial: Using Intel® RealSense™ Technology in the Unreal Engine* 3 - Part 1

Download PDF 1.38 MB

Part 2

Introduction

Epic Games (https://epicgames.com/) Unreal Game Engine 3 (UE3) is a popular PC game engine. Intel® RealSense™ Technology is used for face and hand movement tracking to enrich the gaming experience. In the UE3 environment, using an Unreal script in the Unreal Development Kit (UDK) is the only recommendation and custom functions should be added into the UDK as a plug-in, but the Intel® RealSense™ SDK doesn’t provide an UDK plugin.

This article describes how to add Intel RealSense SDK features into a game character in massively multiplayer online role-playing games (MMORPGs) on UE3 by using C++, not an Unreal script. The common term for determining and modifying a character’s facial structure is “face-rigging.” There are several ways to handle face-rigging in the game, but we are focusing on using the Intel RealSense SDK as the manipulation method of the characters’ facial bone structure for performance and workload issues.

Key points covered in the article include the following:

• A description of the face-rigging method
• How to use Autodesk 3ds Max *as part of the Unreal Engine
• A description of the coordinate system in Unreal: X-Y-Z, Euler (Pitch-Yaw-Roll) and Quaternion
• How to enable the Intel RealSense SDK on the Unreal Engine
• Mapping the algorithm of the Intel RealSense SDK to the game engine

Face-Rigging within the Game

There are several ways to modify the underlying bone structure of characters, which we call face-rigging, in the game.

• Animation with script. Pre-defined animation is the normal method in games, but it is hard to implement for real-time face-rigging. If you want to make a simple emotion expressed on a face, this method would be the best and easiest way. You can control animation using an Unreal script or Matinee.
• Commercial face-rigging tool – FaceFX*. FaceFX is Unreal’s commercial face-rigging tool from OC3 Entertainment (https://www.facefx.com). It is prelicensed on Unreal Engine 3. FaceFX incorporates full body and face changes for characters.
• Morph targeting. The previous Intel RealSense SDK with Unity face-rigging sample code (named Angie) used Morph. It is a simple way to implement the Intel RealSense SDK within the game, but it has to make a morph target for each character. In the case of this MMORPG, there are from three to six tribes, and the player can modify a character’s face and body, so there are several thousands of combinations. It requires several thousands of Morph face resources and make less performance compared to bone manipulation.
• Bone manipulation. If the Intel RealSense SDK can determine bone manipulation, it should be a good method on a real game. Even with several thousands of character combinations, there are comparatively few face structures (tribes x males/females). Also, this method will not impact on rendering and has minimum impact on gaming performance.

For example, the MMORPG game Bless* (http://bless-source.com/, http://bless.pmang.com/) has 10 tribes but there are only eight face bones types for Elf (male/female), Lupus (male/female), Human (male/female), and Mascu (male/female). A full list of bone names is available in the Face Bone Structure section at the end of the document.

Figure 1: Game characters in MMORPG.

Environment

• Tested machine: Intel® Core™ i7-5775C processor, 16 GB DDR, 256 GB solid-state drive (SSD). Recommended machine: higher CPU like the 6th generation Intel® Core™ i7 processor (code-named Skylake) and a high-performance external graphic card and SSDs with large space storage (more than 50 GB). SSDs rather than hard-disk drives are recommended for I/O bandwidth. Intel® RealSense™ camera (F200) or 2D web camera.
• Microsoft Windows* 7 64 bit
• Autodesk 3ds Max 2014
• Epic Games Unreal Engine 3 source code (required license)
• Microsoft Visual Studio* 2010
• Intel RealSense SDK 6.0.21.6598 and Intel® RealSense™ Depth Camera Manager 1.4.27.41944

Setup procedure

1. Clean install Windows 7 64 bit on the machine and update Windows and drivers for each device
2. Copy the UE3 source code to the local drive.
3. Install Microsoft Visual Studio 2010 and update it. A debugging script must be included if you need debugging on Visual Studio 2010. Refer to the backup – Autoexpand setup for debugger.
4. (Optional) Install Autodesk 3ds Max 2014 if you need to export the FBX file from the MAX file.

Export MAX File to FBX File for Importing UE3

Most common 3D modeling tools like Autodesk 3ds Max or Maya* can export their 3D modeling to the Unreal Engine or Unity* through the FBX file format.

These steps are based on Autodesk 3ds Max 2014.

1. Open the MAX file that contains the bone structure. You can see the bone positions and outlook as well.

   
   Figure 2:Open the MAX file.

2. Export the file to FBX. Set the “by-object” mode to export it correctly if encountered the warning screen of an unsupported “by-layer” mode.

   
   Figure 3:Export to FBX.

3. Select all objects, then right-click and select “Object Properties”.

   
   Figure 4:Export option.

4. Click “By Object" button to change to “by-layer” mode

   
   Figure 5: Export option – the by-layer mode.

5. Select Menu, and then select Export. Enter the export name in the FBX export option. You can select animation and bake animation to test the animation in UE3.

   
   Figure 6:Export with animation.

Import the FBX File into the UE3 Editor
If you are using standalone type UE3 with standard option (DirectX* 9, 32-bit), you can run the Unreal editor.

1. Run the UE3 Editor with the following commands:

   Run CMD and go to your UE3 folder (in my case, C:\UE3)
   Go to \Binaries\Win32\
   Run examplegame.exe editor -NoGADWarning

   
   Figure 7: Unreal Editor startup.

2. In Content Browser, click Import, and then select the FBX file. Click OK to import. Once imported, you can see Animset, SkeletonMesh, and others.

   
   Figure 8:Unreal Editor - Content Browser.

3. To check your imported FBX, right-click Animset and then select Edit Using AnimSet Viewer.

   
   Figure 9:Unreal editor - AnimSet Viewer.

4. You can adjust the scale and position of the face using the mouse buttons (left: rotation, middle: position, right: zoom). You can see the bone names on the left side and skeletons on the right side. If you play the animation, the time frame and delta of position and rotation are also visible.

   
   Figure 10:AnimSet Viewer.

   
   Figure 11:AnimSet Viewer - Adjust scale.

5. Select the bone you want (these images use BN_Ear_R 53) and the X-Y-Z coordinate system. To move, drag each X-, Y-, or Z-axis arrow.

   
   Figure 12:AnimSet Viewer - Check Bone

6. To test rotation with Euler (pitch-yaw-roll), press the space bar. Changing the coordinate system displays the Euler coordinate system on the right ear. You can adjust the rotation as you drag each P-Y-R circle.

   
   Figure 13:Change the coordinate system.

Map and Level Creation in UE3 Editor

You can skip this section if you plan to use another existing map file or level. The steps in this section will make simple cubes and light, camera, and actor ( face bone).

1. Run the UE3 Editor using the following commands:
   
   Run CMD and go to your UE3 folder (in my case: C:\UE3)
   Go to \Binaries\Win32\
   Run examplegame.exe editor -NoGADWarning

2. Use one of the template levels, or make yourself a super basic level. Right-click the BSP Cube button. In the pop-up, enter 1024 for X, Y, Z, and enable “Hollow?” Then on the left toolbar, click Add.

   
   Figure 14: Unreal Editor - make layer.

3. Fly into the cube using the WASD/arrow keys and the mouse, or alternatively drag around while holding the left/right/both mouse buttons to move the camera.

   
   Figure 15: Unreal Editor – start location.

4. To add the game start location, right-click the floor, and then select Add Actor, then select Add Playerstart.

5. To add light, right-click the wall, and then select Add Actor, then select Add Light(Point).

   
   Figure 16: Unreal Editor - Add light.

6. To add an actor: face bone, press the Tab key to move to the contents browser and drag the skeleton mesh into the UE editor.

   
   Figure 17: Unreal Editor - Add Actor.

7. To adjust scaling, enter a scaling number on the bottom right.

8. To adjust position, select the Translation mode icon on the upper-left side; move your character with X-Y-Z.

9. To adjust rotation, select the Rotation mode icon on the upper-left side; move your character with P-Y-R.

   
   Figure 18: Unreal Editor - Adjust rotation.

10. Save the level with a name. In this instance, I used “test.umap” in <UE source>\ExampleGame\Content\Maps

    
    Figure 19: Unreal Editor – save.

11. Finally, build it all. From the menu, select Build, and then select Build All.

12. To check your map, click Play or press Alt+F8.

    
    Figure 20: Unreal Editor – Build.

13. Save and exit the UE Editor.

About the Authors

Chunghyun Kim is an application engineer in the Intel Software and Services Group. He focuses on game and graphic optimization on Intel® architecture.

Peter Hong is an application engineer at the Intel Software and Services Group. He focuses on enabling the Intel RealSense SDK for face, hand tracking, 3D scanning, and more.

Part 2