Interactive games all have some functionality that game developers must implement: game objects should adhere to the laws of physics and collide with other game objects; a game object should trigger events, such as playing a sound and counting the score; and it should respond to user inputs like the joystick, mouse, and keyboard. Oftentimes, developers must re-implement this functionality for every target platform, which can be time-consuming. To make life easier, developers can choose a game engine that includes functions to perform common tasks, freeing them to focus on the more creative aspects of their game.
The cross-platform game engine Unity* 3D from Unity Technologies is an attractive solution. Its targeted platforms are computers, game consoles, mobile devices, and web browsers. Unity 3D also supports different programming languages like C++, C#, Unity Script (similar to JavaScript*), and Boo.
This article is appropriate for both beginner and experts. For those who have never worked with Unity before, there is a short introduction in the first part of the article. Following that, I demonstrate how to use hand tracking with the Intel® RealSense™ SDK to develop a little game in C# with the Intel RealSense 3D Camera. To participate you`ll need: Visual Studio and Unity installed on your PC, the Intel RealSense SDK, and the Intel RealSense 3D camera.
Download, Install, and Configure Visual Studio*
You can download Unity for free. Unity runs on the Microsoft Windows* OS and Apple Mac* OS X*. After Unity is installed, you should complete the free registration process.
The code shown in this article is modelled on the C# programming language. The default cross-platform IDE Unity provides is MonoDevelop. If you are a .NET developer, you’ll be happy to know that Unity and Microsoft Visual Studio work perfectly together. The first step is to connect Unity with Visual Studio. To do this, you have to select Edit -> Preferences… from the drop-down menu. A Unity Preferences dialog will open. Choose External Tools on the left side (Figure 1). For the external script editor click on MonoDevelop within the ComboBox and find the Visual Studio .exe file using the Browse… button. Install Visual Studio using the default settings; the location of the program should be:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\devenv.exe.
These are all necessary steps to successfully connect.
Figure 1. Connect Visual Studio* with Unity*
The Environment
Now is a good time to configure certain features of the Unity environment. The main screen consists of different windows. To find your ideal arrangement either choose the settings manually or use the default settings. Let's choose the popular setting Tall from the Layout drop-down menu in upper right.
Figure 2. Choose the position of the window
The Scene Window and the Game Window
The first big scene window (Figure 3) is used for positioning and to lay out your game world. The standard for visualizing content is in 3D. Game objects can be positioned easily via drag & drop. You can see the game tab directly above the main screen. The game tab displays a preview of the rendered game, as it will be shown on the desired platform.
Figure 3. The scene window
Figure 4. The game window
The Hierarchy Window
The hierarchy window (Figure 5) contains the relevant content objects, for example, game objects, camera views, light objects, UI elements, and audio. New content objects are created in the menu under "GameObject" or directly in the window with "Create".
Figure 5. The hierarchy window
The Inspector Window
When a content object in the hierarchy window is selected, the inspector window displays, showing its properties (Figure 6). It’s possible to extend functions by adding components that are available on the 'add component' button. A possible component would be a rigid body for example, which gives a game object its gravity.
Figure 6. The inspector window
The Project Window
Project files such as game scenes, GameObjects, audio, images, and script files are structured in the project window (Figure 7). When selecting a file, the inspector displays the appropriate properties and preview information. Applied script files can be opened directly with their own development environment by double-clicking the file name.
Figure 7. The project window
Developing a simple game
In the early 1970s Atari founder Nolan Bushnell published the successful game named “Pong.” The principle of the game is simple and similar to that of table tennis: a ball moves back and forth on the screen. Each player controls a vertical paddle. If a player fails to return the ball, the opponent receives a point. The next steps show how to develop a Ping Pong game in Unity.
With File -> New Project create a new project named PingPongUnity.
Figure 8. Create a new project.
The Paddle
First, to add the paddles, click on Create -> 3D Object -> Cube in the hierarchy window. To change the properties for the shape and position of the paddles, you have to select the generated cube and change it in the inspector window. We change the name to Paddle1 and customize the shape in the transform section. The necessary scale-values are: X: 0.5, Y:3, Z:1. Set the Position to: X: -9, Y: 1, Z: 0. The first paddle is ready now. Right click on the paddle in the hierarchy window and then click Duplicate in the context menu. This creates an identical copy of the paddle. Now you only have to rename the copy to Paddle2. Then change the X-value of Position to 9.
For the paddles to have the correct physical behavior, we need to add a RigidBody component by clicking the “Add Components” button for each paddle. This allows us to move the paddles within the code, allowing the paddles to collide with the ball. If the ball collides with a paddle now, it would be hurled out of the field. That is not the behavior we want. To prevent that, we need to set the constraints property of each added RigidBody. The paddles are just allowed to move up and down, so we need to select x, y and z at the freeze rotation and only x at the freeze position.
Figure 9. The generated paddles
The Walls
For ping pong we need two walls, one above and one below, so that the ball bounces off them and remains in the area. For this, we use the cube object again. Click the Create -> 3D Object -> Cube in the hierarchy window again. The name will be WallBelow. The shape of the wall is now slightly longer and gets the scale values: X: 23, Y: 1, Z: 1.The position gets: X: 0, Y: -5, Z: 0.
To generate the second wall you only have to duplicate the current wall again and rename it to WallAbove. Now just change the Y-Position to 7.
Figure 10. The generated walls
The ball
The last and most important object of the game is, of course, the ball. Unity already offers one in the desired shape. Just click Create -> 3D Object -> Sphere in the hierarchy window. Change the name to Ball and set the Position to X: 0, Y: 1, Z: 0. To add the appropriate physics, you need to add the rigid body component using "Add Components" and remove the property selection "Use gravity”.
Figure 11. The generated ball
Change the camera perspective and let there be light
The scene display and the game window are still not ideal for our game. We need to change the camera settings. In the hierarchy window, select the standard camera Main Camera. Then change the current 3D view to a 2D view. This is done in the camera Projection properties. Change from Perspective (3D) to Orthographic (2D). So we get a perspective from the front. Also change the new Size property to 10.
If the representation of the scene window is still not ideal, you can rotate the view by clicking the coordinate’s axis.
Figure 12. The rotation of the view will be enabled by clicking the coordinate’s axis.
Now, the camera is equipped with an ideal setting. The game objects themselves are still quite dark. To add some light, switch to the hierarchy window and add the light by clicking on Create -> Light -> Directional Light. You can see the changes in the game window now.
Figure 13. The game in the game window
Let’s move the ball
The field is ready, but it still needs some action. If you run the game, the ball falls to the ground and that’s it. What we need are a few lines of C# code to bring the ball in motion. To do this, we create a C# script file named “Ball” by clicking the Create button in the project window. Then we create a binding between the script file and the game object. This is done by dragging & dropping the file to the game object in the hierarchy or scene window. If you select the ball object, the script properties are shown in the inspector window.
Double-clicking the C# file automatically starts the development environment you set up, either MonoDevelop or Microsoft Visual Studio. The generated C# class begins with a standard structure. This includes the MonoBehavior base class, which is an interface to the Unity API. The “Start” method is used for a one-time initialization and preparation. Game frameworks are working with several pictures per second, just like when you’re watching TV or a flip-book. A single picture is called a frame. The “Update” method is called every time before a frame is drawn. This repetition is what we call a game loop, which means that other game objects can act irrespective of the player actions.
In our game we want to set the direction in which the ball may fall, once at the start of the game. The “Start” method seems to be perfect for this. You can find the required code in Listing 1.
using UnityEngine;
public class Ball : MonoBehaviour
{
private float _x;
private float _y;
// Use this for initialization
void Start ()
{
_x = GetRandomPositionValue();
_y = GetRandomPositionValue();
GetComponent<Rigidbody>().velocity = new Vector3(Random.Range(5,10) * _x, Random.Range(5,10) * _y, 0);
Debug.Log("x: " + _x);
Debug.Log("y: " + _y);
}
private float GetRandomPositionValue()
{
if (Random.Range(0, 2) == 0)
{
return -1;
}
return 1;
}
// Update is called once per frame
void Update ()
{
}
}
Listing 1. Let the ball fall in one direction with C#
The real magic of the game object is hidden behind the components section. There is one component for every possible game logic. The C# code in Listing 1 accesses the Rigidbody component of the current game object by using the GetComponent method. Rigidbody expands the game object with the gravity functionality. When setting the velocity property, a new Vector3 instance is assigned. This defines a random direction of the current gravity.
Testing the game again shows that the ball randomly falls in different directions. But if the ball collides against a wall, it slides up off along the screen, which is not how we want it to behave. It should be more like a rubber ball that bounces off the walls.
For this behavior, the Physic Material is suitable. We create a Physic Material by clicking the Create button in the project window. We change the name to Bouncy. There are some adjustments to the properties required for the desired bounce behavior. In the inspector window, the properties Dynamic Friction and Static Friction are given the value 0. The Bounciness is set to 1. The Friction Combine requires the minimum value, and Bounce Combine, the maximum.
Figure 14. Adding and setting up the Physic Material
For all game objects to use our generated Physic Material, you must set the general physic settings. To do this click on the menu Edit -> Project Settings -> Physics. The Physic Material Bouncy is dragged & dropped in the Default material property. The property Bounce Threshold needs to be set to 0.1. If you start the game now, the ball behaves like a rubber ball.
Figure 15. Set “Bouncy“ within the PhysicsManager
Control the paddle with your hand
The game is close to being finished. The game objects are where they belong and the ball is in motion. The only thing that’s missing now is the logic to control the paddles. For this, there’s nothing better than using an innovative input device, namely a 3D camera, which became popular on the Microsoft Xbox* with Kinect*. Some of the latest tablets and Ultrabook™ devices are being equipped with 3D cameras. If you’re don’t already own one, these devices are definitely affordable. For development you’ll need to download the free Intel RealSense SDK. After the installation you can find some Unity samples in the Intel RealSense SDK folder.
Figure 16– The Intel RealSense SDK samples
Unity 3D supports third-party plug-ins, and one has been created for the Intel RealSense SDK. To include the Intel RealSense SDK it’s necessary to add the SDK DLL files to the project. In Unity select Assets -> Import package -> custom package.... Then choose the RSUnityToolkit.unitypackage file, which is under C:\Program files (x 86) \Intel\RSSDK\framework\Unity. A dialog window like the one shown in Figure 17 will then open. All of the files are selected by default. Click None to clear the selections and select only the plug-ins directory. If you’re developing on a 64-bit computer, you still need to replace the imported DLL files manually. You’ll find them in the following directory: C:\Program Files (x86)\Intel\RSSDK\bin\x64
Figure 17– Integrate the Intel RealSense SDK into Unity* by using "import package"
After the integration, it’s time to start the development of the paddle logic. We need another C# file, which can be generated by clicking the Create button in the project window. The name of the file is Paddle. Only one general instance needs to be generated during the game to respond to both paddles. This is very important because the Intel RealSense 3D camera can only connect with one instance, not multiple. For this reason, we’ll connect the new file by dragging & dropping it to the "main camera"game object in the hierarchy window.
Next double-click the Paddles.cs file. The paddle game objects are required, even though the file is connected with the main camera. The connection between multiple game objects in Unity is simply working via the properties. In C# you need to declare the additional properties using “get” and “set”, which is not necessary in Unity. All we need is to create one public variable of type GameObject for each paddle, as shown in Listing 2. The game objects can now be dragged & dropped to the created paddle properties to connect them.
public class Paddles : MonoBehaviour
{
public GameObject Paddle1;
public GameObject Paddle2;
..
Listing 2 – Self written properties
Figure 18. Declare game objects for the Paddles.cs file with Unity*
To create a permanent connection with the camera, the Intel RealSense SDK offers a class named PXCMSenseManager. The start function will only be called once from the Unity engine, so it’s ideal for the general preparation of an instance. That’s why we initialize the PXCMSenseManager class within the start function. For that purpose, we’ll give it a QueryHand module first, which allows us to read hand gestures. To recognize the face and voice, a separate module is needed. The instance to the camera is declared outside the start function as _pxcmSenseManager field. The code is shown in Listing 3.
public class Paddles : MonoBehaviour
{
public GameObject Paddle1;
public GameObject Paddle2;
private PXCMSenseManager _pxcmSenseManager;
private PXCMHandModule _pxcmHandModule;
// Use this for initialization
private void Start()
{
_pxcmSenseManager = PXCMSenseManager.CreateInstance();
if (_pxcmSenseManager == null)
{
Debug.LogError("SenseManager Initialization Failed");
}
else
{
pxcmStatus pxcmResult = _pxcmSenseManager.EnableHand();
if (pxcmResult != pxcmStatus.PXCM_STATUS_NO_ERROR)
{
Debug.LogError("EnableHand: " + pxcmResult);
}
else
{
_pxcmHandModule = _pxcmSenseManager.QueryHand();
_pxcmSenseManager.Init();
PXCMHandConfiguration configuration = _pxcmHandModule.CreateActiveConfiguration();
configuration.EnableAllGestures();
configuration.ApplyChanges();
configuration.Dispose();
}
}
}
...
Listing 3. Connect permanently with the Intel RealSense 3D camera
A direct and up-to-date query of the camera data takes place in the update function (game loop) so we can determine if the camera sees a hand or not. The left hand is used for the left paddle and the right hand for the right paddle. Therefore, you need to assign the computed value to the respective paddle. Because the logic is very similar, we create a private function named MoveBall for it.
If the player quits or restarts the game, the direct instance to the camera needs to be disconnected. The OnDisable function, which will be called automatically by the Unity engine, is perfect in this case.
// Update is called once per frame
private void Update()
{
if (_pxcmSenseManager == null)
{
return;
}
_pxcmSenseManager.AcquireFrame(false, 0);
_pxcmHandModule = _pxcmSenseManager.QueryHand();
PXCMHandData handData = _pxcmHandModule.CreateOutput();
handData.Update();
MoveBall(handData, PXCMHandData.AccessOrderType.ACCESS_ORDER_LEFT_HANDS, Paddle1);
MoveBall(handData, PXCMHandData.AccessOrderType.ACCESS_ORDER_RIGHT_HANDS, Paddle2);
_pxcmSenseManager.ReleaseFrame();
}
private void MoveBall(PXCMHandData handData, PXCMHandData.AccessOrderType accessOrderType, GameObject gameObject)
{
PXCMHandData.IHand pxcmHandData;
if (handData.QueryHandData(accessOrderType, 0, out pxcmHandData) ==
pxcmStatus.PXCM_STATUS_NO_ERROR)
{
PXCMHandData.JointData jointData;
if (pxcmHandData.QueryTrackedJoint(PXCMHandData.JointType.JOINT_CENTER, out jointData) ==
pxcmStatus.PXCM_STATUS_NO_ERROR)
{
gameObject.GetComponent<Rigidbody>().velocity = new Vector3(-9, jointData.positionWorld.y*100f, 0);
}
}
}
private void OnDisable()
{
_pxcmHandModule.Dispose();
_pxcmSenseManager.Dispose();
}
}
Listing 4. Move a paddle with your hand
Restart the game
The game objects are generated, the logic is all coded, and now it’s time to test the game. If a hand is recognized in a 1 to 5 meter range from the camera, the paddles will move. If the ball hits one of the paddles, it will bounce off and move in the opposite direction. If the ball leaves the field without touching a paddle, there’s nothing the desperate player can do. In that case, the game should restart automatically if the ball leaves the game field.
Only a few lines of code are necessary to implement that logic in the Ball.cs file. The Update function should check if the ball is still in sight. If the ball is out of sight, the Application.LoadLevel function restarts the game, as shown in Listing 5.
private bool _loaded;
// Update is called once per frame
void Update ()
{
if (GetComponent<Renderer>().isVisible)
{
_loaded = true;
}
if (!GetComponent<Renderer>().isVisible && _loaded)
{
Application.LoadLevel(Application.loadedLevel);
}
}Listing 5. Ball.cs: Restart the game, if ball is out of sight
Publish the game
The first version of the game is ready and of course we want to share it with others. The menu item File | Build Settings (Ctrg + Shift + B) opens the Build dialog. You can choose which platforms you want your game to run from a huge selection in the Platform box. We will choose Windows and set x86_64 as architecture. Simply click Build to start the compiler and generate an .exe file for Windows. To play the game, you need to have an Intel RealSense 3D camera including the pre-installed driver.
Figure 19. Create an .exe file to publish the game
Now it’s your turn
I’ve shown you how to create a fun game, but it could still use some perfecting. So show me what you’ve got. Build in a high score, sounds, and some cool special effects. Upload a video on YouTube*, and send me an e-mail so I can see what you’ve created. I’m excited about your ideas. And now, have a lot of fun with your first Unity game and the Intel RealSense 3D camera.
Figure 20. Playing the game
Figure 21. Game UI
About the Author
Gregor Biswanger (Microsoft MVP,Intel Black Belt, and Intel Software Innovator) is the founder of CleverSocial.de and a freelance lecturer, consultant, trainer, author, and speaker. He advises large and medium-sized companies, organizations, and agencies on software architecture, agile processes, XAML, Web- and hybrid-app development, cloud solutions, content marketing, and social media marketing.
He has published several video trainings about these topics at video2brain and Microsoft. He’s also a freelance author and writes online for heise Developer as well as articles for trade magazines. He has penned the book “cross-platform development” from entwickler.press.
You can find Gregor often on the road attending or speaking at international conferences. In addition Intel asked him to be a technology consultant for the Intel® Developer Zone. He’s the leader of the INdotNET (Ingolstaedter .NET Developers Group).You can find his contact information here: https://about.me/gregor.biswanger/