In this article you will learn how a mixed reality game was built using the Intel® RealSense™ Camera (Rear Facing), Intel® RealSense™ SDK, Unity 3d™ game engine and dynamically changing terrain (sand). Game makers are constantly looking for new ways to improve the user’s experience, allowing the player to become a more integral part of the game. From avatars and in-depth game options to terrain editors and customizable mini-games, developers are continually working to give users the capability to manifest their own virtual existence. Intel® RealSense™ technology unlocks new and exciting ways for players to interact with computers. TANKED!, an augmented sandbox arcade game, harnesses this technology into an exciting multiplayer battle. At Design Mill we built a customized sandbox and utilize Intel RealSense technology and an image projector to allow users to manipulate the topography of the map with their hands, dynamically creating a virtual environment via real world sand.
The Game
TANKED! is an arcade-style one-on-one tank battle video game. The players start the game by building a virtual environment using real world sand. Once the terrain is modified, a tank is dropped into the environment for each player and the game begins. Players maneuver their tank across the dynamically generated terrain using the slopes to their advantage while they dodge the other player’s projectiles. Tanks can dip into valleys for cover or climb up hills to change their cannon’s trajectory, all while chasing power-ups placed at strategic intervals throughout the game. Health boosts, increases in tank speed, and cannon fire rate may make all the difference in battle. The first player to land enough hits on his or her opponent wins the game.
However, there is an added twist: the mine sweep mini-game. Players are faced with a warning of incoming mines and the tanks stop battling. Mines are placed at random points where the Intel® RealSense™ camera has determined that the depth of the sand is sufficient for mine placement. Before time runs out each player must dig into the sand in search of the mines to deactivate them. Once uncovered, the player deactivates the mine by hitting his or her action button. If a player does not successfully deactivate all of the mines in his or her area a significant amount of damage is applied to his or her tank.
The Hardware
The Intel RealSense camera and an image projector are mounted above the rectangular sand table designed for two players. Each player has a joystick to maneuver his or her tank, an action button for firing projectiles at his or her opponent or deactivating mines, and a start button.
The Intel RealSense camera is calibrated to focus on the inside of the sandbox, ignoring the walls and anything outside of them. We then project the image of the created virtual terrain back on to the sand.
Using the Intel® RealSense™ SDK to acquire the depth data of the sand allows for completely unique haptic control over a virtual terrain. Utilizing the Intel RealSense SDK we were able to calibrate the sensor and the projector, read the sand depths, and smooth and replace invalid depths.
Based on the depth readings from the Intel RealSense SDK, the dynamically created terrain affects the path of the cannon projectile. If the tank is on a hill, the projectile will be shot up into the air at a much more drastic angle rather than straight ahead. Along with this, a player can build hills strategically for protection. Therefore, each game play is unique because of the dynamic terrain the players create using the sand. Players must be aware of their vertical position in world space and the distance to their enemy, and whether their shots will make it far enough to hit.
Development
Along with the Unity* 3D game engine, the Intel RealSense camera goes to work creating the terrain dynamically. Since the Intel RealSense camera is mounted directly above the sand table, we utilize the COORDINATE_SYSTEM_REAR_OPENCV SETTING for the sensor and capture the STREAM_TYPE_DEPTH stream. For each frame, Unity’s Update method calls the code, which instructs the Intel RealSense camera to pass the depth readings for each point on the sand table to Unity. To improve performance, the depth readings that fall into the acceptable range are averaged inside Unity over a specified number of frames. Once averaged, the depths are smoothed, rotated, offset, and cropped as necessary. These depths are then scaled up to fill an array the size of the heightmap resolution of the Unity terrain. This array of heights is then pushed to the TerrainData.SetHeights method in Unity. The custom splatmap of textures we have provided in Unity are then updated to reflect these new heights.
Finally, the dynamically generated terrain with textures and foliage is projected over the sand table; this gives each player the experience of driving his or her tank in a world they created. This process, although it may sound like a lot, actually takes no time at all, and feels seamless in the flow of the game.
During the mine sweep mini-game the terrain is recalibrated after the players have moved the sand around in search of the mines. This showcases possibilities with dynamic environments created with the Intel RealSense camera. Players can experience hidden worlds and secrets buried in the landscape they created.
/// <summary>
/// Gathers depth data from the sensor, adding it to the points you are accumulating for averaging purposes
/// If the number of frames accumulated has reached the number you want to average, the terrain itself if refreshed.
/// </summary>
private void updateTerrain()
{
if (pp == null || !autoUpdate)
return;
pxcmStatus sts = pp.AcquireFrame (false, 0);
if (sts == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
PXCMCapture.Sample sample = pp.QuerySample();
If (sample.depth != null)
{
cumulativeFramesCount = GetRGB32PixelsDepth2 (sample.depth);
if (cumulativeFramesCount >= averagePasses)
cumulativeFramesCount = RefreshData2();
}
pp.ReleaseFrame();
}
else
pp.ReleaseFrame();
}
/// <summary>
/// Populates the pointsToAverage list with the data from the sensor
/// </summary>
/// <returns> The number of frames accumulated
/// <param name=”image”>Depth sample from the sensor</param>
private int GetRBG32PixelsDepth2 (PXCMImage image)
{
PCXMImage.ImageData cdata;
UInt16[] dpixels = null;
if (image.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH, out cdata) >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
Int32 dpitch = cdata.pitches[0] / sizeof(Int16);
Int32 dwidth = (Int32) image.info.width;
Int32 dheight = (Int32) image.info.height;
dpixels = cdata.ToShortArray(0, dpitch * dheight);
image.ReleaseAccess(cdata);
cumulativeFramesCount += 1;
float testValue;
foreach (PointForAveraging point in pointsToAverage)
{
//initialize based on the coordinates
testValue = dpixels[point.heightCoord * dpitch + point.widthCoord];
if (testValue != depthLowConfidenceValue
&& testValue >= realSenseSettings.MinDepthValue&& testValue <= realSenseSettings.MaxDepthValue)
{
point.sumZValues += testValue;
point.countZValues += 1;
}
}
}
return cumulativeFramesCount;
}
Challenges
The location and position of the Intel RealSense camera and the projector are critical to the success of the game. The Intel RealSense camera has to be mounted far enough above the sand table to capture all four walls. We created a calibration utility that allows the player to select the area inside the sandbox walls by setting offsets for the x- and y-axes and the width of the area to display. These offsets specify whether or not the sensor should be mirrored and whether the depth points need to be rotated 180 degrees before displaying them on the sand. This utility calculates the height of the area to display based on the selected width and the aspect ratio of the Intel RealSense camera.
The calibration settings determine the depth points that will be utilized in the actual game. Depth points are gathered for each frame in Unity 3D. To improve game performance, these depth point readings are averaged over a specified number of frames. Low confidence readings from the depth sensor and depth points out of the specified range are ignored. Any ignored depth values are repopulated with the depth value from the closest valid pixel.
The Unity 3D terrain is a square object that we need to make fill the rectangular sandbox. Therefore, before we update the heights in the terrain we scale the valid depth points to match the terrain resolution by using the number of valid width and depth pixels versus the heightmap resolution. The scaled values are used to create an array of the heightmap resolution and then pushed to the TerrainData.SetHeights() method. Then the terrain textures and trees are refreshed based on these new heights, and the player sees their updated terrain.
Occasionally, there were holes in the terrain from bad depth data, which caused the tanks to fall through the map. We corrected the problem by reading an average of the surrounding pixels for the tank movement rather than a single point. In addition, a tank-leveling script was created allowing the tank to move smoothly along the surface.
Looking Forward
Having a dynamic environment creation framework built utilizing the Intel RealSense SDK opens up a whole new world of possibilities with the digital environments we can create. This applies to not only entertainment games, but also educational interactive experiences. Whether the gameplay made focuses on entertainment or education, a virtual sand table powered by Intel RealSense technology can provide a completely new experience to play and to learn.
The development of TANKED! has been a fun and educational process, and we look forward to continue working on this game. We will continue to further develop out the gameplay, as well as optimize for a better frame rate and seamless controls to heighten the user experience.
In addition to the development of TANKED!, we have developed and are releasing an interpretive projector system developer kit called Torch. This will allow the developer community to create their own interactive sandbox and tabletop experiences. Find out more about the Torch platform at www.ExperienceTorch.com.
Intel® Tools and Support
The Intel RealSense SDK documentation, Samples Viewer, and code examples were integral to the development of TANKED!. The Raw Streams and Capture Viewer examples portrayed the color, depth, and IR streams from the Intel RealSense camera at different dimensions and frame rates. These examples showed us what the camera could capture, so we could use it to our advantage in our game. The Unity and C# code examples provided in the SDK were invaluable as a starting point for our development efforts. The Intel Developer Zone and personal support from the Intel RealSense technology team, in conjunction with the strong community and support for the Unity gaming engine, provided the additional tools and resources to make this game a reality.
About Design Mill (www.designmillinc.com)
Design Mill, Inc. is a strategic systems integrator delivering pioneering solutions for industry-leading organizations and the Department of Defense that transforms integration of cutting-edge interactive hardware and software through innovative design, development, and processes. Focusing on next-generation initiatives for virtual reality and augmented reality, Design Mill enables clients to unlock revolutionary customer engagement approaches and untapped growth strategies to generate synergies for a new level of success.
Notices
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.
This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.
The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request.
Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.
Intel, the Intel logo, and Intel RealSense are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others
© 2015 Intel Corporation.
This sample source code is released under the Intel Sample Source Code License Agreement.