Download PDF
[PDF 832KB]
By Garret Romaine
After their Space Between* demo won second place in the Intel® Perceptual Computing 2013 Challenge, Justin Link and Ryan Clark, the two-man coding team at Chronosapien Interactive, quickly upgraded to the newer Intel® RealSense™ SDK. They now have a soon-to-be released commercial title that takes full advantage of gesture controls available with the Intel RealSense SDK.
Using the Unity3D* engine plus some clever coding solutions, the result is an exciting game that highlights some key capabilities of Intel® RealSense™ technology. In a stunning mix of sound, art, and inputs, Space Between allows a player to guide sea creatures through a series of tasks in an underwater world (Figure 1). The work of Link and Clark, which showcases Intel RealSense technology and highlights best practices and smart design considerations, is worth a deeper look.
Figure 1. In the realm of Space Between* from the demo here.
Users Mimic Swimming Motions to Guide Sea Creatures
Gameplay for Space Between is ingeniously simple. Player movement is monitored by the Intel® RealSense™ 3D cameras, which interpret the motions and use the feedback to guide turtles, fish, and other denizens of the deep. The Chronosapien team created several scenarios that challenge users to learn new motions, which in turn enabled the team to gather some interesting data points on how to implement Intel RealSense technology.
Figure 2. In this scene from the Sunlight zone portion of their demo, cofounder Ryan Clark shows how moving his hand influences a swimming sea turtle.
The full commercial title incorporates five modules, each representing a different zone of the ocean and using different gestures:
- Sunlight zone. Use your open palm to guide a turtle through the water (Figure 2), snatching points as it moves.
- Twilight zone. Use head motions in the deep, darker environment, moving a whale and capturing points.
- Midnight zone. Use gesture control to move a school of fish around. Open and close your fist to control the size and speed of the school.
- Abyss. Use gesture control to catch points and avoid predators.
- Trenches. Use gesture control in both hands, guiding two robotic arms that are used to combat deep sea crabs and collect artifacts.
Figure 3. Space Between* uses recognizable icons to guide the user through the level-selection process.
Key to Good Gameplay: User Analysis
The team put special emphasis on the user experience as it relates to developing for the gestures. For example, they initially used a side-facing hand with a swing gesture to guide a swimming turtle in the Sunlight zone. Initially, Link and Clark thought the motion would be similar to riding in a car and holding a hand out in the wind, surfing the air, and would easily translate to a swimming motion as the user guided the turtle. However, they found that in practice this gesture was far too tiring on the shoulder. After some online research, the team found that the gesture is a difficult one to expect users to perform repeatedly. “You can only keep it up about 90 or so seconds,” Link said. So they replaced that specific move.
According to Clark, they originally began developing with the early versions of the Intel® Perceptual Computing SDK code base. The Intel Perceptual Computing SDK was limited to monitoring 2D face positions, and because orientation wasn’t available in the Unity SDK, 3D hand tracking was limited to just position data in Unity.
“At the time, we couldn't track the angle or the orientation of the hand without using a separate block for tracking, which wasn't an option for us. Over time, with the turtle moving, it helped to create a sine wave, which is where we got the rhythm of the game. As we were able to get [Intel] RealSense [technology] in, we transitioned to using palm orientation.”
Link also had some advice about testing, which is to put as many people as possible in front of the game to get the most valuable feedback.“ We put anybody we could find in front of a screen, at every opportunity we got,” Link said. “It was challenging because we were basically testing five different games at once. It took us a while to get that first iteration where we could test all of them. We didn't have the resources for a large-scale QA effort. But we finally got to a place where we used our friends to check specific features that we were looking for in terms of testing and validation.”
The team also faced the challenge of how to educate users about the gesture controls (Figure 4). While allowing the user to stumble a bit initially can lead to a greater sense of triumph once the interface is mastered, this can also lead to frustration, causing the user to give up and walk away. With no real precedent on how to train users with gesture control, the team faced a big hurdle.
Figure 4. In this scene from the Midnight zone portion of their demo, cofounder Ryan Clark shows how he opens and closes his fist to control a school of fish.
After debating how much guidance to give the user and how much to assume the user already knows based on the technical capabilities of the camera, the team settled on creating an instruction screen.
“It’s like a fact sheet (Figure 5) because Space Between has a documentary feel to it,” said Link. They also show the user icons, which represent the types of modalities they will be using, such as 2D position tracking, 2D openness, or head-position tracking. “We don't tell them exactly what it is that they're doing inside of the game, so they still have some discovery, but we tell them how they can control it so they can determine what it should be responding to.”
Figure 5. Space Between* features several instruction screens to teach the user how to play.
Link and Clark also learned through the demo process and testing that having both visual and aural feedback helps the user realize that something is happening.
“With gesture cameras, you don't have tactile feedback like you do with the mouse and keyboard,” said Clark. “You must do a lot more in the application to explain that the game is responding to something the user is doing so that they know they're moving correctly. Users need that feedback.”
Speech Recognition and False Positives
When using speech recognition, a device’s microphone picks up sound from the user in their environment. The problem with this is that the user is often not the only sound generator in the environment. This problem is exacerbated with games because the application itself is producing sound effects and music, which the microphone picks up unless the user is using headphones. In the Space Between demo, the result was that either speech wasn’t being detected when it needed to be, or it was detected when it shouldn’t be. Because of the false positives, they focused instead on gesture control and palm orientation, and rather than using complete facial recognition they tracked head movements.
Managing Data Load with Reusable Modules
One key design element the team implemented was to streamline data handling (Figure 6). Instead of requesting and printing the data as needed, they created a manager class that gets all the Intel RealSense SDK data that the subscribers might require. The manager also formats the data and smoothes it as necessary. The result is that any data the game requests has already been handled appropriately.
public void UpdateHands(PXCMHandData HandOutput) {
// Query the width of the map so we know what the position data actually means.
if (RightHand.ScreenSize == default(Vector2)) {
// RealSenseInputManager handles gathering raw data from the RealSense SDK.
Vector2 depthFeedSize = RealSenseInputManager.Instance.ImageSize(PXCMCapture.StreamType.STREAM_TYPE_DEPTH);
RightHand.ScreenSize = depthFeedSize;
LeftHand.ScreenSize = depthFeedSize;
}
// Do hand node tracking
PXCMHandData.IHand handData = null;
// Right hand handling
if (HandOutput.QueryHandData(PXCMHandData.AccessOrderType.ACCESS_ORDER_RIGHT_HANDS, 0, out handData) >= pxcmStatus.PXCM_STATUS_NO_ERROR) {
// Position broadcasting
PXCMPointF32 handPosition = handData.QueryMassCenterImage();
Vector3 handPositionFormatted = new Vector3(handPosition.x, handPosition.y, 0.0f);
BroadcastViewportPosition(RightHand, handPositionFormatted);
// Switch the depth back to millimeters for consistency
BroadcastDepth(RightHand, (int) (handData.QueryMassCenterWorld().z * 1000);
//...
}
//...
}
/// <summary>
/// Takes in a Vector3 corresponding to camera coordinates and converts it to viewport position. Broadcasts this value to any subscribers.
/// </summary>
/// <param name="BodyPart">The body part to broadcast for (left hand, right hand, face).</param>
/// <param name="Position">PXCUPipeline's PXCMPoint3DF32 converted to Vector3.</param>
private void BroadcastViewportPosition(PositionInfo BodyPart, Vector3 Position) {
// Unity uses (0,0) as the bottom-left position, but the RealSense SDK uses it as the top-right. (Exception is a non-mirrored camera.)
float x = isInvertingInput ? Position.x / BodyPart.ScreenSize.x : Mathf.Abs(1.0f - Position.x / BodyPart.ScreenSize.x);
float y = Mathf.Abs(1.0f - Position.y / BodyPart.ScreenSize.y);
Position = new Vector3(x, y, 0.0f);
// Add the new (unsmoothed) position to the body part's instance of PositionInfo. PositionInfo handles smoothing of data.
BodyPart.AddPosition(Position);
// If there are any subscribers, broadcast the smoothed (averaged) position, which is calculated in PositionInfo.
if (BodyPart.PositionCaller != null)
BodyPart.PositionCaller(BodyPart.AveragedPosition);
}
Figure 6. Chronosapien formatted and smoothed its data prior to handling, resulting in less formatting.
“We don't have to constantly figure out how to change that for each of our games,” Link said. “We can do it just once and then apply it for each. Using compartmentalized code means you don’t have to format everything repeatedly.”
During the design process, it’s also important to get the code in manageable forms (Figure 7) before you start implementing into the game. “We like to call them managers,” Clark said. “They are hailing data. In each game, when you're handling that data, it doesn't matter whether it's coming from a game pad or a gesture. You're implementing it the same way.”
public override void SubscribeInput() {
if (InputType == InputTypes.RealSense) {
RealSenseInputProcessor.Instance.RightHand.PositionBroadcast += PositionPrimary;
RealSenseInputProcessor.Instance.RightHand.DepthBroadcast += DepthPrimary;
RealSenseInputProcessor.Instance.RightHand.OpennessBroadcast += OpennessPrimary;
RealSenseInputProcessor.Instance.RightHand.OrientationBroadcast += FacingPrimary;
}
else if (InputType == InputTypes.Controller) {
GamepadInputProcessor.Instance.JoystickLeft.PositionBroadcast += PositionPrimary;
GamepadInputProcessor.Instance.TriggerRight.PositionBroadcast += DepthPrimary;
GamepadInputProcessor.Instance.TriggerLeft.PositionBroadcast+= OpennessPrimary;
GamepadInputProcessor.Instance.JoystickRight.PositionBroadcast += FacingPrimary;
}
}
Figure 7. Using data managers compartmentalized the code and made it easier to manipulate certain modules without breaking things elsewhere.
Link said that learning to use singletons and data managers early in his career paid off when the time came to work with the Intel RealSense SDK and different parts of the reference data. “It's a fairly common technique in application development to have a single place and have your code be modular, so that you can upgrade one part of it without breaking another. You don't want to process or format the data for every use. It's nice to set up something that is already doing that so when you drop the application on top, the app is ready to go. Likewise, you don't have to change anything if the input does change.”
In one instance, the team’s foresight helped to streamline the development cycle. They knew that halfway through the project cycle they would need to switch to the Intel RealSense SDK from the original Intel Perceptual Computing SDK code. All the code snippets that fit Intel Perceptual Computing were their own special module. Thus, when they did the Intel RealSense SDK module, they put them side-by-side. The team easily turned modules on and off to verify that everything worked the same way. Then as they finished working with the Intel Perceptual Computing SDK, they removed it without having to change anything throughout the other scripts.
Clark and Link appreciated their interaction with Intel’s support team. “Honestly, I think that one of the best parts about this project has been working with Intel. Everyone has been incredibly nice and helpful. From our perspective, they did everything they could to support us,” said Clark.
Link added that other contest winners were also a big help. They became fast friends with Jacob and Melissa Pennock, founders of Unicorn Forest Games, who entered their spell-casting game, Head of the Order*, into the Intel Perceptual Computing Challenge. “We met them at CES; they actually beat us in the contest, but we've been talking with them ever since because they're doing something very similar in terms of a game in Unity* using [Intel] RealSense [technology]. Usually they helped us, and vice versa.”
Contest Entry Was Crucial for Expertise
The team said that entering the Intel Perceptual Computing Challenge contest was an important way to jump-start learning the technology. Their advice for anyone thinking of getting started is to dive right in. “Just get in there and check out the samples, and get your hands dirty as quickly as possible,” Link said. “See what the capabilities are. After that, you can start messing around with the toolkit to see the features. Then once you hit the wall of where you can get with the out-of-the-box features, start pushing it by dipping into the back-end of that experience itself.”
Clark agreed. “One important thing that we found when we designed these games is that a lot of the gameplay actually came from existing games that people already know. I recommend that you come up with a design based on something that you are familiar with.”
Working out of a small office, Link and Clark are pleased to be approaching the finish line. “Everything's coming together, and it's fun seeing everything as a whole,” Link said. “Recently we've been taking our game to expos, so we’re watching people finally play it out for the first time. Because it's a gesture-controlled game, it has its own uniqueness. Seeing people's responses to our work is very cool.”
Link said that the most satisfying aspect of creating Space Between has been the ability to create a truly immersive experience. He marvels at the way people get into the game and explore the virtual world that he and his partner created. “The level of immersion that you get from using [Intel] RealSense [technology] gesture controls is amazing,” he said. “We're putting everything together in one package, aligning the visual experience, the audio tracks, and the hands-on interaction for the user. There is a level of immersion that happens now that wasn't possible with traditional input media.”
The team is already planning for the next contest, even as they put the finishing touches on their current title. “It’s a brand-new idea,” Clark said. “Our next application is called The Risen, where the user takes on the role of an evil skeleton master, controlling minions to do their bidding while interacting with the world using their hands. We’ll be focusing on many of the new capabilities that are in the latest [Intel] RealSense SDK, including more emotion detection, user segmentation, and updated voice recognition.”
About the App Authors
Justin Link and Ryan Clark first worked together at the University of Central Florida in the Media Convergence Laboratory, creating interactive kiosks and simulations. From there, it was an easy step to consider designing interactive games, and the Intel Perceptual Computing Challenge spurred them to think big.
Link traces his gaming bona fides back to the early Nintendo* days, and now spends his precious gaming cycles on the Warcraft* title Hearthstone*, a digital card game. Clark enjoys gaming in person, spending his rare downtime playing Warmachine*, a popular turn-based tabletop title.
Intel RealSense Technology Links
Intel Developer Zone for Intel RealSense Technology