By John Tyrrell
Fusing the traditions of Western and Southeast Asian shadow-theater, Ombre Fabula is a prototype app that uses the Intel® RealSense™ SDK to create a gesture-controlled interactive shadow play. During the collaborative development process, the Germany-based team comprised of Thi Binh Minh Nguyen and members of Prefrontal Cortex overcame a number of challenges. These included ensuring that the Intel® RealSense™ camera accurately sensed different hand-shapes by using custom blob-detection algorithms, and extensive testing with a broad user base.
The title screen of Ombre Fabula showing the opening scene in grandma’s house (blurred in the background) before the journey to restore her eyesight begins.
Originally the bachelor’s degree project of designer Thi Binh Minh Nguyen—and brought to life with the help of visual design and development team Prefrontal Cortex—Ombre Fabula was intended to be experienced as a projected interactive installation running on a PC or laptop equipped with a user-facing camera. Minh explained that the desire was to “bring this art form to a new interactive level, blurring the boundaries between audience and player”.
Players are drawn into an enchanting, two-dimensional world populated by intricate “cut-out” shadow puppets. The user makes different animals appear on screen by forming familiar shapes—such as a rabbit—with their hands. The user then moves the hand shadow to guide the child protagonist as he collects fragments of colored light on his quest to restore his grandmother’s eyesight.
Ombre Fabula was entered in the 2014 Intel RealSense App Challenge, where it was awarded second place in the Ambassador track.
Decisions and Challenges
With previous experience working with more specialized human interface devices, the team was attracted to Intel RealSense technology by the breadth of possibilities it offers in terms of gesture, face-tracking and voice recognition, although they ultimately used only gesture for the Ombre Fabula user interface (UI).
Creating the UI
Rather than being adapted from another UI, the app was designed from the ground up with hand- and gesture-tracking in mind. Ombre Fabula was also designed primarily as a wall-projected interactive installation, in keeping with the traditions of shadow-theater and in order to deliver room-scale immersion.
Ombre Fabula was designed to be used as a room-scale interactive installation to maximize the player’s immersion in the experience. Here the user is making the bird form.
For the designers, the connection between the real world and the virtual one that users experience when their hands cast simulated shadows on the wall, is crucial to the immersive experience. This experience is further enhanced by the use of a candle, evoking traditional shadow plays.
The UI of Ombre Fabula is deliberately minimal—there are no buttons, cursors, on-screen displays, or any controls whatsoever beyond the three hand-forms that the app recognizes. The app was designed to be used in a controlled installation environment, where there is always someone on hand to guide the user. The designers often produce this type of large-scale interactive installation, and with Ombre Fabula they specifically wanted to create a short, immersive experience that users might encounter in a gallery or similar space. The intentionally short interaction time made direct instructions from a host in situ more practical than an in-game tutorial in order to quickly bring users into the game’s world. The shadow-only UI augments the immersion and brings the experience closer to the shadow theater that inspired it.
Here, the user moves the hand to the right to lead the protagonist out of his grandma’s house.
Implementing Hand Tracking
In the initial stage of development, the team focused on using the palm- and finger-tracking capabilities of the Intel RealSense SDK to track the shape of the hands as they formed the simple gestures for rabbit, bird, and dragon. They chose these three shapes because they are the most common to the style of shadow theater that inspired that app, and because they are the most simple and intuitive for users to produce. The rabbit is produced by pinching an O with the thumb and finger of a single hand and raising two additional fingers for the ears; the bird is formed by linking two hands at the thumbs with the out-stretched fingers as the wings; and dragon is made by positioning two hands together in the form of a snapping jaw.
The basic hand gestures used to produce the rabbit, bird, and dragon.
However, it was discovered that the shapes presented problems for the Intel RealSense SDK algorithms because of the inconsistent visibility and the overlapping of fingers on a single plane. Essentially, the basic gestures recognized by the Intel RealSense camera—such as five fingers, a peace sign, or a thumbs-up, for example—were not enough to detect the more complex animal shapes required.
As a result, the team moved away from the Intel RealSense SDK’s native hand-tracking and instead used the blob detection algorithm, which tracks the contours of the hands. This delivers labeled images—of the left hand, for example—and then the Intel RealSense SDK provides the contours of that image.
Here, the hand gestures for bird, dragon and rabbit are shown with the labeled hand contours used to allow Ombre Fabula to recognize the different gestures.
At first, extracting the necessary contour data from the Intel RealSense SDK was a challenge. While the Unity* integration is excellent for the hand-tracking of palms and fingers, this wasn’t what was required for effective contour tracking. However, after spending time with the documentation and working with the Intel RealSense SDK, the team was able to pull the detailed contour data required to form the basis for the custom shape-detection.
The user forms a rabbit-like shape with the hand in order to make the rabbit shadow-puppet appear on-screen.
The user then moves the hand to advance the rabbit through the game environment and collect the dots of yellow light.
Using the Custom Blob Detection Algorithm
Once the blob data was pulled from the Intel RealSense SDK, it needed to be simplified in order for the blob detection to be effective for each of the three forms—rabbit, bird, and dragon. This process proved more complex than anticipated, requiring a great deal of testing and iteration of the different ways to simplify the shapes in order to maximize the probability of them being consistently and accurately detected by the app.
// Code snippet from official Intel "HandContour.cs" script for blob contour extraction
int numOfBlobs = m_blob.QueryNumberOfBlobs();
PXCMImage[] blobImages = new PXCMImage[numOfBlobs];
for(int j = 0; j< numOfBlobs; j++)
{
blobImages[j] = m_session.CreateImage(info);
results = m_blob.QueryBlobData(j, blobImages[j], out blobData[j]);
if (results == pxcmStatus.PXCM_STATUS_NO_ERROR && blobData[j].pixelCount > 5000)
{
results = blobImages[j].AcquireAccess(PXCMImage.Access.ACCESS_WRITE, out new_bdata);
blobImages[j].ReleaseAccess(new_bdata);
BlobCenter = blobData[j].centerPoint;
float contourSmooth = ContourSmoothing;
m_contour.SetSmoothing(contourSmooth);
results = m_contour.ProcessImage(blobImages[j]);
if (results == pxcmStatus.PXCM_STATUS_NO_ERROR && m_contour.QueryNumberOfContours() > 0)
{
m_contour.QueryContourData(0, out pointOuter[j]);
m_contour.QueryContourData(1, out pointInner[j]);
}
}
}The contour extraction code used to pull the contour data from the Intel RealSense SDK.
The simplified data was then run through a freely available software algorithm called $P Point-Cloud Recognizer*, which is commonly used for character recognition of pen strokes and similar tasks. After making minor modifications and ensuring its good functioning in Unity, the developers used the algorithm to detect the shape of the hand in Ombre Fabula. The algorithm decides–with a level of probability of around 90 percent–which animal the user’s hand-form represents, and the detected shape is then displayed on screen.
// every few frames, we test if and which animal is currently found
void DetectAnimalContour () {
// is there actually a contour in the image right now?
if (handContour.points.Count > 0) {
// ok, find the most probable animal gesture class
string gesture = DetectGestureClass();
// are we confident enough that this is one of the predefined animals?
if (PointCloudRecognizer.Distance < 0.5) {
// yes, we are: activate the correct animal
ActivateAnimalByGesture(gesture);
}
}
}
// detect gesture on our contour
string DetectGesture() {
// collect the contour points from the PCSDK
Point[] contourPoints = handContour.points.Select (x => new Point (x.x, x.y, x.z, 0)).ToArray ();
// create a new gesture to be detected, we don't know what it is yet
var gesture = new Gesture(contourPoints, "yet unknown");
// the classifier returns the gesture class name with the highest probability
return PointCloudRecognizer.Classify(gesture, trainingSet.ToArray());
}
// This is from the $P algorithm
// match a gesture against a predefined training set
public static string Classify(Gesture candidate, Gesture[] trainingSet) {
float minDistance = float.MaxValue;
string gestureClass = "";
foreach (Gesture template in trainingSet) {
float dist = GreedyCloudMatch(candidate.Points, template.Points);
if (dist < minDistance) {
minDistance = dist;
gestureClass = template.Name;
Distance = dist;
}
}
return gestureClass;
}This code uses the $P algorithm to detect which specific animal form is represented by the user’s gesture.
Getting Player Feedback Early Through Testing and Observation
Early in the development process, the team realized that no two people would form the shapes of the different animals in exactly the same way—not to mention that every individual’s hands are different in size and shape. This meant that a large pool of testers was needed to fine-tune the contour detection.
Conveniently, Minh’s situation at the university gave her access to just such a pool, and approximately 50 fellow students were invited to test the app. For the small number of gestures involved, this number of testers was found to be sufficient to optimize the app’s contour-detection algorithm and to maximize the probability that it would display the correct animal in response to a given hand-form.
Moving the hands left or right moves the camera and causes the protagonist to follow the animal character through the shadow world.
In addition to creating something of a magical moment of connection for the user, the simulated shadow of the user’s hands on screen delivered useful visual feedback. During testing, the developers observed that if the animal displayed was other than the one the user intended, the user would respond to the visual feedback of the shadow on screen to adjust the form of their hands until the right animal appeared. This was entirely intuitive for the users, requiring no prompting from the team.
A common problem with gesture detection is that users might make a rapid gesture—a thumbs up, for example—which can give rise to issues of responsiveness. In Ombre Fabula, however, the gesture is continuous in order to maintain the presence of the animal on the screen. Testing showed that this sustained hand-form made the app feel responsive and immediate to users, with no optimization required in terms of the hand-tracking response time.
Ombre Fabula has been optimized for a short play-time of 6–10 minutes. Combined with a format in which users naturally expect to keep their hands raised for a certain period of time, the testers didn’t mention any hand or arm fatigue.
Under the Design Hood: Tools and Resources
The previous experience that Minh and Prefrontal Cortex have of creating interactive installations helped them make efficient decisions regarding the tools and software required to bring their vision to life.
Intel RealSense SDK
The Intel RealSense SDK was used to map hand contours, for which the contour tracking documentation provided with the SDK proved invaluable. The developers also made use of the Unity samples provided by Intel, trying hand-tracking first. When the hand-tracking alone was found to be insufficient, they moved on to images and implemented the custom blob detection.
Unity software
Both Minh and Prefrontal Cortex consider themselves designers first and foremost, and rather than invest their time in developing frameworks and coding, their interest lies in quickly being able to turn their ideas into working prototypes. To this end, the Unity platform allowed for fast prototyping and iteration. Additionally, they found the Intel RealSense Unity toolkit within the Intel RealSense SDK a great starting point and easy to implement.
$P Point-Cloud Recognizer
The $P Point-Cloud Recognizer is a 2D gesture-recognition software algorithm which detects, to a level of probability, which line-shape is being formed by pen strokes and other similar inputs. It’s commonly used as a tool to support rapid prototyping of gesture-based UIs. The developers lightly modified the algorithm and used it in Unity to detect the shape the user’s hands are making in Ombre Fabula. Based on probability, the algorithm decides which animal the shape represents and the app then displays the relevant visual.
The dragon is made to appear by forming a mouth-like shape with two hands, as seen in the background of this screenshot.
What’s Next for Ombre Fabula
Ombre Fabula has obvious potential for further story development and adding more hand-shadow animals for users to make and control, although the team currently has no plans to implement this. Ultimately, their ideal scenario would be to travel internationally with Ombre Fabula and present it to the public as an interactive installation—its original and primary purpose.
Intel RealSense SDK: Looking Forward
Felix Herbst from Prefrontal Cortex is adamant that gesture-control experiences need to be built from the ground up for the human interface, and that simply adapting existing apps for gesture will, more often than not, result in an unsatisfactory user experience. He emphasizes the importance of considering the relative strengths of gestural interfaces—and of each individual technology—and developing accordingly.
Those types of appropriate, useful, and meaningful interactions are critical to the long-term adoption of human interface technologies. Herbst’s view is that if enough developers create these interactions using Intel RealSense technology, then this type of human interface has the potential to make a great impact in the future.
About the Developers
Born in Vietnam and raised in Germany, Minh Nguyen is an experienced designer who uses cutting-edge technologies to create innovative, interactive multimedia installations and games. She is currently completing her masters in multimedia and VR design at Burg Giebichenstein University of Art and Design Halle in Germany. In addition to her studies, Minh freelances under the name Tibimi on such projects as Die besseren Wälder from game studio The Good Evil. Based on an award-winning play, the game is designed to encourage children and teens to consider the meaning of being ‘different’.
Prefrontal Cortex is a team of three designers and developers―Felix Herbst, Paul Kirsten and Christian Freitag―who use experience and design to astound and delight users with uncharted possibilities. Their award-winning projects have included the world-creation installation Metaworlds*, the [l]ichtbar interactive light projection at the Farbfest conference, the multi-touch image creation tool Iterazer*, and the award-winning, eye-tracking survival shooter game Fight with Light*. Their large-screen, multiplayer game Weaver* was a finalist in the Intel App Innovation Contest 2013. In addition to all these experimental projects, they create interactive applications for various industry clients using virtual and augmented reality.
Both Minh and the Prefrontal Cortex team intend to continue exploring the possibilities of human interface technologies, including the Intel RealSense solution.
Additional Resources
The video demonstrating Ombre Fabula can be found here.
For more information about the work of the Ombre Fabula creators, visit the Web sites of Minh Nguyen (Tibimi) and Prefrontal Cortex.
The Intel® Developer Zone for Intel® RealSense™ Technology provides a detailed resource for developers who want to learn more about Intel RealSense solutions. Developers can also download the Intel RealSense SDK and Intel RealSense Developer Kit, along with a number of useful Intel RealSense Technology Tutorials.