Look around. In our three-dimensional world, you can see objects from any angle, gauge their size, and perceive how near or far they are relative to their surroundings. But when it comes to capturing and showing those objects to others, we often turn to 2D still or moving pictures. The reason for that is simple: capturing 3D images has been too complicated and expensive for anyone but skilled professionals and committed enthusiasts.
Recent innovations are changing that. With little effort, shoppers can use tablets to see how they look wearing different pairs of virtual eyeglasses. Similar Android* and iOS* apps let people preview—and interact with—virtual furniture, clothes, toys, and more. In most cases, the 3D inventory used in these apps has been limited to pre-built content created by pros.
Intel® RealSense™ 3D cameras will soon put 3D scanning, printing, and sharing into the hands of the masses. Combined with hand and finger tracking, voice commands, facial analysis, and augmented reality capabilities facilitated by Intel® RealSense™ technology, people will interact naturally with real and virtual objects as well as print them in 3D. Potential use cases range from practical applications (sales, medical, design, education) to entertainment (games, movie making, social engagement).
3D Systems, Autodesk, and DotProduct are trailblazing companies with deep roots in professional 3D content creation and printing. They’re implementing Intel RealSense technology with the goal of bringing 3D capture and share to a huge audience.
Seeing and Printing in 3D
Founded by Chuck Hull, the inventor of 3D printing, 3D Systems has been serving the automotive, aerospace, industrial design, and medical industries since 1986. The company is currently focused on democratizing 3D printing. “The Intel RealSense 3D camera is a natural fit,” Ping Fu, 3D Systems Chief Entrepreneurial Officer, said. “Two years ago, most 3D scanners cost 10,000 to 50,000 USD. With Intel RealSense 3D cameras, users will have access to 3D scanners embedded in their PC, 2-in-1, AIO, Ultrabook™ device, or tablet.
3D Systems is developing two desktop applications that make use of the Intel RealSense technology in PCs: 3DMe* and SENSE*. Both were built using the Intel® RealSense™ SDK 2014.
3DMe allows anyone to put their face on a favorite character; for example, an NBA player, Star Trek’s Spork, a snowboarder, a princess, and more (Figure 1). Once captured, the character can be 3D printed, animated, or shared through social media.
Figure 1: User’s face scanned into a character using the 3DMe* app
SENSE is object-scanning software. “You can scan your head, entire body, or objects—for example a favorite toy—and either 3D print or export them,” Ping Fu explained. “Gaming is an obvious use case. Designers and makers can use the capability to create reference models for detailed designs, and casual users can scan their favorite object, share with friends, play with it in games, or 3D-print their own creation.
Use cases for personally scanned objects that are 3D-printed run the gamut. “Doll makers want to create personalized dolls to match girls’ faces,” Ping Fu said. “Sports enthusiasts want helmets that exactly fit the shape of their heads. People can scan their wrists and hands to easily make casts that ease the discomfort of carpal tunnel.”
Both 3DMe and SENSE capture objects (Figure 2) as editable 3D meshes, complete with a texture map and color information. 3DMe users will be linked to cubify.com to print a likeness of the person; SENSE users will be able to print their models using 3D Systems’ CUBE* line of personal 3D printers or upload them to the cloud and print them using a service bureau. The CUBE personal printer supports printing in a few materials, whereas higher-end professional and industrial printers support printing in “…metal, ceramic, acrylic, plastic, and even edible chocolate and sugar,” said Ping Fu.
Figure 2: Object scanning with SENSE* software
The SENSE app supports the exporting of meshes as OBJ, DXE, PLY, and other CAD-friendly file formats. “Most CAD users prefer to work with parameterized meshes so that when they adjust geometry, those changes propagate across surfaces, textures, and so on,” Ping Fu continued. “SENSE is compatible with 3D Systems’ Geomagic Solution software suite, which provides the fastest way to get from scan data to CAD/CAM parametric models.”
According to Ping Fu, Intel RealSense technology can hold global accuracy to within 1 mm. “That is good enough for many personalized products—custom-fit eyeglasses, shoes, or medical devices such as casts to mend broken limbs. For industrial design use cases, people will bring 3D scans into a CAD environment to refine reference design in greater detail.”
Ping Fu’s experience with 3D scanning dates back to 1996, when she was co-founder and CEO of Geomagic, a world-leading 3D scanning software company acquired by 3D Systems in 2013. She has always wanted to democratize 3D scanning and was dedicated to connect 3D scanning to 3D printing. In Intel, she found a kindred spirit. “Our relationship with Intel has been one of our most fruitful, meaningful collaborations,” she said. “We see eye-to-eye. Intel RealSense technology promotes intuitive interaction, and Intel RealSense 3D cameras capture raw data. Our software takes that point cloud and turns it into a full-color textured model for multiple down-stream usage. We both bring the required expertise to make perceptual computing accessible, easy, and fun to use in 3D applications.”
Ping Fu cited Intel’s expertise in integrating hardware with optimized software as being key. “The Intel RealSense SDK team is fantastic. They know how to support the developer community.” Each company contributed to the other’s efforts. 3D Systems engineers used the Intel RealSense SDK to build their apps, tapping the SDK for its camera control, depth image, and facial recognition capabilities. “We also got great help from working closely with Intel’s User Experience team, creating an intuitive, easy-to-use app.”
3D Systems also contributed to the Intel RealSense SDK, providing scan data processing functions that will give other developers the ability to take point cloud data and turn it into a mesh. “One of the challenges with 3D scanning is that raw data is often noisy,” Ping Fu said. “We provided very sophisticated cleanup functions, including the ability to fill holes based on the curvature of a surface. It gives you a very natural look without having to re-scan an object.” 3D Systems also provided the ability to down-sample objects while retaining their overall shape—a function that game developers use to deliver low-polygon-count objects for easier on-the-fly rendering. The most unique algorithm is the “what you see is what you get” (WYSIWYG) scanning process, making scanning easy for consumers and modeling automatic.
What were the 3D System teams’ key takeaways from the development experience? “Getting 3D scanning right is difficult,” Ping Fu said. “For a new technology to gain acceptance, user experience is key. We learned a tremendous amount with Intel’s help on performance tuning, user testing, and market feedback. Intel also introduced us to other ISVs, PC OEMs, and retailers. Those connections have turned into valuable partnerships.”
Seeing in 3D to Enhance 2D Pictures
Autodesk, a world leader in 3D design, engineering, and entertainment software, is using the technology to capture 3D image data so that consumers can easily enhance 2D photographs. “Intel RealSense technology is a great fit for our Pixlr* image editing app,” Gareth Pennington, senior software manager for Autodesk, said. “We’re especially interested in background segmentation and the ability to get a new sense of dimension out of 2D images.”
Background segmentation uses depth data to determine or “segment” what's in the foreground and background, and then extract, replace, or enhance different aspects of the image—all in real time (Figure 3). The segmentation algorithms of the Intel RealSense SDK look at color and depth data together, determine what to display and extract, and then deliver the results in 60-fps HD video.
Figure 3: Background segmentation using the Pixlr* app
“We will deliver very playful use cases—for example, giving people the ability to capture only their head and automatically remove the background so that their head can be converted into an image stamp that can then be layered onto other images,” Pennington said. “Or create personalized smiley faces and other emoticons for use in email or texts. For example, if you’re on a boat or in a plane, you could snap a picture of yourself and apply a green filter that affects only you—not what’s around you—to indicate you’re seasick.”
Pennington advises other developers to “explore the code samples as soon as possible, because when you’re looking at a huge SDK with so many capabilities, it’s difficult to know where to focus.”
Pixlr is currently available both as an online web app and as a native app for Windows* and Mac OS, iOS, and Android. The release supporting Intel RealSense technology will be a native Windows app. Looking ahead, Pennington said, “It’s fun to think about what we could do with all of the other capabilities enabled by Intel RealSense technology—motion tracking, facial recognition, and then also looking at being able to measure things and potentially re-focusing... those are all really compelling. There was a lot left in the Intel RealSense SDK for us to explore. Over time, as more devices with Intel RealSense technology become available to consumers, we’ll spur their creativity. We're excited about the future.”
Seeing in 3D on Tablets
From its inception in 2012, DotProduct has served professional markets and early adopters of 3D laser scanning—oil, gas, and power companies; engineering firms; as well as the forensics and historical documentation industries. “Those industries are dependent on 3D imaging,” Brian Ahern, CEO of DotProduct explained. “They design and operate in 3D, maintain and upgrade facilities in 3D, and de-commission in 3D.”
Recognizing that small-form factor 3D sensors offered an affordable alternative to industrial-strength scanners, DotProduct created a hand-held Android tablet-based solution that produces accurate 3D models in the field. That system carried a 5,000 USD price tag. Now, the company is working closely with Intel in developing an even lower-cost tablet solution based on Intel RealSense technology.
“We’re excited about Intel RealSense technology,” Ahern said. “We expect our market to grow exponentially when an Intel RealSense 3D camera is embedded in off-the-shelf tablets.” Ahern envisions DotProduct becoming a software-centric company, licensing its technology to OEMs and ODMs that are integrating Intel RealSense technology.
Like 3D Systems and Autodesk, DotProduct is working to make its solution more consumer-friendly to attract a broader audience. “The 2.0 beta version features a completely rebuilt UI. Its navigation, viewing, and rendering features are intuitive to use [Figure 4].”
Figure 4: DotProduct’s rebuilt user interface.
Ahern believes prosumers will be the first to adopt the new product, which will focus on capturing and sharing 3D images. “One of our competitive advantages is a proprietary, lossless image-compression algorithm that allows us to email 3D point clouds without having to build a memory-hungry mesh.” A free viewer app will allow people to share, for example, a model of their living room, and open it on a tablet that will convert the point cloud to a mesh for viewing from any angle and be measured.
DotProduct’s core technology is based on Android, but “…it’s not dependent on it. Android is a wrapper—our core code is in C++. We’re now porting to Windows,” Ahern said. When the product is released in 2015, it will run on both platforms. Intel RealSense technology product manager Kenn Walker added, “When you look at apps that take advantage of Intel RealSense technology, some will be built on higher levels of the SDK. The DotProduct app runs closer to the metal, handling data in its raw form.”
What kind of processing power is needed for handling that data? “More of everything—CPU, GPU, RAM—is great,” Ahern said. “More horsepower gives us better results.”
Ahern envisions all manner of use cases, ranging from gaming to sales to edutainment and more. “In some design businesses, sales people have to manually measure objects, rooms, and facilities. That’s a time-consuming process. Being able to simply point a tablet at an object (Figure 5) or a room while moving around, under, and over things, and then pressing ‘Finish’ to display your model on the tablet will be a game changer.”
Figure 5: Scanning with a tablet
Describing the key takeaways from the development experience, Ahern said, "We learned that even engineers sometimes underestimate the speed of process in consumer tech. We knew embedded 3D mobile devices were coming, but it happened faster than we anticipated."
Ahern and his team at DotProduct also realized that it's not enough to get customers excited about new technology. "You have to either create a solution that seamlessly integrates with their existing workflow or provide them with new and intuitive processes for using data in work and play environments. We spent the last couple of years doing just that—it takes persistence to get all the details right."
Resources
Explore Intel RealSense technology further, learn about Intel RealSense SDK for Windows Beta, and download a Developer Kit here.
Is your project ready to demonstrate? Join the Intel® Software Innovator Program. It supports developers who have forward‐looking projects and provides speakership and demo opportunities.
Get started at the Developer Resource Center.
Read part 1, part 2, and part 3 of this “Transforming the UI—Designing Tomorrow’s Interface Today” series.