By Konstantin Popov of Cappasity
Cappasity has been developing 3D scanning technologies for two years. This year we are going to release a scanning software product for Ultrabook™ devices and tablets with Intel® RealSense™ cameras: Cappasity Easy 3D Scan*. Next year we plan to create hardware and software solutions to scan people and objects.
As an Intel Software Innovator and with the help of the Intel team, we were invited to show the prototype of the people scanning system much earlier than planned. We had limited time for preparations, but still we decided to take on the challenge. In this article I'll explain how we created our demo for the Intel Developer Forum 2015 held August 18– 20 in San Francisco.
Our demo is based upon previously developed technology that combines the multiple depth cameras and the RGB cameras into a single scanning system (U.S. Patent Pending). The general concept is as follows: we calibrate the positions, angles, and optical properties of the cameras. This calibration allows us to merge the data for subsequent reconstruction of the 3D model. To capture the scene in 3D we can place the cameras around the scene, rotate the camera system around the scene, or rotate the scene itself in front of the cameras.
We selected the Intel RealSense camera because we believe that it's an optimum value-for-money solution for our B2B projects. At present we are developing two prototype systems using several Intel RealSense cameras: a scanning box with several 3D cameras for instant scanning and a system for full-body people scanning.
We demonstrated both prototypes at IDF 2015. The people scanning prototype operated with great success for the three days of the conference, scanning many visitors who came to our booth.
Now let's see how it works. We attached three Intel RealSense cameras to a vertical bar so that the bottom camera is aimed at the feet and lower legs, the middle camera captures the legs and the body, and the top-most camera films the head and the shoulders.
Each camera is connected to a separate Intel® NUC computer, and all the computers are connected to the local area network.
Since the cameras are mounted onto a fixed bar, we used a rotating table to rotate the person being filmed. The table construction is quite basic: a PLEXIGLAS® pad, roller bearings, and a step motor. The table is connected to the PC via an Intel® Edison board; it receives commands through the USB port.
We also used a simple lighting system to steadily illuminate the front of a person being filmed. In the future, all these components will be built into a single box, but at present we were just demonstrating an early prototype of the scanning system, so we had to assemble everything using a commercially available component.
Our software operates based on the client-server architecture, but the server part can be run on almost any modern PC. That is, any computer that performs our calculations is a "server" in our system. We often use an ordinary Ultrabook with Intel® HD Graphics as a server. The server sends the recording command to the Intel NUC computers, gets the data from them, then analyzes and rebuilds the 3D model.
Now, let's look at some particular aspects of the task we are trying to solve. The 3D rebuilding technology that we use in the Cappasity products is based upon our implementation of the Kinect* Fusion algorithm. But in this case our challenge was much more complex: we had only one month to create an algorithm to reconstruct the data from several sources. We called it "Multi-Fusion." In its present state the algorithm can merge the data from an unlimited number of sources into a single voxel volume. For scanning people three data sources were enough.
Calibration is the first stage. The Cappasity software allows the devices to be calibrated pairwise. Our studies from the year we spent in R&D came in pretty handy in preparation for IDF 2015. In just a couple of weeks we reworked the calibration procedure and implemented support for voxel volumes after Fusion. Previously the calibration process was more involved with processing the point cloud. The system needs to be calibrated just once, after the cameras are installed. Calibration takes no more than 5 minutes.
Then we had to come up with a data-processing approach, and after doing some research we chose post-processing. That is, first we record the data from all cameras, then we upload the data to the server via the network, and then we begin the reconstruction process. All cameras record color and depth streams. As a result, we have the complete data cast for further processing. It is convenient considering that the post-processing algorithms are constantly improved, and the ones we're using were written in just a couple of days before IDF.
Compared to the Intel RealSense camera (F200), the Intel RealSense camera (long-range R200) performs better with black color and complex materials. We had few glitches in tracking. The most important thing, however, is that the cameras allow us to capture the images at the required range. We have optimized the Fusion reconstruction algorithm for OpenCL™ to achieve good performance even on Intel HD Graphics 5500 and later. To remove the noise we used Fusion plus additional data segmentation after a single mesh was composed.
In addition, we have refined the high-resolution texture mapping algorithm. We use the following approach: we capture the image at the full resolution of the color camera, and then we project the image onto the mesh. We are not using voxel color since it causes the texture quality to degrade. The projection method is quite complex to implement, but it allows us to use both built-in and external cameras as color sources. For example, the scanning box we are developing operates using DSLR cameras to get high-resolution textures, which is important for our e-commerce customers.
However, even the built-in Intel RealSense cameras with RGB provide perfect colors. Here is a sample after mapping the textures:
We are developing a new algorithm to eradicate the texture shifting. We plan to have it ready by the release of our Easy 3D Scan software product.
Our seemingly simple demo is based upon complex code allowing us to compete with expensive scanning systems at USD 100K+ price range. The Intel RealSense cameras are budget-friendly, which will help them revolutionize the B2B market.
Here are the advantages of our people scanning system:
- It is an affordable solution, and it’s easy to setup and operate. Only a press of a button is needed.
- Small size: the scanning system can be placed in retail areas, recreational centers, medical institutions, casinos, and so on.
- The quality of the 3D models is suitable for 3D printing and for developing content for AR/VR applications.
- The precision of the resulting 3D mesh is suitable for taking measurements.
We understand that the full potential of the Intel RealSense cameras is yet to be uncovered. We are confident that at CES 2016 we'll be able to demonstrate significantly improved products.