“When the Shoe Fits…” Using 3D Imaging to Improve Shoe Sizing

Abstract

This case study shows how Volumental used Intel® RealSense™ technology with its cutting-edge 3D modeling and virtualization software to scan a customer’s feet in real time so consumers could select an appropriately sized pair of shoes. With Volumental's software and the Intel® RealSense^TM F200 camera and SDK, capturing three angles in 3D is sufficient to size feet within 2mm (1/12^th of an inch). Now that the Intel Real Sense camera(s) will be available in  Ultrabook™, 2 in 1, and all-in-one systems, Volumental sees a world with custom fitting shoes, clothes and even medical devices will be within reach of the mass market. 

Introduction

If you’ve ever shopped for children’s shoes, you know that buying the right-sized pair of shoes is challenging. Not only is your child growing fast, but you have to get your child to sit still to try on shoes. Then you have to wonder about the accuracy of his or her response about the fit of the shoes, given the shoes are associated with a favorite cartoon character.

What if your child’s feet could be quickly and accurately measured before trying on any shoes so that you could limit the selection to shoes in that size?

Volumental and Intel® RealSense™ Technology

Volumental (https://www.volumental.com/) has a plan to realize that concept using Intel RealSense technology. And 3D scanning for accurate measurement of the body won’t just be for measuring feet and buying shoes. Caroline Walerud, CEO says: “At Volumental we believe that everything you wear should be made to fit you, not the other way around.” Volumental believes that 3D scanning, most notably of the body and body parts, will open up the field of product customization, not just better fitting clothes and shoes but medical devices as well. And now with Intel RealSense technology, this reality will be within the reach of the mass market.  

Volumental is a 3D body scanning startup, a spinoff from a doctoral research project in Computer Vision at the Royal Institute of Technology in Stockholm. When Intel announced Intel® RealSense^TM hardware and software, Volumental took notice and immediately saw the technology as a vehicle for their cutting-edge 3D modeling and visualization software. But moving from a broad concept to an actual project took some innovation.

Intel led with the user-facing Intel® RealSense^TM F200 camera, integrated into Ultrabook™, 2 in 1, and all-in-one systems, while Volumental wrote the code to measure and create 3D models of feet, first for children (the more difficult) and then adults. 

Volumental did user testing to see how consumers would “naturally” try to measure their feet using a laptop or tablet.; Intuitively, users thought they would move a device around their foot to perform a scan (Figure 1), rather than place a device on the floor.

But once shown a visual with the device on the floor, users agreed that made more sense. It became obvious that a 2 in 1 PC in tent mode (or stand mode) provided the the best experience (Figure 2).  It was important that users should be able to scan bare or socked feet, in the interest of creating a product that wouldn’t feel awkward to use, and that was easy and intuitive.  By adding user instructions for the position for each of the 3 poses,  it’s possible to get good results in a home setting.

For retail stores, where shoe departments want to allow self-service but not risk customers walking off with the scanning device, a foot scanning kiosk, like the one shown in Figure 3, was the right solution. Diagrams on the bottom pad show the user how and where to stand.

Using the Intel RealSense depth and color cameras, Volumental has created an application that takes 3D images of the feet, either bare or in socks. The application scans the feet using three views, detecting each foot and the direction it is facing (Figure 4). 

The Intel RealSense 3D camera provides both a color image and corresponding depth information for each pixel. The Volumental software analyzes the data from the three snapshots and provides the length, width, and girth around the instep, creating a useful 3D model of the customer’s feet.  The resulting model captures thousands of measurements and can be used for both size recommendation and the making of bespoke shoes.  Figure 5 shows an example of a customer’s 3D model, and shoe stores can provide a catalog of footwear available in that size.

How it works

Using pre-beta release hardware and the Intel RealSense SDK 2014, Volumental, Intel, and PLAE, a San Francisco-based children’s shoe manufacturer, showed a demo of the possibilities of measuring children’s feet at the Intel Developer Forum in San Francisco in 2014. And early in 2015, Volumental and Intel showed off their latest version—a freestanding kiosk computer with the Intel RealSense F200 camera at foot level, which will be placed in some retail shoe departments in 2015.

The ShoeSizer software combines computer vision and machine learning to create accurate 3D models of feet in a few seconds. It’s ok if the feet are moving (slightly) as Volumental uses algorithms that keep track of the 3D trajectory and make sure the result is accurate.

The software’s algorithms are based on machine learning, and the algorithms are trained on large databases of feet to ensure the creation of accurate models even when there is image noise or missing data. While the algorithms and actual processing are Volumental’s intellectual property, Intel was able to help improve performance by focusing on multithreading using OpenMP*. The resulting models contain more information related to the foot’s size than just a conversion to shoe size.

Other Challenges

Touchscreen systems requiring the use of the hands aren’t helpful in this usage that focuses on feet. However, the additional capabilities of Intel RealSense technology including the capability to use voice commands quickly became valuable. The kiosk accepts user’s voice commands to activate the scanning process.