Introduction
In the spirit of the Maker Movement and “America’s Greatest Makers” TV show coming this spring, this article describes a project I constructed and programmed: a pan-tilt person-tracking camera rig using an Intel® RealSense™ camera (R200) and a few inexpensive electronic components. The goal of this project was to devise a mechanism that extends the viewing range of the camera for tracking a person in real time.
The camera rig (Figure 1) consists of two hobby-grade servo motors that are directly coupled using tie wraps and double-sided tape, and a low-cost control board.
Figure 1. DIY pan-tilt camera rig.
The servos are driven by a control board connected to the computer’s USB port. A Windows* C# app running on a PC or laptop controls the camera rig. The app uses the Face Tracking and Person Tracking APIs contained in the Intel® RealSense™ SDK for Windows*.
The software, which you can download using the link on this page, drives the two servo motors in real time to physically move the rig nearly 180° degrees in two axes to center the tracked person in the field of view of the R200 camera. You can see a video of the camera rig in action here: https://youtu.be/v2b8CA7oHPw
Why?
The motivation to build a device like this is twofold: first, it presents an interesting control systems problem wherein the camera that’s used to track a moving person is also moving at the same time. Second, a device like this can be employed in interesting use cases such as:
- Enhanced surveillance – monitoring areas over a wider range than is possible with a fixed camera.
- Elderly monitoring – tracking a person from a standing position to lying on the floor.
- Robotic videography – controlling a pan-tilt system like this for recording presentations, seminars, and similar events using a mounted SLR or video camera.
- Companion robotics – controlling a mobility platform and making your robot follow you around a room.
Scope (and Disclaimer)
This article is not intended to serve as a step-by-step “how-to” guide, nor is the accompanying source code guaranteed to work with your particular rig if you decide to build something similar. The purpose of this article is to chronical one approach for building an automated person-tracking camera rig.
From the description and pictures provided in this document, it should be fairly evident how to fasten two servo motors together in a pan-tilt arrangement using tie wraps and double-sided tape. Alternatively, you can use a kit like this to simplify the construction of a pan-tilt rig.
Note: This is not a typical (or recommended) usage of the R200 peripheral camera. If you decide to build your own rig, make certain you securely fasten the camera and limit the speed and range of the servo motors to prevent damaging it. If you are not completely confident in your maker skills, you may want to pass on building something like this.
Software Development Environment
The software developed for this project runs on Windows 10 and was developed with Microsoft Visual Studio* 2015. The code is compatible with the Intel® RealSense™ SDK version 2016 R1.
This software also requires installation of the Pololu USB Software Development Kit, which can be downloaded here. The Pololu SDK contains the drivers, Control Center app, and samples that are useful for controlling servo motors over a computer’s USB port. (Note: this third-party software is not part of the code sample that can be downloaded from this page.)
Computer System Requirements
The basic hardware requirements for running the person-tracking app are:
- 4th generation (or later) Intel® Core™ processor
- 150 MB free hard disk space
- 4GB RAM
- Intel® RealSense™ camera (R200)
- Available USB3 port for the R200 camera
- Additional USB port for the servo controller board
Code Sample
The software developed for this project was written in C#/WPF using Microsoft Visual Studio 2015. The user interface (Figure 2) provides the color camera stream from the R200 camera, along with real-time updates of the face and person tracking parameters.
Figure 2. Custom software user interface.
The software attempts to track the face and torso of a single person using both the Face Tracking and Person Tracking APIs. Face tracking alone is performed by default, as it currently provides more accurate and stable tracking. If the tracked person’s face goes out of view of the camera, the software will resort to tracking the whole person. (Note that the person tracking algorithm is under development and will be improved in future releases of the RSSDK.)
To keep the code sample as simple as possible, it attempts tracking only if a single instance of a face or person is detected. The displacement of a bounding rectangle’s center to the middle of the image plane is used to drive the servos. The movements of the servos will attempt to center the tracked person in the image plane.
Servo Control Algorithm
The first cut at controlling the servos in software was to derive linear equations that effectively scale the servo target positions to the coordinate system shared by the face rectangle and image, as shown in the following code snippet.
Servo.cs
public class Servo
{
public const int Up = 1152;
public const int Down = 2256;
public const int Left = 752;
public const int Right = 2256;
.
.
.
}
MainWindow.xaml.cs
private const int ImageWidthMin = 0;
private const int ImageWidthMax = 640;
private const int ImageHeightMin = 0;
private const int ImageHeightMax = 480;
.
.
.
ushort panScaled = Convert.ToUInt16((Servo.Right - Servo.Left) * (faceX –
ImageWidthMin) / (ImageWidthMax - ImageWidthMin) + Servo.Left);
ushort tiltScaled = Convert.ToUInt16((Servo.Down - Servo.Up) * (faceY –
ImageHeightMin) / (ImageHeightMax - ImageHeightMin) + Servo.Up);
MoveCamera(panScaled, tiltScaled);Although this approach came close to accomplishing the goal of centering the tracked person in the image plane, it resulted in oscillations that occurred as the servo target position and face rectangle converged. These oscillations could be dampened by reducing the speed of the servos, but this made the camera movements too slow to effectively keep up with the person being tracked. A PID algorithm or similar solution could have been employed to tune-out the oscillations, or perhaps employing inverse kinematics to determine the camera position parameters, but I decided to use a simpler approach instead.
The chosen solution simply compares the center of the face (faceRectangle) or person (personBox) to the center of the image plane in a continuous thread and then increments or decrements the camera position in both x and y axes to find a location that roughly centers the person in the image plane. Deadband regions (Figure 3) are defined in both axes to help ensure the servos stop “hunting” for the center position when the camera is approximately centered on the person.
Figure 3. Incremental tracking method.
Building the Code Sample
The code sample has two dependencies that are not redistributed in the downloadable zip file, but are contained in the Pololu USB Software Development Kit:
- UsbWrapper.dll (located in pololu-usb-sdk\UsbWrapper_Windows\)
- Usc.dll (located in pololu-usb-sdk\Maestro\Usc\precompiled_obj\)
These files should be copied to the ServoInterface project folder (C:\PersonTrackingCodeSample\ServoInterface\), and then added as references as shown in Figure 4.
Figure 4. Third-party dependencies referenced in Solution Explorer.
Note that this project uses an explicit path to libpxcclr.cs.dll (the managed RealSense DLL): C:\Program Files (x86)\Intel\RSSDK\bin\win32. This reference will need to be changed if your installation path is different. If you have problems building the code samples, try removing and then re-adding this library reference.
Control Electronics
This project incorporates a Pololu Micro Maestro* 12-channel USB servo controller (Figure 5) to control the two servo motors. This device includes a fairly comprehensive SDK for developing control applications targeting different platforms and programming languages. To see how a similar model of this board was used refer to robotic hand control experiment article.
Figure 5. Pololu Micro Maestro* servo controller.
I used Parallax Standard Servo motors in this project; however, similar devices are available that should work equally well for this application. The servos are connected to channels 0 and 1 of the control board as shown in Figure 5.
Servo Controller Settings
I configured the servo controller board settings before starting construction of the camera rig. The Pololu Micro Maestro SDK includes a Control Center app (Figure 6) that allows you to configure firmware-level parameters and save them to flash memory on the control board.
Figure 6. Control Center channel settings.
Typically, you should set the Min and Max settings in Control Center to match the control pulse width of the servos under control. According to the Parallax Standard Servo data sheet, these devices are controlled using “pulse-width modulation, 0.75–2.25 ms high pulse, 20 ms intervals.” The Control Center app specifies units in microseconds, so Min would be set to 750 and Max set to 2250.
However, the construction of this particular device resulted in some hard-stops (i.e., positions that result in physical binding of the servo horn that can potentially damage the component). The safe operating range of each servo was determined experimentally, and these values were entered for channels 0 and 1 to help prevent it from inadvertently being driven to a binding position.
Summary
This article gives an overview of one approach to building an automated camera rig capable of tracking a person’s movements around a wide area. Beyond presenting an interesting control systems programming challenge, practical applications for a device like this include enhanced surveillance, elderly monitoring, etc. Hopefully, this project will inspire other makers to create interesting things with the Intel RealSense cameras and SDK for Windows.
Watch the Video
To see the pan-tilt camera rig in action, check out the YouTube video here: https://youtu.be/v2b8CA7oHPw
Check Out the Code
Follow the Download link to get the sample code for this project.
About Intel® RealSense™ Technology
To learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/intel-realsense-sdk.
About the Author
Bryan Brown is a software applications engineer at Intel Corporation in the Software and Services Group.