Our Work and Motivation
The HELIOS* project is focused on enhancing and complementing human sensory functions with cutting-edge vision technologies.
A study published by the World Health Organization reveals an estimated 285 million visually impaired people worldwide: 39 million blind and 246 million with low vision.
We believe it is very important to improve mobility, safety, and access to knowledge for people with sight deficiencies.
Using Computer Vision, Artificial Intelligence, and Intel® RealSense™ technology, we are working on innovative solutions to help visually impaired individuals overcome several challenges they face on a daily basis. Our approach centers on the development of smart headgear to assist with partial or complete vision loss.
HELIOS Headgear Models and Features
The HELIOS headgear provides a series of accessibility features for visually impaired individuals, empowering them to perform actions and tasks with more ease and confidence.
HELIOS Touch
HELIOS Touch addresses individuals suffering from severe or complete blindness. It uses our HTI interface to translate visual data to the user via haptic signals, granting nearby environment localization and obstacle avoidance capabilities.
HELIOS Touch 3D representation
HELIOS Light
HELIOS Light addresses individuals suffering from low vision, using AR/VR technology to enhance the user’s level of visual perception, capitalizing on the Intel RealSense RGB and depth data streams to provide an adaptable vision aid for performing a variety of daily tasks.
HELIOS Light 3D representation
A key feature of HELIOS is to give the user better awareness of the nearby environment, substantially improving freedom of movement and safety.
Understanding non-Braille text is another important functionality. HELIOS can read the content of books, magazines, or other printed material, such as restaurant menus.
Furthermore, HELIOS delivers a new layer of context for interpersonal interaction by recognizing friendly faces and social cues.
Hardware breakdown
Intel® RealSense™ Technology
Intel® RealSense™ cameras have RGB-D capability and versatile sensors that provide HELIOS with high-quality depth and RGB streams. Their features, performance, and small form factor make them optimal for integration into the HELIOS headgear.
Intel® RealSense™ camera R200. Learn more about it in this article.
The Razer* Stargazer, which is a third-party version of the Intel® RealSense™ camera SR300.
HTI* Haptic interface
HTI is a hardware component of HELIOS Touch, developed by our team. It is designed to translate visual data into haptic feedback, providing the user with an extra layer of information, in a precise, non-intrusive form.
HTI test board
Open Source Virtual Reality
The Razer OSVR Hacker Development Kit is a highly customizable platform for Virtual and Augmented Reality. It is an ideal off-the-shelf component for HELIOS Light given its open source, extendable nature, and compelling hardware design.
Razer OSVR HDK
Intel Next Unit of Computing
Intel’s latest generation of small form factor PCs provides a robust platform for running HELIOS software components in real-time, with emphasis on performance, power efficiency, and portability.
Intel® NUC
Software. Intel® RealSense™ SDK
The Intel RealSense SDK is a central piece of HELIOS's software components. Out-of-the-box it facilitates access to high-frame-rate RGB, and depth and IR streams and provides a comprehensive set of computer vision algorithms for tasks such as person tracking, facial recognition, or 3D mapping. The SDK ships with a great set of sample projects and extensive online documentation.
The following code sample reveals the key components for developing a Text-to-Speech module with RealSense and UWP (Universal Windows Platform):
public async void StartRealSenseStreaming()
{
Status streamingStatus;
// Set RealSense sample reader and bind SetOcrFrame event
SampleReader sampleReader = SampleReader.Activate(senseManager);
sampleReader.SampleArrived += SetOcrFrame;
// Set RGB stream profile and device info filter
Dictionary<StreamType, PerceptionVideoProfile> profiles = new Dictionary<StreamType, PerceptionVideoProfile>();
profiles[StreamType.STREAM_TYPE_COLOR] = ColorProfile;
sampleReader.EnableStreams(profiles);
readers.Add(sampleReader);
if (currentRealSenseDevice != null)
senseManager.CaptureManager.FilterByDeviceInfo(currentRealSenseDevice.DeviceInfo);
// Set streaming status message
if ((streamingStatus = await senseManager.InitAsync()) == Intel.RealSense.Status.STATUS_NO_ERROR)
{
if ((streamingStatus = senseManager.StreamFrames()) == Intel.RealSense.Status.STATUS_NO_ERROR)
{
StatusMessage = "Streaming started";
}
else
{
StatusMessage = "Failed to stream: " + streamingStatus.ToString();
}
}
else
{
StatusMessage = "Initialization failed: " + streamingStatus.ToString();
}
IsStreaming = true;
}
private void SetOcrFrame(Object module, SampleArrivedEventArgs args)
{
// Setting current frame for OCR processing
Sample sample = args.Sample;
if (sample == null) return;
var localOcrFrame = sample.Color;
if (localOcrFrame == null) return;
lock (sample)
{
ocrFrame = localOcrFrame.SoftwareBitmap;
}
}
private async void TextToSpeech()
{
// setup OCR engine for English
OcrEngine ocrEngine = OcrEngine.TryCreateFromLanguage(new Language("en"));
// recognize text from the RealSense OcrFrame
var ocrResult = await ocrEngine.RecognizeAsync(RealSense.OcrFrame);
if (!String.IsNullOrEmpty(ocrResult.Text))
{
// setup speech synthesizer
var voice = SpeechSynthesizer.AllVoices;
using (SpeechSynthesizer speechSynthesizer = new SpeechSynthesizer())
{
speechSynthesizer.Voice = voice.First(v => v.Gender == 0);
var voiceStream = await speechSynthesizer.SynthesizeTextToStreamAsync(ocrResult.Text);
// setup playback of voice synthesis
PlaybackVoice(voiceStream);
}
}
}Testing and validation
Mihai Leoveanu was born with severe sight deficiency, but that didn’t stop him from becoming the outstanding person he is today.
Mihai has a strong can-do attitude and is one of the best students in his graduating class. He is currently working on his Master’s Thesis that proposes a series of Accessibility improvements for the Targoviste Royal Court historical site. These improvements would allow visually impaired tourists to have a richer visiting experience.
Mihai is the first person to review the capabilities of our headgear.
Mihai field testing HELIOS
Mihai reading with HELIOS
During the experimentation process, Mihai provided observations for each feature of HELIOS he was using. He seamlessly adopted the new information sources, and in a matter of minutes he was successfully using the headgear to gain a more accurate description of his surroundings.
Conclusion
The results from development and testing are very positive. Functionalities like environment perception and non-Braille reading are much easier for the user. With further development, HELIOS has the potential to make a real difference for individuals with sight impairment, becoming a true complement to their senses.
About the Authors
Silviu-Tudor Serban, Cristian Dragomir, and Andrei Nistor are Intel RealSense technology experts with backgrounds in Computer Vision, Artificial Intelligence, Software Development, and IoT. Find out more at Helios Vision and Intel Devmesh.