PhonoPaper is a technology and an application that allows you to play audio converted into a picture of a special format, called a spectrogram on paper or any other surface. The process is roughly as follows: 10 seconds of audio (voice, song excerpt, etc.) are converted into a picture of a special format. The printed picture can, for example, be stuck to a wall. Passersby, noticing the code, launch the PhonoPaper scanner on their phones, aim the camera at the image and in an instant begin to hear the sound encoded in it. The user is fully involved in the process—the direction and speed of playback depend on the movement of the user's hands (although there is also an automatic mode). All the necessary information is stored in the image and Internet access is not required.
An example of Phonopaper in action:
PhonoPaper aroused keen interest among musicians, artists, and fans of unusual experiments. In the 3rd quarter of 2014, the application took first place in the Intel Developer Rating project on the Apps4All.ru website. Thanks to an x86-based Android* tablet Intel loaned me for testing, the app has been improved and optimized. Here, I’ll show how I optimized PhonoPaper.
Video capture
The first thing I did was to connect the Intel® INDE Media for Mobile set of libraries, specifically, the GLCapture class which captures video with an OpenGL ES* surface in real time (in HD sound quality). Why is this necessary? First, the process of capturing and playing PhonoPaper codes is a fun, exciting spectacle like playing an unusual musical instrument. Second, PhonoPaper can operate in free mode, when the sound is converted indiscriminately, such as with a carpet and cat. Both would be great to record and upload to YouTube*.
Free mode example: any camera image is perceived as a sound spectrum.
PhonoPaper codes, drawn by hand
The process of connecting GLCapture has been described in several articles (1, 2 ). I'll just mention a few points that are good to know before starting.
You should use Android version 4.3 or later. For older devices, I created a cross-platform MJPEG recorder, the speed and quality of which is of course much inferior to hardware-accelerated GLCapture which writes in mp4. But still, it gets the job done.
The application must be built on the basis of OpenGL ES 2.0. My program historically used version 1.1, so I had to rewrite the code. But the transition to OpenGL ES 2.0 ultimately had a positive impact on productivity, as it gave the opportunity to manually adjust the shaders.
GLCapture can write sound from a microphone. This is good if you want video that accompanies your comments. If you need high-quality sound directly from the application, you should record it in a separate file and then combine it with the mp4. For combining them, you can use MediaComposer with the SubstituteAudioEffect effect from the Media for Mobile set. Another way is recording to WAV, encoding WAV to AAC, and adding the AAC track to the mp4 file using the mp4parser library.
Since PhonoPaper is written in Pixilang programming language, the function of capturing video can later be used with other pixilang-based applications (PixiTracker, PixiVisor, Nature - Oscillator, Virtual ANS), and most importantly, will be available for all developers using Pixilang. At the same time, access is very easy (there are only a few options: start capture, stop, and save).
Pixilang is an open cross-platform programming language, customized to work with sound and graphics. The language syntax is highly minimalistic and is a hybrid of BASIC and C, which together with other features (the ability to write code without functions and universal containers for storing any data) reduces the entry threshold.
Intel® C++ Compiler and optimization
The next step was to assemble the x86 Android version of PhonoPaper using the Intel® C Compiler and compare the results with GCC 4.8. I use Debian Linux*, and a rather old version at that. Therefore, the first problem was to find the appropriate version of the Intel C++ Compiler. Fortunately, the right installation package was found – Intel® System Studio 2015. Despite the warning during installation, everything worked well and the first assembly was successful.
The compilation works with the following keys: -xATOM_SSSE3 -ipo -fomit-frame-pointer -fstrict-aliasing -finline-limit=300 -ffunction-sections -restrict. To test the performance of the Pixilang virtual machine (it is the basis of all my applications) small tests were written, and the sources and results of them can be viewed in this archive (zip). As a result, even without preparation, some code fragments were accelerated 5-fold(!). This is quite impressive!
In PhonoPaper, most of the load is for the spectral synthesizer function (table-wave, not FFT) – wavetable_generator (). For it, a single test was written that renders an audio stream with a random spectrum within four seconds. At the end of the test, the highest possible sampling frequency is produced. Unfortunately, the Intel C Compiler did not perform well here: 105 kHz compared to 100 kHz on GCC. By adding the -qopt-report=2 key during compilation, this message displays in the report:
loop was not vectorized: vector dependence prevents vectorization.
The main loop within our function could not be vectorized because input data indicators can point to overlapping memory areas:
int* amp = (int*)amp_cont->data int* amp_delta = (int*)amp_delta_cont->data;
Viewing the code, I see that at this point the intersection has been eliminated and I just need to tell the compiler. In C/C++ there is a special keyword, restrict, stating that the declared indicator points to a block of memory, which is not pointed to by any other indicator. Therefore, I replace the above code with this:
int* restrict amp = (int*)amp_cont->data; int* restrict amp_delta = (int*)amp_delta_cont->data;
Then, I assemble the application and see that the cycle is successfully vectorized. With some additional changes (in the process, it proved that it is possible to eliminate a few bit operations), the result is now 190 kHz. Taking these amendments into account, GCC gave 130 kHz—a 1.46-fold performance increase!
What is next?
As you can see, the results are very positive. PhonoPaper now runs faster (thanks largely to the Intel C++ compiler) and was extended with the functionality of the video capture. In addition, the video recording will appear in a few simple functions in the next Pixilang 3.6 update.
This optimization helped with even doing things like recording, printing, and then playing back voice codes.
There’s lots to explore with PhonoPaper!
About the Author
Alexander Zolotov is a software developer, demoscener, composer, and sound and graphics designer. He is the owner of WarmPlace.ru website. He developed several audio-visual applications: SunVox (modular synth and tracker), Virtual ANS, PhonoPaper, PixiTracker, and Pixilang programming language.