Java* Parser for Depth Photo

Download PDF [PDF 684 KB]
Download The Code Samples [ZIP 25 KB]

Abstract

The Java parser for depth photo parses eXtensible Device Metadata (XDM) image files and extracts metadata embedded in image files to generate XML files. In addition, this app analyzes XML files to extract color image data and depth map data. It is a fundamental building block for depth photography use cases, like the image viewer, refocus feature, parallax feature, and measurement feature.

The input to this app is an XDM image file and outputs include XML file(s), color image file(s), and depth map file(s).

XDM

First, we describe the input to this app: XDM image files. XDM is a standard for storing metadata in a container image while maintaining the compatibility with existing image viewers. It is designed for Intel^® RealSense^TM technology. The metadata includes device-related information, like a depth map, device and camera poses, a lens perspective model, vendor information, and a point cloud. The following figure shows an example where the XDM file stores the depth map (right) as metadata with the color image (left).

Adobe XMP* Standard

Currently the XDM specification supports four types of container image formats: JPEG, PNG, TIFF, and GIF. XDM metadata is serialized and embedded inside a container image file, and its storage format is based on the Adobe Extensible Metadata Platform (XMP) standard. This app is specifically developed for the JPEG format. Next we briefly describe how XMP metadata is embedded in JPEG image files and how the parser parses XMP packets.

In the XMP standard, 2-byte markers are interspersed among data. The marker types 0xFFE0–0xFFEF are generally used for application data, named APPn. By convention, an APPn marker begins with a string identifying the usage, called a namespace or signature string. An APP1 marker identifies Exif and TIFF metadata; an APP13 marker designates Photoshop* Image Resources that contains IPTC metadata, another or multiple APP1 markers designate the location of the XMP packet(s).

The following table shows an entry format for the StandardXMP section in the JPEG, including:

• 2-byte APP1 marker 0xFFE1
• Length of this XMP packet, 2-bytes long
• StandardXMP namespace, http://ns.adobe.com/xap/1.0/, 29-bytes long
• XMP packet, less than 65,503 bytes

If the serialized XMP packet becomes larger than 64 KB, it can be divided into a main portion (StandardXMP) and an extended portion (ExtendedXMP), stored in multiple JPEG marker segments. The entry format for the ExtendedXMP section is similar to that for StandardXMP, except that the namespace is http://ns.adobe.com/xmp/extension/.

The following image shows how StandardXMP and ExtendedXMP are embedded in a JPEG image file.

This app defines three classes as follows:

1. XDMJavaParser. The main class that parses the XDM image, outputs XML files, and parses the XMP XML file to output color image(s) and depth map image(s).
2. XMLJavaParser. The class that parses the XMP XML file.
3. XMLJavaParserHandler. The handler class that implements the parsing of the XMP XML file.

The following code snippet shows two functions that are used to parse the XDM image in the XDMJavaParser class:

• findMarker. Parse the JPEG file (that is, buffer) from the specified location (that is, position) and search 0xFFE1 marker. If it is found, return the marker position; otherwise, return -1.
• findHeader. Look for StandardXMP namespace (http://ns.adobe.com/xap/1.0/) and ExtendedXMP namespace (http://ns.adobe.com/xmp/extension/) in the JPEG file (that is, buffer) from the specified location (that is, position). If found, return the corresponding namespace; otherwise, return an empty string.

XML

XMP metadata can be directly embedded within an XML document⁴. According to the XDM specification, the XML data structure is defined as follows:

The image file contains the following items as shown in the above table, formatted as RDF/XML. This describes the general structure:

• Container image. The image external to the XDM, visible to normal non-XDM apps.
• Device. The root object of the RDF/XML document as in the Adobe XMP standard.
  • Revision - Revision of the XDM specification
  • VendorInfo - Vendor-related information for the device
  • DevicePose - Device pose with respect to the world
  • Cameras - RDF sequence of one or more camera entities
    • Camera - All the information for a given camera. There must be a camera for any image. The container image is associated with the first camera, which is considered the primary camera for the image.
      • VendorInfo - Vendor-related information for the camera
      • CameraPose - Camera pose relative to the device
      • Image - Image provided by the camera
      • ImagingModel - Imaging (lens) model
      • Depthmap - Depth-related information including the depth map and noise model
        • NoiseModel - Noise properties for the sensor
      • PointCloud - Point-cloud data

The following code snippet is the main function of this app in the XDMJavaParser class, which parses the input JPEG file by searching APP1 marker 0xFFE1. If it is found, search the StandardXMP namespace string and ExtendedXMP namespace string. If the former, calculate metadata size and starting point, extract the metadata, and form the StandardXMP XML file. If the latter, calculate metadata size and starting point, extract the metadata, and form the ExtendedXMP XML file. The app’s outputs are two XML files.

The following code snippet is XMLJavaParser, which is the class that parses XMP XML file:

The following code snippet parses the XML file and extracts the color image and depth map for depth photography purposes in the XMLJavaParserHandler class. It is very straightforward. The function startElement() searches the attribute named IMAGE:DATA and extracts the corresponding data into a JPEG file, which is the color image. If multiples are found, multiple JPEG files are created. The function also searches the attribute named DEPTHMAP:DATA and extracts the corresponding data into a PNG file, which is the depth map. If multiples are found, multiple PNG files are created, too. The app’s outputs are JPEG file(s) and PNG file(s).

Conclusion

This white paper described the XDM file format, Adobe XMP standard, and XML data structure. The Java parser app for the depth photo parses the XDM image file and output StandardXMP XML file and ExtendedXMP XML file. Then it parses the XML files to extract color image file(s) and depth map file(s). This app does not depend on any other programs. It is a basic building block for any depth photography use cases.  

References

 “The eXtensible Device Metadata (XDM) specification, version 1.0,” /en-us/articles/the-extensible-device-metadata-xdm-specification-version-10

 Intel^® RealSense^TM Technology: http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html

Adobe XMP Developer Center. http://www.adobe.com/devnet/xmp.html

 “XML 1.0 Specification,” World Wide Web Consortium. Retrieved 2010-08-22.

Attribution and Thanks

This project uses SAX (http://www.saxproject.org) API for XML in Java. SAX project is not in the public domain and its copyright status can be found at the following link: http://www.saxproject.org/copying.html

About The Author

Yu Bai is an application engineer in SSG, working with external ISVs to ensure their applications run well on Intel® platforms. Before joining SSG, she worked for Rudolph Technologies as a senior software engineer, developing applications used in the operation of precision photolithograph equipment for the semiconductor capital equipment industry. Prior to Rudolph, she worked for Marvell Semiconductor as a staff engineer working on power analysis and power modeling for the company's application processors. She joined Marvell through the company's acquisition of Intel XScale® technology in 2006.

Yu received her master’s and doctorate degrees in Electrical Science and Computer Engineering from Brown University. Her graduate research focused on high-performance and low-power computer architecture design. Yu holds six U.S. patents and has published 10+ journal and international conference papers on power/performance management and optimization.