Sustainability of Digital Formats: Planning for Library of Congress Collections

The Polynomial Texture Map (PTM) file format is an openly documented image file format published by Tom Malzbender and Dan Gelb, when they worked for Hewlett-Packard Laboratories (HP Labs). The current version (1.2) of the PTM specification was published in November 2001. The special functionality of the PTM file format and the related RTI file format is that, with specialized viewer software, images can be adjusted interactively on screen by a user as if viewing an object with light from varying directions. This makes the formats very useful for studying objects that are relatively flat, but with an uneven surface. This description focuses on the use of the format in cultural heritage institutions to generate digital surrogates of artifacts in their collections. Examples of artifacts for which interactive viewing using PTM images has been particularly valuable include cuneiform tablets, fossils, coins, and the Antikythera mechanism. The images are sometimes described as 2D+. They are generated by photographing an object multiple times with lighting direction varying in a controlled fashion between images and fitting a mathematical function to the set of image data points for each pixel. When Malzbender, Gelb and colleagues developed this imaging technique and the associated PTM file format, HP Labs had an active group working in the area. See Polynomial Texture Mapping (PTM) at HP Labs. Page available via Internet Archive.

Polynomial Texture Mapping, as an image capture and processing technique, was first described in Enhancement of Shape Perception by Surface Reflectance Transformation, a technical report from HP Labs in 2000. For that first explanation, forty images were acquired of a stationary sample, all with identical camera position, but varying in the position of a light source. A per-pixel reflectance model is derived from these images and transformed to enhance the perception of surface detail and shape. A PTM file holds functions incorporating effects of lighting direction, not just single color values in RGB or some other color representation, such as YCbCr. In a conventional image, each pixel contains static values for red, green, and blue channels. In a PTM, each pixel (sometimes called a textel or texel, for "texture element") holds coefficients that specify a function that calculates the red, green, blue values of that textel as a function of two independent parameters, Lu and Lv, representing the direction from the light source to the textel/pixel. The curious or mathematically inclined will find links to the underlying mathematics in Notes below.

Typically, PTMs are used for displaying the appearance of an object under varying lighting directions with Lu and Lv specifying the direction of a virtual point light source. A typical PTM image file stores a set of coefficients per color channel and/or luminance per textel. These values can be derived mathematically (e.g., by least mean squares methods) from the data for the corresponding pixel in each of the source images.

As described below, the PTM file format supports several different encoding formats for color and luminance. In the specification for the PTM file format, the term "format" refers to these different encodings; in this description we use "encoding format" for clarity, to distinguish from references to the overall PTM file format.

A PTM file (always in little-endian form) begins with the following sections, separated by newline characters (aka line-feed or LF) with an optional space before the newline:

• Header String: The ASCII string 'PTM_1.2' appears as the first line of the file, identifying the file as a PTM file of the specified version.
• Format String: The ASCII string on the second line identifying the encoding format used in the file. See Notes below for discussion of the encoding formats permitted. The encoding format options are for different models for color and luminance changes with lighting direction and for whether or not the data is compressed. The most commonly used encoding formats are identified by PTM_FORMAT_LRGB (aka LRGB PTM) and PTM_FORMAT_RGB (aka RGB PTM). The LRGB PTM encoding format focuses on changes in luminance with lighting direction, using static values for the three colors for a textel. This requires less data than the RGB PTM encoding format, which models the changes in the three colors with lighting direction. In Polynomial Texture Maps (presented at Siggraph 2001), the inventors recommend the LRGB PTM variant because "the chromaticity of a particular pixel is fairly constant under varying light source direction; it is largely the luminance that varies." They also state, "An RGB PTM is capable of representing color shifts in materials. This characteristic is not common. For the majority of materials an LRGB PTM is adequate and always more efficient."
• Image Size: The next line consists of an ASCII string containing the width and height of the PTM map in pixels. Optionally, a newline may separate the two numbers.
• Scale and Bias: The PTM coefficients are stored in the file as a single byte per coefficient, with range of 0 to 1. Scale and bias values are used to allow the stored values to be mapped to the proper values. This technique is commonly used in graphics applications to optimize computational precision for pixel data given a constrained size for data elements. A total of 6 bias and 6 scale values are provided, applicable to the 6 polynomial coefficients. The six ASCII floating point scale values appear first, followed by the six ASCII integer bias values, all separated by spaces.

The exact representation for the actual image data depends on the encoding format in use. The specification indicates, that for the uncompressed encoding formats, the coefficient data is organized by pixel/textel in reversed scanline order (i.e., from bottom to top by row) without line separators.

In addition to supporting interactive viewing by varying the virtual light source, the PTM file format supports, with computational efficiency, additional techniques that are useful for enhancing the contrast in an image to study details such as inscriptions. See section 4.1 of Polynomial Texture Maps for discussion of specular enhancement and diffuse gain.

At least two non-profit organizations exist that provide imaging services or consulting to cultural heritage organizations or projects that will benefit from PTM or RTI images. Cultural Heritage Imaging (CHI) is based in California. The Visual Computing Lab (VCL-ISTI) at ISTI (Istituto di Scienza e Tecnologie dell’Informazione), an institute of the National Research Council of Italy located in Pisa, has supported projects throughout Europe, including the Ariadne Visual Media Service. Other projects and organizations that have developed expertise include the Smithsonian Museum Conservation Institute's Imaging Studio, the West Semitic Research Project (WSRP) at the University of Southern California, and the University of Southampton Archaeological Computing Research Group.

For a small selection of projects at museums and other archival institutions that have employed PTM imaging, see Notes below.

Instructions and kits for building and using equipment to photograph sets of images suitable for assembling PTM or RTI image files are available from various sources, e.g., WSRP; Historic England; Affordable Reflectance Transformation Imaging Dome by Leszek Pawlowicz; Cultural Heritage Science Open Source (CHSOS) and Cultural Heritage Imaging | Gear (CHI).

Free software is available for building PTM files from sets of images. Executables of PTM Builder and PTM Fitter software can be downloaded from HP Labs. The download pages for this software are available via Internet Archive.

Viewers available include: the original PTM Viewer from HP Labs as executables for Windows, Mac, and Linux (download page available via Internet Archive); a port of the original HP Labs code to Java, supporting viewing of PTMs over the web and with source code available; and RTIViewer, an open-source viewer for PTM and RTI images. See RTIViewer at CHI to download executables of the viewer or request copies of the source code and RTI Viewer at VCL-ISTI, for information about sources of technical and financial support for development of the viewer.

Applications of PTM outside the cultural heritage domain include generation of surfaces responsive to light direction that can be wrapped over models of physical objects used in video games, animated films, or 3-D models used by architects, designers, etc. PTM "shaders" in the OpenGL Shading Language (used for video games) were developed by Brad Ritter of HP Labs. Section 14.4 of OpenGL Shading Language: Polynomial Texture Mapping with BRDF Data states, "One use of PTMs is for representing materials that vary spatially across the surface. Materials such as brushed metal, woven fabric, wood, and stone, all reflect light differently depending on the viewing angle and light source direction. They may also have interreflections and self-shadowing. The PTM captures these details and reproduces them at runtime." PTM is not the only models used for such shading, also known as "texture mapping." Texture mapping models that respond to variation in viewing direction and light source vary in data size and computational power needed for real-time rendering. See Bidirectional Texture Function Modeling: A State of the Art Survey from 2009 for a comparison of various models. The compilers of this resource have not determined how widely used the PTM format is used in video games or design applications. Comments welcome.

PTM image encoding formats: The PTM 'formats' described in the PTM specification and supported by HP's PTM Viewer are listed in a table in Image-Based Empirical Information Acquisition, Scientific Reliability, and Long-Term Digital Preservation for the Natural Sciences and Cultural Heritage, from which the information below is derived. The encoding formats most commonly used are the first two listed below:

• PTM_FORMAT_LRGB. Luminance as a polynomial multiplied by unscaled RGB. 9 (6+3) bytes per pixel.
• PTM_FORMAT_RGB. Polynomial coefficients for each color channel. 18 (3x6) bytes per pixel.
• PTM_FORMAT_LUM. YCrCb color space, only Y as a polynomial. 1 or 2 bytes per pixel.
• PTM_FORMAT_PTM_LUT. Index to a lookup table that contains RGB values plus polynomial coefficients. 4 (3+1) bytes per pixel.
• PTM_FORMAT_PTM_C_LUT. RGB values plus an index to a lookup table that contains only polynomial coefficients. Variable bytes per pixel.
• PTM_FORMAT_JPEG_RGB. JPEG compression of PTM_FORMAT_RGB. Variable bytes per pixel.
• PTM_FORMAT_JPEG_LRGB. JPEG compression of PTM_FORMAT_LRGB. Variable bytes per pixel.
• PTM_FORMAT_JPEGLS_RGB. JPEGLS compression of PTM_FORMAT_RGB. Variable bytes per pixel.
• PTM_FORMAT_JPEGLS_LRGB. JPEGLS compression of PTM_FORMAT_LRGB. Variable bytes per pixel.

Mathematics underlying PTM: The mathematical details underlying the PTM methodology (for the LRGB PTM encoding format) are described in section 3.2 (Polynomial Color Dependence) in Polynomial Texture Maps, a paper by the inventors at Siggraph 2001, available via Internet Archive. For each pixel with position (u, v) in the image plane, luminosity (L) is modelled with a biquadratic polynomial as a function of incoming light direction. The direction is given by the projection (Lu, Lv) of the light vector (normalized to unit length) onto the image plane (see diagram from the article Virtual whitening of fossils using polynomial texture mapping). The six coefficients for the polynomial (usually referred to as a0 through a5) are fitted to the photographic data per texel/pixel and stored in the PTM. Malzbender et al computed the best fit using singular value decomposition (SVD) to solve for a0-a5. The SVD is computed only once given an arrangement of light sources for the set of capture photographs and then can be applied per pixel. The PTM format was specifically designed to allow surface colors to be efficiently reconstructed from these coefficients and light directions with minimal fixed-point hardware.

A useful summary is found in a sequence of slides from a presentation by Pradeep Rajiv entitled Image based PTM synthesis for realistic rendering of low resolution 3D models. Slide 32 illustrates the use of multiple images (in this case, only three images) from the same viewpoint, but lit from different directions; Slide 33 summarizes the information stored in the PTM (in this case, an LRGB PTM); Slide 34 shows the equations used to calculate the luminance (L) and the color channel intensities (R, G, and B) from the PTM for each texel/pixel given a virtual light direction. Note that, in this last slide, the term "texture" is used instead of "image," because the author is using the PTM technique to generate a 2-dimensional "texture" to wrap over a 3-dimensional model (as in virtual reality applications) rather than a 2-dimensional view on a screen.

Use of the PTM file format by archival institutions: Described briefly below are some examples of use of the PTM technique and format to create digital surrogates of artifacts in collections of museums and other archival institutions. The surrogates are used both for scholarly study and for presentations for the general public.

The Freer Gallery of Art and the Arthur M. Sackler Gallery Archives (Freer|Sackler Archives) hold a significant collection of 393 squeezes from ancient archaeological sites in the Near East. A squeeze is a series of sheets of paper that are layered on top of each other and moistened to create a wet pulp. This substance is pressed onto the inscriptions, creating a paper mold and capturing the writing for a 3-dimensional effect. In 2010, the Freer|Sackler Archives received a grant from the Smithsonian Institution's Collections Care and Preservation Fund to aid in the preservation of the squeezes and the 3-D information they contain. The Freer|Sackler Archives collaborated with the Smithsonian Institution Museum Conservation Institute (MCI) to create a digital preservation surrogate for each squeeze, using Reflectance Transformation Imaging (RTI). See Squeeze Imaging Project for information about the project, available via Internet Archive.

In 2005, the HP Labs team worked with the Antikythera Mechanism Research Project to generate PTMs of the fragments of Antikythera Mechanism. For information about this project, see the Antikythera page at HP Labs and Full resolution PTM at the Antikythera Mechanism Research Project in Greece. These pages are available via Internet Archive.

PTM and RTI have been used effectively by archaeologists to document stone tools. See Reflectance Transformation Imaging For Lithics from Dr. Leszek Pawlowicz.

Cultural Heritage Imaging (CHI) has worked with curators and art conservators to demonstrate the potential use of RTI to help document and conserve works of art in their collections. See Documentary: RTI and Art Conservation (2010) and Art Conservation (at CHI).

CHI experimented in the imaging of coins using PTM in collaboration with Tom Malzbender and Dan Gelb at Hewlett-Packard Labs and then explored imaging possibilities further in a project documented in Reflection Transformation Imaging and Virtual Representations of Coins from the Hospice of the Grand St. Bernard. See also Numismatics: The Study of Coins at Cultural Heritage Imaging.

A 2017 article, Underwater reflectance transformation imaging: a technology for in situ underwater cultural heritage object-level recording describes successful experiments with the use of RTI techniques under water, using the PTM format for the images.