Sustainability of Digital Formats: Planning for Library of Congress Collections

The RTI image file format is a file format used for output from a process of Reflectance Transformation Imaging, a method of computational photography. The format was developed under an IMLS National Leadership grant (LG-25-06-0107-06) to the University of Southern California (USC) and Cultural Heritage Imaging (CHI). The project was titled "Developing Advanced Technologies for the Imaging of Cultural Heritage Objects." The Visual Computing Lab of ISTI-CNR [VCL ISTI-CNR] in Pisa, Italy was a sub-contractor responsible for developing the RTI file format and an associated viewer. The RTI file format was designed as a more general successor format to PTM, which was named for its first application (polynomial texture mapping).

RTI and PTM image files are both created from multiple digital photographs of a subject shot from a stationary camera position. In each photograph, light is projected from a different known, or knowable, direction. See Notes below for two popular methods for generating a controlled set of captures. The result is a series of images of the same subject with varying highlights and shadows. Lighting information from the images is mathematically manipulated to fit a model of the surface to the data in the images. A primary objective is to enable a user to (virtually) re-light the RTI or PTM image interactively and examine its surface on a screen. In addition, the image files can be used to create enhanced visualizations. The PTM format (with file extension .ptm) is based on fitting a binomial quadratic equation to the light data for each pixel. The RTI format (with file extension .rti) indicates support for a variety of additional fitting methods, currently including Spherical Harmonics (SH), Hemi-Spherical Harmonics (HSH), and Adaptive Basis PTM. All these methods derive a per-pixel reflectance model represented by a set of coefficients that define the fitted function that takes into account the direction of the lighting. The stated concept motivating the RTI specification is that it can be "universal" and support any similar basis for encoding the "reflectance properties of the depicted surface." However, as of early 2018, RTI Builder appears only to support PTM and HSH. The curious or mathematically inclined will find comparisons of PTM and HSH and links related to the underlying mathematics for HSH in Notes and Useful References, below.

The specification for the RTI File format, available from the CHI Forum, is titled "RTI Format - Draft 0.9." The authors are listed as Massimiliano Corsini and Prabath Gunawardane of the Visual Computing Lab of ISTI-CNR [VCL ISTI-CNR] in Pisa, Italy. It is dated December 2008, with an update in December 2010 by Carla Schroer of CHI. An RTI file has three main components:

• Header: This component is in plain text lines, terminated with CR LF (0x0D 0x0a). Comment lines, beginning with #, may precede the primary content of the header. The primary content of the header consists of three lines with numeric values that function as structural metadata that will permit an application to interpret the Data component appropriately. A sample header is

  #HSH1.2
  3
  1908 3148 3
  4 2 1

  • First in the example is a single comment indicating the fitting model (HSH) and version (1.2).
  • Line 1 contains a single integer, a code indicating the fitting model used for the RTI image (RTI_Type). The specification lists four values: 1 (PTM); 2 (SH); 3 (HSH); and 4 (Adaptive Basis).
  • Line 2 contains three integers defining: image width in pixels; image height in pixels; and number of color channels (e.g., 3 for RGB, 1 for grayscale). This last item is called color_dimensions.
  • Line 3 contains three integers, defining: number of coefficients (basis_terms) in the fitting function (e.g., 6 for PTM, 9 for a second order HSH); code indicating whether coefficients are calculated for a single composite luminance value (1) or independently for each color channel (2); data element size in bytes, typically 1, 2, or 4.

  If RTI_Type is "Adaptive Basis," there will be additional lines, one per color channel. The compilers of this resource have not encountered explicit use of this feature. Comments welcome.

• Data: All image data is stored in little-endian binary format as a contiguous block without separators. The main data is preceded by floating point values for scale and bias for each coefficient (basis term). Assuming a value of N for basis_terms, there will be N scale values followed by N bias values. The actual value for a coefficient is calculated from the stored coefficient by Cactual = Cstored*scale + bias. This technique is commonly used in graphics applications to optimize computational precision for pixel data given a constrained size for data elements, and is also used in the PTM format. The pixel data is stored in row order, with basis_terms*color-dimensions elements for each pixel.
• Metadata: The last component is a single XMP packet. The specification suggests element sets for basic and extended metadata, but also has a note indicating that the metadata section of the specification needs to be modified to take into account the entities/properties of the CIDOC Conceptual Reference Model."

Specialized viewing applications, such as RTIViewer or the ISF Standalone Viewer link via Internet Archive (developed for the InscriptiFact project at USC), offer the capability to change the virtual light source or rotate the virtual object, revealing more detail than possible using traditional photography.

A link to a draft specification (last updated in 2010) for the RTI Format is posted in the CHI forum under Where can I find the file format specifications for RTI and PTM?. The link to RTI Format - Draft 0.9 will yield a request to sign in or register. Registration is free.

An explanatory note in the posting says that it is "marked 'draft' because it contemplates a .xmp structure embedded in the file structure, and that part was still under discussion when this document was created. The format for the RTI data is final, and this is what is used by the RTIViewer and other viewers that support this format."

The RTI format is supported by the same organizations and tools that support the PTM format. The widely used RTIBuilder software produces PTM format files if the PTM fitter is used and RTI format files if the HSH fitter is used. The format is used in the same cultural heritage domains as PTM, especially those for which the HSH fitting model has demonstrated a particular advantage.

Examples of domains for which HSH has advantages include:

• Digital surrogates for coins, because HSH handles shiny surfaces and specular highlights (bright spots of light appearing on shiny objects) better than PTM, which yields matte images. See Storytelling of a Coin Collection by Means of RTI Images: the Case of the Simoneschi Collection in Palazzo Blu.
• Palimpsests and other illegible documents, because HSH enables use of the light spectrum outside the visible range. See Jubilees Palimpsest Project.
• Conservation of Paintings, because a third order HSH has been shown experimentally as producing more reproducible results than PTM. See Measuring Changes in Cultural Heritage Objects with Reflectance Transform Imaging (2013) and A New Quantitative Method for the Non-Invasive Documentation of Morphological Damage in Paintings Using RTI Surface Normals (2014)

In the abstract for his 2016 tutorial on Reflectance Transformation Imaging (RTI) for Cultural Heritage Artefacts, Konstantinos Papadopoulos stated, "The method has been extensively used on a wide range of cultural objects, such as inscriptions, manuscripts, rock art, paintings, numismatics and any possible material including metal, stone, leather, paper, wax, bone, and clay." Resources that provide examples of the variety of cultural heritage applications of RTI, include: videos and lists of presentations from Illumination of Material Culture: A Symposium on Computational Photography and Reflectance Transformation Imaging (RTI)", a two-day symposium hosted by the Metropolitan Museum of Art in March 2017; and Highlight-Reflectance Transformation Imaging (H-RTI) for Heritage Applications (2016) from the Colour and Space in Cultural Heritage (COSCH) project (2012-16). Link via Internet Archive.

An important development leading to widespread adoption of the RTI imaging method and file format in cultural heritage institutions is the introduction of open-source tools to deliver effective interactive viewing of re-lighting, etc. over the web. For more on webRTI, see Notes, below.

See also Adoption for PTM.

Accessibility Features

Accessibility features for 3D model formats include audio descriptions for screen readers to enable user interactivity events such as object selection, rotation and zoom, alt text for image forms, color contrast definition as well as caption and subtitle support. RTI has no specific support for accessibility or assistive technology although it may depend on implementation and software features. Comments welcome.

RTI image capture methods: An RTI file is created from a sequence of digital images of the subject with a specific set of lighting angles. Two commonly used methods to produce the image sequence are described below. Either method allows specialized software, such as RTIBuilder, to construct an RTI file.

• Highlight Method; An image-capture technique for Reflectance Transformation Image (RTI), in which you include an image of at least two reflective spheres in each shot. The reflection of the light source on the spheres enables the processing software to calculate the lighting angle for that image. This light source information is used during the generation of an RTI or algorithmic rendering image. The highlight method allows great flexibility in subject size and location. It involves manually moving a light source to different angles relative to an object (at a constant distance), and using a shiny reflective ball to determine the lighting angle. While this non-automated technique doesn't require much in the way of special equipment, it can be slow and often requires two people to perform. For an in-depth discussion, see the RTI Guide to Highlight Image Capture. For examples of application in different contexts and for different types of object, see Highlight-Reflectance Transformation Imaging (H-RTI) for Heritage Applications (2016) from the Colour and Space in Cultural Heritage (COSCH) project (link via Internet Archive).
• Light Array or Dome Method: An apparatus (typically dome-shaped) that has a set of lights at fixed positions. Control software flashes each light in a fixed sequence in order to produce the image sequence needed to produce an RTI file. Because each light is at a known position, the software can associate each image with the lighting angle used for it. The technique referred to as Dome-RTI involves putting the object of interest under a dome with multiple light sources on the inside, and a camera looking down through a hole at the object. An electronic controller turns lights on and off in sequence, and fires the camera shutter in sync with the lights. Because this can be fully automated, and because the lighting angles are always the same, obtaining the required set of images can be faster and easier than with the Highlight-RTI technique. See Affordable Reflectance Transformation Imaging Dome for instructions and specifications for a particular kit using the Dome-RTI approach.

Common objective for HSH and PTM image fitting models: The PTM and HSH models are two examples of Reflectance Transformation Imaging fitting methods, in which the distribution of light reflected from an object surface is modeled as functions of angles of lighting and viewing and of spectrum as captured by a device such as a camera. The term bidirectional reflectance distribution function (BRDF) is used to describe the way light from a single light source reflects off an opaque surface and is observed by a viewer or camera. The motivation behind both PTM and HSM was to provide a compact representation of an approximation to the actual BRDF for an object surface that would support real-time interpolation to simulate relighting the object surface from different points. In both cases, the model can be used to generate synthetic computer graphics or to build a digital reproduction of a real object surface from a set of images captured using methods such as those described above. PTM and HSH are both widely used in cultural heritage contexts.

Comparisons of HSH and PTM image fitting models/algorithms: Comparisons located by the compilers of this resource are primarily based on use of RTIBuilder software from CHI. See Reflectance Transformation Imaging: Guide to Highlight Image Processing Version 1.4 (manual for RTIBuilder version 2.0.2). RTIBuilder offers two "fitters" (PTM and HSH) and provides output in the PTM (.ptm) file format for the former and in the RTI (.rti) file format for the latter. RTIBuilder offers a choice of LRGB and RGB for the PTM fitter; see the description PTM for a discussion of these variants, which both use 6 coefficients for the fitted polynomials. RTIBuilder options for the HSH fitter are for polynomials of order 1, 2, and 3, with 4, 9, and 16 coefficients respectively. More coefficients provide more accuracy in fitting the surface geometry, but require more space for storage and more computational power for building and for viewing.

In HSH or PTM - How to choose the best fitter from the Cultural Heritage Imaging Forums, Carla Schroer of CHI stated in 2012, "we have found that the HSH approach is superior in terms of rendering more sculptural objects that have self shadowing - where in the ptm in specular enhancement mode you might get just a black area where there is a lot of shadowing, an HSH will have better data. Also HSH does a better job of rendering shiny material as shiny. PTMs are always matte. Sometimes a matte surface is easier to see what you are looking for - so like most things it is a trade-off. Finally, I will note that HSH files are generally bigger than lrgb ptms. PTMs use 6 coefficients of a bi-quadratic polynomial to describe the surface normal. Tom Malzbender wanted a compact format for this. In HSH, this number is determined by the "order" that you choose when you create it. a first order HSH uses 4, a 2nd order 9 and a third order 16. So this affects the file size of the result, but it's also why the HSH can render shiny surfaces more accurately." Later in the thread, she says of HSH, "its main disadvantage is that we don't have all the rendering modes in the viewer implemented for it." The RTIViewer software available for download in early 2018 still has fewer rendering modes for RTI files based on HSH.

RTI Files from Reflectance Transformation Imaging For Lithics echoes Schroer's assessment, "There are two kinds of RTI data files. *.ptm files are a fitting of a binomial quadratic equation to the light curve, while *.rti use a 2nd or 3rd order Legendre polynomial for the fit. *.rti files are a better fit, but there are more visualization modes available with *.ptm files."

In Measuring Changes in Cultural Heritage Objects with Reflectance Transform Imaging (2013), Marcello Manfredi et al, states, "There are two main ways to describe the reflectance function: using polynomial texture map (PTM) and using Hemispherical Harmonics (HSH). The HSH approach is superior in term of 3D rendering because PTM uses only 6 coefficients of a biquadratic polynomial to describe the surface normal while a third order HSH uses 16 coefficients." Their experience led them to state, "The aim of this research was the development of a reliable RTI imaging method for monitoring changes in cultural heritage objects. RTI provides detailed information on the geometry of the object surface. We showed that RTI can be reproducible and quantified to the level where it can be used as practical method for measuring change in objects. We found that computing surface normals with HSH is more reproducible than using the older PTM method, and is therefore more sensitive to changes."

In A New Quantitative Method for the Non-Invasive Documentation of Morphological Damage in Paintings Using RTI Surface Normals (2014), Manfredi et al. propose and demonstrate an RTI method for the non-invasive detection of color changes and morphological changes (cracks, holes, bends, etc.) in paintings, arguing that the HSH approach is superior to PTM for this application.

Links to the mathematics underlying the HSH fitting approach: The HSH methodology for rendering surfaces was first published as A Novel Hemispherical Basis for Accurate and Efficient Rendering in 2004 by Pascal Gautron et al. As part of work under a grant to CHI in 2007-2009, PhD students Prabath Gunawardane and Oliver Wang working with James Davis at University of California, Santa Cruz took the HSH method and developed a software module to fit the HSH model to image data from an RTI capture. There is no publication based on that particular activity (although the source code is available from CHI on request). The CHI Forum had a thread in 2014 with topic Scientific Method HSH vs. PTM in 2014. It included the question, "Do you have any papers regarding the hsh?" There were several expert responses with useful information, but no citation to a paper. The compilers of this resource have also found no official documentation for the underlying model and fitting algorithm employed in the HSH fitter in RTIBuilder. Comments welcome.

The compilers of this resource have identified some articles that include useful discussion of the mathematics of surface modeling and the HSH fitting method, including explanatory diagrams.  Lindsay MacDonald's 2014 article, Colour and Directionality in Surface Reflectance, has a discussion of PTM and HSH, with a straightforward summary of the formulae for the coefficients used for HSH of order 1, 2 , and 3. Representation of Cultural Objects by Image Sets with Directional Illumination (2015), also by MacDonald, has a fuller discussion. Another author providing helpful explanations of reflectance modeling in general and HSH in particular is Shireen Elhabian. See her article On the Use of Hemispherical Harmonics for Modeling Images of Objects under Unknown Distant Illumination (2011) and a discussion of her research interests in Appearance Modeling. Many resources that discuss the issues of surface rendering and introduce the terminology are available online, for example The Phong Model and the concepts of Illumination Models and BRDF from Scratchapixel.

WebRTI: An important contribution to the adoption of RTI is the development of WebRTIViewer, a viewer requiring HTML5 and building on Web-GL, for high resolution RTI images based on PTM or HSH fitting models. For the web visualization, an RTI image must be preprocessed with the command line tool webGLRTIMaker, which creates a tiled multi-resolution format to mount on a web server. The tools also work with high-resolution images in widely used raster formats (JPEG, PNG, and TIFF).

The compilers of this resource have been unable to locate a specification document for the multi-resolution format produced by WebGLRTIMaker. Comments welcome. The webGLRTIMaker source code is available on GitHub. An example with the webRTIviewer source code has a number of separate JPEG images for 15 tiles of the same pixel dimensions (1_1.jpg, 1_2.jpg, 1_3.jpg, 2_1.jpg, etc.) and a file named info.xml that holds structural metadata, including indication of tile arrangement and positions. In Storytelling of a Coin Collection by Means of RTI Images: the Case of the Simoneschi Collection in Palazzo Blu, Gianpaolo Palma, the main developer of webRTIviewer, states that webGLRTImaker "subdivides the RTI image in nine layers, one layer for each HSH coefficient, where the i-th layer contains the i-th coefficient of the three RGB colour channels. Then for each layer it creates a multiresolution quad-tree and produces a tile for each node of the tree. Finally, it saves each tile in a different JPG image. With this format to visualize a specific pixel, we need to load the nine JPG images that contain its HSH coefficients. The main advantage is the out-of-core loading of the node of the quad-tree making immediately available the interaction with the coin without awaiting the complete loading of the data. This means that at the beginning the user interacts with a low-resolution version of the coin, which is progressively refined as soon as the higher resolution data are loaded. The loading of the tiles at the different resolutions is guided by the zoom and pan operation of the user."

The widely used "dome" technique for constructing RTI images was developed by Tom Malzbender and Dan Gelb, research scientists at Hewlett-Packard Labs. A paper describing the first tools and methods, using was presented at SigGraph 2001: Polynomial Texture Maps. See also Polynomial Texture Mapping (PTM) (link via Internet Archive). In March 2017, Malzbender spoke at a symposium organized by CHI and the Metropolitan Museum of Art, "Illumination of Material Culture: A Symposium on Computational Photography and Reflectance Transformation Imaging (RTI)". His talk gave a brief history of RTI and presented some new possibilities. See video of Day 2: Afternoon Session at time 44.09.

The first distributed draft of the RTI format was dated August 27, 2008 as an output of the 2006 IMLS grant (Award Number LG-25-06-010706). Draft 0.9 was published in December 18, 2008 and updated in minor ways in December 2010 by CHI.