Sustainability of Digital Formats: Planning for Library of Congress Collections

ASTEM E57 3D file format (E57), built using a combination of binary and XML, XML 1.0, formats, is an open-source, fully documented, data file exchange format for 3D imaging data. E57 files are capable of storing 3D point data, point data attributes, and 2D imagery, as well as having a defined extension for addressing future aspects of 3D imaging. The E57 file format was established by the ASTM committee E57, a technical committee that develops and maintains ASTM standards and is documented in the ASTM E2807-11(2019) e1, Standard Specification for 3D Imaging Data Exchange, Version 1.0 (referenced throughout this document). The E57 standard is available for purchase through the ASTM website.

The website, libE57.org (Copyright 2011), has a collection of information and software tools, including a brief description of the E57 format, libE57 Software, and links to documentation. Daniel Huber, wrote a conference paper on the overall design and implementation of the E57 format, The ASTM E57 File Format for 3D Imaging Data Exchange (PDF), January 2011.

Structure of E57:

The E57 standard uses ‘Trees’ to describe the structure of the XML data and to index data in binary sections. Trees are defined as “data structures that represent an acyclic graph...consisting of nodes, which store some information, and edges (also known as arcs) that connect the nodes.

The E57 format is divided into 2 or more sections in this order:

File Header Section: required, contains fields such as, fileSignature, version, and location of XML section(s).

• File Header has a length of 48 bytes and includes byte fields: fileSignature, verisonMajor, versionMinor, fileLength, xmlOffset, xmlLength, and pageSize.

Binary Section(s): optional, encoded using the little-endian byte order. Binary sections, two types, are present when XML section contains data types Blob or CompressedVector (described in more detail below).

• According to the specification, the binary format is "composed of a sequence of pages...each page shall be composed of 1020 bytes of data (known as the payload) followed by a 32-bit cyclic redundancy check (CRC) checksum computed on the preceding payload...The length of an E57 file shall be an integral multiple of 1024 bytes. Any unused bytes in the payload of the final page in the file shall be filled with 0 values."

XML Section(s): required, contains a single XML 1.0 document using UTF-8 encoding. It describes the data hierarchy which contains a set of XML elements, called E57 elements, that are built in a specific format, following particular grammatical rules described in the specification, and are defined in a common, five level coordinate system constructed from the eight fundamental data types (described later in the documentation).

E57 standard defines a set data types that support 3D point data and 2D imagery, including: E57Root, Data3D, PointRecord, PointGroupingScheme, RigidBodyTransform, Image2D, IntensityLimits, ColorLimits, DateTime, Globally Unique Identifiers (GUIDs), and CoordinateMetadata.

• Image data stored in either JPEG or PNG.
• ColorLimits - specify limits for the value of red, green, and blue color that a sensor is capable of producing.
• Date and time are encoded based on Global Positioning System (GPS).
• Standard does not specify the format or generation of GUIDs.
• CoordinateMetadata encodes a geodetic datum, geoid, coordinate system, and map projection that allows stored data to be referenced in a standardized Coordinate Reference System (CRS), which is specified by a well-known text (WKT) string for a spatial reference system specification developed by the Open Geospatial Consortium (OGC).

E57 supports eight fundamental data types, five terminal and three non-terminal. Data types of the elements are indicated by the required type XML attribute (some others are optional). Data types are documented in the WSC standard XML Schema Part 2, Data types Second Edition. E57 element data types include (order inconsequential):

• Integer – terminal, stores integer values.
• ScaledInteger – terminal, stores numbers with fractional parts.
• Float – terminal, stores point values.
• String – terminal, stores text.
• Blob – terminal, stores opaque blocks of binary data interpreted by the reader, indicates the size and location of the binary section of the Blob.
• Structure – non-terminal, represents unordered groups of E57 elements.
• Vector – non-terminal, stores ordered list of items (records).
• CompressedVector – non-terminal, stores ordered lists of identically typed items (records) in a compressed binary format, records are divided into chunks, and the sections is composed of a section header and a sequence of binary packets; Index, Data, and Ignored.
  • Index Packet – organized in a tree with a header and tree nodes connected using fields in the address entries.
  • Data Packet – contains actual data from the records stored in a CompressedVector. Composed of a header and bytestream buffers lengths and bytestream buffers.
  • Ignored Packet – used to reserve space in the binary section for future use.

Uses of E57:

According to the website, FileInfo.com, “E57 files can be used for rendering images of the real-world objects, such as buildings, atmospheric entities (e.g. clouds), and geological surfaces. This can be useful in construction, surveying, engineering, and research.”

ASTM E57 3D Imaging Systems Committee – An Update (March 2008), states the “ASTM committee E57 was established to develop standards for the performance evaluation of 3D imaging systems. The committee’s initial focus is on standards for 3D imaging systems typically used for applications including, but not limited to, construction and maintenance, surveying, mapping and terrain characterization, manufacturing, transportation, mining, mobility, historic preservation, and forensics...The ASTM E57 3D Imagery committee was formed specifically to develop standard terminology, test methods, best practices and data interoperability specification for these instruments.”

According to Cicely Enright in Standards for the Third Dimension, March/April 2013, Subcommittee E57 “needed to prove that the format would work by creating software to implement the standard... In the time since the file format has become available, major hardware and software vendors in the 3D imaging area have incorporated it in their products.” Lecia, Bentley Systems, Riegl and others support the E57 format and have implemented it.

The LibE57.org website has information on the libE57 software, an open-source implementation of the of E57 Standard in the C++ language. “It is intended to lower the barrier to adoption of the standard and to provide a reference to compare other implementations against.” Also, the website link to E57 supporting partners and products.

According to Nicole Herber in What are Point Clouds?, September 2021, “The E57 format in particular has virtually become the industry standard. Almost all registration software of the laser scanner manufacturers can output the format and processing software for point clouds can also read the format.”

The E57 format is a combination of binary and XML sections, not easily viewable by human readers combined. LibE57 software provides a utility tool for extracting the XML section from the E57 file, “e57xmldump”.

Comments welcome.

None beyond availability of supporting software.

Comments welcome.

IANA Security Considerations states: “Per their specification, E57 files do not contain executable content. However, in addition to binary data, they contain an XML-encoded stream. Therefore, they have all of the general security considerations presented in section 10 of [RFC7303].”

Comments welcome.

E57 files store geometry point data, including point clouds and rigid body transforms, produced by 3D imaging systems.

Comments welcome.

E57 files store 3D point data attributes, such as color and intensity, and 2D imagery.

According to Daniel Huber in The ASTM E57 File Format for 3D Imaging Data Exchange (PDF), January 2011, “Frequently, 3D imaging sensors will capture imagery, either using built in color sensors or through a conventional digital camera. These images can be a valuable asset for visualization or analysis of 3D data sets.”

Comments welcome.

According to Daniel Huber in The ASTM E57 File Format for 3D Imaging Data Exchange (PDF), January 2011, “Images are most useful if they are calibrated with respect to the 3D point data so that individual 3D points can be projected onto the images (or, conversely, the images can be projected onto the 3D point data). To support this functionality, the standard defines four different image representations. These representations correspond to different common camera models.”

As described in the E57 standard there are four image representations possible:

• Visual Reference
• Pinhole
• Spherical
• Cylindrical

Images may be used for visual reference and/or to match 3D points to image pixels (projection).

Comments welcome.

None known information about Animation.

Comments welcome.

Preserving 3D – Data Type Series, “E57m is a technical metadata standard for the E57 file format available through the libE57 reference implementation libraries. It describes 3D scans and /or 2D images contained in E57 files.”

DURAARK – DURAble ARchitectural Knowledge, was a three-year research project (February 2013 - January 2016) “aimed at the development of methods for the sustainable long-term preservation of building data, which includes 3D point cloud scans and building information models (BIM) as well as metadata, related knowledge and Web data.” The project used the ASTM standardized E57 format for the point-could scans. See link to The DURAARK Project PDF for more information.

According to Paul Bourke in E57 Proposed Standard for Laser Scan Data, June 2017, “Broadly an e57 file consists of a 48-byte header, a series of data chunks, and finally an XML section. The whole file, including the header is split into 1024 byte "pages", 1020 bytes of data and 4 bytes at the end as a checksum... In order to extract the xml section one simply reads from the start of the file in 1024-byte pages, once the file location reaches header.xmlPhysicalOffset save the remaining 1020 bytes of each 1024 page.”

In 2006, ASTM committee E57 was established to develop standards for the performance evaluation of 3D imaging systems. The committee’s initial focus is on standards for 3D imaging systems typically used for applications including, but not limited to, construction and maintenance, surveying, mapping and terrain characterization, manufacturing (e.g., aerospace, shipbuilding), transportation, mining, mobility, historic preservation, and forensics.

The ASTM E57 3D Imaging committee was formed specifically to develop standard terminology, test methods, best practices and data interoperability specifications for these instruments.