Sustainability of Digital Formats: Planning for Library of Congress Collections

JPEG XS is an image and video coding system that is visually lossless and prioritizes low-latency, low complexity, and interoperability. JPEG XS has been designed to specifically target applications that use uncompressed video to provide high quality images with less power consumption than other moving-image codecs such as MPEG-4_AVC and HEVC. Per intoPIX, lead developer of the codec, "XS" of the JPEG XS codec stands for "extra small" and "extra speed" to reflect compression efficiency. Use cases and targeted applications for JPEG XS include virtual reality applications, live production and broadcasting over IP, mobile devices, autonomous cars and other automotive image applications, drones, and online gaming.

The key features of the JPEG XS as defined by the JPEG XS White paper are:

• Visual lossless quality: minimal or imperceptible flickering between original and compressed image.
• Multi-generation robustness: no significant quality degradation for up to 10 encoding and decoding cycles.
• Multi-platform interoperability: the use cases of the JPEG XS codec require real-time implementations on a variety of platforms such as Central Processing Unit (CPU), Graphics Processing Unit (GPU), Field Programmable Gate Array (FPGA), and an Application Specific Integrated Circuit (ASIC). To support different platforms, the JPEG XS codec needs to support various end-to-end parallelization.
• Low complexity: in both hardware and software. Low complexity implementations are critical for JPEG XS to replace uncompressed video. JPEG XS has been designed so that an i7 processor can process 4k 4:4:4 60p content in real time.
• JPEG XS is designed for minimal delay between signal and human interaction. JPEG XS offers a scalable algorithmic latency ranging from a small number of lines down to less than a single line.

Additional features that are highlighted in the JPEG XS Use Cases and Requirements paper are:

• Bit rate allocation: JPEG XS allows for variable bit rate (VBR) allocation which allows time for rate-distortion optimization and improvements in visual quality.
• High dynamic range (HDR) content: JPEG XS currently supports bit depths up to 16 bits per component and allows for efficient compression of HDR content.

ISO/IEC 21122-2 is the JPEG XS standard that defines profiles and identifies various points of interoperability to target specific use cases. The JPEG XS encoding is broken down into four distinct profiles, which allows different levels of latency and complexity. JPEG’s published overview of the coding system evaluations provides more in depth comparisons of the test configurations for each JPEG XS profile while Figure 1 provides a block diagram of the JPEG XS codec.

• Main profile: the default profile for JPEX XS and one vertical Discrete Wavelet Transform (DWT). Supports Natural, CGI, Screen content and targets broadcast, pro-av, frame buffer, and display link applications.
• Light or Low logic: Less complex and slightly less efficient with one vertical DWT. Supports natural content for broadcast, industrial camera, and in-camera compression applications.
• Light-subline or Low Memory: Less buffering and less efficient profile with no vertical DWT. Also supports natural content for cost-sensitive applications.
• High: More quality than the other profiles with two vertical DWT. Supports natural, CGO, and screen content and targets similar applications to the "Main" profile but mostly geared towards high-end devices and cinema-remote production.

There are also standardized transport and container formats for the JPEG XS codestream, making it possible for storage and transportation of JPEG XS images. Read more about the profiles and associated formats on page 6 of the JPEG XS White paper and consult Tables 2 and 3 for comparisons across profiles and formats. Those include:

• JPEG XS File format - storage of single images. See JXS.
• MPEG-4 File Format - storing video. See ISO_BMFF and MP4_FF_2.
• High Efficiency Image File Format - storage of mixed image and video content. See HEIF.
• Material Exchange Format - serve as a video container format for SMPTE standard professional video. See MXF and this intoPIX blog post about mapping JPEG XS codestreams to the MXF container.
• MPEG Transport stream - an MPEG-2 transport stream for JPEG XS. See MPEG-2.
• RTP payload format - the IP transport for JPEG XS.
• SMPTE 2110 system stream - the encapsulation of compressed video stream.

JPEG XS, unlike other codecs such as JTK_C and MPEG-2, was not designed with a focus on compression efficiency, which serves as the primary difference between them. From an interview with Antonin Descampe, co-founder of intoPIX and developer of JPEG XS, the key question for JPEG XS is "How can we ultimately replace uncompressed video?" JPEG XS focuses more so on allowing an increase in resolution, frame rates, and stream numbers while also maintaining the advantages of uncompressed streaming utilizing the key features discussed above. JPEG XS relies on intra-frame or "i-frame" compression, where every frame is encoded individually. This is critical for JPEG XS' functionality to use less computing power (discussed more below) as intra-frame codecs require less power to decode during playback and exploits redundancy within the frames. It requires less processing time because the computer is dealing with a single frame.

One of the prominent use cases for JPEG XS is the transport over video links and IP networks. Per this intoPIX blog post and Page 3 of the JPEG XS White paper, uncompressed video is unsuitable for the high frame-rates of 4K, 8K video and beyond given the bandwidth and infrastructural limitations. JPEG XS offers compression ratios ranging from 2:1 to 6:1, even up to 10:1 for visually lossless quality. See Table 1 on pages 3-4 in the JPEG XS Use Cases and Requirements for a detailed comparison of bandwidth requirements and compression rations for video formats.

The lower complexity feature of the JPEG XS codec gives it an advantage over MPEG codecs for video including H.264, known as MPEG-4_AVC, and H.265 known as HEVC. JPEG XS has a more balanced complexity between encoder and decoder, suiting environments that have the same number of encoders and decoders. JPEG XS codecs require much less power than the H.264 or HEVC codecs do because of memory consumption. H.264 and H.265 codecs require more memory and are highly complex whereas JPEG XS would not have memory requirement issues as they use line-based compression. As mentioned above, MPEG 4 and HEVC codecs use considerably more power because they are use inter-frame compression, which exploits redundancies between sucessive frames. More processing time is needed to examine several frames. Lastly, the JPEG XS code’s low-latency presents an advantage over these MPEG codecs. Some of the targeted JPEG XS applications require real-time transmission such as autonomous driving systems with lengthy delays rendering this codec useless. H.264 and H.265 codecs, with multiple encoding and decoding steps, would lead to a compiled latency of many seconds. See the intoPIX blog post for the full interview with Antonin Descampe and comparison graphs between JPEG XS and other intro-frame and distribution codecs.

Comments welcome.

As of May 2022, this standard is in published status. The ISO/IEC 21122-1 format specification is broken down into four parts. Parts 1 through 3 released updated versions of their specifications in 2022. The 2022 versions of Parts 4 and 5 remain in development, but the 2019 versions remain available.

Part 1 carries the running title, Part 1: Core coding system, which "Defines the JPEG XL codestream and decoder, which can be used for lossy encoding, lossless encoding, and lossless re-compression of existing JPEG images."

Part 2 carries the running title, Part 2: Profiles and buffer models. Part 2, "specifies an extensible box-based file format which adds support for metadata (e.g. Exif and JUMBF) and legacy JPEG bitstream reconstruction data."

Part 3 carries the running title, Part 3: Transport and container. Part 3 defines how to embed the codestream into a more descriptive file format and "contains all definitions are that necessary to transport a codestream."

Part 4 carries the running title: Part 4, Conformance testing, which defines the conformance testing for JPEG XS. A second edition is under development that permits faster decoder implementations.

Part 5 carries the running title: Part 5, Reference software. This outlines the JPEG XS reference software.

The JPEG XS Use Cases and Requirements details the applications and real-time implementations of JPEG XS. JPEG XS has been specifically designed to meet the requirements of broadcast and other digital cinema workflows, VR gaming applications, and sensor compression. Additional applications for the JPEG XS encoding include:

• Medical imaging
• Video surveillance
• Camera manufacturers
• Automotive Infotainment
• Mobile video applications
• Live-production

In a 2018 interview, Antonin Descampe, one of the lead engineers and co-founders of intoPIX, stated that, "the purposes of the JPEG XS standard is not to replace the JPEG. It is rather in specific areas that we want to intervene, such as proposing a standard that compresses very slightly without visual loss for existing infrastructure."

Comments welcome.

JPEG XS’s patented technology is made available for licensing via the JPEG XS Patent Portfolio License. The Fraunhofer-Institute for Integrated Circuits and intoPIX collaborated to develop the encoding standard and released the JPEG XS PPL license in 2021. The JPEG XS PPL uses three license models:

• Model 1: Pay Per Instance/Product
• Model 2: Pay Per Activation
• Model 3: Pay Per Use

More in depth information about the license models and reference terms are provided in a downloadable license overview.

Multi-platform interoperability serves as a key feature of the codec so that encoding and decoding on different platforms can support the required real-time implementations. The JPEG XS codec is designed to allow parallelization, which enables encoding and subsequent decoding in real time. JPEG XS also offers no significant quality in degradation for up to 10 encoding-decoding cycles.

Comments welcome.

ISO/IEC 21122-3 defines metadata information such as: color spaces, mastering display metadata (MDM), as well as EXIF metadata to facilitate transport, editing, and presentation.

The core coding requirements for JPEG XS detail multiple color space support (as part of the support for HDR content) including:

• Rec. BT.709 - a ITU-R standard for image encoding and signal characteristics of high-definition television.
• Rec. BT.2020 - another ITU-R standard that defines various aspects of ultra-high def television and SDR including frame rates, bit depths, color primaries, and RGB color representations.
• P3D65 - an RGB color space, used for digital motion picture distribution.
• RGB
• CFA
• YCbCr
• grayscale

Another key feature of JPEG XS is support for Bayer and CFA (color filter array) compression. Direct compression of original RAW sample values is possible without the need to covert to RGB samples.

The JPEG group initiated a call for proposals in early 2016 for a low-latency lightweight image coding system at the 71st JPEG meeting. The Call for Proposals document provides an overview of the development and communication timeline as well as the use cases, requirements, and key features that have been described above. Engineers and developers at intoPIX created a lightweight image compression algorithm that was eventually standardized into the JPEG XS encoding system.

Additional relevant news and timelines about the JPEG XS codec's development are maintained in press releases archived on JPEG’s site, which include summaries of annual JPEG meetings.

Older format specifications and related addendum that have since been replaced with updated versions are listed here::

• JPEG XS Part 1 (ISO/IEC 21122-1) Information technology — JPEG XS low-latency lightweight image coding system — Part 1: Core coding system.
• JPEG XS Part 2 (ISO/IEC 21122-2) Information technology — JPEG XS low-latency lightweight image coding system — Part 2: Profiles and buffer models.
• JPEG XS Part 3 (ISO/IEC 21122-3) Information technology — JPEG XS low-latency lightweight image coding system — Part 3: Transport and container formats.
• ISO/IEC 21122-1/CD Amd 1 Information technology — JPEG XS low-latency lightweight image coding system — Part 1: Core coding system — Amendment 1: Profile and sublevel for 4:2:0 content.