Sustainability of Digital Formats: Planning for Library of Congress Collections

High Efficiency Video Coding (HEVC) is an international standard defined jointly by ISO/IEC (as ISO/IEC 23008-2) and ITU-T (as H.265) and also referred to as MPEG-H Part 2. It was first published in 2013. HEVC was seen as an evolution of earlier video coding specifications, namely ITU-T H.261 (MPEG-1), ITU-T H.262 (MPEG-2), ITU-T H.263 (MPEG-4, Visual Coding) and ITU-T H.264 (MPEG-4, Advanced Video Coding). HEVC was developed in response to the growing need for higher compression of moving pictures for various applications such as streaming, videoconferencing, digital storage media and television broadcasting. The HEVC syntax was designed to enable the use of the coded video representation in a flexible manner for a wide variety of network environments and to enable a high compression capability for a desired image or video quality.

The family of HEVC encodings comprises a large number of profiles (see below) aimed, according to the specification, at different application areas, including, but not limited to: broadcast (cable TV on optical networks / copper, satellite, terrestrial, etc.); camcorders; content production and distribution; digital cinema; home cinema; internet streaming, download and play; medical imaging; mobile streaming, broadcast and communications; real-time conversational services (videoconferencing, videophone, telepresence, etc.); remote video surveillance; storage media (optical disks, digital video tape recorder, etc.); and wireless display. Each profile is defined by constraints on the syntax as defined in the full HEVC (H.265) specification.

A concise summary of the structures available in the HEVC syntax is provided in High Efficiency Video Coding (HEVC) Test Model 16 (HM 16) Encoder Description Update 13 (January 2020): "HEVC has a block-based hybrid coding architecture, combining inter [temporal, see Notes below] and intra [spatial, see Notes below] prediction and transform coding with high efficiency entropy coding. However, in contrast to previous video coding standards, HEVC employs a quadtree coding block partitioning structure that enables a flexible use of large and small coding, prediction, and transform blocks. HEVC also allows for improved intra prediction and coding, adaptive motion parameter prediction and coding, a new loop filter and an enhanced version of context-adaptive binary arithmetic coding (CABAC) entropy coding over that defined by previous standards. New high-level structures have also been designed to aid parallel processing." The introduction to HEVC in a 2017 video titled Introducing HEIF and HEVC, presented at the Apple WWDC 2017 conference for developers at which Apple announced the new default formats being introduced for image and video capture on Apple mobile devices, discusses some key features that make HEVC different from the AVC (H.264) codec that preceded it. Evaluation of High Efficiency Video Coding (HEVC) for 3GPP services, a technical report from the 3rd Generation Partnership Project, also provides a useful description of HEVC, focusing on comparison to AVC.

HEVC is a important advance on AVC (H.264), in that it supports higher bit depths and enhanced chroma formats, including the use of full-resolution chroma (4:4:4). It might be assumed that a greater number of pixels in the source data signal leads to larger files and greater demand for transmission bandwidth. However, key innovations designed to support efficient compression mean that this is not necessarily the case. Chapter 9 of High Efficiency Video Coding (HEVC): Algorithms and Architectures describes in detail comparisons using objective and subjective measures in compression performance of the reference software for HEVC and AVC for a varied test set of video sequences. The objective evaluation used average differences of R-D (rate-distortion) curves based on PSNR (peak signal to noise ratio) measured against bit rate. Based on this objective measure, an overall performance improvement of of HEVC over AVC was between 22% and 43% for four different choices of coding parameters aimed at different use cases. Results of subjective evaluation tests indicated that an even higher bit rate saving in the ranges of 55–87% could be achieved. A separate objective evaluation of video sequences in the larger 4k resolution (3,840 x 2,160 pixels) showed up to 76% improvement.

Innovations highlighted in descriptions of HEVC include the following:

• One of the major features that contribute significantly to the coding efficiency of H.265 (HEVC), particularly for high resolution formats, is the usage of flexible block sizes for coding and transforms. Unlike H.264 (AVC), where the basic coding block is a macroblock of fixed size 16x16, H.265 (HEVC) defines a Coding Tree Unit (CTU) of a maximum size of 64x64. Each CTU can be divided into smaller units in a hierarchical manner (known as a quad-tree) and can represent smaller blocks of size 4x4. Similarly, the transforms used in H.265 (HEVC) can have different sizes, starting from 4x4 and going up to 32x32.
• Intra coding in HEVC allows 35 prediction modes – the DC [discrete cosine] mode, the planar mode and 33 directional (or "angular") modes. See subclause 8.4.2 of version 7 of the HEVC specification. AVC included 9 modes - the DC mode and 8 directional modes. See subclause 8.3.1.1 of version 13 (6/2019) of the AVC specification. The added angular modes can model different directional lines and structures in an image more accurately and the planar mode can smooth regions with gradually changing colors.  The increased flexibility improves both objective coding efficiency and visual quality as edges can be better predicted and ringing artifacts around the edges are reduced. To avoid contouring artifacts, a new interpolative prediction option is included to improve the visual quality.  Discrete Sine Transform (DST) is utilized instead of traditional Discrete Cosine Transform (DCT) for 4x4 intra transform blocks.  For full details, see Intra Coding of the HEVC Standard (2012).
• Prediction residuals after inter coding within individual blocks may be further compressed to remove spatial correlation inside the block before it is quantized, typically discarding less important visual information while forming a close approximation to the source data.
• HEVC has greater flexibility and adaptivity in its transform and quantization design and includes some additional coding modes in which the transform stage and sometimes also the quantization stage are skipped altogether. HEVC allows certain part of the coded picture to be coded in a lossless manner.
• In common with its AVC predecessor, HEVC contains an in-loop deblocking filter, where the blocking artifacts around the transform edges in the reconstructed picture are smoothed to improve the picture quality and compression efficiency. This filter has been simplified and made more parallel-friendly for HEVC.  Moreover, HEVC introduces a new filtering stage called the sample-adaptive offset (SAO) filter, which can provide both an objective and subjective improvement in video quality.
• HEVC uses a single entropy coding engine, based on Context Adaptive Binary Arithmetic Coding (CABAC). CABAC in HEVC shares many similarities with CABAC as specified for AVC, but contains several modifications that improved coding efficiency and lowered implementation complexity, especially for parallel architectures.
• Any bitstream compliant with HEVC is organized into network abstraction layer (NAL) units, which are self-contained packets that allow the video layer to be identical for different transmission environments. Each NAL unit is associated with a particular temporal layer. The syntax is constrained in such a way that a decoder capable of decoding at a given frame rate is expected to discard NAL units associated with intermediate pictures used for higher frame rate decoding. This feature is known as "temporal scalability." See section on hierarchical encoding into temporal layers from an Apple presentation for developers at WWDC 2017 or the diagram Two HEVC encoder methods for block artifact reduction: Fig. 4.

Versions and profiles: As of November 2020, there are seven versions of the HEVC specification. Most of the updates have introduced new profiles, as listed below. In a profile name that ends in a number, that number indicates the maximum bit depth per channel for color representation. If a profile does not include a chroma indication, and is not for a monochrome picture, the profile is limited to 4:2:0 chroma support.

• ITU-T H.265 (V1) (04/2013). Version 1 of the HEVC specification introduced Main, Main Still Picture, and Main 10 profiles. These are all limited to the 4:2:0 chroma format. The Main and Main Still Picture profiles are limited to 8 bits per color channel.
• ITU-T H.265 (V2) (10/2014). Version 2 of the HEVC specification introduced a large number of profiles, of which most extended bit-depth and/or chroma support and are often referred to as a group, as Range Extension (RExt) profiles. 4:2:2 profiles addressed needs for camera-captured video for broadcast. 4:2:0 and 4:2:2 with bit depth 12 addressed expectations for ultra-high definition (UHD) equipment and services, for which support for high dynamic range (HDR) video would be needed. See Overview of the Range Extensions for the HEVC Standard: Tools, Profiles, and Performance (2016). The new profiles were:
  • Monochrome, Monochrome 12, Monochrome 16
    The Monochrome profile, which allows 8 or 10 bits per pixel, is suitable for alpha (transparency) channels and depth maps. Monochrome images with higher bit depth are used in medical imaging applications.
  • Main 12
  • Main 4:2:2 10, Main 4:2:2 12
  • Main 4:4:4, Main 4:4:4 10, Main 4:4:4 12
    Supporting both inter-picture and intra-picture coding. 4:4:4 chroma variants are important not only for for studio and professional content production, but also for applications involving remote screen-sharing.
  • Main Intra, Main 10 Intra,  Main 12 Intra,  Main  4:2:2 10 Intra, Main 4:2:2 12 Intra, Main 4:4:4 Intra, Main 4:4:4 10 Intra, Main 4:4:4 12 Intra, Main 4:4:4 16 Intra
    The Intra profiles are only permitted to use intra-picture prediction, resulting in coding that only includes I-frames.
  • Main 4:4:4 Still Picture, Main 4:4:4 16 Still Picture
  • High Throughput 4:4:4 16 Intra
    Intended for codecs embedded in professional cameras.
  Additional profiles added in version 2 of HEVC include:
  • Multiview Main (referred to as MV-HEVC), in Annex G.
    The first profile from the JCT-3V group. It allows efficient coding of multiple camera views and associated auxiliary pictures. An important application is for stereoscopic images. See Overview of the Multiview and 3D Extensions of High Efficiency Video Coding.
  • Scalable Main, Scalable Main 10 (referred to as SHVC), in Annex H.
    Scalability of video coding refers to the encoding of a high-quality video bitstream that contains one or more subset bitstreams that can themselves be decoded with a reasonable quality at a lower transmission rate. The subset bitstream is derived by dropping packets from the larger bitstream. In addition to the temporal scalability already provided by the first version of HEVC, SHVC provided spatial, signal-to-noise ratio, bit depth, and color gamut scalability functionalities, as well as combinations of any of these. See Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard (2015) and SHVC, the Scalable Extensions of HEVC, and Its Applications.
• ITU-T H.265 (V3) (04/2015). Version 3 of the HEVC specification introduced one profile.
  • 3D Main in Annex I. This profile provides support for 3D high efficiency video coding (3D-HEVC), specifying a syntax and associated decoding process for efficient coding of video textures and depth maps for 3D video applications. See Overview of the Multiview and 3D Extensions of High Efficiency Video Coding.
• ITU-T H.265 (V4) (12/2016). Version 4 of the HEVC specification introduced several new profiles and defined additional supplemental enhancement information (SEI) payloads and additional color representation identifiers. This revision also corrected various minor defects in the prior content of the specification. The following profiles were added to Annex A.
  • High Throughput 4:4:4, High Throughput 4:4:4 10, High Throughput 4:4:4 14
  • Screen-Extended Main, Screen-Extended Main 10. The screen-extended profiles, referred to collectively as HEVC-SCC, take advantage of the fact that screen content frequently contains no sensor noise, and such content may have large uniformly flat areas, repeated patterns, highly saturated or a limited number of different colors, and numerically identical blocks or regions among a sequence of pictures. These characteristics offer opportunities for significant improvements in compression efficiency over a coding system primarily designed for camera-captured content. See, for example, Overview of Screen Content Video Coding: Technologies, Standards, and Beyond (2016) or Overview of the Emerging HEVC Screen Content Coding Extension (2015).
  • Screen-Extended Main 4:4:4, Screen-Extended Main 4:4:4 10
  • Screen-Extended High Throughput 4:4:4, Screen-Extended High Throughput 4:4:4 10, Screen-Extended High Throughput 4:4:4 14
  The following scalable profiles were added to Annex H:
  • Scalable Monochrome, Scalable Monochrome 12, Scalable Monochrome 16
  • Scalable Main 4:4:4
• ITU-T H.265 (V5) (02/2018). Version 5 of the HEVC specification added SEI message payloads, including messages to support omnidirectional 360° video as used in augmented reality (AR) and virtual reality (VR) applications. See , for example, Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation (2017). Minor defects in the prior content of the specification were corrected. Profiles added were:
  • Monochrome 10
  • Main 10 Still Picture
• ITU-T H.265 (V6) (06/2019). Version 6 of the HEVC specification adds additional SEI messages that include SEI manifest and SEI prefix messages, and corrections to various minor defects in prior content. No profiles were added.
• ITU-T H.265 (V7) (11/2019). Version 7 of the HEVC specification adds additional SEI messages for fisheye video information and annotated regions, and also includes corrections to various minor defects in prior content. No profiles were added.

To support "islands of interoperability," HEVC defines not only profiles, but also levels and tiers. The discussion here is based not only on the HEVC specification but also on content found in High Efficiency Video Coding (HEVC): Algorithms and Architectures. While profiles define the syntax and coding features that can be used for the video content, a significant consideration is the degree of capability within a given feature set. Levels are levels of capability defined to establish the picture resolution, frame rate, bit rate, buffering capacity, and other aspects that are matters of degree. The lowest levels have only low resolution and low frame rate capability—e.g., a typical video format for level 1 may be only 176x144 resolution at 15 frames per second, whereas level 4.1 would be capable of 1920x1080 HDTV at 60 frames per second and level 6.1 would be capable of 8192x4320 video resolution at up to 120 frames per second. As of Version 7, Table A.8 of the HEVC specification lists the following general levels: 1, 2, 2.1, 3, 3.1, 4, 4.1, 5, 5.1, 5.2, 6, 6.1, and 6.2. Note that when a bitstream is indicated to conform to a specific level it is also considered as conforming to higher levels. Levels higher than 4 have variants for Main and High tiers, with a key difference being that the High tier defines a much higher maximum size for coded picture buffers (CPB), which increases the bit rates decoders must be capable of handling. The Still Picture profiles may use a special level (8.5) that indicates no limit on picture size or complexity.

One use of levels is in specifications based on HEVC encoding. For example, DICOM specifies the use of Main or Main 10 profiles at level 5.1 and ATSC 3.0 - A/341 refers to levels 3.1, 4.1, 5, 5.1, and 5.2 for Main 10 and Scalable Main 10 profiles. The compilers of this resource have not made an attempt to explore the relative adoption of particular profiles, tiers, and levels. Comments welcome.

The HEVC specification is published as ISO/IEC 23008-2: Information technology — High efficiency coding and media delivery in heterogeneous environments — Part 2: High efficiency video coding (aka MPEG-H Part 2) and as ITU-T H.265: High efficiency video coding. See ITU Recommendation H.265 (approved 11/2019) for full text of Version 7 of ITU-T H.265. See ISO/IEC 23008-2:2020 catalog record and preview for the equivalent ISO/IEC specification published in August 2020.

The HEVC section of the MPEG website has been a rich source of drafts, white papers, and intermediate documentation associated with the initial development of HEVC and its steady extension to cover additional use cases by adding profiles. The website has been largely frozen as of June 2020 as a result of the re-organization of ISO/IEC JTC1/SC 29.

While the specification describes the syntax and a suggested decoding process for coded video compliant with the standard, a possible encoding process is described in documentation for reference software, often referred to as HM (for HEVC Test Model). See, for example, High Efficiency Video Coding (HEVC) Test Model 16 (HM 16) Encoder Description Update 13 (January 2020), which serves as an overview of HEVC Version 1 and the Range Extensions of HEVC Version 2, and also provides an encoder-side description of the HM-16.21 software version. The HM software is open source, maintained at https://vcgit.hhi.fraunhofer.de/jct-vc/HM.

Two books by members of the JCT-VC development group were published in 2014 and 2105 as informative companions to the formal HEVC specification.

• High Efficiency Video Coding (HEVC): Algorithms and Architectures, Editors: Vivienne Sze, Madhukar Budagavi, Gary J. Sullivan. Has separately authored chapters on different aspects of HEVC, including: block structures and parallelism; intra-picture prediction; inter-picture prediction; transform and quantization; in-loop filters; and entropy coding. Also includes compression performance analysis in comparison to AVC (H.264) using subjective and objective methods.
• High Efficiency Video Coding: Coding Tools and Specification, by Mathias Wien. Includes history and technical background on video coding and comparisons with AVC in all the chapters that relate to aspects of video coding (such as temporal and spatial predictive coding, color representation, etc.). Introduces the range extension (RExt) profiles included in version 2 of the HEVC specification.

Some standards organizations other than ISO/IEC and ITU have incorporated support for particular HEVC profiles into their own standards:

• The Advanced Television Systems Committee, Inc. (ATSC) published its ATSC Standard: Video – HEVC (A/341) in 2019. Chip makers and software vendors supporting this standard include Cobalt Digital Inc. and Unisoft (links via Internet Archive) with its ATEME TITAN Live HEVC encoder with DASH packager and other products.
• The 3rd Generation Partnership Project (3GPP) incorporates use of some profiles of HEVC into the following standards: IP Multimedia System (IMS) Messaging and Presence; Media formats and codecs (TS 26.141); IP Multimedia Subsystem (IMS); Multimedia telephony; Media handling and interaction (TS 26.114); Multimedia Messaging Service (MMS); Media formats and codecs (TS 26.140); Multimedia Broadcast/Multicast Service (MBMS); Protocols and codecs (TS 26.346); and three standards related to Transparent end-to-end Packet-switched Streaming Service (PSS) -- Protocols and codecs (TS 26.234), 3GPP file format (3GP -- TS 26.244), and Progressive Download and Dynamic Adaptive Streaming over HTTP (3GP-DASH -- TS 26.247). See 3GPP supports High Efficiency Video Coding.
• DICOM (Digital Imaging and Communications in Medicine), a widely used standard for medical images, incorporates two variant substypes of HEVC as of November 2020 in DICOM PS3.5 2020e - Data Structures and Encoding. In particular, see PS3.5 -- 8.2.10 HEVC/H.265 Main Profile / Level 5.1 Video Compression and PS 3.5 -- 8.2.11 HEVC/H.265 Main 10 Profile / Level 5.1 Video Compression. These encoding options were added to DICOM in 2016. See DICOM Supplement 195: HEVC/H.265 Transfer Syntax.

HEVC has been adopted in many parts of the video creation and distribution business. The Wikipedia entry for High Efficiency Video Coding lists many of the early adopters. A selection of other examples follows:

• Socionext sells a number of chips for hardware encoding and decoding of HEVC video. See Socionext | H.264/H.265 Codec.
• Allegro DVT has developed and licenses decoder software to test compliance with many HEVC profiles and quality of encoded video. See Allegro | H.265/HEVC Elementary Streams. The company also licenses multiformat encoder and decoder software suitable for integration into System-On-Chip (SoC) designs. Several HEVC profiles are supported in a variety of such software products from Allegro.
• Camcorders such as the Canon XF705 can record straight to HEVC, in this case in 4:2:2 10-bit.
• The Ultra HD Blu-ray format (aka 4K Ultra HD) is based on HEVC encoding. See Wikipedia entry for Ultra HD Blu-ray.
• The first versions of Apple's operating systems to support viewing, editing, and conversion of HEVC video were iOS 11 and Mac OS 10.13 (High Sierra), released in 2017. See Using HEIF or HEVC media on Apple devices. For playback, support is for Main 10 Profile, Level 5.0, High Tier. Apple usually stores captured HEVC video content in its Quicktime container format with the .mov extension, but also supports HEVC in an MPEG-4 container. An important application of HEVC for Apple is in HTTP Live Streaming (HLS). Apple's developer tools, such as AVFoundation, support HEVC, which enables use in contexts that use the AVFoundation API; see How to make HEVC, H265 and VP9 videos with an alpha channel for the web.
• Support is available for HEVC in Windows 10, but not without an explicit choice to add an HEVC codec. See How to Install Free HEVC Codecs on Windows 10 (for H.265 Video), recognizing that an October 2020 update to the article indicates that the free encoder formerly available from Microsoft has been withdrawn but the codec costing 99 cents is available.
• Adobe products for video encoding and editing that support HEVC include: Premiere Pro, After Effects and Media Encoder. Apple's Final Cut Pro also supports HEVC (see Media formats supported in Final Cut Pro), including in Canon's XF-HEVC container.
• The online video platform Brightcove supports HEVC. See Brightcove's HEVC solution.

MainConcept provides a plugin HEVC encoder for the widely used FFmpeg video toolkit. An encoding library for HEVC is available from x265.com. x265 source code is available for developers who would like to contribute to the project, or build open-source applications under the terms of the GNU GPL v2 license. Commercial licenses are available for companies who want to use x265 in video hardware and software products. An open source encoder for HEVC has also been made available by the VideoLAN project. See Videolan -- H.265. The compilers of this resources have not determined whether these projects are connected. Comments welcome.

As is the case for many moving image formats, the intra-picture features of HEVC can be used to compress still images. One such use of the HEVC is in the default format with a .heic extension used by Apple in its iPhones and iPads since 2017. This format employs the High Efficiency Image File Format (HEIF) as a container for a single image or collection of images, where the individual image items are compressed using certain HEVC profiles (Main, Main Still Image, and Main 10). See HEIF_HEIC for information about use of HEVC for still images in cameras from Sony and Canon and cellphones from Samsung.

The underlying HEVC encoding is heavily encumbered with patents. According to HEVC Advance Passes 10,000 Patent Milestone - announces Toshiba Corp. Joins as a Licensor, a March 2020 press release, there are around 17,000. Patents have proved a general problem for video coding schemes. See, for example, Patents, Standards Brew Licensing Woes (June 2019) from EE Times. Rather than there being a single patent pool, there are three, as shown in a blog post from January 2018 by Leonardo Chiariglione.

In January 2019, a study of HEVC / H.265 standard-essential patent royalty rates was released by Unified, an international membership organization that seeks to improve patent quality and deter unsubstantiated or invalid patent assertions in defined technology sectors (Zones). One zone for Unified's activities is video codecs. The report, based on a methodology Unified describes as objective economic evaluation (OVAL), claims that it "demonstrates that HEVC rates should be comparable to or less than the cost of licensing the main patent pool for AVC (MPEGLA), HEVC’s predecessor codec." The report found that "each of the three major licensors are asking for licensing rates that likely exceed what the total HEVC rate should be, both individually and when taken together."

It is clear to the compilers of this resource that the patent situation has been an impediment to adoption. In particular, the lack of built-in support for HEIF formats in browsers and in operating systems other than Apple's iOS and Mac OS seems likely to be related to the challenges and risks associated with the patent landscape. Comments welcome. Although related to the earlier AVC (H.264) coding scheme, the patent-related battle between Nokia and Lenovo in Germany in 2020 illustrates the risks. See, for example, Setback for Nokia in German patent battle with Lenovo, a November 2, 2020 post by Reuters. Nokia has not joined any of the HEVC patent pools.

Terminology The terminology used in the HEVC specification and descriptions of the encoding is complex and sometimes extends or modifies terms found in glossaries such as the Glossary of Video Terms and Acronyms from Tektronix, the FADGI Glossary, or the Wikipedia entry for Video compression picture types. The notes below relate to terms for which the compilers of this resource did not find definitions in these sources that covered HEVC usage.

• Quadtree (often hyphenated). A mathematical term for a tree in which a parent node can be split into four child nodes, each of which may become parent node for another split into four child nodes. For an example of a quadtree as used in HEVC, see Block Structures and Parallelism Features in HEVC: Figure 3.
• CTU (Coding Tree Unit). In HEVC a picture is divided into CTUs, which are square blocks except possibly for the bottom row and right-hand column. A CTU may be further divided into 4 child CTUs.
• Tiles and Slices. HEVC has two additional types of partition that may be used for a picture. They consist of sequences of CTUs. A tile is a rectangular region of CTUs within a rectangular grid of tiles that form a picture; tiles are treated like independent pictures. A slice is a sequence of CTUs that can be decoded independently from other slices of the same picture. A slice can either be an entire picture or a region of a picture. One of the main purposes of slices is resynchronization in the event of data losses. In contrast to common usage, a slice in HEVC is not limited to a single row of blocks in a rectangular grid. See subclause 6.3 in the HEVC specification or Block Structures and Parallelism Features in HEVC: Figure 6.
• I-slices (intra slices) are independently compressed HEVC slices. An I-slice is decoded using intra prediction only. An extension of the more common term, I-frame.
• P-slices (predictive slices) employ inter-picture coding, based on differences from previous pictures in the video source. An extension of the more common term, P-frame.
• B-slices (bi-predictive slices) are bidirectionally predicted slices using differences between the current picture and both preceding and following pictures to derive coded content for the current slice. An extension of the more common term, B-frame.
• Intra coding uses various spatial prediction modes to exploit spatial statistical dependencies in the source signal for a single picture or tile.
• Inter coding uses motion vectors for block-based prediction to exploit temporal statistical dependencies between different pictures. In HEVC, inter prediction can apply to block and slice substructures, not only to entire pictures or tiles.
• SEI: supplemental enhancement information. An SEI message contains information (often referred to as the message payload) that is not necessary simply to decode the samples of coded pictures, but is a standardized, structured message to a decoder that can assist in processes related to decoding, display, or other purposes. Some advanced features, such as 3-D, will not function as intended in an application that does not recognize the corresponding SEI messages.
• Levels consist of a defined set of constraints on the values that may be taken by HEVC syntax elements and variables or the value of a transform coefficient prior to scaling. A common "general" set of levels is defined for all HEVC profiles. Hence most aspects of the definition of each level are common across different profiles. Within a coded video sequence, profiles, levels, and tiers are identified in very terse fashion in parameter sets and not accessible to simple tools. See File type signifiers above.
• Tiers are specified categories of level constraints imposed on values of the syntax elements in the bitstream, where the level constraints are nested within a tier and a decoder conforming to a certain tier and level would be capable of decoding all bitstreams that conform to the same tier or the lower tier of that level or any level below it.
• CPB (coded picture buffer) is a buffer containing decoding units in decoding order specified in the hypothetical reference decoder in Annex C of the HEVC spec. The size of CPB in content coding generated by an encoder is an important factor for decoders. Levels specify a maximum CPB size in bits.
• Parameter sets in HEVC are similar to the parameter sets in H.264/AVC, and share the same basic design goals—namely bit rate efficiency, error resiliency, and supporting systems layer interfaces. There is a hierarchy of parameter sets in HEVC, including the Sequence Parameter Set (SPS) and Picture Parameter Set (PPS) which are similar to their counterparts in AVC. HEVC introduced a new type of parameter set called the Video Parameter Set (VPS). The parameter sets are stored in special network abstraction layer (NAL) units, with specific NAL unit types encoded as binary integers at the beginning of the unit. See Table 7-1 in the HEVC specification or Table I in Overview of the High Efficiency Video Coding (HEVC) Standard for a mapping of the integer codes to human-readable names. For compression efficiency, these parameter sets are very tersely coded, primarily using 1-bit flags and binary integers. Note that if an HEVC coded video sequence or still image is embedded in an ISO_BMFF container as specified in ISO/IEC 14496-15, the VPS and SPS parameter sets will be found in an HEVC configuration item property in a box of type hvcC.
• SPS (Sequence Parameter Set). Contains parameters that apply to an entire coded video sequence, and do not change from picture to picture within a coded video sequence. The SPS has codes for characteristics such as bit-depth, chroma format, picture width and picture height. Has NAL unit type of 33.
• VPS (Video Parameter Set). A new parameter set defined in HEVC, which applies to all of the layers of a bitstream. A layer may contain multiple temporal sub-layers. Has NAL unit type of 32 and is stored before the SPS.
• PPS (Picture Parameter Set). There is one PPS per picture. Among the details stored in this parameter set are: number of tile rows; number of tile columns; flags related to whether particular extensions, e.g., for 3D or screen content coding(SCC) are used; and other parameters needed for decoding the picture. Has NAL unit type of 34.

The article titled Overview of the High Efficiency Video Coding (HEVC) Standard, published in 2012 before the HEVC specification had been approved as a standard, discussed the antecedent standards. The summary that follows is adapted from that discussion.

Some important video coding standards have evolved in a series of well-known ITU-T and ISO/IEC standards. The ITU-T produced H.261 and H.263; ISO/IEC produced MPEG-1 and MPEG-4 Visual; and the two organizations jointly produced the H.262/MPEG-2 Video and H.264/MPEG-4 Advanced Video Coding (AVC) standards. Throughout this evolution, efforts have been made to maximize compression capability, while considering the computational resources that were practical for use in products at the time of anticipated deployment of each standard.

The two standards that were jointly produced had been widely deployed. The standard preceding HEVC, H.264/MPEG-4 AVC, was developed in the period between 1999 and 2003, and extended in several important ways from 2003–2009. By 2012, AVC had been widely used for many applications, including broadcast of high definition (HD) TV signals over satellite, cable, and terrestrial transmission systems, video content acquisition and editing systems, camcorders, security applications, Internet and mobile network video, Blu-ray Discs, and real-time conversational applications such as video chat, video conferencing, and telepresence systems. By 2012, AVC had largely displaced the older H.262/MPEG-2 Video standard within its more limited application domains.

Key drivers behind the development of HEVC were: the growing popularity of HD video, and the emergence of yet larger formats (e.g., 4k×2k or 8k×4k resolution); and the networking issues caused by the growing volume of video applications targeting mobile devices and the transmission needs for video-on-demand services. HEVC was designed with a focus on two key issues: increased video resolution and increased use of parallel processing architectures, while still addressing applications that had been supported by AVC.

The first version of the HEVC (H.265) specification was published in April 2013. Editions of the standard are listed below:

• ITU-T H.265 (V1) (04/2013) http://handle.itu.int/11.1002/1000/11885
• ITU-T H.265 (V2) (10/2014) http://handle.itu.int/11.1002/1000/12296
• ITU-T H.265 (V3) (04/2015) http://handle.itu.int/11.1002/1000/12455
• ITU-T H.265 (V4) (12/2016) http://handle.itu.int/11.1002/1000/12905
• ITU-T H.265 (V5) (02/2018) http://handle.itu.int/11.1002/1000/13433
• ITU-T H.265 (V6) (06/2019) http://handle.itu.int/11.1002/1000/13904
• ITU-T H.265 (V7) (11/2019) http://handle.itu.int/11.1002/1000/14107

Two more recent video encodings have been developed, namely Versatile Video Coding (VVC) and Essential Video Coding (EVC). VVC, also known as ITU-T H.266, aims at improved compression over HEVC, yet higher spatial resolution, and full support for immersive applications. The first edition was approved in 2020 as ITU-T H.266 (08/2020). See also VVC Overview from Fraunhofer HHI. The objective of EVC is to be a royalty-free standard.