Sustainability of Digital Formats: Planning for Library of Congress Collections

Introduction | Sustainability Factors | Content Categories | Format Descriptions | Contact
Format Description Categories >> Browse Alphabetical List

High Efficiency Image File (HEIF) Format, MPEG-H Part 12

>> Back
Table of Contents
Format Description Properties Explanation of format description terms

Identification and description Explanation of format description terms

Full name ISO/IEC 23008-12 Information technology — High efficiency coding and media delivery in heterogeneous environments — Part 12: Image File Format (formal name); High Efficiency Image File Format -- HEIF (common name)
Description

The High Efficiency Image File Format (HEIF) is an international standard defined by MPEG-H Part 12 (ISO/IEC 23008-12), first published by ISO in 2017. The Scope section of ISO/IEC 23008-12:2017 states that the formats defined in the document enable the interchange, editing, and display of images, as well as the carriage of metadata associated with those images. The format defines an interoperable storage format for a single image, a collection of images, and sequences of images and also defines normative structures used to contain metadata, how to link that metadata to the images, and how metadata of certain forms is carried. HEIF is a special case of the ISO Base Media File Format (ISO_BMFF, ISO/IEC 14496-12). The recommended pronunciation of HEIF was clarified in the 2020 corrigendum to ISO/IEC 23008-12, which adds the following to the Introduction, "HEIF is suggested to be pronounced "heaff" (like heath with an ff ending)." Equivalently, other sources suggest it rhymes with "beef."

Nokia provides a useful introduction to HEIF at HEIF: Technical information, which states, "In ISO_BMFF, a continuous or timed media or metadata stream forms a track, whereas static media or metadata is stored as items. Building on this structure, HEIF has the following basic design:

  • Still images are stored as items. Typically image items are independently coded, and do not depend on any other item in their decoding. If predictively coded image items with coding dependencies are present, this is clearly signaled. Any number of image items can be included in the same file.
  • Image sequences are stored as tracks. An image sequence track can be indicated to be displayed either as a timed sequence or in a non-timed manner, such as a gallery of images. An image sequence track may be used instead of image items when there is coding dependency between images."

Nokia has provided another useful summary at https://github.com/nokiatech/heif from which the following is adapted: HEIF is a media container format. It is not an image or video encoder per se. Hence, the quality of the visual media depends highly on the proper usage of encoders. The initial standard supports image bitstreams encoded in HEVC (including its SHVC and MV-HEVC extensions), AVC, or JPEG. This can be extended to support future encodings. HEIF has features not present in other image file formats. Some of these features are:

  • Encapsulation of images coded using HEVC, SHVC, MV_HEVC, AVC, and JPEG
  • Encapsulation of image sequences coded using HEVC, SHVC, MV-HEVC, and AVC
  • Support for computational photography use cases
  • Support for both lossy and lossless image data storage
  • A way to distribute still images, image collections and related metadata.

The original 2017 specification for HEIF has a clause 9, entitled "Extensions to the ISO base media file format." In the ISO_BMFF format, all data is encapsulated in typed boxes. Boxes defined for HEIF in addition to those in ISO_BMFF include the following box types:

  • iprp: the ItemPropertiesBox enables the association of any item with an ordered set of item properties. Each property association may be marked as either essential or non-essential. A reader shall not process an item that is associated with a property that is not recognized or not supported by the reader and that is marked as essential to the item. A reader may ignore an associated item property that is marked non-essential to the item. Examples of properties are width and height (spatial extents); decoder configuration and initialization; pixel aspect ratio; color information (e.g., ICC profile or colour_type codes as defined for JPEG_XR); bit depth; and number of color channels.
  • grp1: the GroupsListBox provides a mechanism for grouping items or tracks

Additionally, enhancements to ISO_BMFF box types support image sequences. They include:

  • Support for repeating edits, using an enhancement to the EditListBox.
  • Support for sample-to-item sample grouping, so that samples of a track can be linked to one or more metadata items using an enhancement to the MetaBox.

The extensions described above were moved from the HEIF spec (ISO/IEC 23008-12) to the ISO_BMFF spec (ISO/IEC 14496-12) in 2020.

Excellent resources describing the features and structures of the HEIF format in an informal but technical way are videos of sessions at Apple's WWDC 2017 developer conference: Introducing HEIF and HEVC; High Efficiency Image File Format; Working with HEIF and HEVC (HEIF section begins at 20:00).

A summary of the requirements Apple laid out for picking an image format to use as a successor to the ubiquitous JPEG encoding (typically wrapped in a JFIF or JPEG_EXIF container) is to be found 3 minutes into the High Efficiency Image File Format video. The bullet points below are adapted slightly from the transcript. Note that some of the points are specific to the advantages of HEVC compression. However, many are related to the structured HEIF container format, which Apple engineers believed would address their requirements.

  • The new format needs to support all the features available in JPEG, but at the same time provide better performance. It needs to be friendly to professional photography tools, the web and the cloud. The new format also needs to be flexible and extensible to cope with the ever-changing photography ecosystem. Here is a list of features Apple considered paramount.
  • The compression needs to be state-of-the-art, both on the lossy and lossless front. It needs to be competitive with natural images, but also when compressing text or graphics. The format needs to be friendly to hardware accelerated code and decode operations on modern CPUs, GPUs and ESPs. Performance and power is very high in the list of requirements.
  • It needs to support high bit depth and wide color gamut which is the new frontier for images captured on consumer devices. It needs to be able to compress 4:4:4 color sampling and also describe HDR [High Dynamic Range] content, including HDR metadata, transfer function and color space definitions.
  • Auxiliary images, for example for alpha [channel for transparency] or depth [i.e. depth plane data], need to have a commonly defined place in image files. New editing tools will be able to utilize auxiliary data for new presentation and editing experiences.
  • In recent years, new ways to present and display animated images have been developed. Apple Live Photo is one example. Apple Live Photo includes animated content together with static images. The new common format needs to store animated information efficiently, ideally using temporal compression techniques, and be able to instruct players about the presentation intent for example, a looping sequence.
  • The new format needs also to support multiple images in the same file. For example, multi-exposure stacks or stereo images. This is to aid the development and implementation of new computational photography algorithms.
  • Multiple representations of the same image matching the same file are of great importance. For example, multi-resolution, including progressively increasing level of details or the ability to represent the same image encoded with different codecs.
  • Tiles are an important tool the new format must implement, allowing for scalable operations on images of any size.
  • The new format needs support for rich metadata associated to each image in the file. And also support for time metadata for example, for a sequence of images. There is also desire for a new format to be able to include other metadata types for example, audio or text.
  • Last but not least, the new format should be flexible and extensible enough to provide a solid foundation for the future.

Demonstrations of advantages of tiling and advanced editing using an auxiliary depth plane image can be seen in the Working with HEIF and HEVC video, from 22:30 through 26:30.

Production phase May be applied in initial-state picture creation in cameras or used for middle- and final-state archiving or end-user delivery.
Relationship to other formats
    Subtype of ISO_BMFF, ISO Base Media File Format
    May contain MPEG-4_AVC, MPEG-4, Advanced Video Coding (Part 10) (H.264)
    May contain JPEG, JPEG Image Encoding Family
    May contain HEVC, High Efficiency Video Coding (HEVC) Family, H.265, MPEG-H Part 2. HEVC can be used for still images or video.
    Has subtype HEIF_HEIC, High Efficiency Image File Format, HEIC/HEIX brands
    Has subtype MIAF (Multi-Image Application Format), defined in ISO/IEC 23000-22. Not described separately on this website. See https://nokiatech.github.io/heif/technical.html.

Local use Explanation of format description terms

LC experience or existing holdings The Library of Congress has a handful of images in the HEIC format (a subtype of the HEIF format) in its digital storage system. These are likely to have been acquired in the course of archiving websites.
LC preference Neither HEIF nor any of its subtypes are listed in the Library of Congress Recommended Formats Statement (RFS) for Still Image content in its collections.

Sustainability factors Explanation of format description terms

Disclosure International standard, fully disclosed. Developed and maintained under the auspices of the Motion Picture Experts Group (MPEG), which functioned as ISO/IEC JTC1/SC 29/WG11. After re-organization of SC 29 in 2020, maintenance of the HEIF standard will be by a new working group, perhaps WG3 (MPEG Systems).
    Documentation MPEG-H Part 12. See ISO/IEC 23008-12:2017 from the ITTF list of publicly available standards for full text and ISO/IEC 23008-12:2017 catalog record and preview for the initial specification. Support for HEVC, AVC, and JPEG encodings were defined in the initial 2017 specification. See Format specifications and Notes on history below for more details, particularly with respect to amendments and corrigenda.
Adoption

The HEIF file format has been adopted as an in-camera format for a number of cameras and mobile devices. In June 2017, Apple announced the adoption of HEIF at its WWDC 2017 developer conference. Built-in support for the format was introduced for capture on new iOS 11 phones. Qualcomm announced its first system on a chip (SOC) supporting HEIF in late 2018. See With Snapdragon 855 chip, Android phones get iPhone's photo-packing ability. In January 2020, Qualcomm introduced the lower-end SOC, Snapdragon 662, which also supports HEIF. Samsung introduced support for HEIF capture in its Galaxy S10 phones in 2019. See Here’s how to make Galaxy S10 camera photos take less storage space. Canon's EOS-1D X Mark III camera, introduced in January 2020, has the option of using HEIF internally as do the EOS R5 and R6 cameras, introduced later in 2020.

Support in operating systems began in 2017 with Apple's iOS 11 and Mac OS High Sierra. As of 2020, opening a .heic image taken on an iPhone in Preview or importing to the Photos app works as for JPEG, PNG, etc., and the Finder shows thumbnails. Inserting a .heic image into a Microsoft Word document works, with the image being silently converted to JPEG in the process. Support is available in Windows 10, but not without an explicit choice to add codecs. See Windows 10: Add Support for HEIC to Photos App. In Frequently Asked Questions about IrfanView, the answer to "How to load HEIC files in IrfanView? " offers equivalent advice. HEIF (.HEIC) Image Support for Linux/Ubuntu [quick guide] describes a mechanism for handling HEIF/HEIC files in Linux/Ubuntu, using the heif-gdk-pixbuf package.

Software libraries available include: HEIF parsers/readers/writers from Nokia; libheif, a decoder and encoder library for HEIF (with HEVC encoding, aka H.265) and AVIF (AV1 Image File Format, with AVC encoding, aka H.264); libde265, used by libheif. Abydos and qt-heif-image-plugin incorporate the libheif library.

CopyTrans HEIC for Windows (free for personal use) lets you view and print HEIC images, convert from HEIC to JPEG and insert HEIC images into MS Office documents. iMazing HEIC Converter will convert HEIC files to JPEG and PNG, available for Mac and Windows. See Wikipedia entry for High Efficiency Image File Format for a list of image editing applications with support for HEIF. Adobe Photoshop is not on the list; it appears from a long-running (3 years or so) discussion on Adobe's Photoshop community forum on the topic of Photoshop: HEIC Support, that the Mac version supports HEIC for import, although not for export. The compilers of this resource suspect that lack of support for .heic files in Windows in general, and in Photoshop for Windows in particular, is likely related to the licensing issues associated with HEVC compression codecs mentioned immediately below. Comments welcome.

    Licensing and patents

Since the HEIF container format is a special case of ISO_BMFF, see ISO_BMFF for information on its patent situation.

There are patents associated with implementations of encodings used within the container, including the encodings for which ISO/IEC 23008-12 includes support specifications in annexes. There are portfolio licenses available for HEVC (H.265) and AVC (H.264). See AVC/H.264 Patent Portfolio License and HEVC Patent Portfolio License. See also MP4_FF_2.

However, not all holders of relevant patents join these pools. For example, Nokia, which claims patents associated with H.264, aka Advance Video Coding (AVC), is not listed as a licensor in the patent pool linked in the previous paragraph. Nokia has deposited a number of patent statements with ISO related to ISO/IEC 23008-12 stating that it is prepared to grant a license to an unrestricted number of applicants on a worldwide non-discriminatory basis and on reasonable terms and conditions to make, use and sell implementations of the document. Note that, in 2020, Nokia is suing Lenovo for infringement of 20 patents in several countries. See, for example, Nokia seeks to block Lenovo sales in Germany over patent licensing (October 20, 2020) from Reuters. Lenovo is claiming that the terms are not reasonable.

Transparency Depends upon complex algorithms and tools to read; will require sophistication to build tools.
Self-documentation The HEIF specification supports the carriage of metadata written in various metadata schematic languages, including Exif, XMP, and MPEG-7 in XML form. Metadata can be applied to individual image items or to image sequences. See Annex A. Note that HEIF-compliant readers do not need to support these metadata formats. Also supported are a mechanism for recording integrity checks and timed metadata tracks for image sequences.
External dependencies The HEIF format inherits from ISO_BMFF the capability for media data (e.g., tracks and image items) to be stored outside the primary file and referred to by URL. The compilers of this resource have not determined whether this capability is used for HEIF files in practice. Comments welcome.
Technical protection considerations Content protection is allowed in the HEIF specification, following the ISO_BMFF capability, implemented in the Item Protection Box, with box type ipro. One possible approach is using the ISO Common Encryption standard defined in ISO/IEC 23001-7:2016 Information technology — MPEG systems technologies — Part 7: Common encryption in ISO base media file format files.

Quality and functionality factors Explanation of format description terms

Still Image
Normal rendering HEIF is a container format for raster images. Image quality depends on encoding. The container is designed to support images with large dimensions and high bit depth.
Color maintenance The HEIF format inherits the ability to incorporate ICC colour profiles from the ISO_BMFF format. Specifically, the box with type code colr uses the syntax of the ColourInformationBox defined in ISO/IEC 14496-12 (subclause 12.1.5.2). The syntax supports ICC profiles distinguishing two classes of profile, stating that, "an ICC profile may be a restricted one, under the code 'rICC', which permits simpler processing. That profile shall be of either the Monochrome or Three‐Component Matrix‐Based class of input profiles, as defined by ISO 15076‐1. If the profile is of another class, then the 'prof' indicator must be used." See ICC.1:2010 (Profile version 4.3.0.0), which is equivalent to ISO 15076‐1.
Support for multispectral bands The intent of the developers of the HEIF format did not include support for multispectral images and there is no specific support for incorporating collections of aligned images in individual frequency bands, as for example, in the FITS format. However, the number of channels per pixel does not appear to be limited by the HEIF file format. The number of channels will usually be constrained by a specific encoding format or profile. Some experiments with satellite imagery (captured using visible and non-visible frequencies) have been performed, but experiments such as those described at Lossy Compression of Multispectral Satellite Images with Application to Crop Thematic Mapping: A HEVC Comparative Study (May 2020) have focused on the compression efficiency for individual images in particular frequency bands. This particular paper compared compression efficiency using HEVC and JPEG 2000, finding HEVC superior. The compilers of this resource are not aware of any adoption of the HEIF file format for storing multispectral images. Comments welcome.
Functionality beyond normal rendering The container format supports image sequences and unsequenced image collections as well as single still image items. Inherits capabilities from ISO_BMFF to incorporate timed text or subtitles for image sequences.  ISO/IEC 14496-30:2018 specifies how to incorporate timed text into ISO_BMFF, specifically Timed Text Markup Language 1 (TTML1) and WebVTT: The Web Video Text Tracks Format, both from W3C.

File type signifiers and format identifiers Explanation of format description terms

Tag Value Note
Filename extension See note.  The file extension and Internet Media Type used for a file in the HEIF format will often depend on the encoding used for image compression. See examples below.
Pronom PUID fmt/1101
See https://www.nationalarchives.gov.uk/PRONOM/fmt/1101.
Wikidata Title ID Q24907733
See https://www.wikidata.org/wiki/Q24907733
Tag Value Note
Filename extension heif
heifs
.heif and .heifs are specified in Annex C of ISO/IEC 23008-12 for use as the extension for a file in the HEIF format with any compression encoding. .heif is used for one or more single images. .heifs is the extension used for a sequence of images.
Internet Media Type image/heif
See IANA registration.
Tag Value Note
Filename extension heic
heics
.heic is used for HEIF file containing one or more HEVC coded image items. This is the format introduced by Apple in 2017 as an in-camera format for iOS devices. .heics is used for image sequences.
Internet Media Type image/heic
See IANA registration.
Tag Value Note
Filename extension hif
Used by Canon for its HEIF format which uses HEVC compression. Added to Annexes C and D of ISO/IEC 23008-12 in a corrigendum in 2020; appears to apply to single images or image sequences. The compilers of this resource have not determined whether the compression encoding is significantly different from that used by Apple in its HEIC format. Comments welcome.

Notes Explanation of format description terms

General

HEIF, HEIC, HEVC, and more -- relationships and distinctions: The relationships among these three acronyms is confusing. According to the Introduction to ISO/IEC 23008-12, the file format specified in the document is referred to as the High Efficiency Image File Format (HEIF). ISO/IEC 23008-12 specifies both structural brands that can be used with any codec and brands specific to High Efficiency Video Coding (HEVC). HEVC is specified in ISO/IEC 23008-2 and the equivalent ITU-T Rec. H.265. Two HEVC-specific brands for single image items (heic and heix) are described in Annex B of ISO/IEC 23008-12; these brands are for encapsulating image items encoded using HEVC. An image item typically corresponds to a single photograph, but may include derivatives or auxiliary images. Brand heic is to be used when the encoding for the image item conforms to the Main profile or the Main Still Picture profile of HEVC. This is the most widely used of the HEVC-specific brands. Brand heix is to be used when the encoding for the image item conforms to the Main 10 profile or any of the format range extensions profiles of HEVC. Corresponding brands for image sequences, hevc and hevx, are also defined in Annex B. Further brands are defined for layered images encoded in HEVC: heim, heis, hevm, and hevs. Some brands have associated file extensions and internet media types registered through IANA. However, many brands are considered subtypes of another. For example, the IANA registration for image/heic indicates that image/heic is appropriate for a HEIF file conforming to the requirements for the 'heic', 'heix', 'heim', or 'heis' brand. When the requirements of the HEVC-specific brands defined in Annex B are obeyed, the file format can be referred to as the HEVC Image File Format. For an Apple-specific attempt to explain the confusing acronyms, see What the HEIC? Apple’s Highly Compressed Image Format Still Confuses (June 2020).

Subtype of HEIF: The MIAF (Multi-Image Application Format), defined in ISO/IEC 23000-22, builds on the HEIF specification, defining additional constraints aimed at furthering interoperability. The additional constraints limit the encodings to specified HEVC and AVC encoding profiles and levels and require specific metadata, including EXIF. MIAF Profiles include: MIAF HEVC Basic profile; MIAF HEVC Advanced profile; MIAF HEVC Extended profile; and MIAF AVC Basic profile. See Technical information summary from Nokia for an informal summary.

Considerations for archival use and forensic analysis: two analyses based on images found in practice appear to have useful information for those charged with long-term preservation of digital photographs.

First, Assessing the Potential Use of High Efficiency Video Coding (HEVC) and High Efficiency Image File Format (HEIF) in Archival Still Images, a 2018 analysis by Michael Bennett, evaluates HEVC’s attributes as both a mathematically lossless and as a visually lossless standard and compares HEVC to the JPEG 2000 specification which has gained steady adoption among cultural heritage institutions over time. Basic techniques for successfully encapsulating HEVC bitstreams in the new HEIF wrapper format are also examined in detail. The study which focused on detailed analysis of five very different image examples, yielded inconclusive findings with respect to the HEVC/HEIF’s tangible advantages as an archival still image format. Surprisingly, the compression possible using available tools for HEVC/HEIF and JPEG 2000, was greater for JPEG 2000 for visually lossless (42db PSNR) compressed images created from the same source TIFF files. The author found that comparisons were visually intriguing, offering glimpses into the unique underlying compression techniques that each specification employs. He also highlighted shortcomings in current software tools for applying HEVC compression in relation to profiles and options that the specification allows and argued for further analysis as more (and more mature) software becomes available.

Second, Forensic Considerations for the High Efficiency Image File Format (HEIF), a 2020 analysis by Sean McKeown and Gordon Russell of Napier University in Edinburgh, states "HEIF is a complex container format, much different from traditional still picture formats, this leaves forensics practitioners exposed to risks of potentially mishandling evidence. This paper describes the forensically relevant features of the HEIF format, including those which could be used to hide data, or cause issues in an investigation, while also providing commentary on the state of software support for the format. Finally, suggestions for current best-practice are provided, before discussing the requirements of a forensically robust HEIF analysis tool." This article provides very useful examples and analysis of software support at the time of writing. It highlights the fact that the HEIF format does not have a convenient file signature, stating, "Instead, the ftyp box must be located and the brands matched with appropriate HEIF brands, which contained the 'heic' brand for all of our test images, though this may not always be the case in reality. The 'pict' type in hdlr is also a good indication." The authors see particular challenges for forensic investigation in relation to HEIF's support for "hidden" images as well as image sequences. Also, many familiar tools can only preview the image identified as "primary". For example, Exiftool can only extract metadata for the primary image.

History

The ISO Base Media File Format (ISO_BMFF), on which HEIF is based, was first defined in 2001, published as ISO/IEC 14496-12 and ISO/IEC 15444-12 as a shared part of MP4 and JPEG 2000. Work on developing ISO/IEC 23008-12, the specification for HEIF began around 2013. A committee draft (CD) was presented at the 107th MPEG meeting in San José, California in January 2014. By MPEG 112, held in Warsaw in June 2015, according to the press release issued after the meeting, the specification for HEIF was considered technically complete. The first published edition of ISO/IEC 23008-12 was released in late 2017. Apple and Nokia had been heavily involved in the development process. For resources related to the development phase of HEIF and early adoption, see Useful References below.

A corrigendum to ISO/IEC 23008-12 was published in 2020; see ISO/IEC 23008-12:2017. Technical corrigendum 1 (2020) to download text or the corresponding ISO/IEC catalog record which provides the entire text as a preview. As well as minor clarifications and corrections, this corrigendum removes Clause 9, (entitled Extensions to the ISO Base Media File Format), replacing it with a reference to the updated version of ISO/IEC 14496-12 to which the clause content has been moved. The 2020 corrigendum also added .hif as a permitted file extension. See also ISO/IEC 23008-12:2017. Amendment 2 (2020): Support for predictive image coding, bursts, bracketing and other improvements.


Format specifications Explanation of format description terms


Useful references

URLs


Last Updated: 06/17/2021