Sustainability of Digital Formats: Planning for Library of Congress Collections

Introduced at the NAB conference in 2018, Apple ProRes RAW and Apple ProRes RAW HQ, which together make up the Apple ProRes RAW Codec Family, are proprietary, lossy compressed video codecs designed to support high-dynamic-range (HDR) raw image sensor data content. In the ProRes RAW White Paper (dated November 2020 as of this writing but updated frequently), Apple states that the ProRes RAW family is based on the same principles and underlying technology as existing Apple ProRes 442 Family and Apple ProRes 4444 Family codecs, most notably support for intraframe (I-frame) only and variable bit rate. ProRes RAW expands the bit depth capability up to 16-bit sample per image channel (beyond the 10-bit limit for ProRes 422 and 12-bit for ProRes 4444) and supports any frame size up to 8K (beyond the previous 5K limit) at full resolution. The unique feature of ProRes RAW is that this technology is "applied to a camera sensor’s pristine raw image data rather than conventional image pixels."

As the ProRes RAW White Paper explains with a helpful chart on p.4, single sensor digital cameras capture color at each photosite (or pixel) in a filter arrangement known as a Bayer pattern in which "each two-by-two group of photosites consists of one red, one blue, and two green sensor elements. In contrast, conventional RGB images consist of not one but three primary color values (red, green, and blue) at each pixel location." However this Bayer pattern raw data is not viewable on monitors as it does not contain any information about color or brightness at a specific pixel. Every camera captures this raw image data but typically, it's not accessible or exportable and the raw data needs to be converted for viewing and use from RAW to RBG. This conversion is called demosaicing (also known as de-mosaicing or debayering) and the process by which it happens is determined by the camera's built in firmware and its assigned algorithm to convert the image data. Since each algorithm has strengths and weaknesses (i.e., some work well for natural scenery while other work well for low noise imagery), this lack of choice can be limiting for the final camera output.

A benefit of ProRes RAW, and similar RAW image format including Canon Cinema RAW Light, Blackmagic RAW, REDCODE and ARRIRAW, is that it directly encodes the Bayer pattern image captured from the camera sensors so that processing, such as demosaicing, linear-to-log conversions, and custom LUTs, is deferred to the time of playback and are performed by application software. This allows the user to "take advantage of raw processing enhancements and demosaicing algorithm improvements in future software" and while also preserving more detail and dynamic range for editing in post production. Shooting in raw captures the camera sensor’s full gamut and preserves more colors and dynamic range than can be displayed in ITU-R BT.709-6. Shawn C. Steiner of B and H Photo summarizes "This means that when you pull it up on your computer, your files contain the maximum possible amount of information. Need to salvage some nearly blown highlights? Done. Want to pull the detail out of your shadows dramatically? You are set. How about keeping your lens’s unique vignetting or distortion for creative effect? All good. The original data is preserved for you to make all the decisions later." ProRes RAW is useful for storing video direct from camera onto an external recorder, raw data workflows and HDR while ProRes 422 and 4444 codecs will continue to be useful as intermediate codecs.

ProRes RAW's advantages over other RAW formats is that when used in Final Cut Pro, ProRes RAW offers superior performance in both playback and rendering enabling the same real-time, multistream video editing performance. Apple reports that "compared to other raw video formats supported by Final Cut Pro, ProRes RAW offers superior performance in both playback and rendering" with both ProRes RAW and ProRes RAW HQ able to support 33 simultaneous 4K streams as opposed to Canon Cinema RAW Light's 3 and REDCODE 5.1's 2 streams. It also has a significant speed advantage over the same two formats for a variety of export options: for HEVC, 3.0x faster than Canon Cinema RAW Light and 2.8x faster than REDCODE 5.1; H.264, 3.6x faster than Canon Cinema RAW Light and 3.5x faster than REDCODE 5.1; and render speeds of 7.7x faster than Canon Cinema RAW Light and 7.5x faster than REDCODE 5.1.

Target data rates for the two compression levels are variable depending on the content. According to Larry Jordan, "data rate will vary as the complexity of the image increases. This variable data rate means that files sizes for the same duration video may vary as well." The The Tiliam Blog explains that for data rates, "ProRes RAW falls between ProRes 422 and ProRes 422 HQ because "the Bayer requires 1 sample per pixel but YCbCr 4:2:2 requires the equivalent of 2 samples per pixel" and ProRes RAW HQ falls between ProRes 422 HQ and ProRes 4444 because "YCbCr plus alpha requires 4 samples per pixel but RAW requires 1 sample." ProRes Raw does not have a version with an alpha channel (as found in ProRes 4444) which creates transparency in compositing or layering workflows by defining how much of the RGB or YCbCr pixels of a foreground image should be blended with the pixels in the background image. See Notes for more information comparing the ProRes data rates. Larry Jordan states that "ProRes RAW is about 1/3 the size of an uncompressed RAW file and can be easily edited on a relatively recent MacBook Pro laptop. It also supports multicam editing, with the number of simultaneous streams depending upon your hardware."

A note on the capitalization of the word "raw". The word is capitalized when referring to image formats including ProRes RAW, ARRIRAW, and the like. RAW is not an acronym; discussion in the StackExchange forum suggests the capitalization is following the typical pattern of other file format names like JPEG or TIFF (although these are acronyms for Joint Photographic Experts Group and Tagged Image File Format respectively). The capitalization is helpful because "One can speak about raw data, which is unedited, that is shot in RAW format, and the capitalization specifies it."

ProRes codecs were usually contained within the QuickTime "mov" wrapper but starting with Final Cut Pro 10.3 released on October 27, 2016, the option for using ProRes in the MXF Generic Container was added as an option for broadcast delivery. Previously, MXF import and export was only available through third-party plug-ins starting with version 10.0.6 released in October 23, 2012. See Notes for more information.

Apple introduced ProRes RAW in 2018 with the release of Final Cut 10.4.1. Apple maintains the full list of ProRes RAW licensed products and Final Cut Pro integrated products such as Motion and Compressor. Non-Apple products include Telestream's Switch multiplatform media player, Assimilate's SCRATCH colorgrading and compositing software (starting April 2019), Colorfront's On-Set Dailies and Express Dailies products (also starting April 2019), FilmLight's Baselight version 5.2 grading software and Daylight dailies platform for shot management and high-performance transcoding, DJI's Zenmuse X7 camera and Inspire 2 drone, MTI Film's Cortex cameras and a variety of Atomos monitors-recorders including Sumo 19, Ninja V, Ninja V Pro Kit and Shogun 7.

Because ProRes RAW is an acquisition format created by cameras and video recorders with RAW data straight from the camera sensor, it's not possible to create ProRes RAW files from video software. To export ProRes RAW files to other formats, use Final Cut Pro, Motion, or Compressor. ProRes RAW files can not be exported to other formats with QuickTime Player.

Apple licenses and certifies specific third party products and workflows. The full list of licensed products is available. Licenses are not required for content creators.

Apple's White Paper states that compared to other raw video formats supported by Final Cut Pro, ProRes RAW offers superior performance in both playback and rendering enabling the same real-time, multistream video editing performance with Apple ProRes RAW as with standard media.

The performance of Apple ProRes codecs scales in multiprocessor environments so that the decoding time per frame goes down as the number of processors increases. When the system spends less time decoding each frame, it has time for more real-time effects processing.

In addition, its scalar quantization allows for the possibility to decode a video at different partial resolutions (½ or ¼ etc size of the full frame size) which lowers the data throughput while maintaining high image quality. This functionality allows the user advantages in playback of additional tracks of video and higher quality previews.

The 2019 What Is ProRes RAW article in Full Exposure includes details comparing data rates for ProRes RAW against other ProRes codecs. The data rates are variable depending on the content including scene dynamic range and color complexity but as a general guide for estimating file size for a 4K UHD 60p 10-bit file:

ProRes RAW family:

• ProRes RAW (1,680mb/s ÷ 8000 x 60sec) = 12.6GB file per 60 seconds
• ProRes RAW HQ (2,388mb/s ÷ 8000 x 60sec) = 17.91GB file per 60 seconds

ProRes 4444 family:

• ProRes 4444 (2656mb/s) = 19.9GB file per 60 seconds
• ProRes 4444 XQ (3976mb/s) = 29.8GB file per 60 seconds

ProRes 422 family:

• ProRes 422 LT (821mb/s) = 6.1GB file per 60 seconds
• ProRes 422 (1178mb/s) = 8.8GB file per 60 seconds
• ProRes 422 HQ (1768mb/s) = 13.3GB file per 60 seconds

In summary, this breaks down to approximately a file storage saving of 37% for ProRes RAW over a comparable ProRes 4444, 58% smaller file size for the ProRes RAW over ProRes 4444 XQ. "We see a 10% smaller file size compared to ProRes 4444 and a 40% smaller file size compared to ProRes 4:4:4:4 XQ which is massive considering that ProRes RAW HQ is the equivalent of an uncompressed RAW format."

Again, as reported in What Is ProRes RAW, the space saving for uncompressed video is even more dramatic: "Taking our 1 minute 4K UHD 60p clip, will result in a 80GB file when shooting in Uncompressed RAW 10bit 4:2:2, where as it will only be an 17.91GB file when shooting in ProRes RAW HQ, which is a 3.5x smaller file size and bit rate requirement...Taking a 1 minute 4K UHD 60p clip as an example, will result in a 134GB file when shooting in Uncompressed RAW 12bit 4:4:4. Yes you read that right, for every one minute of footage you require a whopping 134GB of storage space, whereas it will only be an 17.91GB file when shooting in ProRes RAW HQ."

A major competitor for Apple ProRes RAW is Blackmagic RAW which is "freely available and includes a developer SDK so anyone can add support for Blackmagic RAW to their own software." Blackmagic RAW also supports constant bit rate as well as variable bit rate.

While ProRes codecs of all flavors use the QuickTime wrapper, the higher technical capabilities of the ProRes4444 codecs lead themselves to application in the broadcast community including MXF users. To facilitate this, Apple authored SMPTE RDD 44:2017-11: SMPTE Registered Disclosure Document - Material Exchange Format — Mapping and Application of Apple ProRes (available for a fee from IEEE) which maps ProRes bitstreams into a subset of the MXF Generic Container (SMPTE ST 379-2). MXF support was added to Final Cut Pro 10.3 in October, 2016. This document has not been updated to include ProRes RAW codecs. Comments welcome.

The history of the development of RAW formats for filmmaking is an interesting one according to Ben Allan in Newsshooter:"RAW had been around for many years as an option for stills photographers when Oakley sunglasses billionaire Jim Jannard wanted to shift his hobby from photography to filmmaking. The story goes that he was shopping around for a moving picture camera and was disappointed that despite the price being virtually no issue, it was impossible to get the kind of resolutions, bit depth and RAW recording that was so readily available in DSLR’s. Of course, the big challenge here is that moving pictures require processing a constant stream of data at what were impossible frame rates for stills cameras of the day. Not only was it not available to buy at any price but none of the existing manufacturers even said they had a roadmap for when it might be available. So Jim Jannard being Jim Jannard decided to start a company [RED Digital Cinema] to make digital cinema cameras that recorded high bit depth, 4K RAW images." Jannard introduced in the first RED ONE camera at the NAB show in 2006 and a year later, in 2007, released the RED ONE camera with Mysterium sensor that could record 4K REDCODE RAW to a Compact Flash card at a cost of $17,500.

Apple filed an unsuccessful petition to the United States Patent and Trademark Office (USPTO) in 2019 to review and challenge RED’s patent regarding REDCODE. Although the motivation of the petition is not stated in the official USPTO documentation, Yossy Mendelovich in Y.M. Cinema Magazine suggests two: "to examine the possibility of using the REDCODE technology to further developing of Apple’s ProRes RAW. The other purpose is to prepare the right conditions for establishing a partnership between RED and Apple to combine RED’s codec into Apple’s Metal framework." The USPTO denied the request stating: “Petitioner (Apple) has not shown a reasonable likelihood that it would prevail in showing that any challenged claim is unpatentable. Thus, we deny the Petition and do not institute an inter-partes review”.