The world of video compression codecs is busy and competitive. Matt Murray breaks down what’s going on and what it means for manufacturers out there.

For this edition, I was invited to express my thoughts about video compression standards from the perspective of a hardware manufacturer, regarding their trajectory, the potential impact on the CEDIA channel and how video compression can affect communication platforms as a whole.

Compression type is a topical subject, as video is consumed on an array of devices ranging simply from mobile phones to Dolby Cinema auditoriums, using a variety of imaging technologies and delivery platforms. Fundamentally, the underlying mechanism that delivers video to consumers is video compression codecs.

What most consumers are unaware of is the fierce competition among allied companies for video compression codec dominance and the mitigating complexities enveloping compression standards licensing royalties that block some codecs from universal deployment across all media platforms. I will briefly touch on that aspect of video compression competition further below.

The tail wagging the dog

From hardware device manufacturers upward through the installation chain and culminating with the end user, I think it is agreed to by all, that getting a signal in front of the viewer’s eyes, ideally unchanged from the source or at least perceptually matching it as close as possible, is one of the primary objectives every stratum in our industry wakes up each morning striving to succeed at.

This was thought to have been all but accomplished when video matched audio’s precedent by breaking free from its analogue domain tethers, to become digitally based. And in like kind, following audio’s footsteps, video developed an equally unquenchable technological thirst for higher resolution and greater bit-depths. During what eventually turned out to be the transition period into the formats presently in use today, from the perspective of the content creation community, the prevailing thought was the roiling cataclysm was placed into motion prematurely, and unfairly, by marketing-motivated display manufacturers who fiercely promoted 4K to stimulate television and projector sales, after the failure of 3D home viewing.

Dolby, SMPTE and Hollywood (including studios worldwide) were discussing and already working toward making pixels better, through HDR, Dolby Vision and other recommended practices involving colour quality, but ultimately and less than subtly, were coerced into amassing more pixels to match the consumer marketplace. Unforeseen at the time by those in the eye of the typhoon, it may now be said, the outcome we enjoy today made the maelstrom worthwhile.

Here we go… again?

As was seen with the rollout of 4K, now playing out once more with 8K is the causality dilemma of ‘the chicken or the egg’, or perhaps described better here as ‘the content or the device’. This highlights the time-honoured consumer electronics equation that the pace of format adoption and acceptance is proportionate to content availability, no differently than at any other juncture throughout the industry’s ten-decade history.

Unsurprisingly, competition between video compression codecs should not be viewed differently than other ‘format wars’ that have been waged, such as Beta vs VHS and HD-DVD vs Blu-ray. In the past five years, new video codecs have been released only to be met with mixed reactions. In some circumstances, conflict has not arisen from the perspective of content availability, but rather, from a lack of available hardware that can effectively process the codec.

Why are compression codecs needed?

Looking at how and where video compression methods are used will show that a favourably viewed video compression codec remains dormant today, despite being introduced three years ago. The scenario created below centres on Dolby Vision cinematic content and will serve to illustrate where the various codecs consumers are aware of are employed, also isolating why one codec has yet to become popular inside the sphere of video compression codecs.

Shooting for a major cinematic title had been completed and has just wrapped up post-production. With the content creation process complete, how content will be distributed determines the video compression standard to be applied. The cinema targets and trim pass for the Dolby Vision “hero grade” of this theatrical release are uncompressed RGB 4:4:4, 108-nit, P3, D65, ST. 2084, comprising the professional cinema file. This file, as are all cinematic files, is unusable by any other playback medium outside of the post-production environment. In this scenario, our Dolby Vision file version, encoded for theatrical presentation, can only be decoded by the Integrated Media Block, built-in to the dual Christie 4K 6P RGB Laser cinema theatrical projectors, with data served from a Doremi Show Vault (the combination of the IMB and the Show Vault having gone through an elaborate security-linking process called marrying, in one of a myriad of steps necessarily performed to combat potential piracy).

However, for content creators, specifically colourists, this file is also used as the base grade for making subsequent passes to create deliverables in Rec. 709, HDR10 and Consumer Dolby Vision formats for the varied tiers of distribution, based on the content holder’s customer and the customer’s distribution providers.

An example might be, Warner Bros. Studios owns and holds the content but licenses it to Disney, which puts it on the American broadcast network ABC and streaming service HULU. At the end of the theatrical run, distribution typically includes network broadcasts, direct broadcast satellite tiers ranging from pay-per-view and then on to packaged services such as HBO, channels on regional cable networks (here in the United States, FX or TNT) and national or global online streaming services. Each facet of distribution requires different, specific video compression codecs.

Making sausage (well, sort of)

Continuing our scenario, the title is prepared in advance for post-theatric distribution (with the process that follows below, expressed in an extremely simplified manner).

The cinematic file is converted into a Dolby Vision mezzanine file. Before encoding, proprietary Dolby-specific pre-processing for Dolby Vision metadata is performed, along with video elementary stream mixing and Dolby Vision post-processing, all three electronically coupled to standard HEVC encoding, by devices widely used for non-Dolby Vision UHD encoding, such as ATEME’s Titan or Harmonic’s XOS. From this point, customer-specific renditions for different bitrates or resolutions are processed. Files are then sent to an HLS or DASH packager and depending on the target audience of the content holder’s customer, the appropriate video codec is applied.

Again, that is an extremely simplified fast-track through a sample Dolby Vision encoding. One significant takeaway that speaks to the lack of universality in deployment involves the video compression encoders mentioned above. Neither is yet to offer encoding for Versatile Video Codec H.266, announced nearly three years ago in July 2020, although the most codec agnostic between the two is DASH. DASH is an industry acronym for Dynamic Adaptive Streaming over HTTP, also referred to as MPEG-DASH. Like Apple’s HLS, video files are split into small segments. But unlike HLS, which is Apple-centric, DASH supports H.264, H.265 and even Google’s VP9 for online playback with most browsers.

Putting on my manufacturer’s hat

This is a good place to interject that while I will address compression types in a general sense, many are used almost universally across multiple platforms including in online browsers and especially with streaming applications. For clarification, AVPro Edge is not in the streaming business, we are in the business of distributing video after it has been streamed.

As a manufacturer of devices primarily for the transport and distribution of HDMI signals, AVPro Edge’s mission is to ensure that video signals leaving our devices are pixel-for-pixel identical to how they were received at the input stage. Our Murideo testing and certification instrumentation is designed to confirm the video is displayed as intended.

Codecs are decoded by source devices, such as DBS satellite receivers or Apple TV, so compression does not play a major role in product strategy for our HDMI-centric devices, as the HDMI signal is transported from the source and ultimately delivered to a video display. When HDMI signal extension devices are forced to compress an HDMI signal for transport, that compression type differs entirely from the forms of compression referred to above, that format a file into a particular video compression codec. HDMI compression, such as with HDBaseT, is designed to reduce bandwidth below a certain threshold.

AVPro Edge dabbles lightly in video codecs as a manufacturer. AV-over-IP is a product category where we faced the prospect of deciding on compression algorithms during product development. For the three distinct performance tiers comprising our AV-over-IP platform, a different compression type was selected for each.

In that respect, as a manufacturer, we must remain aware of compression technologies when in a long-term product development cycle. In almost every circumstance, we are beholden to chip manufacturers who ultimately decide which video compression codecs their designs support. A chip manufacturer may elect to support all codecs available for the target market application segment, or reactions to market analysis may steer it to omit certain types that have not yet found widespread acceptance. Some chip makers may design their architecture in such a way to allow “headroom” to accommodate an update for a pending codec that represents a major market disruption (Samsung’s 4K panel and 1080p-to-4K Evo Box upgrade nearly ten years ago is an example of timing disruption in the market, with a hardware change made in reaction to a codec change for 4K timing and colour space. But this is not an example of a simple codec-only update to an existing chip).

With the video codecs embedded onboard the chipsets that we use and selected based on overall performance for their intended use, we do not do anything separately outside of the architecture of those chipsets.

Perhaps worth a mention is that as a manufacturer planning new products, difficulties in longer-term strategies may arise when new compression types are introduced. When a new codec is announced, generally accompanied by much fanfare, the intention is to acquaint the public with the proposed specifications that are targeted to be met, but often, details are scarce or completely omitted regarding hardware capability or availability. While these ambitions are recognised as necessary and represent advocacy for technological advancement, for the manufacturing community, an entirely different scenario unfolds.

Such announcements tend to “freeze” the status of current products. As previously mentioned, manufacturers rely on third-party chipsets for product design. A new codec prompts questions from distributors, dealers and end-users, asking when the codec will appear in the product. And if not at first, can it be updated? (Typically, not, since chipsets are seldom open-source or lack sufficient headroom). While this scenario happens less frequently in the consumer electronics world as current formats have seemingly settled into place, new video compression algorithms proposed for 8K content may once again contribute to market disruption.

Compression types

For the Dolby Vision scenario described above, what follows are the various video compression codecs that may be used to distribute this title across multiple available platforms.

H.264/AVC (advanced video coding)

H.264 (also known as MPEG-4 Part 10) is the most widely used video streaming codec and is supported by virtually all devices. It is an encoding format that can be used for Blu-ray Discs, all major streaming services, terrestrial HDTV broadcasting, satellite, cable systems and with web software. It’s typically used in lossy compression form.

H.264 has restrictions with patents owned by various entities and is administered by MPEG LA (MPEG Licensing Administration). Royalties are required, though some free use of H.264 is possible when streaming internet video content that is free of charge to end users.

H.265/HEVC (high-efficiency video coding)

H.265 (also known as MPEG-H Part 2) was designed to be the successor to H.264. By comparison, after data compression with a bit rate reduction as great as 62% for a 2160p signal, video quality was considered equivalent. A downside to H.265 is resource consumption, using nearly three times more than H.264. It is capable of 8K resolution, and royalty intensive. An MPEG-4 H.265 derivative is the encode/decode choice for the United States subscription satellite service, DirecTV.

AV1 (AOMedia Video 1)

AV1 is not the newest codec to appear, but it has garnered the most attention. Unlike AVC/HEVC, AV1 is open-source and royalty-free. The consortium was founded in 2015, a little more than a month after H.265 was announced and the MPEG LA announced two patent pools, with a third forming, and some patent holders refusing to license either pool. These actions were on top of an increase in HEVC royalty fees over those for AVC.

AOMedia announced their explicit purpose was to create a royalty-free platform, but also with an eye on reducing wherever possible the strain to the internet’s worldwide data load with more efficient encoding and decoding and also contributing to reducing energy consumption around the globe.

AV1 is backed by a cross-section of industry heavyweights which includes Amazon, Cisco, Intel, Microsoft, Mozilla and Netflix. Prominent additional members include Apple, Google and Nvidia.  Initially targeting Web video and browser applications, AV1 is successfully in use by all top browsers except Safari and iOS, macOS. As the codec matured, Smart TVs including Android TV and Google TV effectively handle AV1 as do LG, Samsung, Sony and Amazon Fire TV.

AOMedia is rapidly growing, as it can deliver high-quality video with smaller file sizes and lower bandwidth consumption. HDR is not yet confirmed but with the powerhouse resources joining the AOMedia consortium, their targets are HEVC 265 and VVC H.266. In the foreseeable future, AV1 will remain part of a multi-codec scenario, but AMD, Intel and Samsung are devoting chip-level resources to AV1 support which should advance the codec’s ability substantially.

VVC/H.266/MPEG-I Part 3 (Versatile Video Coding)

VVC was finalised on 6 July 2020 and designed to replace H.265. The goals for VVC were to significantly reduce bitrate with no loss in fidelity in comparison to H.265 and to increase application versatility. The latest testing reveals bit rate savings near 50%, which is significant, yet H.266 has not received widespread deployment nearly three years into its existence.

While attempts were made to ward off the royalty fiasco MPEG LA experienced with H.265, the waters only seemed to get murkier. Despite what may be a vastly improved codec, it has been bogged down by perpetual litigation to the extent that hardware manufacturers have thus far seemingly steered clear, save for fewer than a handful, giving the content side no place to land and no incentive for adoption.

Additionally, with AV1 now a force making marketplace gains, companies like Amazon, Apple and Google see no merit or inclination to make VVC operable on their respective platforms. It’s Game of Thrones, the silicon version.

Final musings

For integrators, whether in the CEDIA channel or strictly doing commercial projects, how content of any type gets to and through devices is not prone to changing in any transparent way. There may be a ripple effect for clients that are web-centric, who may occasionally see “the file format is not supported”, certainly something not attributable to the performance of your firm.

The original suggestion for this piece focused on a hypothesis regarding a conjectural timeline for an H.267 codec, based on the anticipated technological turnover governed by the calendar and not the marketplace.

H.267 might be on a theoretical drawing board and in some stage of analysis, however, with VVC H.266 bending a bit in the wind from royalty matters and hardware attrition, plus AV1 kinetically charging forward, there is not a video compression codec poised to fully render its competition irrelevant. The status quo may waver slightly, but it is unlikely that it will appear any differently at this time next year.