H.264 Under the Hood

H.264 has been decoded by more devices than any other video codec, so its quirks are worth understanding even where a newer standard takes the top rung. This sheet walks the encode path from a fixed grid of macroblocks to a packed bitstream.

The fixed grid

H.264 divides a frame into macroblocks of sixteen by sixteen samples. Each macroblock predicts from a neighbour inside the same frame or from a previously reconstructed frame, then encodes only the residual. The fixed size is the codec's main limit and its main gift: it is simple to parallelise and easy to decode in silicon, but it cannot adapt its partition to the content as flexibly as later designs.

Prediction and the residual

Intra prediction copies a pattern from already coded neighbours, which handles flat skies and walls well. Inter prediction searches a window for a matching block and codes the motion along with a small residual. A good match shrinks the residual, and a small residual survives heavy quantisation without visible artefacts. The encoder therefore spends much of its time on the match search, and that search is where quality and time trade against each other.

Transform, quantisation, entropy

The residual goes through an integer transform that concentrates energy into few coefficients. Quantisation then discards low-value precision, guided by the rate factor. The last stage codes the symbols without loss, using a context-adaptive scheme that adapts to local statistics. Because quantisation sits in the middle, a change in the rate factor moves both size and distortion at once.

Where it still pays

At low resolution and low rungs, H.264 delivers a distortion that newer codecs beat only slightly, while demanding far less encode time. That balance keeps it at the base of most ladders. Its decoder base is also unmatched, so a stream that must run everywhere usually ends up here.

Setting it up

Anchors for the 1080p tier are on the 1080p settings sheet. The rate factor itself is explained on the constant quality page.