Benchmark Protocol
A measurement is only useful if a stranger can repeat it. This protocol sets out how the lab records a run so that the numbers travel with enough context to be trusted, tested and, when needed, contradicted.
Sampling
Every clip in a bench set is cut from a larger master, and the cut is documented to the frame. We keep a short clip for iteration and a longer clip for the recorded run, and we never edit the source between them. Frame rate, duration and crop are written into the run note before the encoder is touched. Where a subject is rare, such as high-motion sport or low-light grain, we take two independent cuts so a single unlucky frame cannot drive the result.
The run note
Each recorded run carries a compact text sheet: source identity, frame count, colour profile, encoder identity, rate control mode, all quality and tool flags, threading class and the wall-clock start and end. The note is the anchor for everything downstream. If a later operator changes one flag, that is a new note and a new result.
| Field | Purpose |
|---|---|
| source id | Proves which master was used. |
| frame count | Fixes the sample length. |
| colour tag | Matrix, primaries, transfer. |
| rate mode | Constant quality or target bitrate. |
| flag set | Every option that changed output. |
| host class | Groups decode and encode timing. |
Error bars and repeats
A single run is a single draw. For every headline figure the lab performs three repeats and reports the median with the spread of the other two. When the spread is wide, the cause is named: frame-threaded decodes often vary by a few percent, and accelerated engines vary more when the driver batches work. Reporting the spread keeps a small honest difference from being sold as a large one.
Reproducibility checks
Before a result is published, a second operator repeats it from the note alone, without discussion. If the two medians differ beyond the stated spread, the result is held and the note is examined for a hidden variable such as a driver revision or a thermal limit. Only results that survive this pass reach the tables.
What the protocol forbids
No result may be quoted without its option list. No metric may be quoted without its crop and its definition. No ladder may be published without the source class it was tuned on. These three rules are the reason the desk can compare statements across a year without the tables rotting.
