Skip to content

Benchmark methodology

This is the contributor guide for collecting and interpreting measurements. For comparisons, start with benchmark results.

The benchmark site compares public JPEG operations with TurboJPEG on the same host. It measures JPEG encode/decode, not other codecs, application I/O, or image editing.

Reproduce the full comparison

make bench-setup
make bench
make docs-benchmark
make docs-build

The setup target downloads the official libjpeg-turbo 3.2.0 source archive, checks its SHA-256, and builds a release oracle under target/. It does not add a native runtime dependency to the Rust crate. CI uses the same target. Select a different existing installation with TURBOJPEG_PREFIX; its identity must be recorded and comparable before making a regression claim.

Use a new BENCH_OUTPUT for each repeat. The runner requires an odd number of rounds, at least three; the maintained default is five. CI records the complete 20-configuration encode/decode matrix.

For an x86 SIMD comparison, run both builds on the same x86-64 host and use a separate Cargo target directory and result directory for each build. For example, on an AVX2-capable Linux host:

RUSTFLAGS="-C target-cpu=x86-64" \
  make bench BENCH_TARGET_DIR=target/jpeg-sse2 \
  BENCH_OUTPUT=target/benchmarks/jpeg/sse2

RUSTFLAGS="-C target-cpu=x86-64 -C target-feature=+avx2" \
  make bench BENCH_TARGET_DIR=target/jpeg-avx2 \
  BENCH_OUTPUT=target/benchmarks/jpeg/avx2

The runner records explicit Rust build environment flags in metadata.json. The AVX2 build must only run on an AVX2-capable CPU. Keep workload, oracle, host, rounds, and correctness checks identical when comparing the two outputs; cross-compilation and assembly inspection alone do not establish a speedup. If CARGO_BUILD_TARGET is set, it must match the compiler host because the runner executes the built binary.

The x86-simd-lint CI job runs strict Clippy for the SSE2 baseline and AVX2 builds, then checks public JPEG and AVIF encode/decode parity in both runtime lanes. Those instrumented runs verify correctness and branch reachability; they are not performance measurements. AVIF has parity coverage there, but the paired timing comparison below measures JPEG only.

Local AVIF sequence decode probe

The repository also includes examples/avif_perf_probe.rs for timing the public AVIF sequence decoder on one fixture. Run it from a repository checkout:

cargo run --release --locked --no-default-features --features avif \
  --target-dir target/avif-perf-probe \
  --example avif_perf_probe -- \
  tests/fixtures/input/images/avif/animated_motion_temporal_mv_range_2048x128.avif 1

The example reads the input once, then prints one iteration nanoseconds row per timed decode. The optional third argument sets the number of unmeasured warm-up decodes (default 1). Each timed call includes decode_sequence and destruction of its decoded output; file I/O, warm-up, and compilation are outside the timer. This probe does not compare results with an oracle. Use make coverage-selected with exact parity cases for correctness. For performance comparisons, collect equal-length runs in alternating A–B–B–A order on one host with identical release settings, and report the workload, machine, Rust version, run medians, and sample counts. This probe is a local diagnostic; it is separate from the published JPEG benchmark matrix and its results do not establish SSE2 or AVX2 runtime speed.

Optional native x86 profiling

The Profile x86 JPEG workflow runs for pull requests that change the JPEG implementation, its build inputs, or the profiling workflow. It can also be run manually from Actions. It builds optimized SSE2 and AVX2 binaries with debug line information, then samples the public encode call for the existing 512×512 and 1024×1024 RGB Q85 4:2:0 workloads. The job installs the Linux perf tools matching its booted kernel and records the resolved package and tool versions. Its artifact retains the actual profiled executables, an immutable Git source archive, source revision, build logs, and a SHA-256 inventory alongside host and build metadata, raw perf data, actual sample counts, inline-aware reports, annotated sampled assembly, source-mapped converter disassembly with ISA counts scoped to the aligned converter, and output hashes. The profiling build pins Cargo release debug information, disables stripping and split debug files, and verifies the actual ELF debug sections and converter source-line mapping in src/codecs/jpeg/encode/mod.rs. Inlined kernels.rs line mentions are recorded separately when emitted by the compiler; they are not required for the converter's own source mapping. Sample counts come from bounded raw perf records rather than rounded report headers; lost samples or compressed records select the timing fallback. The job records and temporarily sets the per-process user-space sampling policy to 2 on its fresh hosted VM, then restores the original setting in an unconditional cleanup step. It prefers hardware cycles:u events and uses software cpu-clock:u samples when cycle counters are unavailable; the recorded event identifies which was collected. Both sampling modes require DWARF call graphs. If preflight or sampling cannot collect enough usable samples, the job records the reason and falls back to public-API timing and linked assembly. That fallback does not establish sampled hotspots.

Treat these profiles as hotspot attribution only. They include conversion data movement as well as arithmetic, and sampling overhead makes them unsuitable for comparing latency. Use the paired five-round matrix above for timing comparisons, and do not add a kernel candidate unless the profile shows conversion is a material cost.

Timing boundary

Each timed encode starts from the same deterministic RGB, grayscale, or CMYK bytes and includes operation-owned output destruction. Each decode starts from the same complete JPEG byte slice and includes decoded-output destruction.

Rust calls its public codec API. C creates and configures a TurboJPEG 3 handle, calls the public operation, frees output, and destroys the handle. Input generation, sample storage, build work, and one-time fixture I/O are outside both timed operations.

Both use accurate integer DCT behavior. Normal TurboJPEG SIMD stays enabled; the comparison does not disable it to improve the Rust ratio. Rust uses ordinary locked release builds, and the thin C harness uses -O3. There is no hidden LTO, PGO, or native-CPU tuning in the default comparison.

Samples and correctness

Five rounds alternate implementation order. The summary is the median of the five round medians. Raw records retain each round's median, P95, minimum, result length/hash, order, and host load. A percentile of these medians would not equal the pooled per-operation percentile; the public table leaves unavailable spread unmeasured instead of inventing it.

The configurations vary dimensions, quality, chroma sampling, progressive and optimized encoding, restart markers, grayscale, CMYK, and odd edges. FNV output-hash and byte-length comparisons are diagnostic equality checks, not substitutes for the full Pillow parity suite.

CMYK has a specific convention difference: Pillow/image-slash-star expose conventional CMYK, while direct TurboJPEG decode exposes JPEG's stored Adobe sample convention. Unequal CMYK hashes must remain visible with that explanation. Do not label every timing row pixel-parity-proven.

Interpret results

Lower latency means less time only for the same operation and configuration. The original rust_over_turbo ratio below 1 means Rust took less time; above 1 means TurboJPEG took less time. Do not reverse the ratio or average unrelated operations into a project-wide claim.

Control host load, power, compiler/build flags, linked library, inputs, and repeat policy. Hosted-runner observations are not a calibrated performance lab. Keep all rows and raw samples, repeat noisy experiments, and use separate profiling runs to investigate causes.

Public data and CI

The Benchmark workflow retains metadata.json, summary.csv, and raw.jsonl. Its validated ARM64 public snapshot feeds this repository's Pages workflow. A separate x86-64 job measures SSE2 and AVX2 sequentially on one AVX2-capable host, checks that both builds produce the same outputs for all 40 workloads, and retains both raw result sets as a CI artifact. Those hosted-runner measurements are investigation evidence; they do not publish a speed claim or gate changes on noisy timing differences. The original report hashes, source revision, environment, measurement policy, and every result remain identifiable. For pull requests, that job also compares the base and candidate revisions with six interleaved A-B-B-A cohorts per workload under SSE2 and AVX2. The RGB 4:2:0 workloads cover 128×128 Q10, 512×512 Q10, 128×128 Q85, 512×512 Q85, 1024×1024 Q85, 128×128 Q100, and the odd 63×65 Q85 image. Complete encoded outputs must match before timing. Each sample records its executable SHA-256; the summary retains all cohort ratios and reports their paired median at revision-abba/summary.json. These remain focused hosted-runner observations.

Both revisions build sequentially from retained Git source archives at the same absolute source path, in a temporary directory under the output directory. Fresh target directories prevent stale reuse when archive timestamps predate an earlier build. The runner creates no worktree and leaves Git state unchanged. The archive digests, source-file hashes, immutable revisions, tree identities, actual build commands, compiler versions, and logs accompany the measurements. Identical archived source must produce identical executable hashes before timing can start.

The image-slash-star/jpeg-revision-abba@3 report retains both actual timed executables under revision-abba/executables/, plus the producer scripts and workflow. Optional --sse2-target-dir and --avx2-target-dir directories supply prebuilt binaries for untimed exact-output checks; these are retained and compared with the freshly archived candidate. Revision timings always use the two archived builds. Local output must stay inside the repository. The runner accepts --cases for exact matrix workloads and --cohorts for one to twenty repeated quartets; its default remains one quartet, while CI explicitly selects six. Existing version-2 reports retain their original build policy and meaning.

Explicit patch experiments keep unaccepted implementation changes out of the shipping Rust sources. scripts/benchmark_jpeg_patch_experiment.py consumes a committed plan and patch, builds the same immutable revision twice, and applies the patch only inside the candidate's private source archive. The runner retains the original archive inventory separately from the patched compiled file hashes and complete changed source files. Version-4 reports identify the experimental patch explicitly; patched bytes are never represented as committed Rust blobs. Complete JPEG outputs must match before timing starts.

The native x86 workflow measures a patch experiment when its committed plan or patch changes in a pull request or a push to main. The reservoir-capacity gate requires at least 3% primary paired gain in both SSE2 and AVX2, all six primary cohorts faster, and no guard median slowdown above 2%. experiment-result.json records the result separately from CI success. A rejected candidate is a completed diagnostic experiment. A successful timing gate still requires exact pinned Pillow parity and an architecture policy before adopting source changes. The earlier failed ARM64 reservoir screen remains recorded in its plan; it does not establish x86 latency or justify changing ARM64 shipping code.

The same workflow measures the JPEG entropy-reader bulk-refill candidate from decoder-refill.json and its pinned patch. Its primary workload is 512×512 grayscale 4:2:0, with grayscale 4:2:2, small baseline RGB, and a progressive long-scan image as guards. The gate requires at least 3% paired gain in both SSE2 and AVX2, all six primary cohorts faster, and no guard slowdown above 2%; complete decoded pixel buffers must match before timing. The plan records a local ARM64 screen and selected parity results as supporting evidence only. Those results do not establish native x86 speed or full-suite parity. If the committed decoder source changes, the workflow also checks its SHA-256 against the measured candidate recorded in the plan.

The decoder-refill candidate was rejected by the native x86 run on 2026-11-03 on an AMD EPYC 9V45 host. Across six paired cohorts, the SSE2 primary gain was -0.16% (two cohorts faster), and its progressive guard was 2.22% slower. The AVX2 primary gain was 0.80% (three cohorts faster). Both missed the 3% gain and six-cohorts-faster requirements; the SSE2 guard also exceeded its 2% slowdown limit. All eight complete decoded pixel-buffer pairs matched. The candidate source was not adopted despite its positive local ARM64 screen. The retained artifact ZIP has SHA-256 130cec9518190d2c5163f272ed4c73c21125aa3d80bae052397ba9213422b2a8.

The native reservoir-capacity experiment was also rejected in run 41533813985. Across six paired cohorts, SSE2 primary gain was -1.0652% (zero cohorts faster) and AVX2 gain was 1.0000% (six cohorts faster). Worst guard slowdowns were 2.1212% and 4.0758%, respectively. Neither architecture met the declared gates. All fourteen complete JPEG pairs matched. The artifact audit reconciled 336 samples, 84 ABBA cohorts, four executables and 676 archived-source comparisons. Execution revision 5c9f3273f470439325e971020f9e4c49c14cfd8a is the GitHub test merge with the same complete tree as PR #165's head 69ec5f624; original receipts retain the execution revision. The original artifact ZIP has SHA-256 2cc0906baffa319183ffc6cb98a975d7022c019104540fe0c425625b052881e4. No shipping JPEG optimization was adopted. The original artifact and audit remain in .coverage-mcp/local-evidence/performance/jpeg-native-patch-69ec5f624-20261103/.

For local verification of the experiment runner, the revision tool accepts --candidate-patch and explicit --variants native. Native diagnostics on ARM64 do not establish SSE2/AVX2 timing. The default variants remain SSE2 and AVX2 and require an x86-64 host. Untracked user files are preserved and excluded from both Git archives; tracked producer changes must be committed before execution. Neither mode creates a worktree or changes the shipping sources.

This policy follows a source-identical native x86 calibration that observed candidate/control differences from -0.07% to +2.39%. A separate ARM64 probe built identical archived bytes three times: two builds at one source path had identical executable hashes, while a build at a different source path did not. All seven production JPEG workloads matched live pinned Pillow bytes on all three executables. This exposes a build-identity confound; it does not establish an x86 speedup or attribute every timing difference to source paths. The original calibration and build-path probe remain under .coverage-mcp/local-evidence/ci/pr159-final-x86-benchmark-20261102/ and .coverage-mcp/local-evidence/performance/jpeg-build-identity-20261102/.

A 2026-11-02 AVX2 eight-block reciprocal experiment was rejected by this procedure. On the AMD EPYC 7763 host, RGB 512×512 Q85 4:2:0 gained 0.19% by median paired ratio across six cohorts, below its predeclared 3% minimum. The SSE2 RGB 128×128 Q10 guard was 2.43% slower, above its 2% limit. Complete JPEG bytes matched for all fourteen revision/workload comparisons, and all forty SSE2/AVX2 public output hashes and byte lengths matched. The original arithmetic was restored. These observations identify executable revision 485e879fea0208bfbfd323288cf2d5b48a6f4691 (PR merge tree for candidate 19d6e0df75dcb935663e0134106184f41f6e56df) in native benchmark run 41472576351. The raw artifact and audited cohort ratios are retained locally in .coverage-mcp/local-evidence/ci/pr159-x86-benchmark-20261102/. Local paths and hostnames are omitted from the public presentation.

Fixture-harness timing from benchmark_fixture_workloads.py measures harness execution cost and artifact sizes. It is not a codec throughput comparison and is not mixed into these tables.

References: libjpeg-turbo build guidance, Rust Performance Book, and Criterion analysis.

Reading the comparison table

The public page places one workload on each row and implementations in columns. Column headings sort the underlying time values, independent of the displayed ns/µs/ms units. Workload and implementation filters retain the baseline and update the visible comparison counts. Details expose sample counts, percentiles and recorded correctness; the full measurements and JSON remain downloadable.

A speed factor is baseline median divided by project median. For example, 20 µs versus 10 µs is 2× faster; 10 µs versus 20 µs is 2× slower. Each row's bars share a linear scale; different workloads do not share a scale. Counts and the lowest-median highlight describe observations, not a statistical significance test or an overall project score. Small differences may be noise. Timing-only rows remain labeled; failed execution, differing output, missing baseline, incompatible measurement policies and unconfirmed GPU completion do not receive a comparative speed label. A fallback names the backend actually used. The UI never changes samples, acceptance thresholds or correctness results.

The presentation takes cues from Artificial Analysis and MLPerf Endpoints: make the comparison clear while keeping task, environment and quality context visible. These projects are references for presentation, not validators of these results.