Skip to content

test(CI): Benchmark regression detection #412

Description

@mario4tier

No CI job measures time. The nightly passes --no-perftest --no-direct-bench in all
three places it drives regtest.py, and ta_bench_direct appears in no workflow file.
A perf regression is caught only when someone runs regtest.py by hand.

Goal: a safety net that detects >10% regressions across the ~965 C entry points
(201 batch + 201 _Open + 201 _OpenAndFill + 201 _Update + 161 _Peek). Not a
measurement tool; the devbox stays the oracle for any figure we quote.

Why not wall clock

Every runner is GitHub-hosted. ta_bench_direct already defaults --max-spread=25 and
exits non-zero when the box is busy; its colour band is 1.20x because binary layout alone
moves the ratio past 10%; stream_ab.py --min-ratio sits at 0.35 because anything near
1.0 flaps. A 10% wall-clock threshold on a shared vCPU is alarm noise, not a gate.

Proposal

CodSpeed simulation mode: valgrind-based, each benchmark run once, <1% variance
regardless of system load, free for open source. It runs on our own runner, so
public-repo Actions minutes are the only cost (the 600 min/month budget is for their
bare-metal walltime runners, which we would not use).

Report-only on push: dev plus pull_request first, then enable the 10% threshold.
push: dev matters: 82% of what lands on dev is a direct push that on: pull_request
never sees.

Blocking pre-check

120 functions carry TA_FMA_MULTIVERSION, i.e. target_clones("default","fma")
(src/ta_func/ta_utility.h:27). Under valgrind the CPUID is emulated. If it does not
report FMA, all 120 fused indicators get measured on the default clone, which no
post-2013 x86 user executes. Silent, systematic, and concentrated exactly on the
functions where the FMA contract is load-bearing.

Verify before any harness work: a probe printing __builtin_cpu_supports("fma"), run
natively and under valgrind --tool=cachegrind. If they disagree, either pin the clone
for the benchmark build or record the blind spot explicitly. x86 only; FMA is baseline
on ARMv8.

Scope

  • In: C. The Rust crate optionally, later.
  • Out: Java and C#. CodSpeed supports Python, Rust, JS/TS, C++ and Go. Those tiers
    stay with scripts/stream_ab.py on the devbox.
  • Out: Apple Silicon. Valgrind has no ARM64/Darwin port, so macos-latest cannot run
    simulation. ARM64 Linux is supported, so ubuntu-24.04-arm is the ARM arm if we want
    one. Instruction counts are not comparable across ISAs; that means a separate baseline
    per architecture, and an ARM-only regression is invisible unless we run the ARM suite.

What it will not catch

Out-of-order effects, port pressure, dependency-chain latency. CodSpeed's own team
concedes valgrind "does aggravate the branching cost" on branchy code, so a branchless
rewrite like #410 may read as a regression rather than the win it is.

Magnitudes are not usable at all. On astral-sh/uv#21569 the same benchmark on the same
commit read -15.25% under simulation and -5.35% under wall time, a 3x discrepancy. A
CodSpeed percentage never goes in a release note.

What this does buy is the algorithmic class: a lost fast path, an extra pass over the
window, an un-inlined call. Those are arch-independent and do transfer.

Work

  1. A third emitter in ta_codegen/generator/src/bench_gen.rs, beside write_c_bench and
    write_c_stream_bench, emitting one google-benchmark registration per entry point.
    The emitted .cc joins the regen-clean contract. This is the bulk of the effort.
  2. A ta_bench_codspeed CMake target modelled on ta_bench_direct
    (CMakeLists.txt:844): links ${TA_LIB_TOOL_TARGET}, sits inside
    if(BUILD_BENCHMARKS). Benches are CMake-only, so no Makefile.am obligation.
  3. Workflow, ~25 lines, tokenless for a public repo via OIDC.

Gotchas, none hard, all silent if missed:

  • CodSpeed's C++ guide says CMAKE_BUILD_TYPE=RelWithDebInfo, which for gcc is
    -O2 -g. We ship -O3 and force it on both existing bench targets. Set -O3 -g
    explicitly or the suite measures code nobody ships.
  • Link the static library, never a single-TU -flto build. gcc re-decides inlining
    corpus-wide there, which would move hundreds of unrelated benchmarks every commit.
  • Their FetchContent example pins GIT_TAG main. Pin a real tag, and keep it inside
    BUILD_BENCHMARKS so a plain scripts/build.py stays offline and needs no network.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions