No CI job measures time. The nightly passes --no-perftest --no-direct-bench in all
three places it drives regtest.py, and ta_bench_direct appears in no workflow file.
A perf regression is caught only when someone runs regtest.py by hand.
Goal: a safety net that detects >10% regressions across the ~965 C entry points
(201 batch + 201 _Open + 201 _OpenAndFill + 201 _Update + 161 _Peek). Not a
measurement tool; the devbox stays the oracle for any figure we quote.
Why not wall clock
Every runner is GitHub-hosted. ta_bench_direct already defaults --max-spread=25 and
exits non-zero when the box is busy; its colour band is 1.20x because binary layout alone
moves the ratio past 10%; stream_ab.py --min-ratio sits at 0.35 because anything near
1.0 flaps. A 10% wall-clock threshold on a shared vCPU is alarm noise, not a gate.
Proposal
CodSpeed simulation mode: valgrind-based, each benchmark run once, <1% variance
regardless of system load, free for open source. It runs on our own runner, so
public-repo Actions minutes are the only cost (the 600 min/month budget is for their
bare-metal walltime runners, which we would not use).
Report-only on push: dev plus pull_request first, then enable the 10% threshold.
push: dev matters: 82% of what lands on dev is a direct push that on: pull_request
never sees.
Blocking pre-check
120 functions carry TA_FMA_MULTIVERSION, i.e. target_clones("default","fma")
(src/ta_func/ta_utility.h:27). Under valgrind the CPUID is emulated. If it does not
report FMA, all 120 fused indicators get measured on the default clone, which no
post-2013 x86 user executes. Silent, systematic, and concentrated exactly on the
functions where the FMA contract is load-bearing.
Verify before any harness work: a probe printing __builtin_cpu_supports("fma"), run
natively and under valgrind --tool=cachegrind. If they disagree, either pin the clone
for the benchmark build or record the blind spot explicitly. x86 only; FMA is baseline
on ARMv8.
Scope
- In: C. The Rust crate optionally, later.
- Out: Java and C#. CodSpeed supports Python, Rust, JS/TS, C++ and Go. Those tiers
stay with scripts/stream_ab.py on the devbox.
- Out: Apple Silicon. Valgrind has no ARM64/Darwin port, so
macos-latest cannot run
simulation. ARM64 Linux is supported, so ubuntu-24.04-arm is the ARM arm if we want
one. Instruction counts are not comparable across ISAs; that means a separate baseline
per architecture, and an ARM-only regression is invisible unless we run the ARM suite.
What it will not catch
Out-of-order effects, port pressure, dependency-chain latency. CodSpeed's own team
concedes valgrind "does aggravate the branching cost" on branchy code, so a branchless
rewrite like #410 may read as a regression rather than the win it is.
Magnitudes are not usable at all. On astral-sh/uv#21569 the same benchmark on the same
commit read -15.25% under simulation and -5.35% under wall time, a 3x discrepancy. A
CodSpeed percentage never goes in a release note.
What this does buy is the algorithmic class: a lost fast path, an extra pass over the
window, an un-inlined call. Those are arch-independent and do transfer.
Work
- A third emitter in
ta_codegen/generator/src/bench_gen.rs, beside write_c_bench and
write_c_stream_bench, emitting one google-benchmark registration per entry point.
The emitted .cc joins the regen-clean contract. This is the bulk of the effort.
- A
ta_bench_codspeed CMake target modelled on ta_bench_direct
(CMakeLists.txt:844): links ${TA_LIB_TOOL_TARGET}, sits inside
if(BUILD_BENCHMARKS). Benches are CMake-only, so no Makefile.am obligation.
- Workflow, ~25 lines, tokenless for a public repo via OIDC.
Gotchas, none hard, all silent if missed:
- CodSpeed's C++ guide says
CMAKE_BUILD_TYPE=RelWithDebInfo, which for gcc is
-O2 -g. We ship -O3 and force it on both existing bench targets. Set -O3 -g
explicitly or the suite measures code nobody ships.
- Link the static library, never a single-TU
-flto build. gcc re-decides inlining
corpus-wide there, which would move hundreds of unrelated benchmarks every commit.
- Their
FetchContent example pins GIT_TAG main. Pin a real tag, and keep it inside
BUILD_BENCHMARKS so a plain scripts/build.py stays offline and needs no network.
No CI job measures time. The nightly passes
--no-perftest --no-direct-benchin allthree places it drives
regtest.py, andta_bench_directappears in no workflow file.A perf regression is caught only when someone runs
regtest.pyby hand.Goal: a safety net that detects >10% regressions across the ~965 C entry points
(201 batch + 201
_Open+ 201_OpenAndFill+ 201_Update+ 161_Peek). Not ameasurement tool; the devbox stays the oracle for any figure we quote.
Why not wall clock
Every runner is GitHub-hosted.
ta_bench_directalready defaults--max-spread=25andexits non-zero when the box is busy; its colour band is 1.20x because binary layout alone
moves the ratio past 10%;
stream_ab.py --min-ratiosits at 0.35 because anything near1.0 flaps. A 10% wall-clock threshold on a shared vCPU is alarm noise, not a gate.
Proposal
CodSpeed simulation mode: valgrind-based, each benchmark run once, <1% variance
regardless of system load, free for open source. It runs on our own runner, so
public-repo Actions minutes are the only cost (the 600 min/month budget is for their
bare-metal walltime runners, which we would not use).
Report-only on
push: devpluspull_requestfirst, then enable the 10% threshold.push: devmatters: 82% of what lands on dev is a direct push thaton: pull_requestnever sees.
Blocking pre-check
120 functions carry
TA_FMA_MULTIVERSION, i.e.target_clones("default","fma")(
src/ta_func/ta_utility.h:27). Under valgrind the CPUID is emulated. If it does notreport FMA, all 120 fused indicators get measured on the
defaultclone, which nopost-2013 x86 user executes. Silent, systematic, and concentrated exactly on the
functions where the FMA contract is load-bearing.
Verify before any harness work: a probe printing
__builtin_cpu_supports("fma"), runnatively and under
valgrind --tool=cachegrind. If they disagree, either pin the clonefor the benchmark build or record the blind spot explicitly. x86 only; FMA is baseline
on ARMv8.
Scope
stay with
scripts/stream_ab.pyon the devbox.macos-latestcannot runsimulation. ARM64 Linux is supported, so
ubuntu-24.04-armis the ARM arm if we wantone. Instruction counts are not comparable across ISAs; that means a separate baseline
per architecture, and an ARM-only regression is invisible unless we run the ARM suite.
What it will not catch
Out-of-order effects, port pressure, dependency-chain latency. CodSpeed's own team
concedes valgrind "does aggravate the branching cost" on branchy code, so a branchless
rewrite like #410 may read as a regression rather than the win it is.
Magnitudes are not usable at all. On astral-sh/uv#21569 the same benchmark on the same
commit read -15.25% under simulation and -5.35% under wall time, a 3x discrepancy. A
CodSpeed percentage never goes in a release note.
What this does buy is the algorithmic class: a lost fast path, an extra pass over the
window, an un-inlined call. Those are arch-independent and do transfer.
Work
ta_codegen/generator/src/bench_gen.rs, besidewrite_c_benchandwrite_c_stream_bench, emitting one google-benchmark registration per entry point.The emitted
.ccjoins the regen-clean contract. This is the bulk of the effort.ta_bench_codspeedCMake target modelled onta_bench_direct(
CMakeLists.txt:844): links${TA_LIB_TOOL_TARGET}, sits insideif(BUILD_BENCHMARKS). Benches are CMake-only, so noMakefile.amobligation.Gotchas, none hard, all silent if missed:
CMAKE_BUILD_TYPE=RelWithDebInfo, which for gcc is-O2 -g. We ship-O3and force it on both existing bench targets. Set-O3 -gexplicitly or the suite measures code nobody ships.
-fltobuild. gcc re-decides inliningcorpus-wide there, which would move hundreds of unrelated benchmarks every commit.
FetchContentexample pinsGIT_TAG main. Pin a real tag, and keep it insideBUILD_BENCHMARKSso a plainscripts/build.pystays offline and needs no network.