Repository navigation
Help wanted: add before-vs-after proof entries for kernel skills #1
Description
What this is
The proof/ directory holds empirical evidence that skill files produce measurably better kernel code — same model, same prompt, with and without the skill file injected into context.
There is currently one entry:
proof/cuda/softmax/— validateswrite-cuda-softmax-kernel(RTX 4070, Claude Sonnet 4.6, 2 bug classes caught)
Every other skill in this repo has no proof entry yet.
What we are looking for
Run a before-vs-after benchmark for any skill in skills/. The bar is intentionally low — a screenshot, a correctness table, or a chart is enough.
Skills that most need proof entries
| Skill | Category | Priority | Status |
|---|---|---|---|
write-cuda-reduction-kernel |
cuda | high | ✅ |
write-cuda-gemm-kernel |
cuda | high | ✅ |
write-cuda-layernorm-kernel |
cuda | high | ✅ |
write-triton-softmax-kernel |
triton | high | ✅ |
write-triton-attention-kernel |
triton | high | ✅ |
write-int8-quantized-kernel |
quantization | high | ✅ |
write-fp8-kernel |
quantization | high | ✅ |
avoid-warp-divergence |
cuda | medium | |
write-numerically-stable-kernel |
patterns | medium | |
handle-boundary-conditions |
patterns | medium | |
port-cuda-kernel-to-triton |
portability | medium |
How to contribute a proof
- Pick a skill from the table above (comment below to claim it so no one duplicates effort).
- Generate a kernel without the skill file using any capable coding model.
- Generate the same kernel with the skill file injected into context. Same model, same base prompt.
- Run both. Compare correctness and/or performance.
- Create
proof/<category>/<kernel-name>/and drop in your artifacts. - Open a PR.
Full instructions: proof/README.md
Minimum bar
- Same model, same base prompt — only the skill file differs between the two runs.
- At least one correctness check (not just speed numbers).
- Hardware model + shapes tested noted somewhere.
A chart is nice but not required. Raw numbers in a table are fine. A screenshot works.
What a strong entry looks like
See the existing softmax proof as a reference:
- proof/cuda/softmax/softmax-correctness.md
- Includes: pass/fail matrix, error cliff chart, adversarial failure count, root-cause explanation, animated diff
You do not need to match that level of polish for a first entry. Correctness and reproducibility matter more than visual quality.
Activity
titoatwork commentedon Aug 21, 2026
Claiming
port-cuda-kernel-to-triton.Plan is to port a CUDA warp-shuffle reduction.
__shfl_down_synchas no direct Triton analogue, and a naive port tends to either try to emulate the shuffle or quietly lose the reduction axis, so the failure shows up as wrong numbers rather than only as a slower kernel. That satisfies the "at least one correctness check, not just speed" bar inproof/README.md.Method:
Landing under
proof/portability/reduction/, following the existingcode-diff.mdplus<topic>-proof.mdlayout.Hardware will be a Colab T4 unless that turns out to be a bad fit for the comparison, in which case I will say so rather than quietly switch.
If you would rather I take a different kernel from the table, or you have a preferred base prompt so results stay comparable with the existing entries, happy to follow that instead.
teddytennant commentedon Aug 30, 2026
I'd like to take
write-numerically-stable-kernelfor a proof entry.No GPU on my machine, so this would be correctness-only with no timing numbers. The design: 5 kernel generations from a base prompt and 5 from the byte-identical prompt with
SKILL.mdprepended, on a streaming per-row mean and variance task where the single-pass constraint forces the algorithm choice, checked against a float64 reference over 75 cases each.The split is clean. 5/5 without the skill write
sum(x*x)/L - mean*mean, 5/5 with it write Welford merged with Chan's parallel rule. At mean 10000 the naive form returns negative variance, so1/sqrt(var+eps)is NaN. I cross-checked in plain NumPy fp32 to confirm it's IEEE-754 and not aTRITON_INTERPRETartifact.The skill's Performance requirements section asks for the cost of stabilization to be quantified, and that's the half I can't do. Is a correctness-only entry worth having, with the performance half left for someone with a CUDA device, or would you rather wait for both together?
docs: record the deviations between the filed plan and this entry
titoatwork commentedon Sep 1, 2026
Follow-up on my claim above, since the promise was made here.
I said hardware would be a Colab T4 and that I would say so rather than quietly switch. It ended up on a local RTX 3050 and I did not flag it. That is now recorded in the proof doc itself, along with three smaller deviations from the plan I filed. Details in #12.
Outcome, briefly: the skill file made Opus 5 fail all 11 shapes it passes unaided, through a
.to(tl.int64)call on a stride argument that Triton specializes to a Pythonintfor any contiguous tensor. Sonnet 5 passed 11/11 both ways, so it is model-specific rather than a blanket problem with the skill. #12 carries the fix toSKILL.mdand the full repro.