NTT-CRT MFA: fuse transpose into row-FFT pass (~11% at 40M-100M digits)

[mmurshed]

What

The Matrix-Fourier (Bailey 6-step) NTT multiply ran each axis as a full out-of-place Transpose immediately followed by a full row-FFT sweep over the just-written buffer. At MFA sizes that scratch buffer is 64–256 MB per prime, so each transpose write + row-FFT read is two extra full round-trips through DRAM per axis. Large MFA on Apple silicon is memory-bandwidth bound (previously profiled), so those round-trips dominate.

This fuses them: gather a block of TILE output rows into an L2-resident tile, run the row FFT (and cross-twiddle) there, and write the result once. Collapses (transpose + row-FFT) into a single streaming pass — transform traffic drops from 8n → 4n bytes per forward/inverse. Per-row FFT math is byte-identical to the old step2/step5, so output is bit-exact.

Scope / safety

• Only the single-level case (n1,n2 <= LEAF) is fused — covers every in-practice MFA length (2^24..2^26 all factor to sub-FFTs ≤ 2^13). The recursive case still needs a live as sub-FFT scratch and falls back to the original transpose path.
• Gated by BIGMATH_NTT_MFA_FUSE (default on); tile size BIGMATH_NTT_MFA_FUSE_TILE (default 16 rows = 512 KB at the 2^13 leaf). Set -DBIGMATH_NTT_MFA_FUSE=0 to restore the old path.
• All four column↔row index maps were hand-derived against the original Transpose+step semantics.

Verification

• Bit-exact vs the unfused path (FNV checksum of product limbs) at 2M / 4M / 5.2M / 6M / 8M limbs.
• mult_correctness passes (18/0).
• Best-of-7 interleaved, fresh processes, M1 Max (base 2^64): 4M +11.24%, 8M +11.27%. Best-of-5 across 4M–10M: all +9–11%.

🤖 Generated with Claude Code