perf(PULPOpen): let Gemm take a one-dimensional bias instead of widening it - #50
Closed
runwangdl wants to merge 3 commits into
Closed
perf(PULPOpen): let Gemm take a one-dimensional bias instead of widening it#50runwangdl wants to merge 3 commits into
runwangdl wants to merge 3 commits into
Conversation
…ing it GEMMLayer.computeShapes widens the C operand to [M, N] unconditionally, because PULP_Gemm_fp32_fp32_fp32_fp32 walked the bias as a full matrix (pDstC[i * O + j]). A Linear's bias is O values shared by every output row, so the widened operand stores the same vector M times: on CCT-2 at 64 tokens a 128-wide bias occupies 32 KB to carry 512 B. Give the kernel a bias row stride. O reproduces the previous walk exactly, 0 makes every row re-read the same vector. PULPFloatGEMMTemplate picks the stride from the C operand's own rank, PULPGEMMLayer skips the widening when the bias arrives one-dimensional, and the tile constraint then ties only the trailing dimension to the output instead of requiring an M extent the bias does not have. Registered on both PULPOpen and GAP9. GAP9 keeps its own operator mapping in Deeploy/Targets/GAP9/Platform.py, so registering a layer in the PULPOpen mapping alone has no effect on a GAP9 deployment -- worth knowing before measuring, since an unregistered change looks exactly like an ineffective one. Targets/Generic/Layers.py is untouched, so other targets keep the [M, N] bias their kernels expect. Verified on GAP9 gvsoc (CCT linear probing, --l1 122000 --defaultMemLevel L3): Errors: 0, 37.69M train cycles. Scope of the saving: this reaches graphs whose Linear layers are exported as Gemm. The single-block CCT variants measured alongside this are exported as MatMul + Add and contain no Gemm nodes at all, so their 252 KB of widened bias is untouched here; that path is addressed separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
runwangdl
force-pushed
the
feat/gemm-vector-bias
branch
from
July 28, 2026 01:22
a0d7198 to
1b01f9c
Compare
The one-dimensional bias reached the kernel correctly but was tiled as if it were
still [M, O]. serializeTilingSolution built the C cube straight from the output
cube:
CCube = HyperRectangle(tuple(cube.offset), tuple(cube.dims))
so the DMA was told to move M * O elements out of a buffer holding O. On CCT at 64
tokens that is 32768 B read from a 512 B buffer, 64x past the end, and the tile the
kernel then read was garbage.
It only showed up on the L3 and promote configurations because those are the ones
that actually transfer the operand; with everything resident in L2 there is no copy
and the oversized cube costs nothing. That is why siracusa-training-tiled-l2 passed
while the four L3 and promote jobs each failed with 4 errors out of 4, and why the
GAP9 jobs, whose smaller L1 budget yields a different tiling solution, did not catch
it either.
A broadcast bias now gets a cube of its own rank: the trailing O extent tracks the
output tile, the leading dimensions stay 1. Measured on the generated code for
Models/Training/CCT/cct_train at --l1 128000 --defaultMemLevel L3, node_84:
before C_transfer_size = {28672, 4096} 32768 B
after C_transfer_size = {448, 64} 512 B
512 B is the whole bias, O = 128 floats, split across the two O tiles of 112 and 16.
A widened [M, O] bias is untouched and still tiles like the output.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # Deeploy/Targets/PULPOpen/Platform.py
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GEMMLayer.computeShapeswidens the C operand to[M, N]unconditionally, becausePULP_Gemm_fp32_fp32_fp32_fp32walked the bias as a full matrix (pDstC[i * O + j]). A Linear's bias isOvalues shared by every output row, so the widened operand stores the same vectorMtimes — on CCT-2 at 64 tokens, 32 KB to carry 512 B.What changes
The kernel takes a bias row stride.
Oreproduces the previous walk exactly;0makes every row re-read the same vector, so one kernel serves both layouts and the existing behaviour is bit-identical when the stride isO.TargetLibraries/PULPOpen/src/Gemm.cbiasStrideparameter, 4 indexing sitesTargetLibraries/{PULPOpen,GAP9}/inc/...Templates/FloatGemmTemplate.pyPULPOpen/Layers.pyPULPGEMMLayerskips the widening for a 1-D bias{PULPOpen,GAP9}/Platform.pyTileConstraints/GEMMTileConstraint.pyTargets/Generic/Layers.pyis untouched, so other targets keep the[M, N]bias their kernels expect.One thing worth knowing
GAP9 keeps its own operator mapping in
Deeploy/Targets/GAP9/Platform.py. Registering a layer only in the PULPOpen mapping has no effect on a GAP9 deployment, and an unregistered change is indistinguishable from an ineffective one when you measure it. Both mappings are updated here.Tiling
The kernel side is only half of it.
serializeTilingSolutionbuilt the C cube fromthe output cube, so a
[O]bias was still transferred as if it were[M, O]-- onCCT at 64 tokens, 32768 B moved out of a 512 B buffer. A broadcast bias now gets a
cube of its own rank, with the trailing extent tracking the output tile and the
leading dimensions held at 1. Measured on the generated code for
cct_trainat--l1 128000 --defaultMemLevel L3, node_84:C_transfer_size{28672, 4096}{448, 64}512 B is the entire bias,
O = 128floats across the two O tiles of 112 and 16.This is invisible whenever the operand is already resident, which is why
siracusa-training-tiled-l2passed while the four L3 and promote jobs each failedwith 4 errors out of 4, and why the GAP9 jobs, whose smaller L1 budget produces a
different tiling solution, did not catch it either.
Verification
GAP9 gvsoc, CCT linear probing,
--l1 122000 --defaultMemLevel L3: Errors: 0, 37.69M train cycles.Scope
This reaches graphs whose Linear layers are exported as
Gemm. The single-block CCT variants measured alongside it are exported asMatMul + Addand contain noGemmnodes, so their 252 KB of widened bias is untouched here — that path is addressed separately.🤖 Generated with Claude Code