Skip to content

perf(PULPOpen): let Gemm take a one-dimensional bias instead of widening it - #50

Closed
runwangdl wants to merge 3 commits into
develfrom
feat/gemm-vector-bias
Closed

perf(PULPOpen): let Gemm take a one-dimensional bias instead of widening it#50
runwangdl wants to merge 3 commits into
develfrom
feat/gemm-vector-bias

Conversation

@runwangdl

@runwangdl runwangdl commented Jul 28, 2026

Copy link
Copy Markdown
Owner

GEMMLayer.computeShapes widens the C operand to [M, N] unconditionally, because PULP_Gemm_fp32_fp32_fp32_fp32 walked the bias as a full matrix (pDstC[i * O + j]). A Linear's bias is O values shared by every output row, so the widened operand stores the same vector M times — on CCT-2 at 64 tokens, 32 KB to carry 512 B.

What changes

The kernel takes a bias row stride. O reproduces the previous walk exactly; 0 makes every row re-read the same vector, so one kernel serves both layouts and the existing behaviour is bit-identical when the stride is O.

file change
TargetLibraries/PULPOpen/src/Gemm.c biasStride parameter, 4 indexing sites
TargetLibraries/{PULPOpen,GAP9}/inc/... declarations
Templates/FloatGemmTemplate.py stride chosen from the C operand's rank
PULPOpen/Layers.py PULPGEMMLayer skips the widening for a 1-D bias
{PULPOpen,GAP9}/Platform.py register it
TileConstraints/GEMMTileConstraint.py tie only the trailing dimension

Targets/Generic/Layers.py is untouched, so other targets keep the [M, N] bias their kernels expect.

One thing worth knowing

GAP9 keeps its own operator mapping in Deeploy/Targets/GAP9/Platform.py. Registering a layer only in the PULPOpen mapping has no effect on a GAP9 deployment, and an unregistered change is indistinguishable from an ineffective one when you measure it. Both mappings are updated here.

Tiling

The kernel side is only half of it. serializeTilingSolution built the C cube from
the output cube, so a [O] bias was still transferred as if it were [M, O] -- on
CCT at 64 tokens, 32768 B moved out of a 512 B buffer. A broadcast bias now gets a
cube of its own rank, with the trailing extent tracking the output tile and the
leading dimensions held at 1. Measured on the generated code for cct_train at
--l1 128000 --defaultMemLevel L3, node_84:

C_transfer_size total
before {28672, 4096} 32768 B
after {448, 64} 512 B

512 B is the entire bias, O = 128 floats across the two O tiles of 112 and 16.

This is invisible whenever the operand is already resident, which is why
siracusa-training-tiled-l2 passed while the four L3 and promote jobs each failed
with 4 errors out of 4, and why the GAP9 jobs, whose smaller L1 budget produces a
different tiling solution, did not catch it either.

Verification

GAP9 gvsoc, CCT linear probing, --l1 122000 --defaultMemLevel L3: Errors: 0, 37.69M train cycles.

Scope

This reaches graphs whose Linear layers are exported as Gemm. The single-block CCT variants measured alongside it are exported as MatMul + Add and contain no Gemm nodes, so their 252 KB of widened bias is untouched here — that path is addressed separately.

🤖 Generated with Claude Code

…ing it

GEMMLayer.computeShapes widens the C operand to [M, N] unconditionally, because
PULP_Gemm_fp32_fp32_fp32_fp32 walked the bias as a full matrix (pDstC[i * O + j]).
A Linear's bias is O values shared by every output row, so the widened operand
stores the same vector M times: on CCT-2 at 64 tokens a 128-wide bias occupies
32 KB to carry 512 B.

Give the kernel a bias row stride. O reproduces the previous walk exactly, 0 makes
every row re-read the same vector. PULPFloatGEMMTemplate picks the stride from the
C operand's own rank, PULPGEMMLayer skips the widening when the bias arrives
one-dimensional, and the tile constraint then ties only the trailing dimension to
the output instead of requiring an M extent the bias does not have.

Registered on both PULPOpen and GAP9. GAP9 keeps its own operator mapping in
Deeploy/Targets/GAP9/Platform.py, so registering a layer in the PULPOpen mapping
alone has no effect on a GAP9 deployment -- worth knowing before measuring, since
an unregistered change looks exactly like an ineffective one.

Targets/Generic/Layers.py is untouched, so other targets keep the [M, N] bias
their kernels expect.

Verified on GAP9 gvsoc (CCT linear probing, --l1 122000 --defaultMemLevel L3):
Errors: 0, 37.69M train cycles.

Scope of the saving: this reaches graphs whose Linear layers are exported as Gemm.
The single-block CCT variants measured alongside this are exported as MatMul + Add
and contain no Gemm nodes at all, so their 252 KB of widened bias is untouched
here; that path is addressed separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
runwangdl and others added 2 commits July 28, 2026 23:26
The one-dimensional bias reached the kernel correctly but was tiled as if it were
still [M, O]. serializeTilingSolution built the C cube straight from the output
cube:

    CCube = HyperRectangle(tuple(cube.offset), tuple(cube.dims))

so the DMA was told to move M * O elements out of a buffer holding O. On CCT at 64
tokens that is 32768 B read from a 512 B buffer, 64x past the end, and the tile the
kernel then read was garbage.

It only showed up on the L3 and promote configurations because those are the ones
that actually transfer the operand; with everything resident in L2 there is no copy
and the oversized cube costs nothing. That is why siracusa-training-tiled-l2 passed
while the four L3 and promote jobs each failed with 4 errors out of 4, and why the
GAP9 jobs, whose smaller L1 budget yields a different tiling solution, did not catch
it either.

A broadcast bias now gets a cube of its own rank: the trailing O extent tracks the
output tile, the leading dimensions stay 1. Measured on the generated code for
Models/Training/CCT/cct_train at --l1 128000 --defaultMemLevel L3, node_84:

    before   C_transfer_size = {28672, 4096}   32768 B
    after    C_transfer_size = {448, 64}         512 B

512 B is the whole bias, O = 128 floats, split across the two O tiles of 112 and 16.
A widened [M, O] bias is untouched and still tiles like the output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts:
#	Deeploy/Targets/PULPOpen/Platform.py
@runwangdl

Copy link
Copy Markdown
Owner Author

Folded into #54 so this ships as a single merge. The branch is kept, and the content is in #54 with its description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant