Skip to content

perf(PULPOpen): add a vector bias without materialising the broadcast - #51

Closed
runwangdl wants to merge 2 commits into
feat/gemm-vector-biasfrom
feat/add-vector-bias
Closed

perf(PULPOpen): add a vector bias without materialising the broadcast#51
runwangdl wants to merge 2 commits into
feat/gemm-vector-biasfrom
feat/add-vector-bias

Conversation

@runwangdl

Copy link
Copy Markdown
Owner

AddLayer.computeShapes rewrites the lower-rank operand to the higher-rank one's shape. A Linear exported as MatMul + Add hands the Add a bias of shape [O] against an activation of [1, M, O], so the bias ends up stored as M identical rows — on CCT-2 at 64 tokens, 32 KB to carry 512 B, and the eight live at the peak account for 252 KB.

What changes

Keep the bias at [O] and teach the three places that assumed equal shapes:

PULPFloatAddTemplate walks the second operand with a wrapped row index — advanced and reset, not taken modulo, since there is no hardware divide on this core
PULPBroadcastAddTileConstraint (new) constrains only the trailing axis, and gives the broadcast operand the trailing slice of the output cube instead of the whole cube
PULPAddLayer skips the rewrite when the smaller operand has a single non-unit axis

Each falls back to the inherited elementwise path when the operands do have equal shapes, so an ordinary Add is untouched. Targets/Generic is not modified.

Measured

GAP9, --l1 122000 --defaultMemLevel L3, L3 peak, address-deduplicated:

model before after
CCT linear probing 1954.9 KB 1702.6 KB −12.9%
CCT LoRA on one block 1965.5 KB 1713.0 KB −12.8%
CCT full FT on one block 2376.8 KB 2252.0 KB −5.3%

The bias's own L3 allocation drops from cl_ram_malloc(32768) to cl_ram_malloc(512), and the activation side is unchanged at 512.0 KB — the expected shape of the saving, since a bias is persistent rather than an activation.

Verification, including what does not pass

  • linear probing — Errors: 0, 37.68M train cycles
  • one-block full fine-tuning — Errors: 0, 54.94M train cycles
  • one-block LoRA — does not build, with or without this change. hoistConstant asserts one consumer per Constant, and a LoRA graph's frozen transposed weight feeds both the forward Gemm and its gradient. A control run on this branch with this commit removed fails with the identical assertion, so the limitation is pre-existing. Its figure in the table therefore comes from the allocation plan, not from a run.

Stacked on #50, which adds the GAP9 registration this builds on.

🤖 Generated with Claude Code

runwangdl and others added 2 commits July 28, 2026 01:27
AddLayer.computeShapes rewrites the lower-rank operand to the higher-rank one's
shape. A Linear exported as MatMul + Add hands the Add a bias of shape [O] against
an activation of [1, M, O], so the bias is stored as M identical rows: on CCT-2 at
64 tokens a 128-wide bias occupies 32 KB to carry 512 B, and the eight live at the
peak account for 252 KB.

Keep the bias at [O] and teach the three places that assumed equal shapes:

  PULPFloatAddTemplate      walks the second operand with a wrapped row index. The
                            index is advanced and reset rather than taken modulo,
                            because there is no hardware divide on this core.
  PULPBroadcastAddTileConstraint
                            constrains only the trailing axis, and gives the
                            broadcast operand the trailing slice of the output cube
                            instead of the whole cube.
  PULPAddLayer              skips the rewrite when the smaller operand has a single
                            non-unit axis.

Each falls back to the inherited elementwise behaviour when the operands do have
equal shapes, so an ordinary Add is untouched. Registered on both the PULPOpen and
GAP9 mappings; GAP9 keeps its own, and a layer registered only in PULPOpen's has no
effect on a GAP9 deployment.

Measured on GAP9, --l1 122000 --defaultMemLevel L3, L3 peak, address-deduplicated:

  CCT linear probing        1954.9 -> 1702.6 KB   (-12.9%)
  CCT LoRA on one block     1965.5 -> 1713.0 KB   (-12.8%)
  CCT full FT on one block  2376.8 -> 2252.0 KB   ( -5.3%)

The bias's own L3 allocation drops from cl_ram_malloc(32768) to cl_ram_malloc(512),
and the activation side is unchanged at 512.0 KB, which is the expected shape of the
saving: a bias is persistent, not an activation.

Verified on gvsoc: linear probing Errors: 0 (37.68M train cycles), one-block full
fine-tuning Errors: 0 (54.94M). The LoRA variant does not build, with or without
this change: hoistConstant asserts one consumer per Constant, and a LoRA graph's
frozen transposed weight feeds both the forward Gemm and its gradient. A control
run on this branch without this commit fails with the identical assertion, so that
limitation is pre-existing; its memory figure above therefore comes from the
allocation plan, not from a run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…add51

# Conflicts:
#	Deeploy/Targets/GAP9/Platform.py
#	Deeploy/Targets/PULPOpen/Platform.py
@runwangdl

Copy link
Copy Markdown
Owner Author

Folded into #54 so this ships as a single merge. The branch is kept, and the content is in #54 with its description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant