perf(PULPOpen): add a vector bias without materialising the broadcast - #51
Closed
runwangdl wants to merge 2 commits into
Closed
perf(PULPOpen): add a vector bias without materialising the broadcast#51runwangdl wants to merge 2 commits into
runwangdl wants to merge 2 commits into
Conversation
AddLayer.computeShapes rewrites the lower-rank operand to the higher-rank one's
shape. A Linear exported as MatMul + Add hands the Add a bias of shape [O] against
an activation of [1, M, O], so the bias is stored as M identical rows: on CCT-2 at
64 tokens a 128-wide bias occupies 32 KB to carry 512 B, and the eight live at the
peak account for 252 KB.
Keep the bias at [O] and teach the three places that assumed equal shapes:
PULPFloatAddTemplate walks the second operand with a wrapped row index. The
index is advanced and reset rather than taken modulo,
because there is no hardware divide on this core.
PULPBroadcastAddTileConstraint
constrains only the trailing axis, and gives the
broadcast operand the trailing slice of the output cube
instead of the whole cube.
PULPAddLayer skips the rewrite when the smaller operand has a single
non-unit axis.
Each falls back to the inherited elementwise behaviour when the operands do have
equal shapes, so an ordinary Add is untouched. Registered on both the PULPOpen and
GAP9 mappings; GAP9 keeps its own, and a layer registered only in PULPOpen's has no
effect on a GAP9 deployment.
Measured on GAP9, --l1 122000 --defaultMemLevel L3, L3 peak, address-deduplicated:
CCT linear probing 1954.9 -> 1702.6 KB (-12.9%)
CCT LoRA on one block 1965.5 -> 1713.0 KB (-12.8%)
CCT full FT on one block 2376.8 -> 2252.0 KB ( -5.3%)
The bias's own L3 allocation drops from cl_ram_malloc(32768) to cl_ram_malloc(512),
and the activation side is unchanged at 512.0 KB, which is the expected shape of the
saving: a bias is persistent, not an activation.
Verified on gvsoc: linear probing Errors: 0 (37.68M train cycles), one-block full
fine-tuning Errors: 0 (54.94M). The LoRA variant does not build, with or without
this change: hoistConstant asserts one consumer per Constant, and a LoRA graph's
frozen transposed weight feeds both the forward Gemm and its gradient. A control
run on this branch without this commit fails with the identical assertion, so that
limitation is pre-existing; its memory figure above therefore comes from the
allocation plan, not from a run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…add51 # Conflicts: # Deeploy/Targets/GAP9/Platform.py # Deeploy/Targets/PULPOpen/Platform.py
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AddLayer.computeShapesrewrites the lower-rank operand to the higher-rank one's shape. A Linear exported asMatMul + Addhands the Add a bias of shape[O]against an activation of[1, M, O], so the bias ends up stored asMidentical rows — on CCT-2 at 64 tokens, 32 KB to carry 512 B, and the eight live at the peak account for 252 KB.What changes
Keep the bias at
[O]and teach the three places that assumed equal shapes:PULPFloatAddTemplatePULPBroadcastAddTileConstraint(new)PULPAddLayerEach falls back to the inherited elementwise path when the operands do have equal shapes, so an ordinary Add is untouched.
Targets/Genericis not modified.Measured
GAP9,
--l1 122000 --defaultMemLevel L3, L3 peak, address-deduplicated:The bias's own L3 allocation drops from
cl_ram_malloc(32768)tocl_ram_malloc(512), and the activation side is unchanged at 512.0 KB — the expected shape of the saving, since a bias is persistent rather than an activation.Verification, including what does not pass
hoistConstantasserts one consumer per Constant, and a LoRA graph's frozen transposed weight feeds both the forward Gemm and its gradient. A control run on this branch with this commit removed fails with the identical assertion, so the limitation is pre-existing. Its figure in the table therefore comes from the allocation plan, not from a run.Stacked on #50, which adds the GAP9 registration this builds on.
🤖 Generated with Claude Code