Repository navigation
fix(kernels): int64 index offsets in SWITCH LoRA expand (int32 overflow on large intermediate_size, granite-4.1-30b) - #161
Conversation
…2 overflow on large intermediate_size
The SWITCH fused-LoRA expand kernels compute the packed lora_B element
offset as `lb_off_X + a*(N*X) + rX + n_range*X`. These index expressions
were int32, so with a large `N` (= intermediate_size) the offset exceeds
2^31 and the kernel issues an out-of-bounds load:
switch_lora_kernel.py _accum_slice_delta / _switch_lora_expand_kernel
RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered
This reproduces on granite-4.1-30b (intermediate_size=32768) for any
composed adapter at inference time; compose succeeds, the crash is on the
first forward. granite-4.1-3b and -8b (intermediate_size 12800) stay under
the int32 limit, which is why only the 30B class is affected. The trigger
is intermediate_size, not hidden_size (both 8B and 30B are 4096) or layer
count.
Fix: promote the index tensor operands to int64 so the whole offset
arithmetic is 64-bit — `n_range`/`h_range`/`m_range` and the per-tier
`r16..r512` ranges via `.to(tl.int64)` — across the three LBPACK kernels
(`_switch_lora_expand_kernel`, `_accum_slice_delta`,
`_switch_lora_shrink_expand_kernel`). The `lb_off_*` scalars stay Python
ints; promotion happens because a tensor operand in each sum is now int64.
(Note: `tl.int64(x)` is not callable — the cast must be `tensor.to(tl.int64)`.)
Validated on BlueVela (H200): granite-4.1-3b load+generate unchanged (no
regression on the common path), and granite-4.1-30b now loads and
generates where it previously hit the illegal memory access; a 12-domain
30B eval sweep then ran with zero illegal-memory failures.
Signed-off-by: Alexei Karve <karve@us.ibm.com>
|
Thanks for tracking this down and for the clear write-up. The fix is good: moving the index math to int64 resolves the 30B crash, and we'd like to merge it. Before we do, we have two small requests. 1. Root cause in the description We think the overflow comes from the The term that does overflow is Your patch already casts
That way a later cleanup won't drop the 2. Regression test Could you add a GPU test that runs the expand kernel with Alternative If you don't have time for either, that's fine. Just say so here, and we'll open a follow-up issue for the test and the doc fix and handle them ourselves. In that case we'll merge this PR as it is after our GPU test suite and benchmark pass. Thanks again! |
What
Fixes an int32 index overflow in the SWITCH fused-LoRA expand kernels that causes a CUDA illegal memory access when serving a composed adapter on large-
intermediate_sizebases (granite-4.1-30b).Fixes #160.
Why
The packed
lora_Belement offsetlb_off_X + a*(N*X) + rX + n_range*Xwas computed in int32. WithN = intermediate_size = 32768(granite-4.1-30b) it exceeds 2^31 → out-of-bounds load:granite-4.1-3b / -8b (
intermediate_size12800) stay under the limit, so only the 30B class is affected. The trigger isintermediate_size, nothidden_size(both 8B/30B are 4096) or layer count.How
Promote the index tensor operands to int64 (
n_range/h_range/m_rangeandr16..r512via.to(tl.int64)) in the three LBPACK kernels (_switch_lora_expand_kernel,_accum_slice_delta,_switch_lora_shrink_expand_kernel). The whole offset then promotes to 64-bit;lb_off_*scalars stay Python ints. No change to logic or results on paths that already fit int32.Validation
On H200:
Notes
tl.int64(x)is not callable (it's a dtype) — the cast istensor.to(tl.int64).