bench: mark decode 'n/a' when the ANE cannot run it - #152
Merged
Conversation
The decode column showed a bare '-' on M1 / M2 Pro, indistinguishable from 'not measured'. In fact decode ran but the ANE could not: the 32000-vocab lm_head matmul exceeds those families' 16384 max matmul dimension (NotImplementedError, recorded in the decode row's status). Now the table shows 'n/a' with a footnote explaining it is a real per-generation limit, distinct from '-' (not submitted).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The decode column in ROOFLINES.md showed a bare
-on M1 and M2 Pro, which reads as 'not measured'. It is actually a real per-generation ANE limit: the decode benchmark's 32000-vocab lm_head matmul(1, 32000)exceeds those families' 16384 max matmul dimension, so it can't run untiled (NotImplementedError, already recorded in the decode row'sstatus). M5 (family 5) runs it -> 117 tok/s.Now the table shows
n/afor the dim-limited case with a footnote, distinct from-(simply not submitted). Aggregator-only; ruff clean,--checkgreen.Getting an actual number on those chips would need tiling the lm_head into <=16384-wide chunks in
decode_measurement.py-- doable, but that's a paper-repro script and a tiled lm_head isn't strictly comparable to M5's untiled one, so it's a separate, careful change (best done as a fallback that only triggers when the untiled compile fails, leaving M5 unchanged).