Repository navigation
Metal MTP: lift the MTP window to runtime after the current draft work #1135
Description
We plan MTP speculative decode on the Metal backend. It should reuse the MTP window, not copy it.
crates/app/src/mtp/policy.rs and most of MtpDraft hold no card types. The plan is to move them to runtime::mtp behind one Host: Verify trait. CUDA then implements Host for Session<B>, and Metal implements it for its session.
We will start this only after your current work on crates/app/src/mtp* and crates/gpu/src/draft* lands. Until then we work on Metal-only parts (M-row mat-vec, Verify, the MTP layer).
If you add code there now, it helps to keep card types out of policy.rs and the MtpDraft state. If you see a better cut, please say so here.
Activity
midagedev commentedon Nov 4, 2026
Done.
MtpDrafttoruntime::mtp, behind aHosttrait. The CUDA side isimpl Host for Session<B>in app, and its behaviour and kernels are unchanged.SessionimplementHost, sogenerate_metal --draft mtpuses the same draft code.On an M1 Pro with Qwen3.8-27B MLX 4-bit, MTP decodes ×1.12–1.22 faster than plain decode. The ids equal plain greedy. The 2-row verify pass is the next lever.