MLXVLM: resolve the LFM2VL image token id from the vocabulary - #2
Open
Siddhesh2377 wants to merge 1 commit into
Open
Siddhesh2377 wants to merge 1 commit into
Siddhesh2377 wants to merge 1 commit into
Conversation
… of hardcoding 396
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LFM2VLProcessor.prepareexpands the image placeholder by scanning the templatedprompt for a hardcoded token id of 396. That id belongs to LFM2-VL. LFM2.5-VL-3B
declares
image_token_id124907, which is<image>in its vocabulary.On that model the scan finds nothing, so the prompt keeps the single placeholder
the chat template emitted. Inference then looks for the id from the model config,
finds one position, and compares it against 1536 features from the vision tower:
That is a
fatalError, so the host process dies outright. There is no error tocatch and no crash report is written, which makes it look like an unexplained
disappearance rather than a model problem.
The two halves of the file already disagreed about where the id comes from:
LFM2VLConfigurationparsesimage_token_idand inference uses it, while theprocessor guessed. This reads the id from the vocabulary instead and keeps 396 as
the fallback for a tokenizer with no
<image>entry, following thetokenId(_:default:)pattern already used inMuseGlimmer.swift.Verified against
mlx-lfm2.5-vl-3b-4biton macOS: a 2048x1324 screenshot thatpreviously killed the process now comes back with a full description.
🤖 Generated with Claude Code