Skip to content

MLXVLM: resolve the LFM2VL image token id from the vocabulary - #2

Open
Siddhesh2377 wants to merge 1 commit into
mainfrom
siddhesh/lfm2vl-image-token-id
Open

Siddhesh2377 wants to merge 1 commit into
mainfrom
siddhesh/lfm2vl-image-token-id

Conversation

@Siddhesh2377

Copy link
Copy Markdown
Collaborator

LFM2VLProcessor.prepare expands the image placeholder by scanning the templated
prompt for a hardcoded token id of 396. That id belongs to LFM2-VL. LFM2.5-VL-3B
declares image_token_id 124907, which is <image> in its vocabulary.

On that model the scan finds nothing, so the prompt keeps the single placeholder
the chat template emitted. Inference then looks for the id from the model config,
finds one position, and compares it against 1536 features from the vision tower:

MLXVLM/LFM2VL.swift:960: Fatal error: Image features and image tokens do not match: tokens: 1, features 1536

That is a fatalError, so the host process dies outright. There is no error to
catch and no crash report is written, which makes it look like an unexplained
disappearance rather than a model problem.

The two halves of the file already disagreed about where the id comes from:
LFM2VLConfiguration parses image_token_id and inference uses it, while the
processor guessed. This reads the id from the vocabulary instead and keeps 396 as
the fallback for a tokenizer with no <image> entry, following the
tokenId(_:default:) pattern already used in MuseGlimmer.swift.

Verified against mlx-lfm2.5-vl-3b-4bit on macOS: a 2048x1324 screenshot that
previously killed the process now comes back with a full description.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant