Skip to content

fix(bridge): honor stop_at_eos=False in hf_generate - #1826

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/hf-generate-disable-eos
Sep 28, 2026
Merged

jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/hf-generate-disable-eos

Conversation

@emerardd

Copy link
Copy Markdown
Contributor

Description

Honor stop_at_eos=False in TransformerBridge.hf_generate() by explicitly passing eos_token_id=None to HuggingFace generation. Previously the wrapper omitted the argument, allowing HF to inherit model.generation_config.eos_token_id and stop early despite the Bridge flag.

An offline zero-weight tiny GPT-2 with EOS 0 demonstrates the bug on dev at 11224618: requesting three new tokens for [[1]] with stop_at_eos=False returns [[1, 0]]; disabling EOS explicitly in HF returns [[1, 0, 0, 0]].

Changes

  • Disable configured EOS stopping explicitly when stop_at_eos=False, without mutating the model's generation configuration.
  • Ignore an explicitly supplied EOS id when the stopping switch is false, matching native generate() semantics. This intentionally corrects the previous inconsistent precedence.
  • Preserve the existing explicit/tokenizer EOS selection when the switch is true.
  • Document the precedence and that custom stopping criteria still apply.

This is independent of the seq2seq streaming fix in #1825 and is based directly on dev.

Type of change

  • Bug fix

Validation

  • Before the fix, the new regression module reported 7 failed, 12 passed.
  • After the fix and an additional custom-stopping regression, the affected surface reported 63 passed, 2 dependency warnings, with no skips. All 20 new tests passed.
  • The tests use a locally constructed tiny GPT-2 and tokenizer, with no model downloads. They cover true/false stopping, default and explicit EOS ids, tensor/string/batched-list input, unmodified generation config, ModelOutput scores, and custom stopping criteria.
  • Existing HF batch-padding, generation-capability and native stopping-criteria tests also passed.
  • Changed-file pycln/isort/Black checks passed; mypy . passed for 385 source files.
  • The full test suite was not rerun. The reported warnings are SWIG dependency deprecations.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Only the affected test surface was run, so the complete unit-suite checkbox is left unchecked. The existing interface tests were not rewritten.

@jlarson4

Copy link
Copy Markdown
Collaborator

Looks good, solid solution @emerardd, merging now

@jlarson4
jlarson4 merged commit 37da09c into TransformerLensOrg:dev Sep 28, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants