Skip to content

fix: reject template options where chunk_overlap >= chunk_size - #92

Merged
yifanfeng97 merged 1 commit into
yifanfeng97:mainfrom
Frankie-Xu:fix/template-chunk-overlap-validation
Sep 11, 2026
Merged

yifanfeng97 merged 1 commit into
yifanfeng97:mainfrom
Frankie-Xu:fix/template-chunk-overlap-validation

Conversation

@Frankie-Xu

Copy link
Copy Markdown
Contributor

Problem

YAML options.chunk_size and options.chunk_overlap have no cross-field check. When overlap is greater than or equal to size, character splitting degenerates (empty or non-progressing chunks).

Fix

Validation runs on the unified Options model used by parse_option:

  • chunk_size, when set, must be >= 1
  • chunk_overlap, when set, must be >= 0
  • when both are set, chunk_overlap must be strictly smaller than chunk_size (equality fails)

Constructing an AutoType by hand is out of scope.

Tests

Parser tests cover valid pairs, overlap equal to size, overlap greater than size, non-positive size, negative overlap, and omitted fields. Bundled presets still load and parse_option still succeeds. HE-T rules are unchanged.

Compatibility

Templates that omit both fields behave as before. Rebased onto v0.9.0 / d8fce60. Does not touch ingest or provenance.

Made with Cursor

…chunk_size

Co-authored-by: Cursor <cursoragent@cursor.com>
@yifanfeng97

Copy link
Copy Markdown
Owner

Merged — verified locally that degenerate combos (overlap == size and overlap > size) are rejected with a clear author-facing message while valid pairs are unaffected. No bundled preset violates the constraint. One follow-up idea: he template validate doesn't currently exercise parse_option, so these invalid combos still pass validate and only fail at Template.create — surfacing the check there too would catch them even earlier.

@yifanfeng97
yifanfeng97 merged commit 266cb4c into yifanfeng97:main Sep 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants