feat: make inline serializers concatenate runs faithfully

[cau-git]

Summary

InlineGroup runs now carry their own significant whitespace, and the serializers concatenate them faithfully instead of joining with a hard " ".

The join was the defect. It invented spaces the model never asked for, and while the separator lived in the serializer those artifacts could never be removed by fixing a producer:

source	before	after
2<sup>nd</sup>	2 nd	2nd
O<sub>2</sub>.	O 2 .	O2.
**Advanced Topics** in *Machine Learning*	**Advanced Topics** in *Machine Learning*	**Advanced Topics** in *Machine Learning*
This is *italic text*.	This is *italic text* .	This is *italic text*.

The ODF backend already emitted contract-honouring runs, so its committed Markdown goldens carried the mirror-image artifact — X 2 + Y 2 = Z — which is the shortest proof that the serializer, not the producer, was wrong.

What changed

Serializers. Markdown, HTML, LaTeX and DocLang join inline parts with "". DocLang keeps its delimiter between the element head and the body only; <content> remains its whitespace channel.

Whitespace hoisting. In DOCX and HTML the boundary space frequently lives inside a formatted run. Emitting it verbatim produces **bold ** tail, which CommonMark renders as literal asterisks. Markdown, HTML and LaTeX now split leading/core/trailing whitespace, apply the complete decoration stack to the core, and restore the edges. Doing this inside an individual formatter would not work — the hyperlink would immediately recapture the space ([**bold** ](url)). DocLang deliberately opts out and keeps the whitespace inside the markup, in <content>.

DocLang deserializer.

• _get_text stripped every fragment before joining, collapsing Advanced <bold>Topics</bold> into AdvancedTopics. It now trims the outer block boundary only, and never trims whitespace delivered by <content>.
• Bare text nodes use the standard XML heuristic: a whitespace run containing a newline is pretty-print indentation, one without it is content. Serialized DocLang uses <content> for whitespace-bearing runs, so this only governs hand-written and VLM-emitted input.
• Element-head children (<caption>, <description>, <summary>, <custom>) no longer bleed into visible body text, where they were also being double-modelled in item.meta.

Also fixed as a prerequisite (first commit, independently releasable): <text>2<superscript>nd</superscript></text> deserialized to a single item "nd" — the leading 2 was silently discarded. That is data loss, not a whitespace artifact.

Compatibility

Space-less runs are in users' saved .json files, and release coordination cannot fix a file on disk. The schema minor is bumped to 1.11.0 and a model_validator(mode="before") re-injects the legacy separator for documents stamped below it.

The goal is narrow and exact: preserve the old serializer's visible output, including its existing extra-space bugs. The old document does not contain enough information to recover semantic whitespace — ["H", "2", "O"] and ["Normal", "italic"] have the same shape but need different boundaries.

["Normal", "italic"] -> ["Normal ", "italic"] -> "Normal italic"
["H", "2", "O"]      -> ["H ", "2 ", "O"]     -> "H 2 O"        # legacy bug retained
["left ", "right"]   -> ["left  ", "right"]   -> "left  right"  # legacy double space retained

• It must run mode="before", because check_version_is_compatible rewrites version to CURRENT_VERSION and destroys the evidence.
• Versionless input is left alone rather than guessed at.
• The validator works on a copy, so the caller's dict is never mutated and re-validating the same raw value cannot compound separators.
• A separator never lands inside a delimiter: code and formula runs are wrapped in `/$, so the space goes to the plain neighbour, and when both neighbours are delimited it becomes a plain run of its own — `ab`` and$a$$b$` are both corrupt.
• One-way and idempotent; warns once per process.

This does not cover live conversion against an old docling backend, which builds the document in memory at the current version with no old stamp to key on. That remains an accepted, documented risk, mitigated by the paired docling change.

Behavioural changes to expect

• Chunk text changes. ChunkingDocSerializer defines no inline serializer, so it inherits the Markdown join. This is not golden-file cosmetics: it changes embeddings, so anyone with a persisted vector index gets silent retrieval drift on upgrade. It needs its own line in the release notes.
• Markdown / plaintext / HTML / LaTeX / DocLang exports are re-baselined.
• DocTags is untouched and explicitly out of scope. Its golden gains one empty <text></text> for the legacy fixture, because that document now carries an extra whitespace run and DocTags renders every run stripped — pre-existing behaviour, reproducible without any migration.

The failure mode inverts

Worth flagging for reviewers: today a producer that omits boundary whitespace yields extra spaces — ugly, harmless, unnoticed for as long as it has existed. After this change the same omission yields merged words. The cost of getting it wrong goes from cosmetic to severe, and it lands on third-party backends and on anything constructing DoclingDocuments by hand. add_inline_group's docstring now states the contract so the next backend author sees it.

Testing

602 passed, 6 skipped.

New test/test_inline_whitespace_contract.py holds the contract as a table — one set of inline-run sequences checked against md / itxt / html / latex / doclang, including whitespace inside a formatted run and inside a formatted run with a hyperlink, plus a whitespace-only run. It also covers:

• DocLang round-trip identity over the same table, both pretty-printed and compact;
• load normalization for old-schema documents, including the full plain/code/formula pair matrix;
• that the migration does not mutate caller-owned input and is stable across repeated validation;
• chunk text, pinned so the embedding drift cannot recur silently.

Pre-contract fixtures deliberately keep their old version stamp, so their goldens are byte-identical to before this PR and act as the regression test for the migration.

Release notes

• Ships as a minor, not a major. A 3.0 is not affordable on either package, so the commits deliberately avoid feat!: / BREAKING CHANGE: — python-semantic-release 7.34.6 would otherwise bump to 3.0.
• Paired with feat: honor the InlineGroup whitespace contract in the backends docling#3891, which fixes the producers. Merge this first: the dangerous combination is a new docling-core under old backends, which merges words.
• Downstream repos pinning docling-core independently (docling-serve, docling-eval, docling-mcp, docling-jobkit) need a floor bump in the same wave.

[github-actions]

✅ DCO Check Passed

Thanks @cau-git, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require two reviewer for test updates	👀 reviews
🟢	Enforce conventional commit	—

🔴 Require two reviewer for test updates

Waiting for

• #approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

---

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

❌ Patch coverage is 95.27559% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling_core/types/doc/document.py	91.80%	5 Missing ⚠️
docling_core/transforms/deserializer/doclang.py	97.29%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!