Skip to content

[v3.0.2] List rendering robustness fixes (itemize / enumerate) - #422

Draft
OlgaRedozubova wants to merge 34 commits into
masterfrom
dev/olga/list-padding-block-item-marker
Draft

[v3.0.2] List rendering robustness fixes (itemize / enumerate)#422
OlgaRedozubova wants to merge 34 commits into
masterfrom
dev/olga/list-padding-block-item-marker

Conversation

@OlgaRedozubova

@OlgaRedozubova OlgaRedozubova commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Two groups of renderer fixes, shipped together as one 3.0.2 release:

  • Lists (1–4): four unrelated inputs made LaTeX itemize / enumerate lists render incorrectly; the fixes live in the list-env and footnote block rules.

  • Code blocks (5): code-text styles were absolute (px / rem), so a code block did not scale when a consumer sizes a rendered block by a single font-size; the styles are now relative.

  • Version bump: 3.0.1 → 3.0.2

  • Specs: pr-specs/2026-07-list-rendering-robustness.md, pr-specs/2026-07-code-block-font-scaling.md

  • Changelog: doc/changelog.md

  • Full test suite green.


1. Marker padding for block-content items

A top-level list gets its padding-inline-start from the widest custom \item[...] marker, but the width was only measured on the inline item path. Items whose content is a block environment (\begin{figure}, \begin{tabular}, a code fence) were skipped, so a list whose long-marker items all hold block content lost its padding.

MMD example

\begin{itemize}
\item[11.33] In Fig. 11-20, when pump YA delivers 5.00 cfs.
\begin{figure}
\includegraphics[alt={},max width=\textwidth]{https://cdn.mathpix.com/snip/images/QxUdugRlfaW7h_IxvYNbgRDXjd0egDGrEXiSjKH7FGU.original.fullsize.png}
\caption{Fig. 11-20}
\end{figure}
\item[11.34] In Fig. 11-21, which system has the greater capacity.
\begin{figure}
\includegraphics[alt={},max width=\textwidth]{https://cdn.mathpix.com/snip/images/QxUdugRlfaW7h_IxvYNbgRDXjd0egDGrEXiSjKH7FGU.original.fullsize.png}
\caption{Fig. 11-21}
\end{figure}
\end{itemize}
Before After
Screenshot 2026-07-28 at 18 32 50 Screenshot 2026-07-28 at 18 32 18

2. Marker width: fullwidth/CJK, math, and wrapped content

Marker width summed String.length over top-level text tokens only, so several markers were undercounted and the list got too small an indent (marker overlaps the content):

  • Fullwidth / CJK (\item[11.], U+FF0E) counted as narrow ASCII — now East-Asian Wide/Fullwidth chars count as 2.
  • Math (\item[$x^4+x^4$]) contributed 0 — now uses the token's rendered widthEx.
  • Wrapped content (\item[\textbf{…}]) contributed 0 — now measured through the wrapper's children.

ASCII text markers are unchanged.

MMD examples

\begin{itemize}
\item[I] Item 1.
\item[11.] Item 11..
\item[I] Item 1.
\end{itemize}
\begin{itemize}
\item[$x^4 + x^4$] a footnote note or item body
\end{itemize}
\begin{itemize}
\item[\textbf{x^4 + x^4}] a footnote note or item body
\end{itemize}
Case Before After
fullwidth 11. Screenshot 2026-07-28 at 18 09 56 Screenshot 2026-07-28 at 18 10 13
math $x^4+x^4$ Screenshot 2026-07-28 at 18 27 20 Screenshot 2026-07-28 at 18 27 11
bold \textbf{…} Screenshot 2026-07-28 at 18 29 21 Screenshot 2026-07-28 at 18 29 06

3. No env-state leak from an aborted list parse

The list block rule parses speculatively into a buffered state that shares env by prototype. On abort (unclosed list) or a silent probe it returned without restoring env.isBlock (and env.inheritedListType); the leaked isBlock = true then let the inline list fallback fire on the following content, so an unclosed itemize before a tabular rendered a broken partial list with empty <> item bodies. Those transient fields are now restored on both paths, so the unclosed list degrades to plain text — exactly as it does without a tabular.

MMD example

\begin{itemize}
\item[(d1)] Asad Ali Khan
\item[(d2)] Asad Ali Khan
\begin{tabular}{|l|l|}
cell
\end{tabular}
\begin{itemize}
Before After
Screenshot 2026-07-28 at 18 34 13 Screenshot 2026-07-28 at 18 34 27

4. Footnote block rules stop at a list start

The \footnote / \footnotetext block rules scan forward for their open tag, terminating at block boundaries so they don't swallow following blocks. The LaTeX list rule was not a terminator, so a \begin{itemize} between a paragraph and a later footnote (no blank line) did not stop the scan: the list was swallowed and rendered as literal text (a blank line masked the bug). The LaTeX list rule is now a terminator for both — \footnotetext via its full set, \footnote via a minimal fence + Lists (keeping its cheap scan).

This is also a performance fix: on repeated paragraph + list-with-footnotetext units without blank separators the missing terminator made the scan run across every list into the rest of the document — O(N²), seconds on large inputs; terminating at the list makes it linear (guarded by a scaling test).

MMD example

Wiener's life work, its enormous range notwithstanding, exhibits a coherence of thought from start to finish, reminiscent of a great work of art. To write a biography of a person of so towering a stature is an
\begin{itemize}
\item[] \footnotetext{
1 Fortunately, this correspondence is now being studied by Dr. Albert C. Lewis of the Bertrand Russell Editorial Project, McMaster University, Ontario, Canada, and may see the light of day within a few years.
}
\end{itemize}
Before After
Screenshot 2026-07-28 at 18 35 45 Screenshot 2026-07-28 at 18 35 55

5. Code-block styles scale with the em context

Code-text styles were pinned to absolute values, so when a consumer scales a rendered block by setting a single font-size on the container (e.g. image export), everything scaled except the code — px is fixed and rem resolves against the root, not the block's em. The four properties are now relative, calibrated so a 16px base is pixel-identical to before (only code padding moves 16px → 15px):

  • #setText pre { font-size: 0.9375em; } (was 85%)
  • #setText pre code { font-size: inherit; } (was 15px)
  • #setText pre code { line-height: 1.6; } (was 24px)
  • #setText pre code { padding: 1em; } (was 1rem)

Styles only — no change to lstlisting / fenced-code markup.

MMD example

\begin{lstlisting}
def area(r):
    return 3.14159 * r * r
\end{lstlisting}

Scale the rendered block by setting a large font-size on the container (as image export does).

Before After
Screenshot 2026-07-28 at 18 37 00 Screenshot 2026-07-28 at 18 37 13

Testing

List cases in tests/_data/_lists/_data.js and tests/_list-marker-padding.js: block-content markers (figure/fence), fullwidth 11., math and \textbf markers, an unclosed list + tabular, a list after a paragraph with a multiline \footnote{} / \footnotetext{}, and a markdown list not swallowed before a \footnote.

Guards: a scaling test (tests/_footnotes_latex.js) rejecting the O(N²) footnote scan; a silent-Lists env invariant (tests/_parse-isolation.js); selector-scoped code-style assertions (tests/_styles.js).

Full suite green.

Non-goals

  • The list padding heuristic itself (px-per-char, the > 3 threshold) is unchanged.
  • Malformed list input still degrades gracefully; the goal is plain text, not a partial list.
  • Nested lists keep the default indent even with long markers (pre-existing; separate ticket).
  • Code blocks: rendering at the default 16px base is unchanged apart from the 1px code padding; inline code, pre scroll/overflow, highlight colors, and table-cell padding are untouched.

These .d.ts files were emitted by an older tsc and only differ in tuple
formatting (e.g. [number, number][]); regenerate them with the pinned
4.9.5 compiler so committed declarations match the toolchain.
A top-level itemize/enumerate only got its inline-start padding when the
widest \item[...] marker was measured. That measurement lived only in the
inline item path, so items whose content is a block environment
(\begin{figure}, \begin{tabular}, code fence) were skipped. When every
long-marker item held block content, the list lost its padding entirely.

Extract the marker-width calc into a shared computeMarkerPadding helper and
apply it in the block item path too (setTokenListItemOpenBlock now returns
the created token). Add regression tests for fenced-code and figure items.
Marker padding summed String.length, so a fullwidth/CJK marker like
\item[11.] (U+FF0E, length 3) was undercounted versus its ASCII twin
\item[11.42] (length 5) and fell under the padding threshold, leaving the
list without indentation while a neighbouring ASCII list got it.

Count East-Asian Wide/Fullwidth characters as width 2 (iterating by code
point so surrogate pairs count once). ASCII markers are unaffected.
The Lists block rule parses speculatively into a buffered state, but that
state shares env by prototype, so parsing mutates the real env.isBlock (and
inheritedListType). On an unclosed list the rule returned false without
restoring them; a leaked isBlock=true then let the inline list fallback fire
on the following text, so an unclosed itemize before a tabular rendered a
broken partial list with empty <> item bodies instead of plain text.

Restore isBlock and inheritedListType when the speculative parse aborts, so
the unclosed list degrades to text exactly as it does without a tabular.
The \footnote/\footnotetext block rules scan forward for their open tag; their
terminator set included the core markdown list rule but not the LaTeX list rule,
so a \begin{itemize} between a paragraph and a later footnote (no blank line) was
swallowed and rendered as literal text. Add the LaTeX list rule to the footnote
terminator set and use it for both footnote block rules.

Bumps version to 3.0.2; adds the list-rendering-robustness spec and changelog
entry covering this branch's list fixes.
@OlgaRedozubova OlgaRedozubova self-assigned this Jul 28, 2026
Code-text styles were pinned to absolute values (`pre code` font-size 15px /
line-height 24px / padding 1rem, `pre` font-size 85%), so a code block did not
scale when a consumer sizes a rendered block via a single font-size on the
container (e.g. image export): everything else scaled but the code stayed ~15px.

Make them relative: `pre` font-size 0.9375em; `pre code` font-size inherit,
line-height 1.6, padding 1em. Calibrated so a 16px base is pixel-identical
except code padding (16px -> 15px). Styles only; no HTML change.

Regenerate style snapshots, add a regression gate that rejects absolute
font-size/line-height and rem padding for `pre code`; spec + changelog under 3.0.2.
…inators, perf guard

- Restore env.isBlock/inheritedListType on both the abort and silent paths of the
  Lists block rule, so a silent terminator probe never mutates shared state.
- \footnote uses a minimal fence+Lists terminator set (not the full set): fixes the
  list swallow with one cheap extra probe, avoiding a per-line cost regression.
- Add a first-char fast bail to the Lists rule (no substring allocation) since it now
  runs as a per-line terminator in paragraph/footnote scans.
- Measure markers by display width (East-Asian wide chars count as 2, BMP only).
- Tests: silent env invariant, marker width edge cases (math/emoji/nested), footnote
  recognition after block constructs, and a scaling guard that rejects the O(N^2)
  footnote scan across unterminated lists.
- Selector-scoped code-style gate; spec/changelog updated.
- Measure math markers by their rendered widthEx and wrapped markers (e.g. \textbf)
  by recursing into children, so such markers no longer get too small an indent
  (previously only top-level text tokens counted).
- Extract the char-based width primitives (isWideChar, displayWidth, tokenDisplayWidth)
  into common/display-width.ts — the char-based counterpart of getTextWidthByTokens;
  computeMarkerPadding now delegates to them.
- Round the padding px (marker width can be fractional once math width is included).
- Tests: wide-math and bold marker padding, markdown list not swallowed before a
  \footnote; explicit non-empty guard on the code-style assertions.
- Spec and changelog updated.
Empty `<>` item bodies from a leaked env.isBlock are fixed on both paths:
the speculative list parse restores its transient env fields on every exit
(abort/silent/commit/exception), and a tabular inside a list no longer carries
those flags into its envToInline snapshot (which core-inline replays onto the
shared env). Adds common/env-transient.ts as the single source of truth.

Marker padding is emitted in ex, not px, so it scales with the container
font-size like the marker's math SVG; this also fixes math markers being
clipped (exact widthEx + gap, both ex). The marker gap (.li_level padding-right)
and the default list indent move to ex/em. Marker width also trims edge
whitespace and falls back to source length when widthEx is absent.

Merges the two footnote terminator resolvers into resolveEnabledRuleFns.

Adds tests/_display-width.js, an env-leak adversarial matrix, exact ex-value
assertions, and updates fixtures/snapshot/specs/changelog.
Emit list marker padding-inline-start in em (was ex), converted from the ex
measurement via the default font metrics (EX_TO_EM = exDef/fonSizeDef). Custom
padding is emitted only when it exceeds the default (2.5em), so it never resolves
below it — no CSS max() or cross-unit comparison. The marker gap (0.625em) and
default indent (2.5em) move to em too; the attribute now carries its unit
("3.32em"). Math without a widthEx (non-SVG output) keeps the default indent
rather than a fabricated source-length estimate.

Text markers use 1.3 ex/char (reserve is per character cell, not per glyph);
this is ~5-14% tighter than 3.0.1 and wide all-caps markers can overlap —
documented in the spec Non-Goals and changelog.

Sync docs/comments with the code, add tests: EX_TO_EM vs fontMetrics, exact em
values, glyph-width limitation, MARKER_GAP_EM vs CSS.
A speculative (silent/aborted) list parse runs the body's \begin{figure} /
\begin{table}\caption and bumps the module-global caption counters, but its
tokens are discarded — so a \footnote after such a list, and the pre-existing
paragraph-terminator probe, shifted Figure N / Table N. Snapshot and restore the
counters when the parse is discarded: in the Lists block rule's finally and in
parseListEnvRawToTokens (inline path). A figure in a list is now Figure 1.

Extract the counters to the leaf module common/caption-counters.ts so the list
rule no longer imports the heavy begin-table module (removes an import-cycle
edge); the deep-import helpers Clear{Table,Figure}Numbers move there, renamed to
clear{Table,Figure}Numbers. Move the default font metrics (DEFAULT_FONT_SIZE_PX
/ DEFAULT_EX_PX) to consts as the single source for EX_TO_EM and FontMetrics.
Treat combining U+3099/U+309A as zero width in both isWideChar and displayWidth;
set committed before flush; drop the now-redundant CSS guard.

Bump 3.0.1 -> 3.1.0 (minor): the data-padding-inline-start attribute is now an
em value and caption numbering changes for existing docs (see changelog/spec).

Add regression tests: caption numbering (absolute, multi-float order, nested
list, \ref matches the caption number), attribute format contract, 3-char
threshold, combining marks in isWideChar and displayWidth.
Replace the flat per-character reserve with a per-glyph-class estimate in em
(narrow / normal / wide / extra-wide W@% / East-Asian full-width; combining
marks 0). Measured against real font metrics, the old flat 14px/char
over-reserved narrow/digit markers by 36-70% and under-reserved all-caps by
~14%; the class estimate holds a +11..+27% margin — a tighter, correct indent
for narrow/normal markers and a wider, safer one for all-caps. Math still uses
the exact widthEx; short markers fall to the 2.5em default; the reservation is
clamped at 20em so a pathological OCR marker can't blow out the content column.

Only text-like leaves (text / code_inline / text_special) are measured, so a
`code`-span marker now contributes its width while an html_inline marker's raw
markup is not counted. Remove the now-unused displayWidth; dedupe the combining
-mark predicate.

Add a font-metrics lock test (Arial fixture): the rendered indent is never
smaller than the marker's true glyph width + gap. Update fixtures, docs, and
the attribute-format / clamp / mixed-marker tests.
A list's marker padding was accumulated as the max over all markers at every
depth and written to the outermost list token, so a wide marker several levels
deep drove the top-level list's indent (e.g. a deep [3.1.1.1] gave the outer
list 4.31em while its own 1.–5. markers are narrow). Track a stack of open list
tokens and attribute padding to the current (innermost) list instead.

Nested lists now also emit and apply their own padding (drop the top-level-only
gate in finalizeListItems and the level>1 suppression in the list renderers), so
each level reserves for its own markers — a wide nested marker indents its own
list rather than overflowing the container. The outer list reflects only its own
markers (default when they're narrow).

Add regression tests (per-level attribution; deep marker not bubbled up) and
update changelog/spec.
Marker padding (on top of the per-nesting-level attribution): keep the
default 2.5em indent on every list and emit a custom padding-inline-start
only when a marker overflows the accumulated ancestor indent plus that
default, reserving just the shortfall (need - ancestorIndent) so nested
markers don't compound. Flat lists are unchanged (e.g. [11.33] -> 3.51em);
ordinary nested lists (numbering, bullets) stay at the default; only a
genuinely wide nested marker reserves extra, and less than its full width.

Fix \item detection: the item-command regexes matched too loosely -
LATEX_ITEM_COMMAND_INLINE_RE matched the bare word "item" (no backslash),
and LATEX_ITEM_COMMAND_RE / LATEX_LIST_BOUNDARY_INLINE_RE matched any
\item-prefixed command (\itemsep, \itemindent). A list-item body containing
such text split into a spurious item, and a multiline \footnote/\footnotetext
inside an item was broken mid-brace and rendered as literal text. All three
now require \item plus a command boundary (\item[...] or \item(?![a-zA-Z])).

Update changelog and the list-rendering pr-spec; add regression tests
(_lists.js: \itemsep/\itemindent; _footnotes_latex.js: "item" in a footnote
body; _list-marker-padding.js: per-level B2 attribution).
…ate padding style

Resolve per-list padding in one top-down pass (resolveListPadding) after the whole
environment is parsed, instead of emitting during finalize. finalizeListItems now only
records each list's widest marker width; the pass computes the indent once every list's
final width is known, so item order no longer skews a nested list (a wide parent item
after the sublist resolves the same as before it).

Clamp the total (ancestor + own), not just the added shortfall, to LIST_MAX_INDENT_EM,
so cumulative indent never exceeds the clamp on a pathological nested marker.

Validate data-padding-inline-start against /^\d+(\.\d+)?em$/ before inlining it as a
style, so only a bare em value can reach the rendered CSS.

Docs: the attribute is now emitted on nested lists (Migration); an empty parent item
followed by a wide sublist marker overlaps as LaTeX does (Non-Goals, verified in
Overleaf); image markers reserve by alt text; correct the code-block "16px" note for
PreviewStyle's 17px base. Add regression tests (order independence, cumulative clamp,
bold reserve lock).
Core-markdown <ul>/<ol> (no .itemize/.enumerate class) had no padding-inline-start,
so browsers used their 40px default, which doesn't scale when a consumer sizes
content by font-size. Add a scoped rule (#preview-content ul, #setText ul, ... ol)
setting padding-inline-start: 2.5em, matching the LaTeX-list default and scaling with
the container. Scoped only per 2026-03-mmd-css-scoping.md — a bare ul/ol would restyle
the host page. LaTeX lists are unaffected (their class rules and inline padding win by
specificity, same value); TOC is safe (separate #toc_container; the in-content
.table-of-contents is display:none).

Regenerate the listsStyles snapshot; update changelog and the list-rendering pr-spec.
…oped-only

Follow-up to the markdown-list padding rule (bc57340):
- The rule also covers the generated footnotes list (<ol class="footnotes-list">),
  which had no explicit padding — disclosed in changelog and the pr-spec/Migration.
- Migration: the wrapped ul/ol padding floor rises from the UA default (specificity 0)
  to (1,0,1), so a consumer's class-specificity rule no longer overrides the indent.
- Spec: the #toc_container-over-markdown precedence rests on bundle order (lists before
  toc), not specificity (both 1,0,1) — documented, with the specificity-boost option.
- Test: assert listsStyles has no bare ul/ol selector, so a future edit can't quietly
  regenerate the snapshot with one.
The markdown-list rule (#setText ul { padding-inline-start: 2.5em }) and
#toc_container ul { padding: 0 } are both (1,0,1), so which wins depended on bundle
order (lists before toc). TocStyle is only emitted when toc is enabled, so a consumer
that renders a visible #toc_container inside the wrapper without TocStyle got 2.5em on
the TOC list.

Add a guard in the always-emitted lists module — #preview-content #toc_container ul/ol,
#setText #toc_container ul/ol { padding-inline-start: 0 }, specificity (2,0,1) — so the
TOC list stays at 0 regardless of order or whether TocStyle is present. It matches any
ul/ol inside a wrapper-nested #toc_container (so it also outranks .itemize (1,1,0); a
theoretical LaTeX-in-TOC would be zeroed too — accepted over a :not(.itemize) carve-out).

Also address review follow-ups: correct the clamp wording (the reserved indent is
clamped to 20em, the per-level default step adds on top); widen the no-bare-ul/ol
tripwire to catch post-comma and combinator forms; regenerate the listsStyles snapshot;
update the pr-spec.
… test fixtures

Attribute marker padding through one shared registry (openTokens + allListTokens)
seeded in ListOpen and threaded into ListItems/processListChildToken, so all list
forms resolve identically — the block loop (own-line nested), the same-line form,
and the fully single-line / in-cell form (which the block Lists rule bails on and
ListOpen now handles, resolving on its single-line early-return). A wide nested
marker no longer inflates the outer list regardless of how the list is written,
and a single-line / in-cell list reserves for its marker too.

Review follow-ups: remove the dead `padding` plumbing (ListItemsResult /
ListInlineContext field, fallback branches, ListOpen local); make openTokens /
allListTokens required (no silent-no-op default); fill skipped depth before the
ancestor sum and clamp depth >= 0 in resolveListPadding (no RangeError / NaN);
fix the stale "for top-level lists" render comments; note the inline-opened
*_list_open prentLevel (0 -> depth) in Migration.

Convert the rendered-HTML tests to full-HTML fixtures (catch visual regression the
way the _data-driven tests do): marker padding, B2 nesting, empty-<>, \item
detection and footnote-terminator cases -> tests/_data/_lists/_data.js; caption
numbering -> _data/_captions/_data.js; "item"-word footnote body ->
_data/_footnotes_latex/_data-footnotetext.js. Keep the non-renderable tests
(display-width units, config-varying renders, env-state, the Arial glyph-width
invariant, resolveListPadding unit, the scaling test) as targeted assertions.
Changelog: cut spec-level internals (per-glyph constants, px measurements, the
3.0.1 over-reservation percentages, internal probe/env mechanism) from the 3.1.0
list-fix bullets, keeping what changed plus the migration — matching the concise
style of earlier entries.

pr-spec: rewrite the Testing section, which described tests that were moved into
full-HTML fixtures (the list-rendering, caption and footnote-body cases), to match
the actual layout; drop an intermediate example value from a Non-Goal.
Follow-up fixes on the 3.1.0 list work, addressing review findings.

Table-cell export:
- A list written entirely on one line inside a table cell unwraps its item
  body to bare inline tokens (not an `inline` wrapper), which the cell renderer
  only sent to HTML — so table-markdown/tsv/csv/smoothed dropped the body
  (`\item[x] d` exported as `x `). The leaf branch now collects the whole
  consecutive run through renderTableCellContent, matching the multi-line form.
- A markdown link in a `tabular` cell skipped two tokens too many, leaving its
  `</a>` unemitted and absorbing following siblings. It now walks to its own
  link_close by depth. Pre-existing; unrelated to lists.

Speculative-parse hardening:
- Caption env (caption/captionPos/captionIsLabelFormatEmpty/
  captionIsSingleLineCheck) written by nested float rules is now snapshotted and
  restored on a non-committing exit, alongside the caption counters; closed by
  audit (no reproduced defect). Shared snapshotEnvKeys/restoreEnvKeys helpers.
- openTokens pop on an inline `\end` is now by token identity, so the block and
  inline paths can't pop the same list twice.
- Silent Lists probe result is memoized per state (invalidated on state.src
  reassignment), so paragraph/footnote terminator scans don't re-run the whole
  speculative parse per line. Output is byte-identical with and without the memo.

Cleanups:
- resolveEnabledRuleFn resolves the single list terminator without a Set/array.
- resolveListPadding uses a local instead of the vestigial token.indentEm.
- Merge duplicate consts imports.

Tests: single-line list in a cell (table-markdown/tsv), link-in-cell, the
padding-style sanitization guard, and silent-probe memo isolation. Docs:
changelog + specs (clamp wording, margin range/font, migration notes).
Widen the speculative-parse env rollback beyond the caption keys. A list body
parsed speculatively (silent probe / aborted parse) runs nested float and
tabular rules that write more shared-env keys than caption alone: begin-table
also writes envType/align/alignEnvBlock/number/type, and begin-tabular writes
isInline/subTabular/tabulare. A silent probe leaked all of these.

- Rename LIST_SPECULATIVE_CAPTION_ENV_KEYS -> LIST_SPECULATIVE_ENV_KEYS and add
  the float/tabular keys, so the set can't drift from its write sites. Still
  restored only on a non-committing exit; a committed float keeps them, as it
  does without a list (pre-existing).

Tests:
- _parse-isolation: a silent Lists probe over a body holding a figure / an
  unclosed table leaves env untouched (full key+value snapshot) — fails without
  the widened set.
- _list-marker-padding: the default-indent overflow threshold (narrow glyphs
  straddle it).
- _lists fixtures: sibling sublists resolve independently in either order (only
  the wide one reserves).

Docs: spec fix 3 lists every key and its write site; changelog/migration note
the committed-float persistence and the concrete table-markdown/marker diffs.
Three behaviour-preserving optimizations for list-heavy parses, plus docs.

- textReserveEm: measure ASCII (the common case) through a precomputed
  glyph-class lookup table instead of running the class regexes per char. The
  table is built from the same regexes so it can't drift, and iteration moves to
  code units with an explicit surrogate skip (astral chars still count once).
  Verified equal to the old path across ASCII/combining/CJK/astral/surrogates.
- restoreEnvKeys: set previously-absent keys to `undefined` instead of
  `delete`-ing them; `delete` moved the shared parser `env` into dictionary mode
  for the rest of the parse. All readers test the value, so behaviour is
  unchanged — but a consumer inspecting its own `env` after a parse now sees the
  keys present with value `undefined` (documented in changelog/spec/migration).
- safeAssignToken: hoist its skip Set to a module constant (was rebuilt per
  flushed token). Kept the constant at the top of the file so the exported
  function keeps its JSDoc in the generated .d.ts.

Tests updated to assert env leakage at the value level. Docs note the perf
change and the consumer-visible `undefined`-vs-absent env difference.
Docs only; no code change.

- changelog: re-measure the parse-speed bullet and drop the earlier overclaim.
  The memo helps where a terminator scan re-probes the same line, so the win is
  on repeated paragraph + list + footnote units without blank separators
  (200 units 47 -> 24 ms, 400 units 152 -> 43 ms, now linear); a plain
  list-heavy document is roughly unchanged (4119 lines: 60 -> 58 ms). Keep the
  env-rollback migration note.
- spec: state the memo measurement as an on/off isolation (34.4 -> 20.4 ms on
  the re-probed shape; no effect on blank-separated lists), drop the
  unverifiable "33-shape matrix" phrase, and replace "separate ticket" with
  "out of scope here". Refresh the Testing section for the tests that landed
  (sibling sublists, the default-indent threshold, the padding-style guard,
  the figure/table env-isolation cases).
- textReserveEm: classify non-ASCII BMP letters by case (uppercase extra-wide,
  the rest wide) instead of counting them as narrow, so uppercase Cyrillic/
  Greek/accented-Latin markers are no longer clipped by the item text; widen the
  zero-width set to the real combining-mark blocks (a decomposed accent now
  costs only its base glyph); cache the class per code point. ASCII stays on the
  lookup table. Behaviour for ASCII is unchanged.
- ItemsListPush: split a line on a real \item (LATEX_ITEM_SPLIT_RE), so a mid-
  line \itemsep no longer starts a spurious item and drops the space before it.
- resolveListPadding: carry a prefix sum instead of re-reducing the ancestor
  indents per level (equivalent output, O(1) per token).
- snapshotEnvForInline: build the snapshot by copying wanted keys instead of
  spread-then-delete (delete dropped env into dictionary mode); carry symbol
  keys through.
- list-state: warn once per parse for the depth-desync diagnostics, so a silent
  probe can't flood a consumer's log.
- styles-lists: hold the in-content .table-of-contents list at padding 0 too.

Docs: changelog + spec cover the by-case width model (with the measured
under/over-reserve trade-offs), the item-split anchoring, and the perf notes.
Fixture comment corrected (astral emoji land in the wide class, not normal).
- textReserveEm/tokenMarkerWidth: measure code_inline and \texttt{…} markers at
  a flat 0.62em/char (monospace advance) instead of the proportional glyph
  classes, which underreserved narrow chars in a `<code>` face; cache only the
  non-ASCII case test (a Map, covering astral too) rather than a 64 KB table;
  a lone surrogate reserves 0.
- Lists probe memo: include the line's content offset (bMarks+tShift) and
  blkIndent in the key, so a blockquote — which shifts those for the same line
  numbers on one state — can't return a stale answer.
- processListChildToken: on the inline path, pop the shared registry only when
  the top is a list-open of the matching kind, mirroring the block path's
  identity check, so an unpaired close can't misattribute later markers.
- list-state: report each distinct list-depth diagnostic once per parse, capped
  at five, instead of one blanket message — a different depth is new info.

Docs: spec/changelog cover the monospace width model and the probe-key inputs.
The icon demo pages embed a copy of the stylesheet, and 2eb05a7 updated the
list and code-block rules in it. That was pointless: none of the eight pages
contains a <ul>, <ol> or <pre>, so every edited rule matches nothing there.
Restore all five files to their master state.
- resolveListPadding: round the reserve up (Math.ceil), never to nearest, so a
  marker's indent can't land a hundredth below what it needs.
- snapshotEnvForInline: drop undefined-valued LIST_SPECULATIVE_ENV_KEYS from the
  snapshot so replaying it can't clear a key that went live after the rollback
  (reproduced: env.align ended undefined instead of "center" when a list-with-
  tabular preceded \begin{center}). The begin-tabular trio (isInline/subTabular/
  tabulare) is exempt — the tsv/csv export needs their undefined to reach replay.
- tokenMarkerWidth: count monospace cells (code points, combining marks 0) for
  code_inline/\texttt markers instead of UTF-16 length; drop code_inline from
  TEXT_LIKE_TYPES since the monospace branch handles it.
- list-state: warn once per distinct case, no separate cap.
- README: note that lib/ and es5/ are committed build artifacts (build, don't
  compile, before committing) and how to check the tree.

Docs: spec/changelog updated — the env-replay fix (with the reproduced case),
the ceil'd fixture value, the by-case width model, and the release bundling.
- table-markdown getMdLink: build a link label by walking to the matching
  link_close instead of reading only the token after link_open, so a formatted
  label survives the export (`[**b** x](url)` was `[](url)`). Per inner token:
  text with `]` re-escaped, code_inline/smiles with their markers, else the
  getMdForChild marker or its content (keeps image alt / inline-math LaTeX).
- render-table-cell-content: also feed the consumed link tokens to the smoothed
  accumulator, so pptx no longer keeps an opening `<a>` with no text or close.
- Lists probe memo: key on the first line's end offset (eMarks) instead of
  blkIndent — the rule never reads blkIndent, and eMarks pins the content more
  tightly; the geometry still distinguishes a blockquote's shifted lines.
- render-tabular: only build the leaf run when an export (tsv/csv/md/smoothed)
  is requested; an HTML-only render was doing it for nothing.
- block-rule: fold the single list terminator back into resolveEnabledRuleFns
  (one resolver, not two).

Docs: changelog/spec cover the link-label composition (with the image/math/
smiles/escaping behaviour) and the probe-key inputs (blkIndent excluded, tested).
Marker width (display-width.ts):
- Extend isWideChar to the astral Wide blocks, so an emoji or rare-ideograph
  marker reserves the ideograph width (\item[😀😀] now indents 3.03em, was no
  custom indent).
- Rename isZeroWidthCombining -> isZeroWidthChar and add the joiners/bidi marks
  (U+200B–200F) and variation selectors (U+FE00–0F): they compose their
  neighbours rather than taking a cell, so an emoji ZWJ sequence no longer
  reserves a full glyph per joiner (👨‍👩‍👧 5.40em -> 3.60em, ❤️ 1.80em -> 0.90em).
- Cache the cased class in a fixed Uint8Array over the dense range instead of a
  growing Map (no lifecycle owner; recomputed above the bound).

Closer-lookahead perf (quadratic -> flat on malformed input):
- Lists and findFirstCompleteListEnv answer false for an unclosed env without
  parsing to EOF, by checking for a closing \end first (cached per state).
- BeginTabular does the same for an unclosed \begin{tabular} — same probe-per-
  line pattern, much higher ceiling.
Both lookaheads are exact (sweep regexes built from the parser's own patterns);
guarded by scaling tests.

Cell export: share smoothedFor between the main and link loops; thread the real
isSubTable into the leaf-run render instead of hardcoding it.

Docs: changelog/spec cover the width model, the closer lookahead (with the
honest residual ceiling from newTheoremBlock/lheading), and the tabular fix.
- render-tabular: a paragraph boundary (paragraph_open/close) ends a cell leaf
  run — the `block` flag can't gate it, since createBufferedState marks the
  unwrapped inline leaves as block too. Keeps the export re-render to pure inline
  content; pinned by a test that the second render doesn't disturb module state.
- render-table-cell-content: the link loop feeds `content` the forPptx-smoothed
  form, matching the main loop.
- env-transient: copy only enumerable symbol keys into the inline snapshot, the
  same reach the old `{...env}` spread had.
- display-width: fold \t\r\n into the narrow class — they collapse to a space
  when rendered.
- Comments: the tabular closer sweep uses the wider unanchored regex on purpose
  (over-reporting a closer only keeps the bail conservative); the emitted padding
  matches PADDING_EM_RE by construction.

Docs: changelog condensed to one line per fix with a single consolidated
Migration section; no content dropped.
…width

- Extract the three per-state "last match offset in src" caches (footnote token
  sweep, tabular closer, list closer) into common/src-pos-cache.ts
  (lastMatchPosCached). Same behaviour — Symbol-keyed per state, invalidated on
  state.src reassignment, empty-match guarded — with one implementation.
- monoCells: an East-Asian Wide glyph takes two cells in a monospace face, so a
  code_inline/\texttt marker over CJK reserves double-width (`\texttt{漢字}`
  → 2.48em, was 1.24em).

Docs: spec covers the mono double-width, why the leaf-run predicate is a
deny-list (lstlisting is a block leaf that must stay in), and the depth-rollback
asymmetry; changelog footnotes-list note refined.
…ation space

- table-markdown getMdLink: emit the link destination through mdHref — the
  angle-bracket form (<…>) when it holds whitespace, </>, or unbalanced parens
  (a bare (…) destination truncates at the first unmatched ')'), bare otherwise;
  and re-escape `]` in a raw html_inline label node (arrives unescaped).
- render-tabular: when a cell's inline token is followed by an unwrapped leaf
  run (a continuation line whose line break is a separate, dropped token),
  restore the single space between them so the Markdown export reads "a b", not
  "ab".

Docs: getMdLink spec covers the html_inline label branch and the mdHref
destination handling.
Rewrite pr-specs/2026-07-list-rendering-robustness.md to the monorepo spec
template (Context / Goal / Non-Goals / Current / Desired Behavior / Breaking
change / Testing / Done When), one line per fix. Drops the measurements,
profiling and internal-mechanism detail that had grown it to ~55 KB; keeps every
consumer-facing breaking change and non-goal. ~55 KB -> ~10 KB, no content that
a consumer needs is lost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant