Skip to content

Phase 2: chunk .rxp literals to ≤63 bytes for bytecode compilation - #5

Closed
mgrossmann wants to merge 3 commits into
mainfrom
feature/rxp-literal-chunking
Closed

Phase 2: chunk .rxp literals to ≤63 bytes for bytecode compilation#5
mgrossmann wants to merge 3 commits into
mainfrom
feature/rxp-literal-chunking

Conversation

@mgrossmann

Copy link
Copy Markdown
Contributor

Closes #4. First Phase 2 item ("literals", spec §5).

Problem

The .rxp transpiler emitted one REXX string literal per HTML run. rexx370's
bytecode compiler caps each stored constant at 63 bytes (IRXBC_STR_MAX), so
ordinary HTML lines returned IRXBC_ERR_STRTOOLONG and the engine fell back to
the token-walk interpreter — a doomed compile attempt paid on every request.

Change

line_close_lit (src/rxptrans.c) now splits each pending literal into chunks of
at most 63 value bytes, each a quoted string joined by the existing ||.

  • Value bytes, not source bytes: an embedded ' is one value byte (written
    doubled), so the stored constant never exceeds the limit.
  • Non-breaking: a literal ≤63 bytes emits a single part, identical to before;
    all pre-existing tests are unchanged. Only >63-byte literals split, and
    'ab'||'c' concatenates to the same bytes.
  • Transpiler-only (.rxp); .rexx direct execution is untouched.

Supersedes the spec's _LIT.-stem-via-vpool alternative, which is blocked on the
current rexx370 (IRXEXCOM is not an installed load module).

Verification

  • make test-host: 17/17 (5 new: split boundaries 63/64/130, boundary
    quote-doubling, expression composition).
  • End-to-end with the real transpiler on a realistic page (long HTML lines),
    before vs. after, same rendered bytes:
    • old output → [bc] exec=0 fallback=1 (interpreter fallback)
    • new output → [bc] exec=1 fallback=0 (compiles + runs bytecode)
  • Host measurement: ~38% faster per request vs. the fallback path. Absolute
    µs are host-only; the ratio is the transferable part. Details and the full
    four-cell matrix in doc/phase2-status.md.

Docs

Spec bumped to 1.2 (§5 rewritten to the real mechanism, §10 phase plan
re-scoped into unblocked vs. rexx370-blocked). doc/phase2-status.md tracks the
running Phase 2 state.

Transpiled .rxp pages emitted one REXX string literal per HTML run. rexx370's
bytecode compiler caps each constant at 63 bytes (IRXBC_STR_MAX), so ordinary
HTML lines forced IRXBC_ERR_STRTOOLONG and a fallback to the token-walk
interpreter on nearly every page -- a doomed compile paid on every request.

line_close_lit now splits each literal into <=63 value-byte chunks joined by
||, keeping every constant compilable. Chunk boundaries are counted in value
bytes (an embedded quote is one byte, written doubled); output is byte-identical
to one long literal, and literals under 63 bytes are unchanged. Transpiler-only;
.rexx direct execution is unaffected.

Measured on a realistic page: the same transpiled output goes from interpreter
fallback to a successful bytecode compile, ~38% faster per request (host). Adds
5 tests (split boundaries, boundary quote-doubling, expression composition);
17/17 host tests pass. Spec bumped to 1.2; phase2-status.md tracks state.

Closes #4
…tion

Adversarial stress testing found that chunking a page with >512 distinct
63-byte constants (~32 KB of distinct content, cumulative across the page)
overflows rexx370's program-wide BCOM_MAX_CONSTS table. The overflow returns
IRXBC_ERR_STOR, which bc_err_is_fallback() does not classify as fallback-
eligible, so irx_exec_run returns it fatally -- a 500 where the unchunked page
rendered via interpreter fallback. Confirmed on a realistic 50 KB page.

The proper fix is upstream: classify the compile-time capacity error as
fallback-eligible in rexx370 (same pattern as the 63-byte STRTOOLONG fix).
PR #5 is not strictly safe to merge until that is in place.
@mgrossmann
mgrossmann marked this pull request as draft July 3, 2026 16:08
@mgrossmann

Copy link
Copy Markdown
Contributor Author

Blocked on mvslovers/rexx370#212. Post-merge stress testing found that chunking a page with >512 distinct 63-byte constants (~>32 KB distinct content, cumulative — the program-wide BCOM_MAX_CONSTS table) overflows and returns IRXBC_ERR_STOR, which bc_err_is_fallback() treats as fatal. Result: a realistic 50 KB page that renders via interpreter fallback today would return a 500 once chunked. This change is a strict win only after rexx370 classifies the compile-time table overflow as fallback-eligible (rexx370#212). Converting to draft until that lands. Details: doc/phase2-status.md.

@mgrossmann

Copy link
Copy Markdown
Contributor Author

Superseded by rexx370 and closing. rexx370 moved long-literal handling into the bytecode compiler itself: #208 (e3d187c on main — chunk long string literals in the compiler instead of falling back) plus #212/#213 (compile-time fixed-table overflow now returns IRXBC_ERR_CAPACITY and falls back gracefully).

Verified host-side against rexx370 main: the unchunked transpiler output (no PR #5) now behaves identically to the chunked output — small pages compile natively (exec=1, byte-identical to the interpreter), and a 50 KB page falls back gracefully and renders (no more fatal 500). So transpiler-level chunking is redundant, and the concern belongs in the engine, not in every caller. The #4 investigation drove the fix to the right layer.

Closing; src/rxptrans.c stays simple (one literal per line).

@mgrossmann mgrossmann closed this Jul 3, 2026
@mgrossmann
mgrossmann deleted the feature/rxp-literal-chunking branch July 3, 2026 18:06
mgrossmann added a commit that referenced this pull request Jul 10, 2026
…nspiler

Records the resolution of the Phase 2 "literals" item. rexx370 now handles long
SAY literals in the bytecode compiler itself (#208 compile-time chunking) and
falls back gracefully on fixed-table overflow (#212/#213, IRXBC_ERR_CAPACITY),
so transpiled .rxp pages compile with no transpiler change. The transpiler-side
chunking attempt (issue #4 / PR #5) was closed as superseded.

- New doc/phase2-status.md: living Phase 2 tracker (scope, decisions, evidence).
- Spec bumped to 1.2: §5 and §10 updated; transpiler stays one-literal-per-line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 2: .rxp transpiler literal-chunking (make server pages bytecode-compilable)

1 participant