[Bug]: Generated interactive scenes can render but remain unusable after script/runtime failures

[Qnh233]

Summary

Generated interactive scenes can render correctly while being unusable because the generated JavaScript fails either:

1. before execution (syntax / parse-time failure), or
2. during interaction (runtime exception in generated state/data logic).

The current iframe runtime already captures runtime errors, but a broken interactive scene can still reach the user and remain effectively dead.

This is broader than the syntax-only case previously tracked in #1582 / #1583.

Real reproductions

I reproduced two independent real generated artifacts in a sandboxed browser path.

Case A: Flink realtime simulator — parse-time failure

The page visually renders a complete Flink simulation UI with sliders, Watermark / Checkpoint indicators, and a Start Simulation button.

The generated main script contains:

state counts = new Array(10).fill(0);

Browser result:

SyntaxError: Unexpected identifier 'counts'

Because the entire classic script fails to parse, handleMainButton() is never defined. Clicking the visible button then produces:

ReferenceError: handleMainButton is not defined

User-facing symptom: the page looks complete, but clicking Start Simulation does nothing.

Case B: data-pipeline game — runtime initialization failure

A separate generated game has syntactically valid JavaScript and renders normally.

After clicking Start Challenge, generated data references _REST API_, while the component registry contains REST API. The lookup returns undefined, then generated code reads comp.name and throws:

TypeError: Cannot read properties of undefined (reading 'name')

The interaction starts, partially initializes, then aborts.

Why this matters

These two artifacts have the same user-facing failure mode but different causes:

render succeeds
  -> control is visible
  -> generated JS is unusable
  -> interaction appears dead / partially dead

A syntax-only fix addresses Case A but not Case B.

Current behavior

• OpenMAIC already captures iframe runtime errors into the scene runtime error store.
• Current main does not appear to consume those errors for recovery.
• fix(playback): queue widget messages until iframe ready #1532 already fixed the separate iframe-ready message queue race, so this issue is not intended to duplicate that work.
• [Research] Interactive replay: deterministic HTML contract inspector #1053 / [Task] Interactive actions: consume runtime visibility contract #1055 cover deterministic replay / runtime visibility contracts and are related, but this report is specifically about preventing or recovering generated interactive scenes that are already known to be broken.

Proposed reliability boundary

I locally validated a two-layer approach:

Layer 1 — generation-time classic inline JS syntax preflight

Before persisting an interactive widget:

• compile-check classic inline <script> bodies without executing them;
• skip data scripts such as application/json;
• skip external src scripts;
• skip type="module" scripts;
• route failures through the existing invalid-model-output retry path.

The real Flink artifact is rejected before persistence with:

script #9: Unexpected identifier 'counts'

Layer 2 — runtime-error-informed targeted repair

For syntax-valid pages that fail during interaction:

• reuse the existing iframe runtime-error capture;
• provide the current HTML plus concrete fatal runtime errors as repair context;
• regenerate only the interactive HTML;
• keep other scene data unchanged;
• run the repaired output through the same syntax preflight.

With the full runtime-broken game artifact, a local targeted repair produced a page that:

• passed syntax preflight;
• successfully started after clicking;
• rendered all expected component cards;
• emitted no page/runtime errors.

I intentionally kept the repair trigger user-initiated during local validation so normal playback does not automatically spend model quota.

Validation performed locally

Relevant regression coverage passed:

• iframe / picker / keep-alive / sandbox / repair-path tests: 42/42
• generation tests excluding the existing Windows-only assets path assertion: 149/149
• selected PBL / quiz / vocational routing regressions: 39/39
• generation package build: pass
• generation package typecheck: pass
• root tsc --noEmit: pass
• focused ESLint: pass
• git diff --check: pass

Syntax preflight cost on the two real artifacts was sub-millisecond (~0.07–0.20 ms per validation).

Architecture question

The generation-time syntax preflight is a narrow bug-fix boundary.

For runtime recovery, I would like maintainer guidance on the preferred integration point:

• a lightweight runtime failure affordance near the interactive renderer, or
• the existing MAIC Editor / agent editing path (e.g. the current interactive HTML editing plumbing).

I do not want to hard-code the UX/ownership before alignment, especially with #646 evolving the editor/agent architecture.

Related

• [Bug]: Generated interactive HTML with invalid inline JS is persisted as an unresponsive widget #1582 — previous syntax-only issue, closed after discovering the broader runtime failure class
• fix(generation): reject invalid interactive scripts #1583 — previous syntax-only PR, withdrawn for the same reason
• fix(playback): queue widget messages until iframe ready #1532 — merged iframe-ready message queue fix; separate failure mode
• [Research] Interactive replay: deterministic HTML contract inspector #1053 — deterministic interactive replay / contract inspector
• [Task] Interactive actions: consume runtime visibility contract #1055 — runtime visibility contract consumption
• [MAIC Editor] Phase 2 — universal scene-DSL editor + AI editing (MAIC Agent) #646 — MAIC Editor / agent architecture

Expected behavior

A generated interactive scene that is known to be invalid should not be persisted as an apparently functional widget.

If a syntactically valid scene fails at runtime, the captured error should be surfaced into an explicit recovery path rather than leaving a visually complete but unusable interaction.

[wyuc]

Thanks @Qnh233 — two clean reproductions, and the framing is right: render success is not interaction success, and a syntax-only check (#1582 / #1583) can't catch Case B. Accepting this as a bug.

On direction, one constraint first: we don't want to fix this by heuristically rewriting generated JS after the fact (e.g. auto-correcting state → let, or fuzzy-matching _REST API_ to REST API). That kind of repair layer grows forever and silently masks what the model actually produced. Preferred shapes, roughly in order:

1. Validate at generation time, then retry or fail loudly. Parse-time checks cover Case A; a short headless smoke run (load, click the primary control, collect runtime errors) would cover the common Case B shape. A failed check triggers regeneration of that scene, and if it still fails, the scene is marked broken rather than shipped as a dead widget.
2. Consume the runtime errors the iframe already captures. Surface them to the learner/teacher ("this interactive failed to run") instead of leaving a silent, dead control.
3. Repair through the agent, not through code. This already works today: Pro mode's agent can read the scene source and patch the interactive HTML in place (patch_stage with str_replace on /content/html), so a broken scene can be fixed by asking the agent, with the captured error as context. Wiring the captured runtime errors into that flow — e.g. a "fix this interactive" entry that hands the error to the agent — would turn detection into recovery without any hand-written repair heuristics.

If you'd like to take one of these on, (1) or the (2)+(3) wiring are both welcome — please sketch the approach here first, since the smoke-run cost and where it sits in the pipeline are worth agreeing on before code.

[gurpreetlucky]

New reproduction: diagram widget — convertLatexDelimiters regex mangles JS template literals (and the script-protection regex doesn't catch truncated <script> tags)

Adding a third independent reproduction for #1622 Case A — parse-time failure of generated interactive JS that the existing findInteractiveScriptSyntaxFailure validator cannot catch, with a different root cause than the Flink-simulator example already on the thread.

Symptom (user-facing)

In a production classroom built on the v1.1.0 pin, generating a 2-scene interactive lesson ("Amazing Animal World") produced:

• Scene 1 (slide, "Welcome to the Animal Kingdom") — renders correctly, 8/8 audio assets persist, all good.

• Scene 2 (interactive, widgetType: diagram, 7 nodes / 6 edges, "Classify the Animals") — the diagram frame renders but every node transform and every edge path is silently broken, and clicking the Next button does nothing. Browser console:

  Uncaught SyntaxError: Unexpected identifier 'data' (about:srcdoc:737)
  

  (line 737 is the truncation point in the iframe srcdoc, not the storage line; see below for the line-number offset math.)

Root cause — two compounding bugs in packages/@openmaic/generation/src/interactive-post-processor.ts

Bug 1 — the LaTeX regex at line 50 mangles JS template-literal interpolation

// line 48-51, convertLatexDelimiters in interactive-post-processor.ts
// Convert inline math: $...$ -> \(...\)
// Use non-greedy match and exclude newlines to avoid false positives
processed = processed.replace(/\$([^$\n]+?)\$/g, '\\($1\\)');

This regex is intended to convert LaTeX inline math $E=mc^2$ into KaTeX inline math \(E=mc^2\). The character class [^$\n]+? is non-greedy and excludes $ and newlines, so the regex is happy to span across a JS template-literal ${...} interpolation boundary and capture content between two distinct ${...} markers as if they were a single LaTeX span.

When the model writes valid JS like:

g.setAttribute('transform', `translate(${pos.x}, ${pos.y})`);

the regex matches $, then [^$\n]+? lazily consumes {pos.x}, (no $, no newline), then closes on the next $, producing:

g.setAttribute('transform', `translate(\({pos.x}, \){pos.y})`);

That mangled string is exactly what was persisted to the database. Verified by querying the document_scenes.data->'content'->>'html' column for this classroom — both setAttribute('transform', ...) and setAttribute('d', ...) lines for nodes and edges carry the broken form. 7 node transforms + 6 edge paths all corrupted, so the SVG renders as 13 boxes without connectors.

Bug 2 — the script-protection regex at line 40 requires a closing </script>

// line 40-44, convertLatexDelimiters in interactive-post-processor.ts
let processed = html.replace(/<script[^>]*>[\s\S]*?<\/script>/gi, (match) => {
  scriptBlocks.push(match);
  return `__SCRIPT_BLOCK_${scriptBlocks.length - 1}__`;
});

[\s\S]*? (non-greedy) followed by <\/script> only matches well-formed <script>...</script> blocks. If the model's output is truncated mid-script (the third <script> in this widget ends with const nextBtn = document and no closing tag — length: 14502 bytes, very close to a model output token limit), the protection never fires and the broken tail of the script is fed straight into the LaTeX regex.

Why findInteractiveScriptSyntaxFailure doesn't catch it

packages/@openmaic/generation/src/scene-generator.ts line 1339:

const scriptSyntaxFailure = findInteractiveScriptSyntaxFailure(html);
if (scriptSyntaxFailure) {
  // log + onFailure + return null
}

interactive-script-validator.ts lines 122-129 use node:vm's Script constructor:

function classicScriptSyntaxError(source: string): string | null {
  try {
    new Script(source);
    return null;
  } catch (error) {
    return error instanceof Error ? error.message : String(error);
  }
}

Reproduction in Node v22.23.3 (V8 v12.4.254.21, our container's runtime) shows the broken template literal parses cleanly:

$ node -e 'const {Script}=require("node:vm"); \
  new Script("`translate(\\\\({pos.x}, \\\\){pos.y})`");'
# → no throw, VALID

V8 v12.4 silently treats \( in a template literal as the literal two characters \ + (, so the validator's compile-check passes. A stricter engine or a newer browser V8 throws on the same input — that's where the user's Unexpected identifier 'data' ultimately surfaces. So validator and runtime disagree on the same script, and findInteractiveScriptSyntaxFailure returns null even though the resulting JavaScript is broken.

A truncated script (const nextBtn = document at EOF) similarly compiles — ASI produces const nextBtn; document; and V8 accepts both as separate statements. So the validator also returns null for the truncation. The breakage only surfaces at runtime when the click handler is missing.

Evidence — exact stored bytes

$ psql -c "SELECT substring(data->'content'->>'html' FROM 11700 FOR 250) \
    FROM document_scenes WHERE stage_id = 'E2zudOaoeB' AND scene_order = 2;"

  g.setAttribute('transform', `translate(\({pos.x}, \){pos.y})`);
  ...
  path.setAttribute('d', `M \({pts.sx} \){pts.sy} L \({pts.ex} \){pts.ey})`);

The \( and \) artifacts are present on every node transform and every edge path in the persisted widget HTML. Pre-conversion, the model's raw response carried translate(${pos.x}, ${pos.y}) — the LaTeX post-processor is what introduced the \ escapes.

The truncation at 14502 bytes lands mid-statement (const nextBtn = document) with the <script> tag still open. After patchHtmlForIframe injects three shim scripts (STORAGE_SHIM 22 lines + ERROR_CAPTURE_SHIM 45 lines + ELEMENT_PICKER_SHIM 260 lines + iframe css ≈ 10 lines ≈ 337 lines total) into the iframe <head>, the truncation point moves from stored-line 397 to roughly patched-line 734, which matches the user's reported about:srcdoc:737.

Proposed fix (minimal, scoped to interactive-post-processor.ts)

1. Make the script-protection regex tolerate unclosed <script> blocks so the LaTeX transforms can never see script content, truncated or not:

   // before
   let processed = html.replace(/<script[^>]*>[\s\S]*?<\/script>/gi, (match) => {
   // after — `$` alternation accepts a <script> that runs to EOF unclosed
   let processed = html.replace(/<script\b[^>]*>(?:[\s\S]*?<\/script>|$)/gi, (match) => {

   This single-token change means a truncated widget script is protected from the regex pass (and the truncation's downstream runtime breakage is the only remaining symptom, not silent JS corruption).

2. Tighten convertLatexDelimiters's regex to refuse spanning JS template-literal interpolation, even as a defense-in-depth after (1):

   // before
   processed = processed.replace(/\$([^$\n]+?)\$/g, '\\($1\\)');
   // after — break the match on `{` immediately following the opening `$`,
   //          because `${...}` is unambiguously a template-literal interpolation
   processed = processed.replace(/\$(?!\{)([^$\n]+?)(?<!\})\$/g, '\\($1\\)');

   The (?!\{) lookahead on the opening $ skips any ${, and the (?<!\}) lookbehind on the closing $ requires the captured content not to end on } from a closed interpolation. Together they stop the regex from spanning ${...} boundaries.

3. Optional: also reject template-literal sequences containing \\( or \\) outside the small set of valid JS escape sequences — findInteractiveScriptSyntaxFailure already runs V8 over the script, but V8 versions disagree on \( in template literals, so a pattern-level pre-check catches what a strict engine would.

Why this fits under #1622

This is a Case A parse-time failure for a different root cause than the Flink-simulator example. The user-facing symptom is identical: the diagram widget's frame renders, the controls look interactive, but the generated JavaScript cannot run. The validator added for #1582 / #1583 is necessary but not sufficient — it can compile-check the script, but it can't see the regex munging that happens in the post-processor before the script reaches the validator, and it doesn't catch truncated scripts whose bodies still parse.

I'll leave the actual fix to the maintainer's preferred direction (gen-time validate + retry/fail loudly, surface runtime errors, repair via the agent) rather than reworking convertLatexDelimiters heuristically — happy to send a PR if you'd like one once the direction is settled.

[Qnh233]

@wyuc Thanks — I'd like to take the (2) + (3) path first, and keep the MVP as wiring rather than introducing another repair layer.

I checked the current main after the partial #1622 fix. The pieces already seem to line up well:

• InteractiveIframeHost already captures per-scene runtime errors and InteractiveRuntimeErrorBanner surfaces the latest one.
• Pro editing already supports reading the exact persisted scene source and patching an interactive via patch_stage / str_replace on /content/html.
• When the scene HTML changes, the iframe reloads and the current runtime-error state is cleared.

Proposed MVP

1. Add a small “Fix with agent” action to the existing runtime-error banner (next to Dismiss).
2. The action does no heuristic JS rewriting and adds no new repair API.
3. It opens/targets the existing Pro editing conversation for the current course and hands it a small diagnostic context:
   • target sceneId
   • latest captured runtime error
   • instruction to inspect the persisted source and make the smallest necessary repair
4. The agent follows the existing edit path:
   • read_stage ... detail:"source"
   • diagnose from source + captured error
   • patch_stage (prefer a narrow str_replace on /content/html)
   • re-read/verify
5. The updated HTML reloads the iframe; if the repair is correct, the existing error banner naturally disappears on the fresh document.

For the first version, I'd prefer the button to prefill the agent composer rather than auto-send, so the user still explicitly starts the model call and can see exactly what evidence is being handed over. That also keeps quota/cost behavior unchanged until the user presses Send.

So the MVP boundary would intentionally exclude:

• headless smoke-run / generation-pipeline changes from (1)
• handwritten JS repair heuristics
• automatic background repair
• a new repair-specific server/tool API

The main regression I would use is the runtime Case B from this issue: captured TypeError → “Fix with agent” → existing agent reads the scene → patches /content/html → iframe reloads without the same runtime error.

Is this the (2)+(3) wiring you had in mind? In particular, would you prefer the first MVP to prefill + explicit Send as above, or should “Fix with agent” immediately submit the repair request once clicked?

[wyuc]

@Qnh233 Yes, this is the (2)+(3) wiring I had in mind. Please go ahead with the MVP as scoped: a "Fix with agent" action on the existing runtime-error banner that hands the scene ID and the captured error to the existing Pro editing agent. No new repair API, no heuristic rewriting, no background repair.

Prefill + explicit Send. The user sees exactly what is handed over, and no model call (or quota spend) happens on the click itself.

Three requirements for the PR:

1. Only show the action when it can actually work. The banner renders in every playback surface, including classic mode, shared/viewer links and learner views. The Pro editing agent is only available to someone who can edit the course. Render "Fix with agent" only when the current user can edit this stage and the Pro editing conversation is available. Everywhere else, keep today's banner with Dismiss only. Please add a test for the hidden case.
2. Treat the captured error as untrusted evidence. The error text comes from inside the generated page, so the page controls its content. When you prefill the composer, frame it as quoted data rather than instructions: fence it, cap its length (the banner already truncates to 180 chars; something similar is fine), and label it as a runtime error reported by the page. Keep the instruction part fixed and written by us.
3. Regression: your Case B flow is the right one. Captured TypeError → "Fix with agent" → the agent reads the scene source → patch_stage on /content/html → the iframe reloads without the error. Please also cover dismiss and re-trigger: after a successful patch, the banner should not reappear for the same, now stale, error.

Please link this issue from the PR.

@gurpreetlucky thanks for the detailed diagram-widget reproduction. That one is a separate bug in our own post-processor: convertLatexDelimiters damages output that the model produced correctly. It only happens when a <script> is left unclosed (truncated output), but it should still be fixed. A PR would be welcome, scoped to your items (1) (protect unclosed <script> blocks through EOF) and (2) (don't let the inline-math match span ${…}), with tests for both, including a truncated-script fixture. Please leave out (3): a pattern-level pre-check on template-literal escapes is the kind of heuristic repair layer we're avoiding.

Direction (1), a generation-time headless smoke run, is still open if anyone wants to sketch it separately.

[Ethereal49]

Thanks for clarifying the split. I'd like to take the independent post-processor scope for #1622: protect unclosed <script> blocks through EOF and prevent inline-math matching from spanning ${...}, with focused regressions for both, including a truncated-script fixture. I'll leave out the template-literal escape pre-check and the "Fix with agent" UI work. If @gurpreetlucky is not already taking this scope, please assign it to me.

[gurpreetlucky]

Thanks @Ethereal49 , Yes pls proceed from here ..