Skip to content

Add an html control arm to the bench - #4

Open
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1787848165-html-protocol-arm
Open

Add an html control arm to the bench#4
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1787848165-html-protocol-arm

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

Answers "can we run the bench with HTML generation": yes — node run.ts html works alongside the three protocols, and this adds the fourth protocol folder plus the harness wiring (run.ts format, score.ts Fmt/regex/rank, token and cost tools, README). Committed results and raws are untouched, so the parity test still reproduces them byte for byte.

The interesting part is what the arm can and cannot measure, and it is documented in the README rather than left for the reader to discover:

  • HTML has no SDK verdict to borrow. Its shipped renderer is the browser's HTML5 parser, which is error-tolerant by specification: parse5 reports no error for <section><h2>Hi<p>x</p> (unclosed) or a stray </div>, and unknown tags render as inert inline boxes. So evaluate runs parse5 for the spec-level errors it does report, then adds a tagBalanceErrors scan of the tags the model actually wrote — unclosed containers and stray end tags, skipping void and optional-end-tag elements — because that is the only way to see markup a browser silently repaired.

  • Even with that, completion saturates. Probe on 5 briefs, one per band (uncommitted, BENCH_ONLY=b1-invoice,b2-sales,b3-crm,b4-bank,b5-exec, BENCH_PROVIDER=openai):

    Model openui json-render html
    gpt-5.6-luna, 4 reps 40/40 (plain and tailwind)
    gpt-5-nano, 2 reps 2/10 0/10 10/10

    A weak model that fails 8 of 10 OpenUI runs and all 10 json-render runs writes 10/10 valid HTML, because there is no closed vocabulary to violate. Read the arm as a floor, not as a competitor on the reliability axis.

  • Where it is informative is length. Same model, same briefs, mean output: 876 tokens OpenUI Lang → 1,413 plain HTML (1.6x) → 2,796 Tailwind-styled HTML (3.2x). BENCH_HTML_STYLE=plain|tailwind selects the condition, since styling dominates output size and "plain semantic markup" is not what models emit unprompted.

Prompt shape follows the other protocols (rules + the same two worked screens, here in HTML). REQUIRED_ATTRS is deliberately thin — img@alt, a@href, standalone label@for — so the gate stays a "required prop" check and does not drift into being an accessibility linter; an earlier draft included option@value and flagged 77 perfectly valid options.

New dependency: parse5 8.0.1, the reference HTML5 parser (the one jsdom ships).

Link to Devin session: https://app.devin.ai/sessions/b774bc8429a54ca8b609c1e8ae86e576
Open in Devin Desktop: https://app.devin.ai/desktop/session/b774bc8429a54ca8b609c1e8ae86e576?variant=devin
Requested by: @zahlekhan

No catalog and no SDK: the verdict is parse5 plus the shared completeness layer, with a tag-balance scan because the HTML5 parser recovers from unclosed and stray tags silently.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant