Add an html control arm to the bench - #4
Open
devin-ai-integration[bot] wants to merge 1 commit into
Open
Conversation
No catalog and no SDK: the verdict is parse5 plus the shared completeness layer, with a tag-balance scan because the HTML5 parser recovers from unclosed and stray tags silently. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Answers "can we run the bench with HTML generation": yes —
node run.ts htmlworks alongside the three protocols, and this adds the fourth protocol folder plus the harness wiring (run.tsformat,score.tsFmt/regex/rank, token and cost tools, README). Committed results and raws are untouched, so the parity test still reproduces them byte for byte.The interesting part is what the arm can and cannot measure, and it is documented in the README rather than left for the reader to discover:
HTML has no SDK verdict to borrow. Its shipped renderer is the browser's HTML5 parser, which is error-tolerant by specification:
parse5reports no error for<section><h2>Hi<p>x</p>(unclosed) or a stray</div>, and unknown tags render as inert inline boxes. Soevaluateruns parse5 for the spec-level errors it does report, then adds atagBalanceErrorsscan of the tags the model actually wrote — unclosed containers and stray end tags, skipping void and optional-end-tag elements — because that is the only way to see markup a browser silently repaired.Even with that, completion saturates. Probe on 5 briefs, one per band (uncommitted,
BENCH_ONLY=b1-invoice,b2-sales,b3-crm,b4-bank,b5-exec,BENCH_PROVIDER=openai):gpt-5.6-luna, 4 repsgpt-5-nano, 2 repsA weak model that fails 8 of 10 OpenUI runs and all 10 json-render runs writes 10/10 valid HTML, because there is no closed vocabulary to violate. Read the arm as a floor, not as a competitor on the reliability axis.
Where it is informative is length. Same model, same briefs, mean output: 876 tokens OpenUI Lang → 1,413 plain HTML (1.6x) → 2,796 Tailwind-styled HTML (3.2x).
BENCH_HTML_STYLE=plain|tailwindselects the condition, since styling dominates output size and "plain semantic markup" is not what models emit unprompted.Prompt shape follows the other protocols (rules + the same two worked screens, here in HTML).
REQUIRED_ATTRSis deliberately thin —img@alt,a@href, standalonelabel@for— so the gate stays a "required prop" check and does not drift into being an accessibility linter; an earlier draft includedoption@valueand flagged 77 perfectly valid options.New dependency:
parse58.0.1, the reference HTML5 parser (the one jsdom ships).Link to Devin session: https://app.devin.ai/sessions/b774bc8429a54ca8b609c1e8ae86e576
Open in Devin Desktop: https://app.devin.ai/desktop/session/b774bc8429a54ca8b609c1e8ae86e576?variant=devin
Requested by: @zahlekhan