Repository navigation
test(tui): wait for the run to end before a scenario submits - #118
Merged
Merged
Conversation
The lsp-tools and lsp-diagnostics scenarios sent /diagnostics right after the tool-call line, which renders before the tool runs. While a run is in flight the TUI refuses a submit, so the Enter was dropped and the command stayed in the input box; the assertion then timed out on a frame showing the answered run next to the unexecuted command. Run #298 lost exactly that race while run #297 passed on the identical tree (tree d3fe4d3), which is what makes this a scenario defect and not an LSP regression. Waiting for the stub's final answer — the signal tool-call.scenario and tool-multi.scenario already use — puts the submit after the run, and a short stable absorbs the StreamDone that flips the input back on. The lsp scenarios need a PTY, which this machine's sandbox denies (/dev/ptmx is "operation not permitted"), so the scenario run itself is verified by CI's lsp job; locally only the parse was checked (both files reach the open step). Refs #117
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
tui/testdata/scenarios/lsp-tools.scenarioandtui/testdata/scenarios/lsp-diagnostics.scenario: after the tool-call line, wait for the stub's final answer (stub-final-ok) and a shortstablebefore sending/diagnostics.Closes #117.
Why
The two LSP scenarios sent
/diagnosticsas soon as the tool-call line appeared, but that line renders before the tool runs. The TUI refuses a submit while a run is in flight (tui/update.go:m.status != StatusStreaming && !m.runIsActive()), so the Enter was dropped, the command stayed in the input box, and the assertion timed out on a frame showing the finished answer next to the unexecuted text. Waiting for the model's final answer — the signaltool-call.scenarioandtool-multi.scenarioalready use — puts the submit after the run;stable 200msabsorbs theStreamDonethat flips the input back on.This is a scenario defect, not an LSP regression. Run #297 passed and run #298 failed on byte-identical trees:
lsp-diagnostics.scenariohas the same shape (wait --text "read_file"then the command) and is exposed to the same race even though it did not lose it in #298; it is fixed here too.Verification
Mechanism probe (pretend the run is slow). A stub whose second response sleeps 8s keeps the run active after the tool-call line is on screen:
/diagnosticssent on the tool-call linestub-final-ok+┃ /diagnosticsin the input — byte-identical to run #298's failure framestub-final-ok+stable 200mssaw LSP not available, 15 steps, rc=0lsp-tools.scenariounder the fix: 6/6 pass locally withgopls v0.23.0. The pre-fix content also passed 6/6 — the window is too narrow to hit by repetition here, which is why the probe above exists.go test ./... -count=1(redirectedHOME/GOPATH/GIT_CONFIG_GLOBAL) → 13 packages ok / 0 failed / 0 ignored. The changed files are scenario data, not Go sources, so this suite does not exercise them; it is recorded for completeness.CI's
lspjob (make test-tui-lsp) is the gate; result below.Honest scope
The TUI scenarios need a PTY. Under the default
workspace-writesandbox, macOS Seatbelt denies the character devices (/dev/ptmx,/dev/zero,/dev/tty→EPERM;/dev/nullis allowed), so the first attempts reportedpty: start /bin/sh: operation not permitted; the earlier note that widening did not help was wrong — that retry had not taken effect. With full access the PTY works, and the runs above were made there. CI remains the only environment that runs them under the normal policy.Notes
ubuntu-latestnotice per job). Reviewed withannotations_countper job.CI
All 8 checks pass on
11f71df, includingLSP integration (gopls + typescript)— the gate for this fix, which runsmake test-tui-lsp— andTUI visual. Annotations match the documented baseline (1 per job; 2 forbrowserandtui-visual), so no new annotations.