Skip to content

[agent] Bound the OpenAI-compatible stream after finish_reason - #178

Merged
yusiwen merged 1 commit into
masterfrom
fix/176-openai-stream-idle-bound
Oct 10, 2026
Merged

yusiwen merged 1 commit into
masterfrom
fix/176-openai-stream-idle-bound

Conversation

@yusiwen

@yusiwen yusiwen commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Closes #176. Regression introduced by #175, found while delivering #174.

What changed

  • agent/provider_openai.go now carries the same pair of bounds the Ollama provider does:
    openAIRequestTimeout (120 s) for one non-streaming request, and openAIIdleTimeout (2 min) for
    the silence between lines of a stream, reset by every line the scanner delivers. Both are package
    variables so a test can shrink them, and the cancel cause names the bound (errOpenAISilent), the
    same idiom as provider_ollama.go (issue agent: the Ollama provider has no request timeout, so a stalled endpoint hangs the run #1).
  • A failed read returns an error — openai stream: … — instead of the partial answer with a nil
    error, so a cut stream cannot be mistaken for a finished one. That is also what the Ollama provider
    has done since issue agent: the Ollama provider has no request timeout, so a stalled endpoint hangs the run #1.
  • openAIRequestError names the bound that fired and keeps errors.Is(err, context.Canceled) for a
    caller cancellation, which is how a TUI run renders as interrupted.
  • chatStream takes the watchdog as an argument; the parser-level tests pass nil.
  • Docs: CHANGELOG.md, CODEBASE.md.

Why

#175 had to read past finish_reason to reach the usage chunk, and removed the early return there
without replacing it. Measured against an endpoint that completes its answer and then holds the
connection open:

Endpoint ends with before after
data: [DONE] 2 ms, "Hi", err=nil 3 ms, "Hi", err=nil
nothing (held open) 1.502 s, "Hi", err=nil 1.5 s, resp=nil, err=openai stream: context deadline exceeded

The 1.5 s there is the caller's deadline in the repro; neither entry point passes one, so the real
wait was openAIRequestTimeout — the whole 120 s. The worse half is the second column: a truncated
stream was reported as a successful answer.

Rejected alternative: stopping at finish_reason once usage has been seen. An endpoint that never
reports usage would then return at the wrong point and cut the flush that follows the marker, which
is the defect #175 fixed.

Verification

  • go test ./... -count=1 → 13 packages ok, 0 failed. go test ./... -count=1 -race → same.

  • make staticcheck (v0.8.1) → exit 0. gofmt -l . → no output.

  • The new gate was mutation-checked four ways:

    Mutation Result
    watchdog never fires the silence test times out at 25 s — that is the read blocking without the bound
    no per-line reset a stream producing a chunk every 100ms failed: openai stream: stream went silent for 400ms
    a cut stream returns its partial text a stream that went quiet after finish_reason returned success with "Hi"
    drop the caller-cancellation classification on a cancelled context, error = http2: stream closed, want errors.Is(err, context.Canceled)
  • The last row is why openAIRequestError has a direct unit test: on HTTP/1.1 the read error already
    wraps context.Canceled, so removing the helper's classification does not change the streaming
    subtest — the mutation survived until the helper itself was pinned with an HTTP/2-shaped error.

Honest scope

  • No live provider call was made; the endpoints used are httptest servers.
  • The idle bound is uniform, so a stream that goes quiet after a complete answer still waits the
    full idle bound (2 min) before failing. Distinguishing "the answer is complete but the sentinel
    never came" from "the model is still thinking" would need a shorter post-finish_reason bound,
    which this change deliberately does not add.
  • A stream that closes without [DONE] is still accepted as complete, deliberately:
    OpenAI-compatible servers vary, and failing their working answers would cost more than the
    truncation it would catch. [DONE] still ends the read early.
  • Behaviour change worth naming: a mid-stream caller cancellation now surfaces as an error rather
    than as partial content that looked like a finished answer. An interrupted run still reads as
    interrupted (pinned above), but the transcript difference is real.
  • Reproduction harness: /tmp/dsh-repro-175 (scratch module, replace → this repository).

Reading to `data: [DONE]` (issue #174) is what makes the usage chunk reachable.
It also removed the early return at `finish_reason` without putting anything in
its place: an endpoint that sends a complete answer and then holds the
connection open left the step waiting for the whole-request client timeout, and
`chatStream` returned the text it had with a nil error, so the loop could not
tell a cut stream from a finished one.

Measured against an endpoint that completes its answer and then goes quiet:
2 ms when the stream ends with `data: [DONE]`, 1.502 s at a 1.5 s caller
deadline without it — and neither entry point passes a deadline of its own, so
in production the wait is the full 120 s.

- `openAIIdleTimeout` (2 min), reset by every line the scanner delivers, plus
  `openAIRequestTimeout` (120 s) for the batch path: the same two-bounds shape,
  and the same cancel-cause idiom, as `provider_ollama.go` (issue #1). Both are
  package variables so a test can shrink them.
- A failed read is now an error (`openai stream: …`) instead of the partial
  answer with a nil error, which is also what the Ollama provider does.
- `openAIRequestError` names the bound that fired, and keeps
  `errors.Is(err, context.Canceled)` for a caller cancellation — that is how the
  TUI renders an interrupted run, and it is pinned by a direct unit test rather
  than left to the transport: an HTTP/1.1 read error happens to wrap
  context.Canceled already, an HTTP/2 stream close does not.

A stream that *closes* without `[DONE]` is still accepted as complete.
OpenAI-compatible servers vary here, and failing their working answers would
cost more than the truncation this catches. `[DONE]` remains the marker that
ends the read early.

Closes #176.

Tests: `go test ./... -count=1` and `-race` green (13 packages); gofmt and
staticcheck v0.8.1 clean. Four mutations of the new behaviour fail on an
assertion — or, with the bound disabled, on the test timeout, which is itself
the demonstration that the read blocks without it.
@yusiwen yusiwen added the bug Something isn't working label Oct 10, 2026
@yusiwen
yusiwen merged commit 49964e2 into master Oct 10, 2026
9 checks passed
@yusiwen
yusiwen deleted the fix/176-openai-stream-idle-bound branch October 10, 2026 06:40
yusiwen pushed a commit that referenced this pull request Oct 11, 2026
Pin that a non-streaming `Chat` against an endpoint that accepts the connection and never answers fails
within `openAIRequestTimeout`, mirroring `TestOllamaBatchTimesOutOnSilentEndpoint`.

Test only: the code half of #176 already landed as `49964e2` (#178) — the same idle-timeout work this PR
set out to do — so the author's force-push (`7222e40`, whose base already carried that fix) left the
branch with this test alone. Authored by AdamMagued in #177.

Verified: the test fails against a silent `httptest` endpoint at the shrunk bound (0.15 s locally,
Go 1.27.1 darwin/arm64); `go test ./agent -count=1 -race` green; CI on `7222e40` 9/9 jobs pass with
annotations at the per-job baseline (1, or 2 for browser and tui-visual).

Refs #185, #186.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[agent] Bound the SSE read after finish_reason so a stream without [DONE] cannot stall the call

1 participant