feat(agents): local integration_test authoring agent

[ayushnangia]

Summary

The 01-coding-template documents hud eval <env-name> integration_test as the shipping check for every task: pre-stage the golden solution, run the graders, require Reward 1.0. Until now that agent only existed on the platform — the local CLI rejected the name (previously patched with a helpful error in #585). This PR implements the real thing locally.

What it does

• AgentType.INTEGRATION_TEST + IntegrationTestConfig (no model, timeout_seconds bounds the staging pass).
• IntegrationTestAgent — an Agent that makes no LLM calls: it opens the task's own MCP capabilities, replays every Task.validation tool call through them (recording ToolSteps like any other agent), and ends the trace with an empty answer so the environment's scenario graders grade the staged workspace. String-arguments steps are rejected without executing, unknown/invalid steps are skipped with warnings, and a missing-capability call surfaces as an error result.
• Run.validation — rollout attaches task.validation to the live run so the agent can consume it.
• The authoring gate — after the rollout, the CLI enforces the documented contract: every run must grade Reward 1.0; anything lower (or a grading error) exits non-zero and names the failing reward/raw grade, so the authoring loop fails loudly instead of shipping a task whose golden doesn't pass its own hidden graders.
• gateway_provider raises for this type (it's not a gateway shortcut — no LLM).

Validation

• New tests: hud/agents/tests/test_integration_test.py (dispatch + ToolStep recording, empty-validation no-op, invalid-entry tolerance) and CLI tests (config accepts the name, the reward gate exits on <1.0 and on grading errors).
• Full local suite: 392 passed (hud/tests + capabilities + patches + agents + cli).
• ruff + ty clean.

Notes

• Supersedes the fix(cli): point integration_test users at the platform flow #585 error path (the name now parses; the error only remains in spirit for the gateway shortcut). fix(cli): point integration_test users at the platform flow #585 can be closed as superseded if this lands.
• No platform changes: sync payloads and the platform-side agent are untouched.
• Template contract: the 01-coding-template README's authoring loop is the spec this implements.

---

Note

Medium Risk
Touches eval rollout, grading exit behavior, and remote command execution over MCP/SSH for golden staging; mistakes could block authoring or run unintended shell on workspaces, though scope is limited to the integration_test path.

Overview
Adds a local, no-LLM integration_test agent so hud eval <taskset> integration_test can serve as the documented shipping check: replay Task.validation through the task’s MCP or SSH capabilities, then let scenario graders score the staged workspace.

IntegrationTestAgent opens manifest MCP/ssh/2 bindings, coerces validation entries to MCPToolCall, dispatches tool calls (including bash -lc over SSH), records ToolSteps, and honors a configurable staging timeout. Invalid or unsupported validation steps are skipped with warnings; create_agent rejects this type as a gateway model shortcut.

Rollout and CLI: Run.validation is populated from task.validation during rollout. After eval, _enforce_integration_test_reward requires every run to reach reward 1.0 (or exits non-zero on grading errors), matching the authoring-loop contract.

AgentType.INTEGRATION_TEST, IntegrationTestConfig, and tests cover agent dispatch, SSH paths, config parsing, and the reward gate.

^{Reviewed by Cursor Bugbot for commit c9601f2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[ayushnangia]

Thanks Bugbot — three findings, addressed on the new head f7c84bd:

1. Missing config_cls arm (6943937d) — was on the first push only; the CI-hardening commit d45100b added the INTEGRATION_TEST arm + a case _ raise (the repo's ty check --error-on-warning lint caught it). No longer applicable on the current head.
2. SSH golden staging (1858b6e9) — real gap for SSH-published workspaces, fixed: the agent now opens ssh/2 bindings alongside mcp ones and runs bash validation steps via bash -lc <command> over SSHClient.run, with the exit code mapped to isError. Note the coding-template itself publishes MCP (env.add_tool(bash_tool.mcp)), but SSH-publishing envs were indeed unserved — now both protocols stage correctly. New tests cover both directions.
3. create_agent gateway cast (b904078c) — real; create_agent("integration_test") now raises (mirrors gateway_provider) with a pointer to hud eval <taskset> integration_test. Test added.

Full suite 395 passed, ruff + ty clean.

[ayushnangia]

Fixed on c9601f2: _run_over_ssh now passes a single remote command string (bash -lc <command>), letting asyncssh shlex-split it while preserving the command's own quoting — the multi-arg form could lose grouping on space-join. Test updated to pin the single-string form.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c9601f2. Configure here.}

[cursor]

SSH staging command misquoted

High Severity

Building the remote string as bash -lc {command} without shlex.quote does not pass the golden script as one -c argument. On ssh/2 workspaces, shell_argv already wraps the exec string in bash -lc, and working SSH tools just run the raw command. Typical validation steps with spaces or redirects therefore stage the wrong remote command, so the authoring gate can fail (or pass) for the wrong reason.

Additional Locations (1)

• hud/agents/tests/test_integration_test.py#L108-L109

 

^{Reviewed by Cursor Bugbot for commit c9601f2. Configure here.}

[ayushnangia]

Review-readiness summary, since the PR went through several CI rounds:

• Head is settled: all four Bugbot findings addressed (config_cls arm = stale on the first push, fixed in d45100b; SSH golden staging = implemented for ssh/2 workspaces; create_agent gateway cast = now raises; single-string SSH command form = applied on c9601f2).
• CI fully green: ruff, ty, tests 3.11 + 3.12 on the final head.
• Contract: the 01-coding-template README's authoring loop is the spec; the Reward-1.0 gate exits non-zero naming the failing raw grade.
• Relation to fix(cli): point integration_test users at the platform flow #585: this implements what that PR's error message pointed at — if this lands, fix(cli): point integration_test users at the platform flow #585 can close as superseded.

No rush — flagging only because the diff is now stable.