Skip to content

feat(cli): add deterministic task preflight checks - #590

Open
ChinmayK0607 wants to merge 6 commits into
mainfrom
chinmay/hud-2527-embed-deterministic-checks
Open

feat(cli): add deterministic task preflight checks#590
ChinmayK0607 wants to merge 6 commits into
mainfrom
chinmay/hud-2527-embed-deterministic-checks

Conversation

@ChinmayK0607

@ChinmayK0607 ChinmayK0607 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Added live task manifest validation to hud task list, start, and grade, including deployed environments, canonical environment names, and consistent task/ references.
  • Added hud task grade --dry-run for environment, task, grader, and reward checks. Normal grading also rejects malformed or out-of-range rewards.
  • Added full local source and deployed task-manifest validation to hud sync tasks, including verifier task IDs.
  • Reused the task source/runtime resolver from hud eval, documented hud deploy as the build-and-publish path, and added regression coverage. There are no UI changes.
  • Supersedes the broader standalone-check design in Add production Task lifecycle check command #531.

Why

Deterministic lifecycle failures were discovered only after task upload or rollout. The previous standalone checker duplicated execution lifecycle code and included nondeterministic agent behavior beyond this requirement. These checks now live in the existing commands that own each lifecycle stage.

Impact

Environment and task authors can catch invalid build sources, missing deployed tasks, startup failures, broken graders, and malformed rewards before syncing or running agent evaluations. CI can use existing command exit statuses without adopting a separate QA protocol.

Validation

  • uv --no-config run pytest -q (1122 passed, 17 deselected)
  • uv --no-config run ruff check .
  • uv --no-config run ruff format . --check
  • uv --no-config run --extra dev --extra train --extra modal --extra daytona ty check

Note

Medium Risk
Touches CLI task execution, platform env lookup, and sync/deploy gates. Failures now block uploads and grading, so false positives could stall CI, but this is not auth or data-path logic.

Overview
Adds deterministic preflight to existing CLI commands so authors catch missing tasks, broken graders, and junk sources before sync or agent evals.

hud task start / grade / list can attach to a deployed env (--env) or a live control channel, and they verify the live environment actually exposes the requested task. hud task grade --dry-run walks env → task → grader → reward (empty answer); a low but in-range score still passes. Normal grading now rejects non-finite or out-of-range rewards.

hud sync tasks validates local source and deployed manifests (including verifier task ids) before upload. Source validation also rejects .DS_Store, __MACOSX, and broken symlinks, which hud deploy already runs via EnvironmentSource.validate(). Shared spawn/resolution lives in task_runtime.py.

Reviewed by Cursor Bugbot for commit 72480b9. Bugbot is set up for automated code reviews on this repo. Configure here.

Embed environment, task, grader, and reward validation in existing task and sync commands so broken definitions fail before rollout or upload.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread hud/cli/task.py Outdated
Apply the documented task/ prefix consistently across local, attached, and deployed task commands.
Run full source checks before task sync and reject invalid grader results in the normal task lifecycle, so deterministic failures surface in their owning commands.
Comment thread hud/cli/task.py Outdated
Normalize deployed environment references consistently, gate HUD runtime commands on authentication, and simplify task validation without exposing internal tracebacks.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a6e021f. Configure here.

Comment thread hud/cli/sync.py
Only normalize environment names during task sync so uppercase and brace-wrapped registry UUIDs keep resolving through the ID path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant