Skip to content

feat: add optimization for duration#128

Open
andrewklatzke wants to merge 1 commit intoaklatzke/AIC-2118/add-additional-validation-to-chaos-modefrom
aklatzke/AIC-2154/optimize-for-duration
Open

feat: add optimization for duration#128
andrewklatzke wants to merge 1 commit intoaklatzke/AIC-2118/add-additional-validation-to-chaos-modefrom
aklatzke/AIC-2154/optimize-for-duration

Conversation

@andrewklatzke
Copy link
Copy Markdown
Contributor

@andrewklatzke andrewklatzke commented Apr 8, 2026

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions

Describe the solution you've provided

Adds additional instructions and data so that the variation generation can optimize for speed when prompted to. Uses a regex of various words ("speed" "snappy" etc.) that inform the prompt whether or not to do the duration optimization.

Describe alternatives you've considered

This could be added as a top-level option to the prompt (to optimize this specific dimension) but I think that would quickly explode into a ton of options. Instead I'm just relying on the users' intent with the acceptance statement to determine whether or not to include the instructions.

Duration tolerance might be tweaked in the future/made an option, but for now this achieves the request in the PRD.

Additional context

Example (abridged) output when including "make it faster" in the prompt:

      "rationale": "... Latency was 2410ms, which is 2515ms faster than the 4925ms baseline, so it also meets the speed goal well.",


Note

Medium Risk
Adds a new latency-based pass/fail gate and injects timing context into judge/variation prompts, which can change when optimizations succeed and may increase retries/iterations when timing data is present.

Overview
Adds duration-aware optimization that activates when any judge acceptance_statement contains latency keywords (via new _acceptance_criteria_implies_duration_optimization).

When active, the client now (1) passes agent_duration_ms into acceptance-statement judges so they can reference current vs baseline latency in their rationale, (2) includes per-iteration duration_ms in variation-generation prompt history plus an extra Duration Optimization section, and (3) enforces a new duration gate (_evaluate_duration) requiring candidates to be < 0.80 * baseline (history[0]) in chaos, validation, and ground-truth sample evaluation.

Adds extensive tests covering keyword detection, judge instruction injection, duration gating edge cases, and end-to-end wiring in both chaos and ground-truth modes.

Reviewed by Cursor Bugbot for commit 5d76276. Bugbot is set up for automated code reviews on this repo. Configure here.

@andrewklatzke andrewklatzke requested a review from jsonbailey April 8, 2026 22:39
@andrewklatzke andrewklatzke requested a review from a team as a code owner April 8, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants