Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions config/personas/quality/references/checks.conf
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,8 @@ IMMEDIATE="uv run python scripts/validate_schemas.py"
# Before each commit: affected integration tests and a small eval smoke set.
CHECKPOINT="uv run pytest -m integration -q && uv run python -m evals.smoke"

# Tests the checkpoint runs: related (default) runs only the tests the commit's
# changes reach, leaving the full suite to CI; full runs each affected
# package's whole suite. Use full only when the suite takes seconds.
# CHECKPOINT_TESTS=full
# Hooks do not automatically run tests. Set CHECKPOINT explicitly for any
# tests this repo needs before an agent commit, or use your own pre-commit hook.

# Nightly CI (quality.yml schedule): E2E, broad model matrix, load tests, the
# full eval suite.
Expand Down
Binary file modified dist/macols-personas-claude-plugin.zip
Binary file not shown.
38 changes: 14 additions & 24 deletions docs/hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,20 @@ Each check runs at the cheapest loop that still catches the problem:

| Loop | What runs | When |
|---|---|---|
| Immediate | Per edit: format, lint and type check on the file. Per turn: related unit tests, types, layer rules, duplication, and the project's `IMMEDIATE` command | Every agent iteration |
| Local checkpoint | The tests the commit's changes reach (each affected package's whole suite with `CHECKPOINT_TESTS=full`), the same lint/type/layer checks, and the project's `CHECKPOINT` command (integration tests, an eval smoke set) | When the agent runs `git commit`; a failure blocks the commit |
| Immediate | Per edit: format, lint and type check on the file. Per turn: types, layer rules, duplication, and the project's `IMMEDIATE` command | Every agent iteration |
| Local checkpoint | Lint/type/layer checks, synthesis, review, and the project's explicit `CHECKPOINT` command | When the agent runs `git commit`; a failure blocks the commit |
| PR | Full tests, semgrep, dependency audits, jscpd threshold (the `quality` skill's CI workflow) | Before human review |
| Merge/nightly | The project's `NIGHTLY` command: E2E, broad model matrix, load tests, full eval suite | Scheduled CI job |
| Production | Traces, user failures, latency and cost, sampled quality evals | Continuously; see the `sre` skill |

Projects declare their own commands in `.macols/checks.conf` (template in
`config/personas/quality/references/checks.conf`). The hooks read it as data and
run the commands from the repo root in trusted projects only. The
checkpoint uses the turn-end test selection over the whole uncommitted change
and leaves the full suite to CI; set `CHECKPOINT_TESTS=full` there to run each
affected package's whole suite instead. The checkpoint
skips when the tree matches the last passing checkpoint, when the command
hooks never automatically run tests. Repos own test execution through their
own pre-commit/CI workflow or an explicit `CHECKPOINT` command. Legacy
`CHECKPOINT_TESTS` and test-scope environment settings do not enable automatic
tests. Existing `IMMEDIATE` or `CHECKPOINT` commands still run as configured.
The checkpoint skips when the tree matches the last passing checkpoint, when the command
passes `--no-verify`, or with `MACOLS_CHECKPOINT=off`.

## What Runs
Expand All @@ -50,13 +51,10 @@ Every tool gets the findings in its own model-visible format
For Go that means a compile or type error in the edited package shows up
with the edit, not at turn end; go vet builds the package, so it runs in
trusted projects only.
- post-task: at the end of a turn, tests related to the changed files, lint
and strict types, golangci-lint, jscpd duplication, and layer rules
(import-linter, dependency-cruiser) when the project configures them. Test
selection: pytest-testmon when the project's venv has it (coverage-based),
otherwise `test_<module>.py` by name; `vitest related` / `jest
--findRelatedTests` by import graph (any other `npm test` waits for the
checkpoint, since it may be E2E); `go test` on the changed packages. When
- post-task: at the end of a turn, lint and strict types, golangci-lint,
jscpd duplication, CDK synthesis, and layer rules (import-linter,
dependency-cruiser) when the project configures them. Built-in Python,
Node, Flutter and Go tests do not run. When
something fails, the agent gets one more step with the findings and a fix
instruction. It only re-runs when the tree changed since its last run, so it
can't loop and Q&A turns stay fast (under a second on small projects).
Expand Down Expand Up @@ -142,20 +140,16 @@ first run). Pass `--no-cli` to skip the whole step.

Every turn-end and checkpoint run appends one JSON line per check to
`.git/macols-checks.jsonl` (shared by worktrees, capped at 10k lines):
duration, pass or fail, and which loop ran it. At the checkpoint, failing
Python tests that the name-based turn-end selection wouldn't have run are
recorded as escapes.
duration, pass or fail, and which loop ran it. Historical test escapes remain readable in older logs.

```bash
macols-check-stats # per loop and check: runs, total/p95 time, fail %, fail streak, escapes
macols-check-stats --days 7
macols-check-stats --escapes # which tests escaped the fast path
```

Read it as: a slow check that never fails belongs in CI, a long fail streak
means the agent is going round in circles, and escapes mean the fast path
needs pytest-testmon or better-named tests. `MACOLS_CHECK_LOG=off` stops the
logging.
A slow check that never fails is a candidate for CI. A long fail streak
means the agent is going round in circles. `MACOLS_CHECK_LOG=off` stops logging.

## Trusted Projects

Expand Down Expand Up @@ -190,16 +184,12 @@ workflow. Existing layer breaks are approved explicitly: a

| Variable | Effect |
|---|---|
| `MACOLS_PYTEST_SCOPE=changed\|full\|off` | Python test scope at turn end (default `changed`) |
| `MACOLS_TESTMON=off` | Ignore pytest-testmon and select Python tests by name |
| `MACOLS_GO_TEST_SCOPE=changed\|module` | Go tests at turn end: only changed packages (default) or the whole changed module via the test cache |
| `MACOLS_DEPLOY_DIFF=off` | Don't run `cdk diff` before a deploy prompt |
| `MACOLS_DEPLOY_DIFF_TIMEOUT` | Seconds the pre-deploy `cdk diff` may take (default 240) |
| `MACOLS_CHECK_LOG=off` | Don't log check timings |
| `MACOLS_DUPLICATION=off` | Skip the jscpd duplication check |
| `MACOLS_DUP_MIN_LINES` | Minimum clone size jscpd reports (default 10) |
| `MACOLS_MAX_FILE_LINES` | File-length limit (default 500, `0` to turn off) |
| `MACOLS_GO_RACE=1` | Run Go tests with the race detector |
| `MACOLS_SEMGREP=1` | Run semgrep locally |
| `MACOLS_CHECKS_VERBOSE=1` | Also print notes such as "pytest not installed" |
| `MACOLS_HOOKS_DIR` | Where the hook scripts live |
Expand Down
14 changes: 2 additions & 12 deletions hooks/checks/lang_checks.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash
#
# lang_checks.sh — the remaining per-language batteries: cdk synth, Flutter
# (scoped flutter test + dart analyze), Go (golangci-lint/go vet + go test),
# (dart analyze), Go (golangci-lint/go vet),
# and shellcheck for scripts.
#
# Sourced by post_task.sh AFTER common.sh and its helpers (add_critical_issue,
Expand Down Expand Up @@ -100,10 +100,7 @@ _changed_go_packages() {
}

# golangci-lint (the project's .golangci.yml: dupl, gocyclo, funlen, depguard…)
# or go vet when it isn't installed, on the changed packages; then go test on
# the changed packages (./... per module at the checkpoint).
# MACOLS_GO_TEST_SCOPE=module widens turn-end tests to the whole module, which
# also catches dependents (cheap when the test cache can replay them).
# or go vet when it isn't installed, on the changed packages.
run_go_checks() {
command -v go &> /dev/null || { add_warning "go not installed - skipping Go checks"; return 0; }
local pkgs
Expand All @@ -122,13 +119,6 @@ run_go_checks() {
out=$(cd "$mod" && go vet "${dirs[@]}" 2>&1) || ec=$?
report_check_result "go vet ($label)" "$ec" "$out" "^.+\.go:[0-9]+(:[0-9]+)?: |^vet: " "issues"
fi
ec=0
local -a race=()
[ "${MACOLS_GO_RACE:-0}" = "1" ] && race=(-race)
local -a test_targets=(./...)
[ "${MACOLS_GO_TEST_SCOPE:-changed}" = "changed" ] && ! full_suite && test_targets=("${dirs[@]}")
out=$(cd "$mod" && ${TIMEOUT_CMD:+$TIMEOUT_CMD $MAX_TEST_TIME} go test ${race[@]+"${race[@]}"} "${test_targets[@]}" 2>&1) || ec=$?
[ "$ec" -ne 0 ] && report_test_failure "go test ($label)" "$ec" "$out"
done
return 0
}
Expand Down
16 changes: 6 additions & 10 deletions hooks/checks/post_task.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,7 @@
# per-turn loop (semgrep, pip-audit/npm audit/govulncheck — see the `quality`
# skill's CI template).
#
# Tests pytest (scoped, or pytest-testmon when installed),
# jest/vitest related tests, flutter test (scoped), go test
# over changed modules (the test cache skips the rest),
# cdk synth
# Synthesis cdk synth
# Lint ruff, eslint, dart analyze, golangci-lint (or go vet),
# and shellcheck, each with the project's own limits
# Types pyright or mypy (whichever the project configures),
Expand All @@ -42,9 +39,8 @@
# Only findings are recorded as CRITICAL_ISSUES, each with a fix instruction.
# WARNINGS hold notes (tool missing, nothing to test) for verbose runs.
#
# Switches: MACOLS_PYTEST_SCOPE=changed|full|off, MACOLS_TESTMON=off,
# MACOLS_DUPLICATION=off, MACOLS_SEMGREP=1 (opt back in to a local semgrep
# scan), MACOLS_GO_RACE=1, MACOLS_GO_TEST_SCOPE=changed|module,
# Tests run only through explicit project commands, never automatically.
# Switches: MACOLS_DUPLICATION=off, MACOLS_SEMGREP=1,
# MACOLS_CHECK_LOG=off (see log_check_run in common.sh),
# MACOLS_LGREVIEW=off (and its BLOCK/TIMEOUT knobs — lgtmaybe_review.sh).
#
Expand Down Expand Up @@ -208,14 +204,14 @@ run_post_task_checks() {

local -a checks=()
if [ "$has_python" = "true" ]; then
checks+=(run_python_tests run_ruff_check run_python_typecheck run_import_linter)
checks+=(run_ruff_check run_python_typecheck run_import_linter)
fi
if [ "$has_node" = "true" ]; then
checks+=(run_node_tests run_eslint_check run_tsc_check run_dependency_cruiser)
checks+=(run_eslint_check run_tsc_check run_dependency_cruiser)
fi
[ "$has_cdk" = "true" ] && checks+=(run_cdk_tests)
if [ "$has_flutter" = "true" ]; then
checks+=(run_flutter_tests run_dart_analyze)
checks+=(run_dart_analyze)
fi
[ "$has_go" = "true" ] && checks+=(run_go_checks)
checks+=(run_shellcheck run_duplication_check run_file_length_check run_semgrep_scan)
Expand Down
10 changes: 4 additions & 6 deletions hooks/post_task_hook.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,12 @@
# changed nothing is allowed to stop, so the hook cannot loop.
# - Never when the tool says this stop is already a hook-driven continuation
# (stop_hook_active) — one nudge per turn, then the user decides.
# - pytest/jest/vitest/go test are scoped to what the changed files reach
# (pytest-testmon or file names for Python, the test cache for Go). The full
# suite is the commit checkpoint's and CI's job.
# MACOLS_PYTEST_SCOPE=full|changed|off overrides the Python scope.
# - Built-in tests never run. Repos own test execution through explicit
# project commands or their own pre-commit/CI workflow.
# - Every run appends per-check timings to .git/macols-checks.jsonl
# (bin/macols-check-stats summarises them).
#
# Only critical findings (failing tests, lint/type/gate errors) are sent to the
# Only critical findings (failing project commands, lint/type/gate errors) are sent to the
# model; notes such as "pytest not installed" print only with
# MACOLS_CHECKS_VERBOSE=1.
#
Expand Down Expand Up @@ -78,7 +76,7 @@ fi

TRUST_NOTE=""
if [ "${UNTRUSTED_SKIPPED:-0}" = "1" ]; then
TRUST_NOTE="Tests, eslint, tsc, mypy, cdk synth, go and layer checks were skipped: $(project_root) is not a trusted project, and those run code the repo controls. To enable them, run $(dirname "$SHARED_HOOKS_ROOT")/bin/macols-trust in the repo, or add its path to $MACOLS_TRUST_FILE."
TRUST_NOTE="Project commands, eslint, tsc, mypy, cdk synth, go and layer checks were skipped: $(project_root) is not a trusted project, and those run code the repo controls. To enable them, run $(dirname "$SHARED_HOOKS_ROOT")/bin/macols-trust in the repo, or add its path to $MACOLS_TRUST_FILE."
fi

if [ ${#CRITICAL_ISSUES[@]} -eq 0 ]; then
Expand Down
7 changes: 2 additions & 5 deletions hooks/pre_commit_check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@
#
# pre_commit_check.sh <command-string> — protocol-neutral core of the local
# checkpoint loop: when the agent runs `git commit`, run the checkpoint
# battery (the tests the change reaches, or whole suites with
# CHECKPOINT_TESTS=full; lint, types, layer rules, and the project's
# battery (lint, types, layer rules, and the project's explicit
# CHECKPOINT command) before the commit lands.
#
# Prints the findings when the checkpoint fails and nothing otherwise, and
Expand Down Expand Up @@ -49,9 +48,7 @@ fp=""
review_ready || fp=$(_check_fingerprint || true)
[ -n "$fp" ] && [ -f "$fp_file" ] && [ "$(cat "$fp_file")" = "$fp" ] && exit 0

CHECKPOINT_MODE=1 MACOLS_PYTEST_SCOPE=full
full_suite || MACOLS_PYTEST_SCOPE=changed
export CHECKPOINT_MODE MACOLS_PYTEST_SCOPE
export CHECKPOINT_MODE=1
# shellcheck disable=SC2034 # read by the sourced post_task.sh checks
MAX_TEST_TIME="${MACOLS_CHECKPOINT_TIMEOUT:-900}"
run_post_task_checks || exit 0
Expand Down
2 changes: 2 additions & 0 deletions openspec/changes/remove-hook-test-runs/.openspec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: macols
created: 2026-10-20
9 changes: 9 additions & 0 deletions openspec/changes/remove-hook-test-runs/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Design
## Context
The shared orchestrator dispatches Python, Node and Flutter test checks; Go combines lint and tests in one check. Both turn-end and pre-commit use this battery.
## Goals / Non-Goals
Remove automatic test execution. Preserve static checks, synthesis, review, trust boundaries and explicit project commands. Do not provision repo-specific test hooks.
## Decisions
Remove test dispatch from the shared orchestrator and the test portion of Go checks. Keep existing standalone Python/Node/Flutter selection helpers outside the automatic battery to avoid unrelated helper and fixture churn. Remove checkpoint scope overrides because the battery no longer selects tests.
## Risks / Trade-offs
Tests no longer block completion or commits unless a repo explicitly configures them. Existing IMMEDIATE or CHECKPOINT commands can still run tests by design.
20 changes: 20 additions & 0 deletions openspec/changes/remove-hook-test-runs/evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Evidence

## E1. No automatic tests; static checks and configured commands survive
- **Claim**: Both batteries skip built-in tests, including legacy full-scope settings; Go lint and explicit checkpoint failures remain effective. Trust gates and model-visible findings survive.
- **Command**: `bash tests/test_hook_loops.sh`
- **Expected**: All assertions pass using scratch repos and boundary stubs.
- **Fails before**: The new battery regression detects automatic test dispatch; Go assertions detect go test.
- **Recorded**: pass, 41/41 assertions; base HEAD 95a50bdf609186b7b0014593bdff3c24bfcdeca3, working-tree changes tested (2026-10-20).

## E2. Shell and anchor constraints
- **Claim**: C-SHELLCHECK-CLEAN, C-SPEC-ANCHORS-HEALTHY and source-only changes hold.
- **Command**: `bash -n hooks/checks/post_task.sh hooks/checks/lang_checks.sh hooks/post_task_hook.sh hooks/pre_commit_check.sh tests/test_hook_loops.sh && shellcheck -S warning hooks/checks/post_task.sh hooks/checks/lang_checks.sh hooks/post_task_hook.sh hooks/pre_commit_check.sh tests/test_hook_loops.sh && ./scripts/spec_drift_gate.sh --check`
- **Expected**: All commands pass; installer and per-edit code unchanged.
- **Recorded**: pass; base HEAD 95a50bdf609186b7b0014593bdff3c24bfcdeca3, working-tree changes tested (2026-10-20).

## E3. Installer, bundle and change validation
- **Claim**: Codex wiring remains valid, the quality template is bundled, and spec deltas validate.
- **Command**: `./tests/verify_install.sh codex && ./scripts/package_claude_desktop_personas.sh --check && openspec validate remove-hook-test-runs --strict && git diff --check`
- **Expected**: All hard checks pass.
- **Recorded**: pass; base HEAD 95a50bdf609186b7b0014593bdff3c24bfcdeca3, working-tree changes tested (2026-10-20). Other agents' installer verifiers were not run; shared hook wiring is unchanged.
32 changes: 32 additions & 0 deletions openspec/changes/remove-hook-test-runs/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
## Why
Automatic tests can take minutes in large repos and delay every agent reply. Test execution belongs to each repo's chosen pre-commit workflow.

## What Changes
- Remove built-in Python, Node, Flutter and Go test execution from turn-end and commit batteries.
- Keep lint, types, structure checks, CDK synthesis, AI review and explicitly configured project commands.

## Capabilities
### New Capabilities
None.
### Modified Capabilities
- `hook-wiring`: batteries validate code without automatically running tests.

## Impact
Shared batteries affect every installed agent in place. Repos can explicitly configure CHECKPOINT commands for tests.

## Scope
### May change
- `hooks/checks/post_task.sh`, `hooks/checks/lang_checks.sh`, `hooks/post_task_hook.sh`, `hooks/pre_commit_check.sh`
- `tests/test_hook_loops.sh`, `docs/hooks.md`, `config/personas/quality/references/checks.conf`
- `openspec/changes/remove-hook-test-runs/**`
- `dist/macols-personas-claude-plugin.zip`: regenerate the bundled quality template.
### Must not change
- Installer wiring, trust gates, per-edit checks, user-owned configuration and unrelated plugins.

## Constraints
- C-EDIT-SOURCE-NOT-OUTPUT: hooks run from this source checkout.
- C-TRUSTED-REPO-CODE: explicit project commands retain trust checks.
- C-HOOKS-MODEL-VISIBLE: preserve finding delivery.
- C-SHELLCHECK-CLEAN: verify touched shell scripts.
- C-SPEC-ANCHORS-HEALTHY: keep anchors resolving and propose specification deltas.
- No new opt-in setting: automatic tests are removed for all repos.
42 changes: 42 additions & 0 deletions openspec/changes/remove-hook-test-runs/specs/hook-wiring/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
## ADDED Requirements

### Requirement: Hook batteries do not automatically execute tests
Turn-end and commit batteries SHALL retain lint, type, layer, duplication, file-length and synthesis checks without automatically running Python, Node, Flutter or Go tests. Tests SHALL run only when explicitly requested by a repository command or another repository-owned workflow. Legacy test-scope settings SHALL NOT enable automatic test execution.

#### Scenario: Code changes without configured test commands
- **WHEN** either battery runs in a trusted project with changed Python, Node, Flutter or Go code
- **THEN** no built-in test runner executes and applicable static checks still run

## MODIFIED Requirements

### Requirement: A local checkpoint runs before commits
Every tool's PreToolUse(Bash) wiring SHALL pass the command to
`hooks/pre_commit_check.sh`, which on `git commit` (not `--no-verify`, not
`MACOLS_CHECKPOINT=off`, trusted projects only) runs lint/type/layer checks,
synthesis, review and the project's explicit `CHECKPOINT` command from
`.macols/checks.conf`. A failure SHALL deny the commit with the findings as
the reason; a pass is remembered by tree fingerprint so an unchanged tree
is not re-checked. The turn-end battery SHALL run the project's `IMMEDIATE`
command. No tests SHALL run automatically in either battery.
<!-- anchor: hook-wiring.checkpoint -->

#### Scenario: The agent commits a failing change
- **WHEN** the agent runs `git commit` in a trusted repo whose CHECKPOINT command fails
- **THEN** the commit is denied and the command's findings reach the model

#### Scenario: A commit runs only the tests it reaches
- **WHEN** the agent runs `git commit` without an explicit test command in CHECKPOINT
- **THEN** the checkpoint runs its applicable validation checks without invoking tests, including when CHECKPOINT_TESTS=full was previously configured

### Requirement: Every battery run is logged per check
`run_post_task_checks` SHALL append one JSON line per check, plus a `total`
line, to `macols-checks.jsonl` in the repo's common git dir (shared by
worktrees): time, repo, loop (`turn` or `checkpoint`), check, duration in ms
and `pass`/`fail`. The log SHALL stay bounded (newest 5000 lines once it passes
10000), `MACOLS_CHECK_LOG=off` SHALL turn it off, and `bin/macols-check-stats`
SHALL summarise time, failure rate, repeated failures and historical escapes per check.
<!-- anchor: hook-wiring.check-log -->

#### Scenario: The user asks whether the hooks are worth their time
- **WHEN** the user runs `macols-check-stats` in a repo after some agent turns
- **THEN** it prints, per loop and check, runs, total and p95 time, failure rate and historical escaped tests
Loading
Loading