Proposal: ongoing Copilot CLI compatibility framework — tests → report → auto-updated wiki matrix (maister × Copilot × OS)

[robmar-net]

This can be removed, apologies for the clutter.

Goal

A self-serve framework so that every time maister-copilot is (re)generated, anyone can verify how it behaves on a specific Copilot CLI version/OS — and a compatibility matrix on the wiki stays current, keyed by maister version × Copilot CLI version × OS.

Context: maister-copilot is auto-generated by platforms/copilot-cli/build.sh; live testing on Copilot CLI 1.0.73 (2026-07-22) confirmed it is runnable (see #8). The generator-remediation work adds a committed test harness (WS7) — this ticket proposes turning it into an ongoing, self-updating compatibility framework.

Proposed shape — script-first

Recommendation: a committed script is the core, not a skill/prompt. Rationale: the harness must be deterministic, CI-runnable, and free of model-credit cost for orchestration. The few checks that genuinely need a live agent run (the task/skill delegation and hook-firing) are exercised by tiny fixed prompts the script drives via copilot -p … --log-level debug and parses from the logs. A skill/agent layer for the harness itself would add nondeterminism + credit cost for no benefit.

Components

1. Test runner — platforms/copilot-cli/compat-tests/run-compat-tests.sh (the WS7 harness): loads the freshly-generated plugin via copilot --experimental --plugin-dir and asserts the runtime contracts:
   • install/load · skills register with 0 load failures · agents resolve as maister-copilot:* · a live task(agent_type:"maister-copilot:<agent>") and skill("<name>") delegation execute · SessionStart + PreToolUse hooks fire (incl. ${CLAUDE_PLUGIN_ROOT}) · plugin .mcp.json loads.
2. Report — report-<maister>-<copilot>-<os>-<date>.md (+ machine-readable .json), header capturing maister version (plugin.json/git), Copilot CLI version (copilot --version), OS (uname -sm), date, per-contract PASS/FAIL + evidence. Raw reports optionally committed under compat-tests/reports/.
3. Wiki updater — update-wiki.sh: clones SkillPanel/maister.wiki.git, upserts a row keyed by (Copilot version × OS × maister version) into a Copilot-CLI-Compatibility page, commits + pushes. Token-gated in CI; runnable manually.
4. Wiki page — the matrix: rows = (maister × Copilot × OS), columns = the contracts, cells = ✅/❌ with a link to the raw report.

Flow

build maister-copilot → run-compat-tests.sh (drives Copilot, asserts, writes report) → update-wiki.sh (upserts the matrix row) → wiki page current.

Triggers / cadence

On each maister-copilot rebuild (CI), on new Copilot CLI releases, and/or scheduled (e.g. weekly). Always runnable manually.

Open decisions (let's refine together to maximize sense)

• Wiki auth/automation: token in CI to push to *.wiki.git, vs GitHub API, vs manual-for-now.
• Cadence & triggers: per-rebuild / per-Copilot-release / scheduled.
• Matrix scope: how many historical (maister × Copilot) combos to retain; single page vs per-major-version pages.
• CI policy: should a contract regression fail the build, or only report to the matrix?
• Report retention: commit raw reports in-repo vs wiki-only vs artifacts.

Depends on / relates to

• WS7 committed harness from the generator-remediation work (.maister/tasks/development/2026-07-22-maister-copilot-generator-remediation).
• Correction/context in maister-copilot? could I touch it perhaps (the scripts)? #8.

Why separate from the remediation

The harness ships with the fix (build/verify asset). This — the version-matrix + wiki publishing + cadence — is an ongoing maintenance/reporting process with its own lifecycle and infra decisions. Keeping it separate keeps the remediation focused.

[robmar-net]

Storage option: repo vs wiki for the compatibility reports

Worth deciding explicitly — the wiki isn't the only option; the per-run reports can also just live in the repo. Each method has trade-offs:

Committed in-repo (e.g. platforms/copilot-cli/compat-tests/reports/ + a generated COMPATIBILITY.md matrix)

• ➕ Versioned with the code that produced it — every report ties to an exact commit; full history/blame/diff between Copilot versions.
• ➕ PR-reviewable, and no extra auth: build-copilot.yml already auto-commits, so the same flow can drop the report in.
• ➕ Simplest automation (same repo).
• ➖ Adds commit churn/noise over time; can bloat history.
• ➖ Less of a curated "dashboard" than a wiki page (mitigated by generating a COMPATIBILITY.md matrix).

GitHub wiki subpage (Copilot-CLI-Compatibility)

• ➕ Clean, browsable, always-current matrix; separate from code/PR diffs; maintainer-editable without a PR.
• ➖ Wiki is a separate git repo (*.wiki.git) — CI needs a token/extra step to push (more friction).
• ➖ Not versioned with the producing commit; not PR-reviewable.

Suggested default (lowest friction): keep raw reports in-repo (auto-committed by the existing CI) + generate a committed COMPATIBILITY.md matrix; optionally mirror that matrix to the wiki later purely as a nicer landing page — so the wiki becomes optional polish, not a dependency.

Folding "publish target: in-repo vs wiki vs both" into the open decisions above.

[robmar-net]

Update — where this is being prototyped (for now).

To iterate freely against a live Copilot CLI, the testing framework proposed here is being developed in my fork first: robmar-net#3 plans it in layers —

• L0 (wiring) — the committed WS7 runtime harness (make test-copilot), already shipping in Re-engineer the maister-copilot generator into a runnable Copilot CLI plugin robmar-net/maister#2 (upstream Re-engineer the maister-copilot generator into a runnable Copilot CLI plugin #10): asserts plugin/skills/agents register, task+skill delegation, hooks fire, MCP loads. Green on Copilot CLI 1.0.73.
• L1 (hook effects) — deterministic checks that a hook had the right effect (destructive-command guard actually blocks; post-compact reminder actually injected), not just that it fired.
• L2 (differential trace equivalence) — run full workflows on both platforms and compare normalized execution traces, with noise-calibrated thresholds (N runs → within-platform-stable = signal skeleton), across 3 scenarios.

Guiding principle: proof of equivalence ≈ regression detection — since we can't predict where a Copilot version will break, capture the broadest observable trace rather than a narrow feature check.

Happy to upstream the framework here once it stabilizes.