Repository navigation
Test the whole language capability matrix, not a runbook #10
Description
The approved level-c-behavioral-plan.md classifies the language capability rows as RUNBOOK (manual), on the grounds that "async subprocess + a project fixture make headless brittle". That call was made without attempting the automation, and it leaves a distribution shipping language support whose end-to-end paths nobody has exercised.
The 2026-08-31 rust work drove every assertion type those rows need, headlessly and deterministically, in an isolated NVIM_APPNAME against a real cargo crate: LSP attach (proved in both directions, by deleting lsp/gopls.lua and getting no client), server-initialized polling, neotest adapter registration and file claiming, DAP configuration count going 0 -> 5 once codelldb landed, merged conform formatter tables, Mason install state, and run commands executed with asserted stdout.
That is seven of the eight checkpoints demonstrated. Only debug genuinely needs more than a status read, because it wants a live session that stops on a breakpoint.
Proposal: promote the language rows from RUNBOOK to AUTO and build a 7-bundle x 8-checkpoint matrix (56 cells) where every cell is PASS, FAIL, a pre-declared N/A, or a pre-declared UNCOVERED, and none may be silently skipped. Reuses the existing isolated harness (tests/bundle_load.sh) rather than adding a parallel one; adds tests/capability.sh and fixtures under tests/fixtures/lang/.
Two method rules carried over from that session, both learned by getting them wrong: poll for the real signal rather than sleeping (rust-analyzer answers cargo check before the crate graph exists, which made a correct testables look broken), and suspect the harness before the code (two apparent bugs were a mis-called vim.notify stub and a probe hardcoding space as <leader> when it is \).
Full plan, phases, per-checkpoint break conditions and risks: dev-docs/language-matrix-test-plan.md. Contract being tested against: dev-docs/language-bundle-contract.md.
Gates the beta cut: zero FAIL, every N/A justified.
Activity
Chiarandini commentedon Sep 1, 2026
Done, and merged in #13.
The language rows are promoted from RUNBOOK to AUTO.
tests/capability.shgrades eight language rows against the eight checkpoints indev-docs/language-bundle-contract.md, in an isolated NVIM_APPNAME with its own XDG root, and it asserts the feature rather than its registration: a formatter that changes the buffer to known text, a deliberately broken symbol producing a diagnostic on the line it was injected on, one passing and one failing test reported as one of each, a debug session that stops on a breakpoint and reads a local, and a run command executed with its stdout compared.Green on CI:
Only debug needed more than a status read, and it needed no escape hatch: every language starts a session, stops on the expected line and reads a local.
The premise held. Grading against a written contract found four defects that read as correct in the source and that every existing gate passed over: Go and Java attached no language server, Python's debugger pointed at an interpreter without debugpy, and Rust reported passing tests as failed. It then found two more on its first Linux run, including an undeclared
cargo-nextestdependency that was only satisfied locally by accident.It runs on pull requests touching a language bundle, and weekly as a drift canary. Not nightly: everything it depends on lives outside this repository and moves slower than a day, and a check that is usually noise is one nobody reads.