AgentProfileDiff: add behavioral axes (carrier, loop, outputContract, verifier, sampling) — measured as the binding constraints

[drewstone]

Ask

Extend AgentProfileDiff to serialize the behavioral axes the policy compiler already searches but the portable artifact cannot express: carrier (where guidance is injected), loop (maxTurns / retryOnEmptyOutput / verifyEveryTurn / stopRule), outputContract (schema / exact filename / validator), verifier (checks / repairPolicy), sampling (one long run vs k short). Today the diff covers the static surface (prompt, model, tools, mcp, resources, permissions); the compiler optimizes over axes it then can't ship.

Why — measured on two real product domains (deepseek-v4-flash, paired/replicated)

The same executable policy (mounted checker + run-it instruction):

Domain	Adoption (agent actually ran it)	Score effect	Cause
Tax 1040 (guidance via production prompt path)	84–99%	flat (+0.002–0.028, p>0.3)	adopted; errors are epistemic, invisible to a self-check
Harvey legal (guidance as tail ## Learned guidance)	4% (3/76)	flat (+0.021, p=0.75)	never adopted; 75% cap-die at the 8192 output cap before any finish gate

Identical content, adoption swung 84%→4% purely by injection point. A chunked-write variant made no difference (still tail-injected). Better policy text cannot fix a delivery problem — carrier and loop are the binding axes, and neither is serializable today.

Each axis + its ablation neutralizer (feeds the ablation-gate spec)

• carrier: hold text fixed, move to baseline injection point → held score proves content, not placement, mattered
• loop: revert turns/retries/verify to baseline → held score = loop change inert
• outputContract: strip specific rules, keep the shell → held score = presence not content
• verifier: disable the checks → held score = caught nothing real
• sampling: collapse to k=1 → held score = averaging, not policy, moved the mean

Proving ground is live (verified 2026-07-03)

tax-agent-clean51 run_taxcalc_loop.ts --profile-diff <file.json> applies a typed diff onto the real taxAgentProfile via mergeAgentProfiles, mounts its inline resource files, and records profileDiff.axes+hash for per-axis ablation. A diff-driven run graded by_line 0.632 with the mounted checker executed in-sandbox; a 139-cell 3v3 reproduced parity with the bespoke-flag path at 99% adoption. Ready to be the behavioral-axis test bed the moment the type extends.

Blocker

The diff API (applyAgentProfileDiff, pruneAgentProfileDiff, schemas) is unpublished — registry @tangle-network/agent-interface@0.18.0 ships only mergeAgentProfiles; the full API lives on branch feat/agentprofile-diff (self-labeling 0.18.0 = version collision). Publishing it lets the product consumer swap its thin local adapter for the real import.

Full evidence memo: agent-lab/papers/EVIDENCE-behavioral-axes-2026-07-03.md.

[drewstone]

Third confirmation, and it sharpens the ask: tools/mcp schema is the decisive carrier axis for small models

Ran the compile-and-prove pipeline on agent-lab's crit-create (EOPS) — an information-dominant task where handing the agent the transaction schema measurably recovers 86–92% of the lift (vs tax which sits below its noise floor). The compiler authored a correct KB and mounted it as resources.files (skill markdown). Result: full arm == baseline == 0.000 across 18 cells — the small model (deepseek-v4-flash) never opened the mounted file (verified in traces).

The measured 86–92% recovery on that exact task delivers the same information via the tool/function schema (auto-visible in tools-list), not a mounted file. Same content, different carrier, opposite outcome.

Crossed with three surfaces now:

Surface	Carrier	Adoption
Tax	system-prompt injection	99%
Harvey	tail "learned-guidance" block	4%
crit-create	mounted skill file (resources.files)	~0%
crit-create (measured working arm)	tool schema	86–92%

Implication for AgentProfileDiff: resources.files is the wrong artifact shape for small models on a tools-surface harness. The highest-leverage carrier is the tool/mcp schema (function descriptions the model reads automatically). This makes the tools/mcp axis — and where knowledge is injected — the first-class optimizable thing, exactly the behavioral-axis ask above. A compiler that emits mounted files will keep failing to promote on tool-surface agents no matter how good the content is.

Concrete addition to the ask: a carrier dimension on the diff (system-prompt | tool-schema | mounted-file | contract), with the ablation neutralizer holding content fixed while swapping carrier — that isolates "content" from "channel", which this data shows is the dominant variable.

[drewstone]

Demonstrated: authored knowledge on the tool-schema carrier causally lifts a small model +0.54

The behavioral-axes ask above is no longer just argument — here is the end-to-end causal result on agent-lab's crit-create (EOPS) with deepseek-v4-flash.

The self-improving loop, every link with a number:

1. Diagnosis — the agent-eval analyst, reading real failure traces, named the exact root cause: "category defaults to 'inquiry-help', should be 'hardware'; tool-schema enforcement gap on create_incident." Strictly better than score-band clustering.
2. Author + deliver — that fix, authored as tool/field descriptions and delivered on the tool-schema carrier (the composite tool's schema in tools/list), lifted the model:

Arm (same carrier, same fields, only content differs)	mean (12 tasks)
authored docs, correct (category=hardware)	0.625
authored docs, deliberately wrong (category=inquiry-help)	0.25
no docs (raw tools baseline)	0.083

+0.54 lift, and provably causal: only the content differs between 0.625 and 0.25, so the authored content — not the carrier's presence or the tool's built-in docs — is the driver.

Why this nails the axis ask

The same knowledge delivered as a mounted resource file (resources.files) scored 0.00 on this task — the small model never opened it. Delivered as tool-schema it scores 0.625. That is the carrier-dominance result in one controlled comparison: for a small model, resources.files is the wrong artifact shape; the tool/mcp schema is the decisive carrier.

Concrete restated ask for AgentProfileDiff:

• A first-class carrier dimension (tool-schema | system-prompt | resource-file | contract) so a proposer can author onto the channel the model actually reads.
• The neutralizer that made this rigorous: hold content fixed, blank it byte-for-byte, and require the lift to vanish (kill-all ⇒ decorative). agent-eval has gates that prove "beat baseline" but none that prove "the content, not the added prompt bytes, caused it" — that placebo control is the one primitive worth upstreaming alongside the axis.

Artifacts + method: agent-lab grid-r326 harness, EOPS_COMPOSITE_DOCS_FILE carrier overlay (committed agent-lab 78619677), GRID_ARM=I docs channel.