feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session recall (“skill-memory”) and historian auto-extraction. Previously skills had no recall; now opted-in skills append a cache-safe block on load and the historian writes global notes automatically.

• Transparent recall: the skill tool now exposes an optional intent; hooks stash it and append a rendered recall block. Ranking cascades from cosine over on-the-fly intent + stored delta_embedding, to FTS5, then recency×hit. Reads union the skill’s project partition with the global '*' tier; reads increment recall_count. Token budgeting counts per-note framing; pinned budget clamps to min(max_pinned_tokens, max_tokens).
• Write/recall tools: ctx_skill_note dedups by hash and cosine, embeds on write, and rejects writes when a skill hasn’t enabled memory. ctx_skill_recall renders notes on demand. Both are OpenCode-only; Pi does not register them. The historian emits <skill_observations>, and both OpenCode and Pi promote them as global '*' notes (source_type='historian').
• Provenance and safety: accepts plain paths and file://; Windows path normalization; name-based fallback when the trailing provenance line is truncated. Intent stash is keyed by sessionId:callId with TTL and per-session pruning. The skill tool’s model schema advertises the intent param via jsonSchema.
• UX and surface: async executeStatus and the TUI show per-project skill-memory totals. Prompt-surface and the A1 golden include the two new tools; budget fixtures were re-measured to reflect the larger surface.

Rollout

• Fork-lane migrations v10000–v10002 add skill_memory, embeddings + FTS, and historian provenance/global '*' unification. initializeDatabase self-heals (creates table/columns, rebuilds FTS) and runForkMigrations runs on DB open.
• Required action: enable per skill by adding skill-memory: { enabled: true } to SKILL.md. Optional: schedule the distill-skill-memory dreamer task (re-embeds NULL/stale vectors first, then outputs a read-only corpus health report).

^{Written for commit c7c70aa. Summary will update on new commits.}

Greptile Summary

This PR introduces skill-memory — a per-skill cross-session recall system. When a skill's SKILL.md opts in with skill-memory: { enabled: true }, accumulated notes surface in a cache-safe <skill-memory> block appended to the skill tool's result on every load. The historian can also auto-extract universal lessons during compaction and write them as global '*'-partition notes.

• Transparent recall (3-hook augmentation): tool.execute.before stashes the optional intent parameter; tool.execute.after parses provenance, reads frontmatter, populates the SkillLoadRegistry, and calls recallSkillMemoryBlock — a three-rung cascade (cosine over intent+delta embeddings → FTS5 → flat recency×hit) that appends the formatted block to output.output.
• Write paths: ctx_skill_note (agent-authored, OpenCode-only, hash + cosine dedup, enabled-guarded) and ctx_skill_recall (explicit recall on demand). Both share the recallSkillMemoryBlock core.
• Historian auto-extraction: the historian emits <skill_observations> blocks; both OpenCode (compartment-runner-incremental.ts) and Pi (pi-historian-runner.ts) promote observations as global '*' notes (source_type='historian') via promoteSkillObservations. Three fork-lane migrations (v10000–v10002) with an initializeDatabase self-heal net round out the schema work.

Confidence Score: 5/5

• The change is safe to merge. All four previously-flagged issues (intent-stash keying, redundant dynamic import, FTS self-heal rebuild, and historian/project-local orphaning) are correctly addressed. The two remaining comments are non-blocking quality observations.
• The implementation is well-structured and thoroughly tested. The recall cascade, migration runner, self-heal path, and both runners' promotion gates are all sound. The one behavioral asymmetry (historian writes notes for skills without enabled: true, while ctx_skill_note refuses to) is a contained design inconsistency with no user-visible impact beyond mild DB and stats inflation.
• packages/plugin/src/features/magic-context/skill-memory/promote.ts — the historian promotion path bypasses the opt-in enabled check that ctx_skill_note enforces. No other files require special attention.

Important Files Changed

Filename	Overview
packages/plugin/src/features/magic-context/skill-memory/storage.ts	New file — core DB layer for skill-memory. The recallPartitionPredicate union approach correctly surfaces historian-written global notes for project-local skills. Double invocation of recallPartitionPredicate in searchSkillMemoryFts is a minor style issue; all other query logic is correct.
packages/plugin/src/features/magic-context/skill-memory/promote.ts	New file — historian promotion path. Best-effort per-item error handling is correct. However, promoteSkillObservations inserts notes for any skill that appears in a TC marker regardless of whether skill-memory is enabled in its SKILL.md, creating orphaned rows and inflating TUI stats counters.
packages/plugin/src/features/magic-context/skill-memory/recall.ts	New file — three-rung recall cascade (cosine embedding → FTS5 → flat recency×hit). Correct bm25 ASC ordering, budgetFill preserves the pinned-first invariant, and errors are now logged rather than silently swallowed. No issues found.
packages/plugin/src/features/magic-context/fork-migrations.ts	New file — fork-lane migration runner. Version-floor enforcement, per-row presence checks (not watermark), idempotent DDL, and sibling-conflict handling all look correct. The three migrations (v10000–v10002) are self-consistent with the initializeDatabase self-heal net.
packages/plugin/src/hooks/magic-context/hook-handlers.ts	Adds intentByCallId stash (composite sessionId:callId key with TTL + cap), maybeInjectSkillMemory, and createToolExecuteBeforeHook. The previously-flagged duplicate import was hoisted; session-delete pruning uses prefix matching. No new issues found.
packages/plugin/src/features/magic-context/storage-db.ts	Adds skill_memory table, FTS virtual table, and triggers to initializeDatabase; calls runForkMigrations from both openDatabase and openDatabaseAsync; includes guarded FTS rebuild for the self-heal gap (as requested in previous review). Logic is correct.
packages/plugin/src/tools/ctx-skill-note/tools.ts	New agent-callable tool. Enabled-guard (added in 3a1733d), exact-hash dedup, cosine dedup with model-version guard, and race-condition fallback are all correctly wired. originProject is tracked separately from the partition key for global-tier skills.
packages/plugin/src/tools/ctx-skill-recall/tools.ts	New agent-callable explicit recall tool. Registry-first + disk-fallback resolution order is correct; enabled-guard is present; test-seam injection uses internal cast (not public API pollution). Cold-start message is distinct from "no notes" message. No issues found.
packages/plugin/src/features/magic-context/skill-memory/provenance.ts	New file — parses "Base directory for this skill:" provenance line (handles both file:// and plain path), derives tier/source, and implements name-based ancestor-walk fallback. Windows normalization and worktree-root stopping condition are handled. Boundary conditions (HOME exclusion, filesystem root) are correct.
packages/plugin/src/hooks/magic-context/compartment-parser.ts	Adds <skill_observations> block parsing. The pipe-delimited regex correctly captures skill-id, kind, and full lesson text (including pipes in the lesson). Kind validation against SKILL_OBS_KINDS and unescapeXml on both skill-id and lesson are correct. Invalid items are silently dropped, which is the right behavior.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant SkillTool as skill tool
    participant BeforeHook as tool.execute.before
    participant AfterHook as tool.execute.after
    participant ProvenanceReg as SkillLoadRegistry
    participant RecallDB as skill_memory (DB)
    participant CtxSkillNote as ctx_skill_note
    participant Historian as Historian

    Agent->>SkillTool: "call skill("council", intent="...")"
    BeforeHook->>BeforeHook: stashIntent(sessionId:callId, intent)
    SkillTool-->>AfterHook: output with "Base directory: ..."
    AfterHook->>AfterHook: parseSkillProvenance(output)
    AfterHook->>AfterHook: parseFrontmatterConfig(SKILL.md)
    AfterHook->>ProvenanceReg: "set(sessionId:skillId, {tier, frontmatterConfig})"
    AfterHook->>AfterHook: getAndDeleteIntent(sessionId:callId)
    AfterHook->>RecallDB: recallSkillMemoryBlock(skill, intent, tier, projectIdentity)
    RecallDB-->>AfterHook: "<skill-memory> block (or "")"
    AfterHook->>SkillTool: append block to output.output
    SkillTool-->>Agent: "skill content + <skill-memory> block"

    Agent->>CtxSkillNote: ctx_skill_note(skill, intent, kind, delta)
    CtxSkillNote->>ProvenanceReg: getSkillLoad(sessionId, skill)
    CtxSkillNote->>RecallDB: hash-dedup + cosine-dedup → insertSkillMemoryNote
    RecallDB-->>CtxSkillNote: id
    CtxSkillNote-->>Agent: "Skill note saved (id=...)"

    Historian->>Historian: "parse <skill_observations> from chunk"
    Historian->>RecallDB: promoteSkillObservations(db, originProject, obs[])
    Note over RecallDB: tier='global', project_identity='*', source_type='historian'

Loading

_{Reviews (37): Last reviewed commit: "style: apply pinned biome formatting aft..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>