059-digest: consume URLs, files, directories, stdin into knowledge graph

[ahoward]

Summary

brane digest — the universal intake command. Feed brane a URL, file, directory, or stdin and it extracts concepts, edges, and episodes through active lens filters.

Why

Today brane can only ingest code (AST extraction via brane ingest) and Claude Code session logs (brane ingest-sessions). It cannot consume arbitrary information — articles, docs, PDFs, markdown, raw text. Bunny's bny brane digest does this well but stores output as loose markdown files. We want the same intake capability but writing into brane's structured graph + episodes.

What

CLI

brane digest <source>          # file, directory, URL, or - for stdin
brane digest https://example.com/article
brane digest docs/architecture.md
brane digest src/              # recursive directory
cat notes.txt | brane digest -

Flags

• --lens <name> — override active lens for this digest
• --dry-run / -n — show what would be extracted without writing
• --agent <id> — agent ID for created episodes (default: cli)
• --json / -j — JSON output

Flow

1. Load source — URL (curl, SSRF-safe), file, directory (recursive with filters), or stdin
2. Stash raw source — store in body.db with hash for dedup (skip if already digested)
3. Load context — active lens prompt + relevant existing concepts (via vector search, not "load everything")
4. LLM extraction — send source + lens + context to Claude with structured JSON schema
5. Apply results — create/update concepts, edges, and episodes from LLM response
6. Return summary — concepts created/updated, edges created, episodes stored

LLM Response Schema

{
  concepts: { name: string, type: string }[],
  edges: { source: string, target: string, relation: string }[],
  episodes: { observation: string, context: string, tags: string[] }[],
  reasoning: string
}

Key Differences from bunny's digest

• Output goes into structured CozoDB graph, not markdown files
• Dedup via content hash (body.db), not source manifest
• Uses vector search to load relevant existing context, not entire worldview
• Lens is an extraction prompt that shapes LLM behavior, not just metadata

MCP Tool

digest: { source: string, lens?: string, agent_id?: string, dry_run?: boolean }

Depends On

Nothing — can start immediately. Uses existing LLM infrastructure (src/lib/llm.ts).

Acceptance

• Can digest a URL
• Can digest a local file
• Can digest a directory recursively
• Can digest stdin
• Dedup: re-digesting same content is a no-op
• Lens prompt shapes extraction
• Creates concepts, edges, and episodes
• MCP tool exposed
• Spike test passes