From 5b54ed87a9778e4261f0c7240e513726c06cddb0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 7 May 2026 21:12:47 +1200 Subject: [PATCH 01/11] plan: grove --- docs/plans/grove.md | 244 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 244 insertions(+) create mode 100644 docs/plans/grove.md diff --git a/docs/plans/grove.md b/docs/plans/grove.md new file mode 100644 index 00000000..75a1fb1c --- /dev/null +++ b/docs/plans/grove.md @@ -0,0 +1,244 @@ +# Grove (v0) + +## Context + +A "grove" is a group of seedling nodes — one leader, N followers — sharing a small set of leader-published settings. Every grove member eventually receives the leader's signed state, gossiped peer-to-peer over a new ALPN; no RAFT, no quorum, eventually consistent. + +This v0 ships the smallest useful slice: the leader publishes typed **grove params** (name → value) inside a signed payload; each follower can independently map any local app param to a grove param, so when the grove value changes (or a mapping is created) the app param is updated through the same pathway as an operator `set_param`, firing `on_change`. While a mapping exists, manual local set/unset of that app param is rejected. + +Out of scope for v0, but design must not foreclose: +- Replicating whole app definitions/configurations (incl. backup-app pattern from the original ask). +- Secret grove params with envelope encryption to per-member pubkeys. + +Explicitly rejected (do not architect extension points for these; the design space is closed, not deferred): +- **Multi-grove membership per node.** The data model is single-row by design. +- **Grove-wide leader election / RAFT / failover.** No quorum, no election. If RAFT enters the picture later, the user's stated direction is that it would live *inside* a leadership group within a grove node, not span the whole grove — which is a different feature, not a generalisation of this one. v0 leader loss is a documented limitation with an out-of-band recovery path (see "Documented v0 limitations" below). + +User-confirmed decisions: +- One grove per node. +- Reject local set on grove-mapped app params. +- Reuse the existing OI Ed25519 identity (`data_dir/oi.key`) as the leader's signing key. New nodes can be onboarded via *any* current member; the leader fingerprint is supplied out-of-band so the joiner can verify the signature without having to talk to the leader directly. +- Spec lives at `docs/spec/grove.md` with a new `g[...]` namespace, registered in `.config/tracey/config.styx`. +- Common transport considerations (RPK TLS, fingerprint pinning, ALPN as hard-version-wall, JSON-line framing, abort semantics) are extracted into a new shared `docs/spec/transport.md` with a `t[...]` namespace, referenced from both `docs/spec/interface.md` (`bes.seedling/1`) and `docs/spec/grove.md` (`bes.grove/1`). Avoids duplicating the same prose across protocols and pre-pays the cost for any future ALPN. + +## Architecture + +### Code structure — extract a `transport` module first + +The code structure mirrors the spec split. Today `crates/core/src/oi/` holds both transport primitives (quinn endpoint construction, RPK TLS verifier, `TrustedKeys`, ALPN dispatch, JSON-line framing, per-connection state, fingerprint extraction) and OI-specific behaviour (port-forward, log streaming, event subscription, OI handlers). Grove would otherwise duplicate the transport half. + +Refactor (lands before any grove-specific code): +- New `crates/core/src/transport/` module with `endpoint.rs` (quinn endpoint, listen + accept loop, ALPN-keyed handler registry), `auth.rs` (protocol-scoped trust sets — `OperatorTrust` and a registry of additional sets keyed by ALPN — and a verifier that consults the set matching the negotiated ALPN), `connection.rs` (per-connection registry, fingerprint extraction, abort frame), `framing.rs` (JSON-line read/write helpers, currently inline in `oi/server.rs`). +- Stays in `crates/core` rather than promoting to a new `crates/transport` crate for v0; promotion is a follow-up if multiple crates ever need server-side transport. The wire-type half (ALPN constants, dial-side `OiClient`, identity keys) remains in `crates/protocol/`. +- ALPN handlers register with `transport` on startup: OI registers `bes.seedling/1`, grove registers `bes.grove/1`. The existing `oi/server.rs::handle_connection` body splits — generic per-connection logic moves to `transport`, OI-specific stream dispatch stays in `oi/`. + +`OiState` splits along the same seam: +- New `TransportState`: endpoint handle, connection registry, identity, protocol trust registry. Owned by a top-level `Daemon` struct (renamed/replacing the current top-level container that wires everything up). +- `OiState` (slimmed): OI-specific session/handler state — port-forward maps, log subscribers, event broadcasters. +- New `GroveState`: publish mutex, dial-loop handle, dirty-mappings set, signed-payload cache. +- `Daemon` holds `TransportState` + `OiState` + `GroveState`. Handlers receive only what they need (operator handlers don't see grove state, grove handlers don't see OI session state). + +This refactor is pure restructuring — no behavioural change, OI tests stay green. It lands as commit 0 (split into 0a/0b if the diff is large; see phasing). + +### Wire protocol — `bes.grove/1` + +- New ALPN constant `GROVE_ALPN = b"bes.grove/1"` in `crates/protocol/src/lib.rs`. +- Registered with the `transport` module's ALPN handler registry (see "Code structure" below). The transport endpoint advertises both `bes.seedling/1` and `bes.grove/1`; on accept, the negotiated ALPN selects the registered handler. No second port. +- Trust-set is protocol-scoped (see "Trust-set reconciliation" below). The verifier consults `OperatorTrust` for `bes.seedling/1` and `GroveTrust` for `bes.grove/1`; a key authorised for both must be in both sets. +- Transport tuning is shared; the existing 10s keepalive / 30s idle is acceptable for grove gossip. + +JSON-line framed bidi stream, using the framing helpers in `transport/framing.rs` (extracted in commit 0 from the existing `oi/server.rs::handle_bidi_stream` shape). + +Messages: +- `hello`: `{type, grove_id, our_seq, our_payload_hash, our_role, our_fingerprint, protocol_version, nonce}`. `protocol_version` is independent of the ALPN (ALPN = hard wall, in-payload version = soft feature flag). `our_payload_hash` is sha-256 of the canonical signed bytes; same seq + different hash → fork → drop + emit `grove.payload-rejected`. Mismatched `grove_id` or unexpected leader → drop. +- `version`: `{type, payload, signature}`. Sender is whoever holds higher seq. Receiver validates signature against its pinned `leader_fingerprint`, requires strict `seq` increase, and bounds the message at 256 KiB on `recv.read_to_end`. Lower bound is "smaller than OI's 4 MiB" — grove payloads stay tiny in v0. Leader publish path **pre-checks** the canonicalised payload size before signing; if `grove invite`, `grove revoke`, or `grove param set` would push the next payload over the cap (e.g. ≥ 240 KiB to leave headroom), the OI handler / CLI / web returns a structured error (`grove.publish-rejected { reason: "payload_too_large", current_bytes, cap_bytes }`) without bumping seq or persisting. Operators get the feedback at the point of mutation, not via a peer's `abort` after-the-fact. +- `peers`: `{type, entries: [{fingerprint, addresses, last_seen}]}`. Address-hint gossip only — *not* membership. On receive: filter to `members` from latest signed payload (drop entries from non-members), cap entries (e.g. 64) and addresses-per-entry (e.g. 4), treat peer-supplied `last_seen` as a tie-breaker, never as authoritative liveness. +- `abort`: `{type, reason}` where `reason ∈ {grove_mismatch, signature_invalid, seq_regression, leader_mismatch, version_too_old, payload_too_large, ...}`. Sent before close on protocol-level rejection so the operator gets a non-network reason via the event surface. + +Full-state-only for v0; revisit deltas when payloads might exceed a few MiB (i.e. when app definitions are replicated). + +### Signed payload + +Ed25519 signature over **JSON canonicalised with `serde_jcs`** (RFC 8785). JSON is idiomatic in this repo (used by OI, events, CLI), and a canonical JSON form keeps the schema flexible for an early-stage feature — adding or reordering fields in v1 doesn't break the wire encoding. + +Payload is a `serde`-derived struct serialising to canonical JSON: + +```jsonc +{ + "grove_id": "", // generated at `grove init` + "seq": 42, // u64, monotonic, leader-managed + "created_at": "2026-05-07T12:34:56Z", + "leader_fp": "", // of leader SPKI + "members": [ // sorted by fp at serialise time + { "fp": "", "label": "..." } + ], + "params": [ // sorted by name at serialise time + { "name": "...", "kind": "text", "value": "..." } + ], + "secrets": [] // v0: always empty. v1 adds envelope-encrypted entries; no schema break. +} +``` + +Bytes actually signed are `domain_sep || canonical_json_bytes` where `domain_sep = b"bes.grove/sig/v1\0"`. Domain separation is non-negotiable — Ed25519 signatures without it get confused across protocols. + +Encoder, signer, and verifier live in a new `crates/protocol/src/grove.rs` (alongside `keys.rs`) so `seedling-ctl` can verify during `grove join` without depending on `core`. Add `serde_jcs` as a dependency on `crates/protocol`. + +`created_at` validation: receive-time skew check of ±5 minutes. Reject outside window with `version_too_old`. Don't include unvalidated fields inside a signed envelope. + +### Database — migration v53 + +All five tables added in one migration block at the bottom of `crates/core/src/runtime/db.rs`. Never edit shipped migrations. + +- `grove_membership(id INTEGER PRIMARY KEY CHECK(id=1), grove_id BLOB NOT NULL, role TEXT NOT NULL CHECK(role IN ('leader','follower')), leader_fingerprint TEXT NOT NULL, current_seq INTEGER NOT NULL, current_payload BLOB NOT NULL, current_signature BLOB NOT NULL, joined_at TEXT NOT NULL)` — single-row. +- `grove_peers(fingerprint TEXT PRIMARY KEY, label TEXT, addresses_json TEXT NOT NULL, last_seen_at TEXT, last_connected_at TEXT)`. +- `grove_params(name TEXT PRIMARY KEY, kind TEXT NOT NULL, value TEXT NOT NULL, version_seq INTEGER NOT NULL)` — denormalised current params for query convenience. +- `grove_param_mappings(app_name TEXT NOT NULL, app_param_name TEXT NOT NULL, grove_param_name TEXT NOT NULL, PRIMARY KEY(app_name, app_param_name))` — local-only, never replicated. +- `grove_versions(seq INTEGER PRIMARY KEY, payload BLOB NOT NULL, signature BLOB NOT NULL, received_at TEXT NOT NULL)` — historical payloads for replay/debug. UNIQUE on `seq` makes duplicate-apply idempotent (`INSERT OR IGNORE` then check `changes()`). + +All mutations on the leader (publish path) wrap `(load current → mutate → bump seq → sign → persist payload + bump grove_membership)` in one `db.call` closure under a single `parking_lot::Mutex<()>` named `grove_publish_mutex` on `GroveState` (held by `Daemon`, see "Code structure"). + +### Trust-set reconciliation (load-bearing) + +After commit 0 the trust-set machinery lives in `transport/auth.rs` as a protocol-scoped registry, keyed by ALPN. Two strictly-separate sets, neither merged into a global "anyone trusted": +- **Operator-derived** (`OperatorTrust`): from `data_dir/authorized_keys` (loader stays in `oi/auth.rs`). Authorises *only* the OI ALPN (`bes.seedling/1`). +- **Grove-derived** (`GroveTrust`): from the latest signed payload's `members`. Authorises *only* the grove ALPN (`bes.grove/1`). Reconciled on every payload-applied event — additions and revocations both. + +A key authorised for both surfaces must appear in *both* sets (operator key on the leader's machine, also a grove member: explicit on both sides). Grove membership grants no operator authority and vice versa. + +The verifier in `transport/auth.rs` takes the negotiated ALPN and checks only the corresponding set. Constant-time comparison still applies, but only over the set relevant to the protocol being negotiated. + +On revocation, iterate the per-connection registry on `TransportState` and close any grove connection with `client_fp == revoked_fp`. Operator connections from the same fingerprint (if the key is also in `OperatorTrust`) are *not* closed — operator authority is a distinct grant. Without per-protocol close, a revoked member retains an existing grove connection until idle-timeout and can keep gossiping. + +### Operations surface + +CLI, OI handler, web — all three for parity. CLI subcommand tree under `crates/ctl/src/grove.rs`: + +Leader-only mutations (handler rejects on follower with "not the leader; current leader is ", regardless of which surface invoked it): +- `grove init` — generate `grove_id`, persist self-only seq=1 payload. +- `grove invite