Skip to content

feat(stack-encrypt): passthrough, Index/Indexes, Encrypted<Terms> and field types for the plan builder - #1069

Merged
coderdan merged 7 commits into
fix/stack-encrypt-target-owned-plaintextfrom
feat/stack-encrypt-engine-plan-pieces
Oct 5, 2026
Merged

coderdan merged 7 commits into
fix/stack-encrypt-target-owned-plaintextfrom
feat/stack-encrypt-engine-plan-pieces

Conversation

@coderdan

@coderdan coderdan commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

stack-encrypt is our Rust library for field-level encryption: every value is sealed under its own data key from ZeroKMS (our key management service), and can carry search terms (equality, full-text match, order) beside the ciphertext so an encrypted column can still be queried. The planned plan builder (#1057) puts one chained front door on its engine, and is meant to be a thin translation onto the engine's existing building blocks. This PR adds the engine pieces that translation needs and did not have: indexes as types, a passthrough operation, a way to run a decryption held in a variable, an output type for "one value plus its terms", and a declared type per field for language bindings whose host language has no types.

This PR is stacked on #1068 (which lets a value that cannot be copied run through the engine). Its base is that branch, so until #1068 merges the diff here shows only this PR's seven commits; once #1068 merges, GitHub retargets it and it still shows only its own.

Breaking change. stack-encrypt 0.2.0 is published on crates.io, and this PR removes or renames public names, so the next release that carries it must be 0.3.0. Three commits are marked breaking (! and a BREAKING CHANGE: footer): the MatchTerm rename, the removal of dynamic::TermKind, and the field-type swap. packages/stack-encrypt/CHANGELOG.md's ## [Unreleased] section (added by #1068) records each break and each addition.

Changes

Indexes as types (src/target/index.rs, new)

  • Index<S>: one index (Equality, Match, Ore, Ope) as a type, implemented only for the plaintexts its scheme is defined over. A match index on an integer is a compile error. spec() lowers an index to data (IndexSpec) for the FFI and saved plans; a match index's tokenizer and filter options travel inside it.
  • Indexes<S>: one index, or a tuple of two to four. Deliberately not (), so "an indexed field with no index" does not compile; that field is ciphertext() alone.
  • indexed::<S, K, M, X>(indexes): the ciphertext().accepting().zip(..).map(..) composition every caller spelled by hand, done once. Works in both source modes from fix(stack-encrypt): let a non-Clone plaintext run a target operation #1068 (owned mode needs S: Clone, because the ciphertext and the terms share the value).
  • Encrypted<Terms> { ciphertext, terms }: the output. Terms are a tuple read by destructuring, let (eq, ore) = out.terms;, with no named accessors. Decrypting it opens the ciphertext.
  • Indexes::select::<I, _>(): picks one index for the query side (a term with no ciphertext). Its rustdoc now shows, as a doctest, the Select<I, P> bound generic code needs; the position parameter is named P so it no longer reads as the At<N> struct.

Engine operations (src/target/operations.rs)

  • passthrough::<S, K, M, Ctx>(): returns the source unchanged and ignores the context (moves it in owned mode, clones it in borrowed mode). Its rustdoc says it is not authenticated and that a field that must be readable but tamper-evident should be sealed with an equality index instead.
  • KeysetCipher::run_decryption and StackCipher::run_decryption: run a Decryption held in a variable, the counterpart of the existing KeysetCipher::run. Documented together with run.

A declared type per field (src/dynamic/kind.rs, new; src/dynamic/record.rs)

  • The type vocabulary is vitaminc's ValueKind, not a stack-encrypt enum: bool, int32, int64, uint32, uint64, float32, float64, string, bytes, array, object. vitaminc is the cryptography library stack-encrypt is built on, and it owns the value model (FfiValue) and its frozen leaf-tag table. vitaminc 0.5.1 publishes ValueKind with its names, FromStr, tags, holds and FfiValue::kind(). stack-encrypt re-exports it as stack_encrypt::dynamic::ValueKind. (An earlier revision of this PR defined its own FieldType enum duplicating all of that; it is gone.)
  • stack-encrypt adds only the two rules vitaminc leaves to the declaring layer, as free functions in stack_encrypt::dynamic, beside term and context: admits(kind, &IndexSpec) -> bool (which indexes a kind is defined for) and read(kind, FfiValue) -> Result<FfiValue, Error> (read a query value as a kind). Free functions rather than an extension trait: bindings call them with a kind in hand, and a trait would need importing to be found.
  • Plan grammar gains an optional "type" key per field, parsed with ValueKind's FromStr. Building a plan fails on an unknown type name, or on an index the type is not defined for. Encrypt refuses a value whose tag is not the declared type (before any key request); decrypt refuses a field that opens to another type. read converts a number only when the conversion is exact (a JavaScript 34.0 becomes the u64 34 for a uint64 field), and refuses a NaN in both float directions.
  • An indexed field without "type" is still dispatched on each value's own tag. This is transitional, and the plan() rustdoc and the dynamic module docs now say so; stack-encrypt: a plan field with an index but no "type" trusts the binding to tag every value the same way — require "type" once the Go binding declares it #1082 tracks requiring "type" on every field with a term output once the Go binding fills it from struct types.
  • The engine does not call read. Its rustdoc says so: a binding deriving a query term for a typed field must call it first, and wiring it into query(v).using(&plan) is tracked in stack-encrypt: one chained plan builder as the single front door, so writes, queries and reads cannot drift #1057.

Dependencies

  • Every vitaminc crate moves to 0.5.1 (all published on crates.io): the pins in the root Cargo.toml and in languages/golang/stackencrypt/guest/Cargo.toml, and the root and guest lockfiles. The workspaces that build a root crate by path inherit the requirement, so their lockfiles move too, vitaminc entries only, manifests unchanged: packages/eql (whose test:encryption task runs --locked and would otherwise fail), the three fuzz crates and the stackauth guest. protect-ffi's lockfile needs no change.

One data form for an index (src/target/index.rs, src/dynamic/{term,record,kind,mod}.rs, Go guest ops.rs/status.rs)

  • IndexSpec is the only data form of an index. The older dynamic::TermKind enum is removed, along with its From<TermKind> for IndexSpec conversion. Everything that took a TermKind now takes an IndexSpec: Output::Term, Error::Term, Scalar::of, dynamic::term and dynamic::admits (which looks at the kind only; a match index's options do not change which types it applies to), and the Go guest's term ABI (its TERM_MATCH code means a match index under default options).
  • Match options now have a wire form instead of being impossible to send. In a plan's "outputs" list an index is still its key string ("eq", "match", "ore", "ope"), and a bare "match" still means the default options (3-grams, downcased, k = 3, m = 256). So every plan the Go binding sends today parses exactly as before, and the Go side is unchanged. A match index under other options is a one-entry object:
    { "match": { "tokenizer": "standard" | { "ngram": <length> }, "downcase": <bool>, "k": <int>, "m": <int> } }
    
    Every option is optional and defaults as above. Unknown or repeated options, and values the match scheme refuses (k outside 3..=16, m not a power of two in 32..=65536, an n-gram length of 0), are a plan error. IndexSpec::to_value writes the bare key whenever the options are the defaults and the object otherwise; IndexSpec::from_value reads either. The shape is documented on dynamic::record::plan.
  • The dynamic term derivation now honours a match index's options (through a crate-private match_terms_under), so a stored term and a probe built from the same plan agree. Before, every dynamic match term used the defaults.
  • A field still names each output key once, so two match indexes under different options in one field are refused: both would be stored under the one "match" key.

Rename (src/sem/mod.rs and every user)

  • MatchTerm is now MatchTerms (and TermBytesError::OddMatchTermLength is OddMatchTermsLength). MatchTerm stays as a deprecated type alias (#[deprecated(since = "0.3.0")]), so code that names the type still compiles; the enum variant cannot have an alias. A match index produces a set of terms: each token contributes several Bloom-filter positions, and a query matches when its positions are a subset of the stored ones. The singular read like EqualityTerm and OreTerm, which are each one comparand. The plural keeps the family prefix so it still reads beside them. Bytes and the matching() constructor are unchanged.

Verification

All run at the current head on Rust 1.94.1 (the mise.toml pin), after rebasing onto #1068's new head (639858b):

  • cargo fmt --all --check: clean.
  • cargo clippy --locked --no-deps --workspace --all-targets --all-features -- -D warnings (the CI command): clean.
  • cargo clippy --locked -p stack-encrypt --no-default-features --all-targets -- -D warnings: clean.
  • mise x --env test -- cargo nextest run -p stack-encrypt -p stack-encrypt-derive --all-features: 382 passed, 0 failed (includes the trybuild suite).
  • mise x --env test -- cargo nextest run --locked --workspace --all-features: 823 passed, 0 failed.
  • mise run test:doc: all pass (new doctests on admits, read and the generic select).
  • mise run doc and RUSTDOCFLAGS="-D warnings" cargo doc -p stack-encrypt --no-deps --no-default-features: clean.
  • mise run wasm:guest:test (the Go stackencrypt guest, now on vitaminc-aead-value 0.5.1): 50 passed, clean.
  • mise run wasm:auth-guest:test (the stackauth guest, lockfile refreshed): 12 passed.
  • EQL, from packages/eql: cargo test --locked -p eql-bindings --features stack-encrypt, cargo test --locked -p eql-encryption-tests and cargo clippy --locked -p eql-bindings -p eql-encryption-tests --all-features --all-targets -- -D warnings: all pass (that is the test:encryption task). Before the lockfile refresh, the first of these failed with cannot update the lock file ... because --locked was passed.
  • Fuzz crates (packages/{stack-auth,stack-encrypt,stack-kms}/fuzz, detached): cargo check --locked --bins: all compile.
  • cargo mutants --no-shuffle -p stack-auth -p stack-encrypt --in-diff <diff against 639858b61, the new #1068 head> (the CI gate): 137 mutants: 100 caught, 37 unviable, 0 missed.
  • The new decrypt test was shown to bite: changing .zip(&opened) to .zip(plan.fields.iter()) in decrypt makes decrypt_skips_an_index_only_field_when_it_checks_types fail.

New tests in this round: decrypt_skips_an_index_only_field_when_it_checks_types, a_typed_composite_field_round_trips_even_when_empty, check_record_accepts_a_record_sealed_as_another_type, the_plan_reads_a_type_given_before_the_outputs (all in src/dynamic/record.rs), and read_refuses_a_float_at_the_first_value_past_an_integer_range and read_refuses_a_nan_in_both_float_directions (in src/dynamic/kind.rs). The tests that checked FieldType's names, its tag table and holds are dropped: that is vitaminc's job now, and vitaminc tests it.

Byte identity, the load-bearing claim, is tests/index.rs: indexed::<String>((Equality, Ore)) gives the same equality and ORE terms as the hand-composed ciphertext().accepting().zip(equality()).zip(ore()) and as a #[derive(EncryptFrom)] struct with the same fields, under the same context. Ciphertexts are sealed under fresh data keys, so they are never byte-equal across two runs; the test instead opens each path's ciphertext with the other path's reader. Compile-fail tests (tests/ui/indexes_empty_set.rs, match_on_integer.rs, match_in_a_tuple_on_integer.rs) pin that () and a match index on an integer are rejected, and tests/ui/pass/indexes.rs pins the sets that must compile.

Related

Review notes

@changeset-bot

changeset-bot Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 27fe042

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught missed unviable timeout
100 0 50 0

Every mutant in the changed lines was caught by a test.

Comment thread packages/stack-encrypt/src/target/index.rs
Comment thread packages/stack-encrypt/src/dynamic/record.rs
Comment thread packages/stack-encrypt/src/dynamic/field_type.rs Outdated
Comment thread packages/stack-encrypt/src/target/index.rs
Comment thread packages/stack-encrypt/src/target/index.rs
Comment thread packages/stack-encrypt/src/dynamic/record.rs Outdated
Comment thread packages/stack-encrypt/src/target/index.rs Outdated

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: 20eb943 (three commits, diffed against fix/stack-encrypt-target-owned-plaintext)

Comment only. Each finding is also an inline comment on the line.

Blocking

None.

Should fix

  1. src/target/index.rs:88 (IndexSpec::key) with src/dynamic/term.rs:80. IndexSpec::Match(options) has no data form that keeps its options: key()/Display render any Match as "match", From<TermKind> for IndexSpec is one-way and always yields DefaultMatch's options, and the plan grammar has no key for match options. Scenario: the builder lowers a saved Match::<Words> field via spec(), writes "match" to the wire, and the query side re-reads it as DefaultMatch; different k/m/tokenizer, different Bloom positions, contains false for every row. A silent zero-row miss with no error, the #1051 drift class. Either an explicit inverse that refuses a non-default Match, or a doc on key() saying it is the output key and not a serialisation, and that non-default MatchOptions has no wire form yet.

Design points (not blocking; both are known to the PR body, recorded so they are tracked)

  1. src/dynamic/record.rs:343. "type" optional means an untyped indexed field is still dispatched on the value's own tag, so for that field the binding's tagging is trusted, which is what #1056's "engine verifies rather than trusts the binding" and the plan doc's "build() refuses an index on a field whose type it cannot resolve" were meant to end. Scenario: a JS host writes 34 as Float64 and later as Int64 under one "ore" field; both accepted, two different ORE terms, a query finds half the rows. The Go-guest reason is sound for now; a follow-up issue to require "type" on term-bearing fields once #1046 fills it, plus a visible note that this is transitional, would keep it from becoming permanent.
  2. src/dynamic/field_type.rs:185 (FieldType::read). Nothing in the engine calls it; dynamic::term() takes Scalar + TermKind and never sees the FieldPlan, so a binding that forgets read derives a float term against a uint64 field and matches nothing, silently. Belongs with query(v).using(&plan) in #1057, but worth naming there so it is not discovered by the first dynamic query through a typed field.

Nits

  1. src/target/index.rs:398. Indexes stops at four; #1056 says "two or more". Match<A> + Match<B> + three scalar indexes, or the JSON index (#1060), make five plausible. One macro arm.
  2. src/target/index.rs:131. Index::operation requires M: ConsumeSource, but matching() is a view operation needing only SourceMode, so a borrowed match query over a non-Clone text type compiles through matching() and not through Index::operation.
  3. src/dynamic/record.rs:2359. Test names the query term probe; CONTEXT.md on #1052 lists "probe" under Avoid for Query.
  4. src/target/index.rs:297. The "a set holding an index twice does not compile" claim (which the PR body leans on for build() refusing repeats) has no tests/ui compile-fail pinning it, unlike ().

Checked and found fine

Byte identity: tests/index.rs compares equality and ORE terms across indexed, the hand composition and the derive under one CallerContext, and cross-opens each path's ciphertext with the other's reader under the same AEAD half, so descriptor and AAD are covered (a wrong descriptor fails the key retrieve, a wrong AAD fails the open); owned mode asserts one generate_keys call. Indexes is implemented for the four index types and 2/3/4-tuples only; () and Match on u32 (alone and inside a tuple) are pinned compile-fail. passthrough rustdoc says not encrypted and not authenticated and points at indexed + Equality; its build is Pending::ready(cipher, Ok(M::take(source))), ignoring the context, with no key request (tested under () as the context). run_decryption on keyset and client, ForeignKeyset refusal tested. FieldType covers exactly vitaminc's ten non-null tags (tags.rs 0x02..0x0B) plus array/object; of/holds are exact per variant (Int32 vs Int64 vs UInt*, Float32 vs Float64, Bytes vs String); admits is TermKind::supports over types and the test cross-checks every (type, kind) pair; read converts numbers only when exact (i128 widening, fract()==0, f as i128 == i, f32 round-trip) and never across kinds. Encrypt refuses a mismatched tag in check_field before build_row, so before any key request (asserted via the counting KMS); check_source agrees; decrypt checks each opened value against its own field in plan order (record_leaves takes fields by plan order, so the zip(&opened) is aligned). Go module and EQL bindings reference no Rust MatchTerm; the Go MatchTerm is Go's own type. No Linear id in commits, body or diff; all three commits GPG-signed (Good signature). Locally, in a worktree of the head: nextest 369/369 (incl. trybuild), doctests 19/19, cargo doc -D warnings clean, cargo mutants --in-diff over the PR diff 105 mutants: 74 caught, 31 unviable, 0 missed.

coderdan added a commit that referenced this pull request Oct 5, 2026
Decrypt through a plan is open, because StackCipher::decrypt already
exists with another signature. A plan has two starts and takes its
context from exactly one of three sources. The typed verb encrypt_into
and the picker are Rust-only. The derive emits the plan only after its
grammar is narrowed and the plan's widened, so the two stay one grammar.
EQL types are assembled per language from standard outputs, with no
registry and no target name in the data grammar.

The lowering table now names what #1068, #1069 and #1071 shipped.
@coderdan
coderdan added this pull request to stack #1075 October 5, 2026 05:11
@coderdan
coderdan force-pushed the feat/stack-encrypt-engine-plan-pieces branch from 20eb943 to 2f2cb81 Compare October 5, 2026 05:24
@coderdan

coderdan commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (014a04a) with no conflicts and no content change; commit hashes changed, nothing else.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-05T06:04:37.404951Z 2f2cb81 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-05T06:09:22.975442Z 2f2cb81 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Comment thread packages/stack-encrypt/src/dynamic/field_type.rs Outdated

@cipherstash-bot cipherstash-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: 🟡 merge after changes (1 of 4 review job(s) failed)

Before merging, add one test for the type check in dynamic::decrypt. The other findings can wait.

No test has an index-only field before a typed field. A likely change to how decrypt pairs opened values with fields passes every existing test. The findings that can wait are a record of the MatchTerm rename before the next crate version bump, one wrong rustdoc claim on Indexes::select, and two optional tests.

I verified each finding in a scratch worktree. Every proposed test passes on this commit. No finding repeats a point that the PR discussion already raises or answers. Those points include IndexSpec::key and match options, the optional "type" key, the unused FieldType::read, the four-index limit, the ConsumeSource bound, and the duplicate-select compile-fail test.

No security findings.

Other findings not posted as comments

  • Optional: FieldType::read treats a NaN differently in the two float directions. read(Float64, Float32(NaN)) returns Float64(NaN). But read(Float32, Float64(NaN)) refuses the value. Only the refusal has a test (read_refuses_a_number_a_float_type_would_round). The exact_f32 doc says "A NaN is never exact", but exact_f64 accepts a NaN. Decide which result you intend for each direction. Then add a test for it. packages/stack-encrypt/src/dynamic/field_type.rs:254
  • Optional: no test shows that check_record accepts a record sealed as another type. The rustdoc says that only decrypt can check the type, because the type tag is inside the encryption envelope. A test would make sure that this sentence stays true. packages/stack-encrypt/src/dynamic/record.rs:610
  • Optional: no parse test puts "type" before "outputs" in a field spec. Every test puts "type" last. The parser applies the type after the key loop. So key order does not matter today, but no test shows this. packages/stack-encrypt/src/dynamic/record.rs:343
How this review was made
Agent Model Review type Result
claude claude-opus-5-5 test-gap 3 found, 3 posted
claude claude-opus-5-5 rust 2 found, 2 posted
codex gpt-5.6-sol test-gap failed
codex gpt-5.6-sol rust 0 found, 0 posted

Synthesis: claude-opus-5-5 merged the findings, removed duplicates and dropped findings it could not confirm in the code. 0 posted finding(s) were raised by two or more models.

Plain language: claude-opus-5-5 read every comment as a new reader would. 3 comment(s) had a problem that stopped the reader acting; it rewrote 3. It also rewrote the review body.

Stack: position 2 of 5 (#1068, *️⃣ #1069, #1071, #1073, #1074). *️⃣ marks this pull request.

Context loaded: the description, 4 linked issue(s) and 18 discussion entries.

Comment thread packages/stack-encrypt/src/dynamic/record.rs
Comment thread packages/stack-encrypt/src/sem/mod.rs
Comment thread packages/stack-encrypt/src/target/index.rs Outdated
Comment thread packages/stack-encrypt/src/dynamic/record.rs Outdated
Comment thread packages/stack-encrypt/src/dynamic/field_type.rs Outdated

@auxesis auxesis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderdan nice one, thanks for this.

Approving, but will need to see that test coverage issue @cipherstash-bot found addressed before merge.

@coderdan

coderdan commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Folded dynamic::TermKind into IndexSpec (9d10ee7), per the review on index.rs and the plan in #1063's latest comment: IndexSpec is now the only data form of an index. Match options get a wire form: an index in a plan's "outputs" is still its key string and a bare "match" still means the defaults (so the Go binding's plans are unchanged), and a non-default match index is {"match": {"tokenizer", "downcase", "k", "m"}}, documented on dynamic::record::plan. The dynamic term derivation now honours those options; a field still refuses two match outputs (one "match" key). The Go guest swaps the type; no Go code changed. Checks: workspace nextest 819 passed, doctests, rustdoc, clippy (all features and no default features), wasm:guest:test 50 passed, EQL eql-bindings tests, and cargo-mutants on the new commit (66 mutants: 54 caught, 12 unviable, 0 missed). The PR body is updated. #1071, #1073, #1074 and #1076 are rebased onto this head. IndexSpec stays a closed enum until #1063 replaces it with an open record.

@coderdan

coderdan commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Review feedback addressed. New head: 7fe56ec (was 9d10ee7), rebased onto #1068's new head 639858b.

  • FieldType is gone; the field type is vitaminc's ValueKind (d284f62, refactor(stack-encrypt)!: the field type of a plan is vitaminc's ValueKind). It is re-exported as stack_encrypt::dynamic::ValueKind. stack-encrypt keeps only the two rules vitaminc leaves to the declaring layer, as free functions: dynamic::admits(kind, &IndexSpec) and dynamic::read(kind, value). The module is now src/dynamic/kind.rs.
  • vitaminc 0.5.1. Every vitaminc crate moves to 0.5.1, in the root workspace and in the Go stackencrypt guest (pins and lockfiles). Lockfiles that build a root crate by path move too, vitaminc entries only: packages/eql (its --locked test:encryption task failed without it), the three fuzz crates and the stackauth guest.
  • Breaking marks. refactor(stack-encrypt)!: with a BREAKING CHANGE: footer on the MatchTerm rename (76253d3, now with a deprecated MatchTerm alias), the TermKind removal (cacc26d) and the ValueKind swap. packages/stack-encrypt/CHANGELOG.md's ## [Unreleased] section lists each break and addition, so the next release is 0.3.0.
  • Docs (325d2b0):
  • Tests and one fix (7fe56ec):
    • the index-only-before-typed decrypt test (shown to fail if values are paired with every plan field);
    • the typed composite round trip, empty and not;
    • check_record accepting a record sealed as another type;
    • "type" before "outputs";
    • read at the first float past each integer range;
    • read now refuses a NaN in both float directions, with a test.
  • Issue filed: stack-encrypt: a plan field with an index but no "type" trusts the binding to tag every value the same way — require "type" once the Go binding declares it #1082, to require "type" on every field with a term output once the Go binding sends it.

Checks at the new head:

  • nextest for stack-encrypt and stack-encrypt-derive: 382 passed. Workspace: 823 passed.
  • test:doc, doc and rustdoc with no default features are clean. Workspace clippy and no-default-features clippy are clean, and so is fmt.
  • wasm:guest:test: 50 passed. wasm:auth-guest:test: 12 passed.
  • EQL test:encryption, run with --locked: passes.
  • cargo mutants --in-diff against 639858b: 137 mutants, 100 caught, 37 unviable, 0 missed.

The PR body is updated.

A match index produces a set of terms, not one: every token of the value
contributes k Bloom filter positions, and a query matches when its
positions are a subset of the stored ones. The singular name read like
EqualityTerm and OreTerm, which are each one comparand, and the plan
builder is about to name index outputs in its own API (an Index's
associated Term), where the difference shows.

MatchTerms keeps the family's prefix, so it still sorts and reads beside
EqualityTerm, OreTerm and OpeTerm, and the plural says what the value is.
The decode error variant follows (OddMatchTermsLength). The constructor
stays `matching()` and the transport bytes are unchanged; this is a name
change only.

stack-encrypt 0.2.0 on crates.io exports both old names, so the old type
name stays as a deprecated alias and the CHANGELOG records the break: the
next release is 0.3.0.

BREAKING CHANGE: sem::MatchTerm is renamed MatchTerms; MatchTerm remains
as a deprecated type alias, so code naming the type still compiles with a
warning. TermBytesError::OddMatchTermLength is renamed
OddMatchTermsLength, with no alias (an enum variant cannot have one).
The plan builder is meant to be a thin lowering onto the combinators, and
three things it needs were missing.

Indexes. Every caller (the derive, dynamic::record) spelled out
ciphertext().accepting().zip(equality()).zip(..).map(..) itself, with the
nested-pair bookkeeping. Index<S> makes one index a type, implemented only
for the plaintexts its scheme is defined over, so a match index on an
integer does not compile. Indexes<S> is one index or a tuple of two to
four, and deliberately not (), so an indexed field with no index is a
compile error rather than a quiet ciphertext-only field. indexed(idx) does
the composition once and returns Encrypted<Terms>: the ciphertext and the
terms as a tuple, read by destructuring. spec() lowers an index to data
(IndexSpec) for the FFI and saved plans, with a match index's options;
Indexes::select picks one index for the query side. The bytes are the hand
composition's and the derive's, which the tests pin.

passthrough(). A plan can carry a field unsealed. The engine had no
operation that just returns the source; this is it, moving the value in
owned mode and cloning it in borrowed mode, ignoring the context. Its
rustdoc says it is not authenticated and what to use instead.

run_decryption(). KeysetCipher::run already executes an Encryption held in
a variable; this is the Decryption counterpart, on the keyset (which
refuses a foreign leaf) and on the client.
A typed host knows what 34 is; JavaScript, PHP and Ruby do not, and index
semantics depend on it: ORE on the integer 34 and the float 34.0 differ, a
JavaScript number is a float, and match is defined over text alone. So a
plan field may now say what its values are, as `"type": "<name>"`.

The vocabulary is vitaminc's frozen leaf-tag table, one FieldType per
scalar tag, plus the two composite kinds the transport frames (array,
object); not new names. The engine uses it three times:

- building a plan refuses an unknown type name, and an index the type is
  not defined for (match on an integer, equality on a float, any index on
  a composite);
- encrypt refuses a value whose tag is not the declared type, before any
  key is minted, rather than trusting the binding's tagging, and decrypt
  refuses a field that opens to another type, so a typeless host can rely
  on the declaration for what it gets back;
- FieldType::read reads a query value as the field's type, converting a
  number only when the conversion is exact (a JavaScript 34.0 is the u64
  34 for a uint64 field).

The key is optional. A field without it is dispatched on each value's own
tag, as every field was before, so the plans existing bindings send (the
Go guest's) stay valid with no change on their side. No error variant is
added: the failures are Plan, Source and Record, which bindings already
classify as caller input.

TermKind also lowers to the typed side's IndexSpec, with the default match
options a dynamic term derives under.
The engine had two enums for the same thing: dynamic::TermKind, which the
record plan, term derivation and Go guest spoke, and IndexSpec, which the
typed Index set lowers to, joined by a From conversion. Indexes are types in
Rust and one data form at the FFI boundary, so TermKind goes and IndexSpec
takes its place everywhere: Output::Term, Error::Term, Scalar::of, term,
FieldType::admits (kind only; options do not change what an index applies
to) and the guest's term ABI.

Folding them gives match options a wire form instead of making them
impossible to send. An index in a plan's outputs list is still its key
string, and a bare "match" still means the default options, so every plan
the Go binding writes today parses to the same thing. A match index under
other options is a one-entry object, {"match": {tokenizer, downcase, k, m}},
each option optional and checked against the scheme's bounds. The term
derivation now honours those options. A field still names each output key
once, so two match indexes under different options are refused rather than
written under one key twice.

TermKind was public in stack-encrypt 0.2.0 on crates.io, so this is a
break: the CHANGELOG's Unreleased section records it, with the additions
this pull request makes to the target layer and the plan grammar.

BREAKING CHANGE: dynamic::TermKind is removed; target::IndexSpec takes its
place. Output::Term holds an IndexSpec and Output is no longer Copy;
Error::Term's kind is an IndexSpec; Scalar::of and dynamic::term take
&IndexSpec where they took a TermKind by value. A plan's wire form is
unchanged.
…eKind

The previous commit added dynamic::FieldType: an enum of the eleven value
kinds a plan field can declare, with its names, tags, parser and the test
that it covers vitaminc's tag table. vitaminc owns that vocabulary (the
FfiValue model and the frozen tag table are its), and vitaminc 0.5.1 now
publishes it as ValueKind, with name, FromStr, tags, holds and
FfiValue::kind. A second enum in stack-encrypt was a copy that could
drift from the one bindings decode against.

So FieldType is gone and ValueKind is re-exported from
stack_encrypt::dynamic. FieldPlan::with_type takes a ValueKind and
field_type returns one; the plan parser reads "type" with ValueKind's
FromStr, mapping its error to Error::Plan; encrypt and decrypt check a
value with ValueKind::holds. The two rules vitaminc deliberately leaves
to the declaring layer stay here, as free functions beside term and
context: admits(kind, &IndexSpec), which indexes a kind is defined for,
and read(kind, value), which reads a query value as a kind. The module
is renamed field_type -> kind, and its docs now say the vocabulary is
vitaminc's.

The tests that checked the names, the tag table and holds are vitaminc's
job now (it has its own, including that every kinded leaf seals under a
tag its kind names) and are dropped; the admits and read tests are
ported.

Every vitaminc crate in the root workspace and in the Go stackencrypt
guest (a detached workspace) moves to 0.5.1, so the family resolves to
one version: a split would duplicate the aead crate and fail the
FfiValue: Decrypt bound. The workspaces that build a root crate by path
inherit that requirement, so their lockfiles move too, vitaminc entries
only: packages/eql (eql-bindings' stack-encrypt feature, which
test:encryption checks with --locked), the three fuzz crates and the
stackauth guest. Their manifests are unchanged.

BREAKING CHANGE: dynamic::FieldType (added earlier in this change set,
never released) is replaced by vitaminc_aead_value::ValueKind,
re-exported as dynamic::ValueKind. FieldType::admits and FieldType::read
become the free functions dynamic::admits(kind, &index) and
dynamic::read(kind, value). FieldPlan::with_type takes, and
FieldPlan::field_type returns, a ValueKind. stack-encrypt now requires
vitaminc 0.5.1.
…s not call

Three places where the docs promised more than the code does.

An indexed plan field without "type" is still dispatched on each value's
own tag, so for that field the engine trusts the binding's tagging: a 34
sent once as a Float64 and once as an Int64 under one "ore" field stores
two different terms. That keeps the Go binding's current plans valid, and
it is meant to end once that binding fills "type" from its struct types.
The plan() rustdoc, FieldPlan and the dynamic module docs now say so, and
point at #1082, which tracks making "type" required on every field with a
term output.

dynamic::read is the whole query-side type story, and nothing in the
engine calls it: term() takes a Scalar and an IndexSpec and never sees
the field's declared kind. Its rustdoc now says that plainly, that a
binding deriving a query term for a typed field must call it first, and
that wiring it into query(v).using(&plan) is tracked in #1057.

Indexes::select said generic code over X: Indexes<S> can call
indexes.select::<Ore, _>(), which does not compile without a Select
bound. The sentence is replaced by a doctest that shows the bound, and
the method's position parameter is renamed At -> P, so the rendered
signature no longer reads as if it named the At<N> struct.
…ed-field gaps

dynamic::read treated a NaN differently in its two float directions:
read(Float64, Float32(NaN)) returned Float64(NaN), while
read(Float32, Float64(NaN)) refused it, and the doc said "a NaN is never
exact". Converting a NaN is never exact, since it equals nothing, so both
directions now refuse it. A NaN already of the field's own float kind is
still returned as it is, like any value of the kind.

Tests for paths review found unpinned:

- decrypt with an index-only field before a typed field. Opened values
  are paired with the fields that have a ciphertext, not with every plan
  field; pairing with every field makes this test fail (checked), where
  every existing test passed.
- a typed "object" or "array" field round-trips, empty or not, and opens
  as its declared kind.
- check_record accepts a record sealed as another type (it cannot open
  the leaf to see the tag), and decrypt refuses it.
- a field spec with "type" before "outputs" parses and is checked the
  same way.
- read at the first float past each integer range (2^63, 2^64, 2^32) is
  refused and the last value inside converts exactly, so a saturating
  `f as u64` cast would fail.
- read refuses a NaN in both float directions.

The query-term test's variable is renamed from probe to query, the
glossary's word.
@coderdan
coderdan force-pushed the feat/stack-encrypt-engine-plan-pieces branch from 7fe56ec to 27fe042 Compare October 5, 2026 16:35
@coderdan
coderdan merged commit 327b4db into main Oct 5, 2026
48 checks passed
@coderdan
coderdan deleted the feat/stack-encrypt-engine-plan-pieces branch October 5, 2026 17:22
auxesis pushed a commit that referenced this pull request Oct 6, 2026
Decrypt through a plan is open, because StackCipher::decrypt already
exists with another signature. A plan has two starts and takes its
context from exactly one of three sources. The typed verb encrypt_into
and the picker are Rust-only. The derive emits the plan only after its
grammar is narrowed and the plan's widened, so the two stay one grammar.
EQL types are assembled per language from standard outputs, with no
registry and no target name in the data grammar.

The lowering table now names what #1068, #1069 and #1071 shipped.
coderdan added a commit that referenced this pull request Oct 6, 2026
…stack-encrypt has

The crates.io gate fired: `cargo publish --dry-run --all-features` builds the
packaged eql-bindings against the registry stack-encrypt its manifest names,
0.2.0, and `open_target` called `run_decryption`, which landed after that
release (#1069). The rule in Cargo.toml stands: this crate uses only the API
the published stack-encrypt carries, until its next release. `decrypt_as` is
in 0.2.0 on both ciphers and runs the same DecryptInto plan, so opening a
stored EQL value goes through it and maps the plaintext with Pending::map;
nothing observable changes, and the encryption test crate's round trips,
both openers and the refusals pass as before. The dry run passes again and
joins the verification list.

Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stack-encrypt: add the engine pieces the plan builder needs (passthrough, run-by-value, Index trait, per-field types) — 0.3.0

3 participants