feat(golang)!: the generated API replaces the value and record calls

The Go SDK is now what the plan builder design describes: a struct's stash
tags are the declaration, stashgen writes the encrypted type and its
functions, and a program calls users.Encrypt, users.Decrypt and
users.Fields. Everything the old package offered beside that is removed,
not deprecated — it was never released: Cipher.Encrypt/Decrypt and the
Element forms, Client.Decrypt, EncryptRecord(s)/DecryptRecord(s),
EncryptedRecord, EncryptedField, Cipher.Term, RecordOption, WithPlan,
ExtendContext, WithGuest, TermKind, Plan, FieldPlan, NewPlan, PlanFromTags
and every run-time tag reader, the plan package and plantest, factstest,
Context/Label and their constructors, and the four Sealed* storage types.

What replaces them, and why it has the shape it has:

- encrypt.Cipher.Extend appends the caller's parts to every field's
  context in every call through the cipher, so the write, the query and
  the read cannot use different ones. encrypt.Ciphertext is the frozen
  stack-encrypt leaf, the one storage type: every sealed field is one
  leaf. Index is Equality, Ore, Ope, Match() and JSON(); the term types
  stay. Decrypter is what a generated Decrypt takes: *Cipher refuses a
  foreign keyset's row before any key is retrieved, *Client opens each
  row under the keyset that sealed it.
- The record path (Cipher.Seal, Cipher.Open, Client.Open, Cipher.Derive)
  takes the internal record types, so only generated code reaches it.
  internal/record is the data form of a declaration as dynamic::record
  reads it: a field's context is its label (context segments + identity)
  nested under each extension part, "type" is the wire kind, outputs are
  c/eq/match/ore/ope. Both gensupport (running) and stashgen (checking)
  lower to it, so the generator checks the plan the program will run.
- encrypt/gensupport is the real library behind generated code: Declare
  and the verb methods carry the field's wire kind, which stashgen picks
  from the Go type (int8..int32 travel as int32, int/int64 as int64, the
  unsigned likewise, []byte as bytes); Codec.Encrypt/Decrypt send a slice
  as one guest call and one ZeroKMS request; Field[T] seals one value and
  derives the field's terms; Get converts within a kind's family and
  refuses the rest, so a value that opens to another type is an error.
- Passthrough fields stay on the host. The FFI codec cannot carry every
  Go type a program stores beside a ciphertext (time.Time, gorm.DeletedAt,
  any driver.Valuer), and nothing the engine does to a passthrough value
  is observable, so the plan the engine sees has the sealed fields only
  and the generated Seal reads passthrough values back from the Record.
  The plan says passthrough crosses; this is the deviation, recorded here
  and in the record package's doc.
- An opaque struct crosses as one JSON document, declared bytes, because
  the engine seals a composite FfiValue as a tree of leaves and an opaque
  struct is one column. Its fields are what JSON carries; the generator
  refuses a nested struct inside one for now.
- stashgen.GuestEngine is the embedded guest: encrypt.NewChecker
  instantiates it with no credentials and asks se_plan_check one field at
  a time, so a refusal names the field, then for the whole plan. The
  engine produces no EQL type in this build (se_targets is empty), so
  encrypt_into is refused with "EQL types are not available yet".

The guest loses se_encrypt, se_decrypt and the element exports (ADR-0007
as amended: every value crosses under a declaration) and gains
se_plan_check and se_targets. A `deterministic-kms` feature builds a TEST
guest whose keys derive from a seed — the DeterministicSource of
stack-encrypt's tests/common, copied — so the Go tests open the records
Rust sealed in tests/fixtures/record_lowering.json through the generated
testusers package and derive the same term bytes, and run round trips,
foreign-keyset refusal, tampering and the ORE/OPE ordering properties
with no ZeroKMS. The hermetic tests replace the old live ordering tests;
the live suite keeps the real round trips through generated code. The
test build's import gate requires no transport import: with no ZeroKMS
client in it, the linker drops the host module.

Generated *_stash.go files are committed (internal/testusers, example),
and tests-golang.yml runs `go generate ./...` and fails on a diff. The
stashgen goldens and the stub SDK follow the new signatures; the stub
agreement test now covers encrypt too and compares signatures without
parameter names. The example is rewritten to the eight steps; the
explicit-credentials example, which only showed the removed API, is
gone with its task.

Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc

Author: coderdan

.github/workflows/tests-golang.yml

@@ -8,8 +8,10 @@ name: Tests (Go)
 #   wasi-check  the stack crates build for wasm32-wasip1 with no JS-host or
 #               native-HTTP dependencies, the no-http shape passes its tests
 #               and docs, both guests pass lint and tests and are built with
-#               their import surfaces checked, their sha256 is recorded, and
-#               `go:test` runs against them.
+#               their import surfaces checked (the stack-encrypt guest twice:
+#               the real build and the deterministic-kms test build), their
+#               sha256 is recorded, `go:test` runs against them, and
+#               `go generate` leaves the tree unchanged.
 #   go-lint     golangci-lint, Linux only.
 #   go-binding-cross
 #               the same Go tests on macOS and Windows, against the guests
@@ -144,6 +146,14 @@ jobs:
       - name: stack-encrypt guest release build and import-surface gate
         run: mise run wasm:guest:build
 
+      # The deterministic-kms TEST build beside it: the Go tests open the
+      # record fixture Rust sealed through it, and run round trips with no
+      # ZeroKMS. The Go package never embeds it; its tests skip when it is
+      # absent, so a build that forgets this step would pass with less
+      # coverage, which is why the job builds it unconditionally.
+      - name: stack-encrypt guest deterministic test build
+        run: mise run wasm:guest:build:deterministic
+
       - name: Credential guest lint and tests
         run: mise run wasm:auth-guest:test
 
@@ -155,7 +165,7 @@ jobs:
       # same bytes rather than a stale or rebuilt guest.
       - name: Record the guests' checksums
         run: |
-          for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do
+          for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do
             (cd languages/golang && openssl dgst -sha256 "$guest" | awk '{print $NF}' > "$guest.sha256" && echo "$guest sha256 $(cat "$guest.sha256")")
           done
 
@@ -166,6 +176,8 @@ jobs:
           path: |
             languages/golang/encrypt/wasm/stack_encrypt_guest.wasm
             languages/golang/encrypt/wasm/stack_encrypt_guest.wasm.sha256
+            languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm
+            languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm.sha256
             languages/golang/auth/wasm/stack_auth_guest.wasm
             languages/golang/auth/wasm/stack_auth_guest.wasm.sha256
           if-no-files-found: error
@@ -183,6 +195,17 @@ jobs:
           echo "RLIMIT_MEMLOCK: $(ulimit -l) KiB"
           mise run go:test
 
+      # What is encrypted is fixed before the program ships: every generated
+      # file is committed, and this fails when `go generate` would change
+      # one. It runs the real stashgen against the guest just built, so it
+      # also proves the generator and the engine agree on the module's own
+      # examples.
+      - name: Generated code is committed
+        working-directory: languages/golang
+        run: |
+          CGO_ENABLED=0 go generate ./...
+          git diff --exit-code -- .
+
   # Linux only: macOS and Windows would report the same findings. Needs no
   # guests: the packages embed a directory and compile without them.
   go-lint:
@@ -256,7 +279,7 @@ jobs:
       - name: The guests are the ones Linux built and checked
         run: |
           cd languages/golang
-          for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do
+          for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do
             want=$(cat "$guest.sha256")
             got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}')
             if [ "$want" != "$got" ]; then
@@ -320,7 +343,7 @@ jobs:
       - name: The guests are the ones the wasi-check job built and checked
         run: |
           cd languages/golang
-          for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do
+          for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do
             want=$(cat "$guest.sha256")
             got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}')
             if [ "$want" != "$got" ]; then

AGENTS.md

@@ -92,7 +92,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l
   links are provenance only.
 - `packages/stack-auth`, `packages/stack-profile`, `packages/stack-kms`, `packages/stack-encrypt`, `packages/stack-encrypt-derive`, `packages/stack-guest-abi`: The Rust crates imported from `cipherstash/cipherstash-suite` with their history — `stack-auth` and `stack-profile` (published to crates.io, one version group), `stack-kms` (published to crates.io from 0.1.0, its own version group, re-exported by `stack-encrypt` as `stack_encrypt::kms`), `stack-encrypt` and `stack-encrypt-derive` (published to crates.io from 0.1.0, one version group; `eql-bindings`' `stack-encrypt` feature depends on them from the registry), and `stack-guest-abi` (`publish = false`). They are the members of the **root Cargo workspace**, with the three node binding crates below. See "Working on the Rust crates".
 - `languages/typescript/packages/auth`, `languages/typescript/packages/profile`, `languages/typescript/packages/stack-auth-wasm`: The node bindings of those crates. `@cipherstash/auth` (napi-rs v2) and its six `platforms/*` packages are published to npm from this repository by `release.yml` (`auth-artifacts`, `publish-auth`); a change to what it ships, the `stack-auth` crate included, needs an `@cipherstash/auth` changeset (`require-auth-npm-changeset.yml`). `@cipherstash/profile` and its platforms are private and never published; `@cipherstash/stack-auth-wasm` is private and builds the wasm that `@cipherstash/auth` ships. Their `build` and `test` scripts never invoke cargo; `build:native`, `build:debug` and `test:cargo` do.
-- `languages/golang`: The Go module (`encrypt`, `auth`, `internal`), a wazero host with no cgo. Its two WASI guests (`*/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; the `.wasm` files they embed are gitignored. There is no Go release process yet.
+- `languages/golang`: The Go module — the SDK `encrypt` with its generated-code support `encrypt/gensupport` and the policy packages `encrypt/policy` and `encrypt/policy/protosource`; the credential package `auth`; the generator `stashgen` and its command `cmd/stashgen`; and `internal` (the shared guest plumbing and the `record` wire model). A wazero host with no cgo. Its two WASI guests (`encrypt/guest`, `auth/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; `mise run wasm:guest:build:deterministic` builds the seeded TEST build the hermetic Go tests use; the `.wasm` files they embed are gitignored. Generated `*_stash.go` files are committed and CI fails when `go generate ./...` changes one. There is no Go release process yet.
 - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README)
 - `languages/typescript/examples/*`: Working apps (basic, prisma, supabase-worker)
 - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo).

languages/golang/auth/README.md

@@ -3,7 +3,7 @@
 The Go binding of the developer profile — the directory `stash auth login`
 writes — read through the `stack-profile` Rust crate running inside a WASI
 guest under [wazero], with `CGO_ENABLED=0`. It is the credential half of
-the Go SDK: it hands a [`encrypt`](../encrypt) client its client
+the Go SDK: it hands an [`encrypt`](../encrypt) client its client
 key and its bearer token without either package re-deriving the profile's
 layout, and without either importing the other.
 
@@ -16,7 +16,7 @@ cross-process refresh lock for device sessions.
 
 ## Use
 
-Most applications never call this package directly: a `encrypt`
+Most applications never call this package directly: an `encrypt`
 client built with `NewClient(ctx)` and no options resolves its credentials with
 `encrypt.AutoCredentials`, which reads the environment first and then
 the profile, through this package. Use it directly to take the profile
@@ -69,7 +69,7 @@ the strategy are the caller's: the client asks the strategy for a token on
 every request but never closes it, so both stay open until the client is
 closed (the deferred calls above run in that order).
 
-A encrypt client takes its token only from a strategy, never a raw
+An encrypt client takes its token only from a strategy, never a raw
 string: a raw token cannot be refreshed when it expires, and would bypass
 the cross-process lock a device-session refresh holds with the `stash` CLI
 (the IdP revokes a whole refresh-token chain when one is used twice).

languages/golang/cmd/stashgen/README.md

@@ -116,9 +116,9 @@ It runs none of the package's code.
 It ignores its own output file when it loads the package, so a stale file does not stop it.
 The same input always gives the same file: fields keep their declared order, and the file carries no version and no time.
 
-`stashgen` checks each declaration with the engine, and holds no copy of the engine's rules.
+`stashgen` checks each declaration with the engine, and holds no copy of the engine's rules: it runs the WASI guest the SDK embeds and asks it, one field at a time, so the error names the field.
 It asks the engine for the EQL types it holds: each name, its plaintext type, its indexes and its query form.
-The engine produces one EQL type today, `TextEq`.
+This build of the engine produces no EQL type, so `encrypt_into` is refused with "EQL types are not available yet"; the next release adds `TextEq` and `encrypt/eql`.
 Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`.
 
 ## When stashgen stops
@@ -176,8 +176,13 @@ The struct you wrote is not protected: `stashgen` warns when it has sealed field
 `-redact` makes `stashgen` write those two methods on the struct.
 No warning, error or log line holds a plaintext value.
 
+## What crosses the binding
+
+Generated code sends the engine every sealed field with its value, under the declaration lowered to data: each field's label (`<context>/<name>`), its outputs and its wire type (`int64`, `string`, `bytes`, ...), which `stashgen` chose from the field's Go type.
+A passthrough field stays on the host: the engine does nothing to it a program could observe, and the FFI codec cannot carry every Go type a program stores beside a ciphertext.
+An `opaque` struct crosses as one JSON document and is one column; its fields are what JSON carries: scalars, `[]byte`, slices and maps of them.
+
 ## Status
 
-The generator is built and tested against a static stand-in for the engine.
-`stashgen.GuestEngine`, which runs the WASI guest the SDK embeds, is not wired yet, so the command stops with `ErrEngineUnavailable` until it is.
-The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`; `stashgen.Generate` takes one with `WithEngine`.
+The command runs the WASI guest the SDK embeds, so it needs the guest built: `mise run wasm:guest:build`.
+The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`; `stashgen.Generate` takes one with `WithEngine`, and `stashgen/enginetest` has a static one for tests.

languages/golang/cmd/stashgen/main_test.go

@@ -8,6 +8,7 @@ import (
 	"strings"
 	"testing"
 
+	"github.com/cipherstash/stack/languages/golang/encrypt"
 	"github.com/cipherstash/stack/languages/golang/stashgen"
 	"github.com/cipherstash/stack/languages/golang/stashgen/enginetest"
 )
@@ -117,17 +118,32 @@ func TestRunFlags(t *testing.T) {
 	}
 }
 
-func TestRunStopsWithoutAnEngine(t *testing.T) {
+// The real engine: the embedded guest, which produces no EQL type in this
+// build, so a struct with encrypt_into is refused and nothing is written.
+// Skips when the guest is not built.
+func TestRunAsksTheEmbeddedEngine(t *testing.T) {
+	checker, err := encrypt.NewChecker(context.Background())
+	if err != nil {
+		t.Skip(err)
+	}
+	_ = checker.Close()
 	dir := writeModule(t, map[string]string{"model.go": userSource})
 	var stdout, stderr bytes.Buffer
 	if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 1 {
-		t.Fatalf("exit %d, want 1", code)
+		t.Fatalf("exit %d, want 1\n%s", code, stderr.String())
 	}
-	if !strings.Contains(stderr.String(), stashgen.ErrEngineUnavailable.Error()) {
+	if !strings.Contains(stderr.String(), "EQL types are not available yet") {
 		t.Fatalf("stderr = %q", stderr.String())
 	}
 	if _, err := os.Stat(filepath.Join(dir, "user_stash.go")); !os.IsNotExist(err) {
-		t.Fatal("a file was written with no engine")
+		t.Fatal("a file was written after the engine refused")
+	}
+	// Separate columns are what the engine runs today.
+	columns := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt,index=equality;match", 1)
+	dir = writeModule(t, map[string]string{"model.go": columns})
+	stderr.Reset()
+	if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 0 {
+		t.Fatalf("exit %d\n%s", code, stderr.String())
 	}
 }