From 481890a44ed192702942f9a8662675c8bd89d213 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 00:23:21 -0700 Subject: [PATCH 01/30] feat(golang): stashgen generator library and encrypt/gensupport Add the Go half of the Stack Encrypt code generator: the stashgen library reads a struct's stash tags through go/packages and go/types, checks the declaration with an Engine, and writes the encrypted type, Encrypt, Decrypt, Fields and the print methods. encrypt/gensupport holds what only generated code calls and nothing else: the GeneratedVersion1 constant, Redacted and RedactedLog, and the once-per-type stderr notices. The generator reads types, not text, so it sees a type from another package the way the compiler does, and it runs none of the package's code. The output file is parsed as its package clause only, so a stale generated file does not stop regeneration. Output is deterministic: declared field order, no time, no version string beyond the constant. Every refusal about an index, an EQL type or a field type comes from the Engine, not from the generator; the generator holds no copy of the engine's rules (SDK principle 1). GuestEngine is the hole the integration step fills with the embedded WASI guest; until then it returns ErrEngineUnavailable and the static fake in internal/fakeengine stands in for tests, knowing TextEq and the four indexes with their Go-kind rules as stack-encrypt's target/index.rs implements them. The golden tests seed from the hand-written examples in docs/plans/2026-10-04-plan-builder and deviate only where a generator needs one rule where the hand-written files used several: a composite literal with more than one entry is written one entry per line; Seal assigns into `var e` with the passthrough fields first and the sealed fields in declared order, with no local names that could collide with a field; Encrypt and Decrypt carry doc comments on every type, naming the type rather than guessing a singular; and the compile check happens against a signatures-only stub of encrypt, encrypt/eql and encrypt/gensupport under testdata, because those packages do not exist yet. The stub is the contract the integration step implements. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- .../golang/encrypt/gensupport/export_test.go | 20 + .../golang/encrypt/gensupport/gensupport.go | 136 ++++ .../encrypt/gensupport/gensupport_test.go | 74 ++ languages/golang/go.mod | 10 +- languages/golang/go.sum | 8 + languages/golang/stashgen/declaration.go | 226 ++++++ languages/golang/stashgen/emit.go | 510 ++++++++++++ languages/golang/stashgen/engine.go | 51 ++ languages/golang/stashgen/errors.go | 25 + languages/golang/stashgen/generate.go | 114 +++ languages/golang/stashgen/golden_test.go | 140 ++++ languages/golang/stashgen/imports.go | 79 ++ .../internal/fakeengine/fakeengine.go | 111 +++ languages/golang/stashgen/load.go | 150 ++++ languages/golang/stashgen/model.go | 253 ++++++ languages/golang/stashgen/names.go | 67 ++ languages/golang/stashgen/read.go | 734 ++++++++++++++++++ languages/golang/stashgen/tag.go | 255 ++++++ languages/golang/stashgen/tag_test.go | 147 ++++ .../testdata/cases/accounts/account.go | 39 + .../cases/accounts/account_stash.go.golden | 154 ++++ .../stashgen/testdata/cases/accounts/go.mod | 12 + .../testdata/cases/contacts/contacts.go | 76 ++ .../contacts/contactstash_stash.go.golden | 205 +++++ .../testdata/cases/contacts/crm/contact.go | 9 + .../stashgen/testdata/cases/contacts/go.mod | 12 + .../cases/documents/document_stash.go.golden | 88 +++ .../testdata/cases/documents/documents.go | 43 + .../stashgen/testdata/cases/documents/go.mod | 12 + .../stashgen/testdata/cases/users/go.mod | 12 + .../stashgen/testdata/cases/users/model.go | 13 + .../testdata/cases/users/user_stash.go.golden | 140 ++++ .../golang/stashgen/testdata/stubgorm/go.mod | 3 + .../golang/stashgen/testdata/stubgorm/gorm.go | 23 + .../testdata/stubsdk/encrypt/encrypt.go | 77 ++ .../testdata/stubsdk/encrypt/eql/eql.go | 9 + .../stubsdk/encrypt/gensupport/gensupport.go | 121 +++ .../golang/stashgen/testdata/stubsdk/go.mod | 3 + 38 files changed, 4160 insertions(+), 1 deletion(-) create mode 100644 languages/golang/encrypt/gensupport/export_test.go create mode 100644 languages/golang/encrypt/gensupport/gensupport.go create mode 100644 languages/golang/encrypt/gensupport/gensupport_test.go create mode 100644 languages/golang/stashgen/declaration.go create mode 100644 languages/golang/stashgen/emit.go create mode 100644 languages/golang/stashgen/engine.go create mode 100644 languages/golang/stashgen/errors.go create mode 100644 languages/golang/stashgen/generate.go create mode 100644 languages/golang/stashgen/golden_test.go create mode 100644 languages/golang/stashgen/imports.go create mode 100644 languages/golang/stashgen/internal/fakeengine/fakeengine.go create mode 100644 languages/golang/stashgen/load.go create mode 100644 languages/golang/stashgen/model.go create mode 100644 languages/golang/stashgen/names.go create mode 100644 languages/golang/stashgen/read.go create mode 100644 languages/golang/stashgen/tag.go create mode 100644 languages/golang/stashgen/tag_test.go create mode 100644 languages/golang/stashgen/testdata/cases/accounts/account.go create mode 100644 languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden create mode 100644 languages/golang/stashgen/testdata/cases/accounts/go.mod create mode 100644 languages/golang/stashgen/testdata/cases/contacts/contacts.go create mode 100644 languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden create mode 100644 languages/golang/stashgen/testdata/cases/contacts/crm/contact.go create mode 100644 languages/golang/stashgen/testdata/cases/contacts/go.mod create mode 100644 languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden create mode 100644 languages/golang/stashgen/testdata/cases/documents/documents.go create mode 100644 languages/golang/stashgen/testdata/cases/documents/go.mod create mode 100644 languages/golang/stashgen/testdata/cases/users/go.mod create mode 100644 languages/golang/stashgen/testdata/cases/users/model.go create mode 100644 languages/golang/stashgen/testdata/cases/users/user_stash.go.golden create mode 100644 languages/golang/stashgen/testdata/stubgorm/go.mod create mode 100644 languages/golang/stashgen/testdata/stubgorm/gorm.go create mode 100644 languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go create mode 100644 languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go create mode 100644 languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go create mode 100644 languages/golang/stashgen/testdata/stubsdk/go.mod diff --git a/languages/golang/encrypt/gensupport/export_test.go b/languages/golang/encrypt/gensupport/export_test.go new file mode 100644 index 000000000..362e94262 --- /dev/null +++ b/languages/golang/encrypt/gensupport/export_test.go @@ -0,0 +1,20 @@ +package gensupport + +import "io" + +// SetNotices redirects the notices for a test and returns the previous writer. +func SetNotices(w io.Writer) io.Writer { + noticesMu.Lock() + defer noticesMu.Unlock() + prev := notices + notices = w + return prev +} + +// ResetNoticed forgets which notices were printed. +func ResetNoticed() { + noticed.Range(func(k, _ any) bool { + noticed.Delete(k) + return true + }) +} diff --git a/languages/golang/encrypt/gensupport/gensupport.go b/languages/golang/encrypt/gensupport/gensupport.go new file mode 100644 index 000000000..e50fef63d --- /dev/null +++ b/languages/golang/encrypt/gensupport/gensupport.go @@ -0,0 +1,136 @@ +// Package gensupport holds what only code written by stashgen calls. +// +// A program never imports this package. The generated file names +// [GeneratedVersion1], prints through [Redacted] and [RedactedLog], and +// reports the two notices that the generator also prints. No function in this +// package panics. +package gensupport + +import ( + "fmt" + "io" + "log/slog" + "os" + "sort" + "strings" + "sync" +) + +// generatedVersion is the type of the version constants. Only a library +// version that accepts a generated file's layout declares the constant that +// file names, so a file from another version does not compile. +type generatedVersion uint8 + +// GeneratedVersion1 is the layout of files that stashgen writes today. Every +// generated file holds `const _ = gensupport.GeneratedVersion1`. +const GeneratedVersion1 generatedVersion = 1 + +// sealed is what a hidden field prints as. +const sealed = "[sealed]" + +// Redacted formats a value for String. The shown fields print with their +// values, in name order; the hidden fields print as [sealed]. Generated code +// passes the passthrough fields as shown and the sealed fields as hidden, so +// no plaintext reaches the output. +func Redacted(typeName string, shown map[string]any, hidden ...string) string { + var b strings.Builder + b.WriteString(typeName) + b.WriteByte('{') + first := true + for _, name := range sortedKeys(shown) { + if !first { + b.WriteString(", ") + } + first = false + fmt.Fprintf(&b, "%s: %v", name, shown[name]) + } + for _, name := range hidden { + if !first { + b.WriteString(", ") + } + first = false + b.WriteString(name) + b.WriteString(": ") + b.WriteString(sealed) + } + b.WriteByte('}') + return b.String() +} + +// RedactedLog is [Redacted] for slog: a group with one attribute for each +// shown field, in name order, and the string [sealed] for each hidden field. +func RedactedLog(shown map[string]any, hidden ...string) slog.Value { + attrs := make([]slog.Attr, 0, len(shown)+len(hidden)) + for _, name := range sortedKeys(shown) { + attrs = append(attrs, slog.Any(name, shown[name])) + } + for _, name := range hidden { + attrs = append(attrs, slog.String(name, sealed)) + } + return slog.GroupValue(attrs...) +} + +func sortedKeys(m map[string]any) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} + +// notices is where the running program's notices go. Tests replace it. +var ( + noticesMu sync.Mutex + notices io.Writer = os.Stderr + noticed sync.Map // notice key -> struct{} +) + +// NoticeUntagged prints, once for each type, that the unexported fields are +// neither encrypted nor stored. stashgen printed the same notice when it +// wrote the file, and the file names the fields in a comment. The tag +// `stash:"-"` on each field stops all three. +func NoticeUntagged(typeName string, fields []string) { + if len(fields) == 0 { + return + } + notice("untagged:"+typeName, fmt.Sprintf( + "stashgen: %s: not encrypted and not stored: the unexported %s. Tag %s `stash:\"-\"` to confirm that.", + typeName, fieldList(fields), itOrEach(fields))) +} + +// NoticePrintsPlaintext prints, once for each type, that the type prints its +// sealed fields in the clear because it has no String and LogValue methods. +// stashgen printed the same notice when it wrote the file. +func NoticePrintsPlaintext(typeName string) { + notice("prints:"+typeName, fmt.Sprintf( + "stashgen: %s prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", + typeName)) +} + +func notice(key, text string) { + if _, seen := noticed.LoadOrStore(key, struct{}{}); seen { + return + } + noticesMu.Lock() + defer noticesMu.Unlock() + fmt.Fprintln(notices, text) +} + +func fieldList(fields []string) string { + quoted := make([]string, len(fields)) + for i, f := range fields { + quoted[i] = fmt.Sprintf("%q", f) + } + if len(quoted) == 1 { + return "field " + quoted[0] + } + return "fields " + strings.Join(quoted[:len(quoted)-1], ", ") + " and " + quoted[len(quoted)-1] +} + +func itOrEach(fields []string) string { + if len(fields) == 1 { + return "it" + } + return "each" +} diff --git a/languages/golang/encrypt/gensupport/gensupport_test.go b/languages/golang/encrypt/gensupport/gensupport_test.go new file mode 100644 index 000000000..10d706efd --- /dev/null +++ b/languages/golang/encrypt/gensupport/gensupport_test.go @@ -0,0 +1,74 @@ +package gensupport + +import ( + "bytes" + "log/slog" + "strings" + "testing" +) + +func TestRedactedHidesSealedFieldsAndSortsShown(t *testing.T) { + got := Redacted("EncryptedUser", map[string]any{"Nickname": "nick", "ID": int64(7)}, "Email", "Name") + want := "EncryptedUser{ID: 7, Nickname: nick, Email: [sealed], Name: [sealed]}" + if got != want { + t.Fatalf("Redacted = %q, want %q", got, want) + } + if got := Redacted("EncryptedDocument", nil, "Sealed"); got != "EncryptedDocument{Sealed: [sealed]}" { + t.Fatalf("Redacted with no shown fields = %q", got) + } +} + +func TestRedactedLogHidesSealedFields(t *testing.T) { + var buf bytes.Buffer + logger := slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{ReplaceAttr: dropTime})) + logger.Info("saved", "user", RedactedLog(map[string]any{"ID": 7}, "Email")) + line := buf.String() + if !strings.Contains(line, "user.ID=7") || !strings.Contains(line, "user.Email=[sealed]") { + t.Fatalf("log line = %q", line) + } +} + +func dropTime(_ []string, a slog.Attr) slog.Attr { + if a.Key == slog.TimeKey { + return slog.Attr{} + } + return a +} + +func TestNoticesPrintOnceForEachType(t *testing.T) { + var buf bytes.Buffer + prev := SetNotices(&buf) + defer SetNotices(prev) + ResetNoticed() + + NoticeUntagged("Account", []string{"cache"}) + NoticeUntagged("Account", []string{"cache"}) + NoticeUntagged("Other", []string{"a", "b"}) + NoticeUntagged("Empty", nil) + NoticePrintsPlaintext("User") + NoticePrintsPlaintext("User") + + lines := strings.Split(strings.TrimSpace(buf.String()), "\n") + if len(lines) != 3 { + t.Fatalf("got %d notices, want 3:\n%s", len(lines), buf.String()) + } + if want := `stashgen: Account: not encrypted and not stored: the unexported field "cache". Tag it ` + "`stash:\"-\"`" + ` to confirm that.`; lines[0] != want { + t.Fatalf("line 0 = %q\nwant %q", lines[0], want) + } + if !strings.Contains(lines[1], `fields "a" and "b". Tag each`) { + t.Fatalf("line 1 = %q", lines[1]) + } + if !strings.Contains(lines[2], "User prints its sealed fields in the clear") { + t.Fatalf("line 2 = %q", lines[2]) + } +} + +func TestVersionConstantIsTyped(t *testing.T) { + // A generated file holds `const _ = gensupport.GeneratedVersion1`. The + // constant's type is unexported, so no other package can declare a value + // that satisfies the same reference. + var _ generatedVersion = GeneratedVersion1 + if GeneratedVersion1 != 1 { + t.Fatalf("GeneratedVersion1 = %d", GeneratedVersion1) + } +} diff --git a/languages/golang/go.mod b/languages/golang/go.mod index 047e38670..0d57540fc 100644 --- a/languages/golang/go.mod +++ b/languages/golang/go.mod @@ -9,4 +9,12 @@ require ( golang.org/x/oauth2 v0.37.0 ) -require golang.org/x/sys v0.48.0 +require ( + golang.org/x/sys v0.48.0 + golang.org/x/tools v0.51.0 +) + +require ( + golang.org/x/mod v0.41.0 // indirect + golang.org/x/sync v0.23.0 // indirect +) diff --git a/languages/golang/go.sum b/languages/golang/go.sum index 216284c5a..39429dfe2 100644 --- a/languages/golang/go.sum +++ b/languages/golang/go.sum @@ -2,9 +2,17 @@ github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d1 github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:6jpaqAo6f7rjWuxoti0Ymoi9DQZyxpnieWVudhXv38Y= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc h1:vlrjoILAURGfpBWucVZEywK22k4lfky1xGDIKV6kqNg= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:RJODA1DCSm4H+1pbmCzDdT2pAIkxhinwIu6ewjh5sPs= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU= github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0= +golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= +golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98= golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58= +golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk= +golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI= +golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA= diff --git a/languages/golang/stashgen/declaration.go b/languages/golang/stashgen/declaration.go new file mode 100644 index 000000000..05e07acd8 --- /dev/null +++ b/languages/golang/stashgen/declaration.go @@ -0,0 +1,226 @@ +package stashgen + +import ( + "fmt" + "strings" +) + +// Declaration is how each field of one struct is encrypted: the context of +// the struct, and the verb, indexes and EQL type of each field. The generator +// builds it from the stash tags, the engine checks it, and the generated file +// holds it as data that only generated code uses. +type Declaration struct { + // Type is the Go type the declaration is for, as the generator prints it + // in an error: "User" or "crm.Contact". + Type string + // Context is the context of every field, from the `context=` tag. + Context string + // Opaque seals the whole struct as one value. The fields then carry no + // tags and no indexes. + Opaque bool + // Fields are the struct's fields in declared order, omitted ones included. + Fields []Field +} + +// Field is one field of a declaration. +type Field struct { + // Name is the field's name in the declaration, which is its column name. + // An omitted field's Name is its Go name in snake case. + Name string + // GoName is the field's name in the Go struct. + GoName string + // GoType is the field's Go type, for the engine's checks. + GoType GoType + // Verb says what happens to the field. + Verb Verb + // Indexes are the indexes the tag names, for EncryptIndex and Index. + Indexes []Index + // EQLType is the EQL type an EncryptInto field seals into. + EQLType string +} + +// Verb is what happens to a field. +type Verb uint8 + +const ( + // VerbOmit leaves the field out: it does not cross the binding. + VerbOmit Verb = iota + // VerbPassthrough stores the field as it is. + VerbPassthrough + // VerbEncrypt seals the field with no index. + VerbEncrypt + // VerbEncryptIndex seals the field and derives each index beside it. + VerbEncryptIndex + // VerbIndex derives the indexes alone, with no ciphertext. + VerbIndex + // VerbEncryptInto seals the field into one EQL value. + VerbEncryptInto +) + +// String returns the tag word for the verb. +func (v Verb) String() string { + switch v { + case VerbOmit: + return "-" + case VerbPassthrough: + return "passthrough" + case VerbEncrypt: + return "encrypt" + case VerbEncryptIndex: + return "encrypt,index" + case VerbIndex: + return "index" + case VerbEncryptInto: + return "encrypt_into" + } + return fmt.Sprintf("Verb(%d)", uint8(v)) +} + +// Sealed reports whether the field has a ciphertext or a term: every verb +// but omit and passthrough. +func (f Field) Sealed() bool { + switch f.Verb { + case VerbEncrypt, VerbEncryptIndex, VerbIndex, VerbEncryptInto: + return true + } + return false +} + +// HasCiphertext reports whether the field's outputs include a ciphertext. +func (f Field) HasCiphertext() bool { + return f.Verb == VerbEncrypt || f.Verb == VerbEncryptIndex +} + +// Index is one index on a field, with its options. +type Index struct { + Name IndexName + Options []Option +} + +// String spells the index as the tag does: `match` or `match(k=3)`. +func (i Index) String() string { + if len(i.Options) == 0 { + return string(i.Name) + } + opts := make([]string, len(i.Options)) + for n, o := range i.Options { + opts[n] = o.String() + } + return string(i.Name) + "(" + strings.Join(opts, ",") + ")" +} + +// Option is one option of an index, from the parentheses after its name. +type Option struct { + Key string + Value string +} + +// String spells the option as the tag does. +func (o Option) String() string { + if o.Value == "" { + return o.Key + } + return o.Key + "=" + o.Value +} + +// IndexName is the tag word for an index. The words are the Rust API's. +type IndexName string + +// The index names. +const ( + IndexEquality IndexName = "equality" + IndexMatch IndexName = "match" + IndexOre IndexName = "ore" + IndexOpe IndexName = "ope" + IndexJSON IndexName = "json" +) + +// indexNames lists every index, in the order the generated type lists their +// outputs. +var indexNames = []IndexName{IndexEquality, IndexMatch, IndexOre, IndexOpe, IndexJSON} + +// GoName is the index's name in generated code: the field of the output +// struct that holds its term, and the query method of the field entry. +func (n IndexName) GoName() string { + switch n { + case IndexEquality: + return "Equality" + case IndexMatch: + return "Match" + case IndexOre: + return "Ore" + case IndexOpe: + return "Ope" + case IndexJSON: + return "JSON" + } + return string(n) +} + +// GoType describes a field's Go type to the engine without go/types. +type GoType struct { + // Name is the type as Go source in the field's package: "string", + // "int64", "[]string", "time.Time", "crm.Address". + Name string + // Kind is the type's underlying kind. + Kind Kind + // Elem is the element type of a slice, map or pointer, and nil otherwise. + Elem *GoType + // Fields are the fields of a struct kind, in declared order. + Fields []GoType +} + +// String returns the type as Go source. +func (t GoType) String() string { return t.Name } + +// Kind is the underlying kind of a Go type, as far as the engine needs it. +type Kind uint8 + +const ( + // KindOther is a kind the engine cannot seal: a channel, a function, an + // interface, a complex number, or a struct with an unexported field. + KindOther Kind = iota + KindString + KindBool + KindInt + KindUint + KindFloat + // KindBytes is []byte. + KindBytes + KindSlice + KindMap + KindPointer + // KindStruct is a struct whose fields are all exported. + KindStruct +) + +var kindNames = [...]string{ + KindOther: "other", + KindString: "string", + KindBool: "bool", + KindInt: "int", + KindUint: "uint", + KindFloat: "float", + KindBytes: "bytes", + KindSlice: "slice", + KindMap: "map", + KindPointer: "pointer", + KindStruct: "struct", +} + +// String returns the kind's name. +func (k Kind) String() string { + if int(k) < len(kindNames) { + return kindNames[k] + } + return fmt.Sprintf("Kind(%d)", uint8(k)) +} + +// Scalar reports whether the kind is one value an index can take. +func (k Kind) Scalar() bool { + switch k { + case KindString, KindBool, KindInt, KindUint, KindFloat, KindBytes: + return true + } + return false +} diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go new file mode 100644 index 000000000..0a632b8b5 --- /dev/null +++ b/languages/golang/stashgen/emit.go @@ -0,0 +1,510 @@ +package stashgen + +import ( + "fmt" + "go/format" + "strings" +) + +// emit writes the generated file from the model. It writes gofmt-ready Go and +// runs go/format over it, so a formatting mistake is a build error here, not +// a diff in the user's repository. +func emit(f *genFile) ([]byte, error) { + if f.imports.err != nil { + return nil, f.imports.err + } + for _, g := range f.fields { + for _, idx := range g.Indexes { + if len(idx.Options) > 0 { + return nil, fieldErr(f.typeName, g.GoName, "index %s: index options have no Go form yet", idx) + } + } + } + w := &writer{} + w.header(f) + w.encryptedType(f) + w.printMethods(f) + w.shape(f) + w.declaration(f) + w.codec(f) + w.functions(f) + w.fieldEntries(f) + for i := range f.models { + w.model(f, &f.models[i]) + } + src, err := format.Source(w.bytes()) + if err != nil { + return nil, fmt.Errorf("stashgen: the generated file does not format: %w\n%s", err, w.bytes()) + } + return src, nil +} + +type writer struct { + b strings.Builder +} + +func (w *writer) bytes() []byte { return []byte(w.b.String()) } + +func (w *writer) p(format string, args ...any) { + fmt.Fprintf(&w.b, format, args...) + w.b.WriteByte('\n') +} + +func (w *writer) nl() { w.b.WriteByte('\n') } + +func (w *writer) header(f *genFile) { + w.p("// Code generated by stashgen. DO NOT EDIT.") + w.nl() + w.p("package %s", f.pkgName) + w.nl() + w.b.WriteString(f.imports.block()) + w.nl() + w.p("// Stops compiling when the library does not accept this version of generated file.") + w.p("const _ = gensupport.GeneratedVersion1") + for _, n := range f.notices { + w.nl() + w.b.WriteString(wrapComment(n, 80)) + } +} + +func tagSuffix(tags string) string { + if tags == "" { + return "" + } + return " `" + tags + "`" +} + +func (w *writer) encryptedType(f *genFile) { + w.nl() + w.p("type %s struct {", f.encName) + for _, m := range f.members { + if m.embedded { + w.p("\t%s%s", m.typeExpr, tagSuffix(m.tags)) + continue + } + w.p("\t%s %s%s", m.name, m.typeExpr, tagSuffix(m.tags)) + } + w.p("}") + for _, g := range f.fields { + if g.Verb == VerbPassthrough || g.Verb == VerbEncryptInto { + continue + } + w.nl() + w.p("type %s struct {", g.outputType) + for _, o := range g.outputs { + w.p("\t%s %s", o.name, o.typeExpr) + } + w.p("}") + } +} + +// shownAndHidden are the arguments of Redacted: the passthrough members by +// Go name, and the sealed fields' Go names. +func shownAndHidden(f *genFile, recv string) (shown string, hidden string) { + var entries, hiddenNames []string + for _, m := range f.members { + if m.embedded { + entries = append(entries, fmt.Sprintf("%q: %s.%s", m.name, recv, m.name)) + } + } + for _, g := range f.fields { + if g.via != "" { + continue + } + if g.Verb == VerbPassthrough { + entries = append(entries, fmt.Sprintf("%q: %s.%s", g.GoName, recv, g.GoName)) + } else { + hiddenNames = append(hiddenNames, fmt.Sprintf("%q", g.GoName)) + } + } + if f.decl.Opaque { + hiddenNames = []string{`"Sealed"`} + } + shown = "nil" + if len(entries) > 0 { + shown = "map[string]any{" + strings.Join(entries, ", ") + "}" + } + return shown, ", " + strings.Join(hiddenNames, ", ") +} + +func (w *writer) printMethods(f *genFile) { + shown, hidden := shownAndHidden(f, "e") + w.nl() + w.p("func (e %s) String() string {", f.encName) + w.p("\treturn gensupport.Redacted(%q, %s%s)", f.encName, shown, hidden) + w.p("}") + w.nl() + w.p("func (e %s) LogValue() slog.Value {", f.encName) + w.p("\treturn gensupport.RedactedLog(%s%s)", shown, hidden) + w.p("}") + if !f.redact { + return + } + r := f.redactRecv + // On the plaintext struct the passthrough fields are the same, and the + // sealed fields are read from the struct itself. + shown, hidden = shownAndHidden(f, r) + w.nl() + w.p("// Written because of -redact: %s no longer prints its sealed fields.", f.typeName) + w.p("func (%s %s) String() string {", r, f.typeName) + w.p("\treturn gensupport.Redacted(%q, %s%s)", f.typeName, shown, hidden) + w.p("}") + w.nl() + w.p("func (%s %s) LogValue() slog.Value {", r, f.typeName) + w.p("\treturn gensupport.RedactedLog(%s%s)", shown, hidden) + w.p("}") +} + +func (w *writer) shape(f *genFile) { + if f.shapeName == "" { + return + } + w.nl() + w.p("// Stops compiling when %s gains, loses, reorders or retypes a field.", f.typeName) + w.p("var _ = %s(%s)", f.shapeName, f.shapeSource) + w.nl() + w.p("type %s struct {", f.shapeName) + for _, s := range f.shapeFields { + if s.embedded { + w.p("\t%s", s.typeExpr) + } else { + w.p("\t%s %s", s.name, s.typeExpr) + } + } + w.p("}") +} + +func indexExpr(idx Index) string { + switch idx.Name { + case IndexMatch, IndexJSON: + return "encrypt." + idx.Name.GoName() + "()" + } + return "encrypt." + idx.Name.GoName() +} + +func (w *writer) declaration(f *genFile) { + w.nl() + if f.decl.Opaque { + w.p("var %s = gensupport.DeclareOpaque(%q)", f.declVar, f.decl.Context) + return + } + w.p("var %s = gensupport.Declare(%q).", f.declVar, f.decl.Context) + for i, fld := range f.decl.Fields { + end := "." + if i == len(f.decl.Fields)-1 { + end = "" + } + switch fld.Verb { + case VerbOmit: + w.p("\tOmit(%q)%s", fld.Name, end) + case VerbPassthrough: + w.p("\tPassthrough(%q)%s", fld.Name, end) + case VerbEncrypt: + w.p("\tEncrypt(%q)%s", fld.Name, end) + case VerbEncryptInto: + w.p("\tEncryptInto(%q, %q)%s", fld.Name, fld.EQLType, end) + case VerbEncryptIndex, VerbIndex: + call := "EncryptIndex" + if fld.Verb == VerbIndex { + call = "Index" + } + args := make([]string, len(fld.Indexes)) + for n, idx := range fld.Indexes { + args[n] = indexExpr(idx) + } + w.p("\t%s(%q, %s)%s", call, fld.Name, strings.Join(args, ", "), end) + } + } +} + +// literal writes a composite literal: one line with one entry, one entry per +// line otherwise. +func (w *writer) literal(indent, open string, entries []string, close string) { + if len(entries) <= 1 { + w.p("%s%s%s%s", indent, open, strings.Join(entries, ""), close) + return + } + w.p("%s%s", indent, open) + for _, e := range entries { + w.p("%s\t%s,", indent, e) + } + w.p("%s%s", indent, close) +} + +func (w *writer) codec(f *genFile) { + w.nl() + w.p("var %s = gensupport.New(gensupport.Generated[%s, %s]{", f.codecVar, f.typeExpr, f.encName) + w.p("\tTypeName: %q,", f.typeName) + w.p("\tDeclaration: %s,", f.declVar) + if f.printsPlaintext { + w.p("\tPrintsPlaintext: true,") + } + if len(f.unexported) > 0 { + quoted := make([]string, len(f.unexported)) + for i, u := range f.unexported { + quoted[i] = fmt.Sprintf("%q", u) + } + w.p("\tUnexported: []string{%s},", strings.Join(quoted, ", ")) + } + w.source(f) + w.seal(f) + w.open(f) + w.value(f) + w.p("})") +} + +func (w *writer) source(f *genFile) { + w.p("\tSource: func(v %s) gensupport.Values {", f.typeExpr) + if f.decl.Opaque { + entries := make([]string, len(f.opaque)) + for i, g := range f.opaque { + entries[i] = fmt.Sprintf("%q: v.%s", g.Name, g.GoName) + } + w.literal("\t\t", "return gensupport.Values{gensupport.OpaqueField: map[string]any{", entries, "}}") + } else { + entries := make([]string, len(f.fields)) + for i, g := range f.fields { + entries[i] = fmt.Sprintf("%q: v.%s", g.Name, g.GoName) + } + w.literal("\t\t", "return gensupport.Values{", entries, "}") + } + w.p("\t},") +} + +// seal writes Seal: the passthrough fields first, each with its error check, +// then the sealed fields in declared order. +func (w *writer) seal(f *genFile) { + w.p("\tSeal: func(rec gensupport.Record) (%s, error) {", f.encName) + if f.decl.Opaque { + w.p("\t\treturn %s{Sealed: rec[gensupport.OpaqueField].Ciphertext}, nil", f.encName) + w.p("\t},") + return + } + w.p("\t\tvar e %s", f.encName) + hasPassthrough := false + for _, g := range f.fields { + if g.Verb == VerbPassthrough { + hasPassthrough = true + } + } + if hasPassthrough { + w.p("\t\tvar err error") + } + for _, g := range f.fields { + if g.Verb != VerbPassthrough { + continue + } + w.p("\t\tif e.%s, err = gensupport.Passthrough[%s](rec, %q); err != nil {", g.GoName, g.typeExpr, g.Name) + w.p("\t\t\treturn %s{}, err", f.encName) + w.p("\t\t}") + } + for _, g := range f.fields { + switch g.Verb { + case VerbPassthrough: + case VerbEncryptInto: + w.p("\t\te.%s = %s(rec[%q].EQL)", g.GoName, g.outputType, g.Name) + default: + entries := make([]string, len(g.outputs)) + for i, o := range g.outputs { + entries[i] = fmt.Sprintf("%s: rec[%q].%s", o.name, g.Name, o.name) + } + w.literal("\t\t", fmt.Sprintf("e.%s = %s{", g.GoName, g.outputType), entries, "}") + } + } + w.p("\t\treturn e, nil") + w.p("\t},") +} + +func (w *writer) open(f *genFile) { + w.p("\tOpen: func(e %s) gensupport.Record {", f.encName) + if f.decl.Opaque { + w.p("\t\treturn gensupport.Record{gensupport.OpaqueField: {Ciphertext: e.Sealed}}") + w.p("\t},") + return + } + entries := make([]string, len(f.fields)) + for i, g := range f.fields { + switch g.Verb { + case VerbPassthrough: + entries[i] = fmt.Sprintf("%q: {Value: e.%s}", g.Name, g.GoName) + case VerbEncryptInto: + entries[i] = fmt.Sprintf("%q: {EQL: e.%s}", g.Name, g.GoName) + default: + parts := make([]string, len(g.outputs)) + for n, o := range g.outputs { + parts[n] = fmt.Sprintf("%s: e.%s.%s", o.name, g.GoName, o.name) + } + entries[i] = fmt.Sprintf("%q: {%s}", g.Name, strings.Join(parts, ", ")) + } + } + w.literal("\t\t", "return gensupport.Record{", entries, "}") + w.p("\t},") +} + +func (w *writer) value(f *genFile) { + w.p("\tValue: func(e %s, vals gensupport.Values) (%s, error) {", f.encName, f.typeExpr) + fail := func() { w.p("\t\t\treturn %s, err", f.zeroExpr) } + switch { + case f.decl.Opaque: + w.p("\t\tfields, err := gensupport.Get[gensupport.Values](vals, gensupport.OpaqueField)") + w.p("\t\tif err != nil {") + fail() + w.p("\t\t}") + w.p("\t\tvar v %s", f.typeExpr) + for _, g := range f.opaque { + w.p("\t\tif v.%s, err = gensupport.Get[%s](fields, %q); err != nil {", g.GoName, g.typeExpr, g.Name) + fail() + w.p("\t\t}") + } + default: + if f.isPointer { + w.p("\t\tv := &%s{}", strings.TrimPrefix(f.typeExpr, "*")) + } else { + w.p("\t\tvar v %s", f.typeExpr) + } + w.p("\t\tvar err error") + for _, g := range f.fields { + w.p("\t\tif v.%s, err = gensupport.Get[%s](vals, %q); err != nil {", g.GoName, g.typeExpr, g.Name) + fail() + w.p("\t\t}") + } + } + w.p("\t\treturn v, nil") + w.p("\t},") +} + +func (w *writer) functions(f *genFile) { + w.nl() + w.b.WriteString(wrapComment(fmt.Sprintf("%s seals each %s in one ZeroKMS request. The result has one element for each input, in the same order.", f.encryptFn, f.typeName), 80)) + w.p("func %s(ctx context.Context, cipher *encrypt.Cipher, %s []%s) ([]%s, error) {", f.encryptFn, f.paramName, f.typeExpr, f.encName) + w.p("\treturn %s.Encrypt(ctx, cipher, %s)", f.codecVar, f.paramName) + w.p("}") + w.nl() + w.b.WriteString(wrapComment(fmt.Sprintf("%s opens each %s in one ZeroKMS request.", f.decryptFn, f.encName), 80)) + w.p("func %s(ctx context.Context, d encrypt.Decrypter, encrypted []%s) ([]%s, error) {", f.decryptFn, f.encName, f.typeExpr) + w.p("\treturn %s.Decrypt(ctx, d, encrypted)", f.codecVar) + w.p("}") +} + +func (w *writer) fieldEntries(f *genFile) { + var sealed []genField + for _, g := range f.fields { + if g.Sealed() { + sealed = append(sealed, g) + } + } + if len(sealed) == 0 { + return + } + w.nl() + w.p("var %s = struct {", f.fieldsVar) + for _, g := range sealed { + w.p("\t%s %s", g.GoName, g.fieldType) + } + w.p("}{") + for _, g := range sealed { + w.p("\t%s: %s{gensupport.NewField[%s](%s, %q)},", g.GoName, g.fieldType, g.typeExpr, f.declVar, g.Name) + } + w.p("}") + for _, g := range sealed { + w.nl() + w.p("type %s struct {", g.fieldType) + w.p("\tfield gensupport.Field[%s]", g.typeExpr) + w.p("}") + w.nl() + w.p("func (f %s) Encrypt(ctx context.Context, c *encrypt.Cipher, v %s) (%s, error) {", g.fieldType, g.typeExpr, g.outputType) + switch { + case g.Verb == VerbEncryptInto: + w.p("\tout, err := f.field.Encrypt(ctx, c, v)") + w.p("\treturn %s(out.EQL), err", g.outputType) + case len(g.outputs) == 1: + w.p("\tout, err := f.field.Encrypt(ctx, c, v)") + w.p("\treturn %s{%s: out.%s}, err", g.outputType, g.outputs[0].name, g.outputs[0].name) + default: + w.p("\tout, err := f.field.Encrypt(ctx, c, v)") + w.p("\tif err != nil {") + w.p("\t\treturn %s{}, err", g.outputType) + w.p("\t}") + parts := make([]string, len(g.outputs)) + for i, o := range g.outputs { + parts[i] = fmt.Sprintf("%s: out.%s", o.name, o.name) + } + w.p("\treturn %s{%s}, nil", g.outputType, strings.Join(parts, ", ")) + } + w.p("}") + if g.queryType != "" { + w.nl() + w.p("func (f %s) Query(ctx context.Context, c *encrypt.Cipher, v %s) (%s, error) {", g.fieldType, g.typeExpr, g.queryType) + w.p("\tout, err := f.field.Query(ctx, c, v)") + w.p("\treturn %s(out.EQL), err", g.queryType) + w.p("}") + } + for _, o := range g.outputs { + if o.index == "" { + continue + } + w.nl() + w.p("func (f %s) %s(ctx context.Context, c *encrypt.Cipher, v %s) (%s, error) {", g.fieldType, o.name, g.typeExpr, o.typeExpr) + w.p("\treturn f.field.%s(ctx, c, v)", o.name) + w.p("}") + } + } +} + +func (w *writer) model(f *genFile, m *genModel) { + w.nl() + w.p("// The -model flag: %s converts to and from its shape only while the", m.typeExpr) + w.p("// two have the same fields, with the same types, in the same order.") + w.p("type %s struct {", m.shapeName) + for _, mf := range m.fields { + w.p("\t%s %s", mf.name, mf.typeExpr) + } + w.p("}") + w.nl() + w.p("var %s = gensupport.Records(%s,", m.codecVar, f.codecVar) + w.p("\tfunc(e %s) %s {", f.encName, m.typeExpr) + var entries []string + for _, mf := range m.fields { + if mf.source != "" { + entries = append(entries, fmt.Sprintf("%s: %s", mf.name, mf.source)) + } + } + w.literal("\t\t", fmt.Sprintf("return %s(%s{", m.typeExpr, m.shapeName), entries, "})") + w.p("\t},") + w.p("\tfunc(r %s) %s {", m.typeExpr, f.encName) + w.p("\t\ts := %s(r)", m.shapeName) + entries = entries[:0] + for _, g := range f.fields { + switch g.Verb { + case VerbPassthrough, VerbEncryptInto: + entries = append(entries, fmt.Sprintf("%s: s.%s", g.GoName, m.fieldFor(&g, &g.outputs[0]))) + default: + parts := make([]string, len(g.outputs)) + for i := range g.outputs { + parts[i] = fmt.Sprintf("%s: s.%s", g.outputs[i].name, m.fieldFor(&g, &g.outputs[i])) + } + entries = append(entries, fmt.Sprintf("%s: %s{%s}", g.GoName, g.outputType, strings.Join(parts, ", "))) + } + } + w.literal("\t\t", fmt.Sprintf("return %s{", f.encName), entries, "}") + w.p("\t},") + w.p(")") + w.nl() + w.p("func %s(ctx context.Context, cipher *encrypt.Cipher, %s []%s) ([]%s, error) {", m.encryptFn, f.paramName, f.typeExpr, m.typeExpr) + w.p("\treturn %s.Encrypt(ctx, cipher, %s)", m.codecVar, f.paramName) + w.p("}") + w.nl() + w.p("func %s(ctx context.Context, d encrypt.Decrypter, %s []%s) ([]%s, error) {", m.decryptFn, m.paramName, m.typeExpr, f.typeExpr) + w.p("\treturn %s.Decrypt(ctx, d, %s)", m.codecVar, m.paramName) + w.p("}") +} + +// fieldFor is the model field bound to one output. +func (m *genModel) fieldFor(g *genField, out *output) string { + for _, mf := range m.fields { + if mf.field != nil && mf.field.Name == g.Name && mf.output != nil && mf.output.name == out.name { + return mf.name + } + } + return "/* unbound " + g.Name + " */" +} diff --git a/languages/golang/stashgen/engine.go b/languages/golang/stashgen/engine.go new file mode 100644 index 000000000..b43bec277 --- /dev/null +++ b/languages/golang/stashgen/engine.go @@ -0,0 +1,51 @@ +package stashgen + +import ( + "context" + "errors" +) + +// Engine answers the generator's questions about what the Rust engine can do. +// The generator holds no copy of the engine's rules: every refusal about an +// index, an EQL type or a field type comes from here. +// +// The SDK's implementation runs the WASI guest that the SDK embeds. See +// [GuestEngine]. +type Engine interface { + // EQLTypes lists the EQL types the engine can produce: each name, its + // plaintext kind, its indexes and its query type. + EQLTypes(ctx context.Context) ([]EQLType, error) + // Check refuses a declaration the engine cannot run: an index or an EQL + // type that does not apply to a field's Go type, a field type it cannot + // seal, an EQL type it cannot produce, or an index option it cannot + // carry. The error names the field. + Check(ctx context.Context, d Declaration) error +} + +// EQLType is one EQL type the engine produces, such as TextEq. +type EQLType struct { + // Name is the Go type name in encrypt/eql, and the value of encrypt_into. + Name string + // Plaintext is the kind of Go value the type seals. + Plaintext Kind + // Indexes are the terms the type carries. + Indexes []IndexName + // Query is the Go type name of the type's query value, such as + // TextEqQuery, or "" for a type with no terms. + Query string +} + +// ErrEngineUnavailable is returned by [GuestEngine] until the SDK's guest +// is wired to the generator. +var ErrEngineUnavailable = errors.New("stashgen: the embedded engine is not available in this build") + +// GuestEngine returns the engine the SDK embeds. +// +// TODO(stack#1046 integration): run the WASI guest that package encrypt +// embeds (the build with the EQL types), ask it for its EQL types and have +// it check each declaration. Until then this returns [ErrEngineUnavailable], +// and stashgen stops before it reads any package. The fake in +// internal/fakeengine is the reference for what an Engine answers. +func GuestEngine(context.Context) (Engine, error) { + return nil, ErrEngineUnavailable +} diff --git a/languages/golang/stashgen/errors.go b/languages/golang/stashgen/errors.go new file mode 100644 index 000000000..eaa25e0db --- /dev/null +++ b/languages/golang/stashgen/errors.go @@ -0,0 +1,25 @@ +package stashgen + +import "fmt" + +// FieldError is a mistake in one field's declaration. Every refusal names the +// type and the field, and never a value. +type FieldError struct { + // Type is the Go type the field belongs to: "User" or "crm.Contact". + Type string + // Field is the Go name of the field, or "" for a mistake about the type. + Field string + // Reason says what is wrong, without the type and field. + Reason string +} + +func (e *FieldError) Error() string { + if e.Field == "" { + return fmt.Sprintf("stashgen: %s: %s", e.Type, e.Reason) + } + return fmt.Sprintf("stashgen: %s.%s: %s", e.Type, e.Field, e.Reason) +} + +func fieldErr(typeName, field, format string, args ...any) error { + return &FieldError{Type: typeName, Field: field, Reason: fmt.Sprintf(format, args...)} +} diff --git a/languages/golang/stashgen/generate.go b/languages/golang/stashgen/generate.go new file mode 100644 index 000000000..ca3ca7470 --- /dev/null +++ b/languages/golang/stashgen/generate.go @@ -0,0 +1,114 @@ +package stashgen + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "strings" +) + +// Request is one run of the generator over one tagged struct. The command +// fills it from its flags. +type Request struct { + // Dir is the directory of the package. "" is the working directory. + Dir string + // Type is the struct that carries the stash tags. + Type string + // Name, when set, writes EncryptName, DecryptName and NameFields. + Name string + // For, as P.F, is the type in another package that Type declares for. + For string + // Models are the -model flags. + Models []ModelRequest + // Redact writes String and LogValue methods on Type. + Redact bool + // Output is the file to write, relative to Dir. "" is the type's name in + // lower case with _stash.go. + Output string +} + +// OutputPath is the file the request writes, relative to Dir. +func (r Request) OutputPath() string { + if r.Output != "" { + return r.Output + } + return strings.ToLower(r.Type) + "_stash.go" +} + +// File is a generated file, ready to write. +type File struct { + // Path is where the file goes. + Path string + // Content is the formatted Go source. + Content []byte + // Notices are what the generator prints to stderr: the unexported fields + // it ignored, and a type that prints its sealed fields. + Notices []string +} + +// Write writes the file. Generated source is committed and read by everyone +// who builds the package, so it gets the mode of any other source file. +func (f *File) Write() error { + return os.WriteFile(f.Path, f.Content, 0o644) //nolint:gosec // source code, not a secret +} + +// FromTags generates the file for the struct named in the request. It loads +// the package with go/packages and reads types, not text; it runs none of the +// package's code; it checks the declaration with the engine; and it returns +// the file without writing it. The same request always gives the same file. +func FromTags(ctx context.Context, engine Engine, req Request) (*File, error) { + if engine == nil { + return nil, errors.New("stashgen: no engine") + } + if req.Type == "" { + return nil, errors.New("stashgen: -type is required") + } + if !isIdent(req.Type) { + return nil, fmt.Errorf("stashgen: -type %q is not a type name in this package", req.Type) + } + if req.Name != "" && (!isIdent(req.Name) || strings.ToUpper(req.Name[:1]) != req.Name[:1]) { + return nil, fmt.Errorf("stashgen: -name %q must be an exported Go name", req.Name) + } + dir := req.Dir + if dir == "" { + dir = "." + } + outPath := filepath.Join(dir, req.OutputPath()) + + eqlTypes, err := engine.EQLTypes(ctx) + if err != nil { + return nil, fmt.Errorf("stashgen: the engine's EQL types: %w", err) + } + + pkg, err := loadPackage(ctx, dir, outPath) + if err != nil { + return nil, err + } + r := &reader{pkg: pkg, req: req, eql: eqlTypes, imports: newImportSet()} + gf, err := r.read() + if err != nil { + return nil, err + } + if err := engine.Check(ctx, gf.decl); err != nil { + return nil, err + } + src, err := emit(gf) + if err != nil { + return nil, err + } + return &File{Path: outPath, Content: src, Notices: gf.stderrNotices()}, nil +} + +// stderrNotices are the notices in the form the generator prints. +func (f *genFile) stderrNotices() []string { + var out []string + if len(f.unexported) > 0 { + out = append(out, fmt.Sprintf("stashgen: %s: not encrypted and not stored: the unexported %s. Tag %s `stash:\"-\"` to confirm that.", f.typeName, fieldList(f.unexported), itOrEach(f.unexported))) + } + if f.printsPlaintext { + out = append(out, fmt.Sprintf("stashgen: %s prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", f.typeName)) + } + return out +} diff --git a/languages/golang/stashgen/golden_test.go b/languages/golang/stashgen/golden_test.go new file mode 100644 index 000000000..548717a11 --- /dev/null +++ b/languages/golang/stashgen/golden_test.go @@ -0,0 +1,140 @@ +package stashgen_test + +import ( + "bytes" + "context" + "flag" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/stashgen" + "github.com/cipherstash/stack/languages/golang/stashgen/internal/fakeengine" +) + +var update = flag.Bool("update", false, "rewrite the golden files from the generator's output") + +// goldenCase is one module under testdata/cases: the example input, the flags +// its go:generate line carries, and the golden file the generator must write. +type goldenCase struct { + dir string // the package directory, relative to the case module + req stashgen.Request + golden string +} + +var goldenCases = map[string]goldenCase{ + "users": {req: stashgen.Request{Type: "User"}, golden: "user_stash.go.golden"}, + "accounts": {req: stashgen.Request{Type: "Account", Redact: true}, golden: "account_stash.go.golden"}, + "contacts": {req: stashgen.Request{Type: "contactStash", For: "crm.Contact", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "ContactRow"}}}, golden: "contactstash_stash.go.golden"}, + "documents": {req: stashgen.Request{Type: "Document"}, golden: "document_stash.go.golden"}, +} + +func TestGolden(t *testing.T) { + for name, c := range goldenCases { + t.Run(name, func(t *testing.T) { + caseDir := filepath.Join("testdata", "cases", name) + req := c.req + req.Dir = filepath.Join(caseDir, c.dir) + file, err := stashgen.FromTags(context.Background(), fakeengine.Engine{}, req) + if err != nil { + t.Fatalf("FromTags: %v", err) + } + goldenPath := filepath.Join(caseDir, c.dir, c.golden) + if *update { + if err := os.WriteFile(goldenPath, file.Content, 0o644); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(goldenPath) + if err != nil { + t.Fatalf("read golden: %v (run with -update to write it)", err) + } + if !bytes.Equal(file.Content, want) { + t.Errorf("generated file differs from %s (run with -update to rewrite it):\n%s", goldenPath, diff(string(want), string(file.Content))) + } + if got := filepath.Base(file.Path); got+".golden" != c.golden { + t.Errorf("output path %q, want %q", got, strings.TrimSuffix(c.golden, ".golden")) + } + compileCase(t, caseDir, c.dir, filepath.Base(file.Path), file.Content) + }) + } +} + +// compileCase copies the case module to a temp dir with the generated file in +// it and builds it against the stub SDK, so uncompilable output fails here. +func compileCase(t *testing.T, caseDir, pkgDir, fileName string, content []byte) { + t.Helper() + tmp := t.TempDir() + if err := copyTree(caseDir, tmp); err != nil { + t.Fatal(err) + } + // The case's go.mod replaces the SDK and gorm with relative paths; from + // the temp dir those must be absolute. + testdata, err := filepath.Abs("testdata") + if err != nil { + t.Fatal(err) + } + gomod, err := os.ReadFile(filepath.Join(tmp, "go.mod")) + if err != nil { + t.Fatal(err) + } + gomod = bytes.ReplaceAll(gomod, []byte("../../stubsdk"), []byte(filepath.Join(testdata, "stubsdk"))) + gomod = bytes.ReplaceAll(gomod, []byte("../../stubgorm"), []byte(filepath.Join(testdata, "stubgorm"))) + if err := os.WriteFile(filepath.Join(tmp, "go.mod"), gomod, 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(tmp, pkgDir, fileName), content, 0o644); err != nil { + t.Fatal(err) + } + cmd := exec.Command("go", "vet", "./...") + cmd.Dir = tmp + cmd.Env = append(os.Environ(), "GOPROXY=off", "GOWORK=off", "GOFLAGS=-mod=mod", "CGO_ENABLED=0") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("generated code does not build: %v\n%s", err, out) + } +} + +func copyTree(src, dst string) error { + return filepath.WalkDir(src, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + rel, err := filepath.Rel(src, path) + if err != nil { + return err + } + target := filepath.Join(dst, rel) + if d.IsDir() { + return os.MkdirAll(target, 0o755) + } + if strings.HasSuffix(path, ".golden") { + return nil + } + data, err := os.ReadFile(path) + if err != nil { + return err + } + return os.WriteFile(target, data, 0o644) + }) +} + +// diff is a line diff good enough to read a golden mismatch. +func diff(want, got string) string { + w, g := strings.Split(want, "\n"), strings.Split(got, "\n") + var b strings.Builder + for i := 0; i < len(w) || i < len(g); i++ { + var wl, gl string + if i < len(w) { + wl = w[i] + } + if i < len(g) { + gl = g[i] + } + if wl != gl { + b.WriteString("-" + wl + "\n+" + gl + "\n") + } + } + return b.String() +} diff --git a/languages/golang/stashgen/imports.go b/languages/golang/stashgen/imports.go new file mode 100644 index 000000000..010e577c6 --- /dev/null +++ b/languages/golang/stashgen/imports.go @@ -0,0 +1,79 @@ +package stashgen + +import ( + "fmt" + "sort" + "strings" +) + +// importSet is the imports of the generated file: each path with the name the +// file uses for it. +type importSet struct { + names map[string]string // path -> name + paths map[string]string // name -> path + err error +} + +func newImportSet() *importSet { + return &importSet{names: map[string]string{}, paths: map[string]string{}} +} + +// add records an import and returns the name to use for it. Two packages +// with one name is an error the generator reports before it writes. +func (s *importSet) add(path, name string) string { + if existing, ok := s.names[path]; ok { + return existing + } + if other, taken := s.paths[name]; taken && other != path { + if s.err == nil { + s.err = fmt.Errorf("stashgen: two imports are named %s: %s and %s", name, other, path) + } + return name + } + s.names[path] = name + s.paths[name] = path + return name +} + +// block writes the import declaration: the standard library first, then the +// rest, each group sorted by path. +func (s *importSet) block() string { + var std, rest []string + for path := range s.names { + if strings.Contains(strings.SplitN(path, "/", 2)[0], ".") { + rest = append(rest, path) + } else { + std = append(std, path) + } + } + sort.Strings(std) + sort.Strings(rest) + var b strings.Builder + b.WriteString("import (\n") + write := func(path string) { + name := s.names[path] + if name != lastElement(path) { + fmt.Fprintf(&b, "\t%s %q\n", name, path) + } else { + fmt.Fprintf(&b, "\t%q\n", path) + } + } + for _, p := range std { + write(p) + } + if len(std) > 0 && len(rest) > 0 { + b.WriteString("\n") + } + for _, p := range rest { + write(p) + } + b.WriteString(")\n") + return b.String() +} + +func lastElement(path string) string { + if i := strings.LastIndexByte(path, '/'); i >= 0 { + return path[i+1:] + } + return path +} diff --git a/languages/golang/stashgen/internal/fakeengine/fakeengine.go b/languages/golang/stashgen/internal/fakeengine/fakeengine.go new file mode 100644 index 000000000..33f050312 --- /dev/null +++ b/languages/golang/stashgen/internal/fakeengine/fakeengine.go @@ -0,0 +1,111 @@ +// Package fakeengine is a static stand-in for the Rust engine in the +// generator's tests. It knows what the engine produces today: the EQL type +// TextEq, and the four indexes with the Go kinds each applies to. +// +// It is a test double, not a copy of the rules the SDK ships: the SDK's +// generator asks the embedded guest. When the engine learns a type or an +// index, update this file and the tests that read it. +package fakeengine + +import ( + "context" + "fmt" + + "github.com/cipherstash/stack/languages/golang/stashgen" +) + +// Engine is the static fake. The zero value is ready to use. +type Engine struct{} + +var _ stashgen.Engine = Engine{} + +// EQLTypes returns TextEq, the one EQL type the engine produces today. +func (Engine) EQLTypes(context.Context) ([]stashgen.EQLType, error) { + return []stashgen.EQLType{ + {Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery"}, + }, nil +} + +// Check applies the engine's rules as they stand today. +func (e Engine) Check(ctx context.Context, d stashgen.Declaration) error { + eqlTypes, _ := e.EQLTypes(ctx) + for _, f := range d.Fields { + if f.Verb == stashgen.VerbOmit { + continue + } + if d.Opaque || f.Sealed() { + if !sealable(f.GoType) { + return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine cannot seal a value of type %s", f.GoType)} + } + } + if f.Verb == stashgen.VerbEncryptInto { + var found *stashgen.EQLType + for i := range eqlTypes { + if eqlTypes[i].Name == f.EQLType { + found = &eqlTypes[i] + } + } + if found == nil { + return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine cannot produce the EQL type %s yet; it produces TextEq", f.EQLType)} + } + if found.Plaintext != f.GoType.Kind { + return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("%s seals a %s, and %s is %s", f.EQLType, found.Plaintext, f.GoType, f.GoType.Kind)} + } + } + for _, idx := range f.Indexes { + if len(idx.Options) > 0 { + return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("index %s: the engine cannot carry index options yet", idx)} + } + if err := indexApplies(idx.Name, f.GoType); err != nil { + return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: err.Error()} + } + } + } + return nil +} + +// indexApplies mirrors the Index impls in stack-encrypt's target/index.rs: +// Equality for every PRF value, Match for text, Ore and Ope for what cllw-ore +// implements (text, bytes, bool, integers and floats). The JSON index is not +// in the engine yet. +func indexApplies(name stashgen.IndexName, t stashgen.GoType) error { + switch name { + case stashgen.IndexEquality: + if t.Kind.Scalar() { + return nil + } + case stashgen.IndexMatch: + if t.Kind == stashgen.KindString { + return nil + } + return fmt.Errorf("match applies to a string, not to %s", t) + case stashgen.IndexOre, stashgen.IndexOpe: + if t.Kind.Scalar() { + return nil + } + case stashgen.IndexJSON: + return fmt.Errorf("the engine cannot derive the json index yet") + } + return fmt.Errorf("%s does not apply to %s", name, t) +} + +// sealable reports whether the engine can seal a value of the type: a scalar, +// or a slice, map, pointer or all-exported struct of sealable types. +func sealable(t stashgen.GoType) bool { + switch t.Kind { + case stashgen.KindString, stashgen.KindBool, stashgen.KindInt, stashgen.KindUint, stashgen.KindFloat, stashgen.KindBytes: + return true + case stashgen.KindSlice, stashgen.KindPointer: + return t.Elem != nil && sealable(*t.Elem) + case stashgen.KindMap: + return t.Elem != nil && sealable(*t.Elem) + case stashgen.KindStruct: + for _, f := range t.Fields { + if !sealable(f) { + return false + } + } + return true + } + return false +} diff --git a/languages/golang/stashgen/load.go b/languages/golang/stashgen/load.go new file mode 100644 index 000000000..6a20ad6e4 --- /dev/null +++ b/languages/golang/stashgen/load.go @@ -0,0 +1,150 @@ +package stashgen + +import ( + "context" + "fmt" + "go/ast" + "go/parser" + "go/token" + "go/types" + "path/filepath" + "strings" + + "golang.org/x/tools/go/packages" +) + +// loadPackage loads the package in dir with its types. It reads types, not +// text, and runs none of the package's code. The file at ignore, the +// generator's own output, is read as its package clause only, so a stale +// generated file does not stop the generator. +func loadPackage(ctx context.Context, dir, ignore string) (*packages.Package, error) { + ignoreAbs, err := filepath.Abs(ignore) + if err != nil { + return nil, err + } + cfg := &packages.Config{ + Context: ctx, + Dir: dir, + Mode: packages.NeedName | packages.NeedFiles | packages.NeedCompiledGoFiles | + packages.NeedImports | packages.NeedTypes | packages.NeedTypesInfo | packages.NeedSyntax, + ParseFile: func(fset *token.FileSet, filename string, src []byte) (*ast.File, error) { + mode := parser.ParseComments | parser.SkipObjectResolution + if sameFile(filename, ignoreAbs) { + mode |= parser.PackageClauseOnly + } + return parser.ParseFile(fset, filename, src, mode) + }, + } + pkgs, err := packages.Load(cfg, ".") + if err != nil { + return nil, fmt.Errorf("stashgen: load %s: %w", dir, err) + } + if len(pkgs) != 1 { + return nil, fmt.Errorf("stashgen: %s holds %d packages, and the generator reads one", dir, len(pkgs)) + } + pkg := pkgs[0] + if pkg.Types == nil || pkg.Types.Scope() == nil { + return nil, fmt.Errorf("stashgen: %s: no types: %s", dir, packageErrors(pkg)) + } + return pkg, nil +} + +func sameFile(a, b string) bool { + absA, err := filepath.Abs(a) + if err != nil { + return false + } + return absA == b +} + +// packageErrors joins the errors go/packages reported, for a message that +// explains why a type could not be read. +func packageErrors(pkg *packages.Package) string { + if len(pkg.Errors) == 0 { + return "no errors reported" + } + msgs := make([]string, 0, len(pkg.Errors)) + for _, e := range pkg.Errors { + msgs = append(msgs, e.Error()) + } + return strings.Join(msgs, "; ") +} + +// lookupStruct finds a named struct type in the package's scope. +func lookupStruct(pkg *packages.Package, name string) (*types.Named, *types.Struct, error) { + obj := pkg.Types.Scope().Lookup(name) + if obj == nil { + return nil, nil, fmt.Errorf("stashgen: package %s has no type %s (%s)", pkg.Name, name, packageErrors(pkg)) + } + tn, ok := obj.(*types.TypeName) + if !ok { + return nil, nil, fmt.Errorf("stashgen: %s.%s is not a type", pkg.Name, name) + } + named, ok := tn.Type().(*types.Named) + if !ok { + return nil, nil, fmt.Errorf("stashgen: %s.%s is not a named struct type", pkg.Name, name) + } + st, ok := named.Underlying().(*types.Struct) + if !ok { + return nil, nil, fmt.Errorf("stashgen: %s.%s is a %s, and the generator reads a struct", pkg.Name, name, named.Underlying()) + } + return named, st, nil +} + +// lookupQualified resolves "crm.Contact" through the package's imports: the +// package whose name is crm, and its type Contact. +func lookupQualified(pkg *packages.Package, qualified string) (*types.Named, *types.Struct, error) { + i := strings.LastIndexByte(qualified, '.') + if i <= 0 || i == len(qualified)-1 { + return nil, nil, fmt.Errorf("stashgen: %q is not P.F, a type F in an imported package P", qualified) + } + pkgName, typeName := qualified[:i], qualified[i+1:] + var found *packages.Package + for _, imp := range pkg.Imports { + if imp.Types != nil && imp.Types.Name() == pkgName { + if found != nil { + return nil, nil, fmt.Errorf("stashgen: %s imports two packages named %s: %s and %s", pkg.Name, pkgName, found.PkgPath, imp.PkgPath) + } + found = imp + } + } + if found == nil { + return nil, nil, fmt.Errorf("stashgen: package %s imports no package named %s", pkg.Name, pkgName) + } + return lookupStruct(found, typeName) +} + +// hasInvalidType reports whether a type mentions the invalid type, which the +// checker uses for what it could not resolve. +func hasInvalidType(t types.Type) bool { + seen := map[types.Type]bool{} + var walk func(t types.Type) bool + walk = func(t types.Type) bool { + if t == nil || seen[t] { + return false + } + seen[t] = true + switch t := t.(type) { + case *types.Basic: + return t.Kind() == types.Invalid + case *types.Named: + return walk(t.Underlying()) + case *types.Pointer: + return walk(t.Elem()) + case *types.Slice: + return walk(t.Elem()) + case *types.Array: + return walk(t.Elem()) + case *types.Map: + return walk(t.Key()) || walk(t.Elem()) + case *types.Struct: + for i := range t.NumFields() { + if walk(t.Field(i).Type()) { + return true + } + } + } + return false + } + return walk(t) +} diff --git a/languages/golang/stashgen/model.go b/languages/golang/stashgen/model.go new file mode 100644 index 000000000..212afe5f5 --- /dev/null +++ b/languages/golang/stashgen/model.go @@ -0,0 +1,253 @@ +package stashgen + +import ( + "fmt" + "go/types" + "strings" +) + +// ModelRequest is one -model flag: a struct with one field for each column, +// for separate columns. +type ModelRequest struct { + // Name is the suffix of the generated functions: EncryptName, DecryptName. + Name string + // Type is the model struct, in this package or as P.T. + Type string + // Declares is a struct in this package that carries the tags for a Type + // that cannot, or "". + Declares string +} + +// ParseModelFlag reads a -model flag: Name=R or Name=R:D. +func ParseModelFlag(s string) (ModelRequest, error) { + name, rest, ok := strings.Cut(s, "=") + if !ok || name == "" || rest == "" { + return ModelRequest{}, fmt.Errorf("stashgen: -model %q is not Name=R or Name=R:D", s) + } + if !isIdent(name) || strings.ToUpper(name[:1]) != name[:1] { + return ModelRequest{}, fmt.Errorf("stashgen: -model %q: %q must be an exported Go name", s, name) + } + typ, declares, _ := strings.Cut(rest, ":") + if typ == "" || (strings.Contains(rest, ":") && declares == "") { + return ModelRequest{}, fmt.Errorf("stashgen: -model %q is not Name=R or Name=R:D", s) + } + return ModelRequest{Name: name, Type: typ, Declares: declares}, nil +} + +// readModel binds a model's fields to the declaration's outputs. +func (r *reader) readModel(m ModelRequest) error { + f := r.file + if f.decl.Opaque { + return fmt.Errorf("stashgen: -model %s: an opaque struct has one output and needs no model", m.Name) + } + modelNamed, modelStruct, err := r.lookupAny(m.Type) + if err != nil { + return err + } + if hasUnexported(modelStruct) && modelNamed.Obj().Pkg().Path() != r.pkg.PkgPath { + return fmt.Errorf("stashgen: -model %s: %s has an unexported field, so Go cannot convert it to a copy of its fields", m.Name, m.Type) + } + tagStruct := modelStruct + tagType := m.Type + if m.Declares != "" { + var declNamed *types.Named + declNamed, tagStruct, err = lookupStruct(r.pkg, m.Declares) + if err != nil { + return err + } + _ = declNamed + tagType = m.Declares + if err := matchFields(tagType, tagStruct, m.Type, modelStruct, r.typeExpr); err != nil { + return err + } + } + tagsByName := map[string]string{} + for i := range tagStruct.NumFields() { + tagsByName[tagStruct.Field(i).Name()] = tagStruct.Tag(i) + } + + gm := genModel{ + name: m.Name, + typeExpr: r.typeExpr(modelNamed), + shapeName: lowerFirst(m.Name) + "Shape", + codecVar: lowerFirst(m.Name) + "Codec", + encryptFn: "Encrypt" + m.Name, + decryptFn: "Decrypt" + m.Name, + paramName: lowerFirst(m.Name), + } + if f.fieldTypePrefix != "" { + gm.shapeName = lowerFirst(f.fieldTypePrefix) + m.Name + "Shape" + gm.codecVar = lowerFirst(f.fieldTypePrefix) + m.Name + "Codec" + } + if gm.paramName == f.paramName || gm.paramName == "ctx" || gm.paramName == "d" || goKeywords[gm.paramName] { + gm.paramName = "models" + } + + bound := map[string]bool{} // "email" or "email,equality" + for i := range modelStruct.NumFields() { + fld := modelStruct.Field(i) + if fld.Name() == "_" { + return fmt.Errorf("stashgen: -model %s: %s has a `_` field, which cannot be a column", m.Name, m.Type) + } + mf := modelField{name: fld.Name(), typeExpr: r.typeExpr(fld.Type())} + raw, ok := tagsByName[fld.Name()] + t, err := parseModelTag(raw) + if !ok || err == errNoTag { + return fieldErr(tagType, fld.Name(), "no stash tag; each field of a model names one output, `stash:\"email\"` or `stash:\"email,equality\"`, or `stash:\"-\"`") + } + if err != nil { + return fieldErr(tagType, fld.Name(), "%v", err) + } + if t.omit { + gm.fields = append(gm.fields, mf) + continue + } + if fld.Embedded() { + return fieldErr(tagType, fld.Name(), "a model's embedded struct is one column per field, which it cannot name; tag it `stash:\"-\"`") + } + key := t.field + if t.index != "" { + key += "," + string(t.index) + } + if bound[key] { + return fieldErr(tagType, fld.Name(), "the output %q is already bound to another field", key) + } + bound[key] = true + g, out := f.lookupOutput(t.field, t.index) + if g == nil { + return fieldErr(tagType, fld.Name(), "%s declares no field named %q", f.typeName, t.field) + } + if out == nil { + if t.index == "" { + return fieldErr(tagType, fld.Name(), "field %q has no ciphertext; it has only indexes", t.field) + } + return fieldErr(tagType, fld.Name(), "field %q has no %s index", t.field, t.index) + } + if got := pathType(fld.Type()); got != out.pathType { + return fieldErr(tagType, fld.Name(), "has type %s, and the %s of %q is %s", r.typeExpr(fld.Type()), outputWord(g, out), t.field, out.typeExpr) + } + mf.field, mf.output = g, out + mf.source = sourceExpr("e", g, out) + gm.fields = append(gm.fields, mf) + } + for _, g := range f.fields { + for i := range g.outputs { + out := &g.outputs[i] + key := g.Name + if out.index != "" { + key += "," + string(out.index) + } + if !bound[key] { + return fieldErr(tagType, "", "no field for the %s of %q", outputWord(&g, out), g.Name) + } + } + } + f.models = append(f.models, gm) + return nil +} + +func outputWord(g *genField, out *output) string { + switch { + case out.index != "": + return string(out.index) + " term" + case g.Verb == VerbPassthrough: + return "value" + case g.Verb == VerbEncryptInto: + return "EQL value" + } + return "ciphertext" +} + +// sourceExpr is the expression that reads one output from the encrypted value e. +func sourceExpr(e string, g *genField, out *output) string { + switch g.Verb { + case VerbPassthrough, VerbEncryptInto: + return e + "." + g.GoName + } + return e + "." + g.GoName + "." + out.name +} + +func (f *genFile) lookupOutput(name string, index IndexName) (*genField, *output) { + for i := range f.fields { + g := &f.fields[i] + if g.Name != name { + continue + } + for j := range g.outputs { + out := &g.outputs[j] + if out.index == index && (index != "" || out.name != "") { + if index == "" && out.index != "" { + continue + } + return g, out + } + } + return g, nil + } + return nil, nil +} + +type modelTag struct { + omit bool + field string + index IndexName +} + +// parseModelTag reads `email`, `email,equality` or `-`. +func parseModelTag(raw string) (modelTag, error) { + value, ok := lookupTag(raw, tagKey) + if !ok { + return modelTag{}, errNoTag + } + if value == "-" { + return modelTag{omit: true}, nil + } + name, index, hasIndex := strings.Cut(value, ",") + if name == "" || strings.ContainsAny(name, "=();") { + return modelTag{}, fmt.Errorf("tag %q: the first part names a field of the declaration", value) + } + t := modelTag{field: name} + if hasIndex { + if !knownIndex(IndexName(index)) { + return modelTag{}, fmt.Errorf("tag %q: %q is not an index; the indexes are equality, match, ore, ope and json", value, index) + } + t.index = IndexName(index) + } + return t, nil +} + +// lookupAny resolves a type name in this package, or P.T through its imports. +func (r *reader) lookupAny(name string) (*types.Named, *types.Struct, error) { + if strings.Contains(name, ".") { + return lookupQualified(r.pkg, name) + } + return lookupStruct(r.pkg, name) +} + +// matchFields checks that a declaring struct names every exported field of +// the struct it declares for, with the same types. +func matchFields(declName string, decl *types.Struct, forName string, target *types.Struct, typeExpr func(types.Type) string) error { + byName := map[string]*types.Var{} + for i := range target.NumFields() { + byName[target.Field(i).Name()] = target.Field(i) + } + named := map[string]bool{} + for i := range decl.NumFields() { + df := decl.Field(i) + tf, ok := byName[df.Name()] + if !ok { + return fieldErr(declName, df.Name(), "%s has no field %s", forName, df.Name()) + } + if !types.Identical(tf.Type(), df.Type()) { + return fieldErr(declName, df.Name(), "has type %s, and %s.%s has type %s", typeExpr(df.Type()), forName, df.Name(), typeExpr(tf.Type())) + } + named[df.Name()] = true + } + for i := range target.NumFields() { + tf := target.Field(i) + if tf.Exported() && !named[tf.Name()] { + return fieldErr(forName, tf.Name(), "not named by %s; every exported field of %s needs a tag there", declName, forName) + } + } + return nil +} diff --git a/languages/golang/stashgen/names.go b/languages/golang/stashgen/names.go new file mode 100644 index 000000000..3c956df74 --- /dev/null +++ b/languages/golang/stashgen/names.go @@ -0,0 +1,67 @@ +package stashgen + +import ( + "strings" + "unicode" +) + +// snakeCase turns a Go name into a column name: ID -> id, CreatedAt -> +// created_at, HTTPServer -> http_server, MedicareNo -> medicare_no. A run of +// capitals is one word until its last letter starts the next word. +func snakeCase(name string) string { + runes := []rune(name) + var b strings.Builder + for i, r := range runes { + if unicode.IsUpper(r) && i > 0 { + prev := runes[i-1] + nextLower := i+1 < len(runes) && unicode.IsLower(runes[i+1]) + if unicode.IsLower(prev) || unicode.IsDigit(prev) || (unicode.IsUpper(prev) && nextLower) { + b.WriteByte('_') + } + } + b.WriteRune(unicode.ToLower(r)) + } + return b.String() +} + +// lowerFirst lowers a name's first letter: User -> user, ContactRow -> +// contactRow. A leading run of capitals lowers as one: ID -> id, HTTPServer +// -> httpServer. +func lowerFirst(name string) string { + runes := []rune(name) + n := 0 + for n < len(runes) && unicode.IsUpper(runes[n]) { + n++ + } + if n > 1 && n < len(runes) { + n-- + } + for i := 0; i < n; i++ { + runes[i] = unicode.ToLower(runes[i]) + } + return string(runes) +} + +// wrapComment writes text as a // comment wrapped at width columns. +func wrapComment(text string, width int) string { + var lines []string + line := "//" + for _, word := range strings.Fields(text) { + if len(line) > 2 && len(line)+1+len(word) > width { + lines = append(lines, line) + line = "//" + } + line += " " + word + } + lines = append(lines, line) + return strings.Join(lines, "\n") + "\n" +} + +// goKeywords are the words a local variable cannot be named. +var goKeywords = map[string]bool{ + "break": true, "case": true, "chan": true, "const": true, "continue": true, + "default": true, "defer": true, "else": true, "fallthrough": true, "for": true, + "func": true, "go": true, "goto": true, "if": true, "import": true, + "interface": true, "map": true, "package": true, "range": true, "return": true, + "select": true, "struct": true, "switch": true, "type": true, "var": true, +} diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go new file mode 100644 index 000000000..14f84e91d --- /dev/null +++ b/languages/golang/stashgen/read.go @@ -0,0 +1,734 @@ +package stashgen + +import ( + "fmt" + "go/types" + "strings" + + "golang.org/x/tools/go/packages" +) + +// The import paths of the SDK packages generated code names. +const ( + encryptPath = "github.com/cipherstash/stack/languages/golang/encrypt" + eqlPath = encryptPath + "/eql" + gensupportPath = encryptPath + "/gensupport" +) + +// genFile is everything the emitter needs for one generated file. The reader +// builds it from the package's types; the emitter turns it into Go source. +type genFile struct { + pkgName string + imports *importSet + + // The plaintext type. + typeName string // how messages and TypeName print it: "User", "crm.Contact" + typeExpr string // the Go type the functions take: "User", "crm.Contact", "*pb.Individual" + zeroExpr string // "User{}", or "nil" for a pointer + isPointer bool + + // The shape check, or none when the type has an unexported field in + // another package and so cannot convert. + shapeName string + shapeSource string // "User{}", "crm.Contact{}" + shapeFields []shapeField + + // Generated names. + encName string // "EncryptedUser" + declVar string // "declaration" + codecVar string // "codec" + fieldsVar string // "Fields" + fieldTypePrefix string // "" or the -name + encryptFn string + decryptFn string + paramName string // the slice parameter of Encrypt + + notices []string + printsPlaintext bool + unexported []string + redact bool + redactRecv string + + decl Declaration + members []encMember // the fields of the encrypted type + fields []genField // every stored field, in declared order + opaque []genField // the fields of an opaque struct + models []genModel +} + +// shapeField is one field of the shape struct, which mirrors the plaintext +// struct field for field. +type shapeField struct { + embedded bool + name string + typeExpr string +} + +// encMember is one member of the encrypted struct: a field, or an embedded +// struct carried through from the plaintext type. +type encMember struct { + embedded bool + name string + typeExpr string + tags string // the other libraries' tags, without stash +} + +// genField is one stored field with what the emitter writes for it. +type genField struct { + Field + typeExpr string // the field's Go type + via string // the embedded passthrough struct it is promoted from, or "" + tags string // the other libraries' tags, without stash + outputType string // the encrypted type's field type: "eql.TextEq" or "EncryptedContactEmail" + outputs []output // the outputs of a sealed field in separate columns + queryType string // "eql.TextEqQuery", or "" when the field has no query + fieldType string // "EmailField" + pathType string // the type with its package path, for model checks +} + +// output is one output of a sealed field in separate columns. +type output struct { + name string // "Ciphertext", "Equality" + typeExpr string // "encrypt.Ciphertext" + pathType string + index IndexName // "" for the ciphertext +} + +// genModel is one -model flag. +type genModel struct { + name string // "Rows" + typeExpr string // "ContactRow" + shapeName string // "rowsShape" + codecVar string // "rowsCodec" + encryptFn string + decryptFn string + paramName string + fields []modelField +} + +// modelField is one field of a model, bound to one output of the declaration. +type modelField struct { + name string // the model's field name + typeExpr string + source string // the expression on the encrypted value, or "" for a field left zero + // For the reverse direction. + field *genField + output *output +} + +// reader builds a genFile from a loaded package. +type reader struct { + pkg *packages.Package + req Request + eql []EQLType + file *genFile + imports *importSet +} + +func (r *reader) qualifier(p *types.Package) string { + if p == nil || p.Path() == r.pkg.PkgPath { + return "" + } + return r.imports.add(p.Path(), p.Name()) +} + +func (r *reader) typeExpr(t types.Type) string { + return types.TypeString(t, r.qualifier) +} + +func pathQualifier(p *types.Package) string { return p.Path() } + +func pathType(t types.Type) string { return types.TypeString(t, pathQualifier) } + +// readStruct reads the tagged struct and, with -for, the type it declares +// for, and builds the file. +func (r *reader) read() (*genFile, error) { + pkg, req := r.pkg, r.req + tagNamed, tagStruct, err := lookupStruct(pkg, req.Type) + if err != nil { + return nil, err + } + f := &genFile{pkgName: pkg.Name, imports: r.imports} + r.file = f + f.imports.add("context", "context") + f.imports.add("log/slog", "slog") + f.imports.add(encryptPath, "encrypt") + f.imports.add(gensupportPath, "gensupport") + + valueNamed, valueStruct := tagNamed, tagStruct + f.typeName = req.Type + if req.For != "" { + valueNamed, valueStruct, err = lookupQualified(pkg, req.For) + if err != nil { + return nil, err + } + f.typeName = req.For + } + baseName := valueNamed.Obj().Name() + f.typeExpr = r.typeExpr(valueNamed) + f.zeroExpr = f.typeExpr + "{}" + f.encName = "Encrypted" + baseName + f.shapeName = lowerFirst(baseName) + "Shape" + f.shapeSource = f.typeExpr + "{}" + if req.Name != "" { + f.encryptFn, f.decryptFn, f.fieldsVar = "Encrypt"+req.Name, "Decrypt"+req.Name, req.Name+"Fields" + f.declVar, f.codecVar = lowerFirst(req.Name)+"Declaration", lowerFirst(req.Name)+"Codec" + f.fieldTypePrefix = req.Name + f.paramName = "values" + } else { + f.encryptFn, f.decryptFn, f.fieldsVar = "Encrypt", "Decrypt", "Fields" + f.declVar, f.codecVar = "declaration", "codec" + f.paramName = pkg.Name + } + if req.Redact { + if req.For != "" { + return nil, fmt.Errorf("stashgen: -redact cannot add print methods to %s, a type in another package", req.For) + } + f.redact = true + f.redactRecv = strings.ToLower(req.Type[:1]) + for _, m := range []string{"String", "LogValue"} { + if hasMethod(types.NewPointer(tagNamed), m) { + return nil, fmt.Errorf("stashgen: -redact: %s already has a %s method", req.Type, m) + } + } + } + + // The context, and the fields. + collected, err := r.collectFields(tagNamed, tagStruct, req.Type, true) + if err != nil { + return nil, err + } + if collected.context == "" { + return nil, fieldErr(req.Type, "", "no `_ struct{}` field with `stash:\"context=...\"` declares the context") + } + f.decl = Declaration{Type: f.typeName, Context: collected.context, Opaque: collected.opaque} + f.unexported = collected.unexported + + if req.For != "" { + if err := r.matchFor(collected, valueNamed, valueStruct); err != nil { + return nil, err + } + } + + // The shape check. A type from another package with an unexported field + // cannot convert, so the file reads each field by name instead, and the + // functions take a pointer. + switch { + case req.For == "": + f.shapeFields = shapeOf(valueStruct, r.typeExpr) + case hasUnexported(valueStruct): + f.shapeName = "" + f.isPointer = true + f.typeExpr = "*" + f.typeExpr + f.zeroExpr = "nil" + f.notices = append(f.notices, fmt.Sprintf("%s has unexported fields, so Go cannot convert it to a copy of its fields. This file reads each field by name: the compiler finds a removed or retyped field, and CI finds an added one.", f.typeName)) + default: + f.shapeFields = shapeOf(valueStruct, r.typeExpr) + } + + if err := r.buildFields(collected); err != nil { + return nil, err + } + if len(f.fields) == 0 && len(f.opaque) == 0 { + return nil, fieldErr(req.Type, "", "stores no field: every field is omitted") + } + + // Printing. + sealedCount := 0 + for _, g := range f.fields { + if g.Sealed() { + sealedCount++ + } + } + if f.decl.Opaque { + sealedCount = 1 + } + if sealedCount > 0 && !f.redact && (!hasMethod(valueNamed, "String") || !hasMethod(valueNamed, "LogValue")) { + f.printsPlaintext = true + if req.For != "" { + f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear, and stashgen cannot add print methods to a type from another package.", f.typeName)) + } else { + f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", f.typeName)) + } + } + if len(f.unexported) > 0 { + f.notices = append(f.notices, "Not encrypted and not stored: the unexported "+fieldList(f.unexported)+". Tag "+itOrEach(f.unexported)+" `stash:\"-\"` to confirm that.") + } + + if err := r.checkDirectives(); err != nil { + return nil, err + } + for _, m := range req.Models { + if err := r.readModel(m); err != nil { + return nil, err + } + } + return f, nil +} + +// collectedField is one field of the tagged struct after its tag is read. +type collectedField struct { + tag tag + goName string + typ types.Type + tags string // the struct tag without the stash key + via string // the embedded struct it is promoted from + viaType types.Type + exported bool +} + +type collected struct { + context string + opaque bool + fields []collectedField + unexported []string +} + +// collectFields reads the tags of a struct. An embedded struct of the same +// package adds its tagged fields; an embedded struct with a tag of its own +// promotes every field under that tag. +func (r *reader) collectFields(named *types.Named, st *types.Struct, typeName string, top bool) (*collected, error) { + c := &collected{} + // The context first, so the loop knows whether the struct is opaque + // wherever the `_` field sits. + for i := range st.NumFields() { + fld := st.Field(i) + t, err := parseTag(st.Tag(i)) + if fld.Name() != "_" { + if err == nil && t.Context != "" { + return nil, fieldErr(typeName, fld.Name(), "context= goes on a `_ struct{}` field") + } + continue + } + if err != nil || t.Context == "" { + return nil, fieldErr(typeName, "_", "a `_` field carries the context tag, `stash:\"context=...\"`") + } + if !top { + return nil, fieldErr(typeName, "_", "an embedded struct declares no context; the outer struct's context applies") + } + if c.context != "" { + return nil, fieldErr(typeName, "_", "the context is declared twice") + } + c.context, c.opaque = t.Context, t.Opaque + } + for i := range st.NumFields() { + fld := st.Field(i) + if fld.Name() == "_" { + continue + } + raw := st.Tag(i) + t, err := parseTag(raw) + if err != nil && err != errNoTag { + return nil, fieldErr(typeName, fld.Name(), "%v", err) + } + hasTag := err == nil + if hasInvalidType(fld.Type()) { + return nil, fieldErr(typeName, fld.Name(), "its type did not resolve: %s", packageErrors(r.pkg)) + } + switch { + case c.opaque: + // An opaque struct seals as one value: its fields carry no tags + // but `-`, and every exported field is part of the value. + switch { + case hasTag && t.Omit: + c.fields = append(c.fields, collectedField{tag: t, goName: fld.Name(), typ: fld.Type(), exported: fld.Exported()}) + case hasTag: + return nil, fieldErr(typeName, fld.Name(), "an opaque struct seals as one value, so its fields carry no tags") + case !fld.Exported(): + c.unexported = append(c.unexported, fld.Name()) + default: + c.fields = append(c.fields, collectedField{tag: tag{Name: snakeCase(fld.Name()), Verb: VerbEncrypt}, goName: fld.Name(), typ: fld.Type(), exported: true}) + } + case fld.Embedded(): + embNamed, embStruct, ok := embeddedStruct(fld.Type()) + if !ok { + return nil, fieldErr(typeName, fld.Name(), "an embedded field must be a struct, not a pointer or an interface") + } + switch { + case hasTag && t.Omit: + c.fields = append(c.fields, collectedField{tag: t, goName: fld.Name(), typ: fld.Type(), exported: fld.Exported()}) + case hasTag && t.Verb == VerbPassthrough && t.Name == "": + for _, pf := range promotedFields(embStruct) { + c.fields = append(c.fields, collectedField{ + tag: tag{Name: snakeCase(pf.Name()), Verb: VerbPassthrough}, + goName: pf.Name(), + typ: pf.Type(), + via: fld.Name(), + viaType: fld.Type(), + exported: true, + }) + } + case hasTag: + return nil, fieldErr(typeName, fld.Name(), "an embedded struct takes `stash:\",passthrough\"` or `stash:\"-\"`, not %q", strings.TrimPrefix(raw, "stash:")) + case embNamed != nil && embNamed.Obj().Pkg() != nil && embNamed.Obj().Pkg().Path() == r.pkg.PkgPath: + inner, err := r.collectFields(embNamed, embStruct, embNamed.Obj().Name(), false) + if err != nil { + return nil, err + } + c.fields = append(c.fields, inner.fields...) + c.unexported = append(c.unexported, inner.unexported...) + default: + return nil, fieldErr(typeName, fld.Name(), "an embedded struct from another package cannot carry tags; tag the field `stash:\",passthrough\"` to store %s, or `stash:\"-\"` to leave them out", fieldNames(embStruct)) + } + case !hasTag && !fld.Exported(): + c.unexported = append(c.unexported, fld.Name()) + case !hasTag: + return nil, fieldErr(typeName, fld.Name(), "no stash tag; every exported field says what happens to it") + default: + c.fields = append(c.fields, collectedField{tag: t, goName: fld.Name(), typ: fld.Type(), tags: stripTagKey(raw, tagKey), exported: fld.Exported()}) + } + } + return c, nil +} + +func embeddedStruct(t types.Type) (*types.Named, *types.Struct, bool) { + named, _ := t.(*types.Named) + st, ok := t.Underlying().(*types.Struct) + return named, st, ok +} + +// promotedFields lists the exported fields of a struct and of the structs it +// embeds, as Go promotes them. +func promotedFields(st *types.Struct) []*types.Var { + var out []*types.Var + for i := range st.NumFields() { + f := st.Field(i) + if f.Embedded() { + if _, inner, ok := embeddedStruct(f.Type()); ok { + out = append(out, promotedFields(inner)...) + continue + } + } + if f.Exported() { + out = append(out, f) + } + } + return out +} + +func fieldNames(st *types.Struct) string { + var names []string + for _, f := range promotedFields(st) { + names = append(names, f.Name()) + } + return strings.Join(names, ", ") +} + +func hasUnexported(st *types.Struct) bool { + for i := range st.NumFields() { + if !st.Field(i).Exported() { + return true + } + } + return false +} + +func hasMethod(t types.Type, name string) bool { + ms := types.NewMethodSet(t) + for i := range ms.Len() { + if ms.At(i).Obj().Name() == name { + return true + } + } + return false +} + +func shapeOf(st *types.Struct, typeExpr func(types.Type) string) []shapeField { + out := make([]shapeField, 0, st.NumFields()) + for i := range st.NumFields() { + f := st.Field(i) + out = append(out, shapeField{embedded: f.Embedded(), name: f.Name(), typeExpr: typeExpr(f.Type())}) + } + return out +} + +// matchFor checks the tagged struct against the type it declares for: each +// field names a field of the same name and type, and every exported field of +// the type is named. +func (r *reader) matchFor(c *collected, valueNamed *types.Named, valueStruct *types.Struct) error { + byName := map[string]*types.Var{} + for i := range valueStruct.NumFields() { + byName[valueStruct.Field(i).Name()] = valueStruct.Field(i) + } + named := map[string]bool{} + for _, cf := range c.fields { + if cf.via != "" { + return fieldErr(r.req.Type, cf.via, "a struct that declares for %s embeds nothing; name each field", r.req.For) + } + vf, ok := byName[cf.goName] + if !ok { + return fieldErr(r.req.Type, cf.goName, "%s has no field %s", r.req.For, cf.goName) + } + if !types.Identical(vf.Type(), cf.typ) { + return fieldErr(r.req.Type, cf.goName, "has type %s, and %s.%s has type %s", r.typeExpr(cf.typ), r.req.For, cf.goName, r.typeExpr(vf.Type())) + } + named[cf.goName] = true + } + var missing []string + for i := range valueStruct.NumFields() { + vf := valueStruct.Field(i) + if vf.Exported() && !named[vf.Name()] { + missing = append(missing, vf.Name()) + } + } + if len(missing) > 0 { + return fieldErr(r.req.For, missing[0], "not named by %s; every exported field of %s needs a tag there (missing: %s)", r.req.Type, r.req.For, strings.Join(missing, ", ")) + } + return nil +} + +// buildFields turns the collected fields into the declaration, the members +// of the encrypted type and the emitter's fields. +func (r *reader) buildFields(c *collected) error { + f := r.file + typeName := r.req.Type + seen := map[string]string{} + embedded := map[string]bool{} + + if c.opaque { + for _, cf := range c.fields { + if cf.tag.Omit { + f.decl.Fields = append(f.decl.Fields, Field{Name: snakeCase(cf.goName), GoName: cf.goName, GoType: r.goType(cf.typ), Verb: VerbOmit}) + continue + } + g := genField{Field: Field{Name: cf.tag.Name, GoName: cf.goName, GoType: r.goType(cf.typ), Verb: VerbEncrypt}, typeExpr: r.typeExpr(cf.typ)} + if prev, dup := seen[g.Name]; dup { + return fieldErr(typeName, cf.goName, "two fields write the name %q: %s and %s", g.Name, prev, cf.goName) + } + seen[g.Name] = cf.goName + f.opaque = append(f.opaque, g) + f.decl.Fields = append(f.decl.Fields, g.Field) + } + f.members = []encMember{{name: "Sealed", typeExpr: "encrypt.Ciphertext"}} + return nil + } + + for _, cf := range c.fields { + t := cf.tag + name := t.Name + if t.Omit { + name = snakeCase(cf.goName) + } + if prev, dup := seen[name]; dup { + return fieldErr(typeName, cf.goName, "two fields write the name %q: %s and %s", name, prev, cf.goName) + } + seen[name] = cf.goName + field := Field{Name: name, GoName: cf.goName, GoType: r.goType(cf.typ), Verb: t.Verb, Indexes: t.Indexes, EQLType: t.EQLType} + if t.Omit { + field.Verb = VerbOmit + } + f.decl.Fields = append(f.decl.Fields, field) + if t.Omit { + continue + } + g := genField{Field: field, typeExpr: r.typeExpr(cf.typ), via: cf.via, tags: cf.tags, pathType: pathType(cf.typ)} + switch field.Verb { + case VerbPassthrough: + g.outputType = g.typeExpr + g.outputs = []output{{name: "Value", typeExpr: g.typeExpr, pathType: g.pathType}} + case VerbEncryptInto: + var eqlType *EQLType + for i := range r.eql { + if r.eql[i].Name == field.EQLType { + eqlType = &r.eql[i] + } + } + if eqlType == nil { + return fieldErr(typeName, cf.goName, "the engine cannot produce the EQL type %s", field.EQLType) + } + f.imports.add(eqlPath, "eql") + g.outputType = "eql." + eqlType.Name + g.outputs = []output{{name: "EQL", typeExpr: g.outputType, pathType: eqlPath + "." + eqlType.Name}} + if eqlType.Query != "" { + g.queryType = "eql." + eqlType.Query + } + default: + g.outputType = f.encName + cf.goName + if g.HasCiphertext() { + g.outputs = append(g.outputs, output{name: "Ciphertext", typeExpr: "encrypt.Ciphertext", pathType: encryptPath + ".Ciphertext"}) + } + for _, idx := range field.Indexes { + g.outputs = append(g.outputs, output{name: idx.Name.GoName(), typeExpr: "encrypt." + idx.Name.GoName() + "Term", pathType: encryptPath + "." + idx.Name.GoName() + "Term", index: idx.Name}) + } + } + if g.Sealed() { + g.fieldType = f.fieldTypePrefix + cf.goName + "Field" + } + f.fields = append(f.fields, g) + if cf.via != "" { + if !embedded[cf.via] { + embedded[cf.via] = true + f.members = append(f.members, encMember{embedded: true, name: cf.via, typeExpr: r.typeExpr(cf.viaType), tags: r.embeddedTags(cf.via)}) + } + continue + } + f.members = append(f.members, encMember{name: cf.goName, typeExpr: g.outputType, tags: cf.tags}) + } + return nil +} + +func (r *reader) embeddedTags(fieldName string) string { + _, st, err := lookupStruct(r.pkg, r.req.Type) + if err != nil { + return "" + } + for i := range st.NumFields() { + if st.Field(i).Name() == fieldName { + return stripTagKey(st.Tag(i), tagKey) + } + } + return "" +} + +// goType classifies a Go type for the engine. +func (r *reader) goType(t types.Type) GoType { + return classify(t, r.typeExpr, map[types.Type]bool{}) +} + +func classify(t types.Type, typeExpr func(types.Type) string, seen map[types.Type]bool) GoType { + g := GoType{Name: typeExpr(t), Kind: KindOther} + if seen[t] { + return g + } + seen[t] = true + defer delete(seen, t) + switch u := t.Underlying().(type) { + case *types.Basic: + switch { + case u.Info()&types.IsString != 0: + g.Kind = KindString + case u.Info()&types.IsBoolean != 0: + g.Kind = KindBool + case u.Info()&types.IsUnsigned != 0: + g.Kind = KindUint + case u.Info()&types.IsInteger != 0: + g.Kind = KindInt + case u.Info()&types.IsFloat != 0: + g.Kind = KindFloat + } + case *types.Slice: + if b, ok := u.Elem().Underlying().(*types.Basic); ok && b.Kind() == types.Byte { + g.Kind = KindBytes + } else { + elem := classify(u.Elem(), typeExpr, seen) + g.Kind, g.Elem = KindSlice, &elem + } + case *types.Array: + elem := classify(u.Elem(), typeExpr, seen) + g.Kind, g.Elem = KindSlice, &elem + case *types.Map: + elem := classify(u.Elem(), typeExpr, seen) + g.Kind, g.Elem = KindMap, &elem + case *types.Pointer: + elem := classify(u.Elem(), typeExpr, seen) + g.Kind, g.Elem = KindPointer, &elem + case *types.Struct: + if hasUnexported(u) { + return g + } + g.Kind = KindStruct + for i := range u.NumFields() { + g.Fields = append(g.Fields, classify(u.Field(i).Type(), typeExpr, seen)) + } + } + return g +} + +// checkDirectives stops when another go:generate line in the package would +// write the same Encrypt function. +func (r *reader) checkDirectives() error { + for _, file := range r.pkg.Syntax { + for _, cg := range file.Comments { + for _, c := range cg.List { + if !strings.HasPrefix(c.Text, "//go:generate") || !strings.Contains(c.Text, "stashgen") { + continue + } + typ, name := directiveFlags(c.Text) + if typ == "" || typ == r.req.Type { + continue + } + if "Encrypt"+name == r.file.encryptFn { + return fmt.Errorf("stashgen: %s and %s would both write %s in package %s; give one of them -name", r.req.Type, typ, r.file.encryptFn, r.pkg.Name) + } + } + } + } + return nil +} + +// directiveFlags reads -type and -name from a go:generate line. +func directiveFlags(line string) (typ, name string) { + args := strings.Fields(line) + for i := 0; i < len(args); i++ { + a := args[i] + next := func() string { + if eq := strings.IndexByte(a, '='); eq >= 0 { + return a[eq+1:] + } + if i+1 < len(args) { + i++ + return args[i] + } + return "" + } + switch { + case a == "-type" || strings.HasPrefix(a, "-type="), a == "--type" || strings.HasPrefix(a, "--type="): + typ = next() + case a == "-name" || strings.HasPrefix(a, "-name="), a == "--name" || strings.HasPrefix(a, "--name="): + name = next() + } + } + return typ, name +} + +// stripTagKey removes one key from a struct tag and keeps the others as +// written, for copying other libraries' tags to the generated type. +func stripTagKey(structTag, key string) string { + var kept []string + rest := structTag + for rest != "" { + rest = strings.TrimLeft(rest, " ") + if rest == "" { + break + } + colon := strings.IndexByte(rest, ':') + if colon <= 0 || colon+1 >= len(rest) || rest[colon+1] != '"' { + break + } + name := rest[:colon] + end := colon + 2 + for end < len(rest) && rest[end] != '"' { + if rest[end] == '\\' { + end++ + } + end++ + } + if end >= len(rest) { + break + } + pair := rest[:end+1] + rest = rest[end+1:] + if name != key { + kept = append(kept, pair) + } + } + return strings.Join(kept, " ") +} + +func fieldList(fields []string) string { + quoted := make([]string, len(fields)) + for i, f := range fields { + quoted[i] = fmt.Sprintf("%q", f) + } + if len(quoted) == 1 { + return "field " + quoted[0] + } + return "fields " + strings.Join(quoted[:len(quoted)-1], ", ") + " and " + quoted[len(quoted)-1] +} + +func itOrEach(fields []string) string { + if len(fields) == 1 { + return "it" + } + return "each" +} diff --git a/languages/golang/stashgen/tag.go b/languages/golang/stashgen/tag.go new file mode 100644 index 000000000..c566a65e5 --- /dev/null +++ b/languages/golang/stashgen/tag.go @@ -0,0 +1,255 @@ +package stashgen + +import ( + "errors" + "fmt" + "reflect" + "strings" +) + +// The key of the struct tag the generator reads. +const tagKey = "stash" + +// tag is one parsed stash tag. +// +// stash:"-" Omit +// stash:"context=users" Context, on a `_ struct{}` field +// stash:"context=documents,opaque" Context and Opaque +// stash:"email,encrypt_into=TextEq" Name, VerbEncryptInto, EQLType +// stash:"notes,encrypt" Name, VerbEncrypt +// stash:"email,encrypt,index=equality;match" Name, VerbEncryptIndex, Indexes +// stash:"attrs,index=json" Name, VerbIndex, Indexes +// stash:"id,passthrough" Name, VerbPassthrough +// stash:",passthrough" VerbPassthrough with no Name, on an embedded struct +type tag struct { + Omit bool + Context string + Opaque bool + Name string + Verb Verb + Indexes []Index + EQLType string +} + +// errNoTag reports a field with no stash tag at all. +var errNoTag = errors.New("no stash tag") + +// parseTag reads the stash key of a struct tag. It returns errNoTag when the +// key is absent, and a descriptive error when the value does not parse. +func parseTag(structTag string) (tag, error) { + value, ok := lookupTag(structTag, tagKey) + if !ok { + return tag{}, errNoTag + } + return parseTagValue(value) +} + +// lookupTag reads one key of a struct tag. The generator is a tool, so +// reflect's tag parser is fine here; generated code uses no reflection. +func lookupTag(structTag, key string) (string, bool) { + return reflect.StructTag(structTag).Lookup(key) +} + +func parseTagValue(value string) (tag, error) { + if value == "-" { + return tag{Omit: true}, nil + } + items, err := splitOutsideParens(value, ',') + if err != nil { + return tag{}, err + } + if strings.HasPrefix(items[0], "context=") { + return parseContextTag(items) + } + t := tag{Name: items[0]} + if strings.ContainsAny(t.Name, "=();") { + return tag{}, fmt.Errorf("tag %q: the first part is the field's name, and %q is not a name", value, t.Name) + } + var sawEncrypt, sawPassthrough, sawIndex, sawInto bool + for _, item := range items[1:] { + switch { + case item == "encrypt": + if sawEncrypt { + return tag{}, fmt.Errorf("tag %q: encrypt is given twice", value) + } + sawEncrypt = true + case item == "passthrough": + if sawPassthrough { + return tag{}, fmt.Errorf("tag %q: passthrough is given twice", value) + } + sawPassthrough = true + case strings.HasPrefix(item, "encrypt_into="): + if sawInto { + return tag{}, fmt.Errorf("tag %q: encrypt_into is given twice", value) + } + sawInto = true + t.EQLType = strings.TrimPrefix(item, "encrypt_into=") + if !isIdent(t.EQLType) { + return tag{}, fmt.Errorf("tag %q: encrypt_into names an EQL type, and %q is not a type name", value, t.EQLType) + } + case strings.HasPrefix(item, "index="): + if sawIndex { + return tag{}, fmt.Errorf("tag %q: index is given twice", value) + } + sawIndex = true + t.Indexes, err = parseIndexes(strings.TrimPrefix(item, "index=")) + if err != nil { + return tag{}, fmt.Errorf("tag %q: %w", value, err) + } + case item == "opaque" || strings.HasPrefix(item, "context="): + return tag{}, fmt.Errorf("tag %q: %s goes on the `_ struct{}` field, as `stash:\"context=...\"`", value, item) + case item == "": + return tag{}, fmt.Errorf("tag %q: an empty part", value) + default: + return tag{}, fmt.Errorf("tag %q: unknown part %q; the parts are encrypt, encrypt_into=, index=, passthrough and -", value, item) + } + } + switch { + case sawPassthrough && (sawEncrypt || sawInto): + return tag{}, fmt.Errorf("tag %q: a field is passthrough or encrypted, not both", value) + case sawPassthrough && sawIndex: + return tag{}, fmt.Errorf("tag %q: a passthrough field has no index", value) + case sawInto && sawEncrypt: + return tag{}, fmt.Errorf("tag %q: encrypt_into already seals the field; drop encrypt", value) + case sawInto && sawIndex: + return tag{}, fmt.Errorf("tag %q: encrypt_into gets its indexes from the EQL type; drop index=", value) + case sawPassthrough: + t.Verb = VerbPassthrough + case sawInto: + t.Verb = VerbEncryptInto + case sawEncrypt && sawIndex: + t.Verb = VerbEncryptIndex + case sawEncrypt: + t.Verb = VerbEncrypt + case sawIndex: + t.Verb = VerbIndex + default: + return tag{}, fmt.Errorf("tag %q: names no verb; add encrypt, encrypt_into=, index= or passthrough", value) + } + if t.Name == "" && t.Verb != VerbPassthrough { + return tag{}, fmt.Errorf("tag %q: a tag with no name goes on an embedded struct, and takes only passthrough", value) + } + return t, nil +} + +func parseContextTag(items []string) (tag, error) { + t := tag{Context: strings.TrimPrefix(items[0], "context=")} + if t.Context == "" { + return tag{}, fmt.Errorf("tag %q: context= names no context", strings.Join(items, ",")) + } + for _, item := range items[1:] { + switch item { + case "opaque": + if t.Opaque { + return tag{}, fmt.Errorf("tag %q: opaque is given twice", strings.Join(items, ",")) + } + t.Opaque = true + default: + return tag{}, fmt.Errorf("tag %q: a context tag takes only opaque, not %q", strings.Join(items, ","), item) + } + } + return t, nil +} + +// parseIndexes reads `equality;match(k=3,m=2048)`. +func parseIndexes(list string) ([]Index, error) { + parts, err := splitOutsideParens(list, ';') + if err != nil { + return nil, err + } + var indexes []Index + seen := map[IndexName]bool{} + for _, part := range parts { + idx, err := parseIndex(part) + if err != nil { + return nil, err + } + if seen[idx.Name] { + return nil, fmt.Errorf("index %s is given twice", idx.Name) + } + seen[idx.Name] = true + indexes = append(indexes, idx) + } + return indexes, nil +} + +func parseIndex(part string) (Index, error) { + if part == "" { + return Index{}, errors.New("index= names no index") + } + name, rest := part, "" + if i := strings.IndexByte(part, '('); i >= 0 { + if !strings.HasSuffix(part, ")") { + return Index{}, fmt.Errorf("index %q: options open with ( and do not close", part) + } + name, rest = part[:i], part[i+1:len(part)-1] + } + idx := Index{Name: IndexName(name)} + if !knownIndex(idx.Name) { + return Index{}, fmt.Errorf("unknown index %q; the indexes are equality, match, ore, ope and json", name) + } + if rest == "" { + if strings.HasSuffix(part, "()") { + return Index{}, fmt.Errorf("index %q: empty options; drop the parentheses", part) + } + return idx, nil + } + for _, opt := range strings.Split(rest, ",") { + key, value, _ := strings.Cut(opt, "=") + if key == "" || !isIdent(key) { + return Index{}, fmt.Errorf("index %q: option %q is not key or key=value", part, opt) + } + idx.Options = append(idx.Options, Option{Key: key, Value: value}) + } + return idx, nil +} + +func knownIndex(name IndexName) bool { + for _, n := range indexNames { + if n == name { + return true + } + } + return false +} + +// splitOutsideParens splits s on sep, except inside parentheses. +func splitOutsideParens(s string, sep byte) ([]string, error) { + var parts []string + depth, start := 0, 0 + for i := 0; i < len(s); i++ { + switch s[i] { + case '(': + depth++ + case ')': + depth-- + if depth < 0 { + return nil, fmt.Errorf("tag %q: a ) with no (", s) + } + case sep: + if depth == 0 { + parts = append(parts, s[start:i]) + start = i + 1 + } + } + } + if depth != 0 { + return nil, fmt.Errorf("tag %q: a ( with no )", s) + } + return append(parts, s[start:]), nil +} + +func isIdent(s string) bool { + if s == "" { + return false + } + for i, r := range s { + switch { + case r == '_', r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z': + case r >= '0' && r <= '9' && i > 0: + default: + return false + } + } + return true +} diff --git a/languages/golang/stashgen/tag_test.go b/languages/golang/stashgen/tag_test.go new file mode 100644 index 000000000..ebf2738fa --- /dev/null +++ b/languages/golang/stashgen/tag_test.go @@ -0,0 +1,147 @@ +package stashgen + +import ( + "errors" + "strings" + "testing" +) + +func TestParseTagEveryFormInTheTable(t *testing.T) { + cases := []struct { + in string + want tag + }{ + {`stash:"context=users"`, tag{Context: "users"}}, + {`stash:"email,encrypt_into=TextEq"`, tag{Name: "email", Verb: VerbEncryptInto, EQLType: "TextEq"}}, + {`stash:"notes,encrypt"`, tag{Name: "notes", Verb: VerbEncrypt}}, + {`stash:"email,encrypt,index=equality;match"`, tag{Name: "email", Verb: VerbEncryptIndex, + Indexes: []Index{{Name: IndexEquality}, {Name: IndexMatch}}}}, + {`stash:"attrs,index=json"`, tag{Name: "attrs", Verb: VerbIndex, Indexes: []Index{{Name: IndexJSON}}}}, + {`stash:"id,passthrough"`, tag{Name: "id", Verb: VerbPassthrough}}, + {`stash:"-"`, tag{Omit: true}}, + {`stash:"context=documents,opaque"`, tag{Context: "documents", Opaque: true}}, + {`stash:",passthrough"`, tag{Verb: VerbPassthrough}}, + // Other libraries' tags around the stash tag do not disturb it. + {`db:"id" stash:"id,passthrough" gorm:"primaryKey"`, tag{Name: "id", Verb: VerbPassthrough}}, + // The order of the parts after the name does not matter. + {`stash:"email,index=equality,encrypt"`, tag{Name: "email", Verb: VerbEncryptIndex, Indexes: []Index{{Name: IndexEquality}}}}, + // Options in parentheses, separated by commas inside them. + {`stash:"email,encrypt,index=equality;match(k=3,m=2048);ore"`, tag{Name: "email", Verb: VerbEncryptIndex, + Indexes: []Index{{Name: IndexEquality}, {Name: IndexMatch, Options: []Option{{"k", "3"}, {"m", "2048"}}}, {Name: IndexOre}}}}, + {`stash:"attrs,index=json(compat)"`, tag{Name: "attrs", Verb: VerbIndex, + Indexes: []Index{{Name: IndexJSON, Options: []Option{{Key: "compat"}}}}}}, + {`stash:"body,context=x"`, tag{}}, // error, see below + } + for _, c := range cases { + got, err := parseTag(c.in) + if c.in == `stash:"body,context=x"` { + if err == nil { + t.Errorf("%s: want error", c.in) + } + continue + } + if err != nil { + t.Errorf("%s: %v", c.in, err) + continue + } + if !tagEqual(got, c.want) { + t.Errorf("%s:\n got %+v\n want %+v", c.in, got, c.want) + } + } +} + +func tagEqual(a, b tag) bool { + if a.Omit != b.Omit || a.Context != b.Context || a.Opaque != b.Opaque || a.Name != b.Name || a.Verb != b.Verb || a.EQLType != b.EQLType { + return false + } + if len(a.Indexes) != len(b.Indexes) { + return false + } + for i := range a.Indexes { + if a.Indexes[i].String() != b.Indexes[i].String() { + return false + } + } + return true +} + +func TestParseTagAbsent(t *testing.T) { + _, err := parseTag(`json:"email"`) + if !errors.Is(err, errNoTag) { + t.Fatalf("err = %v, want errNoTag", err) + } + _, err = parseTag(``) + if !errors.Is(err, errNoTag) { + t.Fatalf("err = %v, want errNoTag", err) + } +} + +func TestParseTagRefusals(t *testing.T) { + cases := map[string]string{ + `stash:""`: "names no verb", + `stash:"email"`: "names no verb", + `stash:"email,encrypt,passthrough"`: "not both", + `stash:"id,passthrough,index=equality"`: "passthrough field has no index", + `stash:"email,encrypt,encrypt"`: "given twice", + `stash:"email,encrypt,encrypt_into=TextEq"`: "drop encrypt", + `stash:"email,encrypt_into=TextEq,index=equality"`: "drop index=", + `stash:"email,encrypt_into="`: "not a type name", + `stash:"email,encrypt_into=eql.TextEq"`: "not a type name", + `stash:"email,shred"`: `unknown part "shred"`, + `stash:"email,encrypt,index=fuzzy"`: `unknown index "fuzzy"`, + `stash:"email,encrypt,index="`: "names no index", + `stash:"email,encrypt,index=equality;equality"`: "given twice", + `stash:"email,encrypt,index=match("`: "( with no )", + `stash:"email,encrypt,index=match)"`: ") with no (", + `stash:"email,encrypt,index=match()"`: "empty options", + `stash:"email,encrypt,index=match(=3)"`: "not key or key=value", + `stash:"email,encrypt,,"`: "an empty part", + `stash:"context="`: "names no context", + `stash:"context=users,encrypt"`: "takes only opaque", + `stash:"context=users,opaque,opaque"`: "given twice", + `stash:"email,opaque"`: "goes on the `_ struct{}` field", + `stash:"email,context=users"`: "goes on the `_ struct{}` field", + `stash:",encrypt"`: "takes only passthrough", + `stash:"a=b,encrypt"`: "is not a name", + } + for in, want := range cases { + _, err := parseTag(in) + if err == nil { + t.Errorf("%s: want an error containing %q, got none", in, want) + continue + } + if !strings.Contains(err.Error(), want) { + t.Errorf("%s: error %q does not contain %q", in, err, want) + } + } +} + +func TestSnakeCase(t *testing.T) { + cases := map[string]string{ + "ID": "id", "CreatedAt": "created_at", "DeletedAt": "deleted_at", "MedicareNo": "medicare_no", + "HTTPServer": "http_server", "UserID": "user_id", "Email": "email", "cache": "cache", + "PhoneNumber": "phone_number", "A1B": "a1_b", "Internal": "internal", + } + for in, want := range cases { + if got := snakeCase(in); got != want { + t.Errorf("snakeCase(%q) = %q, want %q", in, got, want) + } + } +} + +func TestLowerFirst(t *testing.T) { + cases := map[string]string{"User": "user", "ContactRow": "contactRow", "ID": "id", "HTTPServer": "httpServer", "Rows": "rows", "contactStash": "contactStash"} + for in, want := range cases { + if got := lowerFirst(in); got != want { + t.Errorf("lowerFirst(%q) = %q, want %q", in, got, want) + } + } +} + +func TestWrapComment(t *testing.T) { + got := wrapComment("User prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", 80) + want := "// User prints its sealed fields in the clear: it has no String or LogValue\n// method. Write them, or run stashgen with -redact.\n" + if got != want { + t.Fatalf("wrapComment:\n%s\nwant:\n%s", got, want) + } +} diff --git a/languages/golang/stashgen/testdata/cases/accounts/account.go b/languages/golang/stashgen/testdata/cases/accounts/account.go new file mode 100644 index 000000000..748319a79 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/accounts/account.go @@ -0,0 +1,39 @@ +// Package accounts shows an embedded struct, another library's tags, an +// unexported field, and generated print methods. +package accounts + +import ( + "context" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "gorm.io/gorm" +) + +//go:generate go tool stashgen -type Account -redact + +type Account struct { + _ struct{} `stash:"context=accounts"` + + // One tag decides for every field of gorm.Model, which cannot carry tags. + gorm.Model `stash:",passthrough"` + + // stashgen copies the gorm and json tags onto EncryptedAccount. + Email string `stash:"email,encrypt_into=TextEq" gorm:"uniqueIndex" json:"email"` + + // An unexported field with no stash tag is ignored, and stashgen and the + // running program both warn about it. + cache string + + // The tag acknowledges the skip, so nothing warns. + token string `stash:"-"` +} + +func (EncryptedAccount) TableName() string { return "accounts" } + +func Create(ctx context.Context, db *gorm.DB, cipher *encrypt.Cipher, accounts []Account) error { + encrypted, err := Encrypt(ctx, cipher, accounts) + if err != nil { + return err + } + return db.WithContext(ctx).Create(&encrypted).Error +} diff --git a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden new file mode 100644 index 000000000..3a1c3abd4 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden @@ -0,0 +1,154 @@ +// Code generated by stashgen. DO NOT EDIT. + +package accounts + +import ( + "context" + "log/slog" + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" + "gorm.io/gorm" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Not encrypted and not stored: the unexported field "cache". Tag it +// `stash:"-"` to confirm that. + +type EncryptedAccount struct { + gorm.Model + Email eql.TextEq `gorm:"uniqueIndex" json:"email"` +} + +func (e EncryptedAccount) String() string { + return gensupport.Redacted("EncryptedAccount", map[string]any{"Model": e.Model}, "Email") +} + +func (e EncryptedAccount) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"Model": e.Model}, "Email") +} + +// Written because of -redact: Account no longer prints its sealed fields. +func (a Account) String() string { + return gensupport.Redacted("Account", map[string]any{"Model": a.Model}, "Email") +} + +func (a Account) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"Model": a.Model}, "Email") +} + +// Stops compiling when Account gains, loses, reorders or retypes a field. +var _ = accountShape(Account{}) + +type accountShape struct { + _ struct{} + gorm.Model + Email string + cache string + token string +} + +var declaration = gensupport.Declare("accounts"). + Passthrough("id"). + Passthrough("created_at"). + Passthrough("updated_at"). + Passthrough("deleted_at"). + EncryptInto("email", "TextEq"). + Omit("token") + +var codec = gensupport.New(gensupport.Generated[Account, EncryptedAccount]{ + TypeName: "Account", + Declaration: declaration, + Unexported: []string{"cache"}, + Source: func(v Account) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "created_at": v.CreatedAt, + "updated_at": v.UpdatedAt, + "deleted_at": v.DeletedAt, + "email": v.Email, + } + }, + Seal: func(rec gensupport.Record) (EncryptedAccount, error) { + var e EncryptedAccount + var err error + if e.ID, err = gensupport.Passthrough[uint](rec, "id"); err != nil { + return EncryptedAccount{}, err + } + if e.CreatedAt, err = gensupport.Passthrough[time.Time](rec, "created_at"); err != nil { + return EncryptedAccount{}, err + } + if e.UpdatedAt, err = gensupport.Passthrough[time.Time](rec, "updated_at"); err != nil { + return EncryptedAccount{}, err + } + if e.DeletedAt, err = gensupport.Passthrough[gorm.DeletedAt](rec, "deleted_at"); err != nil { + return EncryptedAccount{}, err + } + e.Email = eql.TextEq(rec["email"].EQL) + return e, nil + }, + Open: func(e EncryptedAccount) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "created_at": {Value: e.CreatedAt}, + "updated_at": {Value: e.UpdatedAt}, + "deleted_at": {Value: e.DeletedAt}, + "email": {EQL: e.Email}, + } + }, + Value: func(e EncryptedAccount, vals gensupport.Values) (Account, error) { + var v Account + var err error + if v.ID, err = gensupport.Get[uint](vals, "id"); err != nil { + return Account{}, err + } + if v.CreatedAt, err = gensupport.Get[time.Time](vals, "created_at"); err != nil { + return Account{}, err + } + if v.UpdatedAt, err = gensupport.Get[time.Time](vals, "updated_at"); err != nil { + return Account{}, err + } + if v.DeletedAt, err = gensupport.Get[gorm.DeletedAt](vals, "deleted_at"); err != nil { + return Account{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return Account{}, err + } + return v, nil + }, +}) + +// Encrypt seals each Account in one ZeroKMS request. The result has one element +// for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, accounts []Account) ([]EncryptedAccount, error) { + return codec.Encrypt(ctx, cipher, accounts) +} + +// Decrypt opens each EncryptedAccount in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedAccount) ([]Account, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Email EmailField +}{ + Email: EmailField{gensupport.NewField[string](declaration, "email")}, +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f EmailField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} diff --git a/languages/golang/stashgen/testdata/cases/accounts/go.mod b/languages/golang/stashgen/testdata/cases/accounts/go.mod new file mode 100644 index 000000000..455ad04ab --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/accounts/go.mod @@ -0,0 +1,12 @@ +module example.com/app + +go 1.26 + +require ( + github.com/cipherstash/stack/languages/golang v0.0.0 + gorm.io/gorm v0.0.0 +) + +replace github.com/cipherstash/stack/languages/golang => ../../stubsdk + +replace gorm.io/gorm => ../../stubgorm diff --git a/languages/golang/stashgen/testdata/cases/contacts/contacts.go b/languages/golang/stashgen/testdata/cases/contacts/contacts.go new file mode 100644 index 000000000..0bf3717d3 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/contacts/contacts.go @@ -0,0 +1,76 @@ +// Package contacts encrypts crm.Contact, a type from another package, into +// separate columns: one for the ciphertext and one for each term. +package contacts + +import ( + "context" + "database/sql" + + "example.com/app/crm" + "github.com/cipherstash/stack/languages/golang/encrypt" + "gorm.io/gorm" +) + +//go:generate go tool stashgen -type contactStash -for crm.Contact -model Rows=ContactRow + +// contactStash declares the tags for crm.Contact, which cannot carry them. +// stashgen matches each field to the crm.Contact field with the same name and +// type, and refuses a crm.Contact field this struct does not name. +type contactStash struct { + _ struct{} `stash:"context=contacts"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt,index=equality;match"` + PhoneNumber string `stash:"phone_number,encrypt,index=equality"` + Internal string `stash:"-"` +} + +// ContactRow is a model: one field for each column. Each tag names the output +// the field holds. +type ContactRow struct { + ID int64 `stash:"id"` + Email encrypt.Ciphertext `stash:"email"` + EmailEq encrypt.EqualityTerm `stash:"email,equality"` + EmailMatch encrypt.MatchTerm `stash:"email,match"` + PhoneNumber encrypt.Ciphertext `stash:"phone_number"` + PhoneNumberEq encrypt.EqualityTerm `stash:"phone_number,equality"` +} + +func (ContactRow) TableName() string { return "contacts" } + +// Create writes with database/sql and passes each output itself. +func Create(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, list []crm.Contact) error { + encrypted, err := Encrypt(ctx, cipher, list) + if err != nil { + return err + } + for _, e := range encrypted { + _, err := db.ExecContext(ctx, ` + INSERT INTO contacts (id, email, email_eq, email_match, phone_number, phone_number_eq) + VALUES ($1, $2, $3, $4, $5, $6)`, + e.ID, e.Email.Ciphertext, e.Email.Equality, e.Email.Match, + e.PhoneNumber.Ciphertext, e.PhoneNumber.Equality) + if err != nil { + return err + } + } + return nil +} + +// CreateWithGORM writes the model, which GORM maps one field to one column. +func CreateWithGORM(ctx context.Context, db *gorm.DB, cipher *encrypt.Cipher, list []crm.Contact) error { + rows, err := EncryptRows(ctx, cipher, list) + if err != nil { + return err + } + return db.WithContext(ctx).Create(&rows).Error +} + +func IDByPhone(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, phone string) (int64, error) { + term, err := Fields.PhoneNumber.Equality(ctx, cipher, phone) + if err != nil { + return 0, err + } + var id int64 + err = db.QueryRowContext(ctx, `SELECT id FROM contacts WHERE phone_number_eq = $1`, term).Scan(&id) + return id, err +} diff --git a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden new file mode 100644 index 000000000..fc73074a3 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden @@ -0,0 +1,205 @@ +// Code generated by stashgen. DO NOT EDIT. + +package contacts + +import ( + "context" + "log/slog" + + "example.com/app/crm" + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// crm.Contact prints its sealed fields in the clear, and stashgen cannot add +// print methods to a type from another package. + +type EncryptedContact struct { + ID int64 + Email EncryptedContactEmail + PhoneNumber EncryptedContactPhoneNumber +} + +type EncryptedContactEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} + +type EncryptedContactPhoneNumber struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +func (e EncryptedContact) String() string { + return gensupport.Redacted("EncryptedContact", map[string]any{"ID": e.ID}, "Email", "PhoneNumber") +} + +func (e EncryptedContact) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Email", "PhoneNumber") +} + +// Stops compiling when crm.Contact gains, loses, reorders or retypes a field. +var _ = contactShape(crm.Contact{}) + +type contactShape struct { + ID int64 + Email string + PhoneNumber string + Internal string +} + +var declaration = gensupport.Declare("contacts"). + Passthrough("id"). + EncryptIndex("email", encrypt.Equality, encrypt.Match()). + EncryptIndex("phone_number", encrypt.Equality). + Omit("internal") + +var codec = gensupport.New(gensupport.Generated[crm.Contact, EncryptedContact]{ + TypeName: "crm.Contact", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v crm.Contact) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "email": v.Email, + "phone_number": v.PhoneNumber, + } + }, + Seal: func(rec gensupport.Record) (EncryptedContact, error) { + var e EncryptedContact + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedContact{}, err + } + e.Email = EncryptedContactEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + Match: rec["email"].Match, + } + e.PhoneNumber = EncryptedContactPhoneNumber{ + Ciphertext: rec["phone_number"].Ciphertext, + Equality: rec["phone_number"].Equality, + } + return e, nil + }, + Open: func(e EncryptedContact) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality, Match: e.Email.Match}, + "phone_number": {Ciphertext: e.PhoneNumber.Ciphertext, Equality: e.PhoneNumber.Equality}, + } + }, + Value: func(e EncryptedContact, vals gensupport.Values) (crm.Contact, error) { + var v crm.Contact + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return crm.Contact{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return crm.Contact{}, err + } + if v.PhoneNumber, err = gensupport.Get[string](vals, "phone_number"); err != nil { + return crm.Contact{}, err + } + return v, nil + }, +}) + +// Encrypt seals each crm.Contact in one ZeroKMS request. The result has one +// element for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, contacts []crm.Contact) ([]EncryptedContact, error) { + return codec.Encrypt(ctx, cipher, contacts) +} + +// Decrypt opens each EncryptedContact in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedContact) ([]crm.Contact, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Email EmailField + PhoneNumber PhoneNumberField +}{ + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + PhoneNumber: PhoneNumberField{gensupport.NewField[string](declaration, "phone_number")}, +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedContactEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedContactEmail{}, err + } + return EncryptedContactEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil +} + +func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +type PhoneNumberField struct { + field gensupport.Field[string] +} + +func (f PhoneNumberField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedContactPhoneNumber, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedContactPhoneNumber{}, err + } + return EncryptedContactPhoneNumber{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f PhoneNumberField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +// The -model flag: ContactRow converts to and from its shape only while the +// two have the same fields, with the same types, in the same order. +type rowsShape struct { + ID int64 + Email encrypt.Ciphertext + EmailEq encrypt.EqualityTerm + EmailMatch encrypt.MatchTerm + PhoneNumber encrypt.Ciphertext + PhoneNumberEq encrypt.EqualityTerm +} + +var rowsCodec = gensupport.Records(codec, + func(e EncryptedContact) ContactRow { + return ContactRow(rowsShape{ + ID: e.ID, + Email: e.Email.Ciphertext, + EmailEq: e.Email.Equality, + EmailMatch: e.Email.Match, + PhoneNumber: e.PhoneNumber.Ciphertext, + PhoneNumberEq: e.PhoneNumber.Equality, + }) + }, + func(r ContactRow) EncryptedContact { + s := rowsShape(r) + return EncryptedContact{ + ID: s.ID, + Email: EncryptedContactEmail{Ciphertext: s.Email, Equality: s.EmailEq, Match: s.EmailMatch}, + PhoneNumber: EncryptedContactPhoneNumber{Ciphertext: s.PhoneNumber, Equality: s.PhoneNumberEq}, + } + }, +) + +func EncryptRows(ctx context.Context, cipher *encrypt.Cipher, contacts []crm.Contact) ([]ContactRow, error) { + return rowsCodec.Encrypt(ctx, cipher, contacts) +} + +func DecryptRows(ctx context.Context, d encrypt.Decrypter, rows []ContactRow) ([]crm.Contact, error) { + return rowsCodec.Decrypt(ctx, d, rows) +} diff --git a/languages/golang/stashgen/testdata/cases/contacts/crm/contact.go b/languages/golang/stashgen/testdata/cases/contacts/crm/contact.go new file mode 100644 index 000000000..38dacaf34 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/contacts/crm/contact.go @@ -0,0 +1,9 @@ +// Package crm stands in for a package this program does not own. +package crm + +type Contact struct { + ID int64 + Email string + PhoneNumber string + Internal string +} diff --git a/languages/golang/stashgen/testdata/cases/contacts/go.mod b/languages/golang/stashgen/testdata/cases/contacts/go.mod new file mode 100644 index 000000000..455ad04ab --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/contacts/go.mod @@ -0,0 +1,12 @@ +module example.com/app + +go 1.26 + +require ( + github.com/cipherstash/stack/languages/golang v0.0.0 + gorm.io/gorm v0.0.0 +) + +replace github.com/cipherstash/stack/languages/golang => ../../stubsdk + +replace gorm.io/gorm => ../../stubgorm diff --git a/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden new file mode 100644 index 000000000..e8f9267d8 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden @@ -0,0 +1,88 @@ +// Code generated by stashgen. DO NOT EDIT. + +package documents + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Document prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedDocument struct { + Sealed encrypt.Ciphertext +} + +func (e EncryptedDocument) String() string { + return gensupport.Redacted("EncryptedDocument", nil, "Sealed") +} + +func (e EncryptedDocument) LogValue() slog.Value { + return gensupport.RedactedLog(nil, "Sealed") +} + +// Stops compiling when Document gains, loses, reorders or retypes a field. +var _ = documentShape(Document{}) + +type documentShape struct { + _ struct{} + Title string + Body string + Tags []string +} + +var declaration = gensupport.DeclareOpaque("documents/v2/body") + +var codec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ + TypeName: "Document", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v Document) gensupport.Values { + return gensupport.Values{gensupport.OpaqueField: map[string]any{ + "title": v.Title, + "body": v.Body, + "tags": v.Tags, + }} + }, + Seal: func(rec gensupport.Record) (EncryptedDocument, error) { + return EncryptedDocument{Sealed: rec[gensupport.OpaqueField].Ciphertext}, nil + }, + Open: func(e EncryptedDocument) gensupport.Record { + return gensupport.Record{gensupport.OpaqueField: {Ciphertext: e.Sealed}} + }, + Value: func(e EncryptedDocument, vals gensupport.Values) (Document, error) { + fields, err := gensupport.Get[gensupport.Values](vals, gensupport.OpaqueField) + if err != nil { + return Document{}, err + } + var v Document + if v.Title, err = gensupport.Get[string](fields, "title"); err != nil { + return Document{}, err + } + if v.Body, err = gensupport.Get[string](fields, "body"); err != nil { + return Document{}, err + } + if v.Tags, err = gensupport.Get[[]string](fields, "tags"); err != nil { + return Document{}, err + } + return v, nil + }, +}) + +// Encrypt seals each Document in one ZeroKMS request. The result has one +// element for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, documents []Document) ([]EncryptedDocument, error) { + return codec.Encrypt(ctx, cipher, documents) +} + +// Decrypt opens each EncryptedDocument in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedDocument) ([]Document, error) { + return codec.Decrypt(ctx, d, encrypted) +} diff --git a/languages/golang/stashgen/testdata/cases/documents/documents.go b/languages/golang/stashgen/testdata/cases/documents/documents.go new file mode 100644 index 000000000..09edf3125 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/documents/documents.go @@ -0,0 +1,43 @@ +// Package documents seals each document as one value. Nothing inside it can +// be read or searched on its own, so its fields carry no tags. +package documents + +import ( + "context" + "database/sql" + + "github.com/cipherstash/stack/languages/golang/encrypt" +) + +//go:generate go tool stashgen -type Document + +type Document struct { + _ struct{} `stash:"context=documents/v2/body,opaque"` + Title string + Body string + Tags []string +} + +func Save(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, id int64, doc Document) error { + encrypted, err := Encrypt(ctx, cipher, []Document{doc}) + if err != nil { + return err + } + _, err = db.ExecContext(ctx, `INSERT INTO documents (id, body) VALUES ($1, $2)`, id, encrypted[0].Sealed) + return err +} + +// Load decrypts with the client, which opens each value under the keyset that +// sealed it. A *encrypt.Cipher would also refuse a value from another +// keyset. +func Load(ctx context.Context, db *sql.DB, client *encrypt.Client, id int64) (Document, error) { + var encrypted EncryptedDocument + if err := db.QueryRowContext(ctx, `SELECT body FROM documents WHERE id = $1`, id).Scan(&encrypted.Sealed); err != nil { + return Document{}, err + } + docs, err := Decrypt(ctx, client, []EncryptedDocument{encrypted}) + if err != nil { + return Document{}, err + } + return docs[0], nil +} diff --git a/languages/golang/stashgen/testdata/cases/documents/go.mod b/languages/golang/stashgen/testdata/cases/documents/go.mod new file mode 100644 index 000000000..455ad04ab --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/documents/go.mod @@ -0,0 +1,12 @@ +module example.com/app + +go 1.26 + +require ( + github.com/cipherstash/stack/languages/golang v0.0.0 + gorm.io/gorm v0.0.0 +) + +replace github.com/cipherstash/stack/languages/golang => ../../stubsdk + +replace gorm.io/gorm => ../../stubgorm diff --git a/languages/golang/stashgen/testdata/cases/users/go.mod b/languages/golang/stashgen/testdata/cases/users/go.mod new file mode 100644 index 000000000..455ad04ab --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/users/go.mod @@ -0,0 +1,12 @@ +module example.com/app + +go 1.26 + +require ( + github.com/cipherstash/stack/languages/golang v0.0.0 + gorm.io/gorm v0.0.0 +) + +replace github.com/cipherstash/stack/languages/golang => ../../stubsdk + +replace gorm.io/gorm => ../../stubgorm diff --git a/languages/golang/stashgen/testdata/cases/users/model.go b/languages/golang/stashgen/testdata/cases/users/model.go new file mode 100644 index 000000000..2c1d23961 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/users/model.go @@ -0,0 +1,13 @@ +package users + +//go:generate go tool stashgen -type User + +// Every exported field needs a stash tag, so a new field cannot reach the +// database unencrypted by accident. +type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough" db:"id" gorm:"primaryKey"` + Email string `stash:"email,encrypt_into=TextEq" db:"email"` + Name string `stash:"name,encrypt_into=TextEq" db:"name"` + Internal string `stash:"-"` +} diff --git a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden new file mode 100644 index 000000000..a2af9c6a4 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden @@ -0,0 +1,140 @@ +// Code generated by stashgen. DO NOT EDIT. + +package users + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// User prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedUser struct { + ID int64 `db:"id" gorm:"primaryKey"` + Email eql.TextEq `db:"email"` + Name eql.TextEq `db:"name"` +} + +func (e EncryptedUser) String() string { + return gensupport.Redacted("EncryptedUser", map[string]any{"ID": e.ID}, "Email", "Name") +} + +func (e EncryptedUser) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Email", "Name") +} + +// Stops compiling when User gains, loses, reorders or retypes a field. +var _ = userShape(User{}) + +type userShape struct { + _ struct{} + ID int64 + Email string + Name string + Internal string +} + +var declaration = gensupport.Declare("users"). + Passthrough("id"). + EncryptInto("email", "TextEq"). + EncryptInto("name", "TextEq"). + Omit("internal") + +var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ + TypeName: "User", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v User) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "email": v.Email, + "name": v.Name, + } + }, + Seal: func(rec gensupport.Record) (EncryptedUser, error) { + var e EncryptedUser + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedUser{}, err + } + e.Email = eql.TextEq(rec["email"].EQL) + e.Name = eql.TextEq(rec["name"].EQL) + return e, nil + }, + Open: func(e EncryptedUser) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "email": {EQL: e.Email}, + "name": {EQL: e.Name}, + } + }, + Value: func(e EncryptedUser, vals gensupport.Values) (User, error) { + var v User + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return User{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return User{}, err + } + if v.Name, err = gensupport.Get[string](vals, "name"); err != nil { + return User{}, err + } + return v, nil + }, +}) + +// Encrypt seals each User in one ZeroKMS request. The result has one element +// for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, users []User) ([]EncryptedUser, error) { + return codec.Encrypt(ctx, cipher, users) +} + +// Decrypt opens each EncryptedUser in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Email EmailField + Name NameField +}{ + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + Name: NameField{gensupport.NewField[string](declaration, "name")}, +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f EmailField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} + +type NameField struct { + field gensupport.Field[string] +} + +func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f NameField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} diff --git a/languages/golang/stashgen/testdata/stubgorm/go.mod b/languages/golang/stashgen/testdata/stubgorm/go.mod new file mode 100644 index 000000000..0fd7ae53f --- /dev/null +++ b/languages/golang/stashgen/testdata/stubgorm/go.mod @@ -0,0 +1,3 @@ +module gorm.io/gorm + +go 1.26 diff --git a/languages/golang/stashgen/testdata/stubgorm/gorm.go b/languages/golang/stashgen/testdata/stubgorm/gorm.go new file mode 100644 index 000000000..0e36cd354 --- /dev/null +++ b/languages/golang/stashgen/testdata/stubgorm/gorm.go @@ -0,0 +1,23 @@ +// Package gorm is a TEST STUB of gorm.io/gorm with the types and methods the +// example inputs name. The generator reads gorm.Model's fields through it. +package gorm + +import ( + "context" + "database/sql" + "time" +) + +type DeletedAt sql.NullTime + +type Model struct { + ID uint `gorm:"primarykey"` + CreatedAt time.Time + UpdatedAt time.Time + DeletedAt DeletedAt `gorm:"index"` +} + +type DB struct{ Error error } + +func (db *DB) WithContext(context.Context) *DB { return db } +func (db *DB) Create(any) *DB { return db } diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go new file mode 100644 index 000000000..dbb7b5df3 --- /dev/null +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go @@ -0,0 +1,77 @@ +// Package encrypt is a TEST STUB of the Stack Encrypt Go SDK, signatures +// only. The generator's tests compile generated code against it, so a change +// that emits uncompilable Go fails here. Every symbol it declares is one the +// generated code names; the SDK must declare each with these shapes. +// +// It is not the SDK, and nothing outside stashgen's tests imports it. +package encrypt + +import "context" + +// Cipher is a keyset with any extension of the context. +type Cipher struct{ _ struct{} } + +// Client decrypts under the keyset that sealed each value. +type Client struct{ _ struct{} } + +// Decrypter is what Decrypt takes: a *Client or a *Cipher. +type Decrypter interface{ decrypter() } + +func (*Cipher) decrypter() {} +func (*Client) decrypter() {} + +// Ciphertext is a sealed value in separate columns. +type Ciphertext []byte + +// The term types, one for each index. +type ( + EqualityTerm []byte + MatchTerm []byte + OreTerm []byte + OpeTerm []byte + JSONTerm []byte +) + +// Index is one index on a field, as generated code names it in a declaration. +type Index interface{ index() } + +type namedIndex string + +func (namedIndex) index() {} + +// The indexes. Equality, Ore and Ope take no options; Match and JSON do. +var ( + Equality Index = namedIndex("equality") + Ore Index = namedIndex("ore") + Ope Index = namedIndex("ope") +) + +// MatchOption is an option of the match index. +type MatchOption interface{ matchOption() } + +// Match is the match index with its options. +func Match(...MatchOption) Index { return namedIndex("match") } + +// JSONOption is an option of the json index. +type JSONOption interface{ jsonOption() } + +// JSON is the json index with its options. +func JSON(...JSONOption) Index { return namedIndex("json") } + +// KeysetName names a keyset. +type KeysetName string + +// Keyset returns the cipher for a keyset. +func (*Client) Keyset(KeysetName) *Cipher { return nil } + +// Extend returns a cipher that appends to every field's context. +func (c *Cipher) Extend(string) *Cipher { return c } + +// NewClient opens a client. +func NewClient(context.Context, ...Option) (*Client, error) { return nil, nil } + +// Option configures a client. +type Option interface{ option() } + +// Close closes the client. +func (*Client) Close() error { return nil } diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go new file mode 100644 index 000000000..09f4934a7 --- /dev/null +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go @@ -0,0 +1,9 @@ +// Package eql is a TEST STUB of the generated EQL types: the one type the +// engine produces today, and its query type. +package eql + +// TextEq is an EQL value with an equality term. +type TextEq []byte + +// TextEqQuery is the query value for TextEq. +type TextEqQuery []byte diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go new file mode 100644 index 000000000..6279a9a2b --- /dev/null +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go @@ -0,0 +1,121 @@ +// Package gensupport is a TEST STUB of encrypt/gensupport, signatures only, +// for compiling generated code in the generator's tests. The real package +// lives at languages/golang/encrypt/gensupport; today it holds the version +// constant, Redacted, RedactedLog and the notices, and the integration step +// adds the rest of what is declared here. +package gensupport + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" +) + +type generatedVersion uint8 + +// GeneratedVersion1 is the layout of files stashgen writes today. +const GeneratedVersion1 generatedVersion = 1 + +// OpaqueField is the one field of an opaque declaration. +const OpaqueField = "." + +// Redacted formats a value with its sealed fields hidden. +func Redacted(typeName string, shown map[string]any, hidden ...string) string { return "" } + +// RedactedLog is Redacted for slog. +func RedactedLog(shown map[string]any, hidden ...string) slog.Value { return slog.Value{} } + +// Values are a struct's field values by declared name. +type Values map[string]any + +// Output is what the engine returned for one field. +type Output struct { + Value any + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm + Ore encrypt.OreTerm + Ope encrypt.OpeTerm + JSON encrypt.JSONTerm + EQL []byte +} + +// Record is the engine's output for a struct, by declared name. +type Record map[string]Output + +// Declaration is the data form of a struct's tags. +type Declaration struct{ _ struct{} } + +// Declare starts a declaration with its context. +func Declare(context string) Declaration { return Declaration{} } + +// DeclareOpaque declares a struct sealed as one value. +func DeclareOpaque(context string) Declaration { return Declaration{} } + +func (d Declaration) Passthrough(name string) Declaration { return d } +func (d Declaration) Encrypt(name string) Declaration { return d } +func (d Declaration) EncryptIndex(name string, idx ...encrypt.Index) Declaration { return d } +func (d Declaration) Index(name string, idx ...encrypt.Index) Declaration { return d } +func (d Declaration) EncryptInto(name, eqlType string) Declaration { return d } +func (d Declaration) Omit(name string) Declaration { return d } + +// Generated is what a generated file gives the library for one type. +type Generated[P, E any] struct { + TypeName string + Declaration Declaration + PrintsPlaintext bool + Unexported []string + Source func(P) Values + Seal func(Record) (E, error) + Open func(E) Record + Value func(E, Values) (P, error) +} + +// Codec encrypts and decrypts one type. +type Codec[P, E any] struct{ _ struct{} } + +// New builds the codec for a generated type. +func New[P, E any](Generated[P, E]) *Codec[P, E] { return nil } + +func (*Codec[P, E]) Encrypt(context.Context, *encrypt.Cipher, []P) ([]E, error) { return nil, nil } +func (*Codec[P, E]) Decrypt(context.Context, encrypt.Decrypter, []E) ([]P, error) { + return nil, nil +} + +// Passthrough reads a passthrough field from a record. +func Passthrough[T any](Record, string) (T, error) { var z T; return z, nil } + +// Get reads one value. +func Get[T any](Values, string) (T, error) { var z T; return z, nil } + +// Field is one sealed field's entry. +type Field[T any] struct{ _ struct{} } + +// NewField makes the entry for one field of a declaration. +func NewField[T any](Declaration, string) Field[T] { return Field[T]{} } + +func (Field[T]) Encrypt(context.Context, *encrypt.Cipher, T) (Output, error) { return Output{}, nil } +func (Field[T]) Query(context.Context, *encrypt.Cipher, T) (Output, error) { return Output{}, nil } +func (Field[T]) Equality(context.Context, *encrypt.Cipher, T) (encrypt.EqualityTerm, error) { + return nil, nil +} +func (Field[T]) Match(context.Context, *encrypt.Cipher, T) (encrypt.MatchTerm, error) { + return nil, nil +} +func (Field[T]) Ore(context.Context, *encrypt.Cipher, T) (encrypt.OreTerm, error) { return nil, nil } +func (Field[T]) Ope(context.Context, *encrypt.Cipher, T) (encrypt.OpeTerm, error) { return nil, nil } +func (Field[T]) JSON(context.Context, *encrypt.Cipher, T) (encrypt.JSONTerm, error) { return nil, nil } + +// RecordsCodec encrypts into and decrypts from a model. +type RecordsCodec[P, R any] struct{ _ struct{} } + +// Records wraps a codec with a model's conversions. +func Records[P, E, R any](*Codec[P, E], func(E) R, func(R) E) *RecordsCodec[P, R] { return nil } + +func (*RecordsCodec[P, R]) Encrypt(context.Context, *encrypt.Cipher, []P) ([]R, error) { + return nil, nil +} +func (*RecordsCodec[P, R]) Decrypt(context.Context, encrypt.Decrypter, []R) ([]P, error) { + return nil, nil +} diff --git a/languages/golang/stashgen/testdata/stubsdk/go.mod b/languages/golang/stashgen/testdata/stubsdk/go.mod new file mode 100644 index 000000000..555c134e5 --- /dev/null +++ b/languages/golang/stashgen/testdata/stubsdk/go.mod @@ -0,0 +1,3 @@ +module github.com/cipherstash/stack/languages/golang + +go 1.26 From 698b3af492719d852b5c8c784c29e895981d181a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 00:29:33 -0700 Subject: [PATCH 02/30] feat(golang): stashgen command, refusal tests and the stub contract check Add cmd/stashgen, the go:generate front end: the flags from the plan's reference (-type, -name, -for, -model repeated, -redact, -output), the notices on stderr, and a non-zero exit with no file written for every refusal. run takes the engine constructor so the tests drive it with the static engine while main uses GuestEngine, which still stops with ErrEngineUnavailable: the command is wired, the engine is not. The fake engine moves from internal/fakeengine to the public enginetest package so the command's tests can import it; the Go tool forbids an internal import across the cmd/ and stashgen/ trees. Every entry of the plan's refusal list has a test asserting the error names the field, through *FieldError and errors.As, plus the refusals the list implies: an untagged embedded struct from another package, -redact on a type that already prints, -for on a type with a field the struct does not name, a model field of the wrong type, and a struct that stores no field. Three more golden cases cover what the plan's examples do not: two tagged structs in one package with -name, separate columns on an integer with ore and ope, an embedded struct of the same package, and -for a type with unexported fields, which takes the by-name path and pointers. TestStubAgreesWithGensupport loads the real encrypt/gensupport and the test stub with go/packages and fails when a symbol both declare has two shapes, so the contract the integration step implements cannot drift from what already ships. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/cmd/stashgen/README.md | 152 +++++++++++++ languages/golang/cmd/stashgen/main.go | 103 +++++++++ languages/golang/cmd/stashgen/main_test.go | 144 +++++++++++++ .../encrypt/gensupport/gensupport_test.go | 3 +- languages/golang/stashgen/doc.go | 18 ++ languages/golang/stashgen/engine.go | 4 +- .../enginetest.go} | 14 +- languages/golang/stashgen/golden_test.go | 42 +++- languages/golang/stashgen/read.go | 6 +- languages/golang/stashgen/refusal_test.go | 203 ++++++++++++++++++ languages/golang/stashgen/stub_test.go | 61 ++++++ .../testdata/cases/embedded/embedded.go | 18 ++ .../stashgen/testdata/cases/embedded/go.mod | 12 ++ .../cases/embedded/patient_stash.go.golden | 175 +++++++++++++++ .../stashgen/testdata/cases/foreign/go.mod | 12 ++ .../testdata/cases/foreign/individuals.go | 19 ++ .../foreign/individualstash_stash.go.golden | 182 ++++++++++++++++ .../testdata/cases/foreign/pb/individual.go | 15 ++ .../stashgen/testdata/cases/orders/go.mod | 12 ++ .../cases/orders/order_stash.go.golden | 193 +++++++++++++++++ .../stashgen/testdata/cases/orders/orders.go | 26 +++ .../cases/orders/refund_stash.go.golden | 109 ++++++++++ .../stubsdk/encrypt/gensupport/gensupport.go | 5 + 23 files changed, 1509 insertions(+), 19 deletions(-) create mode 100644 languages/golang/cmd/stashgen/README.md create mode 100644 languages/golang/cmd/stashgen/main.go create mode 100644 languages/golang/cmd/stashgen/main_test.go create mode 100644 languages/golang/stashgen/doc.go rename languages/golang/stashgen/{internal/fakeengine/fakeengine.go => enginetest/enginetest.go} (91%) create mode 100644 languages/golang/stashgen/refusal_test.go create mode 100644 languages/golang/stashgen/stub_test.go create mode 100644 languages/golang/stashgen/testdata/cases/embedded/embedded.go create mode 100644 languages/golang/stashgen/testdata/cases/embedded/go.mod create mode 100644 languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden create mode 100644 languages/golang/stashgen/testdata/cases/foreign/go.mod create mode 100644 languages/golang/stashgen/testdata/cases/foreign/individuals.go create mode 100644 languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden create mode 100644 languages/golang/stashgen/testdata/cases/foreign/pb/individual.go create mode 100644 languages/golang/stashgen/testdata/cases/orders/go.mod create mode 100644 languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden create mode 100644 languages/golang/stashgen/testdata/cases/orders/orders.go create mode 100644 languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md new file mode 100644 index 000000000..122f8e6d6 --- /dev/null +++ b/languages/golang/cmd/stashgen/README.md @@ -0,0 +1,152 @@ +# stashgen + +`stashgen` writes the Go code that encrypts a struct with Stack Encrypt. +You declare how each field is encrypted with `stash` tags. +`stashgen` writes the encrypted type and the functions that encrypt, decrypt and search it. +Your program calls those functions, and it never builds or names a plan. + +## Use the SDK + +1. Add the generator to your module. + This needs Go 1.24 or later. + + ```sh + go get -tool github.com/cipherstash/stack/languages/golang/cmd/stashgen + ``` + +2. Put a `stash` tag on every exported field of the struct, and a `go:generate` comment beside it. + + ```go + //go:generate go tool stashgen -type User + type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt_into=TextEq"` + Name string `stash:"name,encrypt_into=TextEq"` + } + ``` + +3. Run the generator. + It writes `user_stash.go` beside the struct. + + ```sh + go generate ./... + ``` + +4. Commit the generated file. + +5. Call the generated functions where you write and read. + + ```go + encrypted, err := users.Encrypt(ctx, cipher, people) + people, err := users.Decrypt(ctx, cipher, encrypted) + query, err := users.Fields.Email.Query(ctx, cipher, "bob@example.com") + ``` + +6. Store the encrypted type. + Each field is one column, so `database/sql`, pgx, sqlx and GORM take it as it is. + +7. Run the generator again after each change to the struct or to a tag. + A change to the fields of the struct stops the build until you do. + +8. In CI, run the generator and fail when a generated file changes. + + ```sh + go generate ./... && git diff --exit-code + ``` + +The rest of this file is the reference. + +## Struct tags + +The first part of a tag is the field's name, which is the column name in a database. + +| Tag | Meaning | +|---|---| +| `` _ struct{} `stash:"context=users"` `` | the context of every field in the struct | +| `stash:"email,encrypt_into=TextEq"` | seal the field into one EQL value, with the terms that EQL type has | +| `stash:"notes,encrypt"` | seal the field, with no index | +| `stash:"email,encrypt,index=equality;match"` | seal the field, and derive each index beside it | +| `stash:"attrs,index=json"` | derive the index alone | +| `stash:"id,passthrough"` | store the field as it is | +| `stash:"-"` | leave the field out | +| `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value | + +The index names are `equality`, `match`, `ore`, `ope` and `json`. +An index takes its options in parentheses after its name, separated by commas: `index=equality;match(k=3)`. +These words are the same as the Rust API's words for the same behaviour. + +An embedded struct of your own adds its tagged fields to the outer struct. +An embedded struct from another package takes one tag for all of its fields: `stash:",passthrough"` stores them as they are, and `stash:"-"` leaves them out. +Tags for other libraries, such as `gorm`, `db` and `json`, are copied to the same field of the generated type. + +An unexported field with no `stash` tag is ignored, with three notices: `stashgen` prints a warning, the generated file names the field in a comment, and the program prints a warning to stderr once for each type. +`stash:"-"` on the field states the choice, and all three stop. + +## What stashgen writes + +For `-type User`, the file `user_stash.go` holds: + +- `EncryptedUser`, with one field for each stored field of `User`, with the same name. + A passthrough field keeps its Go type. + A field with `encrypt_into` holds one EQL value. + A field with `encrypt` or `index=` holds a struct with one field for each output, such as `Email.Ciphertext` and `Email.Equality`. +- `Encrypt` and `Decrypt`, which take a slice and return a slice, and send one ZeroKMS request for all of it. +- `Fields`, with one entry for each sealed field. + An entry encrypts one value, and it has a query method only for what the field declares: `Query` for `encrypt_into`, and `Equality`, `Match`, `Ore` or `Ope` for `index=`. +- `String` and `LogValue` on `EncryptedUser`, which print the passthrough fields and hide the sealed ones. +- A copy of the fields of `User`, so a change to them stops the build. + +Generated code uses no reflection, and no function in it panics. + +## Flags + +| Flag | Meaning | +|---|---| +| `-type T` | The struct that carries the `stash` tags. Required. | +| `-name N` | Write `EncryptN`, `DecryptN` and `NFields`. A package holds one `Encrypt`; the second struct in a package needs a name. | +| `-for P.F` | `T` declares the tags for `F`, a type in another package. Each field of `T` names a field of `F` with the same name and type, and every exported field of `F` is named. | +| `-model Name=R` | A model `R` for separate columns: one field for each column, each tagged with the output it holds, `stash:"email"` or `stash:"email,equality"`. Writes `EncryptName` and `DecryptName`. Any number. | +| `-model Name=R:D` | The same, for an `R` that cannot carry tags. The struct `D` in your package declares them. | +| `-redact` | Write `String` and `LogValue` methods on `T`. | +| `-output file` | The file to write. The default is the type's name in lower case, with `_stash.go`. | + +`stashgen` loads the package with `golang.org/x/tools/go/packages` and reads types, not text. +It runs none of the package's code. +It ignores its own output file when it loads the package, so a stale file does not stop it. +The same input always gives the same file: fields keep their declared order, and the file carries no version and no time. + +`stashgen` checks each declaration with the engine, and holds no copy of the engine's rules. +It asks the engine for the EQL types it holds: each name, its plaintext type, its indexes and its query form. +The engine produces one EQL type today, `TextEq`. +Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`. + +## When stashgen stops + +`stashgen` stops with an error, and writes no file, for each of these. +The error names the type and the field, and never a value. + +- an exported field with no `stash` tag; +- a tag that does not parse, or two fields with one name; +- a struct with no `context=` field; +- an index or an EQL type that does not apply to the field's Go type, such as `match` on an `int32`; +- a field type that the engine cannot seal; +- an EQL type that the engine cannot produce yet; +- a `passthrough` field that has an index; +- a model with a field that has no tag, or with no field for an output; +- two structs in one package that would both write `Encrypt`; +- an embedded struct from another package with no tag; +- a struct whose every field is left out. + +## Printing + +A generated type hides its sealed fields when a program prints or logs it. +The struct you wrote is not protected: `stashgen` warns when it has sealed fields and no `String` and `LogValue` methods, and the program prints the same warning to stderr once for each type. +`-redact` makes `stashgen` write those two methods on the struct. +No warning, error or log line holds a plaintext value. + +## Status + +The generator is built and tested against a static stand-in for the engine. +`stashgen.GuestEngine`, which runs the WASI guest the SDK embeds, is not wired yet, so the command stops with `ErrEngineUnavailable` until it is. +The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`. diff --git a/languages/golang/cmd/stashgen/main.go b/languages/golang/cmd/stashgen/main.go new file mode 100644 index 000000000..5064338d0 --- /dev/null +++ b/languages/golang/cmd/stashgen/main.go @@ -0,0 +1,103 @@ +// Command stashgen writes the encrypted type and its functions for a struct +// with stash tags. go generate runs it from the struct's package: +// +// //go:generate go tool stashgen -type User +// +// See README.md beside this file for the steps and the flag reference. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "io" + "os" + "strings" + + "github.com/cipherstash/stack/languages/golang/stashgen" +) + +func main() { + os.Exit(run(os.Args[1:], "", os.Stdout, os.Stderr, stashgen.GuestEngine)) +} + +// modelFlags collects the -model flags, which repeat. +type modelFlags []stashgen.ModelRequest + +func (m *modelFlags) String() string { + parts := make([]string, len(*m)) + for i, r := range *m { + parts[i] = r.Name + "=" + r.Type + if r.Declares != "" { + parts[i] += ":" + r.Declares + } + } + return strings.Join(parts, " ") +} + +func (m *modelFlags) Set(s string) error { + r, err := stashgen.ParseModelFlag(s) + if err != nil { + return err + } + *m = append(*m, r) + return nil +} + +// run is main without the process: dir is the package directory ("" for the +// working directory, which is where go generate runs), and newEngine gives +// the engine that checks the declaration. +func run(args []string, dir string, stdout, stderr io.Writer, newEngine func(context.Context) (stashgen.Engine, error)) int { + fs := flag.NewFlagSet("stashgen", flag.ContinueOnError) + fs.SetOutput(stderr) + req := stashgen.Request{Dir: dir} + var models modelFlags + fs.StringVar(&req.Type, "type", "", "the struct that carries the stash tags (required)") + fs.StringVar(&req.Name, "name", "", "write EncryptN, DecryptN and NFields instead of Encrypt, Decrypt and Fields") + fs.StringVar(&req.For, "for", "", "P.F: -type declares the tags for F, a type in package P") + fs.Var(&models, "model", "Name=R or Name=R:D: a model R for separate columns; writes EncryptName and DecryptName (repeatable)") + fs.BoolVar(&req.Redact, "redact", false, "write String and LogValue methods on the -type struct") + fs.StringVar(&req.Output, "output", "", "the file to write (default: the type's name in lower case, with _stash.go)") + fs.Usage = func() { + fmt.Fprintln(stderr, "usage: stashgen -type T [-name N] [-for P.F] [-model Name=R[:D]]... [-redact] [-output file]") + fs.PrintDefaults() + } + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return 2 + } + return 2 + } + if fs.NArg() > 0 { + fmt.Fprintf(stderr, "stashgen: unexpected argument %q\n", fs.Arg(0)) + fs.Usage() + return 2 + } + if req.Type == "" { + fmt.Fprintln(stderr, "stashgen: -type is required") + fs.Usage() + return 2 + } + req.Models = models + + ctx := context.Background() + engine, err := newEngine(ctx) + if err != nil { + fmt.Fprintln(stderr, err) + return 1 + } + file, err := stashgen.FromTags(ctx, engine, req) + if err != nil { + fmt.Fprintln(stderr, err) + return 1 + } + for _, n := range file.Notices { + fmt.Fprintln(stderr, n) + } + if err := file.Write(); err != nil { + fmt.Fprintf(stderr, "stashgen: write %s: %v\n", file.Path, err) + return 1 + } + return 0 +} diff --git a/languages/golang/cmd/stashgen/main_test.go b/languages/golang/cmd/stashgen/main_test.go new file mode 100644 index 000000000..9398a2d5c --- /dev/null +++ b/languages/golang/cmd/stashgen/main_test.go @@ -0,0 +1,144 @@ +package main + +import ( + "bytes" + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/stashgen" + "github.com/cipherstash/stack/languages/golang/stashgen/enginetest" +) + +func fake(context.Context) (stashgen.Engine, error) { return enginetest.Static{}, nil } + +// writeModule writes a module that resolves the SDK to the generator's stub. +func writeModule(t *testing.T, files map[string]string) string { + t.Helper() + stubs, err := filepath.Abs(filepath.Join("..", "..", "stashgen", "testdata")) + if err != nil { + t.Fatal(err) + } + dir := t.TempDir() + gomod := "module example.com/app\n\ngo 1.26\n\nrequire github.com/cipherstash/stack/languages/golang v0.0.0\n\nreplace github.com/cipherstash/stack/languages/golang => " + filepath.Join(stubs, "stubsdk") + "\n" + if err := os.WriteFile(filepath.Join(dir, "go.mod"), []byte(gomod), 0o600); err != nil { + t.Fatal(err) + } + for name, src := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o600); err != nil { + t.Fatal(err) + } + } + return dir +} + +const userSource = `package users + +type User struct { + _ struct{} ` + "`stash:\"context=users\"`" + ` + ID int64 ` + "`stash:\"id,passthrough\"`" + ` + Email string ` + "`stash:\"email,encrypt_into=TextEq\"`" + ` + cache string +} +` + +func TestRunWritesTheFileAndPrintsTheNotices(t *testing.T) { + dir := writeModule(t, map[string]string{"model.go": userSource}) + var stdout, stderr bytes.Buffer + if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, fake); code != 0 { + t.Fatalf("exit %d\n%s", code, stderr.String()) + } + out, err := os.ReadFile(filepath.Join(dir, "user_stash.go")) + if err != nil { + t.Fatal(err) + } + if !bytes.HasPrefix(out, []byte("// Code generated by stashgen. DO NOT EDIT.\n")) { + t.Fatalf("output starts with %q", out[:40]) + } + for _, want := range []string{ + `stashgen: User: not encrypted and not stored: the unexported field "cache". Tag it ` + "`stash:\"-\"`" + ` to confirm that.`, + "stashgen: User prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", + } { + if !strings.Contains(stderr.String(), want) { + t.Errorf("stderr lacks %q:\n%s", want, stderr.String()) + } + } + if stdout.Len() != 0 { + t.Errorf("stdout = %q, want nothing", stdout.String()) + } + + // A second run over the stale file writes the same bytes. + stderr.Reset() + if code := run([]string{"-type", "User", "-redact"}, dir, &stdout, &stderr, fake); code != 0 { + t.Fatalf("second run: exit %d\n%s", code, stderr.String()) + } + again, err := os.ReadFile(filepath.Join(dir, "user_stash.go")) + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(again, []byte("func (u User) String() string")) { + t.Fatal("-redact did not write String on User") + } + if strings.Contains(stderr.String(), "prints its sealed fields") { + t.Errorf("-redact still warns about printing:\n%s", stderr.String()) + } +} + +func TestRunFlags(t *testing.T) { + dir := writeModule(t, map[string]string{"model.go": userSource}) + cases := []struct { + args []string + code int + want string + }{ + {nil, 2, "-type is required"}, + {[]string{"-type"}, 2, "flag needs an argument"}, + {[]string{"-type", "User", "extra"}, 2, `unexpected argument "extra"`}, + {[]string{"-type", "User", "-model", "Rows"}, 2, "is not Name=R or Name=R:D"}, + {[]string{"-type", "User", "-model", "rows=R"}, 2, "must be an exported Go name"}, + {[]string{"-type", "Nobody"}, 1, "has no type Nobody"}, + {[]string{"-type", "User", "-output", "custom_stash.go"}, 0, ""}, + {[]string{"-h"}, 2, "usage: stashgen"}, + } + for _, c := range cases { + var stdout, stderr bytes.Buffer + code := run(c.args, dir, &stdout, &stderr, fake) + if code != c.code { + t.Errorf("%v: exit %d, want %d\n%s", c.args, code, c.code, stderr.String()) + } + if !strings.Contains(stderr.String(), c.want) { + t.Errorf("%v: stderr %q lacks %q", c.args, stderr.String(), c.want) + } + } + if _, err := os.Stat(filepath.Join(dir, "custom_stash.go")); err != nil { + t.Errorf("-output: %v", err) + } +} + +func TestRunStopsWithoutAnEngine(t *testing.T) { + dir := writeModule(t, map[string]string{"model.go": userSource}) + var stdout, stderr bytes.Buffer + if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 1 { + t.Fatalf("exit %d, want 1", code) + } + if !strings.Contains(stderr.String(), stashgen.ErrEngineUnavailable.Error()) { + t.Fatalf("stderr = %q", stderr.String()) + } + if _, err := os.Stat(filepath.Join(dir, "user_stash.go")); !os.IsNotExist(err) { + t.Fatal("a file was written with no engine") + } +} + +func TestModelFlagsRepeat(t *testing.T) { + var m modelFlags + for _, s := range []string{"Rows=ContactRow", "Legacy=userdb.Contact:contactRow"} { + if err := m.Set(s); err != nil { + t.Fatal(err) + } + } + if got := m.String(); got != "Rows=ContactRow Legacy=userdb.Contact:contactRow" { + t.Fatalf("String = %q", got) + } +} diff --git a/languages/golang/encrypt/gensupport/gensupport_test.go b/languages/golang/encrypt/gensupport/gensupport_test.go index 10d706efd..68bd64ea0 100644 --- a/languages/golang/encrypt/gensupport/gensupport_test.go +++ b/languages/golang/encrypt/gensupport/gensupport_test.go @@ -67,7 +67,8 @@ func TestVersionConstantIsTyped(t *testing.T) { // A generated file holds `const _ = gensupport.GeneratedVersion1`. The // constant's type is unexported, so no other package can declare a value // that satisfies the same reference. - var _ generatedVersion = GeneratedVersion1 + accepts := func(generatedVersion) {} + accepts(GeneratedVersion1) if GeneratedVersion1 != 1 { t.Fatalf("GeneratedVersion1 = %d", GeneratedVersion1) } diff --git a/languages/golang/stashgen/doc.go b/languages/golang/stashgen/doc.go new file mode 100644 index 000000000..4185839d2 --- /dev/null +++ b/languages/golang/stashgen/doc.go @@ -0,0 +1,18 @@ +// Package stashgen is the Stack Encrypt code generator for Go, as a library. +// +// A struct's stash tags declare how each field is encrypted. [FromTags] reads +// those tags through go/packages and go/types, checks the declaration with an +// [Engine], and returns the file that the command stashgen writes beside the +// struct: the encrypted type, Encrypt, Decrypt, Fields and the print methods. +// The command at languages/golang/cmd/stashgen is the go:generate front end +// for this package. +// +// The generator reads types, not text, and runs none of the package's code. +// It ignores its own output file when it loads the package, so a stale +// generated file does not stop it. The same input always gives the same file: +// fields keep their declared order, and the file carries no version and no +// time beyond the gensupport.GeneratedVersion1 constant it names. +// +// Every refusal about an index, an EQL type or a field type comes from the +// Engine. The generator holds no copy of the engine's rules. +package stashgen diff --git a/languages/golang/stashgen/engine.go b/languages/golang/stashgen/engine.go index b43bec277..8817d8dea 100644 --- a/languages/golang/stashgen/engine.go +++ b/languages/golang/stashgen/engine.go @@ -44,8 +44,8 @@ var ErrEngineUnavailable = errors.New("stashgen: the embedded engine is not avai // TODO(stack#1046 integration): run the WASI guest that package encrypt // embeds (the build with the EQL types), ask it for its EQL types and have // it check each declaration. Until then this returns [ErrEngineUnavailable], -// and stashgen stops before it reads any package. The fake in -// internal/fakeengine is the reference for what an Engine answers. +// and stashgen stops before it reads any package. The static +// engine in package enginetest is the reference for what an Engine answers. func GuestEngine(context.Context) (Engine, error) { return nil, ErrEngineUnavailable } diff --git a/languages/golang/stashgen/internal/fakeengine/fakeengine.go b/languages/golang/stashgen/enginetest/enginetest.go similarity index 91% rename from languages/golang/stashgen/internal/fakeengine/fakeengine.go rename to languages/golang/stashgen/enginetest/enginetest.go index 33f050312..e3361d1ee 100644 --- a/languages/golang/stashgen/internal/fakeengine/fakeengine.go +++ b/languages/golang/stashgen/enginetest/enginetest.go @@ -1,11 +1,11 @@ -// Package fakeengine is a static stand-in for the Rust engine in the +// Package enginetest is a static stand-in for the Rust engine in the // generator's tests. It knows what the engine produces today: the EQL type // TextEq, and the four indexes with the Go kinds each applies to. // // It is a test double, not a copy of the rules the SDK ships: the SDK's // generator asks the embedded guest. When the engine learns a type or an // index, update this file and the tests that read it. -package fakeengine +package enginetest import ( "context" @@ -14,20 +14,20 @@ import ( "github.com/cipherstash/stack/languages/golang/stashgen" ) -// Engine is the static fake. The zero value is ready to use. -type Engine struct{} +// Static is the fake engine. The zero value is ready to use. +type Static struct{} -var _ stashgen.Engine = Engine{} +var _ stashgen.Engine = Static{} // EQLTypes returns TextEq, the one EQL type the engine produces today. -func (Engine) EQLTypes(context.Context) ([]stashgen.EQLType, error) { +func (Static) EQLTypes(context.Context) ([]stashgen.EQLType, error) { return []stashgen.EQLType{ {Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery"}, }, nil } // Check applies the engine's rules as they stand today. -func (e Engine) Check(ctx context.Context, d stashgen.Declaration) error { +func (e Static) Check(ctx context.Context, d stashgen.Declaration) error { eqlTypes, _ := e.EQLTypes(ctx) for _, f := range d.Fields { if f.Verb == stashgen.VerbOmit { diff --git a/languages/golang/stashgen/golden_test.go b/languages/golang/stashgen/golden_test.go index 548717a11..1c00fbcab 100644 --- a/languages/golang/stashgen/golden_test.go +++ b/languages/golang/stashgen/golden_test.go @@ -11,7 +11,7 @@ import ( "testing" "github.com/cipherstash/stack/languages/golang/stashgen" - "github.com/cipherstash/stack/languages/golang/stashgen/internal/fakeengine" + "github.com/cipherstash/stack/languages/golang/stashgen/enginetest" ) var update = flag.Bool("update", false, "rewrite the golden files from the generator's output") @@ -29,21 +29,32 @@ var goldenCases = map[string]goldenCase{ "accounts": {req: stashgen.Request{Type: "Account", Redact: true}, golden: "account_stash.go.golden"}, "contacts": {req: stashgen.Request{Type: "contactStash", For: "crm.Contact", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "ContactRow"}}}, golden: "contactstash_stash.go.golden"}, "documents": {req: stashgen.Request{Type: "Document"}, golden: "document_stash.go.golden"}, + // Beyond the plan's examples: -name beside an unnamed struct in one + // package, separate columns on an integer, an embedded struct of the same + // package, and -for a type with unexported fields. + "orders": {req: stashgen.Request{Type: "Order", Name: "Order"}, golden: "order_stash.go.golden"}, + "refunds": {dir: "", req: stashgen.Request{Type: "Refund"}, golden: "refund_stash.go.golden"}, + "embedded": {req: stashgen.Request{Type: "Patient"}, golden: "patient_stash.go.golden"}, + "foreign": {req: stashgen.Request{Type: "individualStash", For: "pb.Individual"}, golden: "individualstash_stash.go.golden"}, } func TestGolden(t *testing.T) { for name, c := range goldenCases { t.Run(name, func(t *testing.T) { - caseDir := filepath.Join("testdata", "cases", name) + module := name + if name == "refunds" { + module = "orders" + } + caseDir := filepath.Join("testdata", "cases", module) req := c.req req.Dir = filepath.Join(caseDir, c.dir) - file, err := stashgen.FromTags(context.Background(), fakeengine.Engine{}, req) + file, err := stashgen.FromTags(context.Background(), enginetest.Static{}, req) if err != nil { t.Fatalf("FromTags: %v", err) } goldenPath := filepath.Join(caseDir, c.dir, c.golden) if *update { - if err := os.WriteFile(goldenPath, file.Content, 0o644); err != nil { + if err := os.WriteFile(goldenPath, file.Content, 0o600); err != nil { t.Fatal(err) } } @@ -82,12 +93,27 @@ func compileCase(t *testing.T, caseDir, pkgDir, fileName string, content []byte) } gomod = bytes.ReplaceAll(gomod, []byte("../../stubsdk"), []byte(filepath.Join(testdata, "stubsdk"))) gomod = bytes.ReplaceAll(gomod, []byte("../../stubgorm"), []byte(filepath.Join(testdata, "stubgorm"))) - if err := os.WriteFile(filepath.Join(tmp, "go.mod"), gomod, 0o644); err != nil { + if err := os.WriteFile(filepath.Join(tmp, "go.mod"), gomod, 0o600); err != nil { t.Fatal(err) } - if err := os.WriteFile(filepath.Join(tmp, pkgDir, fileName), content, 0o644); err != nil { + if err := os.WriteFile(filepath.Join(tmp, pkgDir, fileName), content, 0o600); err != nil { t.Fatal(err) } + // A module with two tagged structs builds only with both generated files. + goldens, _ := filepath.Glob(filepath.Join(caseDir, pkgDir, "*.golden")) + for _, g := range goldens { + name := strings.TrimSuffix(filepath.Base(g), ".golden") + if name == fileName { + continue + } + data, err := os.ReadFile(g) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(tmp, pkgDir, name), data, 0o600); err != nil { + t.Fatal(err) + } + } cmd := exec.Command("go", "vet", "./...") cmd.Dir = tmp cmd.Env = append(os.Environ(), "GOPROXY=off", "GOWORK=off", "GOFLAGS=-mod=mod", "CGO_ENABLED=0") @@ -107,7 +133,7 @@ func copyTree(src, dst string) error { } target := filepath.Join(dst, rel) if d.IsDir() { - return os.MkdirAll(target, 0o755) + return os.MkdirAll(target, 0o750) } if strings.HasSuffix(path, ".golden") { return nil @@ -116,7 +142,7 @@ func copyTree(src, dst string) error { if err != nil { return err } - return os.WriteFile(target, data, 0o644) + return os.WriteFile(target, data, 0o600) }) } diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index 14f84e91d..f2d937939 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -213,6 +213,7 @@ func (r *reader) read() (*genFile, error) { // The shape check. A type from another package with an unexported field // cannot convert, so the file reads each field by name instead, and the // functions take a pointer. + var conversionNotice string switch { case req.For == "": f.shapeFields = shapeOf(valueStruct, r.typeExpr) @@ -221,7 +222,7 @@ func (r *reader) read() (*genFile, error) { f.isPointer = true f.typeExpr = "*" + f.typeExpr f.zeroExpr = "nil" - f.notices = append(f.notices, fmt.Sprintf("%s has unexported fields, so Go cannot convert it to a copy of its fields. This file reads each field by name: the compiler finds a removed or retyped field, and CI finds an added one.", f.typeName)) + conversionNotice = fmt.Sprintf("%s has unexported fields, so Go cannot convert it to a copy of its fields. This file reads each field by name: the compiler finds a removed or retyped field, and CI finds an added one.", f.typeName) default: f.shapeFields = shapeOf(valueStruct, r.typeExpr) } @@ -251,6 +252,9 @@ func (r *reader) read() (*genFile, error) { f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", f.typeName)) } } + if conversionNotice != "" { + f.notices = append(f.notices, conversionNotice) + } if len(f.unexported) > 0 { f.notices = append(f.notices, "Not encrypted and not stored: the unexported "+fieldList(f.unexported)+". Tag "+itOrEach(f.unexported)+" `stash:\"-\"` to confirm that.") } diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go new file mode 100644 index 000000000..a26d44c60 --- /dev/null +++ b/languages/golang/stashgen/refusal_test.go @@ -0,0 +1,203 @@ +package stashgen_test + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/stashgen" + "github.com/cipherstash/stack/languages/golang/stashgen/enginetest" +) + +// generate writes one package into a module that resolves the SDK to the +// stub, and runs the generator over it. +func generate(t *testing.T, src string, req stashgen.Request) (*stashgen.File, error) { + t.Helper() + dir := writeModule(t, map[string]string{"model.go": src, "crm/contact.go": crmPackage}) + req.Dir = dir + return stashgen.FromTags(context.Background(), enginetest.Static{}, req) +} + +func writeModule(t *testing.T, files map[string]string) string { + t.Helper() + stubs, err := filepath.Abs("testdata") + if err != nil { + t.Fatal(err) + } + dir := t.TempDir() + gomod := "module example.com/app\n\ngo 1.26\n\nrequire (\n\tgithub.com/cipherstash/stack/languages/golang v0.0.0\n\tgorm.io/gorm v0.0.0\n)\n\nreplace github.com/cipherstash/stack/languages/golang => " + filepath.Join(stubs, "stubsdk") + "\n\nreplace gorm.io/gorm => " + filepath.Join(stubs, "stubgorm") + "\n" + if err := os.WriteFile(filepath.Join(dir, "go.mod"), []byte(gomod), 0o600); err != nil { + t.Fatal(err) + } + for name, src := range files { + if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o750); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o600); err != nil { + t.Fatal(err) + } + } + return dir +} + +// user wraps field lines into a tagged struct named User in package users. +func user(fields string) string { + return "package users\n\nimport (\n\t\"time\"\n\n\t\"example.com/app/crm\"\n\t\"github.com/cipherstash/stack/languages/golang/encrypt\"\n\t\"gorm.io/gorm\"\n)\n\nvar (\n\t_ time.Time\n\t_ crm.Contact\n\t_ encrypt.Cipher\n\t_ gorm.Model\n)\n\ntype User struct {\n" + fields + "\n}\n" +} + +const ctx = "\t_ struct{} `stash:\"context=users\"`\n" + +// TestRefusals covers every entry of the refusal list in the plan's stashgen +// reference, and a few the reference implies. Each refusal names the field. +func TestRefusals(t *testing.T) { + cases := []struct { + name string + src string + req stashgen.Request + field string // the field the error must name, "" for the type + want string + }{ + {"an exported field with no stash tag", user(ctx + "\tEmail string"), stashgen.Request{Type: "User"}, "Email", "no stash tag"}, + {"a tag that does not parse", user(ctx + "\tEmail string `stash:\"email,shred\"`"), stashgen.Request{Type: "User"}, "Email", `unknown part "shred"`}, + {"two fields with one name", user(ctx + "\tEmail string `stash:\"email,encrypt\"`\n\tAlt string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Alt", `two fields write the name "email"`}, + {"an omitted field whose name a column takes", user(ctx + "\tEmail string `stash:\"-\"`\n\tAddr string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Addr", `two fields write the name "email"`}, + {"a struct with no context field", user("\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "", "declares the context"}, + {"a context declared twice", user(ctx + ctx + "\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "_", "declared twice"}, + {"context on a named field", user(ctx + "\tEmail string `stash:\"context=x\"`"), stashgen.Request{Type: "User"}, "Email", "goes on a `_ struct{}` field"}, + {"a _ field with no context", user("\t_ struct{}\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "_", "carries the context tag"}, + {"an index that does not apply to the field type", user(ctx + "\tAge int32 `stash:\"age,encrypt,index=match\"`"), stashgen.Request{Type: "User"}, "Age", "match applies to a string, not to int32"}, + {"an EQL type that does not apply to the field type", user(ctx + "\tAge int64 `stash:\"age,encrypt_into=TextEq\"`"), stashgen.Request{Type: "User"}, "Age", "TextEq seals a string, and int64 is int"}, + {"a field type the engine cannot seal", user(ctx + "\tDone chan int `stash:\"done,encrypt\"`"), stashgen.Request{Type: "User"}, "Done", "cannot seal a value of type chan int"}, + {"a struct with unexported fields the engine cannot seal", user(ctx + "\tAt time.Time `stash:\"at,encrypt\"`"), stashgen.Request{Type: "User"}, "At", "cannot seal a value of type time.Time"}, + {"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "cannot produce the EQL type TextMatch"}, + {"an index on an equality-only kind", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "equality does not apply to map[string]string"}, + {"the json index, not in the engine yet", user(ctx + "\tAttrs map[string]string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"}, + {"an index option the engine cannot carry", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "cannot carry index options"}, + {"a passthrough field that has an index", user(ctx + "\tID int64 `stash:\"id,passthrough,index=equality\"`"), stashgen.Request{Type: "User"}, "ID", "passthrough field has no index"}, + {"an embedded struct from another package with no tag", user(ctx + "\tgorm.Model\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Model", "cannot carry tags; tag the field"}, + {"an embedded struct with a verb other than passthrough", user(ctx + "\tgorm.Model `stash:\",encrypt\"`\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Model", "takes only passthrough"}, + {"an embedded pointer", user(ctx + "\t*gorm.Model `stash:\",passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Model", "must be a struct, not a pointer"}, + {"a struct that stores no field", user(ctx + "\tEmail string `stash:\"-\"`"), stashgen.Request{Type: "User"}, "", "stores no field"}, + {"an opaque struct with a tagged field", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Email", "opaque struct seals as one value"}, + {"two structs that would both write Encrypt", user(ctx+"\tEmail string `stash:\"email,encrypt\"`") + "\n//go:generate go tool stashgen -type Admin\ntype Admin struct {\n" + ctx + "\tEmail string `stash:\"email,encrypt\"`\n}\n", stashgen.Request{Type: "User"}, "", "User and Admin would both write Encrypt"}, + {"-redact on a type with a String method", user(ctx+"\tEmail string `stash:\"email,encrypt\"`") + "\nfunc (User) String() string { return \"\" }\n", stashgen.Request{Type: "User", Redact: true}, "", "already has a String method"}, + {"-for a field the struct names with another type", user(ctx + "\tEmail int `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User", For: "crm.Contact"}, "Email", "has type int, and crm.Contact.Email has type string"}, + {"-for a field the type does not have", user(ctx + "\tPhone string `stash:\"phone,encrypt\"`"), stashgen.Request{Type: "User", For: "crm.Contact"}, "Phone", "crm.Contact has no field Phone"}, + {"-for a field the struct does not name", user(ctx + "\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User", For: "crm.Contact"}, "ID", "not named by User"}, + {"-for with -redact", user(ctx + "\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User", For: "crm.Contact", Redact: true}, "", "cannot add print methods to crm.Contact"}, + {"-for an unknown package", user(ctx + "\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User", For: "nowhere.Contact"}, "", "imports no package named nowhere"}, + {"a model with a field that has no tag", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt,index=equality\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "Email", "no stash tag; each field of a model names one output"}, + {"a model with no field for an output", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt,index=equality\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext `stash:\"email\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "", `no field for the equality term of "email"`}, + {"a model field with the wrong type", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail string `stash:\"email\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "Email", "has type string, and the ciphertext of \"email\" is encrypt.Ciphertext"}, + {"a model field naming an index the field lacks", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext `stash:\"email\"`\n\tEq encrypt.EqualityTerm `stash:\"email,equality\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "Eq", `field "email" has no equality index`}, + {"a model for an opaque struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tEmail string") + "\ntype Row struct{}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "", "needs no model"}, + {"a type that is not a struct", "package users\n\ntype User int\n", stashgen.Request{Type: "User"}, "", "the generator reads a struct"}, + {"a type the package lacks", "package users\n", stashgen.Request{Type: "User"}, "", "has no type User"}, + {"a -name that is not exported", user(ctx + "\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User", Name: "user"}, "", "must be an exported Go name"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + _, err := generate(t, c.src, c.req) + if err == nil { + t.Fatalf("want an error containing %q, got none", c.want) + } + if !strings.Contains(err.Error(), c.want) { + t.Fatalf("error %q does not contain %q", err, c.want) + } + var fe *stashgen.FieldError + if c.field != "" { + if !errors.As(err, &fe) { + t.Fatalf("error %q is not a *FieldError", err) + } + if fe.Field != c.field { + t.Fatalf("error names field %q, want %q: %v", fe.Field, c.field, err) + } + } + }) + } +} + +// crmPackage stands in for a package the program does not own. +const crmPackage = "package crm\n\ntype Contact struct {\n\tID int64\n\tEmail string\n}\n" + +func TestForTypeInAnotherPackage(t *testing.T) { + // crm.Contact lives in its own package; the tagged struct names it. + dir := writeModule(t, map[string]string{ + "crm/contact.go": "package crm\n\ntype Contact struct {\n\tID int64\n\tEmail string\n\tnote string\n}\n", + "contacts/contacts.go": "package contacts\n\nimport \"example.com/app/crm\"\n\nvar _ crm.Contact\n\ntype contactStash struct {\n" + "\t_ struct{} `stash:\"context=contacts\"`\n\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`\n}\n", + }) + file, err := stashgen.FromTags(context.Background(), enginetest.Static{}, stashgen.Request{Dir: filepath.Join(dir, "contacts"), Type: "contactStash", For: "crm.Contact"}) + if err != nil { + t.Fatal(err) + } + src := string(file.Content) + for _, want := range []string{ + "Generated[*crm.Contact, EncryptedContact]", + "contacts []*crm.Contact", + "v := &crm.Contact{}", + "crm.Contact has unexported fields, so Go cannot convert it", + } { + if !strings.Contains(src, want) { + t.Errorf("generated file lacks %q", want) + } + } + if strings.Contains(src, "contactShape") { + t.Error("a type with unexported fields got a shape check") + } +} + +func TestStaleOutputFileIsIgnored(t *testing.T) { + src := user(ctx + "\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt_into=TextEq\"`") + dir := writeModule(t, map[string]string{"model.go": src}) + req := stashgen.Request{Dir: dir, Type: "User"} + first, err := stashgen.FromTags(context.Background(), enginetest.Static{}, req) + if err != nil { + t.Fatal(err) + } + // A stale file: it names a type that no longer exists and redeclares + // Encrypt, so the package does not type-check with it. + stale := "package users\n\nimport \"github.com/cipherstash/stack/languages/golang/encrypt/gensupport\"\n\nconst _ = gensupport.GeneratedVersion1\n\ntype EncryptedUser struct{ Gone Missing }\n\nfunc Encrypt() {}\n" + if err := os.WriteFile(filepath.Join(dir, "user_stash.go"), []byte(stale), 0o600); err != nil { + t.Fatal(err) + } + second, err := stashgen.FromTags(context.Background(), enginetest.Static{}, req) + if err != nil { + t.Fatalf("with a stale output file: %v", err) + } + if string(first.Content) != string(second.Content) { + t.Fatal("the stale output file changed the generated file") + } +} + +func TestOtherLibrariesTagsAreCopied(t *testing.T) { + src := user(ctx + "\tID int64 `db:\"id\" stash:\"id,passthrough\" gorm:\"primaryKey\"`\n\tEmail string `stash:\"email,encrypt_into=TextEq\" json:\"email,omitempty\"`") + file, err := generate(t, src, stashgen.Request{Type: "User"}) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{ + "ID int64 `db:\"id\" gorm:\"primaryKey\"`", + "Email eql.TextEq `json:\"email,omitempty\"`", + } { + if !strings.Contains(string(file.Content), want) { + t.Errorf("generated type lacks %q:\n%s", want, file.Content) + } + } +} + +func TestAStructWithPrintMethodsGetsNoNotice(t *testing.T) { + src := user(ctx+"\tEmail string `stash:\"email,encrypt\"`") + "\nfunc (User) String() string { return \"\" }\nfunc (User) LogValue() any { return nil }\n" + file, err := generate(t, src, stashgen.Request{Type: "User"}) + if err != nil { + t.Fatal(err) + } + if len(file.Notices) != 0 { + t.Fatalf("notices = %q", file.Notices) + } + if strings.Contains(string(file.Content), "PrintsPlaintext") { + t.Fatal("PrintsPlaintext set for a type with String and LogValue") + } +} diff --git a/languages/golang/stashgen/stub_test.go b/languages/golang/stashgen/stub_test.go new file mode 100644 index 000000000..7bc09d9c8 --- /dev/null +++ b/languages/golang/stashgen/stub_test.go @@ -0,0 +1,61 @@ +package stashgen_test + +import ( + "context" + "go/types" + "path/filepath" + "testing" + + "golang.org/x/tools/go/packages" +) + +// TestStubAgreesWithGensupport loads the real encrypt/gensupport and the +// test stub of it, and fails when a symbol both declare has two shapes. The +// stub is the contract the integration step implements; the real package is +// what ships. A symbol that is only in the stub is the integration step's +// work, and a symbol only in the real package is fine. +func TestStubAgreesWithGensupport(t *testing.T) { + real := loadScope(t, filepath.Join("..", "encrypt", "gensupport"), ".") + stub := loadScope(t, filepath.Join("testdata", "stubsdk"), "./encrypt/gensupport") + shared := 0 + for _, name := range real.Names() { + obj := real.Lookup(name) + if !obj.Exported() { + continue + } + stubObj := stub.Lookup(name) + if stubObj == nil { + t.Errorf("gensupport.%s is not in the stub; generated code may not name it", name) + continue + } + shared++ + got := types.TypeString(stubObj.Type(), stripPkg) + want := types.TypeString(obj.Type(), stripPkg) + if got != want { + t.Errorf("gensupport.%s: stub has %s, real package has %s", name, got, want) + } + } + if shared == 0 { + t.Fatal("no shared symbols: the loader found nothing") + } +} + +// stripPkg qualifies nothing, so the two generatedVersion types compare by +// name alone. +func stripPkg(*types.Package) string { return "" } + +func loadScope(t *testing.T, dir, pattern string) *types.Scope { + t.Helper() + cfg := &packages.Config{Context: context.Background(), Dir: dir, Mode: packages.NeedName | packages.NeedTypes} + pkgs, err := packages.Load(cfg, pattern) + if err != nil { + t.Fatal(err) + } + if len(pkgs) != 1 || pkgs[0].Types == nil { + t.Fatalf("loading %s in %s: %d packages", pattern, dir, len(pkgs)) + } + for _, e := range pkgs[0].Errors { + t.Fatalf("loading %s: %v", pattern, e) + } + return pkgs[0].Types.Scope() +} diff --git a/languages/golang/stashgen/testdata/cases/embedded/embedded.go b/languages/golang/stashgen/testdata/cases/embedded/embedded.go new file mode 100644 index 000000000..0a69d568c --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/embedded/embedded.go @@ -0,0 +1,18 @@ +// Package embedded shows an embedded struct of the same package: its tagged +// fields join the outer struct. +package embedded + +//go:generate go tool stashgen -type Patient + +type Person struct { + Name string `stash:"name,encrypt_into=TextEq" json:"name"` + Email string `stash:"email,encrypt,index=equality" json:"email"` + notes string +} + +type Patient struct { + _ struct{} `stash:"context=patients"` + Person + MRN string `stash:"mrn,passthrough" json:"mrn"` + Chart []byte `stash:"chart,encrypt"` +} diff --git a/languages/golang/stashgen/testdata/cases/embedded/go.mod b/languages/golang/stashgen/testdata/cases/embedded/go.mod new file mode 100644 index 000000000..455ad04ab --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/embedded/go.mod @@ -0,0 +1,12 @@ +module example.com/app + +go 1.26 + +require ( + github.com/cipherstash/stack/languages/golang v0.0.0 + gorm.io/gorm v0.0.0 +) + +replace github.com/cipherstash/stack/languages/golang => ../../stubsdk + +replace gorm.io/gorm => ../../stubgorm diff --git a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden new file mode 100644 index 000000000..a63fe9fbd --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden @@ -0,0 +1,175 @@ +// Code generated by stashgen. DO NOT EDIT. + +package embedded + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Patient prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +// Not encrypted and not stored: the unexported field "notes". Tag it +// `stash:"-"` to confirm that. + +type EncryptedPatient struct { + Name eql.TextEq `json:"name"` + Email EncryptedPatientEmail `json:"email"` + MRN string `json:"mrn"` + Chart EncryptedPatientChart +} + +type EncryptedPatientEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +type EncryptedPatientChart struct { + Ciphertext encrypt.Ciphertext +} + +func (e EncryptedPatient) String() string { + return gensupport.Redacted("EncryptedPatient", map[string]any{"MRN": e.MRN}, "Name", "Email", "Chart") +} + +func (e EncryptedPatient) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"MRN": e.MRN}, "Name", "Email", "Chart") +} + +// Stops compiling when Patient gains, loses, reorders or retypes a field. +var _ = patientShape(Patient{}) + +type patientShape struct { + _ struct{} + Person + MRN string + Chart []byte +} + +var declaration = gensupport.Declare("patients"). + EncryptInto("name", "TextEq"). + EncryptIndex("email", encrypt.Equality). + Passthrough("mrn"). + Encrypt("chart") + +var codec = gensupport.New(gensupport.Generated[Patient, EncryptedPatient]{ + TypeName: "Patient", + Declaration: declaration, + PrintsPlaintext: true, + Unexported: []string{"notes"}, + Source: func(v Patient) gensupport.Values { + return gensupport.Values{ + "name": v.Name, + "email": v.Email, + "mrn": v.MRN, + "chart": v.Chart, + } + }, + Seal: func(rec gensupport.Record) (EncryptedPatient, error) { + var e EncryptedPatient + var err error + if e.MRN, err = gensupport.Passthrough[string](rec, "mrn"); err != nil { + return EncryptedPatient{}, err + } + e.Name = eql.TextEq(rec["name"].EQL) + e.Email = EncryptedPatientEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + } + e.Chart = EncryptedPatientChart{Ciphertext: rec["chart"].Ciphertext} + return e, nil + }, + Open: func(e EncryptedPatient) gensupport.Record { + return gensupport.Record{ + "name": {EQL: e.Name}, + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality}, + "mrn": {Value: e.MRN}, + "chart": {Ciphertext: e.Chart.Ciphertext}, + } + }, + Value: func(e EncryptedPatient, vals gensupport.Values) (Patient, error) { + var v Patient + var err error + if v.Name, err = gensupport.Get[string](vals, "name"); err != nil { + return Patient{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return Patient{}, err + } + if v.MRN, err = gensupport.Get[string](vals, "mrn"); err != nil { + return Patient{}, err + } + if v.Chart, err = gensupport.Get[[]byte](vals, "chart"); err != nil { + return Patient{}, err + } + return v, nil + }, +}) + +// Encrypt seals each Patient in one ZeroKMS request. The result has one element +// for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, embedded []Patient) ([]EncryptedPatient, error) { + return codec.Encrypt(ctx, cipher, embedded) +} + +// Decrypt opens each EncryptedPatient in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedPatient) ([]Patient, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Name NameField + Email EmailField + Chart ChartField +}{ + Name: NameField{gensupport.NewField[string](declaration, "name")}, + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + Chart: ChartField{gensupport.NewField[[]byte](declaration, "chart")}, +} + +type NameField struct { + field gensupport.Field[string] +} + +func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f NameField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedPatientEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedPatientEmail{}, err + } + return EncryptedPatientEmail{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +type ChartField struct { + field gensupport.Field[[]byte] +} + +func (f ChartField) Encrypt(ctx context.Context, c *encrypt.Cipher, v []byte) (EncryptedPatientChart, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedPatientChart{Ciphertext: out.Ciphertext}, err +} diff --git a/languages/golang/stashgen/testdata/cases/foreign/go.mod b/languages/golang/stashgen/testdata/cases/foreign/go.mod new file mode 100644 index 000000000..455ad04ab --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/foreign/go.mod @@ -0,0 +1,12 @@ +module example.com/app + +go 1.26 + +require ( + github.com/cipherstash/stack/languages/golang v0.0.0 + gorm.io/gorm v0.0.0 +) + +replace github.com/cipherstash/stack/languages/golang => ../../stubsdk + +replace gorm.io/gorm => ../../stubgorm diff --git a/languages/golang/stashgen/testdata/cases/foreign/individuals.go b/languages/golang/stashgen/testdata/cases/foreign/individuals.go new file mode 100644 index 000000000..636097082 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/foreign/individuals.go @@ -0,0 +1,19 @@ +// Package individuals declares the tags for pb.Individual, a type with +// unexported fields: the generated file reads each field by name and the +// functions take pointers. +package individuals + +import "example.com/app/pb" + +//go:generate go tool stashgen -type individualStash -for pb.Individual + +type individualStash struct { + _ struct{} `stash:"context=individuals"` + Id int64 `stash:"id,passthrough"` + Name string `stash:"name,encrypt"` + Email string `stash:"email,encrypt,index=equality;match"` + MedicareNo string `stash:"medicare_number,encrypt_into=TextEq"` + Nickname string `stash:"nickname,passthrough"` +} + +var _ = pb.Individual{} diff --git a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden new file mode 100644 index 000000000..1fb87876b --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden @@ -0,0 +1,182 @@ +// Code generated by stashgen. DO NOT EDIT. + +package individuals + +import ( + "context" + "log/slog" + + "example.com/app/pb" + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// pb.Individual prints its sealed fields in the clear, and stashgen cannot add +// print methods to a type from another package. + +// pb.Individual has unexported fields, so Go cannot convert it to a copy of its +// fields. This file reads each field by name: the compiler finds a removed or +// retyped field, and CI finds an added one. + +type EncryptedIndividual struct { + Id int64 + Name EncryptedIndividualName + Email EncryptedIndividualEmail + MedicareNo eql.TextEq + Nickname string +} + +type EncryptedIndividualName struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedIndividualEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} + +func (e EncryptedIndividual) String() string { + return gensupport.Redacted("EncryptedIndividual", map[string]any{"Id": e.Id, "Nickname": e.Nickname}, "Name", "Email", "MedicareNo") +} + +func (e EncryptedIndividual) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"Id": e.Id, "Nickname": e.Nickname}, "Name", "Email", "MedicareNo") +} + +var declaration = gensupport.Declare("individuals"). + Passthrough("id"). + Encrypt("name"). + EncryptIndex("email", encrypt.Equality, encrypt.Match()). + EncryptInto("medicare_number", "TextEq"). + Passthrough("nickname") + +var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndividual]{ + TypeName: "pb.Individual", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v *pb.Individual) gensupport.Values { + return gensupport.Values{ + "id": v.Id, + "name": v.Name, + "email": v.Email, + "medicare_number": v.MedicareNo, + "nickname": v.Nickname, + } + }, + Seal: func(rec gensupport.Record) (EncryptedIndividual, error) { + var e EncryptedIndividual + var err error + if e.Id, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedIndividual{}, err + } + if e.Nickname, err = gensupport.Passthrough[string](rec, "nickname"); err != nil { + return EncryptedIndividual{}, err + } + e.Name = EncryptedIndividualName{Ciphertext: rec["name"].Ciphertext} + e.Email = EncryptedIndividualEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + Match: rec["email"].Match, + } + e.MedicareNo = eql.TextEq(rec["medicare_number"].EQL) + return e, nil + }, + Open: func(e EncryptedIndividual) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.Id}, + "name": {Ciphertext: e.Name.Ciphertext}, + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality, Match: e.Email.Match}, + "medicare_number": {EQL: e.MedicareNo}, + "nickname": {Value: e.Nickname}, + } + }, + Value: func(e EncryptedIndividual, vals gensupport.Values) (*pb.Individual, error) { + v := &pb.Individual{} + var err error + if v.Id, err = gensupport.Get[int64](vals, "id"); err != nil { + return nil, err + } + if v.Name, err = gensupport.Get[string](vals, "name"); err != nil { + return nil, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return nil, err + } + if v.MedicareNo, err = gensupport.Get[string](vals, "medicare_number"); err != nil { + return nil, err + } + if v.Nickname, err = gensupport.Get[string](vals, "nickname"); err != nil { + return nil, err + } + return v, nil + }, +}) + +// Encrypt seals each pb.Individual in one ZeroKMS request. The result has one +// element for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, individuals []*pb.Individual) ([]EncryptedIndividual, error) { + return codec.Encrypt(ctx, cipher, individuals) +} + +// Decrypt opens each EncryptedIndividual in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedIndividual) ([]*pb.Individual, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Name NameField + Email EmailField + MedicareNo MedicareNoField +}{ + Name: NameField{gensupport.NewField[string](declaration, "name")}, + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + MedicareNo: MedicareNoField{gensupport.NewField[string](declaration, "medicare_number")}, +} + +type NameField struct { + field gensupport.Field[string] +} + +func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedIndividualName, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedIndividualName{Ciphertext: out.Ciphertext}, err +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedIndividualEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedIndividualEmail{}, err + } + return EncryptedIndividualEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil +} + +func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +type MedicareNoField struct { + field gensupport.Field[string] +} + +func (f MedicareNoField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f MedicareNoField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} diff --git a/languages/golang/stashgen/testdata/cases/foreign/pb/individual.go b/languages/golang/stashgen/testdata/cases/foreign/pb/individual.go new file mode 100644 index 000000000..d7e451335 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/foreign/pb/individual.go @@ -0,0 +1,15 @@ +// Package pb stands in for protoc-gen-go output: exported fields beside +// unexported ones, so the struct cannot convert to a copy of its fields. +package pb + +type state struct{ _ [0]func() } + +type Individual struct { + state state + sizeCache int32 + Id int64 `protobuf:"varint,1,opt,name=id,proto3" json:"id,omitempty"` + Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` + Email string `protobuf:"bytes,3,opt,name=email,proto3" json:"email,omitempty"` + MedicareNo string `protobuf:"bytes,4,opt,name=medicare_no,json=medicareNo,proto3" json:"medicare_no,omitempty"` + Nickname string `protobuf:"bytes,5,opt,name=nickname,proto3" json:"nickname,omitempty"` +} diff --git a/languages/golang/stashgen/testdata/cases/orders/go.mod b/languages/golang/stashgen/testdata/cases/orders/go.mod new file mode 100644 index 000000000..455ad04ab --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/orders/go.mod @@ -0,0 +1,12 @@ +module example.com/app + +go 1.26 + +require ( + github.com/cipherstash/stack/languages/golang v0.0.0 + gorm.io/gorm v0.0.0 +) + +replace github.com/cipherstash/stack/languages/golang => ../../stubsdk + +replace gorm.io/gorm => ../../stubgorm diff --git a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden new file mode 100644 index 000000000..57893c933 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden @@ -0,0 +1,193 @@ +// Code generated by stashgen. DO NOT EDIT. + +package orders + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Order prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedOrder struct { + ID int64 `db:"id"` + Amount EncryptedOrderAmount `db:"amount"` + Note EncryptedOrderNote + Customer EncryptedOrderCustomer +} + +type EncryptedOrderAmount struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Ore encrypt.OreTerm +} + +type EncryptedOrderNote struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedOrderCustomer struct { + Ciphertext encrypt.Ciphertext + Match encrypt.MatchTerm + Ope encrypt.OpeTerm +} + +func (e EncryptedOrder) String() string { + return gensupport.Redacted("EncryptedOrder", map[string]any{"ID": e.ID}, "Amount", "Note", "Customer") +} + +func (e EncryptedOrder) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Amount", "Note", "Customer") +} + +// Stops compiling when Order gains, loses, reorders or retypes a field. +var _ = orderShape(Order{}) + +type orderShape struct { + _ struct{} + ID int64 + Amount int64 + Note string + Customer string +} + +var orderDeclaration = gensupport.Declare("orders"). + Passthrough("id"). + EncryptIndex("amount", encrypt.Equality, encrypt.Ore). + Encrypt("note"). + EncryptIndex("customer", encrypt.Match(), encrypt.Ope) + +var orderCodec = gensupport.New(gensupport.Generated[Order, EncryptedOrder]{ + TypeName: "Order", + Declaration: orderDeclaration, + PrintsPlaintext: true, + Source: func(v Order) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "amount": v.Amount, + "note": v.Note, + "customer": v.Customer, + } + }, + Seal: func(rec gensupport.Record) (EncryptedOrder, error) { + var e EncryptedOrder + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedOrder{}, err + } + e.Amount = EncryptedOrderAmount{ + Ciphertext: rec["amount"].Ciphertext, + Equality: rec["amount"].Equality, + Ore: rec["amount"].Ore, + } + e.Note = EncryptedOrderNote{Ciphertext: rec["note"].Ciphertext} + e.Customer = EncryptedOrderCustomer{ + Ciphertext: rec["customer"].Ciphertext, + Match: rec["customer"].Match, + Ope: rec["customer"].Ope, + } + return e, nil + }, + Open: func(e EncryptedOrder) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "amount": {Ciphertext: e.Amount.Ciphertext, Equality: e.Amount.Equality, Ore: e.Amount.Ore}, + "note": {Ciphertext: e.Note.Ciphertext}, + "customer": {Ciphertext: e.Customer.Ciphertext, Match: e.Customer.Match, Ope: e.Customer.Ope}, + } + }, + Value: func(e EncryptedOrder, vals gensupport.Values) (Order, error) { + var v Order + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return Order{}, err + } + if v.Amount, err = gensupport.Get[int64](vals, "amount"); err != nil { + return Order{}, err + } + if v.Note, err = gensupport.Get[string](vals, "note"); err != nil { + return Order{}, err + } + if v.Customer, err = gensupport.Get[string](vals, "customer"); err != nil { + return Order{}, err + } + return v, nil + }, +}) + +// EncryptOrder seals each Order in one ZeroKMS request. The result has one +// element for each input, in the same order. +func EncryptOrder(ctx context.Context, cipher *encrypt.Cipher, values []Order) ([]EncryptedOrder, error) { + return orderCodec.Encrypt(ctx, cipher, values) +} + +// DecryptOrder opens each EncryptedOrder in one ZeroKMS request. +func DecryptOrder(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedOrder) ([]Order, error) { + return orderCodec.Decrypt(ctx, d, encrypted) +} + +var OrderFields = struct { + Amount OrderAmountField + Note OrderNoteField + Customer OrderCustomerField +}{ + Amount: OrderAmountField{gensupport.NewField[int64](orderDeclaration, "amount")}, + Note: OrderNoteField{gensupport.NewField[string](orderDeclaration, "note")}, + Customer: OrderCustomerField{gensupport.NewField[string](orderDeclaration, "customer")}, +} + +type OrderAmountField struct { + field gensupport.Field[int64] +} + +func (f OrderAmountField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int64) (EncryptedOrderAmount, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedOrderAmount{}, err + } + return EncryptedOrderAmount{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil +} + +func (f OrderAmountField) Equality(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f OrderAmountField) Ore(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type OrderNoteField struct { + field gensupport.Field[string] +} + +func (f OrderNoteField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedOrderNote, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedOrderNote{Ciphertext: out.Ciphertext}, err +} + +type OrderCustomerField struct { + field gensupport.Field[string] +} + +func (f OrderCustomerField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedOrderCustomer, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedOrderCustomer{}, err + } + return EncryptedOrderCustomer{Ciphertext: out.Ciphertext, Match: out.Match, Ope: out.Ope}, nil +} + +func (f OrderCustomerField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +func (f OrderCustomerField) Ope(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} diff --git a/languages/golang/stashgen/testdata/cases/orders/orders.go b/languages/golang/stashgen/testdata/cases/orders/orders.go new file mode 100644 index 000000000..9be74ba0c --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/orders/orders.go @@ -0,0 +1,26 @@ +// Package orders holds two tagged structs. Order carries -name, so the +// package has one Encrypt (for Refund) and one EncryptOrder. +package orders + +//go:generate go tool stashgen -type Order -name Order +//go:generate go tool stashgen -type Refund + +// Order uses separate columns: an ordered amount, and a note with no index. +type Order struct { + _ struct{} `stash:"context=orders"` + ID int64 `stash:"id,passthrough" db:"id"` + Amount int64 `stash:"amount,encrypt,index=equality;ore" db:"amount"` + Note string `stash:"note,encrypt"` + Customer string `stash:"customer,encrypt,index=match;ope"` +} + +// Refund has String and LogValue, so stashgen does not warn about printing. +type Refund struct { + _ struct{} `stash:"context=refunds"` + ID int64 `stash:"id,passthrough"` + Reason string `stash:"reason,encrypt_into=TextEq"` +} + +func (Refund) String() string { return "Refund{...}" } + +func (Refund) LogValue() any { return nil } diff --git a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden new file mode 100644 index 000000000..4f3c53b71 --- /dev/null +++ b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden @@ -0,0 +1,109 @@ +// Code generated by stashgen. DO NOT EDIT. + +package orders + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +type EncryptedRefund struct { + ID int64 + Reason eql.TextEq +} + +func (e EncryptedRefund) String() string { + return gensupport.Redacted("EncryptedRefund", map[string]any{"ID": e.ID}, "Reason") +} + +func (e EncryptedRefund) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Reason") +} + +// Stops compiling when Refund gains, loses, reorders or retypes a field. +var _ = refundShape(Refund{}) + +type refundShape struct { + _ struct{} + ID int64 + Reason string +} + +var declaration = gensupport.Declare("refunds"). + Passthrough("id"). + EncryptInto("reason", "TextEq") + +var codec = gensupport.New(gensupport.Generated[Refund, EncryptedRefund]{ + TypeName: "Refund", + Declaration: declaration, + Source: func(v Refund) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "reason": v.Reason, + } + }, + Seal: func(rec gensupport.Record) (EncryptedRefund, error) { + var e EncryptedRefund + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedRefund{}, err + } + e.Reason = eql.TextEq(rec["reason"].EQL) + return e, nil + }, + Open: func(e EncryptedRefund) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "reason": {EQL: e.Reason}, + } + }, + Value: func(e EncryptedRefund, vals gensupport.Values) (Refund, error) { + var v Refund + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return Refund{}, err + } + if v.Reason, err = gensupport.Get[string](vals, "reason"); err != nil { + return Refund{}, err + } + return v, nil + }, +}) + +// Encrypt seals each Refund in one ZeroKMS request. The result has one element +// for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, orders []Refund) ([]EncryptedRefund, error) { + return codec.Encrypt(ctx, cipher, orders) +} + +// Decrypt opens each EncryptedRefund in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedRefund) ([]Refund, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Reason ReasonField +}{ + Reason: ReasonField{gensupport.NewField[string](declaration, "reason")}, +} + +type ReasonField struct { + field gensupport.Field[string] +} + +func (f ReasonField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f ReasonField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go index 6279a9a2b..0eeaf18e8 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go @@ -119,3 +119,8 @@ func (*RecordsCodec[P, R]) Encrypt(context.Context, *encrypt.Cipher, []P) ([]R, func (*RecordsCodec[P, R]) Decrypt(context.Context, encrypt.Decrypter, []R) ([]P, error) { return nil, nil } + +// NoticeUntagged and NoticePrintsPlaintext are the running program's +// notices; the real New calls them once for each type. +func NoticeUntagged(typeName string, fields []string) {} +func NoticePrintsPlaintext(typeName string) {} From 579eb8d900cc4445892fea1bd5b0f36f4eb6be4e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 00:40:29 -0700 Subject: [PATCH 03/30] feat(golang): declarations from a policy, and protosource Add encrypt/policy, stashgen.Generate and encrypt/policy/protosource: the path for a type that cannot carry stash tags, such as a protobuf message whose field options hold its data categories. A Source gives the facts about each field, the first rule that matches a field decides it, and Generate writes the same file the tag path writes, into a package of the user's own, with pointer-taking functions. A decision spells itself in the tag grammar (Outcome.Tag returns what the field's stash tag would say) and Generate parses it with the tag parser, so the two ways in share one grammar, one reader and one emitter, which is what SDK principle 6 asks of a second way in. That also keeps policy free of an import of stashgen, which Generate needs to import; the index values are policy.Equality, policy.Match() and friends rather than the plan's encrypt.Equality, because package encrypt does not exist in this tree and policy must not import a generator package. Identity joins the declaration as a field a policy alone sets; the tag grammar has no word for it. protosource reads the descriptor through protoreflect, records each extension set on a field's options under the extension's full name, and derives the Go field name the way protoc-gen-go does; Generate matches facts to struct fields through the protobuf tag's name= first and the Go name second. Its fixture is the plan's protoc-gen-go output, renamed. This adds google.golang.org/protobuf to the module. The policy test asserts that the policy path and the tag path write the same body for the same declaration, so the one emitter stays one. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/cmd/stashgen/README.md | 33 +- languages/golang/encrypt/policy/policy.go | 353 ++++++++++++++++++ .../golang/encrypt/policy/policy_test.go | 122 ++++++ .../internal/testpb/classification.pb.go | 94 +++++ .../policy/protosource/internal/testpb/doc.go | 6 + .../internal/testpb/individual.pb.go | 188 ++++++++++ .../encrypt/policy/protosource/protosource.go | 108 ++++++ .../policy/protosource/protosource_test.go | 71 ++++ languages/golang/go.mod | 1 + languages/golang/go.sum | 6 +- languages/golang/stashgen/declaration.go | 4 + languages/golang/stashgen/doc.go | 5 + languages/golang/stashgen/emit.go | 9 +- languages/golang/stashgen/export_test.go | 18 + languages/golang/stashgen/model_test.go | 112 ++++++ languages/golang/stashgen/policy.go | 236 ++++++++++++ languages/golang/stashgen/policy_test.go | 188 ++++++++++ languages/golang/stashgen/read.go | 110 ++++-- languages/golang/stashgen/tag.go | 2 + .../policy_individual_stash.go.golden | 182 +++++++++ .../stubsdk/encrypt/gensupport/gensupport.go | 3 + 21 files changed, 1807 insertions(+), 44 deletions(-) create mode 100644 languages/golang/encrypt/policy/policy.go create mode 100644 languages/golang/encrypt/policy/policy_test.go create mode 100644 languages/golang/encrypt/policy/protosource/internal/testpb/classification.pb.go create mode 100644 languages/golang/encrypt/policy/protosource/internal/testpb/doc.go create mode 100644 languages/golang/encrypt/policy/protosource/internal/testpb/individual.pb.go create mode 100644 languages/golang/encrypt/policy/protosource/protosource.go create mode 100644 languages/golang/encrypt/policy/protosource/protosource_test.go create mode 100644 languages/golang/stashgen/export_test.go create mode 100644 languages/golang/stashgen/model_test.go create mode 100644 languages/golang/stashgen/policy.go create mode 100644 languages/golang/stashgen/policy_test.go create mode 100644 languages/golang/stashgen/testdata/policy_individual_stash.go.golden diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 122f8e6d6..62a7d2380 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -138,6 +138,37 @@ The error names the type and the field, and never a value. - an embedded struct from another package with no tag; - a struct whose every field is left out. +## Declarations from a policy + +A type that a schema generates, such as a protobuf message, cannot carry tags. +For such a type, rules decide how each field is encrypted from what the schema says about it, and `stashgen.Generate` writes the same file from the rules. +The rules live in `github.com/cipherstash/stack/languages/golang/encrypt/policy`; `encrypt/policy/protosource` reads a protobuf message's fields and their options. + +```go +var category = policy.Key("classification.data_categories") + +var Individuals = policy.ForMessage(&pb.Individual{}, policy.Context("individuals"), + policy.FirstOf( + policy.When(policy.Field("id"), policy.Passthrough()), + policy.When(category.Under("user.government_id"), policy.EncryptInto("TextEq")), + policy.When(category.Under("user.contact.email"), policy.EncryptIndex(policy.Equality, policy.Match())), + policy.When(category.Under("user"), policy.Encrypt()), + ), +) + +//go:generate go run ../cmd/genencrypt +func main() { + err := stashgen.Generate(protosource.New(), rules.Individuals, + stashgen.Output("../individuals/individual_stash.go")) + ... +} +``` + +The first rule that matches a field decides it. +Every field needs a decision: a field no rule decides stops the generator with the field's name and its annotations. +`Name` sets the column name, and `Identity` keeps the field's context when its column is renamed. +The generated file goes in a package of your own, and the functions take and return pointers to the message. + ## Printing A generated type hides its sealed fields when a program prints or logs it. @@ -149,4 +180,4 @@ No warning, error or log line holds a plaintext value. The generator is built and tested against a static stand-in for the engine. `stashgen.GuestEngine`, which runs the WASI guest the SDK embeds, is not wired yet, so the command stops with `ErrEngineUnavailable` until it is. -The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`. +The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`; `stashgen.Generate` takes one with `WithEngine`. diff --git a/languages/golang/encrypt/policy/policy.go b/languages/golang/encrypt/policy/policy.go new file mode 100644 index 000000000..d144bcf71 --- /dev/null +++ b/languages/golang/encrypt/policy/policy.go @@ -0,0 +1,353 @@ +// Package policy decides how a type that cannot carry stash tags is +// encrypted, from what its schema says about each field. +// +// Tags are the way to declare a type that you write. A policy is for a type +// that a schema generates, such as a protobuf message whose field options hold +// the field's data categories. A [Source] gives the facts about each field; a +// rule has a [Matcher] and a [Decision]; the first rule that matches a field +// decides it. stashgen.Generate runs the rules and writes the same generated +// file that it writes from tags. Only the generate program runs a policy. +// +// Every field of the message needs a decision. A field that no rule decides +// stops the generator with the field's name and its annotations. A policy has +// an [Otherwise] rule only when its author writes one. +package policy + +import ( + "fmt" + "strings" +) + +// Fact is what a source knows about one field: its name, its Go name, its +// kind and its annotations. A policy reads facts and never learns where they +// came from. +type Fact struct { + // Message is the full name of the message, for errors. + Message string + // Name is the field's name in the schema, such as the proto field name. + // It is the column name unless a rule gives another with [Name]. + Name string + // GoName is the field of the Go struct the schema generated. + GoName string + // Kind is the field's kind in the schema's own words, such as "string" or + // "int64". Rules may match on it. + Kind string + // Annotations are what the schema says about the field, such as its + // data categories. + Annotations []Annotation +} + +// Annotation is one annotation on a field: a key and its values. For a +// protobuf field option the key is the option's full name. +type Annotation struct { + Key string + Values []string +} + +// Values returns every value under key, in order. +func (f Fact) Values(key string) []string { + var out []string + for _, a := range f.Annotations { + if a.Key == key { + out = append(out, a.Values...) + } + } + return out +} + +// String spells the field and its annotations, for the error that names a +// field no rule decides. +func (f Fact) String() string { + var b strings.Builder + fmt.Fprintf(&b, "%s.%s (%s", f.Message, f.Name, f.Kind) + for _, a := range f.Annotations { + fmt.Fprintf(&b, "; %s = %s", a.Key, strings.Join(a.Values, ", ")) + } + b.WriteString(")") + return b.String() +} + +// Source gives the facts about each field of a message. +type Source interface { + // Facts returns one fact for each field of the message, in declared + // order. The message is the value given to [ForMessage]. + Facts(message any) ([]Fact, error) +} + +// SourceFunc is a Source made from a function. +type SourceFunc func(message any) ([]Fact, error) + +// Facts calls the function. +func (s SourceFunc) Facts(message any) ([]Fact, error) { return s(message) } + +// Matcher selects the fields a rule applies to. +type Matcher interface { + Match(Fact) bool +} + +// MatcherFunc is a Matcher made from a function. +type MatcherFunc func(Fact) bool + +// Match calls the function. +func (m MatcherFunc) Match(f Fact) bool { return m(f) } + +// Field matches the field with this schema name. +func Field(name string) Matcher { + return MatcherFunc(func(f Fact) bool { return f.Name == name }) +} + +// Key is an annotation key, such as the full name of a protobuf field option. +type Key string + +// Under matches a field with a value under key that is prefix, or starts +// with prefix and a dot: Key("c").Under("user") matches "user" and +// "user.contact.email", and not "username". +func (k Key) Under(prefix string) Matcher { + return MatcherFunc(func(f Fact) bool { + for _, v := range f.Values(string(k)) { + if v == prefix || strings.HasPrefix(v, prefix+".") { + return true + } + } + return false + }) +} + +// Is matches a field with a value under key equal to value. +func (k Key) Is(value string) Matcher { + return MatcherFunc(func(f Fact) bool { + for _, v := range f.Values(string(k)) { + if v == value { + return true + } + } + return false + }) +} + +// Decision is what a rule decides for a field: one of the tag verbs, or a +// refusal. A decision spells itself in the tag grammar, so the generator +// reads a policy's decision through the same parser as a tag. +type Decision struct { + verb string + indexes []IndexSpec + eqlType string + fail bool + reason string +} + +// Encrypt seals the field with no index. +func Encrypt() Decision { return Decision{verb: "encrypt"} } + +// EncryptIndex seals the field and derives each index beside it. +func EncryptIndex(indexes ...IndexSpec) Decision { + return Decision{verb: "encrypt,index", indexes: indexes} +} + +// Index derives the indexes alone, with no ciphertext. +func Index(indexes ...IndexSpec) Decision { return Decision{verb: "index", indexes: indexes} } + +// EncryptInto seals the field into one EQL value of this type. +func EncryptInto(eqlType string) Decision { return Decision{verb: "encrypt_into", eqlType: eqlType} } + +// Passthrough stores the field as it is. +func Passthrough() Decision { return Decision{verb: "passthrough"} } + +// Omit leaves the field out. +func Omit() Decision { return Decision{verb: "-"} } + +// Fail refuses the field: the generator stops with the reason. +func Fail(reason string) Decision { return Decision{fail: true, reason: reason} } + +// IndexName is one of the tag words for an index: equality, match, ore, ope +// or json. +type IndexName string + +// IndexOption is one option of an index, as the tag writes it in parentheses. +type IndexOption struct { + Key string + Value string +} + +// IndexSpec is one index with its options, as a rule names it: the engine's +// own word for the data form of an index. +type IndexSpec struct { + Name IndexName + Options []IndexOption +} + +// The indexes that take no options. +var ( + Equality = IndexSpec{Name: "equality"} + Ore = IndexSpec{Name: "ore"} + Ope = IndexSpec{Name: "ope"} +) + +// Match is the match index, with its options. +func Match(options ...IndexOption) IndexSpec { return IndexSpec{Name: "match", Options: options} } + +// JSON is the json index, with its options. +func JSON(options ...IndexOption) IndexSpec { return IndexSpec{Name: "json", Options: options} } + +// String spells the index as a tag does: match or match(k=3). +func (i IndexSpec) String() string { + if len(i.Options) == 0 { + return string(i.Name) + } + opts := make([]string, len(i.Options)) + for n, o := range i.Options { + opts[n] = o.Key + if o.Value != "" { + opts[n] += "=" + o.Value + } + } + return string(i.Name) + "(" + strings.Join(opts, ",") + ")" +} + +// Outcome is a decision applied to one field: the decision, and the name +// and identity a rule's options set. +type Outcome struct { + Decision Decision + // Name is the column name, or "" for the field's schema name. + Name string + // Identity is the field's part of its context, or "" for the column + // name. A field whose column is renamed keeps its identity, so data + // written before the change still decrypts. + Identity string +} + +// Tag spells the outcome for a field in the tag grammar: what the field's +// stash tag would say. ok is false for a refusal, which [Outcome.Reason] +// explains, and for the zero Outcome. +func (o Outcome) Tag(name string) (tag string, ok bool) { + d := o.Decision + if d.fail || d.verb == "" { + return "", false + } + if d.verb == "-" { + return "-", true + } + if o.Name != "" { + name = o.Name + } + switch d.verb { + case "encrypt_into": + return name + ",encrypt_into=" + d.eqlType, true + case "encrypt,index", "index": + names := make([]string, len(d.indexes)) + for i, idx := range d.indexes { + names[i] = idx.String() + } + return name + "," + d.verb + "=" + strings.Join(names, ";"), true + } + return name + "," + d.verb, true +} + +// Reason is the reason of a Fail decision, or "". +func (o Outcome) Reason() string { + if !o.Decision.fail { + return "" + } + return o.Decision.reason +} + +// Option modifies a rule's outcome. +type Option func(*Outcome) + +// Name sets the field's column name. +func Name(column string) Option { return func(o *Outcome) { o.Name = column } } + +// Identity keeps the field's context when its column gets a new name. +func Identity(identity string) Option { return func(o *Outcome) { o.Identity = identity } } + +// Rule decides a field, or reports that it does not match. +type Rule interface { + Decide(Fact) (Outcome, bool) +} + +// When is a rule: the fields the matcher selects get the decision. +func When(m Matcher, d Decision, options ...Option) Rule { + return rule{m: m, outcome: outcome(d, options)} +} + +// Otherwise decides every field that no earlier rule matched. +func Otherwise(d Decision, options ...Option) Rule { + return rule{m: MatcherFunc(func(Fact) bool { return true }), outcome: outcome(d, options)} +} + +func outcome(d Decision, options []Option) Outcome { + o := Outcome{Decision: d} + for _, opt := range options { + opt(&o) + } + return o +} + +type rule struct { + m Matcher + outcome Outcome +} + +func (r rule) Decide(f Fact) (Outcome, bool) { + if !r.m.Match(f) { + return Outcome{}, false + } + return r.outcome, true +} + +// Rules are rules tried in order; the first that matches decides. +type Rules []Rule + +// FirstOf returns the rules, tried in order. +func FirstOf(rules ...Rule) Rules { return Rules(rules) } + +// Decide tries each rule in order. +func (r Rules) Decide(f Fact) (Outcome, bool) { + for _, rule := range r { + if rule == nil { + continue + } + if o, ok := rule.Decide(f); ok { + return o, true + } + } + return Outcome{}, false +} + +// OrElse returns the rules, then other for the fields they do not match. +func (r Rules) OrElse(other Rule) Rules { + return append(append(Rules{}, r...), other) +} + +// Context is the context of every field of a message, as the `context=` tag +// gives it for a struct. +type Context string + +// Message is one message with its context and its rules, as [ForMessage] +// builds it and stashgen.Generate reads it. +type Message struct { + message any + context Context + rules Rule +} + +// ForMessage binds a message to its context and the rules that decide its +// fields. The message is a value of the generated Go type, such as +// &pb.Individual{}; the source reads its schema and the generator its type. +func ForMessage(message any, context Context, rules Rule) Message { + return Message{message: message, context: context, rules: rules} +} + +// Message returns the message value given to ForMessage. +func (m Message) Message() any { return m.message } + +// Context returns the message's context. +func (m Message) Context() string { return string(m.context) } + +// Decide runs the rules on one field. +func (m Message) Decide(f Fact) (Outcome, bool) { + if m.rules == nil { + return Outcome{}, false + } + return m.rules.Decide(f) +} diff --git a/languages/golang/encrypt/policy/policy_test.go b/languages/golang/encrypt/policy/policy_test.go new file mode 100644 index 000000000..cc59525b9 --- /dev/null +++ b/languages/golang/encrypt/policy/policy_test.go @@ -0,0 +1,122 @@ +package policy + +import ( + "strings" + "testing" +) + +var category = Key("classification.data_categories") + +func fact(name, kind string, categories ...string) Fact { + f := Fact{Message: "individuals.Individual", Name: name, GoName: strings.ToUpper(name[:1]) + name[1:], Kind: kind} + if len(categories) > 0 { + f.Annotations = []Annotation{{Key: string(category), Values: categories}} + } + return f +} + +var base = FirstOf( + When(category.Under("user.government_id"), EncryptInto("TextEq")), + When(category.Under("user.contact.email"), EncryptIndex(Equality, Match())), + When(category.Under("user"), Encrypt()), +) + +var individuals = ForMessage(nil, Context("individuals"), + FirstOf( + When(Field("medicare_no"), EncryptInto("TextEq"), Name("medicare_number")), + When(Field("id"), Passthrough()), + When(Field("nickname"), Passthrough()), + ).OrElse(base), +) + +func TestTheFirstMatchingRuleDecides(t *testing.T) { + cases := []struct { + f Fact + want string // the field's tag + }{ + {fact("id", "int64"), "id,passthrough"}, + {fact("name", "string", "user.name"), "name,encrypt"}, + {fact("email", "string", "user.contact.email"), "email,encrypt,index=equality;match"}, + {fact("medicare_no", "string", "user.government_id"), "medicare_number,encrypt_into=TextEq"}, + {fact("nickname", "string"), "nickname,passthrough"}, + } + for _, c := range cases { + o, ok := individuals.Decide(c.f) + if !ok { + t.Errorf("%s: no rule decides it", c.f) + continue + } + got, ok := o.Tag(c.f.Name) + if !ok { + t.Errorf("%s: refused: %s", c.f, o.Reason()) + continue + } + if got != c.want { + t.Errorf("%s: %q, want %q", c.f, got, c.want) + } + } + if individuals.Context() != "individuals" { + t.Errorf("Context = %q", individuals.Context()) + } +} + +func TestAFieldNoRuleDecidesIsReported(t *testing.T) { + f := fact("shoe_size", "int32", "system.operations") + if _, ok := individuals.Decide(f); ok { + t.Fatal("a field outside every rule was decided") + } + if got := f.String(); got != "individuals.Individual.shoe_size (int32; classification.data_categories = system.operations)" { + t.Fatalf("String = %q", got) + } +} + +func TestUnderMatchesTheCategoryAndItsChildrenOnly(t *testing.T) { + m := category.Under("user") + if !m.Match(fact("a", "string", "user")) || !m.Match(fact("a", "string", "user.contact.email")) { + t.Fatal("Under does not match the category or a child") + } + if m.Match(fact("a", "string", "username")) || m.Match(fact("a", "string")) { + t.Fatal("Under matches a sibling or a field with no category") + } + if !category.Is("user").Match(fact("a", "string", "user")) || category.Is("user").Match(fact("a", "string", "user.name")) { + t.Fatal("Is does not match exactly") + } +} + +func TestOtherwiseFailAndIdentity(t *testing.T) { + rules := FirstOf( + When(Field("secret"), Fail("never store this field")), + When(Field("old"), Encrypt(), Name("renamed"), Identity("old")), + ).OrElse(Otherwise(Omit())) + o, ok := rules.Decide(fact("secret", "string")) + if !ok || o.Reason() != "never store this field" { + t.Fatalf("Fail: ok=%v reason=%q", ok, o.Reason()) + } + if _, ok := o.Tag("secret"); ok { + t.Fatal("a Fail outcome gave a tag") + } + o, _ = rules.Decide(fact("old", "string")) + if tag, _ := o.Tag("old"); tag != "renamed,encrypt" || o.Identity != "old" { + t.Fatalf("Name and Identity: %q %q", tag, o.Identity) + } + o, ok = rules.Decide(fact("anything", "bytes")) + if tag, _ := o.Tag("anything"); !ok || tag != "-" { + t.Fatalf("Otherwise: ok=%v tag=%q", ok, tag) + } + if tag, _ := EncryptIndex(Equality, Match(IndexOption{"k", "3"}), Ore, Ope).apply("x"); tag != "x,encrypt,index=equality;match(k=3);ore;ope" { + t.Fatalf("indexes spell as %q", tag) + } + if tag, _ := Index(JSON(IndexOption{Key: "compat"})).apply("attrs"); tag != "attrs,index=json(compat)" { + t.Fatalf("json spells as %q", tag) + } + var none Rules + if _, ok := none.Decide(fact("x", "string")); ok { + t.Fatal("empty rules decided a field") + } + if _, ok := ForMessage(nil, "c", nil).Decide(fact("x", "string")); ok { + t.Fatal("a message with no rules decided a field") + } +} + +// apply is Outcome.Tag for a bare decision. +func (d Decision) apply(name string) (string, bool) { return Outcome{Decision: d}.Tag(name) } diff --git a/languages/golang/encrypt/policy/protosource/internal/testpb/classification.pb.go b/languages/golang/encrypt/policy/protosource/internal/testpb/classification.pb.go new file mode 100644 index 000000000..053eb01bd --- /dev/null +++ b/languages/golang/encrypt/policy/protosource/internal/testpb/classification.pb.go @@ -0,0 +1,94 @@ +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.26.0 +// protoc (unknown) +// source: classification.proto + +package testpb + +import ( + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + descriptorpb "google.golang.org/protobuf/types/descriptorpb" + reflect "reflect" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +var file_classification_proto_extTypes = []protoimpl.ExtensionInfo{ + { + ExtendedType: (*descriptorpb.FieldOptions)(nil), + ExtensionType: ([]string)(nil), + Field: 50001, + Name: "classification.data_categories", + Tag: "bytes,50001,rep,name=data_categories", + Filename: "classification.proto", + }, +} + +// Extension fields to descriptorpb.FieldOptions. +var ( + // The Fideslang data categories of a field. + // + // repeated string data_categories = 50001; + E_DataCategories = &file_classification_proto_extTypes[0] +) + +var File_classification_proto protoreflect.FileDescriptor + +var file_classification_proto_rawDesc = []byte{ + 0x0a, 0x14, 0x63, 0x6c, 0x61, 0x73, 0x73, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0e, 0x63, 0x6c, 0x61, 0x73, 0x73, 0x69, 0x66, 0x69, + 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x1a, 0x20, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x64, 0x65, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, + 0x6f, 0x72, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x3a, 0x48, 0x0a, 0x0f, 0x64, 0x61, 0x74, 0x61, + 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x69, 0x65, 0x73, 0x12, 0x1d, 0x2e, 0x67, 0x6f, + 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x46, 0x69, + 0x65, 0x6c, 0x64, 0x4f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0xd1, 0x86, 0x03, 0x20, 0x03, + 0x28, 0x09, 0x52, 0x0e, 0x64, 0x61, 0x74, 0x61, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x69, + 0x65, 0x73, 0x42, 0x1d, 0x5a, 0x1b, 0x65, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x2e, 0x63, 0x6f, + 0x6d, 0x2f, 0x61, 0x70, 0x70, 0x2f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x70, + 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var file_classification_proto_goTypes = []interface{}{ + (*descriptorpb.FieldOptions)(nil), // 0: google.protobuf.FieldOptions +} +var file_classification_proto_depIdxs = []int32{ + 0, // 0: classification.data_categories:extendee -> google.protobuf.FieldOptions + 1, // [1:1] is the sub-list for method output_type + 1, // [1:1] is the sub-list for method input_type + 1, // [1:1] is the sub-list for extension type_name + 0, // [0:1] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_classification_proto_init() } +func file_classification_proto_init() { + if File_classification_proto != nil { + return + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_classification_proto_rawDesc, + NumEnums: 0, + NumMessages: 0, + NumExtensions: 1, + NumServices: 0, + }, + GoTypes: file_classification_proto_goTypes, + DependencyIndexes: file_classification_proto_depIdxs, + ExtensionInfos: file_classification_proto_extTypes, + }.Build() + File_classification_proto = out.File + file_classification_proto_rawDesc = nil + file_classification_proto_goTypes = nil + file_classification_proto_depIdxs = nil +} diff --git a/languages/golang/encrypt/policy/protosource/internal/testpb/doc.go b/languages/golang/encrypt/policy/protosource/internal/testpb/doc.go new file mode 100644 index 000000000..755d37c97 --- /dev/null +++ b/languages/golang/encrypt/policy/protosource/internal/testpb/doc.go @@ -0,0 +1,6 @@ +// Package testpb is protoc-gen-go output for the protosource tests: the two +// messages of docs/plans/2026-10-04-plan-builder/proto, where +// classification.data_categories is a repeated string field option. It is a +// copy of docs/plans/2026-10-04-plan-builder/internal/pb with the package +// renamed; regenerate it from the .proto files there with buf. +package testpb diff --git a/languages/golang/encrypt/policy/protosource/internal/testpb/individual.pb.go b/languages/golang/encrypt/policy/protosource/internal/testpb/individual.pb.go new file mode 100644 index 000000000..561dfdb36 --- /dev/null +++ b/languages/golang/encrypt/policy/protosource/internal/testpb/individual.pb.go @@ -0,0 +1,188 @@ +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.26.0 +// protoc (unknown) +// source: individual.proto + +package testpb + +import ( + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +type Individual struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Id int64 `protobuf:"varint,1,opt,name=id,proto3" json:"id,omitempty"` + Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` + Email string `protobuf:"bytes,3,opt,name=email,proto3" json:"email,omitempty"` + MedicareNo string `protobuf:"bytes,4,opt,name=medicare_no,json=medicareNo,proto3" json:"medicare_no,omitempty"` + Nickname string `protobuf:"bytes,5,opt,name=nickname,proto3" json:"nickname,omitempty"` +} + +func (x *Individual) Reset() { + *x = Individual{} + if protoimpl.UnsafeEnabled { + mi := &file_individual_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Individual) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Individual) ProtoMessage() {} + +func (x *Individual) ProtoReflect() protoreflect.Message { + mi := &file_individual_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Individual.ProtoReflect.Descriptor instead. +func (*Individual) Descriptor() ([]byte, []int) { + return file_individual_proto_rawDescGZIP(), []int{0} +} + +func (x *Individual) GetId() int64 { + if x != nil { + return x.Id + } + return 0 +} + +func (x *Individual) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *Individual) GetEmail() string { + if x != nil { + return x.Email + } + return "" +} + +func (x *Individual) GetMedicareNo() string { + if x != nil { + return x.MedicareNo + } + return "" +} + +func (x *Individual) GetNickname() string { + if x != nil { + return x.Nickname + } + return "" +} + +var File_individual_proto protoreflect.FileDescriptor + +var file_individual_proto_rawDesc = []byte{ + 0x0a, 0x10, 0x69, 0x6e, 0x64, 0x69, 0x76, 0x69, 0x64, 0x75, 0x61, 0x6c, 0x2e, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x12, 0x0b, 0x69, 0x6e, 0x64, 0x69, 0x76, 0x69, 0x64, 0x75, 0x61, 0x6c, 0x73, 0x1a, + 0x14, 0x63, 0x6c, 0x61, 0x73, 0x73, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, + 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0xc2, 0x01, 0x0a, 0x0a, 0x49, 0x6e, 0x64, 0x69, 0x76, 0x69, + 0x64, 0x75, 0x61, 0x6c, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, + 0x52, 0x02, 0x69, 0x64, 0x12, 0x21, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x09, 0x42, 0x0d, 0x8a, 0xb5, 0x18, 0x09, 0x75, 0x73, 0x65, 0x72, 0x2e, 0x6e, 0x61, 0x6d, + 0x65, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x2c, 0x0a, 0x05, 0x65, 0x6d, 0x61, 0x69, 0x6c, + 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x42, 0x16, 0x8a, 0xb5, 0x18, 0x12, 0x75, 0x73, 0x65, 0x72, + 0x2e, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x2e, 0x65, 0x6d, 0x61, 0x69, 0x6c, 0x52, 0x05, + 0x65, 0x6d, 0x61, 0x69, 0x6c, 0x12, 0x37, 0x0a, 0x0b, 0x6d, 0x65, 0x64, 0x69, 0x63, 0x61, 0x72, + 0x65, 0x5f, 0x6e, 0x6f, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x42, 0x16, 0x8a, 0xb5, 0x18, 0x12, + 0x75, 0x73, 0x65, 0x72, 0x2e, 0x67, 0x6f, 0x76, 0x65, 0x72, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x5f, + 0x69, 0x64, 0x52, 0x0a, 0x6d, 0x65, 0x64, 0x69, 0x63, 0x61, 0x72, 0x65, 0x4e, 0x6f, 0x12, 0x1a, + 0x0a, 0x08, 0x6e, 0x69, 0x63, 0x6b, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x08, 0x6e, 0x69, 0x63, 0x6b, 0x6e, 0x61, 0x6d, 0x65, 0x42, 0x1d, 0x5a, 0x1b, 0x65, 0x78, + 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x61, 0x70, 0x70, 0x2f, 0x69, 0x6e, + 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, + 0x33, +} + +var ( + file_individual_proto_rawDescOnce sync.Once + file_individual_proto_rawDescData = file_individual_proto_rawDesc +) + +func file_individual_proto_rawDescGZIP() []byte { + file_individual_proto_rawDescOnce.Do(func() { + file_individual_proto_rawDescData = protoimpl.X.CompressGZIP(file_individual_proto_rawDescData) + }) + return file_individual_proto_rawDescData +} + +var file_individual_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_individual_proto_goTypes = []interface{}{ + (*Individual)(nil), // 0: individuals.Individual +} +var file_individual_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_individual_proto_init() } +func file_individual_proto_init() { + if File_individual_proto != nil { + return + } + file_classification_proto_init() + if !protoimpl.UnsafeEnabled { + file_individual_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Individual); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_individual_proto_rawDesc, + NumEnums: 0, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_individual_proto_goTypes, + DependencyIndexes: file_individual_proto_depIdxs, + MessageInfos: file_individual_proto_msgTypes, + }.Build() + File_individual_proto = out.File + file_individual_proto_rawDesc = nil + file_individual_proto_goTypes = nil + file_individual_proto_depIdxs = nil +} diff --git a/languages/golang/encrypt/policy/protosource/protosource.go b/languages/golang/encrypt/policy/protosource/protosource.go new file mode 100644 index 000000000..8f2901505 --- /dev/null +++ b/languages/golang/encrypt/policy/protosource/protosource.go @@ -0,0 +1,108 @@ +// Package protosource reads the facts about a protobuf message's fields: each +// field's name, its Go name, its kind, and its options as annotations. +// +// A field option's key is the option's full name, such as +// "classification.data_categories", and its values are the option's values +// as strings. A policy matches on them with policy.Key. +package protosource + +import ( + "fmt" + "strings" + + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/cipherstash/stack/languages/golang/encrypt/policy" +) + +// New returns the source. The message given to policy.ForMessage must be a +// generated protobuf message, such as &pb.Individual{}. +func New() policy.Source { return source{} } + +type source struct{} + +// Facts reads the message's descriptor. It returns one fact for each field, +// in field-number order as the descriptor lists them. +func (source) Facts(message any) ([]policy.Fact, error) { + m, ok := message.(proto.Message) + if !ok { + return nil, fmt.Errorf("protosource: %T is not a protobuf message", message) + } + md := m.ProtoReflect().Descriptor() + fields := md.Fields() + facts := make([]policy.Fact, 0, fields.Len()) + for i := range fields.Len() { + fd := fields.Get(i) + facts = append(facts, policy.Fact{ + Message: string(md.FullName()), + Name: string(fd.Name()), + GoName: GoName(string(fd.Name())), + Kind: fd.Kind().String(), + Annotations: annotations(fd), + }) + } + return facts, nil +} + +// annotations reads the extension fields set on the field's options. +func annotations(fd protoreflect.FieldDescriptor) []policy.Annotation { + opts := fd.Options() + if opts == nil { + return nil + } + var out []policy.Annotation + opts.ProtoReflect().Range(func(xd protoreflect.FieldDescriptor, v protoreflect.Value) bool { + if !xd.IsExtension() { + return true + } + a := policy.Annotation{Key: string(xd.FullName())} + if xd.IsList() { + list := v.List() + for j := range list.Len() { + a.Values = append(a.Values, scalar(xd, list.Get(j))) + } + } else { + a.Values = []string{scalar(xd, v)} + } + out = append(out, a) + return true + }) + return out +} + +// scalar spells one option value: an enum by its name, everything else by +// its protoreflect string form. +func scalar(fd protoreflect.FieldDescriptor, v protoreflect.Value) string { + if fd.Kind() == protoreflect.EnumKind { + if ev := fd.Enum().Values().ByNumber(v.Enum()); ev != nil { + return string(ev.Name()) + } + } + return v.String() +} + +// GoName is the Go field name protoc-gen-go gives a proto field: the first +// letter capitalised, and an underscore before a lower-case letter dropped +// with that letter capitalised. medicare_no becomes MedicareNo; foo_1bar +// keeps its underscore, as protoc-gen-go does. +func GoName(protoName string) string { + var b strings.Builder + upper := true + for i := 0; i < len(protoName); i++ { + c := protoName[i] + switch { + case c == '_' && i+1 < len(protoName) && protoName[i+1] >= 'a' && protoName[i+1] <= 'z': + upper = true + case c == '.': + b.WriteByte('_') + case upper && c >= 'a' && c <= 'z': + b.WriteByte(c - 'a' + 'A') + upper = false + default: + b.WriteByte(c) + upper = false + } + } + return b.String() +} diff --git a/languages/golang/encrypt/policy/protosource/protosource_test.go b/languages/golang/encrypt/policy/protosource/protosource_test.go new file mode 100644 index 000000000..c0492451b --- /dev/null +++ b/languages/golang/encrypt/policy/protosource/protosource_test.go @@ -0,0 +1,71 @@ +package protosource + +import ( + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/encrypt/policy" + "github.com/cipherstash/stack/languages/golang/encrypt/policy/protosource/internal/testpb" +) + +func TestFactsReadTheDescriptorAndItsOptions(t *testing.T) { + facts, err := New().Facts(&testpb.Individual{}) + if err != nil { + t.Fatal(err) + } + want := []string{ + "individuals.Individual.id (int64)", + "individuals.Individual.name (string; classification.data_categories = user.name)", + "individuals.Individual.email (string; classification.data_categories = user.contact.email)", + "individuals.Individual.medicare_no (string; classification.data_categories = user.government_id)", + "individuals.Individual.nickname (string)", + } + if len(facts) != len(want) { + t.Fatalf("%d facts, want %d", len(facts), len(want)) + } + for i, f := range facts { + if f.String() != want[i] { + t.Errorf("fact %d = %q, want %q", i, f, want[i]) + } + } + if facts[3].GoName != "MedicareNo" || facts[0].GoName != "Id" { + t.Errorf("GoName: %q %q", facts[0].GoName, facts[3].GoName) + } + if got := facts[2].Values("classification.data_categories"); len(got) != 1 || got[0] != "user.contact.email" { + t.Errorf("Values = %q", got) + } +} + +func TestTheRulesRunOnRealFacts(t *testing.T) { + category := policy.Key("classification.data_categories") + rules := policy.FirstOf( + policy.When(category.Under("user.government_id"), policy.EncryptInto("TextEq")), + policy.When(category.Under("user.contact.email"), policy.EncryptIndex(policy.Equality, policy.Match())), + policy.When(category.Under("user"), policy.Encrypt()), + ) + facts, _ := New().Facts(&testpb.Individual{}) + want := map[string]string{"name": "name,encrypt", "email": "email,encrypt,index=equality;match", "medicare_no": "medicare_no,encrypt_into=TextEq"} + for _, f := range facts { + o, ok := rules.Decide(f) + tag, _ := o.Tag(f.Name) + if w, decided := want[f.Name]; decided != ok || tag != w { + t.Errorf("%s: ok=%v tag=%q, want %q", f.Name, ok, tag, w) + } + } +} + +func TestNotAMessage(t *testing.T) { + _, err := New().Facts(struct{}{}) + if err == nil || !strings.Contains(err.Error(), "not a protobuf message") { + t.Fatalf("err = %v", err) + } +} + +func TestGoName(t *testing.T) { + cases := map[string]string{"id": "Id", "medicare_no": "MedicareNo", "foo_1bar": "Foo_1bar", "Already": "Already", "a_b_c": "ABC", "x__y": "X_Y"} + for in, want := range cases { + if got := GoName(in); got != want { + t.Errorf("GoName(%q) = %q, want %q", in, got, want) + } + } +} diff --git a/languages/golang/go.mod b/languages/golang/go.mod index 0d57540fc..751d6f869 100644 --- a/languages/golang/go.mod +++ b/languages/golang/go.mod @@ -12,6 +12,7 @@ require ( require ( golang.org/x/sys v0.48.0 golang.org/x/tools v0.51.0 + google.golang.org/protobuf v1.36.12 ) require ( diff --git a/languages/golang/go.sum b/languages/golang/go.sum index 39429dfe2..d9d2a09ce 100644 --- a/languages/golang/go.sum +++ b/languages/golang/go.sum @@ -2,8 +2,8 @@ github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d1 github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:6jpaqAo6f7rjWuxoti0Ymoi9DQZyxpnieWVudhXv38Y= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc h1:vlrjoILAURGfpBWucVZEywK22k4lfky1xGDIKV6kqNg= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:RJODA1DCSm4H+1pbmCzDdT2pAIkxhinwIu6ewjh5sPs= -github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= -github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU= github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0= golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= @@ -16,3 +16,5 @@ golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI= golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA= +google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= +google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= diff --git a/languages/golang/stashgen/declaration.go b/languages/golang/stashgen/declaration.go index 05e07acd8..8bca7c2ea 100644 --- a/languages/golang/stashgen/declaration.go +++ b/languages/golang/stashgen/declaration.go @@ -37,6 +37,10 @@ type Field struct { Indexes []Index // EQLType is the EQL type an EncryptInto field seals into. EQLType string + // Identity is the field's part of its context when it differs from Name: + // a column that was renamed keeps the identity it was first written + // under. "" means Name. Only a policy sets it. + Identity string } // Verb is what happens to a field. diff --git a/languages/golang/stashgen/doc.go b/languages/golang/stashgen/doc.go index 4185839d2..a868d1ff3 100644 --- a/languages/golang/stashgen/doc.go +++ b/languages/golang/stashgen/doc.go @@ -13,6 +13,11 @@ // fields keep their declared order, and the file carries no version and no // time beyond the gensupport.GeneratedVersion1 constant it names. // +// [Generate] writes the same file for a type that cannot carry tags, from a +// policy in package encrypt/policy: a source gives the facts about each field +// and the policy's rules decide them. A decision spells itself in the tag +// grammar, so both paths go through one parser and one emitter. +// // Every refusal about an index, an EQL type or a field type comes from the // Engine. The generator holds no copy of the engine's rules. package stashgen diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index 0a632b8b5..5b778a451 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -191,7 +191,7 @@ func (w *writer) declaration(f *genFile) { w.p("var %s = gensupport.Declare(%q).", f.declVar, f.decl.Context) for i, fld := range f.decl.Fields { end := "." - if i == len(f.decl.Fields)-1 { + if i == len(f.decl.Fields)-1 && fld.Identity == "" { end = "" } switch fld.Verb { @@ -214,6 +214,13 @@ func (w *writer) declaration(f *genFile) { } w.p("\t%s(%q, %s)%s", call, fld.Name, strings.Join(args, ", "), end) } + if fld.Identity != "" { + end = "." + if i == len(f.decl.Fields)-1 { + end = "" + } + w.p("\tIdentity(%q, %q)%s", fld.Name, fld.Identity, end) + } } } diff --git a/languages/golang/stashgen/export_test.go b/languages/golang/stashgen/export_test.go new file mode 100644 index 000000000..8b9d4bee4 --- /dev/null +++ b/languages/golang/stashgen/export_test.go @@ -0,0 +1,18 @@ +package stashgen + +import ( + "context" + "io" + + "github.com/cipherstash/stack/languages/golang/encrypt/policy" +) + +// GenerateFor is generateFor for the external tests, which name the message's +// type instead of holding a value of it: the type lives in a module the test +// process cannot import. +func GenerateFor(ctx context.Context, engine Engine, notices io.Writer, output string, source policy.Source, message policy.Message, pkgPath, typeName string) error { + return generateFor(generateConfig{output: output, engine: engine, notices: notices, ctx: ctx}, source, message, pkgPath, typeName) +} + +// MessageType is messageType for the external tests. +var MessageType = messageType diff --git a/languages/golang/stashgen/model_test.go b/languages/golang/stashgen/model_test.go new file mode 100644 index 000000000..1ac3e1ee8 --- /dev/null +++ b/languages/golang/stashgen/model_test.go @@ -0,0 +1,112 @@ +package stashgen_test + +import ( + "context" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/stashgen" + "github.com/cipherstash/stack/languages/golang/stashgen/enginetest" +) + +// build runs go vet over a module the test wrote, after the generated file +// is written into it. +func build(t *testing.T, dir string) { + t.Helper() + cmd := exec.Command("go", "vet", "./...") + cmd.Dir = dir + cmd.Env = append(os.Environ(), "GOPROXY=off", "GOWORK=off", "GOFLAGS=-mod=mod", "CGO_ENABLED=0") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("generated code does not build: %v\n%s", err, out) + } +} + +func TestModelDeclaredByAnotherStruct(t *testing.T) { + // userdb.ContactRow stands in for a sqlc row struct: it cannot carry + // tags, so contactRow in the package declares them. + dir := writeModule(t, map[string]string{ + "userdb/row.go": "package userdb\n\nimport \"github.com/cipherstash/stack/languages/golang/encrypt\"\n\n" + + "type ContactRow struct {\n\tID int64\n\tEmail encrypt.Ciphertext\n\tEmailEq encrypt.EqualityTerm\n\tNote encrypt.Ciphertext\n}\n", + "model.go": "package contacts\n\nimport (\n\t\"example.com/app/userdb\"\n\t\"github.com/cipherstash/stack/languages/golang/encrypt\"\n)\n\nvar _ userdb.ContactRow\n\n" + + "//go:generate go tool stashgen -type Contact -name Contact -model Rows=userdb.ContactRow:contactRow\n" + + "type Contact struct {\n\t_ struct{} `stash:\"context=contacts\"`\n\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt,index=equality\"`\n\tNote string `stash:\"note,encrypt\"`\n}\n\n" + + "type contactRow struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext `stash:\"email\"`\n\tEmailEq encrypt.EqualityTerm `stash:\"email,equality\"`\n\tNote encrypt.Ciphertext `stash:\"note\"`\n}\n\n" + + "// badRow gives Email the wrong type.\ntype badRow struct {\n\tID int64 `stash:\"id\"`\n\tEmail string `stash:\"email\"`\n\tEmailEq encrypt.EqualityTerm `stash:\"email,equality\"`\n\tNote encrypt.Ciphertext `stash:\"note\"`\n}\n", + }) + req := stashgen.Request{Dir: dir, Type: "Contact", Name: "Contact", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "userdb.ContactRow", Declares: "contactRow"}}} + file, err := stashgen.FromTags(context.Background(), enginetest.Static{}, req) + if err != nil { + t.Fatal(err) + } + src := string(file.Content) + for _, want := range []string{ + "type contactRowsShape struct {", + "var contactRowsCodec = gensupport.Records(contactCodec,", + "return userdb.ContactRow(contactRowsShape{", + "EmailEq: e.Email.Equality,", + "func EncryptRows(ctx context.Context, cipher *encrypt.Cipher, values []Contact) ([]userdb.ContactRow, error) {", + "func DecryptRows(ctx context.Context, d encrypt.Decrypter, rows []userdb.ContactRow) ([]Contact, error) {", + "var ContactFields = struct {", + "type ContactEmailField struct {", + } { + if !strings.Contains(src, want) { + t.Errorf("generated file lacks %q", want) + } + } + if err := file.Write(); err != nil { + t.Fatal(err) + } + build(t, dir) + + // The declaring struct must match the model field for field. + _, err = stashgen.FromTags(context.Background(), enginetest.Static{}, stashgen.Request{Dir: dir, Type: "Contact", Name: "Contact", + Models: []stashgen.ModelRequest{{Name: "Rows", Type: "userdb.ContactRow", Declares: "badRow"}}}) + if err == nil || !strings.Contains(err.Error(), "badRow.Email: has type string, and userdb.ContactRow.Email has type encrypt.Ciphertext") { + t.Fatalf("a declaring struct that does not match: %v", err) + } + if err := os.Remove(filepath.Join(dir, "contact_stash.go")); err != nil { + t.Fatal(err) + } +} + +func TestModelWithEQLColumnsAndEmbeddedPassthrough(t *testing.T) { + // A model that carries an EQL value and leaves a column unbound with "-". + dir := writeModule(t, map[string]string{ + "model.go": "package users\n\nimport \"github.com/cipherstash/stack/languages/golang/encrypt/eql\"\n\n" + + "type User struct {\n\t_ struct{} `stash:\"context=users\"`\n\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt_into=TextEq\"`\n}\n\n" + + "type Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail eql.TextEq `stash:\"email\"`\n\tVersion int `stash:\"-\"`\n}\n", + }) + file, err := stashgen.FromTags(context.Background(), enginetest.Static{}, stashgen.Request{Dir: dir, Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(file.Content), "Email: e.Email,") || strings.Contains(string(file.Content), "Version:") { + t.Fatalf("model conversion:\n%s", file.Content) + } + if err := file.Write(); err != nil { + t.Fatal(err) + } + build(t, dir) +} + +func TestParseModelFlag(t *testing.T) { + good := map[string]stashgen.ModelRequest{ + "Rows=ContactRow": {Name: "Rows", Type: "ContactRow"}, + "Rows=userdb.Row:rowDecl": {Name: "Rows", Type: "userdb.Row", Declares: "rowDecl"}, + "LegacyRows=Legacy:legacyD": {Name: "LegacyRows", Type: "Legacy", Declares: "legacyD"}, + } + for in, want := range good { + got, err := stashgen.ParseModelFlag(in) + if err != nil || got != want { + t.Errorf("ParseModelFlag(%q) = %+v, %v; want %+v", in, got, err, want) + } + } + for _, bad := range []string{"", "Rows", "Rows=", "=R", "rows=R", "Rows=R:", "Ro-ws=R"} { + if _, err := stashgen.ParseModelFlag(bad); err == nil { + t.Errorf("ParseModelFlag(%q) accepted", bad) + } + } +} diff --git a/languages/golang/stashgen/policy.go b/languages/golang/stashgen/policy.go new file mode 100644 index 000000000..d8d542cab --- /dev/null +++ b/languages/golang/stashgen/policy.go @@ -0,0 +1,236 @@ +package stashgen + +import ( + "context" + "errors" + "fmt" + "go/types" + "io" + "os" + "path/filepath" + "reflect" + "strings" + + "github.com/cipherstash/stack/languages/golang/encrypt/policy" + "golang.org/x/tools/go/packages" +) + +// GenerateOption configures [Generate]. +type GenerateOption func(*generateConfig) + +type generateConfig struct { + output string + engine Engine + notices io.Writer + ctx context.Context +} + +// Output names the file to write. It is required. The file goes in a package +// of your own, not in the package of the generated type, so the generate +// program never imports a file that it wrote. +func Output(path string) GenerateOption { return func(c *generateConfig) { c.output = path } } + +// WithEngine checks the declaration with this engine instead of the one the +// SDK embeds. +func WithEngine(e Engine) GenerateOption { return func(c *generateConfig) { c.engine = e } } + +// WithNotices sends the generator's notices here instead of stderr. +func WithNotices(w io.Writer) GenerateOption { return func(c *generateConfig) { c.notices = w } } + +// WithContext runs the generator under this context. +func WithContext(ctx context.Context) GenerateOption { return func(c *generateConfig) { c.ctx = ctx } } + +// Generate writes the generated file for a message from a policy: the source +// gives the facts about each field, the message's rules decide each one, and +// the file is the same one stashgen writes from tags. The generated functions +// take and return pointers to the message. +// +// Every field of the message needs a decision. A field that no rule decides +// stops the generator with the field's name and its annotations, and so does +// a field that a rule refuses with Fail. +func Generate(source policy.Source, message policy.Message, opts ...GenerateOption) error { + cfg := generateConfig{notices: os.Stderr, ctx: context.Background()} + for _, o := range opts { + o(&cfg) + } + if cfg.output == "" { + return errors.New("stashgen: Generate needs Output(path)") + } + if cfg.engine == nil { + e, err := GuestEngine(cfg.ctx) + if err != nil { + return err + } + cfg.engine = e + } + pkgPath, typeName, err := messageType(message.Message()) + if err != nil { + return err + } + return generateFor(cfg, source, message, pkgPath, typeName) +} + +// messageType finds the package path and name of the message's struct type. +func messageType(message any) (pkgPath, typeName string, err error) { + if message == nil { + return "", "", errors.New("stashgen: ForMessage got a nil message; give it a value of the generated type, such as &pb.Individual{}") + } + t := reflect.TypeOf(message) + for t.Kind() == reflect.Pointer { + t = t.Elem() + } + if t.Kind() != reflect.Struct || t.Name() == "" || t.PkgPath() == "" { + return "", "", fmt.Errorf("stashgen: the message is a %s, and the generator needs a named struct type", t) + } + return t.PkgPath(), t.Name(), nil +} + +// generateFor is Generate once the message's type is known by name. The +// tests drive it with a type in a module the test process cannot import. +func generateFor(cfg generateConfig, source policy.Source, message policy.Message, pkgPath, typeName string) error { + if message.Context() == "" { + return fmt.Errorf("stashgen: %s: ForMessage needs a Context", typeName) + } + eqlTypes, err := cfg.engine.EQLTypes(cfg.ctx) + if err != nil { + return fmt.Errorf("stashgen: the engine's EQL types: %w", err) + } + facts, err := source.Facts(message.Message()) + if err != nil { + return fmt.Errorf("stashgen: %s: %w", typeName, err) + } + + outDir := filepath.Dir(cfg.output) + outName, outPath, err := outputPackage(cfg.ctx, outDir) + if err != nil { + return err + } + msgPkg, err := loadImport(cfg.ctx, outDir, pkgPath) + if err != nil { + return err + } + if msgPkg.PkgPath == outPath { + return fmt.Errorf("stashgen: %s is in package %s, and the generated file goes in a package of your own", typeName, outPath) + } + named, st, err := lookupStruct(msgPkg, typeName) + if err != nil { + return err + } + display := msgPkg.Name + "." + typeName + + // Facts decide fields; the struct's tags say which Go field a schema + // field became, with the Go name as the fallback. + byProtoName, byGoName := structFields(st) + collected := &collected{context: message.Context()} + seen := map[string]bool{} + for _, fact := range facts { + fld := byProtoName[fact.Name] + if fld == nil { + fld = byGoName[fact.GoName] + } + if fld == nil { + return fieldErr(display, fact.GoName, "the source names the field %q, and the struct has no such field", fact.Name) + } + if seen[fld.Name()] { + return fieldErr(display, fld.Name(), "two facts name this field") + } + seen[fld.Name()] = true + outcome, ok := message.Decide(fact) + if !ok { + return fieldErr(display, fld.Name(), "no rule decides %s", fact) + } + if reason := outcome.Reason(); reason != "" { + return fieldErr(display, fld.Name(), "refused by the policy: %s", reason) + } + tagValue, _ := outcome.Tag(fact.Name) + t, err := parseTagValue(tagValue) + if err != nil { + return fieldErr(display, fld.Name(), "the policy's decision does not parse: %v", err) + } + t.Identity = outcome.Identity + if hasInvalidType(fld.Type()) { + return fieldErr(display, fld.Name(), "its type did not resolve: %s", packageErrors(msgPkg)) + } + collected.fields = append(collected.fields, collectedField{tag: t, goName: fld.Name(), typ: fld.Type(), exported: fld.Exported()}) + } + for i := range st.NumFields() { + fld := st.Field(i) + if fld.Exported() && !seen[fld.Name()] { + return fieldErr(display, fld.Name(), "the source gave no fact for this field, so no rule decided it") + } + } + + r := &reader{pkg: msgPkg, req: Request{Type: typeName}, eql: eqlTypes, imports: newImportSet(), outPkgName: outName, outPkgPath: outPath} + gf, err := r.build(collected, display, named, st, true, nil) + if err != nil { + return err + } + if err := cfg.engine.Check(cfg.ctx, gf.decl); err != nil { + return err + } + src, err := emit(gf) + if err != nil { + return err + } + for _, n := range gf.stderrNotices() { + fmt.Fprintln(cfg.notices, n) + } + return (&File{Path: cfg.output, Content: src}).Write() +} + +// structFields indexes a struct's fields by the proto name in their protobuf +// tag and by Go name. +func structFields(st *types.Struct) (byProtoName, byGoName map[string]*types.Var) { + byProtoName, byGoName = map[string]*types.Var{}, map[string]*types.Var{} + for i := range st.NumFields() { + fld := st.Field(i) + byGoName[fld.Name()] = fld + if tag, ok := lookupTag(st.Tag(i), "protobuf"); ok { + for _, part := range strings.Split(tag, ",") { + if name, ok := strings.CutPrefix(part, "name="); ok { + byProtoName[name] = fld + } + } + } + } + return byProtoName, byGoName +} + +// outputPackage finds the name and import path of the package in dir. A +// directory with no Go files yet is named after itself. +func outputPackage(ctx context.Context, dir string) (name, path string, err error) { + cfg := &packages.Config{Context: ctx, Dir: dir, Mode: packages.NeedName} + pkgs, err := packages.Load(cfg, ".") + if err != nil { + return "", "", fmt.Errorf("stashgen: the output directory %s: %w", dir, err) + } + if len(pkgs) != 1 || pkgs[0].PkgPath == "" { + return "", "", fmt.Errorf("stashgen: the output directory %s is not one package in a module", dir) + } + name = pkgs[0].Name + if name == "" { + abs, err := filepath.Abs(dir) + if err != nil { + return "", "", err + } + name = filepath.Base(abs) + } + return name, pkgs[0].PkgPath, nil +} + +// loadImport loads one import path from the module in dir. +func loadImport(ctx context.Context, dir, importPath string) (*packages.Package, error) { + cfg := &packages.Config{ + Context: ctx, + Dir: dir, + Mode: packages.NeedName | packages.NeedTypes | packages.NeedImports | packages.NeedSyntax, + } + pkgs, err := packages.Load(cfg, importPath) + if err != nil { + return nil, fmt.Errorf("stashgen: load %s: %w", importPath, err) + } + if len(pkgs) != 1 || pkgs[0].Types == nil { + return nil, fmt.Errorf("stashgen: %s did not load as one package from %s", importPath, dir) + } + return pkgs[0], nil +} diff --git a/languages/golang/stashgen/policy_test.go b/languages/golang/stashgen/policy_test.go new file mode 100644 index 000000000..7703def47 --- /dev/null +++ b/languages/golang/stashgen/policy_test.go @@ -0,0 +1,188 @@ +package stashgen_test + +import ( + "bytes" + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/encrypt/policy" + "github.com/cipherstash/stack/languages/golang/stashgen" + "github.com/cipherstash/stack/languages/golang/stashgen/enginetest" +) + +var category = policy.Key("classification.data_categories") + +// individualFacts is what protosource would read from individual.proto. +func individualFacts(any) ([]policy.Fact, error) { + f := func(name, goName, kind string, categories ...string) policy.Fact { + fact := policy.Fact{Message: "individuals.Individual", Name: name, GoName: goName, Kind: kind} + if len(categories) > 0 { + fact.Annotations = []policy.Annotation{{Key: string(category), Values: categories}} + } + return fact + } + return []policy.Fact{ + f("id", "Id", "int64"), + f("name", "Name", "string", "user.name"), + f("email", "Email", "string", "user.contact.email"), + f("medicare_no", "MedicareNo", "string", "user.government_id"), + f("nickname", "Nickname", "string"), + }, nil +} + +var base = policy.FirstOf( + policy.When(category.Under("user.government_id"), policy.EncryptInto("TextEq")), + policy.When(category.Under("user.contact.email"), policy.EncryptIndex(policy.Equality, policy.Match())), + policy.When(category.Under("user"), policy.Encrypt()), +) + +func individualRules(extra ...policy.Rule) policy.Message { + rules := policy.FirstOf( + policy.When(policy.Field("medicare_no"), policy.EncryptInto("TextEq"), policy.Name("medicare_number")), + policy.When(policy.Field("id"), policy.Passthrough()), + policy.When(policy.Field("nickname"), policy.Passthrough()), + ) + rules = append(rules, extra...) + return policy.ForMessage(struct{}{}, policy.Context("individuals"), rules.OrElse(base)) +} + +// policyModule copies the foreign case, whose pb package stands in for +// protoc-gen-go output, and adds an empty individuals directory. +func policyModule(t *testing.T) string { + t.Helper() + dir := t.TempDir() + src := filepath.Join("testdata", "cases", "foreign") + for _, name := range []string{"go.mod", "pb/individual.go"} { + data, err := os.ReadFile(filepath.Join(src, name)) + if err != nil { + t.Fatal(err) + } + if name == "go.mod" { + testdata, _ := filepath.Abs("testdata") + data = bytes.ReplaceAll(data, []byte("../../stubsdk"), []byte(filepath.Join(testdata, "stubsdk"))) + data = bytes.ReplaceAll(data, []byte("../../stubgorm"), []byte(filepath.Join(testdata, "stubgorm"))) + } + if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o750); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, name), data, 0o600); err != nil { + t.Fatal(err) + } + } + if err := os.MkdirAll(filepath.Join(dir, "individuals"), 0o750); err != nil { + t.Fatal(err) + } + return dir +} + +func TestGenerateFromAPolicy(t *testing.T) { + dir := policyModule(t) + out := filepath.Join(dir, "individuals", "individual_stash.go") + var notices bytes.Buffer + err := stashgen.GenerateFor(context.Background(), enginetest.Static{}, ¬ices, out, policy.SourceFunc(individualFacts), individualRules(), "example.com/app/pb", "Individual") + if err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(out) + if err != nil { + t.Fatal(err) + } + golden := filepath.Join("testdata", "policy_individual_stash.go.golden") + if *update { + if err := os.WriteFile(golden, got, 0o600); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(golden) + if err != nil { + t.Fatalf("%v (run with -update to write it)", err) + } + if !bytes.Equal(got, want) { + t.Errorf("generated file differs from %s:\n%s", golden, got) + } + if !strings.Contains(notices.String(), "pb.Individual prints its sealed fields in the clear") { + t.Errorf("notices = %q", notices.String()) + } + // The policy path and the tag path write the same file for the same + // declaration, but for the header comments the tag path cannot know. + tagged, err := os.ReadFile(filepath.Join("testdata", "cases", "foreign", "individualstash_stash.go.golden")) + if err != nil { + t.Fatal(err) + } + if body(string(got)) != body(string(tagged)) { + t.Errorf("the policy path and the tag path disagree:\n%s", diff(body(string(tagged)), body(string(got)))) + } +} + +// body drops everything before the encrypted type: the package clause, the +// imports and the notices, which the two paths word differently. +func body(src string) string { + i := strings.Index(src, "\ntype Encrypted") + if i < 0 { + return src + } + return src[i:] +} + +func TestGenerateRefusals(t *testing.T) { + dir := policyModule(t) + out := filepath.Join(dir, "individuals", "individual_stash.go") + cases := []struct { + name string + source policy.Source + message policy.Message + want string + }{ + {"a field no rule decides", policy.SourceFunc(individualFacts), + policy.ForMessage(struct{}{}, "individuals", policy.When(policy.Field("id"), policy.Passthrough())), + `pb.Individual.Name: no rule decides individuals.Individual.name (string; classification.data_categories = user.name)`}, + {"a field a rule refuses", policy.SourceFunc(individualFacts), + individualRules(policy.When(policy.Field("name"), policy.Fail("names are not stored"))), + "pb.Individual.Name: refused by the policy: names are not stored"}, + {"a fact for a field the struct lacks", policy.SourceFunc(func(any) ([]policy.Fact, error) { + return []policy.Fact{{Message: "m", Name: "shoe_size", GoName: "ShoeSize", Kind: "int32"}}, nil + }), individualRules(policy.Otherwise(policy.Passthrough())), `the source names the field "shoe_size", and the struct has no such field`}, + {"a struct field with no fact", policy.SourceFunc(func(any) ([]policy.Fact, error) { + return []policy.Fact{{Message: "m", Name: "id", GoName: "Id", Kind: "int64"}}, nil + }), individualRules(policy.Otherwise(policy.Passthrough())), "pb.Individual.Name: the source gave no fact for this field"}, + {"no context", policy.SourceFunc(individualFacts), policy.ForMessage(struct{}{}, "", base), "ForMessage needs a Context"}, + {"an index the engine refuses", policy.SourceFunc(individualFacts), + individualRules(policy.When(policy.Field("name"), policy.EncryptIndex(policy.Match(policy.IndexOption{Key: "k", Value: "3"})))), + "pb.Individual.Name: index match(k=3): the engine cannot carry index options"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + err := stashgen.GenerateFor(context.Background(), enginetest.Static{}, &bytes.Buffer{}, out, c.source, c.message, "example.com/app/pb", "Individual") + if err == nil || !strings.Contains(err.Error(), c.want) { + t.Fatalf("error %v, want one containing %q", err, c.want) + } + if _, statErr := os.Stat(out); !os.IsNotExist(statErr) { + t.Fatal("a file was written after a refusal") + } + }) + } +} + +func TestGenerateNeedsOutputAndAMessage(t *testing.T) { + if err := stashgen.Generate(policy.SourceFunc(individualFacts), individualRules()); err == nil || !strings.Contains(err.Error(), "needs Output") { + t.Fatalf("err = %v", err) + } + type local struct{ A int } + pkgPath, name, err := stashgen.MessageType(&local{}) + if err != nil || name != "local" || !strings.HasSuffix(pkgPath, "/stashgen_test") { + t.Fatalf("messageType = %q %q %v", pkgPath, name, err) + } + if _, _, err := stashgen.MessageType(nil); err == nil { + t.Fatal("nil message accepted") + } + if _, _, err := stashgen.MessageType(42); err == nil { + t.Fatal("an int accepted as a message") + } + err = stashgen.Generate(policy.SourceFunc(individualFacts), policy.ForMessage(&local{}, "c", base), stashgen.Output(filepath.Join(t.TempDir(), "x_stash.go"))) + if err == nil || !strings.Contains(err.Error(), "not available in this build") { + t.Fatalf("without an engine: %v", err) + } +} diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index f2d937939..ebf7ba500 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -123,15 +123,26 @@ type reader struct { eql []EQLType file *genFile imports *importSet + // The package the file is written into, when it is not pkg: the policy + // path writes into a package of the user's own. + outPkgName string + outPkgPath string } func (r *reader) qualifier(p *types.Package) string { - if p == nil || p.Path() == r.pkg.PkgPath { + if p == nil || p.Path() == r.outPath() { return "" } return r.imports.add(p.Path(), p.Name()) } +func (r *reader) outPath() string { + if r.outPkgPath != "" { + return r.outPkgPath + } + return r.pkg.PkgPath +} + func (r *reader) typeExpr(t types.Type) string { return types.TypeString(t, r.qualifier) } @@ -140,30 +151,50 @@ func pathQualifier(p *types.Package) string { return p.Path() } func pathType(t types.Type) string { return types.TypeString(t, pathQualifier) } -// readStruct reads the tagged struct and, with -for, the type it declares -// for, and builds the file. +// read reads the tagged struct and, with -for, the type it declares for, and +// builds the file. func (r *reader) read() (*genFile, error) { pkg, req := r.pkg, r.req tagNamed, tagStruct, err := lookupStruct(pkg, req.Type) if err != nil { return nil, err } - f := &genFile{pkgName: pkg.Name, imports: r.imports} - r.file = f - f.imports.add("context", "context") - f.imports.add("log/slog", "slog") - f.imports.add(encryptPath, "encrypt") - f.imports.add(gensupportPath, "gensupport") - valueNamed, valueStruct := tagNamed, tagStruct - f.typeName = req.Type + typeName := req.Type if req.For != "" { valueNamed, valueStruct, err = lookupQualified(pkg, req.For) if err != nil { return nil, err } - f.typeName = req.For + typeName = req.For + } + collected, err := r.collectFields(tagNamed, tagStruct, req.Type, true) + if err != nil { + return nil, err + } + if collected.context == "" { + return nil, fieldErr(req.Type, "", "no `_ struct{}` field with `stash:\"context=...\"` declares the context") + } + if req.For != "" { + if err := r.matchFor(collected, valueNamed, valueStruct); err != nil { + return nil, err + } } + return r.build(collected, typeName, valueNamed, valueStruct, req.For != "", tagNamed) +} + +// build makes the file for a collected declaration. foreign says the value +// type is in another package; tagged is the struct that carried the tags, +// which -redact writes print methods on. +func (r *reader) build(collected *collected, typeName string, valueNamed *types.Named, valueStruct *types.Struct, foreign bool, tagged *types.Named) (*genFile, error) { + pkg, req := r.pkg, r.req + f := &genFile{pkgName: r.pkgName(), imports: r.imports, typeName: typeName} + r.file = f + f.imports.add("context", "context") + f.imports.add("log/slog", "slog") + f.imports.add(encryptPath, "encrypt") + f.imports.add(gensupportPath, "gensupport") + baseName := valueNamed.Obj().Name() f.typeExpr = r.typeExpr(valueNamed) f.zeroExpr = f.typeExpr + "{}" @@ -178,51 +209,40 @@ func (r *reader) read() (*genFile, error) { } else { f.encryptFn, f.decryptFn, f.fieldsVar = "Encrypt", "Decrypt", "Fields" f.declVar, f.codecVar = "declaration", "codec" - f.paramName = pkg.Name + f.paramName = f.pkgName + } + if f.paramName == "encrypt" || f.paramName == "eql" || f.paramName == "gensupport" || f.paramName == "context" || f.paramName == "slog" || f.paramName == "ctx" || f.paramName == "cipher" { + f.paramName = "values" } if req.Redact { - if req.For != "" { - return nil, fmt.Errorf("stashgen: -redact cannot add print methods to %s, a type in another package", req.For) + if foreign || tagged == nil { + return nil, fmt.Errorf("stashgen: -redact cannot add print methods to %s, a type in another package", typeName) } f.redact = true f.redactRecv = strings.ToLower(req.Type[:1]) for _, m := range []string{"String", "LogValue"} { - if hasMethod(types.NewPointer(tagNamed), m) { + if hasMethod(types.NewPointer(tagged), m) { return nil, fmt.Errorf("stashgen: -redact: %s already has a %s method", req.Type, m) } } } - // The context, and the fields. - collected, err := r.collectFields(tagNamed, tagStruct, req.Type, true) - if err != nil { - return nil, err - } - if collected.context == "" { - return nil, fieldErr(req.Type, "", "no `_ struct{}` field with `stash:\"context=...\"` declares the context") - } - f.decl = Declaration{Type: f.typeName, Context: collected.context, Opaque: collected.opaque} + f.decl = Declaration{Type: typeName, Context: collected.context, Opaque: collected.opaque} f.unexported = collected.unexported - if req.For != "" { - if err := r.matchFor(collected, valueNamed, valueStruct); err != nil { - return nil, err - } - } - // The shape check. A type from another package with an unexported field // cannot convert, so the file reads each field by name instead, and the // functions take a pointer. var conversionNotice string switch { - case req.For == "": + case !foreign: f.shapeFields = shapeOf(valueStruct, r.typeExpr) case hasUnexported(valueStruct): f.shapeName = "" f.isPointer = true f.typeExpr = "*" + f.typeExpr f.zeroExpr = "nil" - conversionNotice = fmt.Sprintf("%s has unexported fields, so Go cannot convert it to a copy of its fields. This file reads each field by name: the compiler finds a removed or retyped field, and CI finds an added one.", f.typeName) + conversionNotice = fmt.Sprintf("%s has unexported fields, so Go cannot convert it to a copy of its fields. This file reads each field by name: the compiler finds a removed or retyped field, and CI finds an added one.", typeName) default: f.shapeFields = shapeOf(valueStruct, r.typeExpr) } @@ -231,7 +251,7 @@ func (r *reader) read() (*genFile, error) { return nil, err } if len(f.fields) == 0 && len(f.opaque) == 0 { - return nil, fieldErr(req.Type, "", "stores no field: every field is omitted") + return nil, fieldErr(typeName, "", "stores no field: every field is omitted") } // Printing. @@ -246,10 +266,10 @@ func (r *reader) read() (*genFile, error) { } if sealedCount > 0 && !f.redact && (!hasMethod(valueNamed, "String") || !hasMethod(valueNamed, "LogValue")) { f.printsPlaintext = true - if req.For != "" { - f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear, and stashgen cannot add print methods to a type from another package.", f.typeName)) + if foreign { + f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear, and stashgen cannot add print methods to a type from another package.", typeName)) } else { - f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", f.typeName)) + f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear: it has no String or LogValue method. Write them, or run stashgen with -redact.", typeName)) } } if conversionNotice != "" { @@ -259,8 +279,10 @@ func (r *reader) read() (*genFile, error) { f.notices = append(f.notices, "Not encrypted and not stored: the unexported "+fieldList(f.unexported)+". Tag "+itOrEach(f.unexported)+" `stash:\"-\"` to confirm that.") } - if err := r.checkDirectives(); err != nil { - return nil, err + if pkg != nil { + if err := r.checkDirectives(); err != nil { + return nil, err + } } for _, m := range req.Models { if err := r.readModel(m); err != nil { @@ -270,6 +292,14 @@ func (r *reader) read() (*genFile, error) { return f, nil } +// pkgName is the name of the package the file is written into. +func (r *reader) pkgName() string { + if r.outPkgName != "" { + return r.outPkgName + } + return r.pkg.Name +} + // collectedField is one field of the tagged struct after its tag is read. type collectedField struct { tag tag @@ -517,7 +547,7 @@ func (r *reader) buildFields(c *collected) error { return fieldErr(typeName, cf.goName, "two fields write the name %q: %s and %s", name, prev, cf.goName) } seen[name] = cf.goName - field := Field{Name: name, GoName: cf.goName, GoType: r.goType(cf.typ), Verb: t.Verb, Indexes: t.Indexes, EQLType: t.EQLType} + field := Field{Name: name, GoName: cf.goName, GoType: r.goType(cf.typ), Verb: t.Verb, Indexes: t.Indexes, EQLType: t.EQLType, Identity: t.Identity} if t.Omit { field.Verb = VerbOmit } diff --git a/languages/golang/stashgen/tag.go b/languages/golang/stashgen/tag.go index c566a65e5..10df4a7df 100644 --- a/languages/golang/stashgen/tag.go +++ b/languages/golang/stashgen/tag.go @@ -29,6 +29,8 @@ type tag struct { Verb Verb Indexes []Index EQLType string + // Identity is set by a policy only; no tag spells it. + Identity string } // errNoTag reports a field with no stash tag at all. diff --git a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden new file mode 100644 index 000000000..1fb87876b --- /dev/null +++ b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden @@ -0,0 +1,182 @@ +// Code generated by stashgen. DO NOT EDIT. + +package individuals + +import ( + "context" + "log/slog" + + "example.com/app/pb" + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// pb.Individual prints its sealed fields in the clear, and stashgen cannot add +// print methods to a type from another package. + +// pb.Individual has unexported fields, so Go cannot convert it to a copy of its +// fields. This file reads each field by name: the compiler finds a removed or +// retyped field, and CI finds an added one. + +type EncryptedIndividual struct { + Id int64 + Name EncryptedIndividualName + Email EncryptedIndividualEmail + MedicareNo eql.TextEq + Nickname string +} + +type EncryptedIndividualName struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedIndividualEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} + +func (e EncryptedIndividual) String() string { + return gensupport.Redacted("EncryptedIndividual", map[string]any{"Id": e.Id, "Nickname": e.Nickname}, "Name", "Email", "MedicareNo") +} + +func (e EncryptedIndividual) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"Id": e.Id, "Nickname": e.Nickname}, "Name", "Email", "MedicareNo") +} + +var declaration = gensupport.Declare("individuals"). + Passthrough("id"). + Encrypt("name"). + EncryptIndex("email", encrypt.Equality, encrypt.Match()). + EncryptInto("medicare_number", "TextEq"). + Passthrough("nickname") + +var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndividual]{ + TypeName: "pb.Individual", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v *pb.Individual) gensupport.Values { + return gensupport.Values{ + "id": v.Id, + "name": v.Name, + "email": v.Email, + "medicare_number": v.MedicareNo, + "nickname": v.Nickname, + } + }, + Seal: func(rec gensupport.Record) (EncryptedIndividual, error) { + var e EncryptedIndividual + var err error + if e.Id, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedIndividual{}, err + } + if e.Nickname, err = gensupport.Passthrough[string](rec, "nickname"); err != nil { + return EncryptedIndividual{}, err + } + e.Name = EncryptedIndividualName{Ciphertext: rec["name"].Ciphertext} + e.Email = EncryptedIndividualEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + Match: rec["email"].Match, + } + e.MedicareNo = eql.TextEq(rec["medicare_number"].EQL) + return e, nil + }, + Open: func(e EncryptedIndividual) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.Id}, + "name": {Ciphertext: e.Name.Ciphertext}, + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality, Match: e.Email.Match}, + "medicare_number": {EQL: e.MedicareNo}, + "nickname": {Value: e.Nickname}, + } + }, + Value: func(e EncryptedIndividual, vals gensupport.Values) (*pb.Individual, error) { + v := &pb.Individual{} + var err error + if v.Id, err = gensupport.Get[int64](vals, "id"); err != nil { + return nil, err + } + if v.Name, err = gensupport.Get[string](vals, "name"); err != nil { + return nil, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return nil, err + } + if v.MedicareNo, err = gensupport.Get[string](vals, "medicare_number"); err != nil { + return nil, err + } + if v.Nickname, err = gensupport.Get[string](vals, "nickname"); err != nil { + return nil, err + } + return v, nil + }, +}) + +// Encrypt seals each pb.Individual in one ZeroKMS request. The result has one +// element for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, individuals []*pb.Individual) ([]EncryptedIndividual, error) { + return codec.Encrypt(ctx, cipher, individuals) +} + +// Decrypt opens each EncryptedIndividual in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedIndividual) ([]*pb.Individual, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Name NameField + Email EmailField + MedicareNo MedicareNoField +}{ + Name: NameField{gensupport.NewField[string](declaration, "name")}, + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + MedicareNo: MedicareNoField{gensupport.NewField[string](declaration, "medicare_number")}, +} + +type NameField struct { + field gensupport.Field[string] +} + +func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedIndividualName, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedIndividualName{Ciphertext: out.Ciphertext}, err +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedIndividualEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedIndividualEmail{}, err + } + return EncryptedIndividualEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil +} + +func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +type MedicareNoField struct { + field gensupport.Field[string] +} + +func (f MedicareNoField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f MedicareNoField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go index 0eeaf18e8..2cf9b1673 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go @@ -124,3 +124,6 @@ func (*RecordsCodec[P, R]) Decrypt(context.Context, encrypt.Decrypter, []R) ([]P // notices; the real New calls them once for each type. func NoticeUntagged(typeName string, fields []string) {} func NoticePrintsPlaintext(typeName string) {} + +// Identity gives a field a context part other than its name. +func (d Declaration) Identity(name, identity string) Declaration { return d } From d0c4e2c958bda2ac7d32894edbcb643fa2b37587 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 01:23:17 -0700 Subject: [PATCH 04/30] refactor(golang)!: rename stackencrypt to encrypt and stackauth to auth The two packages are `encrypt` and `auth`, at languages/golang/encrypt and languages/golang/auth, as the plan builder design names them: a package name does not repeat the product, and `encrypt.Cipher` reads as what it is where `stackencrypt.Cipher` repeated itself (SDK principle 13). A move with nothing else in it, so a review can read it as one: `git mv` of both directories, the guest crates with them, the package clauses, the import paths, the selectors, the error prefixes and the prose; every path in mise.toml, the root Cargo.toml exclude list, .gitignore, dependabot, tests-golang.yml, go-binding-test.sh, the scripts/__tests__ guards, AGENTS.md, SECURITY.md, stack-guest-abi's module doc, stack-encrypt's CONTEXT.md and the label-segments fixture comment. The task alias `go:stackencrypt:test` goes with the name it kept. The CI variable STACKENCRYPT_TESTS_REQUIRE_LOCK becomes STACK_ENCRYPT_TESTS_REQUIRE_LOCK. Two renames the compiler forced: the plan package's `encrypt` verdict constant collided with the package it now imports, so it is `sealed`; two test locals named `auth` shadowed the auth package and are `cts`. docs/plans and the ADRs keep the old names as history. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- .github/dependabot.yml | 4 +- .github/workflows/tests-golang.yml | 24 +++--- .gitignore | 8 +- AGENTS.md | 2 +- Cargo.toml | 4 +- SECURITY.md | 2 +- .../golang/{stackauth => auth}/README.md | 32 +++---- .../golang/{stackauth => auth}/clientkey.go | 10 +-- languages/golang/{stackauth => auth}/doc.go | 18 ++-- .../golang/{stackauth => auth}/errors.go | 6 +- languages/golang/{stackauth => auth}/guest.go | 14 +-- .../{stackauth => auth}/guest/.gitignore | 0 .../{stackauth => auth}/guest/Cargo.lock | 0 .../{stackauth => auth}/guest/Cargo.toml | 2 +- .../{stackauth => auth}/guest/src/abi.rs | 0 .../{stackauth => auth}/guest/src/auth.rs | 2 +- .../{stackauth => auth}/guest/src/headers.rs | 0 .../{stackauth => auth}/guest/src/host.rs | 0 .../{stackauth => auth}/guest/src/lib.rs | 4 +- .../{stackauth => auth}/guest/src/ops.rs | 2 +- .../{stackauth => auth}/guest/src/status.rs | 0 .../golang/{stackauth => auth}/lock_unix.go | 6 +- .../{stackauth => auth}/lock_windows.go | 6 +- languages/golang/{stackauth => auth}/mount.go | 2 +- .../golang/{stackauth => auth}/oauth2.go | 6 +- languages/golang/{stackauth => auth}/store.go | 12 +-- .../golang/{stackauth => auth}/store_test.go | 2 +- .../golang/{stackauth => auth}/strategy.go | 8 +- .../{stackauth => auth}/strategy_test.go | 2 +- languages/golang/{stackauth => auth}/token.go | 2 +- .../golang/{stackauth => auth}/transport.go | 4 +- .../golang/{stackauth => auth}/wasm/README.md | 0 .../{stackencrypt => encrypt}/README.md | 64 +++++++------- .../{stackencrypt => encrypt}/cipher.go | 4 +- .../{stackencrypt => encrypt}/client.go | 16 ++-- .../{stackencrypt => encrypt}/clientkey.go | 6 +- .../{stackencrypt => encrypt}/context.go | 10 +-- .../{stackencrypt => encrypt}/credentials.go | 86 +++++++++---------- .../credentials_test.go | 64 +++++++------- .../golang/{stackencrypt => encrypt}/doc.go | 16 ++-- .../{stackencrypt => encrypt}/errors.go | 4 +- .../example/README.md | 14 +-- .../example/explicit/README.md | 10 +-- .../example/explicit/main.go | 32 +++---- .../{stackencrypt => encrypt}/example/main.go | 34 ++++---- .../{stackencrypt => encrypt}/export_test.go | 4 +- .../golang/{stackencrypt => encrypt}/guest.go | 10 +-- .../guest/.gitignore | 0 .../guest/Cargo.lock | 0 .../guest/Cargo.toml | 0 .../guest/src/abi.rs | 0 .../guest/src/config.rs | 0 .../guest/src/headers.rs | 0 .../guest/src/host.rs | 0 .../guest/src/lib.rs | 0 .../guest/src/ops.rs | 0 .../guest/src/options.rs | 0 .../guest/src/response.rs | 0 .../guest/src/status.rs | 0 .../guest/tests/native_ops.rs | 0 .../{stackencrypt => encrypt}/guest_test.go | 8 +- .../{stackencrypt => encrypt}/keyset.go | 6 +- .../golang/{stackencrypt => encrypt}/label.go | 6 +- .../{stackencrypt => encrypt}/label_test.go | 2 +- .../golang/{stackencrypt => encrypt}/leaf.go | 6 +- .../{stackencrypt => encrypt}/live_test.go | 18 ++-- .../memory_linux_test.go | 2 +- .../{stackencrypt => encrypt}/memory_test.go | 10 +-- .../{stackencrypt => encrypt}/options.go | 12 +-- .../{stackencrypt => encrypt}/options_test.go | 34 ++++---- .../order_live_test.go | 2 +- .../{stackencrypt => encrypt}/plan/doc.go | 14 +-- .../{stackencrypt => encrypt}/plan/fact.go | 0 .../{stackencrypt => encrypt}/plan/message.go | 44 +++++----- .../plan/plan_test.go | 8 +- .../plan/plantest/compare.go | 2 +- .../plan/plantest/golden_test.go | 6 +- .../plan/plantest/plantest.go | 2 +- .../plan/plantest/plantest_internal_test.go | 4 +- .../plan/plantest/snapshot.go | 20 ++--- .../testdata/TestPolicies/audits.golden | 0 .../testdata/TestPolicies/individuals.golden | 0 .../{stackencrypt => encrypt}/plan/policy.go | 54 ++++++------ .../policy_plan_test.go | 8 +- .../{stackencrypt => encrypt}/record.go | 68 +++++++-------- .../{stackencrypt => encrypt}/runtime_test.go | 2 +- .../golang/{stackencrypt => encrypt}/term.go | 4 +- .../testdata/cllw_order.txt | 0 .../{stackencrypt => encrypt}/transport.go | 14 +-- .../{stackencrypt => encrypt}/unit_test.go | 4 +- .../{stackencrypt => encrypt}/wasm/README.md | 0 .../golang/internal/factstest/factstest.go | 2 +- .../internal/factstest/factstest_test.go | 2 +- languages/golang/internal/guest/clientkey.go | 4 +- languages/golang/internal/guest/doc.go | 2 +- languages/golang/internal/guest/memory.go | 4 +- .../golang/internal/guest/memory_test.go | 2 +- languages/golang/internal/guesttest/probe.go | 2 +- mise.toml | 44 +++++----- packages/stack-encrypt/CONTEXT.md | 2 +- .../tests/fixtures/label_segments.json | 2 +- packages/stack-guest-abi/src/lib.rs | 2 +- .../__tests__/cargo-lock-freshness.test.mjs | 4 +- .../__tests__/cargo-publish-opt-out.test.mjs | 4 +- scripts/__tests__/crates-ci.test.mjs | 4 +- scripts/go-binding-test.sh | 4 +- 106 files changed, 500 insertions(+), 502 deletions(-) rename languages/golang/{stackauth => auth}/README.md (79%) rename languages/golang/{stackauth => auth}/clientkey.go (51%) rename languages/golang/{stackauth => auth}/doc.go (85%) rename languages/golang/{stackauth => auth}/errors.go (95%) rename languages/golang/{stackauth => auth}/guest.go (93%) rename languages/golang/{stackauth => auth}/guest/.gitignore (100%) rename languages/golang/{stackauth => auth}/guest/Cargo.lock (100%) rename languages/golang/{stackauth => auth}/guest/Cargo.toml (96%) rename languages/golang/{stackauth => auth}/guest/src/abi.rs (100%) rename languages/golang/{stackauth => auth}/guest/src/auth.rs (99%) rename languages/golang/{stackauth => auth}/guest/src/headers.rs (100%) rename languages/golang/{stackauth => auth}/guest/src/host.rs (100%) rename languages/golang/{stackauth => auth}/guest/src/lib.rs (97%) rename languages/golang/{stackauth => auth}/guest/src/ops.rs (99%) rename languages/golang/{stackauth => auth}/guest/src/status.rs (100%) rename languages/golang/{stackauth => auth}/lock_unix.go (85%) rename languages/golang/{stackauth => auth}/lock_windows.go (87%) rename languages/golang/{stackauth => auth}/mount.go (99%) rename languages/golang/{stackauth => auth}/oauth2.go (76%) rename languages/golang/{stackauth => auth}/store.go (97%) rename languages/golang/{stackauth => auth}/store_test.go (99%) rename languages/golang/{stackauth => auth}/strategy.go (97%) rename languages/golang/{stackauth => auth}/strategy_test.go (99%) rename languages/golang/{stackauth => auth}/token.go (99%) rename languages/golang/{stackauth => auth}/transport.go (98%) rename languages/golang/{stackauth => auth}/wasm/README.md (100%) rename languages/golang/{stackencrypt => encrypt}/README.md (90%) rename languages/golang/{stackencrypt => encrypt}/cipher.go (98%) rename languages/golang/{stackencrypt => encrypt}/client.go (97%) rename languages/golang/{stackencrypt => encrypt}/clientkey.go (82%) rename languages/golang/{stackencrypt => encrypt}/context.go (95%) rename languages/golang/{stackencrypt => encrypt}/credentials.go (83%) rename languages/golang/{stackencrypt => encrypt}/credentials_test.go (95%) rename languages/golang/{stackencrypt => encrypt}/doc.go (94%) rename languages/golang/{stackencrypt => encrypt}/errors.go (97%) rename languages/golang/{stackencrypt => encrypt}/example/README.md (87%) rename languages/golang/{stackencrypt => encrypt}/example/explicit/README.md (83%) rename languages/golang/{stackencrypt => encrypt}/example/explicit/main.go (84%) rename languages/golang/{stackencrypt => encrypt}/example/main.go (84%) rename languages/golang/{stackencrypt => encrypt}/export_test.go (95%) rename languages/golang/{stackencrypt => encrypt}/guest.go (95%) rename languages/golang/{stackencrypt => encrypt}/guest/.gitignore (100%) rename languages/golang/{stackencrypt => encrypt}/guest/Cargo.lock (100%) rename languages/golang/{stackencrypt => encrypt}/guest/Cargo.toml (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/abi.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/config.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/headers.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/host.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/lib.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/ops.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/options.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/response.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/src/status.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest/tests/native_ops.rs (100%) rename languages/golang/{stackencrypt => encrypt}/guest_test.go (99%) rename languages/golang/{stackencrypt => encrypt}/keyset.go (95%) rename languages/golang/{stackencrypt => encrypt}/label.go (97%) rename languages/golang/{stackencrypt => encrypt}/label_test.go (99%) rename languages/golang/{stackencrypt => encrypt}/leaf.go (94%) rename languages/golang/{stackencrypt => encrypt}/live_test.go (96%) rename languages/golang/{stackencrypt => encrypt}/memory_linux_test.go (96%) rename languages/golang/{stackencrypt => encrypt}/memory_test.go (97%) rename languages/golang/{stackencrypt => encrypt}/options.go (93%) rename languages/golang/{stackencrypt => encrypt}/options_test.go (89%) rename languages/golang/{stackencrypt => encrypt}/order_live_test.go (99%) rename languages/golang/{stackencrypt => encrypt}/plan/doc.go (89%) rename languages/golang/{stackencrypt => encrypt}/plan/fact.go (100%) rename languages/golang/{stackencrypt => encrypt}/plan/message.go (84%) rename languages/golang/{stackencrypt => encrypt}/plan/plan_test.go (98%) rename languages/golang/{stackencrypt => encrypt}/plan/plantest/compare.go (99%) rename languages/golang/{stackencrypt => encrypt}/plan/plantest/golden_test.go (89%) rename languages/golang/{stackencrypt => encrypt}/plan/plantest/plantest.go (99%) rename languages/golang/{stackencrypt => encrypt}/plan/plantest/plantest_internal_test.go (99%) rename languages/golang/{stackencrypt => encrypt}/plan/plantest/snapshot.go (96%) rename languages/golang/{stackencrypt => encrypt}/plan/plantest/testdata/TestPolicies/audits.golden (100%) rename languages/golang/{stackencrypt => encrypt}/plan/plantest/testdata/TestPolicies/individuals.golden (100%) rename languages/golang/{stackencrypt => encrypt}/plan/policy.go (88%) rename languages/golang/{stackencrypt => encrypt}/policy_plan_test.go (94%) rename languages/golang/{stackencrypt => encrypt}/record.go (92%) rename languages/golang/{stackencrypt => encrypt}/runtime_test.go (99%) rename languages/golang/{stackencrypt => encrypt}/term.go (97%) rename languages/golang/{stackencrypt => encrypt}/testdata/cllw_order.txt (100%) rename languages/golang/{stackencrypt => encrypt}/transport.go (95%) rename languages/golang/{stackencrypt => encrypt}/unit_test.go (99%) rename languages/golang/{stackencrypt => encrypt}/wasm/README.md (100%) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 6cb08638d..6b4c31cb3 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -211,8 +211,8 @@ updates: - /packages/stack-auth/fuzz - /packages/stack-kms/fuzz - /packages/stack-encrypt/fuzz - - /languages/golang/stackencrypt/guest - - /languages/golang/stackauth/guest + - /languages/golang/encrypt/guest + - /languages/golang/auth/guest # Monthly, matching the other two cargo entries. schedule: interval: monthly diff --git a/.github/workflows/tests-golang.yml b/.github/workflows/tests-golang.yml index a15437f37..ca156c389 100644 --- a/.github/workflows/tests-golang.yml +++ b/.github/workflows/tests-golang.yml @@ -122,8 +122,8 @@ jobs: with: workspaces: | . - languages/golang/stackencrypt/guest - languages/golang/stackauth/guest + languages/golang/encrypt/guest + languages/golang/auth/guest # The HTTP-free core compiles for wasm32-wasip1 with no JS-host backend # and no native HTTP/TLS stack in its graph: the invariant the wazero @@ -157,7 +157,7 @@ jobs: # same bytes rather than a stale or rebuilt guest. - name: Record the guests' checksums run: | - for guest in stackencrypt/wasm/stack_encrypt_guest.wasm stackauth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do (cd languages/golang && openssl dgst -sha256 "$guest" | awk '{print $NF}' > "$guest.sha256" && echo "$guest sha256 $(cat "$guest.sha256")") done @@ -166,20 +166,20 @@ jobs: with: name: wasm-guests path: | - languages/golang/stackencrypt/wasm/stack_encrypt_guest.wasm - languages/golang/stackencrypt/wasm/stack_encrypt_guest.wasm.sha256 - languages/golang/stackauth/wasm/stack_auth_guest.wasm - languages/golang/stackauth/wasm/stack_auth_guest.wasm.sha256 + languages/golang/encrypt/wasm/stack_encrypt_guest.wasm + languages/golang/encrypt/wasm/stack_encrypt_guest.wasm.sha256 + languages/golang/auth/wasm/stack_auth_guest.wasm + languages/golang/auth/wasm/stack_auth_guest.wasm.sha256 if-no-files-found: error retention-days: 1 # Format, vet and hermetic tests on amd64 and 386. The guest's memory # lock is best effort, so its test skips where RLIMIT_MEMLOCK refuses - # it; CI raises the limit and sets STACKENCRYPT_TESTS_REQUIRE_LOCK so + # it; CI raises the limit and sets STACK_ENCRYPT_TESTS_REQUIRE_LOCK so # the skip is an error here. - name: Go binding env: - STACKENCRYPT_TESTS_REQUIRE_LOCK: "1" + STACK_ENCRYPT_TESTS_REQUIRE_LOCK: "1" run: | ulimit -l "$(ulimit -H -l)" echo "RLIMIT_MEMLOCK: $(ulimit -l) KiB" @@ -260,7 +260,7 @@ jobs: - name: The guests are the ones Linux built and checked run: | cd languages/golang - for guest in stackencrypt/wasm/stack_encrypt_guest.wasm stackauth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do want=$(cat "$guest.sha256") got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}') if [ "$want" != "$got" ]; then @@ -326,7 +326,7 @@ jobs: - name: The guests are the ones the wasi-check job built and checked run: | cd languages/golang - for guest in stackencrypt/wasm/stack_encrypt_guest.wasm stackauth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do want=$(cat "$guest.sha256") got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}') if [ "$want" != "$got" ]; then @@ -340,7 +340,7 @@ jobs: # call `liveClient`, and not all of them are named `TestLive*`. - name: Go live tests working-directory: languages/golang - run: CGO_ENABLED=0 go test -v ./stackencrypt/... + run: CGO_ENABLED=0 go test -v ./encrypt/... - name: stack-encrypt examples run: | diff --git a/.gitignore b/.gitignore index 65da48ae3..eccbed69f 100644 --- a/.gitignore +++ b/.gitignore @@ -108,7 +108,7 @@ mutants.out/ # The Go module's embedded WASI guests: build outputs of `mise run # wasm:guest:build` and `mise run wasm:auth-guest:build`. -languages/golang/stackencrypt/wasm/*.wasm -languages/golang/stackauth/wasm/*.wasm -languages/golang/stackencrypt/wasm/*.sha256 -languages/golang/stackauth/wasm/*.sha256 +languages/golang/encrypt/wasm/*.wasm +languages/golang/auth/wasm/*.wasm +languages/golang/encrypt/wasm/*.sha256 +languages/golang/auth/wasm/*.sha256 diff --git a/AGENTS.md b/AGENTS.md index ce21666b0..82be8dc52 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,7 +92,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l links are provenance only. - `packages/stack-auth`, `packages/stack-profile`, `packages/stack-kms`, `packages/stack-encrypt`, `packages/stack-encrypt-derive`, `packages/stack-guest-abi`: The Rust crates imported from `cipherstash/cipherstash-suite` with their history — `stack-auth` and `stack-profile` (published to crates.io, one version group), `stack-kms` (published to crates.io from 0.1.0, its own version group, re-exported by `stack-encrypt` as `stack_encrypt::kms`), `stack-encrypt` and `stack-encrypt-derive` (published to crates.io from 0.1.0, one version group; `eql-bindings`' `stack-encrypt` feature depends on them from the registry), and `stack-guest-abi` (`publish = false`). They are the members of the **root Cargo workspace**, with the three node binding crates below. See "Working on the Rust crates". - `languages/typescript/packages/auth`, `languages/typescript/packages/profile`, `languages/typescript/packages/stack-auth-wasm`: The node bindings of those crates. `@cipherstash/auth` (napi-rs v2) and its six `platforms/*` packages are published to npm from this repository by `release.yml` (`auth-artifacts`, `publish-auth`); a change to what it ships, the `stack-auth` crate included, needs an `@cipherstash/auth` changeset (`require-auth-npm-changeset.yml`). `@cipherstash/profile` and its platforms are private and never published; `@cipherstash/stack-auth-wasm` is private and builds the wasm that `@cipherstash/auth` ships. Their `build` and `test` scripts never invoke cargo; `build:native`, `build:debug` and `test:cargo` do. -- `languages/golang`: The Go module (`stackencrypt`, `stackauth`, `internal`), a wazero host with no cgo. Its two WASI guests (`*/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; the `.wasm` files they embed are gitignored. There is no Go release process yet. +- `languages/golang`: The Go module (`encrypt`, `auth`, `internal`), a wazero host with no cgo. Its two WASI guests (`*/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; the `.wasm` files they embed are gitignored. There is no Go release process yet. - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `languages/typescript/examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). diff --git a/Cargo.toml b/Cargo.toml index 4ad564ff7..3da394ab4 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -27,8 +27,8 @@ exclude = [ "packages/stack-encrypt/fuzz", # WASI guests for the Go module, built through `wasm:guest:build` and # `wasm:auth-guest:build`. - "languages/golang/stackencrypt/guest", - "languages/golang/stackauth/guest", + "languages/golang/encrypt/guest", + "languages/golang/auth/guest", ] [workspace.package] diff --git a/SECURITY.md b/SECURITY.md index e63869144..821c8fad4 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -33,7 +33,7 @@ It also carries the source of five Rust crates published to crates.io, `packages/stack-profile`) and **`stack-kms`**, **`stack-encrypt`** and **`stack-encrypt-derive`** (`packages/stack-kms`, `packages/stack-encrypt`, `packages/stack-encrypt-derive`), and of the **Go module** at -`languages/golang` (`stackencrypt` and `stackauth`, over WASI guests built +`languages/golang` (`encrypt` and `auth`, over WASI guests built from the stack-* crates), which has no release yet. All of these are in scope for security reports on the same terms as the npm packages above. diff --git a/languages/golang/stackauth/README.md b/languages/golang/auth/README.md similarity index 79% rename from languages/golang/stackauth/README.md rename to languages/golang/auth/README.md index 396939df3..277251666 100644 --- a/languages/golang/stackauth/README.md +++ b/languages/golang/auth/README.md @@ -1,9 +1,9 @@ -# stackauth +# auth The Go binding of the developer profile — the directory `stash auth login` writes — read through the `stack-profile` Rust crate running inside a WASI guest under [wazero], with `CGO_ENABLED=0`. It is the credential half of -the Go SDK: it hands a [`stackencrypt`](../stackencrypt) client its client +the Go SDK: it hands a [`encrypt`](../encrypt) client its client key and its bearer token without either package re-deriving the profile's layout, and without either importing the other. @@ -16,9 +16,9 @@ cross-process refresh lock for device sessions. ## Use -Most applications never call this package directly: a `stackencrypt` +Most applications never call this package directly: a `encrypt` client built with `NewClient(ctx)` and no options resolves its credentials with -`stackencrypt.AutoCredentials`, which reads the environment first and then +`encrypt.AutoCredentials`, which reads the environment first and then the profile, through this package. Use it directly to take the profile apart yourself: @@ -26,12 +26,12 @@ apart yourself: import ( "context" - "github.com/cipherstash/stack/languages/golang/stackauth" - "github.com/cipherstash/stack/languages/golang/stackencrypt" + "github.com/cipherstash/stack/languages/golang/auth" + "github.com/cipherstash/stack/languages/golang/encrypt" ) func run(ctx context.Context) error { - profile, err := stackauth.Resolve(ctx) // CS_CONFIG_PATH, else ~/.cipherstash + profile, err := auth.Resolve(ctx) // CS_CONFIG_PATH, else ~/.cipherstash if err != nil { return err } @@ -39,7 +39,7 @@ func run(ctx context.Context) error { workspace, err := profile.CurrentWorkspaceStore(ctx) if err != nil { - return err // stackauth.ErrNoCurrentWorkspace: run `stash auth login` + return err // auth.ErrNoCurrentWorkspace: run `stash auth login` } clientID, clientKey, err := workspace.SecretKey(ctx) if err != nil { @@ -50,9 +50,9 @@ func run(ctx context.Context) error { return err } defer source.Close() - client, err := stackencrypt.NewClient(ctx, + client, err := encrypt.NewClient(ctx, // The key is consumed and wiped by NewClient. - stackencrypt.WithCredentials(stackencrypt.NewCredentials(clientID, clientKey, source)), + encrypt.WithCredentials(encrypt.NewCredentials(clientID, clientKey, source)), ) if err != nil { return err @@ -63,31 +63,31 @@ func run(ctx context.Context) error { } ``` -`stackauth.ClientKey` and `stackencrypt.ClientKey` are one type, so the +`auth.ClientKey` and `encrypt.ClientKey` are one type, so the key goes straight from the profile into the credentials. The profile and the strategy are the caller's: the client asks the strategy for a token on every request but never closes it, so both stay open until the client is closed (the deferred calls above run in that order). -A stackencrypt client takes its token only from a strategy, never a raw +A encrypt client takes its token only from a strategy, never a raw string: a raw token cannot be refreshed when it expires, and would bypass the cross-process lock a device-session refresh holds with the `stash` CLI (the IdP revokes a whole refresh-token chain when one is used twice). `workspace.Token(ctx)` still reads the stored token, for inspection. With no profile directory at all (CI, a container, a server authenticating -by federation), `stackauth.OpenWithoutProfile(ctx)` runs the guest with +by federation), `auth.OpenWithoutProfile(ctx)` runs the guest with nothing mounted: the access-key and OIDC strategies work, and every profile read is `ErrNoProfile`. `profile.AccessKey(ctx, crn, key)`, `profile.OIDC(ctx, crn, provider)`, and -`profile.Auto(ctx)` also return strategies that `stackencrypt.NewCredentials` +`profile.Auto(ctx)` also return strategies that `encrypt.NewCredentials` takes. `Auto` checks `CS_CLIENT_ACCESS_KEY` and `CS_WORKSPACE_CRN` first, then the current workspace's stored device session. The OIDC provider is a one-method `Token(context.Context) (string, error)` interface, called on every token fetch for the JWT of the user the call is for; each distinct JWT is exchanged once while its CTS token lasts. Use -`stackauth.OAuth2TokenSource(source)` to adapt a +`auth.OAuth2TokenSource(source)` to adapt a `golang.org/x/oauth2.TokenSource`. `WithAuthBaseURL(url)` overrides service discovery for local tests or a custom CTS host; `WithCacheCapacity(n)` sets how many users' CTS tokens an OIDC strategy keeps (1024 unless set), sized to @@ -107,7 +107,7 @@ session refresh call, on the path `ProfileStore.LockPath` names. A fresh token is read without the lock; on refresh the guest re-reads auth.json after acquisition and saves refreshed tokens before release. -The crypto guest behind `stackencrypt` is not widened by this package +The crypto guest behind `encrypt` is not widened by this package existing: it still has no filesystem and no environment. ## Build diff --git a/languages/golang/stackauth/clientkey.go b/languages/golang/auth/clientkey.go similarity index 51% rename from languages/golang/stackauth/clientkey.go rename to languages/golang/auth/clientkey.go index 30553833b..e880f7a26 100644 --- a/languages/golang/stackauth/clientkey.go +++ b/languages/golang/auth/clientkey.go @@ -1,12 +1,12 @@ -package stackauth +package auth import "github.com/cipherstash/stack/languages/golang/internal/guest" // ClientKey is the ZeroKMS client key as [ProfileStore.SecretKey] reads it // out of secretkey.json: opaque (it prints a redaction under every verb and // hands its bytes to no caller) and wiped once consumed. It is the same -// type as stackencrypt.ClientKey, by identity, so a key read here goes -// straight into stackencrypt.NewCredentials. This package does not import -// stackencrypt: a binary that only wants the profile does not carry the -// crypto guest. (stackencrypt imports this one, for AutoCredentials.) +// type as encrypt.ClientKey, by identity, so a key read here goes +// straight into encrypt.NewCredentials. This package does not import +// encrypt: a binary that only wants the profile does not carry the +// crypto guest. (encrypt imports this one, for AutoCredentials.) type ClientKey = guest.ClientKey diff --git a/languages/golang/stackauth/doc.go b/languages/golang/auth/doc.go similarity index 85% rename from languages/golang/stackauth/doc.go rename to languages/golang/auth/doc.go index 46feff9b0..e98bc4db2 100644 --- a/languages/golang/stackauth/doc.go +++ b/languages/golang/auth/doc.go @@ -1,4 +1,4 @@ -// Package stackauth is the Go binding of the developer profile: the +// Package auth is the Go binding of the developer profile: the // directory `stash auth login` writes (~/.cipherstash, or CS_CONFIG_PATH), // read through the stack-profile crate running unmodified inside a WASI // guest under wazero (CGO_ENABLED=0), so the on-disk layout is never @@ -21,27 +21,27 @@ // one workspace ([ProfileStore.WorkspaceStore], // [ProfileStore.CurrentWorkspaceStore]), and the typed reads of the files a // workspace holds: [ProfileStore.SecretKey] hands out the ZeroKMS client key -// as the opaque [ClientKey] that stackencrypt.NewCredentials takes, +// as the opaque [ClientKey] that encrypt.NewCredentials takes, // [ProfileStore.Token] the stored access token, [ProfileStore.DeviceIdentity] // the identity the CLI created. [ProfileStore.Close] releases the guest; // stores scoped from it are closed with it. // // For authentication and refresh, use [ProfileStore.AccessKey], // [ProfileStore.OIDC], [ProfileStore.DeviceSession], or [ProfileStore.Auto]. -// Each returns a [Strategy], which is what stackencrypt.NewCredentials takes -// for the bearer token: the only way a token reaches a stackencrypt client. +// Each returns a [Strategy], which is what encrypt.NewCredentials takes +// for the bearer token: the only way a token reaches a encrypt client. // A raw token — [ProfileStore.Token]'s included — cannot be refreshed when // it expires, and would bypass the cross-process lock a device-session -// refresh holds with the CLI, so stackencrypt does not accept one. A +// refresh holds with the CLI, so encrypt does not accept one. A // strategy lives in its store's guest, and closing the store closes it. A -// stackencrypt client given one never closes the strategy or the store: +// encrypt client given one never closes the strategy or the store: // both are the caller's, and stay open until the client is closed. // [OAuth2TokenSource] adapts an existing golang.org/x/oauth2.TokenSource // into the OIDC provider interface. // // # Why a second guest // -// The crypto guest behind stackencrypt has no filesystem and no +// The crypto guest behind encrypt has no filesystem and no // environment: a bug or compromise inside it cannot read credentials off // disk. Mounting the profile into it would trade that away, and it is the // module that handles plaintext and data keys. So the profile lives in its @@ -63,9 +63,9 @@ // # Memory // // The guest's memory holds the client key and the token while a read is -// in flight. It is supplied the way stackencrypt's is — reserved once so it +// in flight. It is supplied the way encrypt's is — reserved once so it // never moves, locked in RAM and excluded from core dumps where the // platform allows, wiped before release, none of it depending on Close // running — and [ProfileStore.MemoryLocked] reports whether the lock was // granted. [RequireLockedMemory] makes a refused lock an error from Open. -package stackauth +package auth diff --git a/languages/golang/stackauth/errors.go b/languages/golang/auth/errors.go similarity index 95% rename from languages/golang/stackauth/errors.go rename to languages/golang/auth/errors.go index b5a2746f2..9dfbb7ab9 100644 --- a/languages/golang/stackauth/errors.go +++ b/languages/golang/auth/errors.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "errors" @@ -9,7 +9,7 @@ import ( // Failure kinds the profile reports. The guest reports a status code from // the one table every guest shares, so these are the sentinels of the // shared decoder exposed under this package's names; an error from -// stackencrypt of the same kind is the same value. +// encrypt of the same kind is the same value. var ( // ErrNotFound is a profile file that does not exist in the store asked: // no secretkey.json, auth.json or device.json there. For the workspace's @@ -60,5 +60,5 @@ var ( // ErrNoProfile is a profile directory that does not exist: nothing has // logged in on this machine, or CS_CONFIG_PATH names the wrong place. - ErrNoProfile = errors.New("stackauth: no profile directory; run `stash auth login`") + ErrNoProfile = errors.New("auth: no profile directory; run `stash auth login`") ) diff --git a/languages/golang/stackauth/guest.go b/languages/golang/auth/guest.go similarity index 93% rename from languages/golang/stackauth/guest.go rename to languages/golang/auth/guest.go index dc7239267..2d157f5b0 100644 --- a/languages/golang/stackauth/guest.go +++ b/languages/golang/auth/guest.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" @@ -33,7 +33,7 @@ const guestRoot = "/profile" // ErrGuestNotBuilt is returned by Open when no guest module is embedded and // none was supplied with [WithGuest]. -var ErrGuestNotBuilt = errors.New("stackauth: guest module not built — run `mise run wasm:auth-guest:build`") +var ErrGuestNotBuilt = errors.New("auth: guest module not built — run `mise run wasm:auth-guest:build`") func embeddedGuest() ([]byte, error) { wasm, err := guestFS.ReadFile(guestPath) @@ -82,7 +82,7 @@ type instance struct { // because wazero's default is a fixed seed: the Rust runtime draws through // it (its hash maps are seeded from it, for one), and nothing a guest does // should be predictable across instances. The clocks are the system's for -// the same reason they are in stackencrypt: a deterministic default is the +// the same reason they are in encrypt: a deterministic default is the // wrong default for anything that reads time. Each is pinned by a test. // // A nil mount is a guest with no directory at all ([OpenWithoutProfile]): @@ -123,11 +123,11 @@ func newInstance(ctx context.Context, wasm []byte, hostDir string, policy guest. return nil, err } if _, err := wasi_snapshot_preview1.Instantiate(ctx, runtime); err != nil { - return fail(fmt.Errorf("stackauth: instantiating WASI: %w", err)) + return fail(fmt.Errorf("auth: instantiating WASI: %w", err)) } transport := newAuthTransport(rt) if err := transport.instantiate(ctx, runtime); err != nil { - return fail(fmt.Errorf("stackauth: instantiating host transport: %w", err)) + return fail(fmt.Errorf("auth: instantiating host transport: %w", err)) } mem := guest.NewAllocator(policy) // The guest is a reactor (cdylib): no _start. wazero runs _initialize @@ -142,7 +142,7 @@ func newInstance(ctx context.Context, wasm []byte, hostDir string, policy guest. if g := mem.GrowthRefusal(); g.Refused != 0 { return fail(fmt.Errorf("%w: %w", guest.MemoryLockError(g.Reason), err)) } - return fail(fmt.Errorf("stackauth: instantiating guest: %w", err)) + return fail(fmt.Errorf("auth: instantiating guest: %w", err)) } if policy == guest.Strict { if lerr := mem.LockError(); lerr != nil { @@ -172,7 +172,7 @@ func newInstance(ctx context.Context, wasm []byte, hostDir string, policy guest. } for name, slot := range exports { if *slot = module.ExportedFunction(name); *slot == nil { - return fail(fmt.Errorf("stackauth: guest is missing export %s", name)) + return fail(fmt.Errorf("auth: guest is missing export %s", name)) } } return inst, nil diff --git a/languages/golang/stackauth/guest/.gitignore b/languages/golang/auth/guest/.gitignore similarity index 100% rename from languages/golang/stackauth/guest/.gitignore rename to languages/golang/auth/guest/.gitignore diff --git a/languages/golang/stackauth/guest/Cargo.lock b/languages/golang/auth/guest/Cargo.lock similarity index 100% rename from languages/golang/stackauth/guest/Cargo.lock rename to languages/golang/auth/guest/Cargo.lock diff --git a/languages/golang/stackauth/guest/Cargo.toml b/languages/golang/auth/guest/Cargo.toml similarity index 96% rename from languages/golang/stackauth/guest/Cargo.toml rename to languages/golang/auth/guest/Cargo.toml index 8bb48dba1..d3928e995 100644 --- a/languages/golang/stackauth/guest/Cargo.toml +++ b/languages/golang/auth/guest/Cargo.toml @@ -1,6 +1,6 @@ # The credential guest: `stack-profile` and `stack-auth` strategies under # WASI/wazero, embedded by the Go package -# `stackauth` one directory up. ADR-0005 in packages/stack-encrypt/docs/adr. +# `auth` one directory up. ADR-0005 in packages/stack-encrypt/docs/adr. # # A second module rather than the crypto guest widened: this one is given a # directory of credentials, and that one handles plaintext and data keys. diff --git a/languages/golang/stackauth/guest/src/abi.rs b/languages/golang/auth/guest/src/abi.rs similarity index 100% rename from languages/golang/stackauth/guest/src/abi.rs rename to languages/golang/auth/guest/src/abi.rs diff --git a/languages/golang/stackauth/guest/src/auth.rs b/languages/golang/auth/guest/src/auth.rs similarity index 99% rename from languages/golang/stackauth/guest/src/auth.rs rename to languages/golang/auth/guest/src/auth.rs index beaee521f..cbdbef6a8 100644 --- a/languages/golang/stackauth/guest/src/auth.rs +++ b/languages/golang/auth/guest/src/auth.rs @@ -34,7 +34,7 @@ enum Config { provider: u32, base_url: Option, /// How many distinct JWTs keep a CTS token; the strategy's default - /// when absent. See `stackauth.WithCacheCapacity`. + /// when absent. See `auth.WithCacheCapacity`. cache_capacity: Option, }, DeviceSession { diff --git a/languages/golang/stackauth/guest/src/headers.rs b/languages/golang/auth/guest/src/headers.rs similarity index 100% rename from languages/golang/stackauth/guest/src/headers.rs rename to languages/golang/auth/guest/src/headers.rs diff --git a/languages/golang/stackauth/guest/src/host.rs b/languages/golang/auth/guest/src/host.rs similarity index 100% rename from languages/golang/stackauth/guest/src/host.rs rename to languages/golang/auth/guest/src/host.rs diff --git a/languages/golang/stackauth/guest/src/lib.rs b/languages/golang/auth/guest/src/lib.rs similarity index 97% rename from languages/golang/stackauth/guest/src/lib.rs rename to languages/golang/auth/guest/src/lib.rs index 421057ef0..fb9986d4c 100644 --- a/languages/golang/stackauth/guest/src/lib.rs +++ b/languages/golang/auth/guest/src/lib.rs @@ -33,7 +33,7 @@ //! WASI guest module exposing the developer profile — //! [`stack_profile::ProfileStore`] over one mounted directory — to non-Rust //! hosts. Built for `wasm32-wasip1` and embedded by the Go package -//! `stackauth` in the parent directory (wazero host, `CGO_ENABLED=0`). +//! `auth` in the parent directory (wazero host, `CGO_ENABLED=0`). //! ADR-0005 in `packages/stack-encrypt/docs/adr` is the decision this //! module implements; `docs/plans/stack-encrypt-go-bindings.md` sequences //! it. @@ -49,7 +49,7 @@ //! the mount does not contain: every path is built by `stack-profile` from //! a store directory and a validated filename or workspace id. //! -//! The crypto guest (`bindings/go/stackencrypt/guest`) is unchanged by +//! The crypto guest (`languages/golang/encrypt/guest`) is unchanged by //! this one existing. It has no filesystem and no environment, and it //! handles plaintext and data keys; this module can reach one directory of //! credentials. The split is what makes both of those true at once. diff --git a/languages/golang/stackauth/guest/src/ops.rs b/languages/golang/auth/guest/src/ops.rs similarity index 99% rename from languages/golang/stackauth/guest/src/ops.rs rename to languages/golang/auth/guest/src/ops.rs index 6e5ea6388..4e684cb82 100644 --- a/languages/golang/stackauth/guest/src/ops.rs +++ b/languages/golang/auth/guest/src/ops.rs @@ -39,7 +39,7 @@ use crate::status::{status_for_profile, STATUS_ENCODING, STATUS_INTERNAL}; /// and `stack-kms`'s `SecretKey` reads it: the ZeroKMS client id and the /// client key material, standard padded base64. The key crosses to the /// host in the form the file holds, which is one of the two forms -/// `stackencrypt`'s config takes — as bytes, so the host gets a slice it +/// `encrypt`'s config takes — as bytes, so the host gets a slice it /// can wipe rather than a string it cannot. What is deserialized here is /// wiped when it drops; what is moved out of it into the codec value is /// wiped by the codec's own protected types. diff --git a/languages/golang/stackauth/guest/src/status.rs b/languages/golang/auth/guest/src/status.rs similarity index 100% rename from languages/golang/stackauth/guest/src/status.rs rename to languages/golang/auth/guest/src/status.rs diff --git a/languages/golang/stackauth/lock_unix.go b/languages/golang/auth/lock_unix.go similarity index 85% rename from languages/golang/stackauth/lock_unix.go rename to languages/golang/auth/lock_unix.go index a8bc65c54..2d193c604 100644 --- a/languages/golang/stackauth/lock_unix.go +++ b/languages/golang/auth/lock_unix.go @@ -1,6 +1,6 @@ //go:build !windows -package stackauth +package auth import ( "context" @@ -16,7 +16,7 @@ import ( func withRefreshLock(ctx context.Context, path string, run func() error) error { f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o600) //nolint:gosec // path is inside the caller's profile directory if err != nil { - return fmt.Errorf("stackauth: open refresh lock: %w", err) + return fmt.Errorf("auth: open refresh lock: %w", err) } defer f.Close() for { @@ -28,7 +28,7 @@ func withRefreshLock(ctx context.Context, path string, run func() error) error { break } if !errors.Is(err, unix.EWOULDBLOCK) && !errors.Is(err, unix.EAGAIN) { - return fmt.Errorf("stackauth: acquire refresh lock: %w", err) + return fmt.Errorf("auth: acquire refresh lock: %w", err) } select { case <-ctx.Done(): diff --git a/languages/golang/stackauth/lock_windows.go b/languages/golang/auth/lock_windows.go similarity index 87% rename from languages/golang/stackauth/lock_windows.go rename to languages/golang/auth/lock_windows.go index c327a8336..1d901ef2a 100644 --- a/languages/golang/stackauth/lock_windows.go +++ b/languages/golang/auth/lock_windows.go @@ -1,6 +1,6 @@ //go:build windows -package stackauth +package auth import ( "context" @@ -17,7 +17,7 @@ import ( func withRefreshLock(ctx context.Context, path string, run func() error) error { f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o600) //nolint:gosec // path is inside the caller's profile directory if err != nil { - return fmt.Errorf("stackauth: open refresh lock: %w", err) + return fmt.Errorf("auth: open refresh lock: %w", err) } defer f.Close() h := windows.Handle(f.Fd()) @@ -31,7 +31,7 @@ func withRefreshLock(ctx context.Context, path string, run func() error) error { break } if !errors.Is(err, windows.ERROR_LOCK_VIOLATION) { - return fmt.Errorf("stackauth: acquire refresh lock: %w", err) + return fmt.Errorf("auth: acquire refresh lock: %w", err) } select { case <-ctx.Done(): diff --git a/languages/golang/stackauth/mount.go b/languages/golang/auth/mount.go similarity index 99% rename from languages/golang/stackauth/mount.go rename to languages/golang/auth/mount.go index f4ef72e3d..718b87d79 100644 --- a/languages/golang/stackauth/mount.go +++ b/languages/golang/auth/mount.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "errors" diff --git a/languages/golang/stackauth/oauth2.go b/languages/golang/auth/oauth2.go similarity index 76% rename from languages/golang/stackauth/oauth2.go rename to languages/golang/auth/oauth2.go index 37056a790..be7f83b01 100644 --- a/languages/golang/stackauth/oauth2.go +++ b/languages/golang/auth/oauth2.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" @@ -12,14 +12,14 @@ import ( func OAuth2TokenSource(source oauth2.TokenSource) OIDCProvider { return OIDCProviderFunc(func(context.Context) (string, error) { if source == nil { - return "", errors.New("stackauth: nil oauth2 token source") + return "", errors.New("auth: nil oauth2 token source") } token, err := source.Token() if err != nil { return "", err } if token == nil || token.AccessToken == "" { - return "", errors.New("stackauth: oauth2 source returned no access token") + return "", errors.New("auth: oauth2 source returned no access token") } return token.AccessToken, nil }) diff --git a/languages/golang/stackauth/store.go b/languages/golang/auth/store.go similarity index 97% rename from languages/golang/stackauth/store.go rename to languages/golang/auth/store.go index cc04be9d4..b5baea7a6 100644 --- a/languages/golang/stackauth/store.go +++ b/languages/golang/auth/store.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" @@ -42,7 +42,7 @@ func WithGuest(wasm []byte) Option { // memory cannot be locked in RAM or, on Linux, excluded from core dumps, // instead of continuing with memory that may be swapped or dumped and // reporting so through [ProfileStore.MemoryLocked]. It holds for the life -// of the store, as stackencrypt's WithRequireLockedMemory does for a +// of the store, as encrypt's WithRequireLockedMemory does for a // client. func RequireLockedMemory() Option { return func(o *options) { o.requireLocked = true } @@ -159,7 +159,7 @@ func (s *ProfileStore) hostPath(guestPath string) string { // MemoryLocked reports whether the guest's memory — where the client key // and the token pass through — is locked in RAM and, on Linux, excluded -// from core dumps. See stackencrypt's Client.MemoryLocked for what false +// from core dumps. See encrypt's Client.MemoryLocked for what false // means and what to do about it. func (s *ProfileStore) MemoryLocked() bool { return s.root.inst.mem.LockError() == nil } @@ -175,7 +175,7 @@ func (s *ProfileStore) MemoryLockError() error { // String prints the store's directory and memory state. Nothing secret. func (s *ProfileStore) String() string { - return fmt.Sprintf("stackauth.ProfileStore{dir: %s, memory: %s}", s.Dir(), s.root.inst.mem) + return fmt.Sprintf("auth.ProfileStore{dir: %s, memory: %s}", s.Dir(), s.root.inst.mem) } // LogValue implements slog.LogValuer: the directory, and the memory state. @@ -246,7 +246,7 @@ func (s *ProfileStore) callArgs(ctx context.Context, fn export, args ...guest.Ar // Under RequireLockedMemory a growth that cannot be locked is refused, // and the guest sees only a failed allocation — or, for an allocation // of its own, aborts, and the trap closed the profile above. Name the - // real cause either way, as stackencrypt's Client.call does. The + // real cause either way, as encrypt's Client.call does. The // refusal is this call's, not the store's: the range went back unused, // so MemoryLocked still holds. if g := r.inst.mem.GrowthRefusal(); err != nil && g.Refused != growth.Refused { @@ -355,7 +355,7 @@ func (s *ProfileStore) LockPath(ctx context.Context, filename string) (string, e // SecretKey reads secretkey.json in this store (a workspace store; the // root holds none): the ZeroKMS client id and the client key, the latter as -// the opaque [ClientKey] stackencrypt.NewCredentials takes. The transport copy +// the opaque [ClientKey] encrypt.NewCredentials takes. The transport copy // of the key is wiped once it is in the ClientKey; the key is then the // caller's to consume. func (s *ProfileStore) SecretKey(ctx context.Context) (clientID string, key *ClientKey, err error) { diff --git a/languages/golang/stackauth/store_test.go b/languages/golang/auth/store_test.go similarity index 99% rename from languages/golang/stackauth/store_test.go rename to languages/golang/auth/store_test.go index a07374065..ce5370e1d 100644 --- a/languages/golang/stackauth/store_test.go +++ b/languages/golang/auth/store_test.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" diff --git a/languages/golang/stackauth/strategy.go b/languages/golang/auth/strategy.go similarity index 97% rename from languages/golang/stackauth/strategy.go rename to languages/golang/auth/strategy.go index 2ed0dd196..b44b494bc 100644 --- a/languages/golang/stackauth/strategy.go +++ b/languages/golang/auth/strategy.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" @@ -48,9 +48,9 @@ func strategyConfig(opts []StrategyOption) strategyOptions { } // Strategy is a Rust stack-auth strategy retained inside the credential -// guest: the source of the bearer token stackencrypt.NewCredentials takes. +// guest: the source of the bearer token encrypt.NewCredentials takes. // Close drops its cached credential; closing the parent profile closes all -// its strategies. It is the caller's to close: a stackencrypt client that +// its strategies. It is the caller's to close: a encrypt client that // was given it asks it for tokens but never closes it, so it must stay open // until the client is closed. type Strategy struct { @@ -65,7 +65,7 @@ type Strategy struct { func (s *ProfileStore) newStrategy(ctx context.Context, config any, device bool) (*Strategy, error) { data, err := json.Marshal(config) if err != nil { - return nil, fmt.Errorf("stackauth: encode strategy: %w", err) + return nil, fmt.Errorf("auth: encode strategy: %w", err) } defer guest.Wipe(data) out, err := s.callArgs(ctx, func(i *instance) api.Function { return i.authNew }, guest.BufArg(data)) diff --git a/languages/golang/stackauth/strategy_test.go b/languages/golang/auth/strategy_test.go similarity index 99% rename from languages/golang/stackauth/strategy_test.go rename to languages/golang/auth/strategy_test.go index a98fca173..7fa1c3d95 100644 --- a/languages/golang/stackauth/strategy_test.go +++ b/languages/golang/auth/strategy_test.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" diff --git a/languages/golang/stackauth/token.go b/languages/golang/auth/token.go similarity index 99% rename from languages/golang/stackauth/token.go rename to languages/golang/auth/token.go index 49dccbb3c..3e9825d6c 100644 --- a/languages/golang/stackauth/token.go +++ b/languages/golang/auth/token.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" diff --git a/languages/golang/stackauth/transport.go b/languages/golang/auth/transport.go similarity index 98% rename from languages/golang/stackauth/transport.go rename to languages/golang/auth/transport.go index 0cda9cbb0..78a894548 100644 --- a/languages/golang/stackauth/transport.go +++ b/languages/golang/auth/transport.go @@ -1,4 +1,4 @@ -package stackauth +package auth import ( "context" @@ -106,7 +106,7 @@ func (t *authTransport) instantiate(ctx context.Context, r wazero.Runtime) error return err } -// send is the same host import contract used by stackencrypt: four input +// send is the same host import contract used by encrypt: four input // buffers and two output slots, with a negative status on transport failure. func (t *authTransport) send(ctx context.Context, m api.Module, methodPtr, methodLen, urlPtr, urlLen, headersPtr, headersLen, bodyPtr, bodyLen uint32, diff --git a/languages/golang/stackauth/wasm/README.md b/languages/golang/auth/wasm/README.md similarity index 100% rename from languages/golang/stackauth/wasm/README.md rename to languages/golang/auth/wasm/README.md diff --git a/languages/golang/stackencrypt/README.md b/languages/golang/encrypt/README.md similarity index 90% rename from languages/golang/stackencrypt/README.md rename to languages/golang/encrypt/README.md index 122e86e21..849c635dc 100644 --- a/languages/golang/stackencrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -9,12 +9,12 @@ so there is no cgo and no separate Go port of the cryptography. The package reference is on [pkg.go.dev]; this README covers connecting, what happens to key material, and the errors. -[pkg.go.dev]: https://pkg.go.dev/github.com/cipherstash/stack/languages/golang/stackencrypt +[pkg.go.dev]: https://pkg.go.dev/github.com/cipherstash/stack/languages/golang/encrypt ## Install ```sh -go get github.com/cipherstash/stack/languages/golang/stackencrypt +go get github.com/cipherstash/stack/languages/golang/encrypt ``` Go 1.25 or later. @@ -29,13 +29,13 @@ safe for concurrent use. import ( "context" - "github.com/cipherstash/stack/languages/golang/stackencrypt" + "github.com/cipherstash/stack/languages/golang/encrypt" ) func run(ctx context.Context) error { - client, err := stackencrypt.NewClient(ctx) + client, err := encrypt.NewClient(ctx) if err != nil { - return err // stackencrypt.ErrNoCredentials: nothing configured + return err // encrypt.ErrNoCredentials: nothing configured } defer client.Close() @@ -49,17 +49,17 @@ means: the deadline and cancellation for the work this call does. `NewClient` makes one ZeroKMS round trip, to load the client's default keyset, and `ctx` bounds that request. It has nothing to do with an *encryption* context, which is the value a field is sealed under; that is -`stackencrypt.Context`. Every method that can reach ZeroKMS takes a +`encrypt.Context`. Every method that can reach ZeroKMS takes a `context.Context` first, for the same reason. Everything else is a functional option, and each has a default: ```go -client, err := stackencrypt.NewClient(ctx, - stackencrypt.WithCredentials(stackencrypt.OIDCFederation(crn, provider)), - stackencrypt.WithTransport(rt), - stackencrypt.WithKeysetCacheSize(4096), - stackencrypt.WithRequireLockedMemory(), +client, err := encrypt.NewClient(ctx, + encrypt.WithCredentials(encrypt.OIDCFederation(crn, provider)), + encrypt.WithTransport(rt), + encrypt.WithKeysetCacheSize(4096), + encrypt.WithRequireLockedMemory(), ) ``` @@ -103,15 +103,15 @@ Rust client reads them, so a value exported there for another tool is worth checking. Resolution happens host-side, in Go. The profile and the token strategies -run in `stackauth`'s credential guest; the crypto guest that holds the +run in `auth`'s credential guest; the crypto guest that holds the keys is still given no environment and no filesystem. The credential guest lives as long as the client, and `Close` releases it. To supply the credentials yourself, pass `NewCredentials` with a client -id, a client key and a `stackauth` strategy for the token: +id, a client key and a `auth` strategy for the token: ```go -store, err := stackauth.OpenWithoutProfile(ctx) // or stackauth.Resolve(ctx) for the profile +store, err := auth.OpenWithoutProfile(ctx) // or auth.Resolve(ctx) for the profile if err != nil { return err } @@ -121,8 +121,8 @@ if err != nil { return err } defer strategy.Close() -client, err := stackencrypt.NewClient(ctx, - stackencrypt.WithCredentials(stackencrypt.NewCredentials(clientID, clientKey, strategy)), +client, err := encrypt.NewClient(ctx, + encrypt.WithCredentials(encrypt.NewCredentials(clientID, clientKey, strategy)), ) if err != nil { return err @@ -135,7 +135,7 @@ refreshes it as it needs to. The store and the strategy stay yours: the client never closes them, so keep them open until `client.Close` has returned, as the deferred calls above do. A nil strategy is refused. -Tokens come only from `stackauth` strategies; there is no way to hand the +Tokens come only from `auth` strategies; there is no way to hand the client a raw bearer token. A raw token cannot be refreshed when it expires, and a source outside the strategies would bypass the cross-process lock a device-session refresh holds with the `stash` CLI. @@ -145,10 +145,10 @@ with the workspace CRN and a provider of the IdP's tokens. The provider is asked on every token fetch for the IdP token of the user the call is for; CTS exchanges each distinct IdP token for a CipherStash one, which is cached for that token until it expires, so one client serves many users and none -rides another's token. `stackauth.OAuth2TokenSource` adapts a +rides another's token. `auth.OAuth2TokenSource` adapts a `golang.org/x/oauth2` source. The client key is found as `AutoCredentials` -finds it. `stackauth` strategy options follow the provider: -`OIDCFederation(crn, provider, stackauth.WithAuthBaseURL(cts))` pins the CTS +finds it. `auth` strategy options follow the provider: +`OIDCFederation(crn, provider, auth.WithAuthBaseURL(cts))` pins the CTS endpoint for these credentials, where `CS_CTS_HOST` would pin it for the whole process. @@ -199,15 +199,15 @@ shows it. The report and the policy cover the credential guest too, which holds the token strategy and which the client key may have passed through. `AutoCredentials` and `OIDCFederation` open it under the client's policy. -With `NewCredentials` it is the `stackauth` store you opened: under +With `NewCredentials` it is the `auth` store you opened: under `WithRequireLockedMemory()`, `NewClient` fails with `ErrMemoryLock` if that store's memory is unlocked, but the store's own policy decides its later -growth. Open it with `stackauth.RequireLockedMemory()` as well to keep it +growth. Open it with `auth.RequireLockedMemory()` as well to keep it locked for the life of the client. Production checklist: assert `MemoryLocked()` at startup, or pass `WithRequireLockedMemory()`, and with `NewCredentials` open the store with -`stackauth.RequireLockedMemory()`. Handling `SIGTERM` for a graceful shutdown is +`auth.RequireLockedMemory()`. Handling `SIGTERM` for a graceful shutdown is ordinary Go practice and worth doing for your own reasons; the SDK does not depend on it and installs no signal handler of its own. @@ -226,7 +226,7 @@ subpackage derives it from what the schema already says about each field in Go: ```go -import "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" +import "github.com/cipherstash/stack/languages/golang/encrypt/plan" type Individual struct { ID int64 @@ -249,14 +249,14 @@ var source = plan.SourceFunc(func(msg any) ([]plan.Fact, error) { var category = plan.Key("fides.data_categories") var Base = plan.FirstOf( - plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(stackencrypt.Equality))), - plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(stackencrypt.Equality, stackencrypt.Match))), + plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(encrypt.Equality))), + plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(encrypt.Equality, encrypt.Match))), plan.When(category.Under("user"), plan.Encrypt(plan.EQL())), ) var Individuals = plan.ForMessage(&Individual{}, plan.Table("individuals"), plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(stackencrypt.Equality)), + plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), plan.Column("medicare_number")), ).OrElse(Base), ) @@ -265,7 +265,7 @@ var Individuals = plan.ForMessage(&Individual{}, plan.Table("individuals"), // plan names a field the struct does not have. var individuals = plan.MustPlanFor(source, Individuals) -records, err := cipher.EncryptRecords(ctx, rows, stackencrypt.WithPlan(individuals)) +records, err := cipher.EncryptRecords(ctx, rows, encrypt.WithPlan(individuals)) ``` A policy fails closed: a field with facts that no rule decides is an error @@ -289,7 +289,7 @@ renamed in the database needs no `Identity`. After new writes go to the new column under the old identity: ```go -plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(stackencrypt.Equality)), +plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), plan.Column("medicare_num"), plan.Identity("medicare_number")) ``` @@ -318,7 +318,7 @@ To add the test: 1. Write a test that calls `plantest.Golden` with your source and your policy: ```go - import "github.com/cipherstash/stack/languages/golang/stackencrypt/plan/plantest" + import "github.com/cipherstash/stack/languages/golang/encrypt/plan/plantest" func TestIndividualsPolicy(t *testing.T) { plantest.Golden(t, source, Individuals) @@ -408,7 +408,7 @@ small and reveals nothing about plaintext or key material. | `ErrForeignKeyset` | A keyset-bound `Cipher` was given another keyset's ciphertext. Open it through the `Client`. | | `ErrEncoding` | Malformed input: a value, ciphertext, plan, context or config refused before any cryptography. | | `ErrTerm` | A term could not be derived, for example match text that yields no tokens. | -| `ErrTransport` | ZeroKMS could not be reached, or the token strategy failed. The strategy's own error is wrapped in it, so `errors.Is` finds that too (a refused refresh is `stackauth.ErrInvalidGrant`). | +| `ErrTransport` | ZeroKMS could not be reached, or the token strategy failed. The strategy's own error is wrapped in it, so `errors.Is` finds that too (a refused refresh is `auth.ErrInvalidGrant`). | | `ErrKMS` | Any other ZeroKMS failure. | | `ErrConflict` | ZeroKMS reported a resource conflict. | | `ErrState` | The client has been closed: by `Close`, by a call its context interrupted, or by a guest trap. | @@ -432,7 +432,7 @@ small and reveals nothing about plaintext or key material. is released. - **Two host imports.** The guest imports exactly one HTTP send, served by your `http.RoundTripper`, and one bearer-token fetch, served by the - credentials' `stackauth` strategy. What crosses the boundary per ZeroKMS call is what would + credentials' `auth` strategy. What crosses the boundary per ZeroKMS call is what would cross TLS anyway. The guest sees no environment and no filesystem. - **Real randomness.** The guest draws IVs and nonces from the process CSPRNG. wazero's default random source is deterministic, so the package diff --git a/languages/golang/stackencrypt/cipher.go b/languages/golang/encrypt/cipher.go similarity index 98% rename from languages/golang/stackencrypt/cipher.go rename to languages/golang/encrypt/cipher.go index 3d92aace6..4266848b7 100644 --- a/languages/golang/stackencrypt/cipher.go +++ b/languages/golang/encrypt/cipher.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -98,7 +98,7 @@ func (cph *Cipher) DecryptElement(ctx context.Context, ct any, aad []byte) (any, // ZeroKMS backend that derives terms server-side settles the same way. func (cph *Cipher) Term(ctx context.Context, value any, context Context, kind TermKind, opts ...Option) (any, error) { if context.isZero() { - return nil, fmt.Errorf("stackencrypt: term context is empty") + return nil, fmt.Errorf("encrypt: term context is empty") } var o termOptions for _, opt := range opts { diff --git a/languages/golang/stackencrypt/client.go b/languages/golang/encrypt/client.go similarity index 97% rename from languages/golang/stackencrypt/client.go rename to languages/golang/encrypt/client.go index 8482cb573..d82a3d326 100644 --- a/languages/golang/stackencrypt/client.go +++ b/languages/golang/encrypt/client.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -100,7 +100,7 @@ func NewClient(ctx context.Context, opts ...ClientOption) (_ *Client, err error) if explicit, ok := creds.(*explicitCredentials); ok && err == nil && explicit.token == nil { // Knowable from the credentials as they were built: the one place // a missing token source is decided. - err = fmt.Errorf("%w: NewCredentials needs a stackauth strategy for the token", ErrEncoding) + err = fmt.Errorf("%w: NewCredentials needs a auth strategy for the token", ErrEncoding) } wasm := cfg.guest if err == nil && wasm == nil { @@ -144,7 +144,7 @@ func NewClient(ctx context.Context, opts ...ClientOption) (_ *Client, err error) // here unlocked; NewCredentials' store is the caller's, opened // however the caller chose. if lockErr := resolved.MemoryLockError(); lockErr != nil { - return nil, fmt.Errorf("stackencrypt: the credentials' memory: %w", lockErr) + return nil, fmt.Errorf("encrypt: the credentials' memory: %w", lockErr) } } encoded, err := encodeConfig(initConfig{ @@ -180,7 +180,7 @@ func NewClient(ctx context.Context, opts ...ClientOption) (_ *Client, err error) }) if err != nil { _ = c.Close() - return nil, fmt.Errorf("stackencrypt: cipher init: %w", err) + return nil, fmt.Errorf("encrypt: cipher init: %w", err) } if len(out) != len(KeysetID{}) { _ = c.Close() @@ -224,7 +224,7 @@ func newClient(inst *instance, t *transport) *Client { // MemoryLocked reports whether the memory the client's key material lives // in — this guest's, where the client key and every loaded index key are, -// and whatever the credentials held it in on the way (stackauth's +// and whatever the credentials held it in on the way (auth's // credential guest, for [AutoCredentials]) — is locked in RAM and, on // Linux, excluded from core dumps. False means a lock was refused (on // Linux, most often RLIMIT_MEMLOCK, which defaults to 64 KiB on many @@ -264,7 +264,7 @@ func (c *Client) memoryState() string { // printed. The state is what an operator reading a startup log needs to // see, and [Client.LogValue] gives it structured form. func (c *Client) String() string { - return fmt.Sprintf("stackencrypt.Client{memory: %s}", c.memoryState()) + return fmt.Sprintf("encrypt.Client{memory: %s}", c.memoryState()) } // LogValue implements slog.LogValuer: a group with memory_locked and, when @@ -289,7 +289,7 @@ type initConfig struct { // client key; the caller wipes it, and the key it was read from. func encodeConfig(cfg initConfig) ([]byte, error) { if cfg.clientID == "" || cfg.clientKey.IsZero() { - return nil, errors.New("stackencrypt: the credentials' client id and client key are required") + return nil, errors.New("encrypt: the credentials' client id and client key are required") } // The key crosses as text: the guest's config parser takes the hex or // base64 form as the CS_CLIENT_KEY variable and secretkey.json hold it. @@ -346,7 +346,7 @@ func (c *Client) Close() error { // programming error and panics; the default keyset is [Client.DefaultKeyset]. func (c *Client) Keyset(sel KeysetSelector) *Cipher { if sel == nil { - panic("stackencrypt: Client.Keyset(nil); the default keyset is Client.DefaultKeyset") + panic("encrypt: Client.Keyset(nil); the default keyset is Client.DefaultKeyset") } return &Cipher{client: c, keyset: sel} } diff --git a/languages/golang/stackencrypt/clientkey.go b/languages/golang/encrypt/clientkey.go similarity index 82% rename from languages/golang/stackencrypt/clientkey.go rename to languages/golang/encrypt/clientkey.go index ca7d3f683..597911a0c 100644 --- a/languages/golang/stackencrypt/clientkey.go +++ b/languages/golang/encrypt/clientkey.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import "github.com/cipherstash/stack/languages/golang/internal/guest" @@ -6,9 +6,9 @@ import "github.com/cipherstash/stack/languages/golang/internal/guest" // for the process. It is opaque — it prints a redaction under every verb // and hands its bytes to no caller — and it is wiped once consumed. // -// It is the one type both guest packages share: stackauth reads one out of +// It is the one type both guest packages share: auth reads one out of // the developer profile, and this package consumes it. The alias is what -// makes a key read there the type taken here without stackauth importing +// makes a key read there the type taken here without auth importing // this package, so a binary that only wants the profile does not carry the // crypto guest. type ClientKey = guest.ClientKey diff --git a/languages/golang/stackencrypt/context.go b/languages/golang/encrypt/context.go similarity index 95% rename from languages/golang/stackencrypt/context.go rename to languages/golang/encrypt/context.go index 27387f26f..28f5d2123 100644 --- a/languages/golang/stackencrypt/context.go +++ b/languages/golang/encrypt/context.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "bytes" @@ -81,7 +81,7 @@ func MustContext(part any) Context { // With extends the context by one part, nesting to the left. func (c Context) With(part any) (Context, error) { if c.node == nil { - return Context{}, fmt.Errorf("stackencrypt: cannot extend an empty context") + return Context{}, fmt.Errorf("encrypt: cannot extend an empty context") } if err := checkPart(part); err != nil { return Context{}, err @@ -138,11 +138,11 @@ func checkRootNonEmpty(part any) error { switch p := part.(type) { case string: if p == "" { - return errors.New("stackencrypt: an empty string is an empty context") + return errors.New("encrypt: an empty string is an empty context") } case []byte: if len(p) == 0 { - return errors.New("stackencrypt: an empty byte slice is an empty context") + return errors.New("encrypt: an empty byte slice is an empty context") } } return nil @@ -153,7 +153,7 @@ func checkPart(part any) error { case string, []byte, int32, int64, uint32, uint64, int: return nil default: - return fmt.Errorf("stackencrypt: %T is not a context part (string, []byte or integer)", part) + return fmt.Errorf("encrypt: %T is not a context part (string, []byte or integer)", part) } } diff --git a/languages/golang/stackencrypt/credentials.go b/languages/golang/encrypt/credentials.go similarity index 83% rename from languages/golang/stackencrypt/credentials.go rename to languages/golang/encrypt/credentials.go index c96dad2c7..5e62e6966 100644 --- a/languages/golang/stackencrypt/credentials.go +++ b/languages/golang/encrypt/credentials.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -9,11 +9,11 @@ import ( "os" "sync/atomic" - "github.com/cipherstash/stack/languages/golang/stackauth" + "github.com/cipherstash/stack/languages/golang/auth" ) // Credentials is where a [Client]'s ZeroKMS credentials come from: the -// client id, the client key, and the stackauth strategy that supplies the +// client id, the client key, and the auth strategy that supplies the // bearer token. NewClient resolves them once, host-side — the crypto guest // is never given the environment or a filesystem to look them up itself — // and hands the key to the guest. @@ -22,7 +22,7 @@ import ( // profile, in the Rust client's order. [NewCredentials] takes a client id, // a client key and a strategy explicitly; [OIDCFederation] mints the token // from an identity provider's. Those three are the only implementations: -// the interface is sealed, so a bearer token always comes from a stackauth +// the interface is sealed, so a bearer token always comes from a auth // strategy. A raw token cannot be refreshed when it expires, and a source // outside the strategies would bypass the cross-process refresh lock the // device session shares with the CLI (a refresh token used twice gets the @@ -57,7 +57,7 @@ type resolvedCredentials struct { // ClientKey is the client key. NewClient consumes it whatever the // outcome, as [NewClientKey] describes. ClientKey *ClientKey - // Token supplies the bearer token for every request: a stackauth + // Token supplies the bearer token for every request: a auth // strategy, outside the package's own tests. Token tokenSource // Close, when not nil, releases what the credentials hold open — the @@ -85,11 +85,11 @@ type resolvedCredentials struct { // ErrNoCredentials is [AutoCredentials] finding no token strategy or no // client key in either place it looks. The wrapped error says which, and // what to set. -var ErrNoCredentials = errors.New("stackencrypt: no credentials") +var ErrNoCredentials = errors.New("encrypt: no credentials") // NewCredentials is [Credentials] from explicit values: a client id, a -// client key (from [NewClientKey], or stackauth's typed read), and the -// stackauth strategy that supplies the bearer token (ProfileStore's +// client key (from [NewClientKey], or auth's typed read), and the +// auth strategy that supplies the bearer token (ProfileStore's // AccessKey, DeviceSession, OIDC or Auto). The key is consumed by the first // NewClient given these credentials; a second is refused with // [ErrCredentialsConsumed], as a key is for one client. A nil strategy is @@ -101,7 +101,7 @@ var ErrNoCredentials = errors.New("stackencrypt: no credentials") // has returned, and are the caller's to close after it — the strategy, then // the store (closing the store closes its strategies too). A token asked of // a closed strategy is an error from the operation that needed it. -func NewCredentials(clientID string, key *ClientKey, strategy *stackauth.Strategy) Credentials { +func NewCredentials(clientID string, key *ClientKey, strategy *auth.Strategy) Credentials { c := &explicitCredentials{clientID: clientID, key: key} // A nil *Strategy stored in the interface would be a non-nil source // that fails on first use; left unset, NewClient refuses it up front. @@ -115,7 +115,7 @@ func NewCredentials(clientID string, key *ClientKey, strategy *stackauth.Strateg // the first consumed its key — whether it made a client or refused its // config — so there is nothing left to give. Build new credentials, with a // new key, for another client. -var ErrCredentialsConsumed = errors.New("stackencrypt: the credentials' client key was already consumed by an earlier NewClient") +var ErrCredentialsConsumed = errors.New("encrypt: the credentials' client key was already consumed by an earlier NewClient") // explicitCredentials is NewCredentials. token is the strategy; only this // package's tests put anything else in it. @@ -136,7 +136,7 @@ func (c *explicitCredentials) resolve(context.Context, resolveOptions) (*resolve // A nil token never gets here: NewClient refuses it host-side, before // the guest is read. resolved := &resolvedCredentials{ClientID: c.clientID, ClientKey: c.key, Token: c.token} - if strategy, ok := c.token.(*stackauth.Strategy); ok { + if strategy, ok := c.token.(*auth.Strategy); ok { // The strategy's store is the guest the token lives in and, when // the key was read through it, the key passed through: reported // live, as AutoCredentials reports its profile's. @@ -149,11 +149,11 @@ func (c *explicitCredentials) resolve(context.Context, resolveOptions) (*resolve // String names the credentials' kind and client id; the key prints a // redaction under every verb anyway, but nothing here asks it to. func (c *explicitCredentials) String() string { - return fmt.Sprintf("stackencrypt.NewCredentials{client_id: %s}", c.clientID) + return fmt.Sprintf("encrypt.NewCredentials{client_id: %s}", c.clientID) } // The environment variables AutoCredentials and NewClient read. The profile -// directory's own, CS_CONFIG_PATH, is stackauth's; so is CS_CTS_HOST, which +// directory's own, CS_CONFIG_PATH, is auth's; so is CS_CTS_HOST, which // its strategies take as the authentication endpoint. const ( envClientID = "CS_CLIENT_ID" @@ -189,7 +189,7 @@ var envZeroKMSHost = []string{"CS_ZEROKMS_HOST", "CS_VITUR_HOST"} // directory that does not exist, one that cannot be read, a path that is // not a directory. // -// The profile and the token strategies run in stackauth's credential guest, +// The profile and the token strategies run in auth's credential guest, // not in the crypto guest, which still sees no environment and no // filesystem. The credential guest lives as long as the client, and // Client.Close releases it. @@ -198,24 +198,24 @@ func AutoCredentials() Credentials { return autoCredentials{} } type autoCredentials struct{} // String names the credentials' kind; nothing is resolved to print it. -func (autoCredentials) String() string { return "stackencrypt.AutoCredentials" } +func (autoCredentials) String() string { return "encrypt.AutoCredentials" } func (autoCredentials) resolve(ctx context.Context, opts resolveOptions) (*resolvedCredentials, error) { - return resolveWithStrategy(ctx, opts, func(ctx context.Context, profile *stackauth.ProfileStore, noProfile error) (*stackauth.Strategy, error) { + return resolveWithStrategy(ctx, opts, func(ctx context.Context, profile *auth.ProfileStore, noProfile error) (*auth.Strategy, error) { strategy, err := profile.Auto(ctx) switch { - case errors.Is(err, stackauth.ErrNotAuthenticated): + case errors.Is(err, auth.ErrNotAuthenticated): if noProfile != nil { err = fmt.Errorf("%w: %w", err, noProfile) } return nil, fmt.Errorf("%w: no token: set %s and %s, or run `stash auth login`: %w", ErrNoCredentials, envAccessKey, envWorkspaceCRN, err) - case errors.Is(err, stackauth.ErrAuthConfig) && accessKeyConfigured(): + case errors.Is(err, auth.ErrAuthConfig) && accessKeyConfigured(): // The status covers every configuration fault the guest reports; // name the variables only when they are what was configured. - return nil, fmt.Errorf("stackencrypt: credentials: check %s and %s: %w", envAccessKey, envWorkspaceCRN, err) + return nil, fmt.Errorf("encrypt: credentials: check %s and %s: %w", envAccessKey, envWorkspaceCRN, err) case err != nil: - return nil, fmt.Errorf("stackencrypt: credentials: %w", err) + return nil, fmt.Errorf("encrypt: credentials: %w", err) } return strategy, nil }) @@ -228,37 +228,37 @@ func (autoCredentials) resolve(ctx context.Context, opts resolveOptions) (*resol // token of the user the call is for, and each distinct IdP token is // exchanged once while its CipherStash token lasts, so one client serves // many users without one user ever riding another's token; -// stackauth.OAuth2TokenSource adapts a golang.org/x/oauth2 source. The +// auth.OAuth2TokenSource adapts a golang.org/x/oauth2 source. The // client key is resolved as // [AutoCredentials] resolves it: CS_CLIENT_ID and CS_CLIENT_KEY, else the // developer profile. // // opts configure the federation strategy as they would -// stackauth.ProfileStore.OIDC: stackauth.WithAuthBaseURL pins the CTS +// auth.ProfileStore.OIDC: auth.WithAuthBaseURL pins the CTS // endpoint for these credentials alone (without it, CS_CTS_HOST overrides -// the endpoint, else it is discovered), and stackauth.WithCacheCapacity +// the endpoint, else it is discovered), and auth.WithCacheCapacity // sets how many users' tokens are kept (1024 unless set). -func OIDCFederation(crn string, provider stackauth.OIDCProvider, opts ...stackauth.StrategyOption) Credentials { +func OIDCFederation(crn string, provider auth.OIDCProvider, opts ...auth.StrategyOption) Credentials { return &oidcCredentials{crn: crn, provider: provider, opts: opts} } type oidcCredentials struct { crn string - provider stackauth.OIDCProvider - opts []stackauth.StrategyOption + provider auth.OIDCProvider + opts []auth.StrategyOption } // String names the credentials' kind and workspace; the provider is not // asked for anything to print it. func (c oidcCredentials) String() string { - return fmt.Sprintf("stackencrypt.OIDCFederation{crn: %s}", c.crn) + return fmt.Sprintf("encrypt.OIDCFederation{crn: %s}", c.crn) } func (c oidcCredentials) resolve(ctx context.Context, opts resolveOptions) (*resolvedCredentials, error) { - return resolveWithStrategy(ctx, opts, func(ctx context.Context, profile *stackauth.ProfileStore, _ error) (*stackauth.Strategy, error) { + return resolveWithStrategy(ctx, opts, func(ctx context.Context, profile *auth.ProfileStore, _ error) (*auth.Strategy, error) { strategy, err := profile.OIDC(ctx, c.crn, c.provider, c.opts...) if err != nil { - return nil, fmt.Errorf("stackencrypt: credentials: OIDC federation: %w", err) + return nil, fmt.Errorf("encrypt: credentials: OIDC federation: %w", err) } return strategy, nil }) @@ -273,32 +273,32 @@ func (c oidcCredentials) resolve(ctx context.Context, opts resolveOptions) (*res func resolveWithStrategy( ctx context.Context, opts resolveOptions, - strategy func(ctx context.Context, profile *stackauth.ProfileStore, noProfile error) (*stackauth.Strategy, error), + strategy func(ctx context.Context, profile *auth.ProfileStore, noProfile error) (*auth.Strategy, error), ) (_ *resolvedCredentials, err error) { - authOpts := []stackauth.Option{stackauth.WithRoundTripper(opts.Transport)} + authOpts := []auth.Option{auth.WithRoundTripper(opts.Transport)} if opts.RequireLockedMemory { - authOpts = append(authOpts, stackauth.RequireLockedMemory()) + authOpts = append(authOpts, auth.RequireLockedMemory()) } - profile, err := stackauth.Resolve(ctx, authOpts...) + profile, err := auth.Resolve(ctx, authOpts...) // noProfile is why there is no profile to consult, when there is none: // kept for the errors that would have consulted it, so a profile that // exists but cannot be opened is not reported as "not logged in". var noProfile error - if errors.Is(err, stackauth.ErrNoProfile) { + if errors.Is(err, auth.ErrNoProfile) { // The strategies that need no profile still run, in a guest with // nothing mounted; every profile read on it is ErrNoProfile. noProfile = err - profile, err = stackauth.OpenWithoutProfile(ctx, authOpts...) + profile, err = auth.OpenWithoutProfile(ctx, authOpts...) } if errors.Is(err, ErrMemoryLock) { // Under RequireLockedMemory the guest is refused before anything is // resolved — on a host that cannot lock or reserve its memory at all // (a 32-bit address space, a zero RLIMIT_MEMLOCK). Name which guest, // as the client's own report does. - return nil, fmt.Errorf("stackencrypt: credentials: the credential guest: %w", err) + return nil, fmt.Errorf("encrypt: credentials: the credential guest: %w", err) } if err != nil { - return nil, fmt.Errorf("stackencrypt: credentials: %w", err) + return nil, fmt.Errorf("encrypt: credentials: %w", err) } defer func() { if err != nil { @@ -377,27 +377,27 @@ func clientKeyFromEnv() (string, *ClientKey, error) { // noProfile, when not nil, is why the profile could not be opened; the // store's own answer to a read is then a bare ErrNoProfile, and the reason // is the useful one. -func clientKeyFromProfile(ctx context.Context, profile *stackauth.ProfileStore, noProfile error) (string, *ClientKey, error) { +func clientKeyFromProfile(ctx context.Context, profile *auth.ProfileStore, noProfile error) (string, *ClientKey, error) { notConfigured := func(err error) error { return fmt.Errorf("%w: no client key: set %s and %s, or run `stash auth login`: %w", ErrNoCredentials, envClientID, envClientKey, err) } workspace, err := profile.CurrentWorkspaceStore(ctx) - if errors.Is(err, stackauth.ErrNoProfile) && noProfile != nil { + if errors.Is(err, auth.ErrNoProfile) && noProfile != nil { err = noProfile } - if errors.Is(err, stackauth.ErrNoProfile) || errors.Is(err, stackauth.ErrNoCurrentWorkspace) { + if errors.Is(err, auth.ErrNoProfile) || errors.Is(err, auth.ErrNoCurrentWorkspace) { return "", nil, notConfigured(err) } if err != nil { - return "", nil, fmt.Errorf("stackencrypt: credentials: %w", err) + return "", nil, fmt.Errorf("encrypt: credentials: %w", err) } clientID, key, err := workspace.SecretKey(ctx) - if errors.Is(err, stackauth.ErrNotFound) { + if errors.Is(err, auth.ErrNotFound) { return "", nil, notConfigured(err) } if err != nil { - return "", nil, fmt.Errorf("stackencrypt: credentials: reading the client key: %w", err) + return "", nil, fmt.Errorf("encrypt: credentials: reading the client key: %w", err) } return clientID, key, nil } diff --git a/languages/golang/stackencrypt/credentials_test.go b/languages/golang/encrypt/credentials_test.go similarity index 95% rename from languages/golang/stackencrypt/credentials_test.go rename to languages/golang/encrypt/credentials_test.go index b04911e64..1a022a6dd 100644 --- a/languages/golang/stackencrypt/credentials_test.go +++ b/languages/golang/encrypt/credentials_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -15,8 +15,8 @@ import ( "testing" "time" + "github.com/cipherstash/stack/languages/golang/auth" "github.com/cipherstash/stack/languages/golang/internal/guest" - "github.com/cipherstash/stack/languages/golang/stackauth" ) // Credential resolution, pinned against the Rust client's order. The @@ -45,12 +45,12 @@ const ( profileClientID = "0b1e8a44-5c1d-4d4e-9b52-3f0e6c2f8a17" ) -// authGuestOrSkip skips where stackauth's credential guest is not built, +// authGuestOrSkip skips where auth's credential guest is not built, // as guestOrSkip does for this package's own. func authGuestOrSkip(t *testing.T) { t.Helper() - store, err := stackauth.OpenWithoutProfile(context.Background()) - if errors.Is(err, stackauth.ErrGuestNotBuilt) { + store, err := auth.OpenWithoutProfile(context.Background()) + if errors.Is(err, auth.ErrGuestNotBuilt) { t.Skip(err) } if err != nil { @@ -114,7 +114,7 @@ func newProfile(t *testing.T, files profileFiles) string { } } } - store, err := stackauth.Open(context.Background(), dir) + store, err := auth.Open(context.Background(), dir) if err != nil { t.Fatal(err) } @@ -208,7 +208,7 @@ func TestAutoCredentialsFromTheEnvironmentWithNoProfile(t *testing.T) { authGuestOrSkip(t) // The CI shape: four variables and no profile directory at all. cleanEnv(t, filepath.Join(t.TempDir(), "absent")) - auth := newAuthServer(t) + cts := newAuthServer(t) t.Setenv(envAccessKey, testAccessKey) t.Setenv(envWorkspaceCRN, testCRN) t.Setenv(envClientID, testClientID) @@ -220,8 +220,8 @@ func TestAutoCredentialsFromTheEnvironmentWithNoProfile(t *testing.T) { if resolved.ClientID != testClientID || string(guest.KeyBytes(resolved.ClientKey)) != testClientKey { t.Error("the client key is not the environment's") } - if got := token(t, resolved); got != auth.jwt || auth.calls.Load() != 1 { - t.Errorf("Token = %q after %d exchanges, want the access key's", got, auth.calls.Load()) + if got := token(t, resolved); got != cts.jwt || cts.calls.Load() != 1 { + t.Errorf("Token = %q after %d exchanges, want the access key's", got, cts.calls.Load()) } } @@ -262,14 +262,14 @@ func TestAutoCredentialsPrecedence(t *testing.T) { }) t.Run("the environment's access key wins over the stored session", func(t *testing.T) { cleanEnv(t, newProfile(t, loggedIn("profile-token"))) - auth := newAuthServer(t) + cts := newAuthServer(t) t.Setenv(envAccessKey, testAccessKey) t.Setenv(envWorkspaceCRN, testCRN) resolved, err := resolve(t) if err != nil { t.Fatal(err) } - if got := token(t, resolved); got != auth.jwt { + if got := token(t, resolved); got != cts.jwt { t.Errorf("Token = %q, want the access key's", got) } // The key still comes from the profile. @@ -291,13 +291,13 @@ func TestAutoCredentialsMissing(t *testing.T) { }{ { name: "nothing anywhere", - want: []error{ErrNoCredentials, stackauth.ErrNotAuthenticated}, + want: []error{ErrNoCredentials, auth.ErrNotAuthenticated}, names: envAccessKey, }, { name: "a profile with no login", profile: &profileFiles{}, - want: []error{ErrNoCredentials, stackauth.ErrNotAuthenticated}, + want: []error{ErrNoCredentials, auth.ErrNotAuthenticated}, names: "stash auth login", }, { @@ -311,13 +311,13 @@ func TestAutoCredentialsMissing(t *testing.T) { { name: "a stored session and no secretkey.json", profile: &profileFiles{auth: loggedIn("t").auth}, - want: []error{ErrNoCredentials, stackauth.ErrNotFound}, + want: []error{ErrNoCredentials, auth.ErrNotFound}, names: envClientKey, }, { name: "an access key with no workspace CRN", env: map[string]string{envAccessKey: testAccessKey, envClientID: testClientID, envClientKey: testClientKey}, - want: []error{stackauth.ErrAuthConfig}, + want: []error{auth.ErrAuthConfig}, }, } { t.Run(tc.name, func(t *testing.T) { @@ -355,7 +355,7 @@ func TestAutoCredentialsNamesWhyTheProfileCouldNotBeOpened(t *testing.T) { t.Run("the token", func(t *testing.T) { cleanEnv(t, file) _, err := resolve(t) - for _, want := range []error{ErrNoCredentials, stackauth.ErrNoProfile} { + for _, want := range []error{ErrNoCredentials, auth.ErrNoProfile} { if !errors.Is(err, want) { t.Errorf("error %v, want %v", err, want) } @@ -369,7 +369,7 @@ func TestAutoCredentialsNamesWhyTheProfileCouldNotBeOpened(t *testing.T) { t.Setenv(envAccessKey, testAccessKey) t.Setenv(envWorkspaceCRN, testCRN) _, err := resolve(t) - for _, want := range []error{ErrNoCredentials, stackauth.ErrNoProfile} { + for _, want := range []error{ErrNoCredentials, auth.ErrNoProfile} { if !errors.Is(err, want) { t.Errorf("error %v, want %v", err, want) } @@ -427,7 +427,7 @@ func TestNewClientWithAutoCredentials(t *testing.T) { t.Fatalf("the error carries key material: %q", err) } // A failed NewClient released the credentials it resolved. - if _, err := released.Token.Token(context.Background()); !errors.Is(err, stackauth.ErrState) { + if _, err := released.Token.Token(context.Background()); !errors.Is(err, auth.ErrState) { t.Fatalf("the token source after a failed NewClient: %v, want ErrState", err) } } @@ -676,7 +676,7 @@ func TestAutoCredentialsCloseReleasesTheGuest(t *testing.T) { if err := resolved.Close(); err != nil { t.Fatal(err) } - if _, err := resolved.Token.Token(context.Background()); !errors.Is(err, stackauth.ErrState) { + if _, err := resolved.Token.Token(context.Background()); !errors.Is(err, auth.ErrState) { t.Fatalf("Token after Close: %v, want ErrState", err) } } @@ -684,7 +684,7 @@ func TestAutoCredentialsCloseReleasesTheGuest(t *testing.T) { func TestCredentialsPrintNoKey(t *testing.T) { for _, c := range []Credentials{AutoCredentials(), newTestCredentials(testClientID, NewClientKey([]byte(testClientKey)), staticToken("t"))} { for _, verb := range []string{"%v", "%+v", "%s"} { - if out := fmt.Sprintf(verb, c); strings.Contains(out, testClientKey[:16]) || !strings.HasPrefix(out, "stackencrypt.") { + if out := fmt.Sprintf(verb, c); strings.Contains(out, testClientKey[:16]) || !strings.HasPrefix(out, "encrypt.") { t.Errorf("%s: %q", verb, out) } } @@ -699,7 +699,7 @@ func (f credentialsFunc) resolve(ctx context.Context, opts resolveOptions) (*res func ptr(s string) *string { return &s } -// NewCredentials takes its token only from a stackauth strategy: a nil one +// NewCredentials takes its token only from an auth strategy: a nil one // is refused host-side, before any guest is read, and the key is consumed // all the same — the credentials are spent, as on any refused config. func TestNewCredentialsRefusesANilStrategy(t *testing.T) { @@ -723,13 +723,13 @@ func TestNewCredentialsRefusesANilStrategy(t *testing.T) { func TestNewCredentialsReportsTheStrategysMemoryLock(t *testing.T) { authGuestOrSkip(t) ctx := context.Background() - // Best effort, stackauth's default: the store opens whatever the lock. - store, err := stackauth.OpenWithoutProfile(ctx) + // Best effort, auth's default: the store opens whatever the lock. + store, err := auth.OpenWithoutProfile(ctx) if err != nil { t.Fatal(err) } defer store.Close() - strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, stackauth.WithAuthBaseURL("https://cts.example.com")) + strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, auth.WithAuthBaseURL("https://cts.example.com")) if err != nil { t.Fatal(err) } @@ -761,15 +761,15 @@ func TestRequireLockedMemoryRefusesUnlockedCredentials(t *testing.T) { for name, base := range map[string]func(t *testing.T) Credentials{ "NewCredentials": func(t *testing.T) Credentials { cleanEnv(t, t.TempDir()) - auth := newAuthServer(t) + cts := newAuthServer(t) ctx := context.Background() - // Best effort, stackauth's default. - store, err := stackauth.OpenWithoutProfile(ctx) + // Best effort, auth's default. + store, err := auth.OpenWithoutProfile(ctx) if err != nil { t.Fatal(err) } t.Cleanup(func() { _ = store.Close() }) - strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, stackauth.WithAuthBaseURL(auth.URL)) + strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, auth.WithAuthBaseURL(cts.URL)) if err != nil { t.Fatal(err) } @@ -820,7 +820,7 @@ func TestRequireLockedMemoryRefusesUnlockedCredentials(t *testing.T) { t.Error("the key still holds material after the credentials were refused") } if (*resolved).Close != nil { - if _, err := (*resolved).Token.Token(context.Background()); !errors.Is(err, stackauth.ErrState) { + if _, err := (*resolved).Token.Token(context.Background()); !errors.Is(err, auth.ErrState) { t.Errorf("the token source after the refusal: %v, want it released (ErrState)", err) } } @@ -868,13 +868,13 @@ func TestNewCredentialsLeavesTheStrategyToTheCaller(t *testing.T) { guestOrSkip(t) authGuestOrSkip(t) ctx := context.Background() - store, err := stackauth.OpenWithoutProfile(ctx) + store, err := auth.OpenWithoutProfile(ctx) if err != nil { t.Fatal(err) } defer store.Close() cts := newStub(t, http.StatusUnauthorized, "", "nope") - strategy, err := store.AccessKey(ctx, testCRN, "CSAKtest.key", stackauth.WithAuthBaseURL(cts.URL)) + strategy, err := store.AccessKey(ctx, testCRN, "CSAKtest.key", auth.WithAuthBaseURL(cts.URL)) if err != nil { t.Fatal(err) } @@ -896,7 +896,7 @@ func TestNewCredentialsLeavesTheStrategyToTheCaller(t *testing.T) { // Still open: asked again, it goes back to CTS rather than failing // with ErrState. before := len(cts.requests) - if _, err := strategy.Token(ctx); errors.Is(err, stackauth.ErrState) { + if _, err := strategy.Token(ctx); errors.Is(err, auth.ErrState) { t.Fatalf("the strategy after a failed NewClient: %v, want it still open", err) } if len(cts.requests) == before { diff --git a/languages/golang/stackencrypt/doc.go b/languages/golang/encrypt/doc.go similarity index 94% rename from languages/golang/stackencrypt/doc.go rename to languages/golang/encrypt/doc.go index fa22f8ea1..16d2bb99d 100644 --- a/languages/golang/stackencrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -1,4 +1,4 @@ -// Package stackencrypt is the Go binding of stack-encrypt: ZeroKMS-backed +// Package encrypt is the Go binding of stack-encrypt: ZeroKMS-backed // field-level encryption with searchable index terms, running the Rust // crate unmodified inside a WASI guest under wazero (CGO_ENABLED=0). // @@ -65,7 +65,7 @@ // // The guest imports exactly two host functions: an HTTP send, served by any // [net/http.RoundTripper], and a bearer-token fetch, served by the -// credentials' stackauth strategy. What crosses per ZeroKMS call is what would cross TLS +// credentials' auth strategy. What crosses per ZeroKMS call is what would cross TLS // anyway; derived key material never leaves the guest. Under // [AutoCredentials] and [OIDCFederation] the same RoundTripper also carries // the authentication requests to CTS, so one scoped to the ZeroKMS host @@ -74,19 +74,19 @@ // // # Credentials // -// A [Credentials] supplies the client id, the client key and the stackauth +// A [Credentials] supplies the client id, the client key and the auth // strategy the token comes from, and NewClient resolves it host-side: the crypto guest is never // given the environment or a filesystem to find them in. The default, // [AutoCredentials], mirrors the Rust client — the environment first // (CS_CLIENT_ACCESS_KEY with CS_WORKSPACE_CRN for the token, CS_CLIENT_ID // with CS_CLIENT_KEY for the key), then the developer profile, which it -// reads through stackauth's credential guest, where the token strategies +// reads through auth's credential guest, where the token strategies // also run. CS_ZEROKMS_HOST (or CS_VITUR_HOST) pins the endpoint whatever // the credentials. [NewCredentials] takes a client id, a client key and a // strategy explicitly, and [OIDCFederation] mints the token from an // identity provider's. Pass one with [WithCredentials]. Those three are // the only kinds of Credentials, and none takes a raw token: a token is -// always a stackauth strategy's, since a raw one cannot be refreshed when +// always a auth strategy's, since a raw one cannot be refreshed when // it expires and would bypass the cross-process lock a device-session // refresh holds with the CLI. // Credentials that cannot be resolved fail NewClient with @@ -137,9 +137,9 @@ // [ErrMemoryLock], and closes the client when the growth was the guest's // own allocation rather than a host-staged buffer. The report and the // policy cover the credential guest as well; with [NewCredentials] that is -// the caller's stackauth store, which NewClient refuses under the policy +// the caller's auth store, which NewClient refuses under the policy // when it is unlocked, and which should be opened with -// stackauth.RequireLockedMemory to stay locked (see +// auth.RequireLockedMemory to stay locked (see // [WithRequireLockedMemory]). A Client prints its memory state // ([Client.String]) and logs it ([Client.LogValue]). An embedder running // the guest under its own wazero configuration gets none of this unless @@ -151,4 +151,4 @@ // returns, and every buffer staged for a call is wiped by se_dealloc // before the call's result is returned. The residency tests pin the // second; the first is the Rust crate's own guarantee. -package stackencrypt +package encrypt diff --git a/languages/golang/stackencrypt/errors.go b/languages/golang/encrypt/errors.go similarity index 97% rename from languages/golang/stackencrypt/errors.go rename to languages/golang/encrypt/errors.go index 212c6bb2f..55558da0f 100644 --- a/languages/golang/stackencrypt/errors.go +++ b/languages/golang/encrypt/errors.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import "github.com/cipherstash/stack/languages/golang/internal/guest" @@ -9,7 +9,7 @@ import "github.com/cipherstash/stack/languages/golang/internal/guest" // // They are the sentinels every guest package shares (one status table for // every guest, decoded once), exposed here under this package's names: an -// error from stackauth is the same value, so errors.Is holds across the +// error from auth is the same value, so errors.Is holds across the // two. var ( // ErrAuthentication is an AEAD open failure: a tampered ciphertext, a diff --git a/languages/golang/stackencrypt/example/README.md b/languages/golang/encrypt/example/README.md similarity index 87% rename from languages/golang/stackencrypt/example/README.md rename to languages/golang/encrypt/example/README.md index aa6486e67..aa1f3ca75 100644 --- a/languages/golang/stackencrypt/example/README.md +++ b/languages/golang/encrypt/example/README.md @@ -9,18 +9,18 @@ are meant to fail. ```bash stash auth login # once; the example reads ~/.cipherstash -mise run go:stackencrypt:example # builds both guests, then runs +mise run go:encrypt:example # builds both guests, then runs ``` Or, if you would rather drive it yourself: ```bash mise run wasm:guest:build wasm:auth-guest:build -cd bindings/go && go run ./stackencrypt/example +cd bindings/go && go run ./encrypt/example ``` The guest builds are not optional. This package embeds -`wasm/stack_encrypt_guest.wasm` and `stackauth` embeds +`wasm/stack_encrypt_guest.wasm` and `auth` embeds `wasm/stack_auth_guest.wasm`; both are gitignored, so a fresh checkout has no guests and `NewClient` / `Resolve` fail until they are built — and the Go side will not notice a stale one, so rebuild after any change under either @@ -38,13 +38,13 @@ side will not notice a stale one, so rebuild after any change under either ## Credentials The example calls `NewClient(ctx)` with no options, so it resolves its -credentials with `stackencrypt.AutoCredentials`, which is what any +credentials with `encrypt.AutoCredentials`, which is what any application gets by default. It looks in the environment first and then in the developer profile, in the order the Rust client uses: - **The token.** If `CS_CLIENT_ACCESS_KEY` is set (with `CS_WORKSPACE_CRN`), the access key is exchanged for a token. Otherwise the current workspace's - stored device session is used. Both run in `stackauth`'s credential guest. + stored device session is used. Both run in `auth`'s credential guest. - **The client key.** `CS_CLIENT_ID` and `CS_CLIENT_KEY` if both are set, otherwise the current workspace's `secretkey.json`. - **The endpoint.** `CS_ZEROKMS_HOST` if set, otherwise the token's @@ -61,7 +61,7 @@ credentials are resolved host-side. The device session is **asked on every request and refreshes itself**. A profile token lasts 45 minutes, so a pinned token would give you a program that works for a while and then stops; that is why the client takes tokens -only from `stackauth` strategies and has no way to pass a raw one. The refresh takes the +only from `auth` strategies and has no way to pass a raw one. The refresh takes the same cross-process lock as the `stash` CLI. The IdP rotates refresh tokens and detects replay, so two processes sharing `~/.cipherstash` that both exchanged the same refresh token would get the whole chain revoked; the lock @@ -69,4 +69,4 @@ prevents that. To supply the credentials yourself instead, from a secrets manager and with no `CS_*` variables or profile, see [`explicit/`](explicit/), which uses -`stackencrypt.NewCredentials`. +`encrypt.NewCredentials`. diff --git a/languages/golang/stackencrypt/example/explicit/README.md b/languages/golang/encrypt/example/explicit/README.md similarity index 83% rename from languages/golang/stackencrypt/example/explicit/README.md rename to languages/golang/encrypt/example/explicit/README.md index 4048119ed..b76e09750 100644 --- a/languages/golang/stackencrypt/example/explicit/README.md +++ b/languages/golang/encrypt/example/explicit/README.md @@ -1,6 +1,6 @@ # Explicit credentials -A runnable example of `stackencrypt.NewCredentials`: the application +A runnable example of `encrypt.NewCredentials`: the application supplies every credential itself, and nothing is read from `CS_*` variables or the developer profile. Use this shape when the client key lives in a secrets manager, when one process talks to more than one workspace, or when @@ -18,7 +18,7 @@ Docker secret is mounted: | `access-key` | An access key (`CSAK…`) for the workspace. | ```bash -mise run go:stackencrypt:example:explicit -- \ +mise run go:encrypt:example:explicit -- \ -secrets-dir /run/secrets \ -client-id \ -workspace-crn @@ -28,7 +28,7 @@ Or build both guests and run it yourself from `bindings/go`: ```bash mise run wasm:guest:build wasm:auth-guest:build -cd bindings/go && go run ./stackencrypt/example/explicit -secrets-dir ... -client-id ... -workspace-crn ... +cd bindings/go && go run ./encrypt/example/explicit -secrets-dir ... -client-id ... -workspace-crn ... ``` Optional flags: `-cts-host` pins the authentication endpoint (default: from @@ -42,11 +42,11 @@ that cannot be locked in RAM. The ZeroKMS endpoint comes from the token. key goes straight into `NewClientKey`, which takes ownership of the bytes, and `NewClient` wipes them. - **Who owns what.** The application opens the credential guest - (`stackauth.OpenWithoutProfile`) and the access-key strategy, and closes + (`auth.OpenWithoutProfile`) and the access-key strategy, and closes them after the client. The client asks the strategy for a token on every request, and the strategy re-exchanges the access key as tokens expire. - **Locked memory.** With `-require-locked-memory` the credential guest is - opened with `stackauth.RequireLockedMemory()` and the client with + opened with `auth.RequireLockedMemory()` and the client with `WithRequireLockedMemory()`, so both guests stay locked. The client's printed memory state covers both. - **A key is for one client.** Passing the same credentials to a second diff --git a/languages/golang/stackencrypt/example/explicit/main.go b/languages/golang/encrypt/example/explicit/main.go similarity index 84% rename from languages/golang/stackencrypt/example/explicit/main.go rename to languages/golang/encrypt/example/explicit/main.go index 41ab2e500..785c3d8e3 100644 --- a/languages/golang/stackencrypt/example/explicit/main.go +++ b/languages/golang/encrypt/example/explicit/main.go @@ -6,7 +6,7 @@ // AutoCredentials. // // mise run wasm:guest:build wasm:auth-guest:build # both embedded guests -// go run ./stackencrypt/example/explicit \ +// go run ./encrypt/example/explicit \ // -secrets-dir /run/secrets \ // -client-id 6a70bd18-99ac-4650-b104-37eec3a15b09 \ // -workspace-crn crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY @@ -24,8 +24,8 @@ import ( "os" "path/filepath" - "github.com/cipherstash/stack/languages/golang/stackauth" - "github.com/cipherstash/stack/languages/golang/stackencrypt" + "github.com/cipherstash/stack/languages/golang/auth" + "github.com/cipherstash/stack/languages/golang/encrypt" ) type user struct { @@ -84,12 +84,12 @@ func run(ctx context.Context, cfg config, secrets secrets) error { // it, so the caller chooses its memory policy: under // -require-locked-memory it is locked from the start and stays locked // as it grows. NewClient checks it either way, below. - var storeOpts []stackauth.Option + var storeOpts []auth.Option if cfg.requireLockedMemory { - storeOpts = append(storeOpts, stackauth.RequireLockedMemory()) + storeOpts = append(storeOpts, auth.RequireLockedMemory()) } // No profile: this application's credentials are all explicit. - store, err := stackauth.OpenWithoutProfile(ctx, storeOpts...) + store, err := auth.OpenWithoutProfile(ctx, storeOpts...) if err != nil { return fmt.Errorf("opening the credential guest: %w", err) } @@ -101,9 +101,9 @@ func run(ctx context.Context, cfg config, secrets secrets) error { if err != nil { return err } - var strategyOpts []stackauth.StrategyOption + var strategyOpts []auth.StrategyOption if cfg.ctsHost != "" { - strategyOpts = append(strategyOpts, stackauth.WithAuthBaseURL(cfg.ctsHost)) + strategyOpts = append(strategyOpts, auth.WithAuthBaseURL(cfg.ctsHost)) } // The access key crosses as a string, which Go cannot wipe; the bytes // it was read into can be. @@ -119,13 +119,13 @@ func run(ctx context.Context, cfg config, secrets secrets) error { return err } // NewClientKey takes ownership of the bytes; NewClient wipes them. - creds := stackencrypt.NewCredentials(cfg.clientID, stackencrypt.NewClientKey(keyMaterial), strategy) + creds := encrypt.NewCredentials(cfg.clientID, encrypt.NewClientKey(keyMaterial), strategy) - opts := []stackencrypt.ClientOption{stackencrypt.WithCredentials(creds)} + opts := []encrypt.ClientOption{encrypt.WithCredentials(creds)} if cfg.requireLockedMemory { - opts = append(opts, stackencrypt.WithRequireLockedMemory()) + opts = append(opts, encrypt.WithRequireLockedMemory()) } - client, err := stackencrypt.NewClient(ctx, opts...) + client, err := encrypt.NewClient(ctx, opts...) if err != nil { return fmt.Errorf("connecting to ZeroKMS: %w", err) } @@ -136,7 +136,7 @@ func run(ctx context.Context, cfg config, secrets secrets) error { // A key is for one client. These credentials are spent, and a second // client needs a new key; nothing was sent to find that out. - if _, err := stackencrypt.NewClient(ctx, stackencrypt.WithCredentials(creds)); errors.Is(err, stackencrypt.ErrCredentialsConsumed) { + if _, err := encrypt.NewClient(ctx, encrypt.WithCredentials(creds)); errors.Is(err, encrypt.ErrCredentialsConsumed) { fmt.Println("reusing the credentials is refused: the key was consumed by the first client") } else { return fmt.Errorf("reusing the credentials: got %v, want ErrCredentialsConsumed", err) @@ -148,16 +148,16 @@ func run(ctx context.Context, cfg config, secrets secrets) error { if err != nil { return fmt.Errorf("encrypting records: %w", err) } - emailCtx, err := stackencrypt.ParseLabel("users/email") + emailCtx, err := encrypt.ParseLabel("users/email") if err != nil { return fmt.Errorf("the probe's label: %w", err) } - probe, err := cipher.Term(ctx, "bob@example.com", emailCtx.Context(), stackencrypt.Equality) + probe, err := cipher.Term(ctx, "bob@example.com", emailCtx.Context(), encrypt.Equality) if err != nil { return fmt.Errorf("deriving a probe: %w", err) } for i, r := range records { - if probe.(stackencrypt.EqualityTerm).Equal(r["Email"].Equality) { + if probe.(encrypt.EqualityTerm).Equal(r["Email"].Equality) { fmt.Printf("sealed %d rows; the probe for bob@example.com matches row %d\n", len(records), i) } } diff --git a/languages/golang/stackencrypt/example/main.go b/languages/golang/encrypt/example/main.go similarity index 84% rename from languages/golang/stackencrypt/example/main.go rename to languages/golang/encrypt/example/main.go index acfe03572..1109f167d 100644 --- a/languages/golang/stackencrypt/example/main.go +++ b/languages/golang/encrypt/example/main.go @@ -4,7 +4,7 @@ // // stash auth login // mise run wasm:guest:build wasm:auth-guest:build # both embedded guests -// go run ./stackencrypt/example # from bindings/go +// go run ./encrypt/example # from bindings/go // // It walks the four things the binding does — seal a value, seal a record // with its index terms, probe those terms with a query, and open both again @@ -17,7 +17,7 @@ import ( "os" "sort" - "github.com/cipherstash/stack/languages/golang/stackencrypt" + "github.com/cipherstash/stack/languages/golang/encrypt" "github.com/cipherstash/vitaminc/bindings/go/vcvalue" ) @@ -42,11 +42,11 @@ func run() error { // No options: credentials from AutoCredentials, which is the // environment first (CS_CLIENT_ACCESS_KEY + CS_WORKSPACE_CRN, CS_CLIENT_ID // + CS_CLIENT_KEY), then the developer profile `stash auth login` writes, - // read through stackauth's credential guest. The token is a refreshing + // read through auth's credential guest. The token is a refreshing // device session there, asked on every request, so a long run outlives // one token. The ZeroKMS endpoint: CS_ZEROKMS_HOST if set, else the token's // services claim. - client, err := stackencrypt.NewClient(ctx) + client, err := encrypt.NewClient(ctx) if err != nil { return fmt.Errorf("connecting to ZeroKMS: %w", err) } @@ -77,7 +77,7 @@ func run() error { // A whole value, sealed under an AAD of the caller's choosing. The shape of // the ciphertext mirrors the plaintext, and a field marked Plain rides // alongside it in the clear. -func values(ctx context.Context, cipher *stackencrypt.Cipher) error { +func values(ctx context.Context, cipher *encrypt.Cipher) error { section("a value") aad := []byte("users/v1") @@ -93,7 +93,7 @@ func values(ctx context.Context, cipher *stackencrypt.Cipher) error { return fmt.Errorf("encrypting a value: %w", err) } for name, node := range sealed.(map[string]any) { - if leaf, ok := node.(stackencrypt.Sealed); ok { + if leaf, ok := node.(encrypt.Sealed); ok { fmt.Printf(" %-11s %d bytes of ciphertext\n", name, len(leaf)) } else { fmt.Printf(" %-11s %v (passthrough — in the clear, and unauthenticated)\n", name, node) @@ -118,7 +118,7 @@ func values(ctx context.Context, cipher *stackencrypt.Cipher) error { // A record: every field sealed under its own context, with the index terms // its tag asked for, and all of it from one batched ZeroKMS request. -func recordsAndTerms(ctx context.Context, cipher *stackencrypt.Cipher) ([]stackencrypt.EncryptedRecord, error) { +func recordsAndTerms(ctx context.Context, cipher *encrypt.Cipher) ([]encrypt.EncryptedRecord, error) { section("records, and the terms that index them") users := []user{ @@ -133,25 +133,25 @@ func recordsAndTerms(ctx context.Context, cipher *stackencrypt.Cipher) ([]stacke fmt.Printf(" %d rows sealed in one batched key request\n", len(records)) for i, r := range records { fmt.Printf(" row %d Email: %d-byte ciphertext, eq %x…, match %d positions\n", - i, len(r["Email"].Ciphertext.(stackencrypt.Sealed)), r["Email"].Equality[:6], countPositions(r["Email"].Match)) + i, len(r["Email"].Ciphertext.(encrypt.Sealed)), r["Email"].Equality[:6], countPositions(r["Email"].Match)) fmt.Printf(" Age: %d-byte ciphertext, eq %x…, ore %d bytes\n", - len(r["Age"].Ciphertext.(stackencrypt.Sealed)), r["Age"].Equality[:6], len(r["Age"].Ore)) + len(r["Age"].Ciphertext.(encrypt.Sealed)), r["Age"].Equality[:6], len(r["Age"].Ore)) } // A query probe: the same derivation as the stored term, from the value // being searched for. It never touches the ciphertext — matching is what // the term is for. fmt.Println() - emailCtx, err := stackencrypt.ParseLabel("users/email") + emailCtx, err := encrypt.ParseLabel("users/email") if err != nil { return nil, fmt.Errorf("the probe's label: %w", err) } - probe, err := cipher.Term(ctx, "bob@example.com", emailCtx.Context(), stackencrypt.Equality) + probe, err := cipher.Term(ctx, "bob@example.com", emailCtx.Context(), encrypt.Equality) if err != nil { return nil, fmt.Errorf("deriving a probe: %w", err) } for i, r := range records { - if probe.(stackencrypt.EqualityTerm).Equal(r["Email"].Equality) { + if probe.(encrypt.EqualityTerm).Equal(r["Email"].Equality) { fmt.Printf(" probe for bob@example.com matches row %d\n", i) } } @@ -159,16 +159,16 @@ func recordsAndTerms(ctx context.Context, cipher *stackencrypt.Cipher) ([]stacke // A term is bound to its context. The same value under another field's // context is a different term, which is what stops a match in one column // from being a match in another. - nameCtx, err := stackencrypt.ParseLabel("users/name") + nameCtx, err := encrypt.ParseLabel("users/name") if err != nil { return nil, fmt.Errorf("the probe's label: %w", err) } - wrong, err := cipher.Term(ctx, "bob@example.com", nameCtx.Context(), stackencrypt.Equality) + wrong, err := cipher.Term(ctx, "bob@example.com", nameCtx.Context(), encrypt.Equality) if err != nil { return nil, fmt.Errorf("deriving a probe: %w", err) } fmt.Printf(" the same value under users/name matches nothing: %t\n", - !wrong.(stackencrypt.EqualityTerm).Equal(records[1]["Email"].Equality)) + !wrong.(encrypt.EqualityTerm).Equal(records[1]["Email"].Equality)) var back []user if err := cipher.DecryptRecords(ctx, records, &back); err != nil { @@ -181,7 +181,7 @@ func recordsAndTerms(ctx context.Context, cipher *stackencrypt.Cipher) ([]stacke // ORE terms compare in the plaintext's order without revealing it: sorting // the rows by their Age term sorts them by age. -func ordering(records []stackencrypt.EncryptedRecord) error { +func ordering(records []encrypt.EncryptedRecord) error { section("order, without the values") order := []int{0, 1, 2} @@ -193,7 +193,7 @@ func ordering(records []stackencrypt.EncryptedRecord) error { return nil } -func countPositions(t stackencrypt.MatchTerm) int { +func countPositions(t encrypt.MatchTerm) int { positions, err := t.Positions() if err != nil { return -1 diff --git a/languages/golang/stackencrypt/export_test.go b/languages/golang/encrypt/export_test.go similarity index 95% rename from languages/golang/stackencrypt/export_test.go rename to languages/golang/encrypt/export_test.go index 22b844ae5..a7c3a2029 100644 --- a/languages/golang/stackencrypt/export_test.go +++ b/languages/golang/encrypt/export_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -8,7 +8,7 @@ import ( ) // The test-only ways to give a client a token. The public API takes tokens -// only from stackauth strategies; the tests that talk to httptest stubs +// only from auth strategies; the tests that talk to httptest stubs // need a fixed one, and get it here rather than through anything a caller // could reach. diff --git a/languages/golang/stackencrypt/guest.go b/languages/golang/encrypt/guest.go similarity index 95% rename from languages/golang/stackencrypt/guest.go rename to languages/golang/encrypt/guest.go index 7c521331f..62ff8f062 100644 --- a/languages/golang/stackencrypt/guest.go +++ b/languages/golang/encrypt/guest.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -27,7 +27,7 @@ const guestPath = "wasm/stack_encrypt_guest.wasm" // ErrGuestNotBuilt is returned by NewClient when no guest module is // embedded and none was supplied with WithGuest. -var ErrGuestNotBuilt = errors.New("stackencrypt: guest module not built — run `mise run wasm:guest:build`") +var ErrGuestNotBuilt = errors.New("encrypt: guest module not built — run `mise run wasm:guest:build`") func embeddedGuest() ([]byte, error) { wasm, err := guestFS.ReadFile(guestPath) @@ -106,7 +106,7 @@ func newInstance(ctx context.Context, wasm []byte, t *transport, policy guest.Lo // call that does not currently fail, matching the host transport below. if _, err := wasi_snapshot_preview1.Instantiate(ctx, runtime); err != nil { _ = runtime.Close(ctx) - return nil, fmt.Errorf("stackencrypt: instantiating WASI: %w", err) + return nil, fmt.Errorf("encrypt: instantiating WASI: %w", err) } if err := t.instantiate(ctx, runtime); err != nil { _ = runtime.Close(ctx) @@ -133,7 +133,7 @@ func newInstance(ctx context.Context, wasm []byte, t *transport, policy guest.Lo if g := mem.GrowthRefusal(); g.Refused != 0 { return nil, fmt.Errorf("%w: %w", guest.MemoryLockError(g.Reason), err) } - return nil, fmt.Errorf("stackencrypt: instantiating guest: %w", err) + return nil, fmt.Errorf("encrypt: instantiating guest: %w", err) } if policy == guest.Strict { if lerr := mem.LockError(); lerr != nil { @@ -159,7 +159,7 @@ func newInstance(ctx context.Context, wasm []byte, t *transport, policy guest.Lo for name, slot := range exports { if *slot = module.ExportedFunction(name); *slot == nil { _ = runtime.Close(ctx) - return nil, fmt.Errorf("stackencrypt: guest is missing export %s", name) + return nil, fmt.Errorf("encrypt: guest is missing export %s", name) } } return inst, nil diff --git a/languages/golang/stackencrypt/guest/.gitignore b/languages/golang/encrypt/guest/.gitignore similarity index 100% rename from languages/golang/stackencrypt/guest/.gitignore rename to languages/golang/encrypt/guest/.gitignore diff --git a/languages/golang/stackencrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock similarity index 100% rename from languages/golang/stackencrypt/guest/Cargo.lock rename to languages/golang/encrypt/guest/Cargo.lock diff --git a/languages/golang/stackencrypt/guest/Cargo.toml b/languages/golang/encrypt/guest/Cargo.toml similarity index 100% rename from languages/golang/stackencrypt/guest/Cargo.toml rename to languages/golang/encrypt/guest/Cargo.toml diff --git a/languages/golang/stackencrypt/guest/src/abi.rs b/languages/golang/encrypt/guest/src/abi.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/abi.rs rename to languages/golang/encrypt/guest/src/abi.rs diff --git a/languages/golang/stackencrypt/guest/src/config.rs b/languages/golang/encrypt/guest/src/config.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/config.rs rename to languages/golang/encrypt/guest/src/config.rs diff --git a/languages/golang/stackencrypt/guest/src/headers.rs b/languages/golang/encrypt/guest/src/headers.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/headers.rs rename to languages/golang/encrypt/guest/src/headers.rs diff --git a/languages/golang/stackencrypt/guest/src/host.rs b/languages/golang/encrypt/guest/src/host.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/host.rs rename to languages/golang/encrypt/guest/src/host.rs diff --git a/languages/golang/stackencrypt/guest/src/lib.rs b/languages/golang/encrypt/guest/src/lib.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/lib.rs rename to languages/golang/encrypt/guest/src/lib.rs diff --git a/languages/golang/stackencrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/ops.rs rename to languages/golang/encrypt/guest/src/ops.rs diff --git a/languages/golang/stackencrypt/guest/src/options.rs b/languages/golang/encrypt/guest/src/options.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/options.rs rename to languages/golang/encrypt/guest/src/options.rs diff --git a/languages/golang/stackencrypt/guest/src/response.rs b/languages/golang/encrypt/guest/src/response.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/response.rs rename to languages/golang/encrypt/guest/src/response.rs diff --git a/languages/golang/stackencrypt/guest/src/status.rs b/languages/golang/encrypt/guest/src/status.rs similarity index 100% rename from languages/golang/stackencrypt/guest/src/status.rs rename to languages/golang/encrypt/guest/src/status.rs diff --git a/languages/golang/stackencrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs similarity index 100% rename from languages/golang/stackencrypt/guest/tests/native_ops.rs rename to languages/golang/encrypt/guest/tests/native_ops.rs diff --git a/languages/golang/stackencrypt/guest_test.go b/languages/golang/encrypt/guest_test.go similarity index 99% rename from languages/golang/stackencrypt/guest_test.go rename to languages/golang/encrypt/guest_test.go index 770738ea7..dd830860b 100644 --- a/languages/golang/stackencrypt/guest_test.go +++ b/languages/golang/encrypt/guest_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "bytes" @@ -17,8 +17,8 @@ import ( "testing" "time" + "github.com/cipherstash/stack/languages/golang/auth" "github.com/cipherstash/stack/languages/golang/internal/guest" - "github.com/cipherstash/stack/languages/golang/stackauth" "github.com/cipherstash/vitaminc/bindings/go/vcvalue" "github.com/tetratelabs/wazero" "github.com/tetratelabs/wazero/sys" @@ -959,12 +959,12 @@ func TestTransportSendCounterAndResponseHeaders(t *testing.T) { func ExampleNewClient() { // With no options, NewClient uses AutoCredentials. To supply the - // credentials yourself, the token comes from a stackauth strategy — + // credentials yourself, the token comes from a auth strategy — // here an access key; live_test.go has a real round trip. The caller // opened the store and the strategy, and closes them after the client. ctx := context.Background() err := func() error { - store, err := stackauth.OpenWithoutProfile(ctx) + store, err := auth.OpenWithoutProfile(ctx) if err != nil { return err } diff --git a/languages/golang/stackencrypt/keyset.go b/languages/golang/encrypt/keyset.go similarity index 95% rename from languages/golang/stackencrypt/keyset.go rename to languages/golang/encrypt/keyset.go index 1131d0d10..3ae94a362 100644 --- a/languages/golang/stackencrypt/keyset.go +++ b/languages/golang/encrypt/keyset.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "encoding/hex" @@ -50,11 +50,11 @@ func (id KeysetID) String() string { func ParseKeysetID(s string) (KeysetID, error) { var id KeysetID if len(s) != 36 || s[8] != '-' || s[13] != '-' || s[18] != '-' || s[23] != '-' { - return id, fmt.Errorf("stackencrypt: %q is not a UUID", s) + return id, fmt.Errorf("encrypt: %q is not a UUID", s) } hexed := s[:8] + s[9:13] + s[14:18] + s[19:23] + s[24:] if _, err := hex.Decode(id[:], []byte(hexed)); err != nil { - return id, fmt.Errorf("stackencrypt: %q is not a UUID", s) + return id, fmt.Errorf("encrypt: %q is not a UUID", s) } return id, nil } diff --git a/languages/golang/stackencrypt/label.go b/languages/golang/encrypt/label.go similarity index 97% rename from languages/golang/stackencrypt/label.go rename to languages/golang/encrypt/label.go index fe8dcf416..c48944f3f 100644 --- a/languages/golang/stackencrypt/label.go +++ b/languages/golang/encrypt/label.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "errors" @@ -95,7 +95,7 @@ func (l Label) String() string { return strings.Join(l.segments, labelSeparator) func (l Label) Context() Context { return flatContext(l.segments) } // ErrEmptyLabel is [NewLabel]'s refusal of no segments at all. -var ErrEmptyLabel = errors.New("stackencrypt: a label needs at least one segment") +var ErrEmptyLabel = errors.New("encrypt: a label needs at least one segment") // LabelError says why a string is not a [Label] segment. Index is the // segment's position, counting from zero. @@ -105,7 +105,7 @@ type LabelError struct { } func (e *LabelError) Error() string { - return fmt.Sprintf("stackencrypt: label segment %d %s", e.Index, e.Reason) + return fmt.Sprintf("encrypt: label segment %d %s", e.Index, e.Reason) } // checkSegment is the one definition of plain text, the same as Rust's diff --git a/languages/golang/stackencrypt/label_test.go b/languages/golang/encrypt/label_test.go similarity index 99% rename from languages/golang/stackencrypt/label_test.go rename to languages/golang/encrypt/label_test.go index 929e5342e..b64104243 100644 --- a/languages/golang/stackencrypt/label_test.go +++ b/languages/golang/encrypt/label_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "encoding/json" diff --git a/languages/golang/stackencrypt/leaf.go b/languages/golang/encrypt/leaf.go similarity index 94% rename from languages/golang/stackencrypt/leaf.go rename to languages/golang/encrypt/leaf.go index 702dbbb66..37437e322 100644 --- a/languages/golang/stackencrypt/leaf.go +++ b/languages/golang/encrypt/leaf.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "database/sql/driver" @@ -75,9 +75,9 @@ func scanBytes(kind string, src any) ([]byte, error) { case string: return []byte(v), nil case nil: - return nil, fmt.Errorf("stackencrypt: cannot scan NULL into %s", kind) + return nil, fmt.Errorf("encrypt: cannot scan NULL into %s", kind) default: - return nil, fmt.Errorf("stackencrypt: cannot scan %T into %s", src, kind) + return nil, fmt.Errorf("encrypt: cannot scan %T into %s", src, kind) } } diff --git a/languages/golang/stackencrypt/live_test.go b/languages/golang/encrypt/live_test.go similarity index 96% rename from languages/golang/stackencrypt/live_test.go rename to languages/golang/encrypt/live_test.go index de8c405a3..14280958e 100644 --- a/languages/golang/stackencrypt/live_test.go +++ b/languages/golang/encrypt/live_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "bytes" @@ -9,7 +9,7 @@ import ( "strings" "testing" - "github.com/cipherstash/stack/languages/golang/stackauth" + "github.com/cipherstash/stack/languages/golang/auth" "github.com/cipherstash/vitaminc/bindings/go/vcvalue" ) @@ -23,10 +23,10 @@ import ( // seeded client (every live test); // - STACK_ENCRYPT_TEST_CLIENT_ACCESS_KEY, STACK_ENCRYPT_TEST_WORKSPACE_CRN: // an access key and its workspace, exchanged for a token (every live -// test) — by a stackauth access-key strategy given to NewCredentials +// test) — by a auth access-key strategy given to NewCredentials // (liveClient), and by AutoCredentials from the environment // (TestLiveAutoCredentialsFromTheEnvironment). There is no raw-token -// variable: the client takes tokens only from stackauth strategies; +// variable: the client takes tokens only from auth strategies; // - STACK_ENCRYPT_TEST_ZEROKMS_URL (optional): the ZeroKMS endpoint, else // the token's services claim; // - STACK_ENCRYPT_TEST_CTS_HOST (optional): the authentication endpoint @@ -45,18 +45,18 @@ func liveClient(t *testing.T) *Client { authGuestOrSkip(t) // The explicit path: the caller opens the store and the strategy, and // closes them after the client (cleanups run last-registered first). - store, err := stackauth.OpenWithoutProfile(t.Context()) + store, err := auth.OpenWithoutProfile(t.Context()) if err != nil { - t.Fatalf("stackauth.OpenWithoutProfile: %v", err) + t.Fatalf("auth.OpenWithoutProfile: %v", err) } t.Cleanup(func() { _ = store.Close() }) - var strategyOpts []stackauth.StrategyOption + var strategyOpts []auth.StrategyOption if cts := os.Getenv("STACK_ENCRYPT_TEST_CTS_HOST"); cts != "" { - strategyOpts = append(strategyOpts, stackauth.WithAuthBaseURL(cts)) + strategyOpts = append(strategyOpts, auth.WithAuthBaseURL(cts)) } strategy, err := store.AccessKey(t.Context(), crn, accessKey, strategyOpts...) if err != nil { - t.Fatalf("stackauth access-key strategy: %v", err) + t.Fatalf("auth access-key strategy: %v", err) } t.Cleanup(func() { if err := strategy.Close(); err != nil { diff --git a/languages/golang/stackencrypt/memory_linux_test.go b/languages/golang/encrypt/memory_linux_test.go similarity index 96% rename from languages/golang/stackencrypt/memory_linux_test.go rename to languages/golang/encrypt/memory_linux_test.go index 2577ec0be..d4002aff0 100644 --- a/languages/golang/stackencrypt/memory_linux_test.go +++ b/languages/golang/encrypt/memory_linux_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" diff --git a/languages/golang/stackencrypt/memory_test.go b/languages/golang/encrypt/memory_test.go similarity index 97% rename from languages/golang/stackencrypt/memory_test.go rename to languages/golang/encrypt/memory_test.go index 4df671d1e..ba4da6c30 100644 --- a/languages/golang/stackencrypt/memory_test.go +++ b/languages/golang/encrypt/memory_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -12,9 +12,9 @@ import ( "github.com/tetratelabs/wazero/api" + "github.com/cipherstash/stack/languages/golang/auth" "github.com/cipherstash/stack/languages/golang/internal/guest" "github.com/cipherstash/stack/languages/golang/internal/guesttest" - "github.com/cipherstash/stack/languages/golang/stackauth" ) // The allocator on its own is tested in internal/guest. These are the @@ -135,8 +135,8 @@ func TestRequireLockedMemoryRefusesACallerStoreUnlocked(t *testing.T) { t.Fatalf("lowering RLIMIT_MEMLOCK: %v", err) } ctx := context.Background() - store, err := stackauth.OpenWithoutProfile(ctx) - if errors.Is(err, stackauth.ErrGuestNotBuilt) { + store, err := auth.OpenWithoutProfile(ctx) + if errors.Is(err, auth.ErrGuestNotBuilt) { fmt.Println("case skipped:", err) return } @@ -148,7 +148,7 @@ func TestRequireLockedMemoryRefusesACallerStoreUnlocked(t *testing.T) { fmt.Println("case skipped: mlock succeeds under RLIMIT_MEMLOCK=0") return } - strategy, err := store.AccessKey(ctx, "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY", "CSAKtestKeyId.testKeySecret", stackauth.WithAuthBaseURL("https://cts.invalid")) + strategy, err := store.AccessKey(ctx, "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY", "CSAKtestKeyId.testKeySecret", auth.WithAuthBaseURL("https://cts.invalid")) if err != nil { t.Fatal(err) } diff --git a/languages/golang/stackencrypt/options.go b/languages/golang/encrypt/options.go similarity index 93% rename from languages/golang/stackencrypt/options.go rename to languages/golang/encrypt/options.go index d35b0eef4..fe725f103 100644 --- a/languages/golang/stackencrypt/options.go +++ b/languages/golang/encrypt/options.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import "net/http" @@ -24,7 +24,7 @@ type clientOptions struct { requireLockedMemory bool } -// WithCredentials supplies the client id, the client key and the stackauth +// WithCredentials supplies the client id, the client key and the auth // strategy the token comes from. The default, and what nil means, is // [AutoCredentials]: the environment, then the developer profile. // [NewCredentials] takes the three explicitly, and [OIDCFederation] mints tokens from an identity provider's. @@ -52,12 +52,12 @@ func WithKeysetCacheSize(n int) ClientOption { // WithTransport performs the client's HTTP requests: to ZeroKMS, and, for // [AutoCredentials] and [OIDCFederation], the authentication requests -// stackauth's credential guest makes to CTS: an access-key exchange, a +// auth's credential guest makes to CTS: an access-key exchange, a // device-session refresh, a federation exchange. A RoundTripper scoped to // the ZeroKMS host alone (a pinned client certificate, an egress allowlist) // refuses those; the failure then surfaces as the token strategy's. Under // [NewCredentials] the token exchange runs in the store the caller opened, -// not through this RoundTripper: pass stackauth.WithRoundTripper to that +// not through this RoundTripper: pass auth.WithRoundTripper to that // store instead. Nil means http.DefaultTransport, the default. func WithTransport(rt http.RoundTripper) ClientOption { return func(o *clientOptions) { o.transport = rt } @@ -87,10 +87,10 @@ func WithGuest(wasm []byte) ClientOption { // the client key may have passed through. [AutoCredentials] and // [OIDCFederation] open that guest under the same policy, so it is refused // at NewClient and on every later growth alike. [NewCredentials]' guest is -// the stackauth store the caller opened: NewClient fails with +// the auth store the caller opened: NewClient fails with // ErrMemoryLock if that store's memory is unlocked when it is asked, but // only the store's own policy governs its later growth, so open it with -// stackauth.RequireLockedMemory to hold it locked for the life of the +// auth.RequireLockedMemory to hold it locked for the life of the // client. Client.MemoryLocked reports the store's state live either way. func WithRequireLockedMemory() ClientOption { return func(o *clientOptions) { o.requireLockedMemory = true } diff --git a/languages/golang/stackencrypt/options_test.go b/languages/golang/encrypt/options_test.go similarity index 89% rename from languages/golang/stackencrypt/options_test.go rename to languages/golang/encrypt/options_test.go index d87ea23c0..571ea1401 100644 --- a/languages/golang/stackencrypt/options_test.go +++ b/languages/golang/encrypt/options_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -11,8 +11,8 @@ import ( "sync/atomic" "testing" + "github.com/cipherstash/stack/languages/golang/auth" "github.com/cipherstash/stack/languages/golang/internal/guest" - "github.com/cipherstash/stack/languages/golang/stackauth" ) // Every option sets its one field, and a later option setting the same @@ -94,7 +94,7 @@ func TestTheSameCredentialsPassedTwiceAreNotConsumed(t *testing.T) { // does not panic, whichever constructor made them. func TestCredentialsAreComparable(t *testing.T) { a := OIDCFederation("crn:a", nil) - b := OIDCFederation("crn:b", nil, stackauth.WithAuthBaseURL("https://cts.example.com")) + b := OIDCFederation("crn:b", nil, auth.WithAuthBaseURL("https://cts.example.com")) if a == b || AutoCredentials() != AutoCredentials() { t.Fatal("unexpected comparison result") } @@ -152,12 +152,12 @@ func TestOIDCFederationThroughTheClientTransport(t *testing.T) { guestOrSkip(t) authGuestOrSkip(t) cleanEnv(t, filepath.Join(t.TempDir(), "absent")) - auth := newAuthServer(t) + cts := newAuthServer(t) t.Setenv(envClientID, testClientID) t.Setenv(envClientKey, testClientKey) stub := newStub(t, http.StatusUnauthorized, "", "nope") var idpCalls atomic.Int32 - provider := stackauth.OIDCProviderFunc(func(context.Context) (string, error) { + provider := auth.OIDCProviderFunc(func(context.Context) (string, error) { idpCalls.Add(1) return "idp-token", nil }) @@ -170,13 +170,13 @@ func TestOIDCFederationThroughTheClientTransport(t *testing.T) { if !errors.Is(err, ErrUnauthorized) { t.Fatalf("NewClient: %v, want ErrUnauthorized from the stub", err) } - if idpCalls.Load() != 1 || auth.calls.Load() != 1 { - t.Fatalf("provider calls %d, exchanges %d; want one of each", idpCalls.Load(), auth.calls.Load()) + if idpCalls.Load() != 1 || cts.calls.Load() != 1 { + t.Fatalf("provider calls %d, exchanges %d; want one of each", idpCalls.Load(), cts.calls.Load()) } - if len(stub.requests) != 1 || stub.requests[0].auth != "Bearer "+auth.jwt { + if len(stub.requests) != 1 || stub.requests[0].auth != "Bearer "+cts.jwt { t.Fatalf("ZeroKMS requests = %+v, want one bearing the federated token", stub.requests) } - if rt.count(auth.URL) != 1 || rt.count(stub.URL) != 1 { + if rt.count(cts.URL) != 1 || rt.count(stub.URL) != 1 { t.Fatalf("transport saw %v, want the exchange and the ZeroKMS request", rt.hosts) } } @@ -184,7 +184,7 @@ func TestOIDCFederationThroughTheClientTransport(t *testing.T) { func TestOIDCFederationResolvesTheKeyLikeAuto(t *testing.T) { authGuestOrSkip(t) cleanEnv(t, newProfile(t, loggedIn("profile-token"))) - provider := stackauth.OIDCProviderFunc(func(context.Context) (string, error) { return "idp-token", nil }) + provider := auth.OIDCProviderFunc(func(context.Context) (string, error) { return "idp-token", nil }) resolved, err := OIDCFederation(testCRN, provider).resolve(context.Background(), resolveOptions{Transport: http.DefaultTransport}) if err != nil { t.Fatal(err) @@ -195,7 +195,7 @@ func TestOIDCFederationResolvesTheKeyLikeAuto(t *testing.T) { } // No CRN is a configuration error from the strategy, before any // provider or key is asked. - if _, err := OIDCFederation("", provider).resolve(context.Background(), resolveOptions{Transport: http.DefaultTransport}); !errors.Is(err, stackauth.ErrAuthConfig) { + if _, err := OIDCFederation("", provider).resolve(context.Background(), resolveOptions{Transport: http.DefaultTransport}); !errors.Is(err, auth.ErrAuthConfig) { t.Fatalf("OIDCFederation with no CRN: %v, want ErrAuthConfig", err) } } @@ -206,7 +206,7 @@ func TestOIDCFederationResolvesTheKeyLikeAuto(t *testing.T) { func TestOIDCFederationTakesStrategyOptions(t *testing.T) { authGuestOrSkip(t) cleanEnv(t, filepath.Join(t.TempDir(), "absent")) - auth := newAuthServer(t) + cts := newAuthServer(t) decoy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t.Errorf("CS_CTS_HOST was asked (%s) though WithAuthBaseURL pinned CTS", r.URL.Path) http.NotFound(w, r) @@ -215,18 +215,18 @@ func TestOIDCFederationTakesStrategyOptions(t *testing.T) { t.Setenv("CS_CTS_HOST", decoy.URL) t.Setenv(envClientID, testClientID) t.Setenv(envClientKey, testClientKey) - provider := stackauth.OIDCProviderFunc(func(context.Context) (string, error) { return "idp-token", nil }) - creds := OIDCFederation(testCRN, provider, stackauth.WithAuthBaseURL(auth.URL)) + provider := auth.OIDCProviderFunc(func(context.Context) (string, error) { return "idp-token", nil }) + creds := OIDCFederation(testCRN, provider, auth.WithAuthBaseURL(cts.URL)) resolved, err := creds.resolve(context.Background(), resolveOptions{Transport: http.DefaultTransport}) if err != nil { t.Fatal(err) } defer func() { _ = resolved.Close() }() resolved.ClientKey.Wipe() - if got := token(t, resolved); got != auth.jwt { + if got := token(t, resolved); got != cts.jwt { t.Fatalf("token = %q, want the pinned CTS's", got) } - if auth.calls.Load() != 1 { - t.Fatalf("exchanges at the pinned CTS: %d, want one", auth.calls.Load()) + if cts.calls.Load() != 1 { + t.Fatalf("exchanges at the pinned CTS: %d, want one", cts.calls.Load()) } } diff --git a/languages/golang/stackencrypt/order_live_test.go b/languages/golang/encrypt/order_live_test.go similarity index 99% rename from languages/golang/stackencrypt/order_live_test.go rename to languages/golang/encrypt/order_live_test.go index 6f4e3affe..7747ac37c 100644 --- a/languages/golang/stackencrypt/order_live_test.go +++ b/languages/golang/encrypt/order_live_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "bytes" diff --git a/languages/golang/stackencrypt/plan/doc.go b/languages/golang/encrypt/plan/doc.go similarity index 89% rename from languages/golang/stackencrypt/plan/doc.go rename to languages/golang/encrypt/plan/doc.go index 3614c8457..6cd6b86cd 100644 --- a/languages/golang/stackencrypt/plan/doc.go +++ b/languages/golang/encrypt/plan/doc.go @@ -1,4 +1,4 @@ -// Package plan builds a record [stackencrypt.Plan] from what a domain +// Package plan builds a record [encrypt.Plan] from what a domain // schema already says about its fields, through a policy written in Go. // // Storage decisions do not belong in the schema. The schema carries facts — @@ -12,20 +12,20 @@ // var category = plan.Key("fides.data_categories") // // var Base = plan.FirstOf( -// plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(stackencrypt.Equality))), -// plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(stackencrypt.Equality, stackencrypt.Match))), +// plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(encrypt.Equality))), +// plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(encrypt.Equality, encrypt.Match))), // plan.When(category.Under("user"), plan.Encrypt(plan.EQL())), // ) // // var Individuals = plan.ForMessage(&Individual{}, plan.Table("individuals"), // plan.FirstOf( -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(stackencrypt.Equality)), +// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), // plan.Column("medicare_number")), // ).OrElse(Base), // ) // // var individuals = plan.MustPlanFor(source, Individuals) -// // cipher.EncryptRecords(ctx, rows, stackencrypt.WithPlan(individuals)) +// // cipher.EncryptRecords(ctx, rows, encrypt.WithPlan(individuals)) // // # Facts // @@ -61,7 +61,7 @@ // (ALTER TABLE ... RENAME COLUMN) the rule stores into the new column and // pins the old identity with [Identity]: // -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(stackencrypt.Equality)), +// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), // plan.Column("medicare_num"), plan.Identity("medicare_number")) // // A [Custom] target supplies its context itself; [Column] names only its @@ -69,7 +69,7 @@ // // Nothing on the write path notices a context that changed: a rename with // no pin simply writes new rows under a new context. Check them in with a -// golden test ([github.com/cipherstash/stack/languages/golang/stackencrypt/plan/plantest.Golden]), +// golden test ([github.com/cipherstash/stack/languages/golang/encrypt/plan/plantest.Golden]), // which snapshots what the policy stores each field as and fails, naming // the pin, when a context changes: // diff --git a/languages/golang/stackencrypt/plan/fact.go b/languages/golang/encrypt/plan/fact.go similarity index 100% rename from languages/golang/stackencrypt/plan/fact.go rename to languages/golang/encrypt/plan/fact.go diff --git a/languages/golang/stackencrypt/plan/message.go b/languages/golang/encrypt/plan/message.go similarity index 84% rename from languages/golang/stackencrypt/plan/message.go rename to languages/golang/encrypt/plan/message.go index 2e818dbec..2ebbf9e9d 100644 --- a/languages/golang/stackencrypt/plan/message.go +++ b/languages/golang/encrypt/plan/message.go @@ -5,7 +5,7 @@ import ( "fmt" "reflect" - "github.com/cipherstash/stack/languages/golang/stackencrypt" + "github.com/cipherstash/stack/languages/golang/encrypt" ) var ( @@ -23,7 +23,7 @@ var ( // ErrNothingEncrypted is a message the policy encrypts no field of: // every classified field decided Plaintext, or none classified. Such a // message has no plan to build, and its records are stored without - // one — a [stackencrypt.Plan] always seals at least one field. + // one — a [encrypt.Plan] always seals at least one field. ErrNothingEncrypted = errors.New("the policy encrypts no field of the message") ) @@ -48,7 +48,7 @@ type Message struct { // // var Individuals = plan.ForMessage(&Individual{}, plan.Table("individuals"), // plan.FirstOf( -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(stackencrypt.Equality)), +// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), // plan.Column("medicare_number")), // ).OrElse(Base), // ) @@ -73,11 +73,11 @@ func (m Message) Decide(f Fact) (Decision, bool) { return m.policy.Decide(f) } // // Pure: no I/O, no client. Build is what [PlanFor] runs after reading the // facts, and what a generator or a golden test runs on facts it holds. -func (m Message) Build(facts []Fact) (stackencrypt.Plan, error) { +func (m Message) Build(facts []Fact) (encrypt.Plan, error) { if m.table == "" { - return stackencrypt.Plan{}, fmt.Errorf("plan: %s: a message needs a Table", messageName(m, facts)) + return encrypt.Plan{}, fmt.Errorf("plan: %s: a message needs a Table", messageName(m, facts)) } - var fields []stackencrypt.FieldPlan + var fields []encrypt.FieldPlan var errs []error // An EQL identity is one column's context: two fields sharing one would // bind each other's ciphertexts and terms. NewPlan refuses a shared @@ -101,23 +101,23 @@ func (m Message) Build(facts []Fact) (stackencrypt.Plan, error) { } } if len(errs) > 0 { - return stackencrypt.Plan{}, errors.Join(errs...) + return encrypt.Plan{}, errors.Join(errs...) } if len(fields) == 0 { - return stackencrypt.Plan{}, fmt.Errorf("plan: %s: %w; a message with nothing to encrypt needs no plan", messageName(m, facts), ErrNothingEncrypted) + return encrypt.Plan{}, fmt.Errorf("plan: %s: %w; a message with nothing to encrypt needs no plan", messageName(m, facts), ErrNothingEncrypted) } - p, err := stackencrypt.NewPlan(fields...) + p, err := encrypt.NewPlan(fields...) if err != nil { - return stackencrypt.Plan{}, fmt.Errorf("plan: %s: %w", messageName(m, facts), err) + return encrypt.Plan{}, fmt.Errorf("plan: %s: %w", messageName(m, facts), err) } return p, nil } // field decides one field: its plan field, its EQL identity ("" for a // Custom target), and whether it is planned. -func (m Message) field(f Fact) (stackencrypt.FieldPlan, string, bool, error) { - none := func(err error) (stackencrypt.FieldPlan, string, bool, error) { - return stackencrypt.FieldPlan{}, "", false, err +func (m Message) field(f Fact) (encrypt.FieldPlan, string, bool, error) { + none := func(err error) (encrypt.FieldPlan, string, bool, error) { + return encrypt.FieldPlan{}, "", false, err } d, ok := m.policy.Decide(f) if !ok { @@ -127,7 +127,7 @@ func (m Message) field(f Fact) (stackencrypt.FieldPlan, string, bool, error) { return none(ErrUnmatched) } switch d.verdict { - case encrypt: + case sealed: case plaintext: if d.column != "" { return none(fmt.Errorf("%w: Column(%q) pinned on a Plaintext field", ErrInvalid, d.column)) @@ -170,11 +170,11 @@ func (m Message) field(f Fact) (stackencrypt.FieldPlan, string, bool, error) { context, err := d.target.Context(Identifier{Table: string(m.table), Column: identity}) if err != nil { // Both errors stay reachable: ErrInvalid for the policy's caller, and - // the target's own (a *stackencrypt.LabelError, say) for one that + // the target's own (a *encrypt.LabelError, say) for one that // wants to know which name was wrong. return none(fmt.Errorf("%w: target %v: %w", ErrInvalid, d.target, err)) } - return stackencrypt.FieldPlan{ + return encrypt.FieldPlan{ Field: f.goField(), Name: column, Context: context, @@ -195,21 +195,21 @@ func messageName(m Message, facts []Fact) string { // type. A fact whose GoField the type does not have — a typo in a source, // a generated field renamed — is then an error here, not at the first // record call. -func PlanFor(src Source, m Message) (stackencrypt.Plan, error) { +func PlanFor(src Source, m Message) (encrypt.Plan, error) { if src == nil { - return stackencrypt.Plan{}, errors.New("plan: PlanFor needs a Source") + return encrypt.Plan{}, errors.New("plan: PlanFor needs a Source") } facts, err := src.Facts(m.msg) if err != nil { - return stackencrypt.Plan{}, err + return encrypt.Plan{}, err } p, err := m.Build(facts) if err != nil { - return stackencrypt.Plan{}, err + return encrypt.Plan{}, err } if t := structType(m.msg); t != nil { if err := p.Validate(t); err != nil { - return stackencrypt.Plan{}, fmt.Errorf("plan: %s: %w", messageName(m, facts), err) + return encrypt.Plan{}, fmt.Errorf("plan: %s: %w", messageName(m, facts), err) } } return p, nil @@ -231,7 +231,7 @@ func structType(msg any) reflect.Type { // MustPlanFor is [PlanFor] for startup: it panics when the policy does not // decide every classified field, or the plan does not bind to the message, // so a policy gap stops the process before it writes anything. -func MustPlanFor(src Source, m Message) stackencrypt.Plan { +func MustPlanFor(src Source, m Message) encrypt.Plan { p, err := PlanFor(src, m) if err != nil { panic(err) diff --git a/languages/golang/stackencrypt/plan/plan_test.go b/languages/golang/encrypt/plan/plan_test.go similarity index 98% rename from languages/golang/stackencrypt/plan/plan_test.go rename to languages/golang/encrypt/plan/plan_test.go index 1cdba0ce6..2abb03e85 100644 --- a/languages/golang/stackencrypt/plan/plan_test.go +++ b/languages/golang/encrypt/plan/plan_test.go @@ -7,9 +7,9 @@ import ( "strings" "testing" + se "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" "github.com/cipherstash/stack/languages/golang/internal/factstest" - se "github.com/cipherstash/stack/languages/golang/stackencrypt" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" ) var category = plan.Key("fides.data_categories") @@ -101,7 +101,7 @@ func TestUnmatchedFactFailsTheBuild(t *testing.T) { } // A message the policy encrypts nothing of has no plan to build: the error -// says so, rather than stackencrypt's "at least one field". +// says so, rather than encrypt's "at least one field". func TestNothingEncryptedIsItsOwnError(t *testing.T) { type audit struct { ID int64 @@ -260,7 +260,7 @@ func TestBuildRefusesMalformedDecisions(t *testing.T) { classified := []plan.Annotation{{Key: "k", Values: []string{"v"}}} fact := []plan.Fact{{Field: "a", Annotations: classified}} // Each case fails for its own reason: a sentinel, or the message the - // reason is spelled by where stackencrypt reports it. + // reason is spelled by where encrypt reports it. for name, tc := range map[string]struct { table plan.Table policy plan.Policy diff --git a/languages/golang/stackencrypt/plan/plantest/compare.go b/languages/golang/encrypt/plan/plantest/compare.go similarity index 99% rename from languages/golang/stackencrypt/plan/plantest/compare.go rename to languages/golang/encrypt/plan/plantest/compare.go index ecb892cad..93a48fcec 100644 --- a/languages/golang/stackencrypt/plan/plantest/compare.go +++ b/languages/golang/encrypt/plan/plantest/compare.go @@ -6,7 +6,7 @@ import ( "slices" "strings" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" ) // changes is what differs between a snapshot and the policy now, sorted diff --git a/languages/golang/stackencrypt/plan/plantest/golden_test.go b/languages/golang/encrypt/plan/plantest/golden_test.go similarity index 89% rename from languages/golang/stackencrypt/plan/plantest/golden_test.go rename to languages/golang/encrypt/plan/plantest/golden_test.go index 4d71d86b4..861676cf9 100644 --- a/languages/golang/stackencrypt/plan/plantest/golden_test.go +++ b/languages/golang/encrypt/plan/plantest/golden_test.go @@ -3,10 +3,10 @@ package plantest_test import ( "testing" + se "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" + "github.com/cipherstash/stack/languages/golang/encrypt/plan/plantest" "github.com/cipherstash/stack/languages/golang/internal/factstest" - se "github.com/cipherstash/stack/languages/golang/stackencrypt" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan/plantest" ) var category = plan.Key("fides.data_categories") diff --git a/languages/golang/stackencrypt/plan/plantest/plantest.go b/languages/golang/encrypt/plan/plantest/plantest.go similarity index 99% rename from languages/golang/stackencrypt/plan/plantest/plantest.go rename to languages/golang/encrypt/plan/plantest/plantest.go index 7bdd29172..1dfd06101 100644 --- a/languages/golang/stackencrypt/plan/plantest/plantest.go +++ b/languages/golang/encrypt/plan/plantest/plantest.go @@ -62,7 +62,7 @@ import ( "strings" "testing" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" ) func init() { diff --git a/languages/golang/stackencrypt/plan/plantest/plantest_internal_test.go b/languages/golang/encrypt/plan/plantest/plantest_internal_test.go similarity index 99% rename from languages/golang/stackencrypt/plan/plantest/plantest_internal_test.go rename to languages/golang/encrypt/plan/plantest/plantest_internal_test.go index 0996d0988..d945754e0 100644 --- a/languages/golang/stackencrypt/plan/plantest/plantest_internal_test.go +++ b/languages/golang/encrypt/plan/plantest/plantest_internal_test.go @@ -13,9 +13,9 @@ import ( "strings" "testing" + se "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" "github.com/cipherstash/stack/languages/golang/internal/factstest" - se "github.com/cipherstash/stack/languages/golang/stackencrypt" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" ) var category = plan.Key("fides.data_categories") diff --git a/languages/golang/stackencrypt/plan/plantest/snapshot.go b/languages/golang/encrypt/plan/plantest/snapshot.go similarity index 96% rename from languages/golang/stackencrypt/plan/plantest/snapshot.go rename to languages/golang/encrypt/plan/plantest/snapshot.go index 31b9c26ab..8c9a3580f 100644 --- a/languages/golang/stackencrypt/plan/plantest/snapshot.go +++ b/languages/golang/encrypt/plan/plantest/snapshot.go @@ -8,8 +8,8 @@ import ( "strconv" "strings" - "github.com/cipherstash/stack/languages/golang/stackencrypt" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" ) // header opens every snapshot. One sentence per line: the file is read in @@ -95,8 +95,8 @@ func targetKind(t plan.Target) (string, error) { // a Custom column's is the label [plan.Custom] parses from its argument. // The shape is checked against the context the plan actually binds, so // the file cannot name a context the plan does not use. -func contextText(table string, kind string, d plan.Decision, fp stackencrypt.FieldPlan) (string, error) { - var label stackencrypt.Label +func contextText(table string, kind string, d plan.Decision, fp encrypt.FieldPlan) (string, error) { + var label encrypt.Label switch kind { case kindEQL: identity := d.Identity() @@ -122,7 +122,7 @@ func contextText(table string, kind string, d plan.Decision, fp stackencrypt.Fie if !ok || err != nil { return "", fmt.Errorf("plantest: column %q: cannot spell the context of target %v; a target that does not bind its column identity must be plan.Custom", fp.Name, target) } - if label, err = stackencrypt.ParseLabel(text); err != nil { + if label, err = encrypt.ParseLabel(text); err != nil { return "", fmt.Errorf("plantest: column %q: %w", fp.Name, err) } } @@ -177,14 +177,14 @@ func segmentsOf(spelled string) ([]string, error) { // contextOf is the context a snapshot's column names, rebuilt from its // segments: what [contextText] wrote. -func contextOf(c column) (stackencrypt.Context, error) { +func contextOf(c column) (encrypt.Context, error) { segments, err := segmentsOf(c.context) if err != nil { - return stackencrypt.Context{}, err + return encrypt.Context{}, err } - label, err := stackencrypt.NewLabel(segments...) + label, err := encrypt.NewLabel(segments...) if err != nil { - return stackencrypt.Context{}, err + return encrypt.Context{}, err } return label.Context(), nil } @@ -207,7 +207,7 @@ func take(src plan.Source, m plan.Message) (snapshot, []plan.Fact, error) { if err != nil { return snapshot{}, nil, err } - planned := map[string]stackencrypt.FieldPlan{} + planned := map[string]encrypt.FieldPlan{} for _, fp := range p.Fields() { planned[fp.Field] = fp } diff --git a/languages/golang/stackencrypt/plan/plantest/testdata/TestPolicies/audits.golden b/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/audits.golden similarity index 100% rename from languages/golang/stackencrypt/plan/plantest/testdata/TestPolicies/audits.golden rename to languages/golang/encrypt/plan/plantest/testdata/TestPolicies/audits.golden diff --git a/languages/golang/stackencrypt/plan/plantest/testdata/TestPolicies/individuals.golden b/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/individuals.golden similarity index 100% rename from languages/golang/stackencrypt/plan/plantest/testdata/TestPolicies/individuals.golden rename to languages/golang/encrypt/plan/plantest/testdata/TestPolicies/individuals.golden diff --git a/languages/golang/stackencrypt/plan/policy.go b/languages/golang/encrypt/plan/policy.go similarity index 88% rename from languages/golang/stackencrypt/plan/policy.go rename to languages/golang/encrypt/plan/policy.go index 6e8c8ac97..2811b3fd2 100644 --- a/languages/golang/stackencrypt/plan/policy.go +++ b/languages/golang/encrypt/plan/policy.go @@ -6,7 +6,7 @@ import ( "slices" "strings" - "github.com/cipherstash/stack/languages/golang/stackencrypt" + "github.com/cipherstash/stack/languages/golang/encrypt" ) // Identifier is a field's column identity: the table its message is @@ -26,14 +26,14 @@ type Identifier struct { // [Identifier.Label], which refuses a name this joining would misrender. func (id Identifier) String() string { return id.Table + "/" + id.Column } -// Label is the identifier as the two-segment [stackencrypt.Label] an EQL +// Label is the identifier as the two-segment [encrypt.Label] an EQL // target binds: the shape the Rust derive gives a // `#[stash(struct = T, context = "")]` field, and what EQL's own // Identifier describes. It is refused when either half is not plain — // contains '/', '(' or ')', a control character, or begins with "b64:", a // digit or '-' — since such a name would not render as itself. -func (id Identifier) Label() (stackencrypt.Label, error) { - return stackencrypt.NewLabel(id.Table, id.Column) +func (id Identifier) Label() (encrypt.Label, error) { + return encrypt.NewLabel(id.Table, id.Column) } // Target is what an encrypted field is stored as: the index terms derived @@ -41,34 +41,34 @@ func (id Identifier) Label() (stackencrypt.Label, error) { // AAD, the ZeroKMS data-key binding and the terms' PRF context at once. type Target interface { // Terms lists the index terms to derive, in order. - Terms() []stackencrypt.TermKind + Terms() []encrypt.TermKind // Context returns the field's context given its column identity. An // EQL target binds id.Label(); a custom target returns its own. - Context(id Identifier) (stackencrypt.Context, error) + Context(id Identifier) (encrypt.Context, error) } // EQL is an EQL column target: the field binds its column identity // ([Identifier.Label]) as its context and derives the given terms. Typed // EQL targets (a text-with-equality column, say) are this with the terms // filled in, and implement [Target] the same way. -func EQL(terms ...stackencrypt.TermKind) Target { +func EQL(terms ...encrypt.TermKind) Target { return eqlTarget{terms: slices.Clone(terms)} } -type eqlTarget struct{ terms []stackencrypt.TermKind } +type eqlTarget struct{ terms []encrypt.TermKind } -func (t eqlTarget) Terms() []stackencrypt.TermKind { return slices.Clone(t.terms) } -func (t eqlTarget) Context(id Identifier) (stackencrypt.Context, error) { +func (t eqlTarget) Terms() []encrypt.TermKind { return slices.Clone(t.terms) } +func (t eqlTarget) Context(id Identifier) (encrypt.Context, error) { l, err := id.Label() if err != nil { // The label error names a segment index; the caller gave a table and // a column identity, so say which of those it was. half, name := "table", id.Table - var le *stackencrypt.LabelError + var le *encrypt.LabelError if errors.As(err, &le) && le.Index == 1 { half, name = "column identity", id.Column } - return stackencrypt.Context{}, fmt.Errorf("%s %q cannot name a context: %w", half, name, err) + return encrypt.Context{}, fmt.Errorf("%s %q cannot name a context: %w", half, name, err) } return l.Context(), nil } @@ -78,27 +78,27 @@ func (t eqlTarget) String() string { return "EQL(" + termList(t.terms) + ")" } // column, and derives the given terms. The context is the policy's to // choose and, like any context, must never change once data is written // under it. It is a label of at least two plain segments, written as -// [stackencrypt.ParseLabel] reads it ("notes/v1": the segments notes and +// [encrypt.ParseLabel] reads it ("notes/v1": the segments notes and // v1, rendered as written in the ZeroKMS log), since that is the one shape // of context a planned field binds; the message's table plays no part in // it. A table and a column are an [EQL] target. -func Custom(context string, terms ...stackencrypt.TermKind) Target { +func Custom(context string, terms ...encrypt.TermKind) Target { return customTarget{context: context, terms: slices.Clone(terms)} } type customTarget struct { context string - terms []stackencrypt.TermKind + terms []encrypt.TermKind } -func (t customTarget) Terms() []stackencrypt.TermKind { return slices.Clone(t.terms) } -func (t customTarget) Context(Identifier) (stackencrypt.Context, error) { +func (t customTarget) Terms() []encrypt.TermKind { return slices.Clone(t.terms) } +func (t customTarget) Context(Identifier) (encrypt.Context, error) { if t.context == "" { - return stackencrypt.Context{}, errors.New("an empty string is an empty context") + return encrypt.Context{}, errors.New("an empty string is an empty context") } - l, err := stackencrypt.ParseLabel(t.context) + l, err := encrypt.ParseLabel(t.context) if err != nil { - return stackencrypt.Context{}, fmt.Errorf("context %q is not a label: %w", t.context, err) + return encrypt.Context{}, fmt.Errorf("context %q is not a label: %w", t.context, err) } return l.Context(), nil } @@ -106,7 +106,7 @@ func (t customTarget) String() string { return fmt.Sprintf("Custom(%q%s)", t.context, prefixed(termList(t.terms))) } -func termList(terms []stackencrypt.TermKind) string { +func termList(terms []encrypt.TermKind) string { names := make([]string, len(terms)) for i, k := range terms { names[i] = k.String() @@ -124,7 +124,7 @@ func prefixed(s string) string { type verdict uint8 const ( - encrypt verdict = iota + 1 + sealed verdict = iota + 1 plaintext fail ) @@ -141,7 +141,7 @@ type Decision struct { } // Encrypt decides that the field is encrypted into target. -func Encrypt(target Target) Decision { return Decision{verdict: encrypt, target: target} } +func Encrypt(target Target) Decision { return Decision{verdict: sealed, target: target} } // Plaintext decides that the field is stored as it is: not part of the // plan, never sent to the guest. @@ -152,7 +152,7 @@ func Plaintext() Decision { return Decision{verdict: plaintext} } func Fail(reason string) Decision { return Decision{verdict: fail, reason: reason} } // Target returns the decision's target, and whether it encrypts at all. -func (d Decision) Target() (Target, bool) { return d.target, d.verdict == encrypt } +func (d Decision) Target() (Target, bool) { return d.target, d.verdict == sealed } // Column returns the column the decision stores the field in, or "" for // the field's own name. @@ -166,7 +166,7 @@ func (d Decision) Identity() string { return d.identity } func (d Decision) String() string { var s string switch d.verdict { - case encrypt: + case sealed: s = fmt.Sprintf("Encrypt(%v)", d.target) case plaintext: s = "Plaintext()" @@ -220,7 +220,7 @@ func FirstOf(policies ...Policy) Policy { type RuleOption func(*Decision) // Column names the column an encrypted field is stored in: its record -// key ([stackencrypt.FieldPlan.Name]). It defaults to the field's schema +// key ([encrypt.FieldPlan.Name]). It defaults to the field's schema // name, so a rule needs it only when the two differ — after the field is // renamed in the schema, say. For an EQL target, the column also sets the // field's identity unless [Identity] pins another: on a field never @@ -246,7 +246,7 @@ func Column(name string) RuleOption { // no longer match queries. A database rename (ALTER TABLE ... RENAME // COLUMN) is therefore spelled as the new column and the old identity: // -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(stackencrypt.Equality)), +// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), // plan.Column("medicare_no"), plan.Identity("medicare_number")) // // Only meaningful with an EQL [Encrypt]: a [Custom] target's context is diff --git a/languages/golang/stackencrypt/policy_plan_test.go b/languages/golang/encrypt/policy_plan_test.go similarity index 94% rename from languages/golang/stackencrypt/policy_plan_test.go rename to languages/golang/encrypt/policy_plan_test.go index 4363a576c..006142f3d 100644 --- a/languages/golang/stackencrypt/policy_plan_test.go +++ b/languages/golang/encrypt/policy_plan_test.go @@ -1,4 +1,4 @@ -package stackencrypt_test +package encrypt_test import ( "bytes" @@ -6,9 +6,9 @@ import ( "strings" "testing" + se "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" "github.com/cipherstash/stack/languages/golang/internal/factstest" - se "github.com/cipherstash/stack/languages/golang/stackencrypt" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" ) // Validate refuses a nil type as PlanFromTags does, for the zero plan @@ -28,7 +28,7 @@ func TestValidateRefusesANilType(t *testing.T) { // A plan a policy builds is a Plan like any other: the guest receives // byte-identical input to the equivalent plan built by hand. This test is // here, not in package plan, because the guest encoding is unexported; -// plan imports stackencrypt, so only an external test can hold both. +// plan imports encrypt, so only an external test can hold both. func TestPolicyPlanIsTheHandBuiltPlan(t *testing.T) { type individual struct { ID int64 diff --git a/languages/golang/stackencrypt/record.go b/languages/golang/encrypt/record.go similarity index 92% rename from languages/golang/stackencrypt/record.go rename to languages/golang/encrypt/record.go index a8d882885..ac8651607 100644 --- a/languages/golang/stackencrypt/record.go +++ b/languages/golang/encrypt/record.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -43,28 +43,28 @@ import ( // table and the column; [ParseLabel] refuses a name that would not render // as itself, so check its error: // -// age, err := stackencrypt.ParseLabel("users/age") -// email, err := stackencrypt.ParseLabel("users/email") -// notes, err := stackencrypt.ParseLabel("users/notes") -// plan, err := stackencrypt.NewPlan( -// stackencrypt.FieldPlan{ +// age, err := encrypt.ParseLabel("users/age") +// email, err := encrypt.ParseLabel("users/email") +// notes, err := encrypt.ParseLabel("users/notes") +// plan, err := encrypt.NewPlan( +// encrypt.FieldPlan{ // Field: "Age", // Context: age.Context(), -// Terms: []stackencrypt.TermKind{ -// stackencrypt.Equality, stackencrypt.Ore, +// Terms: []encrypt.TermKind{ +// encrypt.Equality, encrypt.Ore, // }, // }, -// stackencrypt.FieldPlan{ +// encrypt.FieldPlan{ // Field: "Email", // Context: email.Context(), -// Terms: []stackencrypt.TermKind{ -// stackencrypt.Equality, stackencrypt.Match, +// Terms: []encrypt.TermKind{ +// encrypt.Equality, encrypt.Match, // }, // }, -// stackencrypt.FieldPlan{Field: "Notes", Context: notes.Context()}, +// encrypt.FieldPlan{Field: "Notes", Context: notes.Context()}, // ) // records, err := cipher.EncryptRecords( -// ctx, users, stackencrypt.WithPlan(plan), +// ctx, users, encrypt.WithPlan(plan), // ) // // Every planned field is sealed (the `"c"` output). What the guest receives @@ -106,9 +106,9 @@ type RecordOption interface { // Every Option is a RecordOption, so one value serves the encrypt, decrypt // and probe calls alike, and the three cannot drift apart: // -// tenant := stackencrypt.ExtendContext(uint64(tenantID)) +// tenant := encrypt.ExtendContext(uint64(tenantID)) // rows, err := cipher.EncryptRecords(ctx, users, tenant) -// probe, err := cipher.Term(ctx, "bob@example.com", email, stackencrypt.Equality, tenant) +// probe, err := cipher.Term(ctx, "bob@example.com", email, encrypt.Equality, tenant) // err = cipher.DecryptRecords(ctx, rows, &back, tenant) type Option interface { RecordOption @@ -280,7 +280,7 @@ func (f planField) outputs() []string { func NewPlan(fields ...FieldPlan) (Plan, error) { p, err := newPlan(fields) if err != nil { - return Plan{}, fmt.Errorf("stackencrypt: %w", err) + return Plan{}, fmt.Errorf("encrypt: %w", err) } return p, nil } @@ -351,13 +351,13 @@ var tagPlans sync.Map // reflect.Type → Plan // checked by the same validation NewPlan runs. func PlanFromTags(t reflect.Type) (Plan, error) { if t == nil { - return Plan{}, errors.New("stackencrypt: records must be structs, not a nil type") + return Plan{}, errors.New("encrypt: records must be structs, not a nil type") } if cached, ok := tagPlans.Load(t); ok { return cached.(Plan), nil } if t.Kind() != reflect.Struct { - return Plan{}, fmt.Errorf("stackencrypt: records must be structs, not %s", t) + return Plan{}, fmt.Errorf("encrypt: records must be structs, not %s", t) } var fields []FieldPlan for i := 0; i < t.NumField(); i++ { @@ -376,7 +376,7 @@ func PlanFromTags(t reflect.Type) (Plan, error) { switch key { case "label", "context": if ownKey != "" { - return Plan{}, fmt.Errorf("stackencrypt: field %s.%s: %s= and %s= both given; a field has one own context", t, f.Name, ownKey, key) + return Plan{}, fmt.Errorf("encrypt: field %s.%s: %s= and %s= both given; a field has one own context", t, f.Name, ownKey, key) } ownKey, ownValue = key, value var err error @@ -389,23 +389,23 @@ func PlanFromTags(t reflect.Type) (Plan, error) { pf.Context, err = NewContext(value) } if err != nil { - return Plan{}, fmt.Errorf("stackencrypt: field %s.%s: %s=%q: %w", t, f.Name, key, value, err) + return Plan{}, fmt.Errorf("encrypt: field %s.%s: %s=%q: %w", t, f.Name, key, value, err) } case "name": if value == "" { - return Plan{}, fmt.Errorf("stackencrypt: field %s.%s: name must not be empty", t, f.Name) + return Plan{}, fmt.Errorf("encrypt: field %s.%s: name must not be empty", t, f.Name) } pf.Name = value case "index": for _, k := range strings.Split(value, ";") { kind, ok := parseTermKind(k) if !ok { - return Plan{}, fmt.Errorf("stackencrypt: field %s.%s: unknown term kind %q", t, f.Name, k) + return Plan{}, fmt.Errorf("encrypt: field %s.%s: unknown term kind %q", t, f.Name, k) } pf.Terms = append(pf.Terms, kind) } default: - return Plan{}, fmt.Errorf("stackencrypt: field %s.%s: unknown stash tag option %q", t, f.Name, opt) + return Plan{}, fmt.Errorf("encrypt: field %s.%s: unknown stash tag option %q", t, f.Name, opt) } } // Checked after the options so a repeated or doubled option is @@ -417,11 +417,11 @@ func PlanFromTags(t reflect.Type) (Plan, error) { fields = append(fields, pf) } if len(fields) == 0 { - return Plan{}, fmt.Errorf("stackencrypt: %s has no fields tagged for encryption", t) + return Plan{}, fmt.Errorf("encrypt: %s has no fields tagged for encryption", t) } plan, err := newPlan(fields) if err != nil { - return Plan{}, fmt.Errorf("stackencrypt: %s: %w", t, err) + return Plan{}, fmt.Errorf("encrypt: %s: %w", t, err) } tagPlans.Store(t, plan) return plan, nil @@ -448,7 +448,7 @@ func (p Plan) Validate(t reflect.Type) error { return err } if t == nil { - return errors.New("stackencrypt: records must be structs, not a nil type") + return errors.New("encrypt: records must be structs, not a nil type") } _, err := p.bind(t) return err @@ -459,13 +459,13 @@ func (p Plan) Validate(t reflect.Type) error { // a cache keyed by plan would grow with every plan a caller ever built. func (p Plan) bind(t reflect.Type) ([]fieldPlan, error) { if t.Kind() != reflect.Struct { - return nil, fmt.Errorf("stackencrypt: records must be structs, not %s", t) + return nil, fmt.Errorf("encrypt: records must be structs, not %s", t) } bound := make([]fieldPlan, len(p.d.fields)) for i, f := range p.d.fields { sf, ok := t.FieldByName(f.field) if !ok || !sf.IsExported() || len(sf.Index) != 1 { - return nil, fmt.Errorf("stackencrypt: plan field %s is not an exported field of %s", f.field, t) + return nil, fmt.Errorf("encrypt: plan field %s is not an exported field of %s", f.field, t) } bound[i] = fieldPlan{index: sf.Index[0], name: f.name, context: f.context, outputs: f.outputs()} } @@ -522,7 +522,7 @@ func applyOptions(opts []RecordOption) recordOptions { func (cph *Cipher) EncryptRecords(ctx context.Context, rows any, opts ...RecordOption) ([]EncryptedRecord, error) { v := reflect.Indirect(reflect.ValueOf(rows)) if !v.IsValid() || v.Kind() != reflect.Slice { - return nil, fmt.Errorf("stackencrypt: EncryptRecords takes a slice of structs, not %T", rows) + return nil, fmt.Errorf("encrypt: EncryptRecords takes a slice of structs, not %T", rows) } o := applyOptions(opts) plan, err := planFor(v.Type().Elem(), o) @@ -555,7 +555,7 @@ func (cph *Cipher) EncryptRecords(ctx context.Context, rows any, opts ...RecordO func (cph *Cipher) EncryptRecord(ctx context.Context, row any, opts ...RecordOption) (EncryptedRecord, error) { v := reflect.Indirect(reflect.ValueOf(row)) if !v.IsValid() { - return nil, fmt.Errorf("stackencrypt: EncryptRecord takes a struct, not %T", row) + return nil, fmt.Errorf("encrypt: EncryptRecord takes a struct, not %T", row) } o := applyOptions(opts) plan, err := planFor(v.Type(), o) @@ -679,7 +679,7 @@ func (cph *Cipher) DecryptRecord(ctx context.Context, record EncryptedRecord, ou func (c *Client) decryptRecords(ctx context.Context, sel KeysetSelector, records []EncryptedRecord, out any, opts []RecordOption) error { ptr := reflect.ValueOf(out) if ptr.Kind() != reflect.Pointer || ptr.IsNil() || ptr.Elem().Kind() != reflect.Slice { - return fmt.Errorf("stackencrypt: DecryptRecords writes into a pointer to a slice of structs, not %T", out) + return fmt.Errorf("encrypt: DecryptRecords writes into a pointer to a slice of structs, not %T", out) } o := applyOptions(opts) plan, err := planFor(ptr.Elem().Type().Elem(), o) @@ -737,7 +737,7 @@ func commitRecord(target reflect.Value, item any, plan []fieldPlan) error { func (c *Client) decryptRecord(ctx context.Context, sel KeysetSelector, record EncryptedRecord, out any, opts []RecordOption) error { ptr := reflect.ValueOf(out) if ptr.Kind() != reflect.Pointer || ptr.IsNil() || ptr.Elem().Kind() != reflect.Struct { - return fmt.Errorf("stackencrypt: DecryptRecord writes into a pointer to a struct, not %T", out) + return fmt.Errorf("encrypt: DecryptRecord writes into a pointer to a struct, not %T", out) } o := applyOptions(opts) plan, err := planFor(ptr.Elem().Type(), o) @@ -762,7 +762,7 @@ func recordTree(rec EncryptedRecord, plan []fieldPlan) (map[string]any, error) { for _, f := range plan { field, ok := rec[f.name] if !ok || field.Ciphertext == nil { - return nil, fmt.Errorf("stackencrypt: record has no ciphertext for field %q", f.name) + return nil, fmt.Errorf("encrypt: record has no ciphertext for field %q", f.name) } tree[f.name] = map[string]any{"c": field.Ciphertext} } @@ -813,7 +813,7 @@ func assignRecord(target reflect.Value, value any, plan []fieldPlan) error { return fmt.Errorf("%w: decrypted record lacks field %q", ErrInternal, f.name) } if err := assignField(target.Field(f.index), v); err != nil { - return fmt.Errorf("stackencrypt: field %q: %w", f.name, err) + return fmt.Errorf("encrypt: field %q: %w", f.name, err) } } return nil diff --git a/languages/golang/stackencrypt/runtime_test.go b/languages/golang/encrypt/runtime_test.go similarity index 99% rename from languages/golang/stackencrypt/runtime_test.go rename to languages/golang/encrypt/runtime_test.go index 1abdb3905..ab916c806 100644 --- a/languages/golang/stackencrypt/runtime_test.go +++ b/languages/golang/encrypt/runtime_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "bytes" diff --git a/languages/golang/stackencrypt/term.go b/languages/golang/encrypt/term.go similarity index 97% rename from languages/golang/stackencrypt/term.go rename to languages/golang/encrypt/term.go index fbf38436e..d9dc40ee3 100644 --- a/languages/golang/stackencrypt/term.go +++ b/languages/golang/encrypt/term.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "bytes" @@ -74,7 +74,7 @@ type MatchTerm []byte // Positions decodes the term into its token positions. func (t MatchTerm) Positions() ([]uint16, error) { if len(t)%2 != 0 { - return nil, fmt.Errorf("stackencrypt: match term of %d bytes is not a whole number of positions", len(t)) + return nil, fmt.Errorf("encrypt: match term of %d bytes is not a whole number of positions", len(t)) } out := make([]uint16, len(t)/2) for i := range out { diff --git a/languages/golang/stackencrypt/testdata/cllw_order.txt b/languages/golang/encrypt/testdata/cllw_order.txt similarity index 100% rename from languages/golang/stackencrypt/testdata/cllw_order.txt rename to languages/golang/encrypt/testdata/cllw_order.txt diff --git a/languages/golang/stackencrypt/transport.go b/languages/golang/encrypt/transport.go similarity index 95% rename from languages/golang/stackencrypt/transport.go rename to languages/golang/encrypt/transport.go index e9514eb4b..85e349555 100644 --- a/languages/golang/stackencrypt/transport.go +++ b/languages/golang/encrypt/transport.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "context" @@ -23,8 +23,8 @@ const transportModule = "cipherstash_transport" // tokenSource supplies the bearer token the guest presents to ZeroKMS. It // is asked on every request, so a source that rotates tokens needs no // re-initialisation of the client. Outside this package's tests it is -// always a *stackauth.Strategy: minting and refresh stay host-side, out of -// the crypto guest, in stackauth's credential guest. The interface is +// always a *auth.Strategy: minting and refresh stay host-side, out of +// the crypto guest, in auth's credential guest. The interface is // unexported so that no caller can hand the client a raw token, which could // not be refreshed and would bypass the strategies' refresh lock. type tokenSource interface { @@ -69,7 +69,7 @@ func (t *transport) instantiate(ctx context.Context, r wazero.Runtime) error { NewFunctionBuilder().WithFunc(t.tokenGet).Export("token_get"). Instantiate(ctx) if err != nil { - return fmt.Errorf("stackencrypt: instantiating host transport: %w", err) + return fmt.Errorf("encrypt: instantiating host transport: %w", err) } return nil } @@ -176,7 +176,7 @@ type requestBody struct { closed bool } -var errRequestBodyClosed = errors.New("stackencrypt: request body read after close") +var errRequestBodyClosed = errors.New("encrypt: request body read after close") // newRequestBody copies src, which the caller does not keep alive. func newRequestBody(src []byte) *requestBody { @@ -218,7 +218,7 @@ func (t *transport) tokenGet(ctx context.Context, m api.Module, tokenPtrOut, tok return hostFailed } if token == "" { - t.tokenErr = errors.New("stackencrypt: the token source returned an empty token") + t.tokenErr = errors.New("encrypt: the token source returned an empty token") return hostFailed } // The credential's transport copy is wiped once it is in guest memory; @@ -229,7 +229,7 @@ func (t *transport) tokenGet(ctx context.Context, m api.Module, tokenPtrOut, tok // The source did its part; the guest could not take the token (no // allocator, a refused allocation, an out-of-range slot). Say so, // or the failure reads as the source's. - t.tokenErr = errors.New("stackencrypt: the token could not be handed to the guest") + t.tokenErr = errors.New("encrypt: the token could not be handed to the guest") return hostFailed } return 0 diff --git a/languages/golang/stackencrypt/unit_test.go b/languages/golang/encrypt/unit_test.go similarity index 99% rename from languages/golang/stackencrypt/unit_test.go rename to languages/golang/encrypt/unit_test.go index a8b9fa1e5..50d0c9ecc 100644 --- a/languages/golang/stackencrypt/unit_test.go +++ b/languages/golang/encrypt/unit_test.go @@ -1,4 +1,4 @@ -package stackencrypt +package encrypt import ( "bufio" @@ -762,7 +762,7 @@ func TestLeafSetKeepsStackEncryptLeavesDistinct(t *testing.T) { t.Fatal("vcvalue.Sealed accepted as a stack-encrypt leaf") } if _, err := vcffi.MarshalCipherText(vcffi.VCValueLeaves(), Sealed{1}); err == nil { - t.Fatal("stackencrypt.Sealed accepted as a vitaminc leaf") + t.Fatal("encrypt.Sealed accepted as a vitaminc leaf") } } diff --git a/languages/golang/stackencrypt/wasm/README.md b/languages/golang/encrypt/wasm/README.md similarity index 100% rename from languages/golang/stackencrypt/wasm/README.md rename to languages/golang/encrypt/wasm/README.md diff --git a/languages/golang/internal/factstest/factstest.go b/languages/golang/internal/factstest/factstest.go index 38c26e46e..d00406cb7 100644 --- a/languages/golang/internal/factstest/factstest.go +++ b/languages/golang/internal/factstest/factstest.go @@ -12,7 +12,7 @@ import ( "strings" "unicode" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" ) // StructTags is a Go-struct fact source for tests: one fact per exported, diff --git a/languages/golang/internal/factstest/factstest_test.go b/languages/golang/internal/factstest/factstest_test.go index b3e4b7b1c..1c1d9017c 100644 --- a/languages/golang/internal/factstest/factstest_test.go +++ b/languages/golang/internal/factstest/factstest_test.go @@ -5,8 +5,8 @@ import ( "strings" "testing" + "github.com/cipherstash/stack/languages/golang/encrypt/plan" "github.com/cipherstash/stack/languages/golang/internal/factstest" - "github.com/cipherstash/stack/languages/golang/stackencrypt/plan" ) // Recursive embeddings, legal in Go, which a scan of embedded structs must diff --git a/languages/golang/internal/guest/clientkey.go b/languages/golang/internal/guest/clientkey.go index 6773dea68..12beb0de7 100644 --- a/languages/golang/internal/guest/clientkey.go +++ b/languages/golang/internal/guest/clientkey.go @@ -9,10 +9,10 @@ import "fmt" // pointer, hands its bytes only to this package tree, and is wiped once // consumed (see ADR-0005, decision 5). // -// stackauth reads one out of the developer profile; stackencrypt takes it +// auth reads one out of the developer profile; encrypt takes it // in its credentials and wipes it once the key is in guest memory. Both // expose this type as an alias, so a key read by one is the type the other -// takes, without stackauth importing stackencrypt. +// takes, without auth importing encrypt. // // The public packages alias the type, and an alias carries every exported // method with it — Go's internal rule stops the import, not the call. So diff --git a/languages/golang/internal/guest/doc.go b/languages/golang/internal/guest/doc.go index 298e963b8..6e3af747f 100644 --- a/languages/golang/internal/guest/doc.go +++ b/languages/golang/internal/guest/doc.go @@ -4,7 +4,7 @@ // decodes to; and the opaque client key one package reads and the other // consumes. // -// It sits under internal so that stackencrypt and stackauth expose what +// It sits under internal so that encrypt and auth expose what // they need of it — the error sentinels, the ClientKey type — as their own // identifiers (aliases, not copies: an error from either package is the // same value, and a key read by one is the type the other takes) without diff --git a/languages/golang/internal/guest/memory.go b/languages/golang/internal/guest/memory.go index aabbbbeac..6f869727b 100644 --- a/languages/golang/internal/guest/memory.go +++ b/languages/golang/internal/guest/memory.go @@ -39,9 +39,9 @@ import ( // commonly refused, with nothing else lost: the pages can be swapped, and // on a host with no swap not even that. The refusal is recorded and // reported through LockError, which each public package surfaces on its -// client (stackencrypt: Client.MemoryLocked and Client.MemoryLockError) so +// client (encrypt: Client.MemoryLocked and Client.MemoryLockError) so // an operator can see it and raise the limit; Strict turns it into a -// constructor failure (stackencrypt: WithRequireLockedMemory). +// constructor failure (encrypt: WithRequireLockedMemory). // LockPolicy is what a refused lock means for an instance. type LockPolicy uint8 diff --git a/languages/golang/internal/guest/memory_test.go b/languages/golang/internal/guest/memory_test.go index 738230888..e7e556c55 100644 --- a/languages/golang/internal/guest/memory_test.go +++ b/languages/golang/internal/guest/memory_test.go @@ -17,7 +17,7 @@ import ( // The allocator on its own, under the grow probe. What it does for a real // guest, and what a client reports about it, is tested where the guest is -// embedded (stackencrypt's memory tests). +// embedded (encrypt's memory tests). // The whole point of owning the allocation: growth commits more of one // reservation, so the buffer's address is the same before and after, and diff --git a/languages/golang/internal/guesttest/probe.go b/languages/golang/internal/guesttest/probe.go index 23060f077..ad91b1942 100644 --- a/languages/golang/internal/guesttest/probe.go +++ b/languages/golang/internal/guesttest/probe.go @@ -95,7 +95,7 @@ func HostReserves(t *testing.T) bool { // lock cannot quietly go untested. Without it a refused lock is reported // and the assertion skipped: a developer laptop's default limit is not a // bug in these packages. -const RequireLock = "STACKENCRYPT_TESTS_REQUIRE_LOCK" +const RequireLock = "STACK_ENCRYPT_TESTS_REQUIRE_LOCK" // LockOrSkip continues if alloc's memory is locked, skips if the host // refused the lock (or has no reservation to lock), and fails the skip diff --git a/mise.toml b/mise.toml index 97558c9cb..c20ee3766 100644 --- a/mise.toml +++ b/mise.toml @@ -143,7 +143,7 @@ for crate in stack-auth stack-kms stack-encrypt; do done """ -# The stack-encrypt WASI guest (languages/golang/stackencrypt/guest) is a +# The stack-encrypt WASI guest (languages/golang/encrypt/guest) is a # detached workspace — like the fuzz crates — so the workspace-wide tasks # never touch it; these two are its build and test entry points. [tasks."wasm:guest:build"] @@ -153,10 +153,10 @@ run = """ set -euo pipefail rustup target add wasm32-wasip1 root=$(pwd) -cd languages/golang/stackencrypt/guest +cd languages/golang/encrypt/guest cargo build --target wasm32-wasip1 --release module=target/wasm32-wasip1/release/stack_encrypt_guest.wasm -echo "guest module: languages/golang/stackencrypt/guest/$module" +echo "guest module: languages/golang/encrypt/guest/$module" # The security contract is about the *linked* module, which a successful # build says nothing about: the guest may reach the outside world only # through the two host functions the Go embedder provides. Fail-closed — @@ -176,11 +176,11 @@ python3 "$root/scripts/check-wasm-imports.py" "$module" \\ --require wasi_snapshot_preview1:random_get \\ --require cipherstash_transport:transport_send \\ --require cipherstash_transport:token_get -# The Go module embeds the checked artefact (languages/golang/stackencrypt/wasm, +# The Go module embeds the checked artefact (languages/golang/encrypt/wasm, # gitignored): copying it here is what makes `go test` in the binding run # against the guest just built rather than a stale one. cp "$module" ../wasm/stack_encrypt_guest.wasm -echo "embedded into languages/golang/stackencrypt/wasm/stack_encrypt_guest.wasm" +echo "embedded into languages/golang/encrypt/wasm/stack_encrypt_guest.wasm" """ [tasks."wasm:guest:test"] @@ -195,7 +195,7 @@ rustup target add wasm32-wasip1 # a path dependency is not linted from the guest's own workspace below. cargo clippy -p stack-guest-abi --all-targets --target wasm32-wasip1 -- -D warnings RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p stack-guest-abi --target wasm32-wasip1 -cd languages/golang/stackencrypt/guest +cd languages/golang/encrypt/guest cargo fmt --check cargo clippy --all-targets -- -D warnings # The wasm32-only modules (abi, host) only compile for the target; lint @@ -217,10 +217,10 @@ run = """ set -euo pipefail rustup target add wasm32-wasip1 root=$(pwd) -cd languages/golang/stackauth/guest +cd languages/golang/auth/guest cargo build --target wasm32-wasip1 --release module=target/wasm32-wasip1/release/stack_auth_guest.wasm -echo "guest module: languages/golang/stackauth/guest/$module" +echo "guest module: languages/golang/auth/guest/$module" # The credential guest's contract is the mirror image of the crypto # guest's: it may reach the filesystem — that is what it is for, and the # host grants it exactly one directory — and the auth host transport. @@ -236,7 +236,7 @@ python3 "$root/scripts/check-wasm-imports.py" "$module" \\ --require cipherstash_transport:transport_send \\ --require cipherstash_transport:oidc_token_get cp "$module" ../wasm/stack_auth_guest.wasm -echo "embedded into languages/golang/stackauth/wasm/stack_auth_guest.wasm" +echo "embedded into languages/golang/auth/wasm/stack_auth_guest.wasm" """ [tasks."wasm:auth-guest:test"] @@ -245,7 +245,7 @@ shell = "bash -c" run = """ set -euo pipefail rustup target add wasm32-wasip1 -cd languages/golang/stackauth/guest +cd languages/golang/auth/guest cargo fmt --check cargo clippy --all-targets -- -D warnings cargo clippy --target wasm32-wasip1 -- -D warnings @@ -254,9 +254,7 @@ mise x --env test -- cargo nextest run """ [tasks."go:test"] -# The old name, from when the module held one package. -alias = "go:stackencrypt:test" -description = "Format check, vet and test the Go module (languages/golang: stackencrypt, stackauth and the internal packages) against the embedded guests; needs `wasm:guest:build` and `wasm:auth-guest:build` first" +description = "Format check, vet and test the Go module (languages/golang: encrypt, auth and the internal packages) against the embedded guests; needs `wasm:guest:build` and `wasm:auth-guest:build` first" # The body lives in scripts/go-binding-test.sh so the macOS and Windows CI # jobs (which have Go but not mise) run exactly the same checks. One module # at languages/golang holds every Go package (ADR-0005 §4); the guest it @@ -268,27 +266,27 @@ description = "Lint and format-check the Go module (languages/golang) with golan dir = "languages/golang" run = "golangci-lint run ./..." -[tasks."go:stackencrypt:example"] -description = "Run the stack-encrypt Go example (languages/golang/stackencrypt/example) against real ZeroKMS; needs `stash auth login` first" +[tasks."go:encrypt:example"] +description = "Run the stack-encrypt Go example (languages/golang/encrypt/example) against real ZeroKMS; needs `stash auth login` first" shell = "bash -c" -# Both guests: the example reads the profile through stackauth. +# Both guests: the example reads the profile through auth. depends = ["wasm:guest:build", "wasm:auth-guest:build"] run = """ set -euo pipefail cd languages/golang -# Credentials come from stackencrypt.AutoCredentials: the CS_* variables -# if set, else the developer profile. See stackencrypt/example/README.md. -CGO_ENABLED=0 go run ./stackencrypt/example +# Credentials come from encrypt.AutoCredentials: the CS_* variables +# if set, else the developer profile. See encrypt/example/README.md. +CGO_ENABLED=0 go run ./encrypt/example """ -[tasks."go:stackencrypt:example:explicit"] -description = "Run the explicit-credentials Go example (languages/golang/stackencrypt/example/explicit) against real ZeroKMS; pass -secrets-dir, -client-id and -workspace-crn after --" +[tasks."go:encrypt:example:explicit"] +description = "Run the explicit-credentials Go example (languages/golang/encrypt/example/explicit) against real ZeroKMS; pass -secrets-dir, -client-id and -workspace-crn after --" shell = "bash -c" depends = ["wasm:guest:build", "wasm:auth-guest:build"] run = """ set -euo pipefail cd languages/golang # Credentials come only from the flags and the secrets directory: no CS_* -# variables, no profile. See stackencrypt/example/explicit/README.md. -CGO_ENABLED=0 go run ./stackencrypt/example/explicit "$@" +# variables, no profile. See encrypt/example/explicit/README.md. +CGO_ENABLED=0 go run ./encrypt/example/explicit "$@" """ diff --git a/packages/stack-encrypt/CONTEXT.md b/packages/stack-encrypt/CONTEXT.md index cf52e9497..06e677f72 100644 --- a/packages/stack-encrypt/CONTEXT.md +++ b/packages/stack-encrypt/CONTEXT.md @@ -3,7 +3,7 @@ Client-side encryption of values under per-value ZeroKMS data keys, and the derivation of searchable index terms from the same values. Covers `stack-encrypt`, `stack-encrypt-derive`, and the WASI guest in -`languages/golang/stackencrypt/guest` that exposes them to Go. +`languages/golang/encrypt/guest` that exposes them to Go. ## Language diff --git a/packages/stack-encrypt/tests/fixtures/label_segments.json b/packages/stack-encrypt/tests/fixtures/label_segments.json index 988b259a8..5eef3aceb 100644 --- a/packages/stack-encrypt/tests/fixtures/label_segments.json +++ b/packages/stack-encrypt/tests/fixtures/label_segments.json @@ -1,5 +1,5 @@ { - "_comment": "One rule, two suites. A Label segment is plain exactly when the descriptor renders it verbatim. The Rust unit test in src/descriptor.rs and the Go test in languages/golang/stackencrypt/label_test.go both read this file, so the two implementations of the rule cannot drift apart silently.", + "_comment": "One rule, two suites. A Label segment is plain exactly when the descriptor renders it verbatim. The Rust unit test in src/descriptor.rs and the Go test in languages/golang/encrypt/label_test.go both read this file, so the two implementations of the rule cannot drift apart silently.", "plain": [ "users", "email_address", diff --git a/packages/stack-guest-abi/src/lib.rs b/packages/stack-guest-abi/src/lib.rs index fb15536e3..7374c78f7 100644 --- a/packages/stack-guest-abi/src/lib.rs +++ b/packages/stack-guest-abi/src/lib.rs @@ -31,7 +31,7 @@ //! # The guest ABI the Go binding's WASI guests share //! //! The Go binding reaches Rust through WASI modules run by wazero: the -//! crypto guest (`bindings/go/stackencrypt/guest`, `stack-encrypt` over a +//! crypto guest (`bindings/go/encrypt/guest`, `stack-encrypt` over a //! host-provided transport) and, per ADR-0005, the credential guest //! (`stack-profile` and `stack-auth`). Everything a guest needs that is //! *not* about what it does — how the host gets bytes in and out, how a diff --git a/scripts/__tests__/cargo-lock-freshness.test.mjs b/scripts/__tests__/cargo-lock-freshness.test.mjs index ea2424951..288be3628 100644 --- a/scripts/__tests__/cargo-lock-freshness.test.mjs +++ b/scripts/__tests__/cargo-lock-freshness.test.mjs @@ -191,8 +191,8 @@ describe('Cargo.lock records this tree’s crates at their real versions', () => 'packages/stack-auth/fuzz/Cargo.lock', 'packages/stack-kms/fuzz/Cargo.lock', 'packages/stack-encrypt/fuzz/Cargo.lock', - 'languages/golang/stackencrypt/guest/Cargo.lock', - 'languages/golang/stackauth/guest/Cargo.lock', + 'languages/golang/encrypt/guest/Cargo.lock', + 'languages/golang/auth/guest/Cargo.lock', ]), ) }) diff --git a/scripts/__tests__/cargo-publish-opt-out.test.mjs b/scripts/__tests__/cargo-publish-opt-out.test.mjs index 580ed3561..6e1e46545 100644 --- a/scripts/__tests__/cargo-publish-opt-out.test.mjs +++ b/scripts/__tests__/cargo-publish-opt-out.test.mjs @@ -74,8 +74,8 @@ const WORKSPACES = [ 'packages/stack-auth/fuzz', 'packages/stack-kms/fuzz', 'packages/stack-encrypt/fuzz', - 'languages/golang/stackencrypt/guest', - 'languages/golang/stackauth/guest', + 'languages/golang/encrypt/guest', + 'languages/golang/auth/guest', ].map((root) => ({ root, publishable: new Set(), expects: '.' })), ] diff --git a/scripts/__tests__/crates-ci.test.mjs b/scripts/__tests__/crates-ci.test.mjs index 822f78f7d..5cb9371f6 100644 --- a/scripts/__tests__/crates-ci.test.mjs +++ b/scripts/__tests__/crates-ci.test.mjs @@ -222,11 +222,11 @@ const CI_EXEMPT_TASKS = new Map([ 'The fan-out over the two sweeps above, for local use. Both would write mutants.out in the same directory, so CI runs cargo-mutants once over both crates instead.', ], [ - 'go:stackencrypt:example', + 'go:encrypt:example', 'A walkthrough against real ZeroKMS for a developer who has run `stash auth login`. CI exercises the same client through the Go live tests in tests-golang.yml `live`.', ], [ - 'go:stackencrypt:example:explicit', + 'go:encrypt:example:explicit', 'The same walkthrough, with credentials passed as flags after `--`. Nothing for CI to pass; the live tests cover explicit credentials (`liveClient`).', ], ]) diff --git a/scripts/go-binding-test.sh b/scripts/go-binding-test.sh index f77da30d0..8ba565ffe 100755 --- a/scripts/go-binding-test.sh +++ b/scripts/go-binding-test.sh @@ -15,7 +15,7 @@ set -euo pipefail dir=${1:?usage: go-binding-test.sh [...]} shift || true if [ $# -eq 0 ]; then - set -- stackencrypt/wasm/stack_encrypt_guest.wasm stackauth/wasm/stack_auth_guest.wasm + set -- encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm fi cd "$dir" @@ -38,7 +38,7 @@ go vet ./... # Keep the refresh-lock results explicit in CI logs on Linux, macOS, and # Windows. These tests exercise the platform lock implementation with two # independent guest instances sharing one auth.json. -CGO_ENABLED=0 go test -v ./stackauth -run '^TestDeviceRefresh' +CGO_ENABLED=0 go test -v ./auth -run '^TestDeviceRefresh' CGO_ENABLED=0 go test ./... # The transport codec's u32-bound guards are load-bearing where int is 32 From 739ecf1b1e84c6e55fbfc2d2deae8152ee0e72a5 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:00:13 -0700 Subject: [PATCH 05/30] feat(golang)!: the generated API replaces the value and record calls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Go SDK is now what the plan builder design describes: a struct's stash tags are the declaration, stashgen writes the encrypted type and its functions, and a program calls users.Encrypt, users.Decrypt and users.Fields. Everything the old package offered beside that is removed, not deprecated — it was never released: Cipher.Encrypt/Decrypt and the Element forms, Client.Decrypt, EncryptRecord(s)/DecryptRecord(s), EncryptedRecord, EncryptedField, Cipher.Term, RecordOption, WithPlan, ExtendContext, WithGuest, TermKind, Plan, FieldPlan, NewPlan, PlanFromTags and every run-time tag reader, the plan package and plantest, factstest, Context/Label and their constructors, and the four Sealed* storage types. What replaces them, and why it has the shape it has: - encrypt.Cipher.Extend appends the caller's parts to every field's context in every call through the cipher, so the write, the query and the read cannot use different ones. encrypt.Ciphertext is the frozen stack-encrypt leaf, the one storage type: every sealed field is one leaf. Index is Equality, Ore, Ope, Match() and JSON(); the term types stay. Decrypter is what a generated Decrypt takes: *Cipher refuses a foreign keyset's row before any key is retrieved, *Client opens each row under the keyset that sealed it. - The record path (Cipher.Seal, Cipher.Open, Client.Open, Cipher.Derive) takes the internal record types, so only generated code reaches it. internal/record is the data form of a declaration as dynamic::record reads it: a field's context is its label (context segments + identity) nested under each extension part, "type" is the wire kind, outputs are c/eq/match/ore/ope. Both gensupport (running) and stashgen (checking) lower to it, so the generator checks the plan the program will run. - encrypt/gensupport is the real library behind generated code: Declare and the verb methods carry the field's wire kind, which stashgen picks from the Go type (int8..int32 travel as int32, int/int64 as int64, the unsigned likewise, []byte as bytes); Codec.Encrypt/Decrypt send a slice as one guest call and one ZeroKMS request; Field[T] seals one value and derives the field's terms; Get converts within a kind's family and refuses the rest, so a value that opens to another type is an error. - Passthrough fields stay on the host. The FFI codec cannot carry every Go type a program stores beside a ciphertext (time.Time, gorm.DeletedAt, any driver.Valuer), and nothing the engine does to a passthrough value is observable, so the plan the engine sees has the sealed fields only and the generated Seal reads passthrough values back from the Record. The plan says passthrough crosses; this is the deviation, recorded here and in the record package's doc. - An opaque struct crosses as one JSON document, declared bytes, because the engine seals a composite FfiValue as a tree of leaves and an opaque struct is one column. Its fields are what JSON carries; the generator refuses a nested struct inside one for now. - stashgen.GuestEngine is the embedded guest: encrypt.NewChecker instantiates it with no credentials and asks se_plan_check one field at a time, so a refusal names the field, then for the whole plan. The engine produces no EQL type in this build (se_targets is empty), so encrypt_into is refused with "EQL types are not available yet". The guest loses se_encrypt, se_decrypt and the element exports (ADR-0007 as amended: every value crosses under a declaration) and gains se_plan_check and se_targets. A `deterministic-kms` feature builds a TEST guest whose keys derive from a seed — the DeterministicSource of stack-encrypt's tests/common, copied — so the Go tests open the records Rust sealed in tests/fixtures/record_lowering.json through the generated testusers package and derive the same term bytes, and run round trips, foreign-keyset refusal, tampering and the ORE/OPE ordering properties with no ZeroKMS. The hermetic tests replace the old live ordering tests; the live suite keeps the real round trips through generated code. The test build's import gate requires no transport import: with no ZeroKMS client in it, the linker drops the host module. Generated *_stash.go files are committed (internal/testusers, example), and tests-golang.yml runs `go generate ./...` and fails on a diff. The stashgen goldens and the stub SDK follow the new signatures; the stub agreement test now covers encrypt too and compares signatures without parameter names. The example is rewritten to the eight steps; the explicit-credentials example, which only showed the removed API, is gone with its task. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- .github/workflows/tests-golang.yml | 33 +- AGENTS.md | 2 +- languages/golang/auth/README.md | 6 +- languages/golang/cmd/stashgen/README.md | 15 +- languages/golang/cmd/stashgen/main_test.go | 24 +- languages/golang/encrypt/README.md | 534 +++------- languages/golang/encrypt/checker.go | 124 +++ languages/golang/encrypt/cipher.go | 182 +--- languages/golang/encrypt/ciphertext.go | 83 ++ languages/golang/encrypt/client.go | 58 +- languages/golang/encrypt/context.go | 187 ---- languages/golang/encrypt/credentials_test.go | 4 +- languages/golang/encrypt/doc.go | 263 ++--- languages/golang/encrypt/example/README.md | 72 +- .../golang/encrypt/example/explicit/README.md | 53 - .../golang/encrypt/example/explicit/main.go | 170 ---- languages/golang/encrypt/example/main.go | 204 +--- languages/golang/encrypt/example/model.go | 22 + .../golang/encrypt/example/user_stash.go | 169 ++++ languages/golang/encrypt/export_test.go | 92 +- languages/golang/encrypt/fixture_test.go | 153 +++ languages/golang/encrypt/gensupport/codec.go | 344 +++++++ .../golang/encrypt/gensupport/convert.go | 272 ++++++ .../golang/encrypt/gensupport/declaration.go | 221 +++++ languages/golang/encrypt/gensupport/field.go | 104 ++ .../golang/encrypt/gensupport/gensupport.go | 12 +- .../gensupport/gensupport_internal_test.go | 261 +++++ languages/golang/encrypt/guest.go | 27 +- languages/golang/encrypt/guest/Cargo.lock | 1 + languages/golang/encrypt/guest/Cargo.toml | 12 + languages/golang/encrypt/guest/src/abi.rs | 274 ++---- .../golang/encrypt/guest/src/deterministic.rs | 116 +++ languages/golang/encrypt/guest/src/lib.rs | 8 + languages/golang/encrypt/guest/src/ops.rs | 132 +-- languages/golang/encrypt/guest/src/options.rs | 4 +- .../golang/encrypt/guest/tests/native_ops.rs | 329 +------ languages/golang/encrypt/guest_test.go | 306 ++---- .../internal/testusers/document_stash.go | 88 ++ .../encrypt/internal/testusers/probe_stash.go | 280 ++++++ .../encrypt/internal/testusers/user_stash.go | 195 ++++ .../encrypt/internal/testusers/users.go | 45 + languages/golang/encrypt/label.go | 142 --- languages/golang/encrypt/label_test.go | 252 ----- languages/golang/encrypt/leaf.go | 122 --- .../golang/encrypt/live_internal_test.go | 175 ++++ languages/golang/encrypt/live_test.go | 386 +------- languages/golang/encrypt/memory_test.go | 4 +- languages/golang/encrypt/options.go | 7 +- languages/golang/encrypt/options_test.go | 2 +- languages/golang/encrypt/order_live_test.go | 169 ---- languages/golang/encrypt/order_test.go | 194 ++++ languages/golang/encrypt/plan/doc.go | 91 -- languages/golang/encrypt/plan/fact.go | 114 --- languages/golang/encrypt/plan/message.go | 240 ----- languages/golang/encrypt/plan/plan_test.go | 541 ---------- .../golang/encrypt/plan/plantest/compare.go | 450 --------- .../encrypt/plan/plantest/golden_test.go | 47 - .../golang/encrypt/plan/plantest/plantest.go | 222 ----- .../plan/plantest/plantest_internal_test.go | 813 --------------- .../golang/encrypt/plan/plantest/snapshot.go | 454 --------- .../testdata/TestPolicies/audits.golden | 8 - .../testdata/TestPolicies/individuals.golden | 32 - languages/golang/encrypt/plan/policy.go | 374 ------- languages/golang/encrypt/policy_plan_test.go | 98 -- languages/golang/encrypt/record.go | 924 ------------------ languages/golang/encrypt/records.go | 316 ++++++ languages/golang/encrypt/roundtrip_test.go | 258 +++++ languages/golang/encrypt/term.go | 115 ++- languages/golang/encrypt/unit_test.go | 607 +----------- languages/golang/encrypt/wasm/README.md | 9 +- .../golang/internal/factstest/factstest.go | 166 ---- .../internal/factstest/factstest_test.go | 118 --- .../golang/internal/record/fixture_test.go | 39 + languages/golang/internal/record/record.go | 316 ++++++ .../golang/internal/record/record_test.go | 115 +++ languages/golang/stashgen/declaration.go | 3 + languages/golang/stashgen/emit.go | 39 +- languages/golang/stashgen/engine.go | 216 +++- languages/golang/stashgen/policy_test.go | 6 +- languages/golang/stashgen/read.go | 22 +- languages/golang/stashgen/stub_test.go | 85 +- .../cases/accounts/account_stash.go.golden | 2 +- .../contacts/contactstash_stash.go.golden | 4 +- .../cases/embedded/patient_stash.go.golden | 6 +- .../foreign/individualstash_stash.go.golden | 6 +- .../cases/orders/order_stash.go.golden | 6 +- .../cases/orders/refund_stash.go.golden | 2 +- .../testdata/cases/users/user_stash.go.golden | 4 +- .../policy_individual_stash.go.golden | 6 +- .../testdata/stubsdk/encrypt/encrypt.go | 20 +- .../stubsdk/encrypt/gensupport/gensupport.go | 32 +- mise.toml | 46 +- scripts/__tests__/crates-ci.test.mjs | 4 - 93 files changed, 5321 insertions(+), 8589 deletions(-) create mode 100644 languages/golang/encrypt/checker.go create mode 100644 languages/golang/encrypt/ciphertext.go delete mode 100644 languages/golang/encrypt/context.go delete mode 100644 languages/golang/encrypt/example/explicit/README.md delete mode 100644 languages/golang/encrypt/example/explicit/main.go create mode 100644 languages/golang/encrypt/example/model.go create mode 100644 languages/golang/encrypt/example/user_stash.go create mode 100644 languages/golang/encrypt/fixture_test.go create mode 100644 languages/golang/encrypt/gensupport/codec.go create mode 100644 languages/golang/encrypt/gensupport/convert.go create mode 100644 languages/golang/encrypt/gensupport/declaration.go create mode 100644 languages/golang/encrypt/gensupport/field.go create mode 100644 languages/golang/encrypt/gensupport/gensupport_internal_test.go create mode 100644 languages/golang/encrypt/guest/src/deterministic.rs create mode 100644 languages/golang/encrypt/internal/testusers/document_stash.go create mode 100644 languages/golang/encrypt/internal/testusers/probe_stash.go create mode 100644 languages/golang/encrypt/internal/testusers/user_stash.go create mode 100644 languages/golang/encrypt/internal/testusers/users.go delete mode 100644 languages/golang/encrypt/label.go delete mode 100644 languages/golang/encrypt/label_test.go delete mode 100644 languages/golang/encrypt/leaf.go create mode 100644 languages/golang/encrypt/live_internal_test.go delete mode 100644 languages/golang/encrypt/order_live_test.go create mode 100644 languages/golang/encrypt/order_test.go delete mode 100644 languages/golang/encrypt/plan/doc.go delete mode 100644 languages/golang/encrypt/plan/fact.go delete mode 100644 languages/golang/encrypt/plan/message.go delete mode 100644 languages/golang/encrypt/plan/plan_test.go delete mode 100644 languages/golang/encrypt/plan/plantest/compare.go delete mode 100644 languages/golang/encrypt/plan/plantest/golden_test.go delete mode 100644 languages/golang/encrypt/plan/plantest/plantest.go delete mode 100644 languages/golang/encrypt/plan/plantest/plantest_internal_test.go delete mode 100644 languages/golang/encrypt/plan/plantest/snapshot.go delete mode 100644 languages/golang/encrypt/plan/plantest/testdata/TestPolicies/audits.golden delete mode 100644 languages/golang/encrypt/plan/plantest/testdata/TestPolicies/individuals.golden delete mode 100644 languages/golang/encrypt/plan/policy.go delete mode 100644 languages/golang/encrypt/policy_plan_test.go delete mode 100644 languages/golang/encrypt/record.go create mode 100644 languages/golang/encrypt/records.go create mode 100644 languages/golang/encrypt/roundtrip_test.go delete mode 100644 languages/golang/internal/factstest/factstest.go delete mode 100644 languages/golang/internal/factstest/factstest_test.go create mode 100644 languages/golang/internal/record/fixture_test.go create mode 100644 languages/golang/internal/record/record.go create mode 100644 languages/golang/internal/record/record_test.go diff --git a/.github/workflows/tests-golang.yml b/.github/workflows/tests-golang.yml index ca156c389..73b8d8de8 100644 --- a/.github/workflows/tests-golang.yml +++ b/.github/workflows/tests-golang.yml @@ -8,8 +8,10 @@ name: Tests (Go) # wasi-check the stack crates build for wasm32-wasip1 with no JS-host or # native-HTTP dependencies, the no-http shape passes its tests # and docs, both guests pass lint and tests and are built with -# their import surfaces checked, their sha256 is recorded, and -# `go:test` runs against them. +# their import surfaces checked (the stack-encrypt guest twice: +# the real build and the deterministic-kms test build), their +# sha256 is recorded, `go:test` runs against them, and +# `go generate` leaves the tree unchanged. # go-lint golangci-lint, Linux only. # go-binding-cross # the same Go tests on macOS and Windows, against the guests @@ -146,6 +148,14 @@ jobs: - name: stack-encrypt guest release build and import-surface gate run: mise run wasm:guest:build + # The deterministic-kms TEST build beside it: the Go tests open the + # record fixture Rust sealed through it, and run round trips with no + # ZeroKMS. The Go package never embeds it; its tests skip when it is + # absent, so a build that forgets this step would pass with less + # coverage, which is why the job builds it unconditionally. + - name: stack-encrypt guest deterministic test build + run: mise run wasm:guest:build:deterministic + - name: Credential guest lint and tests run: mise run wasm:auth-guest:test @@ -157,7 +167,7 @@ jobs: # same bytes rather than a stale or rebuilt guest. - name: Record the guests' checksums run: | - for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do (cd languages/golang && openssl dgst -sha256 "$guest" | awk '{print $NF}' > "$guest.sha256" && echo "$guest sha256 $(cat "$guest.sha256")") done @@ -168,6 +178,8 @@ jobs: path: | languages/golang/encrypt/wasm/stack_encrypt_guest.wasm languages/golang/encrypt/wasm/stack_encrypt_guest.wasm.sha256 + languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm + languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm.sha256 languages/golang/auth/wasm/stack_auth_guest.wasm languages/golang/auth/wasm/stack_auth_guest.wasm.sha256 if-no-files-found: error @@ -185,6 +197,17 @@ jobs: echo "RLIMIT_MEMLOCK: $(ulimit -l) KiB" mise run go:test + # What is encrypted is fixed before the program ships: every generated + # file is committed, and this fails when `go generate` would change + # one. It runs the real stashgen against the guest just built, so it + # also proves the generator and the engine agree on the module's own + # examples. + - name: Generated code is committed + working-directory: languages/golang + run: | + CGO_ENABLED=0 go generate ./... + git diff --exit-code -- . + # Linux only: macOS and Windows would report the same findings. Needs no # guests: the packages embed a directory and compile without them. go-lint: @@ -260,7 +283,7 @@ jobs: - name: The guests are the ones Linux built and checked run: | cd languages/golang - for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do want=$(cat "$guest.sha256") got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}') if [ "$want" != "$got" ]; then @@ -326,7 +349,7 @@ jobs: - name: The guests are the ones the wasi-check job built and checked run: | cd languages/golang - for guest in encrypt/wasm/stack_encrypt_guest.wasm auth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do want=$(cat "$guest.sha256") got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}') if [ "$want" != "$got" ]; then diff --git a/AGENTS.md b/AGENTS.md index 82be8dc52..da9c497b2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,7 +92,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l links are provenance only. - `packages/stack-auth`, `packages/stack-profile`, `packages/stack-kms`, `packages/stack-encrypt`, `packages/stack-encrypt-derive`, `packages/stack-guest-abi`: The Rust crates imported from `cipherstash/cipherstash-suite` with their history — `stack-auth` and `stack-profile` (published to crates.io, one version group), `stack-kms` (published to crates.io from 0.1.0, its own version group, re-exported by `stack-encrypt` as `stack_encrypt::kms`), `stack-encrypt` and `stack-encrypt-derive` (published to crates.io from 0.1.0, one version group; `eql-bindings`' `stack-encrypt` feature depends on them from the registry), and `stack-guest-abi` (`publish = false`). They are the members of the **root Cargo workspace**, with the three node binding crates below. See "Working on the Rust crates". - `languages/typescript/packages/auth`, `languages/typescript/packages/profile`, `languages/typescript/packages/stack-auth-wasm`: The node bindings of those crates. `@cipherstash/auth` (napi-rs v2) and its six `platforms/*` packages are published to npm from this repository by `release.yml` (`auth-artifacts`, `publish-auth`); a change to what it ships, the `stack-auth` crate included, needs an `@cipherstash/auth` changeset (`require-auth-npm-changeset.yml`). `@cipherstash/profile` and its platforms are private and never published; `@cipherstash/stack-auth-wasm` is private and builds the wasm that `@cipherstash/auth` ships. Their `build` and `test` scripts never invoke cargo; `build:native`, `build:debug` and `test:cargo` do. -- `languages/golang`: The Go module (`encrypt`, `auth`, `internal`), a wazero host with no cgo. Its two WASI guests (`*/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; the `.wasm` files they embed are gitignored. There is no Go release process yet. +- `languages/golang`: The Go module — the SDK `encrypt` with its generated-code support `encrypt/gensupport` and the policy packages `encrypt/policy` and `encrypt/policy/protosource`; the credential package `auth`; the generator `stashgen` and its command `cmd/stashgen`; and `internal` (the shared guest plumbing and the `record` wire model). A wazero host with no cgo. Its two WASI guests (`encrypt/guest`, `auth/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; `mise run wasm:guest:build:deterministic` builds the seeded TEST build the hermetic Go tests use; the `.wasm` files they embed are gitignored. Generated `*_stash.go` files are committed and CI fails when `go generate ./...` changes one. There is no Go release process yet. - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `languages/typescript/examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). diff --git a/languages/golang/auth/README.md b/languages/golang/auth/README.md index 277251666..2b22e327d 100644 --- a/languages/golang/auth/README.md +++ b/languages/golang/auth/README.md @@ -3,7 +3,7 @@ The Go binding of the developer profile — the directory `stash auth login` writes — read through the `stack-profile` Rust crate running inside a WASI guest under [wazero], with `CGO_ENABLED=0`. It is the credential half of -the Go SDK: it hands a [`encrypt`](../encrypt) client its client +the Go SDK: it hands an [`encrypt`](../encrypt) client its client key and its bearer token without either package re-deriving the profile's layout, and without either importing the other. @@ -16,7 +16,7 @@ cross-process refresh lock for device sessions. ## Use -Most applications never call this package directly: a `encrypt` +Most applications never call this package directly: an `encrypt` client built with `NewClient(ctx)` and no options resolves its credentials with `encrypt.AutoCredentials`, which reads the environment first and then the profile, through this package. Use it directly to take the profile @@ -69,7 +69,7 @@ the strategy are the caller's: the client asks the strategy for a token on every request but never closes it, so both stay open until the client is closed (the deferred calls above run in that order). -A encrypt client takes its token only from a strategy, never a raw +An encrypt client takes its token only from a strategy, never a raw string: a raw token cannot be refreshed when it expires, and would bypass the cross-process lock a device-session refresh holds with the `stash` CLI (the IdP revokes a whole refresh-token chain when one is used twice). diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 62a7d2380..a5397f03f 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -116,9 +116,9 @@ It runs none of the package's code. It ignores its own output file when it loads the package, so a stale file does not stop it. The same input always gives the same file: fields keep their declared order, and the file carries no version and no time. -`stashgen` checks each declaration with the engine, and holds no copy of the engine's rules. +`stashgen` checks each declaration with the engine, and holds no copy of the engine's rules: it runs the WASI guest the SDK embeds and asks it, one field at a time, so the error names the field. It asks the engine for the EQL types it holds: each name, its plaintext type, its indexes and its query form. -The engine produces one EQL type today, `TextEq`. +This build of the engine produces no EQL type, so `encrypt_into` is refused with "EQL types are not available yet"; the next release adds `TextEq` and `encrypt/eql`. Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`. ## When stashgen stops @@ -176,8 +176,13 @@ The struct you wrote is not protected: `stashgen` warns when it has sealed field `-redact` makes `stashgen` write those two methods on the struct. No warning, error or log line holds a plaintext value. +## What crosses the binding + +Generated code sends the engine every sealed field with its value, under the declaration lowered to data: each field's label (`/`), its outputs and its wire type (`int64`, `string`, `bytes`, ...), which `stashgen` chose from the field's Go type. +A passthrough field stays on the host: the engine does nothing to it a program could observe, and the FFI codec cannot carry every Go type a program stores beside a ciphertext. +An `opaque` struct crosses as one JSON document and is one column; its fields are what JSON carries: scalars, `[]byte`, slices and maps of them. + ## Status -The generator is built and tested against a static stand-in for the engine. -`stashgen.GuestEngine`, which runs the WASI guest the SDK embeds, is not wired yet, so the command stops with `ErrEngineUnavailable` until it is. -The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`; `stashgen.Generate` takes one with `WithEngine`. +The command runs the WASI guest the SDK embeds, so it needs the guest built: `mise run wasm:guest:build`. +The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`; `stashgen.Generate` takes one with `WithEngine`, and `stashgen/enginetest` has a static one for tests. diff --git a/languages/golang/cmd/stashgen/main_test.go b/languages/golang/cmd/stashgen/main_test.go index 9398a2d5c..274ca0717 100644 --- a/languages/golang/cmd/stashgen/main_test.go +++ b/languages/golang/cmd/stashgen/main_test.go @@ -8,6 +8,7 @@ import ( "strings" "testing" + "github.com/cipherstash/stack/languages/golang/encrypt" "github.com/cipherstash/stack/languages/golang/stashgen" "github.com/cipherstash/stack/languages/golang/stashgen/enginetest" ) @@ -117,17 +118,32 @@ func TestRunFlags(t *testing.T) { } } -func TestRunStopsWithoutAnEngine(t *testing.T) { +// The real engine: the embedded guest, which produces no EQL type in this +// build, so a struct with encrypt_into is refused and nothing is written. +// Skips when the guest is not built. +func TestRunAsksTheEmbeddedEngine(t *testing.T) { + checker, err := encrypt.NewChecker(context.Background()) + if err != nil { + t.Skip(err) + } + _ = checker.Close() dir := writeModule(t, map[string]string{"model.go": userSource}) var stdout, stderr bytes.Buffer if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 1 { - t.Fatalf("exit %d, want 1", code) + t.Fatalf("exit %d, want 1\n%s", code, stderr.String()) } - if !strings.Contains(stderr.String(), stashgen.ErrEngineUnavailable.Error()) { + if !strings.Contains(stderr.String(), "EQL types are not available yet") { t.Fatalf("stderr = %q", stderr.String()) } if _, err := os.Stat(filepath.Join(dir, "user_stash.go")); !os.IsNotExist(err) { - t.Fatal("a file was written with no engine") + t.Fatal("a file was written after the engine refused") + } + // Separate columns are what the engine runs today. + columns := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt,index=equality;match", 1) + dir = writeModule(t, map[string]string{"model.go": columns}) + stderr.Reset() + if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 0 { + t.Fatalf("exit %d\n%s", code, stderr.String()) } } diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index 849c635dc..ba790af47 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -1,441 +1,187 @@ -# stack-encrypt for Go +# Stack Encrypt for Go -Client-side encryption of values under per-value ZeroKMS data keys, and the -derivation of searchable index terms from the same values, for Go. The -`stack-encrypt` Rust crate is compiled to a WASI module and embedded in -this package; Go calls it through wazero, a pure-Go WebAssembly runtime, -so there is no cgo and no separate Go port of the cryptography. +Searchable, field-level encryption for Go structs under per-value ZeroKMS +data keys. You declare what to encrypt with struct tags; `stashgen` writes the +encrypted type and the functions that encrypt, decrypt and search it; the +`stack-encrypt` Rust engine runs unmodified inside a WASI module embedded in +this package, through [wazero], with no cgo. -The package reference is on [pkg.go.dev]; this README covers connecting, -what happens to key material, and the errors. +The package reference is on [pkg.go.dev]; the generator's reference is +[`cmd/stashgen/README.md`](../cmd/stashgen/README.md). +[wazero]: https://wazero.io [pkg.go.dev]: https://pkg.go.dev/github.com/cipherstash/stack/languages/golang/encrypt -## Install +## Use the SDK -```sh -go get github.com/cipherstash/stack/languages/golang/encrypt -``` +1. Add the generator to your module. This needs Go 1.24 or later. -Go 1.25 or later. + ```sh + go get -tool github.com/cipherstash/stack/languages/golang/cmd/stashgen + ``` -## Connect +2. Put a `stash` tag on every exported field of the struct, and a `go:generate` + comment beside it. -A `Client` is one ZeroKMS client: its client key, its default keyset, and -the keysets it has loaded since. Make one per process and share it; it is -safe for concurrent use. + ```go + //go:generate go tool stashgen -type User + type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt,index=equality;match"` + Age uint32 `stash:"age,encrypt,index=equality;ore"` + } + ``` -```go -import ( - "context" +3. Run the generator. It writes `user_stash.go` beside the struct. - "github.com/cipherstash/stack/languages/golang/encrypt" -) + ```sh + go generate ./... + ``` -func run(ctx context.Context) error { - client, err := encrypt.NewClient(ctx) - if err != nil { - return err // encrypt.ErrNoCredentials: nothing configured - } - defer client.Close() +4. Commit the generated file. - // ... - return nil -} -``` +5. Call the generated functions where you write and read. -`ctx` is Go's standard `context.Context`, and it means what it always -means: the deadline and cancellation for the work this call does. -`NewClient` makes one ZeroKMS round trip, to load the client's default -keyset, and `ctx` bounds that request. It has nothing to do with an -*encryption* context, which is the value a field is sealed under; that is -`encrypt.Context`. Every method that can reach ZeroKMS takes a -`context.Context` first, for the same reason. + ```go + client, err := encrypt.NewClient(ctx) + defer client.Close() + cipher := client.Keyset(encrypt.KeysetName("tenant-42")) -Everything else is a functional option, and each has a default: + encrypted, err := users.Encrypt(ctx, cipher, people) // one ZeroKMS request + people, err := users.Decrypt(ctx, cipher, encrypted) + term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com") + ``` -```go -client, err := encrypt.NewClient(ctx, - encrypt.WithCredentials(encrypt.OIDCFederation(crn, provider)), - encrypt.WithTransport(rt), - encrypt.WithKeysetCacheSize(4096), - encrypt.WithRequireLockedMemory(), -) -``` +6. Store the encrypted type. Each field is one or more byte columns, so + `database/sql`, pgx, sqlx and GORM take it as it is: + `e.Email.Ciphertext`, `e.Email.Equality`, `e.Email.Match`. -| Option | Default | -|---|---| -| `WithCredentials(c)` | `AutoCredentials()`: see below. | -| `WithTransport(rt)` | `http.DefaultTransport`. Used for ZeroKMS, and for token requests when the credentials make them. | -| `WithKeysetCacheSize(n)` | 1024 keysets beyond the default one. | -| `WithRequireLockedMemory()` | Off: memory that cannot be locked is reported, not refused. See below. | -| `WithGuest(wasm)` | The embedded guest module. | +7. Run the generator again after each change to the struct or to a tag. A + change to the fields of the struct stops the build until you do. -If an option is given twice, the later one wins. +8. In CI, run the generator and fail when a generated file changes. -### Credentials + ```sh + go generate ./... && git diff --exit-code + ``` + +The rest of this file is the reference. [`example/`](example/) is the eight +steps as a program. + +## Connect -With no `WithCredentials`, `NewClient` finds its credentials the way the -Rust client does, with `AutoCredentials`: the environment first, then the -developer profile that `stash auth login` writes. On a developer machine, logging in is enough. -In CI or a deployment, the environment supplies them. The first two rows -are what a deployment with no profile needs; the rest override what would -otherwise be resolved: - -| Variable | Role | -|---|---| -| `CS_CLIENT_ACCESS_KEY`, `CS_WORKSPACE_CRN` | An access key, exchanged for a token. Without it, the current workspace's stored session is used, and refreshed as it expires. | -| `CS_CLIENT_ID`, `CS_CLIENT_KEY` | The client key, used when both are set. Without them, the current workspace's `secretkey.json` is used. | -| `CS_ZEROKMS_HOST` (or `CS_VITUR_HOST`) | Pins the ZeroKMS endpoint. Otherwise it comes from the token. Read whatever the credentials. | -| `CS_CTS_HOST` | Overrides the authentication endpoint. | -| `CS_CONFIG_PATH` | The profile directory, instead of `~/.cipherstash`. | - -A variable that is set but empty or unusable is an error, not a reason to -look elsewhere. Nothing found is `ErrNoCredentials`, naming what to set; -when the profile would have been consulted, it also says why the profile -could not be opened, so an unreadable or mistyped `CS_CONFIG_PATH` is not -reported as "not logged in". - -The ZeroKMS endpoint comes from the token's services claim; there is no -option to pin it. `CS_ZEROKMS_HOST` (or the legacy `CS_VITUR_HOST`) -overrides it whatever the credentials, `NewCredentials` included, as the -Rust client reads them, so a value exported there for another tool is -worth checking. - -Resolution happens host-side, in Go. The profile and the token strategies -run in `auth`'s credential guest; the crypto guest that holds the -keys is still given no environment and no filesystem. The credential guest -lives as long as the client, and `Close` releases it. - -To supply the credentials yourself, pass `NewCredentials` with a client -id, a client key and a `auth` strategy for the token: +A `Client` is one ZeroKMS client: its client key, its default keyset, and the +keysets it has loaded since. Make one per process and share it; it is safe +for concurrent use. ```go -store, err := auth.OpenWithoutProfile(ctx) // or auth.Resolve(ctx) for the profile +client, err := encrypt.NewClient(ctx) if err != nil { - return err -} -defer store.Close() -strategy, err := store.AccessKey(ctx, crn, accessKey) // or DeviceSession, OIDC, Auto -if err != nil { - return err -} -defer strategy.Close() -client, err := encrypt.NewClient(ctx, - encrypt.WithCredentials(encrypt.NewCredentials(clientID, clientKey, strategy)), -) -if err != nil { - return err + return err // encrypt.ErrNoCredentials: nothing configured } defer client.Close() ``` -The strategy is asked for the bearer token on every request, and mints or -refreshes it as it needs to. The store and the strategy stay yours: the -client never closes them, so keep them open until `client.Close` has -returned, as the deferred calls above do. A nil strategy is refused. - -Tokens come only from `auth` strategies; there is no way to hand the -client a raw bearer token. A raw token cannot be refreshed when it expires, -and a source outside the strategies would bypass the cross-process lock a -device-session refresh holds with the `stash` CLI. - -To authenticate through your own identity provider, pass `OIDCFederation` -with the workspace CRN and a provider of the IdP's tokens. The provider is -asked on every token fetch for the IdP token of the user the call is for; -CTS exchanges each distinct IdP token for a CipherStash one, which is cached -for that token until it expires, so one client serves many users and none -rides another's token. `auth.OAuth2TokenSource` adapts a -`golang.org/x/oauth2` source. The client key is found as `AutoCredentials` -finds it. `auth` strategy options follow the provider: -`OIDCFederation(crn, provider, auth.WithAuthBaseURL(cts))` pins the CTS -endpoint for these credentials, where `CS_CTS_HOST` would pin it for the -whole process. - -`AutoCredentials`, `NewCredentials` and `OIDCFederation` are the only kinds -of `Credentials`: the interface is sealed. - -`ClientKey` is an opaque type, not a string: it prints a redaction under -every verb, so logged credentials never show the key. `NewClientKey` takes -ownership of the slice it is given, and `NewClient` consumes the key — -whatever the outcome, even options it refuses, the key is empty afterwards -and that slice is zero. A key is for one client; build another for another -client. What the SDK cannot reach is what the key was built *from*: a -string read from the environment is Go's, immutable, and lives until -collected. Where an environment variable is the source, treat the process -environment as holding the key for the life of the process. - -### Key material in memory - -The client key enters the instance once, in `NewClient`: it is marshalled -into the config buffer, the `ClientKey` is wiped, the guest copies the key -into its own memory, and the buffer is wiped. From then on the client key, -every loaded index key and every data key in use live in -the wasm instance's memory, and the package owns that memory rather than -leaving it to the runtime's default. It is reserved once and never moves, -so growth never copies a key to somewhere it is not wiped; it is locked in -RAM (`mlock`, `VirtualLock`) so it is never written to swap; on Linux it is -excluded from core dumps (`MADV_DONTDUMP`); and it is wiped before it is -released, on every release path. None of that waits for `Close`. A process -killed by SIGKILL, the OOM killer, a panic on another goroutine or `os.Exit` -runs no deferred call, and the kernel zeroes its pages before anyone else -sees them; the lock and the dump exclusion close the two places a copy -could otherwise outlive the process. - -The lock is best effort. `RLIMIT_MEMLOCK` defaults to 64 KiB on many Linux -hosts and the instance is larger, so the lock is often refused, and the -client then runs with memory the kernel may swap out, which is all that is -lost, and nothing on a host without swap. `client.MemoryLocked()` reports -the outcome and `client.MemoryLockError()` names the limit to raise -(`ulimit -l`, a systemd `LimitMEMLOCK=`, a pod `securityContext`) and the -size the instance holds. For a deployment that would rather not start than -run unlocked, pass `WithRequireLockedMemory()` and `NewClient` fails with -`ErrMemoryLock`. That policy holds for the life of the client: memory the -instance later grows into must lock too, or the call that needed it fails -with `ErrMemoryLock`, so grant a limit with room to grow. A `Client` prints -its memory state with `%v` and logs it as a `slog` group, so a startup log -shows it. - -The report and the policy cover the credential guest too, which holds the -token strategy and which the client key may have passed through. -`AutoCredentials` and `OIDCFederation` open it under the client's policy. -With `NewCredentials` it is the `auth` store you opened: under -`WithRequireLockedMemory()`, `NewClient` fails with `ErrMemoryLock` if that -store's memory is unlocked, but the store's own policy decides its later -growth. Open it with `auth.RequireLockedMemory()` as well to keep it -locked for the life of the client. - -Production checklist: assert `MemoryLocked()` at startup, or pass -`WithRequireLockedMemory()`, and with `NewCredentials` open the store with -`auth.RequireLockedMemory()`. Handling `SIGTERM` for a graceful shutdown is -ordinary Go practice and worth doing for your own reasons; the SDK does not -depend on it and installs no signal handler of its own. - -`Close` runs the guest's own shutdown, wiping every key in place before the -instance is released, and takes no context because it does no I/O. A -`Client` that becomes unreachable without `Close` is released by a runtime -cleanup, which covers the forgot-to-close case in a running process and -nothing at exit. - -## Plans from a policy - -A record plan says which fields to encrypt, under which context, with which -index terms. `stash` tags or `NewPlan` spell it out by hand. The `plan` -subpackage derives it from what the schema already says about each field -(its facts, such as Fideslang `data_categories`), through a policy written -in Go: - -```go -import "github.com/cipherstash/stack/languages/golang/encrypt/plan" +`ctx` is Go's `context.Context`: the deadline and cancellation for the one +ZeroKMS round trip `NewClient` makes. It has nothing to do with an +*encryption* context, which is what a field is sealed under; that is the +`context=` tag. -type Individual struct { - ID int64 - Email string - MedicareNo string -} +Every other setting is a functional option with a default: -// Facts come from a Source, such as the protobuf one planned in CIP-4088. -// Any function returning facts is one. -var source = plan.SourceFunc(func(msg any) ([]plan.Fact, error) { - return []plan.Fact{ - {Field: "id", GoField: "ID"}, - {Field: "email", GoField: "Email", Annotations: []plan.Annotation{ - {Key: "fides.data_categories", Values: []string{"user.contact.email"}}}}, - {Field: "medicare_no", GoField: "MedicareNo", Annotations: []plan.Annotation{ - {Key: "fides.data_categories", Values: []string{"user.government_id"}}}}, - }, nil -}) - -var category = plan.Key("fides.data_categories") - -var Base = plan.FirstOf( - plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(encrypt.Equality))), - plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(encrypt.Equality, encrypt.Match))), - plan.When(category.Under("user"), plan.Encrypt(plan.EQL())), +```go +client, err := encrypt.NewClient(ctx, + encrypt.WithCredentials(encrypt.OIDCFederation(crn, provider)), + encrypt.WithTransport(rt), + encrypt.WithKeysetCacheSize(64), + encrypt.WithRequireLockedMemory(), ) +``` -var Individuals = plan.ForMessage(&Individual{}, plan.Table("individuals"), - plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), - plan.Column("medicare_number")), - ).OrElse(Base), -) +### Credentials -// At startup: panics if a classified field is decided by no rule, or the -// plan names a field the struct does not have. -var individuals = plan.MustPlanFor(source, Individuals) +`AutoCredentials`, the default, reads the environment first +(`CS_CLIENT_ACCESS_KEY` and `CS_WORKSPACE_CRN` for the token, `CS_CLIENT_ID` +and `CS_CLIENT_KEY` for the key), then the developer profile `stash auth login` +writes, through the [`auth`](../auth) package. `NewCredentials` takes a client +id, a client key and an `auth` strategy explicitly; `OIDCFederation` mints the +token from an identity provider's. No credentials take a raw token. The client +key is consumed by `NewClient` and wiped, whatever the outcome. -records, err := cipher.EncryptRecords(ctx, rows, encrypt.WithPlan(individuals)) -``` +## The cipher -A policy fails closed: a field with facts that no rule decides is an error -when the plan is built, naming the field and its facts. There is no default; -write a catch-all, `Plaintext()` included, in the policy. Fields with no -facts are left out and stored as they are. A message the policy encrypts -nothing of has no plan: `PlanFor` reports `ErrNothingEncrypted`, and its -records are stored without one. - -An EQL target's context is its column identity, `"
/"`. The -table is required per message, never derived from its name. A field is -stored in the column named by its schema name (its `Fact.Field`, such as -`medicare_no`: the spelling the Rust derive and the database column share) -unless a rule names another with `plan.Column`, and that column is also its -identity unless the rule pins one with `plan.Identity`. `plan.Field` matches on that same schema name. - -The identity is bound into every stored ciphertext, its data key and its -index terms, so once data is written it must never change. A field never -renamed in the database needs no `Identity`. After -`ALTER TABLE individuals RENAME COLUMN medicare_number TO medicare_num`, -new writes go to the new column under the old identity: +A `Cipher` is the client bound to one keyset, and to any extension of the +context. Every generated function takes one. ```go -plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), - plan.Column("medicare_num"), plan.Identity("medicare_number")) +cipher := client.Keyset(encrypt.KeysetName("tenant-42")).Extend("tenant-42") ``` -`plan.Custom` targets supply their own context: `plan.Column` names only -their record key, and `plan.Identity` is refused. The plan a -policy builds is a `Plan` like any other: the guest receives the same bytes -as for the equivalent hand-built plan. - -### Catch a changed context with a golden test - -A field's context must never change after you write data under it. -The library binds the context into every ciphertext and index term it writes for the field. -If the context changes, the rows you already wrote stop decrypting, and their index terms stop matching queries. - -You get no error when a context changes. -If you rename a proto field or a Go struct field, its context changes too. -Your code then writes new rows under the new context. -To keep the old context, pin the column with `plan.Column` in the field's rule. - -The `plan/plantest` package makes a changed context fail a test. -It gives you a golden test. -A golden test compares the plan with a file you commit, called the golden file. - -To add the test: - -1. Write a test that calls `plantest.Golden` with your source and your policy: - - ```go - import "github.com/cipherstash/stack/languages/golang/encrypt/plan/plantest" - - func TestIndividualsPolicy(t *testing.T) { - plantest.Golden(t, source, Individuals) - } - ``` - - The plantest package defines the `-update` flag. - If your test package defines its own `-update` flag, remove it, and read plantest's flag instead: - - ```go - func update() bool { - f := flag.Lookup("update") - return f != nil && f.Value.String() == "true" - } - ``` - -2. Run the test once with `-update`. - This writes the golden file to `testdata/TestIndividualsPolicy.golden`: - - ```sh - go test -run '^TestIndividualsPolicy$' -update - ``` - -3. Read the golden file, then commit it. - -The golden file names the message's table. -It also lists each field the policy decides: - -- An encrypted field shows its column, context, target, index terms and facts. -- A plaintext field shows its name and facts. +`Extend` appends to the context that the tags declare, for every field, in +every call through the cipher: the write, the query and the read. A row +written through `Extend("tenant-42")` opens and matches only through a cipher +with the same extension. No call takes a keyset or a context, so the three +cannot use different ones. -This is the golden file for the `Individuals` policy above: +`Client.DefaultKeyset()` is the keyset a ZeroKMS administrator set for the +client; `Client.Keyset(encrypt.KeysetName(..))` or `Client.Keyset(id)` any +other, loaded on first use. `Cipher.KeysetID(ctx)` resolves it. -```text -# Written by plantest.Golden: what the policy stores each field it decides as. -# Regenerate it with go test -update; do not edit it by hand. -# A column's context is bound into every ciphertext and query term written under it, so once a row is written it must never change. - -table individuals - -column email - context ["individuals", "email"] - target EQL - terms eq match - fact fides.data_categories user.contact.email - -column medicare_number - context ["individuals", "medicare_number"] - target EQL - terms eq - fact fides.data_categories user.government_id -``` +## Reading -After that, each test run builds the plan the same way `MustPlanFor` does at startup. -If the plan does not match the golden file, the test fails. -The failure lists each change, with the most costly changes first: +`users.Decrypt` takes a `Decrypter`: the `*Cipher`, which refuses a row +another keyset sealed with `ErrForeignKeyset` before any key is retrieved, or +the `*Client`, which opens each row under the keyset that sealed it. -1. **Context changes.** - These lose data. - If a rename caused the change, the failure names the `plan.Column` or `plan.Identity` pin that keeps the old context. -2. **Target changes.** - These need a migration. - For example, the index terms are different, or a plaintext field is now encrypted. -3. **Other changes.** - These do not affect rows you already wrote. - For example, the policy decides a new field. +## What is stored -The golden file lists encrypted fields by column, not by field name. -If you rename a field and your policy pins its column, the file stays the same and the test passes. +A field with `encrypt` and `index=` is a struct with one field for each +output: `Ciphertext` (`encrypt.Ciphertext`, the frozen stack-encrypt leaf), +and `Equality`, `Match`, `Ore` or `Ope` (the term types). A field with +`encrypt` alone has `Ciphertext` only. A passthrough field keeps its Go type. +An `opaque` struct is one `Sealed` field. Every stored type implements +`driver.Valuer` and `sql.Scanner`. -Change a context only before you write rows under it, or ship a migration with the change. -If you made a change on purpose, run the test again with `-update`. -Then read the diff of the golden file before you commit it. +Terms are byte-equal to the ones the Rust crate derives, so a term from +`users.Fields.Email.Equality` compares against a stored term written from any +language. `EqualityTerm.Equal` compares in constant time; `OreTerm.Compare` +and `OpeTerm.Compare` order as the plaintexts; `MatchTerm.Positions` decodes +the token positions. ## Errors -Errors are sentinel values, matched with `errors.Is`. The wasm guest -reports a status code and nothing else, so the vocabulary is deliberately -small and reveals nothing about plaintext or key material. - -| Error | Meaning | -|---|---| -| `ErrAuthentication` | A ciphertext failed to open: tampered, or presented under the wrong context or element derivation. | -| `ErrForbidden` | ZeroKMS refused the request. Also the production form of a wrong-context open, because every data key is bound to its context. | -| `ErrUnauthorized` | ZeroKMS rejected the bearer token: invalid, expired, or for another workspace. | -| `ErrNotFound` | Unknown keyset name or id, or a missing data key. | -| `ErrForeignKeyset` | A keyset-bound `Cipher` was given another keyset's ciphertext. Open it through the `Client`. | -| `ErrEncoding` | Malformed input: a value, ciphertext, plan, context or config refused before any cryptography. | -| `ErrTerm` | A term could not be derived, for example match text that yields no tokens. | -| `ErrTransport` | ZeroKMS could not be reached, or the token strategy failed. The strategy's own error is wrapped in it, so `errors.Is` finds that too (a refused refresh is `auth.ErrInvalidGrant`). | -| `ErrKMS` | Any other ZeroKMS failure. | -| `ErrConflict` | ZeroKMS reported a resource conflict. | -| `ErrState` | The client has been closed: by `Close`, by a call its context interrupted, or by a guest trap. | -| `ErrMemoryLock` | The instance's memory could not be locked in RAM. Returned by `NewClient` under `WithRequireLockedMemory()`, and by a call whose growth could not be locked; otherwise reported by `MemoryLockError`. | -| `ErrNoCredentials` | `NewClient` found no token strategy or no client key, in the environment or the profile. The message names what to set. | -| `ErrCredentialsConsumed` | `NewCredentials` given to a second `NewClient`: the first consumed its key. Build new credentials, with a new key, for another client. | -| `ErrInternal` | An unexpected failure inside the guest. | - -## How it works under the hood - -- **One implementation.** The `stack-encrypt` Rust crate is compiled to a - WASI module and embedded in the package. There is no separate Go port of - the cryptography, so ciphertexts and terms are byte-identical to the Rust - crate's and interchange with every other binding. -- **Key material stays in the guest.** The client key crosses into wasm - memory once at `NewClient`. Data keys are retrieved from ZeroKMS into - guest memory and never surface in Go. That memory is the package's own: - reserved once so it never moves, locked and excluded from core dumps - where the platform allows, wiped before release. `Close` runs the guest's - own shutdown as well, so every key is wiped in place before the instance - is released. -- **Two host imports.** The guest imports exactly one HTTP send, served by - your `http.RoundTripper`, and one bearer-token fetch, served by the - credentials' `auth` strategy. What crosses the boundary per ZeroKMS call is what would - cross TLS anyway. The guest sees no environment and no filesystem. -- **Real randomness.** The guest draws IVs and nonces from the process - CSPRNG. wazero's default random source is deterministic, so the package - configures every instance with `crypto/rand` explicitly. -- **Concurrency.** A wasm instance is single-threaded, so calls on one - `Client` are serialised internally. The `Client` is safe to share. +The generator and the compiler find a mistake in a declaration, so no call +returns an error for one. A call returns an error for a key, for the network, +or for stored data; read them with `errors.Is`: + +- `ErrForeignKeyset`: a `*Cipher` got a row another keyset sealed. +- `ErrForbidden`, `ErrAuthentication`: a ciphertext that does not open under + its field's context (ZeroKMS refuses the key, or the AEAD fails). +- `ErrEncoding`: a stored value or a call that does not fit the declaration. +- `ErrUnauthorized`, `ErrNotFound`, `ErrTransport`, `ErrKMS`: ZeroKMS. +- `ErrState`: a call on a closed client. `ErrMemoryLock`: see below. + +No error, warning or log line holds a plaintext value. A generated type hides +its sealed fields when a program prints or logs it; the struct you wrote does +not, and `stashgen -redact` writes `String` and `LogValue` for it. + +## Key material + +Every key the guest holds lives in the guest's linear memory, which this +package supplies: reserved once, locked in RAM and excluded from core dumps +where the platform allows, and wiped before it is released. The lock is best +effort (`RLIMIT_MEMLOCK` is 64 KiB on many Linux hosts); `Client.MemoryLocked` +reports it, `Client.MemoryLockError` says why not, and +`WithRequireLockedMemory` makes `NewClient` refuse to start unlocked. See the +package documentation for the full account. + +## Building + +The package embeds `wasm/stack_encrypt_guest.wasm`, a build artefact of the +Rust crate in [`guest/`](guest/). It is not committed: run +`mise run wasm:guest:build` (and `mise run wasm:auth-guest:build` for the +credential guest) before `go test`, and +`mise run wasm:guest:build:deterministic` for the test build the hermetic +round-trip and fixture tests use. Without the builds `NewClient` returns +`ErrGuestNotBuilt` and the tests skip. diff --git a/languages/golang/encrypt/checker.go b/languages/golang/encrypt/checker.go new file mode 100644 index 000000000..8041daacc --- /dev/null +++ b/languages/golang/encrypt/checker.go @@ -0,0 +1,124 @@ +package encrypt + +import ( + "context" + "errors" + "fmt" + "net/http" + + "github.com/cipherstash/stack/languages/golang/internal/guest" + "github.com/cipherstash/stack/languages/golang/internal/record" + "github.com/cipherstash/vitaminc/bindings/go/vcffi" + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +// Checker asks the embedded engine about declarations, with no credentials, +// no keyset and no network: what stashgen uses to refuse a declaration the +// engine would refuse, and to learn which EQL types the engine produces. It +// holds a guest instance that was never given a client key, so every other +// operation on it is [ErrState]. +type Checker struct { + c *Client +} + +// NewChecker instantiates the embedded guest for the generator's questions. +func NewChecker(ctx context.Context) (*Checker, error) { + wasm, err := embeddedGuest() + if err != nil { + return nil, err + } + t := &transport{rt: refusingTransport{}, token: noToken{}} + inst, err := newInstance(ctx, wasm, t, guest.BestEffort) + if err != nil { + return nil, err + } + return &Checker{c: newClient(inst, t)}, nil +} + +// Close releases the guest. +func (k *Checker) Close() error { return k.c.Close() } + +// Check refuses a plan the engine would refuse: an index its field's type +// does not admit, a context that is not a label, two fields under one +// identity. The error is [ErrEncoding]; which rule failed is the engine's to +// know, so a caller that wants the field named checks one field at a time. +func (k *Checker) Check(ctx context.Context, plan *record.Plan) error { + if err := plan.Validate(); err != nil { + return fmt.Errorf("%w: %v", ErrEncoding, err) + } + encoded, err := vcffi.Marshal(plan.Wire()) + if err != nil { + return fmt.Errorf("%w: %v", ErrEncoding, err) + } + _, err = k.c.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.planCheck, buf(encoded)) + }) + return err +} + +// Targets lists the EQL types this build of the engine produces: none until +// the EQL target dispatch lands. +func (k *Checker) Targets(ctx context.Context) ([]record.Target, error) { + out, err := k.c.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.targets) + }) + if err != nil { + return nil, err + } + decoded, err := vcffi.Unmarshal(out) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrInternal, err) + } + obj, ok := decoded.(vcvalue.Object) + if !ok || len(obj) != 1 || obj[0].Key != "targets" { + return nil, fmt.Errorf("%w: se_targets returned %T", ErrInternal, decoded) + } + items, ok := obj[0].Value.([]any) + if !ok { + return nil, fmt.Errorf("%w: se_targets returned %T for the list", ErrInternal, obj[0].Value) + } + targets := make([]record.Target, 0, len(items)) + for _, item := range items { + entry, ok := item.(vcvalue.Object) + if !ok { + return nil, fmt.Errorf("%w: a target came back as %T", ErrInternal, item) + } + var t record.Target + for _, f := range entry { + switch f.Key { + case "name": + t.Name, _ = f.Value.(string) + case "kind": + kind, _ := f.Value.(string) + t.Kind = record.Kind(kind) + case "query": + t.Query, _ = f.Value.(string) + case "terms": + terms, _ := f.Value.([]any) + for _, term := range terms { + s, _ := term.(string) + t.Terms = append(t.Terms, record.Output(s)) + } + } + } + if t.Name == "" { + return nil, fmt.Errorf("%w: a target has no name", ErrInternal) + } + targets = append(targets, t) + } + return targets, nil +} + +// refusingTransport fails every request: the checker makes none. +type refusingTransport struct{} + +func (refusingTransport) RoundTrip(*http.Request) (*http.Response, error) { + return nil, errors.New("encrypt: the checker makes no request") +} + +// noToken has no token: the checker needs none. +type noToken struct{} + +func (noToken) Token(context.Context) (string, error) { + return "", errors.New("encrypt: the checker has no credentials") +} diff --git a/languages/golang/encrypt/cipher.go b/languages/golang/encrypt/cipher.go index 4266848b7..3d5f58281 100644 --- a/languages/golang/encrypt/cipher.go +++ b/languages/golang/encrypt/cipher.go @@ -4,21 +4,28 @@ import ( "context" "fmt" - "github.com/cipherstash/vitaminc/bindings/go/vcffi" + "github.com/cipherstash/stack/languages/golang/internal/record" ) -// Cipher is a [Client] bound to one keyset: the Go form of the Rust -// crate's KeysetCipher. It seals values, derives terms and encrypts records -// under that keyset, and opens only that keyset's ciphertexts — a leaf -// sealed under another keyset is refused as [ErrForeignKeyset] before any -// key is retrieved. To open ciphertexts from any keyset, use the Client's -// decrypt methods. +// Cipher is a [Client] bound to one keyset, and to any extension of the +// context: the Go form of the Rust crate's KeysetCipher. Generated code seals +// values, derives terms and opens records through it; a program never calls +// the engine directly. A Cipher opens only its own keyset's ciphertexts — a +// leaf sealed under another keyset is refused as [ErrForeignKeyset] before +// any key is retrieved. To open ciphertexts from any keyset, pass the +// [Client] where a [Decrypter] is taken. // // A Cipher holds no guest state: the keyset is selected on every call, and -// loaded by the guest on first use. +// loaded by the guest on first use, so one is cheap to make per request or +// per tenant. type Cipher struct { - client *Client - keyset KeysetSelector + client *Client + keyset KeysetSelector + extension []any + // err is a refused extension part, reported by the first call rather + // than by Extend: a cipher is made without a request, and no call on + // this package panics. + err error } // Client is the client this cipher belongs to. @@ -37,140 +44,29 @@ func (cph *Cipher) KeysetID(ctx context.Context) (KeysetID, error) { return cph.client.resolveKeyset(ctx, cph.keyset) } -// Encrypt seals v under this keyset. v is encoded through the vcvalue -// model: builtins, slices, maps and structs by reflection, a type -// implementing vcffi.Encryptable by its own encoding, vcvalue.Plain marking -// a passthrough. aad is authenticated but not encrypted, and may be empty; -// the same aad must be presented to Decrypt. The leaves of v seal from -// batched ZeroKMS key requests: one per 500 keyed leaves, so one request -// for any ordinary value. -// -// The ciphertext comes back as ordinary Go values mirroring the -// plaintext's structure: Sealed leaves (and the SealedNone / SealedEmptySeq -// / SealedEmptyMap markers) where fields were encrypted, vcvalue.Plain -// where they passed through, map[string]any for records, []any for -// sequences. -func (cph *Cipher) Encrypt(ctx context.Context, v any, aad []byte) (any, error) { - return cph.encryptValue(ctx, v, aad, false) -} - -// EncryptElement seals v as a sequence element of the collection identified -// by aad — byte-identical to what Encrypt of a whole slice binds per -// element — so a single row inserted this way interchanges with rows -// written by encrypting a slice under the same aad. -func (cph *Cipher) EncryptElement(ctx context.Context, v any, aad []byte) (any, error) { - return cph.encryptValue(ctx, v, aad, true) -} - -// Decrypt opens a ciphertext sealed under this keyset. ct is the shape -// Encrypt returns (any subset of a record's entries decrypts); the -// plaintext is returned in vcvalue's decode shape: Go natives, -// vcvalue.Object for records, vcvalue.Plain for passthrough fields. A leaf -// from another keyset is ErrForeignKeyset. -func (cph *Cipher) Decrypt(ctx context.Context, ct any, aad []byte) (any, error) { - return cph.client.decryptValue(ctx, cph.keyset, ct, aad, false) -} - -// DecryptElement opens a ciphertext sealed as a sequence element — a row -// of a collection encrypted from a slice, or by EncryptElement — under the -// same aad. Elements are authenticated against a derivation of the -// collection's aad, so Decrypt cannot open a lone row. -func (cph *Cipher) DecryptElement(ctx context.Context, ct any, aad []byte) (any, error) { - return cph.client.decryptValue(ctx, cph.keyset, ct, aad, true) -} - -// Term derives one index term for value under context: the probe that -// compares against a term stored by EncryptRecords for a field sealed under -// the same keyset and the same context. value is a scalar: an integer, -// string or byte slice for Equality (floats and booleans have no equality -// encoding); a string for Match; any scalar for Ore and Ope. The result is -// one of EqualityTerm, MatchTerm, OreTerm or OpeTerm. -// -// opts are [Option]s, the options a probe shares with the record calls; a -// [RecordOption] that only a record call takes, such as [WithPlan], does -// not compile here. [ExtendContext] extends context exactly as it extends -// each field's own context in a record call, so a probe for a field -// written under an extension is the field's context plus the same option -// value the rows were written with, never a context spelled by hand. -// -// Term takes a context and returns an error because it may be a ZeroKMS -// round trip: term derivation is asynchronous in the Rust crate, and a -// ZeroKMS backend that derives terms server-side settles the same way. -func (cph *Cipher) Term(ctx context.Context, value any, context Context, kind TermKind, opts ...Option) (any, error) { - if context.isZero() { - return nil, fmt.Errorf("encrypt: term context is empty") - } - var o termOptions - for _, opt := range opts { - opt.applyTerm(&o) - } - context, err := extend(context, o.extension) - if err != nil { - return nil, err - } - encodedValue, err := vcffi.Marshal(value) - if err != nil { - return nil, err - } - // The probe value is plaintext: its transport copy is wiped once it is - // in the guest, as is the context it binds. - defer wipe(encodedValue) - encodedContext, err := vcffi.Marshal(context.value()) - if err != nil { - return nil, err - } - defer wipe(encodedContext) - encodedOpts, err := vcffi.Marshal(options(cph.keyset)) - if err != nil { - return nil, err - } - out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { - return inst.call(ctx, inst.term, buf(encodedValue), buf(encodedContext), scalar(uint64(kind)), buf(encodedOpts)) - }) - if err != nil { - return nil, err - } - return typedTerm(kind, out), nil -} - -func typedTerm(kind TermKind, bytes []byte) any { - switch kind { - case Equality: - return EqualityTerm(bytes) - case Match: - return MatchTerm(bytes) - case Ore: - return OreTerm(bytes) - case Ope: - return OpeTerm(bytes) - default: - return bytes +// Extend returns a cipher that extends the context of every field, in every +// call through it: the write, the query and the read. It appends to the +// context the tags declare and never replaces it, so a field sealed through +// cipher.Extend("tenant-42") opens and matches only through a cipher with +// the same extension. A part is a string, a byte slice or an integer +// (int32, int64, uint32, uint64; Go's int is sent as int64); a byte slice is +// copied. Several parts nest in order: Extend(a, b) is Extend(a).Extend(b). +// An unsupported part type is reported by the first call through the +// cipher, as [ErrEncoding]. +func (cph *Cipher) Extend(parts ...any) *Cipher { + next := &Cipher{client: cph.client, keyset: cph.keyset, err: cph.err} + next.extension = append(next.extension, cph.extension...) + for _, part := range parts { + if err := record.CheckPart(part); err != nil && next.err == nil { + next.err = fmt.Errorf("%w: Extend: %v", ErrEncoding, err) + } + next.extension = append(next.extension, part) } + return next } -func (cph *Cipher) encryptValue(ctx context.Context, v any, aad []byte, element bool) (any, error) { - encoded, err := vcffi.Marshal(v) - if err != nil { - return nil, err - } - // The transport copy of the plaintext is wiped once it is in the guest. - defer wipe(encoded) - opts, err := vcffi.Marshal(options(cph.keyset)) - if err != nil { - return nil, err - } - out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { - fn := inst.encrypt - if element { - fn = inst.encryptElement - } - return inst.call(ctx, fn, buf(encoded), buf(aad), buf(opts)) - }) - if err != nil { - return nil, err - } - // Passthrough (vcvalue.Plain) fields come back in the clear; the - // serialized copy is wiped once decoded. - defer wipe(out) - return unmarshalCipherText(out) +// Extension is the parts the cipher extends every field's context by, in +// order. Empty for a cipher straight from [Client.Keyset]. +func (cph *Cipher) Extension() []any { + return append([]any(nil), cph.extension...) } diff --git a/languages/golang/encrypt/ciphertext.go b/languages/golang/encrypt/ciphertext.go new file mode 100644 index 000000000..7aa4b4fef --- /dev/null +++ b/languages/golang/encrypt/ciphertext.go @@ -0,0 +1,83 @@ +package encrypt + +import ( + "database/sql/driver" + "fmt" + + "github.com/cipherstash/vitaminc/bindings/go/vcffi" +) + +// Ciphertext is one sealed field as a database column holds it: the frozen +// stack-encrypt leaf encoding (version, keyset id, IV, ZeroKMS tag, +// ciphertext). A generated type holds one for each field sealed in separate +// columns, beside that field's terms. It is a distinct type from vitaminc's +// own leaf on purpose: a stack-encrypt leaf is not decryptable by +// vitaminc-encrypt and must never scan or marshal where one belongs. +// +// A field of any Go type seals to one leaf. A scalar seals as the typed leaf +// a Rust record derives; a struct, slice or map seals as one self-describing +// value, which only a dynamic reader opens. +type Ciphertext []byte + +// Value implements driver.Valuer, binding the leaf as a byte column. +func (c Ciphertext) Value() (driver.Value, error) { return []byte(c), nil } + +// Scan implements sql.Scanner, loading a leaf from a byte column. +func (c *Ciphertext) Scan(src any) error { + b, err := scanBytes("Ciphertext", src) + *c = b + return err +} + +// scanBytes copies a driver byte value: drivers may reuse the source slice +// after Scan returns. +func scanBytes(kind string, src any) ([]byte, error) { + switch v := src.(type) { + case []byte: + out := make([]byte, len(v)) + copy(out, v) + return out, nil + case string: + return []byte(v), nil + case nil: + return nil, fmt.Errorf("encrypt: cannot scan NULL into %s", kind) + default: + return nil, fmt.Errorf("encrypt: cannot scan %T into %s", src, kind) + } +} + +// otherLeaf is a leaf of a kind the SDK does not store: the authenticated +// markers for an absent value, an empty sequence and an empty map, which a +// record field never produces. Decoding one is an error at the use site. +type otherLeaf struct { + kind vcffi.LeafKind + bytes []byte +} + +// leaves is the vcffi.LeafSet of this binding's leaf types. +var leaves = vcffi.LeafSet{ + Classify: func(v any) (vcffi.LeafKind, []byte, bool) { + switch n := v.(type) { + case Ciphertext: + return vcffi.LeafSingle, n, true + case otherLeaf: + return n.kind, n.bytes, true + default: + return 0, nil, false + } + }, + Make: func(kind vcffi.LeafKind, bytes []byte) any { + if kind == vcffi.LeafSingle { + return Ciphertext(bytes) + } + return otherLeaf{kind: kind, bytes: bytes} + }, +} + +func marshalCipherText(v any) ([]byte, error) { + return vcffi.MarshalCipherText(leaves, v) +} + +func unmarshalCipherText(buf []byte) (any, error) { + return vcffi.UnmarshalCipherText(leaves, buf) +} diff --git a/languages/golang/encrypt/client.go b/languages/golang/encrypt/client.go index d82a3d326..acaa16868 100644 --- a/languages/golang/encrypt/client.go +++ b/languages/golang/encrypt/client.go @@ -342,11 +342,12 @@ func (c *Client) Close() error { // StackCipher::keyset. No request is made here — Go has no await, so the // keyset is resolved by the guest on the cipher's first use (and cached), // which makes a Cipher cheap to make per call, per tenant or per request. -// [Cipher.KeysetID] is the explicit resolution point. A nil selector is a -// programming error and panics; the default keyset is [Client.DefaultKeyset]. +// [Cipher.KeysetID] is the explicit resolution point. A nil selector gives a +// cipher whose every call fails with [ErrEncoding]; the default keyset is +// [Client.DefaultKeyset]. func (c *Client) Keyset(sel KeysetSelector) *Cipher { if sel == nil { - panic("encrypt: Client.Keyset(nil); the default keyset is Client.DefaultKeyset") + return &Cipher{client: c, keyset: defaultKeyset{}, err: fmt.Errorf("%w: Client.Keyset(nil); the default keyset is Client.DefaultKeyset", ErrEncoding)} } return &Cipher{client: c, keyset: sel} } @@ -381,33 +382,6 @@ func (c *Client) resolveKeyset(ctx context.Context, sel KeysetSelector) (KeysetI return id, nil } -// Decrypt opens a ciphertext produced by any keyset of this client: each -// leaf is opened under the keyset it was sealed with, with batched key -// retrievals per keyset (one per 500 leaves sealed under it). ct is the -// shape Cipher.Encrypt returns; aad must be what the value was sealed -// under. -func (c *Client) Decrypt(ctx context.Context, ct any, aad []byte) (any, error) { - return c.decryptValue(ctx, anyKeyset{}, ct, aad, false) -} - -// DecryptElement is Decrypt for a value sealed as a sequence element; see -// Cipher.DecryptElement. -func (c *Client) DecryptElement(ctx context.Context, ct any, aad []byte) (any, error) { - return c.decryptValue(ctx, anyKeyset{}, ct, aad, true) -} - -// DecryptRecords opens records produced by Cipher.EncryptRecords under any -// keyset of this client, into a slice; see Cipher.DecryptRecords. -func (c *Client) DecryptRecords(ctx context.Context, records []EncryptedRecord, out any, opts ...RecordOption) error { - return c.decryptRecords(ctx, anyKeyset{}, records, out, opts) -} - -// DecryptRecord opens one record under any keyset of this client; see -// Cipher.DecryptRecord. -func (c *Client) DecryptRecord(ctx context.Context, record EncryptedRecord, out any, opts ...RecordOption) error { - return c.decryptRecord(ctx, anyKeyset{}, record, out, opts) -} - // call runs f on the instance under the client's lock. // // A call interrupted by its context (the runtime closes the module on a @@ -462,27 +436,3 @@ func (c *Client) call(ctx context.Context, f func(*instance) ([]byte, error)) ([ } return out, nil } - -func (c *Client) decryptValue(ctx context.Context, sel KeysetSelector, ct any, aad []byte, element bool) (any, error) { - encoded, err := marshalCipherText(ct) - if err != nil { - return nil, err - } - opts, err := vcffi.Marshal(options(sel)) - if err != nil { - return nil, err - } - out, err := c.call(ctx, func(inst *instance) ([]byte, error) { - fn := inst.decrypt - if element { - fn = inst.decryptElement - } - return inst.call(ctx, fn, buf(encoded), buf(aad), buf(opts)) - }) - if err != nil { - return nil, err - } - // The output is plaintext: decode, then wipe the transport copy. - defer wipe(out) - return vcffi.Unmarshal(out) -} diff --git a/languages/golang/encrypt/context.go b/languages/golang/encrypt/context.go deleted file mode 100644 index 28f5d2123..000000000 --- a/languages/golang/encrypt/context.go +++ /dev/null @@ -1,187 +0,0 @@ -package encrypt - -import ( - "bytes" - "errors" - "fmt" - "reflect" -) - -// Context is the encryption context a record field or a term probe binds: -// a domain-separating value that becomes both the ciphertext's AAD (and -// the ZeroKMS descriptor the data key is bound to) and the index terms' -// PRF context. Contexts are identities, so a probe must spell the context -// in exactly the shape the field was sealed under. -// -// A Context is a part or a list of parts. A part is a string, a byte slice -// or an integer (int32, int64, uint32, uint64; Go's int is sent as int64). -// [NewContext] makes a one-part context — the bare part, what a Rust -// `nonempty!("..")` literal binds. [Context.With] extends it as -// Rust's NonEmpty::with does: the result is the two-element list -// [previous, part], nesting to the left. So NewContext("users").With("age") -// is the pair a Rust `struct = .., context = "users"` derive binds its `age` -// field under — and what ParseLabel("users/age") binds — rendering the ZeroKMS -// descriptor users/age; extended With(uint64(7)) it is what a row sealed -// with the chain's .extend(7u64) binds for that field. -// A one-element list is not the bare part, and this type cannot spell one. -// -// Not every Context is a planned field's. A probe ([Cipher.Term]) takes any -// Context, in whatever shape the data was sealed under. A field of a record -// plan ([FieldPlan.Context]) binds a [Label] of at least two plain segments -// and nothing else — the guest lowers a plan into one context per record -// with one identity per field, and refuses any other shape — so [NewPlan] -// and [PlanFromTags] refuse a one-part context, a one-segment label and an -// extended context at construction. A record call extends every field's -// label alike with [ExtendContext]. -// -// A Context owns its parts: a byte-slice part is copied in, so a caller's -// buffer reused once the Context is built does not change it. -// -// Compare two Contexts with [Context.Equal]. Do not use == and do not use a -// Context as a map key: a list context holds a slice, and Go panics when it -// compares those. -type Context struct { - node any -} - -// NewContext makes a one-part context, for a probe ([Cipher.Term]) against -// data sealed under one part, or as the base [Context.With] extends. It is -// not a planned field's context: a field binds a [Label] of two or more -// segments ([ParseLabel]), and [NewPlan] refuses a one-part context with the -// field named. The part must not be empty: a bare -// empty string or empty byte slice is an empty context, and the guest -// proves every context non-empty at the boundary, so such a Context could -// only ever fail — every call, with ErrEncoding. Rust refuses the same -// thing one step earlier: nonempty!("") does not compile. -// -// Emptiness is the whole tree's property, not the part's — a list is empty -// only when every part is — so [Context.With] may still add an empty part -// to a context that already has a non-empty one. Only the root is checked -// here. -func NewContext(part any) (Context, error) { - if err := checkPart(part); err != nil { - return Context{}, err - } - if err := checkRootNonEmpty(part); err != nil { - return Context{}, err - } - return Context{node: ownPart(part)}, nil -} - -// MustContext is [NewContext] for a part known to be valid; it panics -// otherwise, an empty part included. For string literals in probes. -func MustContext(part any) Context { - c, err := NewContext(part) - if err != nil { - panic(err) - } - return c -} - -// With extends the context by one part, nesting to the left. -func (c Context) With(part any) (Context, error) { - if c.node == nil { - return Context{}, fmt.Errorf("encrypt: cannot extend an empty context") - } - if err := checkPart(part); err != nil { - return Context{}, err - } - return Context{node: []any{c.node, ownPart(part)}}, nil -} - -// ownPart is part as a context stores it: a byte slice is copied, so -// neither a Context nor an option that extends one ([ExtendContext]) -// aliases a caller's buffer. Every other part type is a value. -func ownPart(part any) any { - if b, ok := part.([]byte); ok { - return bytes.Clone(b) - } - return part -} - -// value renders the context in the guest's grammar: a scalar or nested -// lists of scalars, ready for the transport codec. -func (c Context) value() any { return c.node } - -// fieldLabel is the context a planned field may bind, as the guest's -// lowering reads it: a label of at least two plain segments and nothing -// else, returned as that Label. A one-part context, an extended context and -// a part that is not text are refused with a reason that says what is -// accepted; a flat list of plain text parts is the label it spells, -// whichever constructor built it. -func (c Context) fieldLabel() (Label, error) { - parts, ok := c.node.([]any) - if !ok { - return Label{}, errors.New(`is one part, not a label; a planned field binds a label of at least two plain segments, ParseLabel("table/column").Context()`) - } - segments := make([]string, 0, len(parts)) - for _, part := range parts { - s, ok := part.(string) - if !ok { - return Label{}, errors.New("is extended, or holds a part that is not text; a planned field binds a plain label, and a record call extends every field's label alike with ExtendContext") - } - segments = append(segments, s) - } - if len(segments) < 2 { - return Label{}, errors.New("has one segment; a planned field binds a label of at least two") - } - l, err := NewLabel(segments...) - if err != nil { - return Label{}, fmt.Errorf("is not a plain label: %w", err) - } - return l, nil -} - -// checkRootNonEmpty refuses the bare parts that are themselves an empty -// context. Integers never are, whatever their value. -func checkRootNonEmpty(part any) error { - switch p := part.(type) { - case string: - if p == "" { - return errors.New("encrypt: an empty string is an empty context") - } - case []byte: - if len(p) == 0 { - return errors.New("encrypt: an empty byte slice is an empty context") - } - } - return nil -} - -func checkPart(part any) error { - switch part.(type) { - case string, []byte, int32, int64, uint32, uint64, int: - return nil - default: - return fmt.Errorf("encrypt: %T is not a context part (string, []byte or integer)", part) - } -} - -// Equal reports whether c and other are the same context: the same parts, -// in the same order, with the same types, so a probe built from one matches -// terms written under the other. This is the supported comparison; == on -// two Contexts panics when either holds a list. -func (c Context) Equal(other Context) bool { return reflect.DeepEqual(c.node, other.node) } - -// isZero reports whether c is the zero Context, which binds nothing: what a -// plan field without a context, or a zero Label, carries. -func (c Context) isZero() bool { return c.node == nil } - -// flatContext is the context a [Label] binds: one segment is the bare part, -// as NewContext makes it; two or more are a flat list of the segments. The -// segments are plain by construction, so no part check is needed, and a -// list is never built from one part (a one-element list is a different -// context from the bare part, and this type cannot spell one). -func flatContext(segments []string) Context { - switch len(segments) { - case 0: - return Context{} - case 1: - return Context{node: segments[0]} - } - parts := make([]any, len(segments)) - for i, s := range segments { - parts[i] = s - } - return Context{node: parts} -} diff --git a/languages/golang/encrypt/credentials_test.go b/languages/golang/encrypt/credentials_test.go index 1a022a6dd..bd12205f2 100644 --- a/languages/golang/encrypt/credentials_test.go +++ b/languages/golang/encrypt/credentials_test.go @@ -801,7 +801,7 @@ func TestRequireLockedMemoryRefusesUnlockedCredentials(t *testing.T) { } creds, resolved := forcedCreds(t) // No crypto guest is needed: the refusal precedes it. - _, err := NewClient(context.Background(), WithCredentials(creds), withZeroKMSURL(stub.URL), WithGuest(wasiProbe), WithRequireLockedMemory()) + _, err := NewClient(context.Background(), WithCredentials(creds), withZeroKMSURL(stub.URL), withGuest(wasiProbe), WithRequireLockedMemory()) if !errors.Is(err, ErrMemoryLock) || !strings.Contains(err.Error(), "credential guest") { t.Fatalf("NewClient under WithRequireLockedMemory: %v, want ErrMemoryLock naming the credential guest", err) } @@ -852,7 +852,7 @@ func TestRequireLockedMemoryAcceptsLockedCredentials(t *testing.T) { } return r, nil }) - _, err := NewClient(context.Background(), WithCredentials(creds), withZeroKMSURL(stub.URL), WithGuest(wasiProbe), WithRequireLockedMemory()) + _, err := NewClient(context.Background(), WithCredentials(creds), withZeroKMSURL(stub.URL), withGuest(wasiProbe), WithRequireLockedMemory()) if asked.Load() == 0 { t.Fatal("the credentials' memory report was not asked") } diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index 16d2bb99d..c1ce88202 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -1,154 +1,167 @@ -// Package encrypt is the Go binding of stack-encrypt: ZeroKMS-backed -// field-level encryption with searchable index terms, running the Rust -// crate unmodified inside a WASI guest under wazero (CGO_ENABLED=0). +// Package encrypt is the Stack Encrypt Go SDK: searchable, field-level +// encryption under per-value ZeroKMS data keys, running the stack-encrypt +// Rust engine unmodified inside a WASI guest under wazero (CGO_ENABLED=0). +// +// # Use the SDK +// +// 1. Add the generator to your module (Go 1.24 or later): +// +// go get -tool github.com/cipherstash/stack/languages/golang/cmd/stashgen +// +// 2. Put a stash tag on every exported field of the struct, and a go:generate +// comment beside it: +// +// //go:generate go tool stashgen -type User +// type User struct { +// _ struct{} `stash:"context=users"` +// ID int64 `stash:"id,passthrough"` +// Email string `stash:"email,encrypt,index=equality;match"` +// Age uint32 `stash:"age,encrypt,index=equality;ore"` +// } +// +// 3. Run the generator. It writes user_stash.go beside the struct: +// +// go generate ./... +// +// 4. Commit the generated file. +// +// 5. Call the generated functions where you write and read: +// +// client, err := encrypt.NewClient(ctx) +// cipher := client.Keyset(encrypt.KeysetName("tenant-42")) +// encrypted, err := users.Encrypt(ctx, cipher, people) // []users.EncryptedUser, one request +// people, err := users.Decrypt(ctx, cipher, encrypted) // []users.User +// term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com") +// +// 6. Store the encrypted type. Each field is one or more columns of bytes +// ([Ciphertext] and the term types), so database/sql, pgx, sqlx and GORM take +// it as it is. +// +// 7. Run the generator again after each change to the struct or to a tag. A +// change to the fields of the struct stops the build until you do. +// +// 8. In CI, run the generator and fail when a generated file changes: +// +// go generate ./... && git diff --exit-code +// +// The rest is the reference. The generator's own reference — the tag +// grammar, the flags, what it writes and what it refuses — is in +// cmd/stashgen/README.md. // // # Shape // // A [Client] is one wasm instance and one ZeroKMS client: [NewClient] -// resolves the client's [Credentials], instantiates the embedded guest, -// hands it the client key once — the [ClientKey] the credentials resolved -// to is consumed and wiped, whatever the outcome — and loads the client's -// default keyset. It takes functional options ([ClientOption]), every one -// with a default, so NewClient(ctx) alone is a working client. Every keyset -// the client uses after that is selected per call through a -// [KeysetSelector] and loaded on first use by the guest's own bounded -// cache; nothing the host could allocate, alias or free crosses the -// boundary. [Client.Close] runs the guest's shutdown so the client key and -// every loaded index key are wiped before the instance is freed — closing a -// wasm instance runs no Rust destructors on its own. Close is hygiene, not -// the security story: see Memory below. +// resolves the client's [Credentials], instantiates the embedded guest, hands +// it the client key once — the [ClientKey] the credentials resolved to is +// consumed and wiped, whatever the outcome — and loads the client's default +// keyset. It takes functional options ([ClientOption]), every one with a +// default, so NewClient(ctx) alone is a working client. [Client.Close] runs +// the guest's shutdown so the client key and every loaded index key are +// wiped before the instance is freed. // // A [Cipher] is the client bound to one keyset ([Client.Keyset] and -// [Client.DefaultKeyset], the Rust crate's StackCipher::keyset and -// default_keyset): it seals values, derives terms and encrypts records -// under that keyset, and opens only that keyset's ciphertexts. The -// [Client] itself opens ciphertexts from any keyset ([Client.Decrypt] and -// friends), fetching batched key retrievals per keyset the leaves were -// sealed under. -// -// # Values -// -// Values cross the boundary in vitaminc's FFI codec ([vcffi]) and are -// modelled as Go natives ([vcvalue]): builtins, slices, maps and structs -// seal by reflection, [vcvalue.Plain] marks a passthrough field, and a -// type implementing [vcffi.Encryptable] drives its own encoding. A -// ciphertext is the same dynamic shape with [Sealed] leaves — a distinct -// type from vcvalue's, because a stack-encrypt leaf is not a vitaminc leaf -// and must never scan or marshal where one belongs. The leaf bytes are the -// frozen stack-encrypt storage format; the transport encoding is not. -// -// # Records and terms -// -// [Cipher.EncryptRecords] is the runtime form of the Rust derive: a struct's -// `stash` tags say, per field, which context to bind and which index terms -// to produce, and one call seals every row of a slice from batched key -// requests. The same plan is a value ([Plan]): [PlanFromTags] is what the -// tags parse to, [NewPlan] builds one for a struct that cannot carry tags -// (generated code), and [WithPlan] runs a record call under it. The plan -// subpackage derives one from a field's schema facts through a policy -// written in Go. Key -// requests are batched 500 keys at a time, in both directions: one request -// for any ordinary value or batch, one more per 500 sealed leaves beyond -// that. Terms ([EqualityTerm], [MatchTerm], [OreTerm], [OpeTerm]) are -// byte-equal to the ones the Rust crate derives, so a probe from -// [Cipher.Term] compares against a stored term from any language. An -// [Option] is the one value that serves encrypt, decrypt and probe alike; -// a [RecordOption], such as [WithPlan], is what only a record call takes. -// [ExtendContext] is an Option: given to a record call and to the probe it -// extends the field's context and the probe's identically, so a -// tenant-scoped probe is the field's own context plus the option value the -// rows were written with, never a context spelled by hand. -// [Cipher.Term] takes a context and returns an error from day one: term -// derivation may be a ZeroKMS round trip. +// [Client.DefaultKeyset]) and to any extension of the context +// ([Cipher.Extend]): what changes from one caller to the next — a tenant, a +// region — attaches here and not to each call, so the write, the query and +// the read cannot use different ones. A Cipher opens only its own keyset's +// records; the [Client] opens records from any of its keysets. Both are a +// [Decrypter], which the generated Decrypt functions take. +// +// # Declarations +// +// A struct's stash tags declare how each field is encrypted: the context of +// the struct, and for each field whether it is sealed, which indexes are +// derived beside it, or whether it passes through as it is. stashgen reads +// the tags and writes the encrypted type, Encrypt, Decrypt and Fields into +// the struct's package; the generated code hands the declaration to the +// engine as data through package gensupport, and the engine runs the same +// plan the Rust chain and the derive run. A program never builds or names a +// plan, and no call takes a context or an index choice of its own. +// +// Passthrough fields never cross the binding: the engine does nothing to a +// passthrough value that a program could observe, and the FFI codec cannot +// carry every Go type a program stores beside a ciphertext (a time.Time, a +// driver.Valuer). An opaque struct crosses as one JSON document and is one +// column. +// +// # Terms +// +// A sealed field with an index gets a term beside its ciphertext: +// [EqualityTerm], [MatchTerm], [OreTerm] or [OpeTerm], byte-equal to the ones +// the Rust crate derives, so a term from a generated Fields entry compares +// against a stored term from any language. The indexes are [Equality], +// [Match], [Ore] and [Ope]; [JSON] is declared and refused until the engine +// derives it. Every stored type implements driver.Valuer and sql.Scanner. // // # Transport and auth // // The guest imports exactly two host functions: an HTTP send, served by any // [net/http.RoundTripper], and a bearer-token fetch, served by the -// credentials' auth strategy. What crosses per ZeroKMS call is what would cross TLS -// anyway; derived key material never leaves the guest. Under +// credentials' auth strategy. What crosses per ZeroKMS call is what would +// cross TLS anyway; derived key material never leaves the guest. Under // [AutoCredentials] and [OIDCFederation] the same RoundTripper also carries -// the authentication requests to CTS, so one scoped to the ZeroKMS host -// alone is not enough. Under [NewCredentials] those requests go through the -// store the caller opened the strategy from. +// the authentication requests to CTS, so one scoped to the ZeroKMS host alone +// is not enough. Under [NewCredentials] those requests go through the store +// the caller opened the strategy from. // // # Credentials // // A [Credentials] supplies the client id, the client key and the auth -// strategy the token comes from, and NewClient resolves it host-side: the crypto guest is never -// given the environment or a filesystem to find them in. The default, -// [AutoCredentials], mirrors the Rust client — the environment first -// (CS_CLIENT_ACCESS_KEY with CS_WORKSPACE_CRN for the token, CS_CLIENT_ID -// with CS_CLIENT_KEY for the key), then the developer profile, which it -// reads through auth's credential guest, where the token strategies -// also run. CS_ZEROKMS_HOST (or CS_VITUR_HOST) pins the endpoint whatever -// the credentials. [NewCredentials] takes a client id, a client key and a -// strategy explicitly, and [OIDCFederation] mints the token from an -// identity provider's. Pass one with [WithCredentials]. Those three are -// the only kinds of Credentials, and none takes a raw token: a token is -// always a auth strategy's, since a raw one cannot be refreshed when -// it expires and would bypass the cross-process lock a device-session -// refresh holds with the CLI. -// Credentials that cannot be resolved fail NewClient with -// [ErrNoCredentials]; credentials that resolve but do not work fail it too, -// at the one ZeroKMS round trip it makes. +// strategy the token comes from, and NewClient resolves it host-side: the +// crypto guest is never given the environment or a filesystem to find them +// in. The default, [AutoCredentials], mirrors the Rust client — the +// environment first (CS_CLIENT_ACCESS_KEY with CS_WORKSPACE_CRN for the +// token, CS_CLIENT_ID with CS_CLIENT_KEY for the key), then the developer +// profile, which it reads through the auth package's credential guest. +// CS_ZEROKMS_HOST (or CS_VITUR_HOST) pins the endpoint whatever the +// credentials. [NewCredentials] takes a client id, a client key and a +// strategy explicitly, and [OIDCFederation] mints the token from an identity +// provider's. None takes a raw token: a token is always a strategy's, since +// a raw one cannot be refreshed when it expires. Credentials that cannot be +// resolved fail NewClient with [ErrNoCredentials]; credentials that resolve +// but do not work fail it too, at the one ZeroKMS round trip it makes. +// +// # Errors +// +// The generator and the compiler find a mistake in a declaration, so no call +// returns an error for one. A call returns an error for a key, for the +// network, or for stored data: [ErrForeignKeyset] when a *Cipher is given a +// record another keyset sealed, [ErrAuthentication] or [ErrForbidden] for a +// ciphertext that does not open under its field's context, [ErrEncoding] for +// a stored value that does not fit its declaration. Read them with errors.Is. +// No error, warning or log line holds a plaintext value; a generated type +// hides its sealed fields when a program prints it. // // # Host runtime // // The guest also imports WASI random_get and clock_time_get, and the // cipher's security rests on the first: ZeroKMS IVs and AEAD nonces are // drawn from it. wazero's defaults for both are deterministic, so every -// instance is configured with the process CSPRNG ([crypto/rand.Reader]) -// and the system clocks. An embedder that instantiates the guest module -// under its own wazero configuration must do the same. +// instance is configured with the process CSPRNG ([crypto/rand.Reader]) and +// the system clocks. // // # Memory // // Every key the guest holds — the client key, each loaded index key, each // data key for the length of a call — lives in the guest's linear memory, -// and the package supplies that memory itself rather than taking wazero's -// default Go slice. It is reserved once at the module's declared maximum, -// so growth never copies it (wazero's default grows with append, which -// would leave an unwiped copy of every key to the garbage collector); -// locked in RAM (mlock, VirtualLock) so it is never written to swap; -// excluded from core dumps on Linux (MADV_DONTDUMP); and wiped before it -// is released, on every release path. -// -// That is deliberately done at allocation, where the caller cannot get it -// wrong, and not at exit, where they cannot be relied on: no deferred -// [Client.Close] runs on SIGTERM without a handler, SIGKILL, the OOM -// killer, a panic on another goroutine or os.Exit, and the package installs -// no signal handler — that is the application's to own, and covers only -// the first of those anyway. The kernel zeroes a dead process's pages -// before anyone else sees them; the lock and the dump exclusion close the -// two places a copy could otherwise outlive the process. -// -// The lock is best effort: RLIMIT_MEMLOCK defaults to 64 KiB on many -// Linux hosts and the guest is larger, so it is commonly refused, and a -// client then works on with memory that may be swapped — which is all -// that is lost, and nothing on a host without swap. [Client.MemoryLocked] +// and the package supplies that memory itself: reserved once so growth never +// copies it, locked in RAM (mlock, VirtualLock) so it is never written to +// swap, excluded from core dumps on Linux (MADV_DONTDUMP), and wiped before +// it is released, on every release path. That is done at allocation, where +// the caller cannot get it wrong, and not at exit, where they cannot be +// relied on. +// +// The lock is best effort: RLIMIT_MEMLOCK defaults to 64 KiB on many Linux +// hosts and the guest is larger, so it is commonly refused, and a client +// then works on with memory that may be swapped. [Client.MemoryLocked] // reports the outcome and [Client.MemoryLockError] the reason, naming the -// limit to raise (ulimit -l, a systemd LimitMEMLOCK=, a pod's -// securityContext). [WithRequireLockedMemory] turns a refusal into a -// [NewClient] failure with [ErrMemoryLock], for deployments that would -// rather not start than run unlocked; it also refuses any later growth of -// the guest's memory that cannot be locked, so the limit granted must -// leave the guest room to grow: a refused growth fails the call with -// [ErrMemoryLock], and closes the client when the growth was the guest's -// own allocation rather than a host-staged buffer. The report and the -// policy cover the credential guest as well; with [NewCredentials] that is -// the caller's auth store, which NewClient refuses under the policy -// when it is unlocked, and which should be opened with -// auth.RequireLockedMemory to stay locked (see -// [WithRequireLockedMemory]). A Client prints its memory state -// ([Client.String]) and logs it ([Client.LogValue]). An embedder running -// the guest under its own wazero configuration gets none of this unless -// it supplies an allocator of its own. -// -// Between calls the guest holds the client key and its keyset cache (each -// keyset's index key) and nothing else: data keys are per-call values in -// the guest's Rust code, wiped by their ZeroizeOnDrop when the export -// returns, and every buffer staged for a call is wiped by se_dealloc -// before the call's result is returned. The residency tests pin the -// second; the first is the Rust crate's own guarantee. +// limit to raise. [WithRequireLockedMemory] turns a refusal into a +// [NewClient] failure with [ErrMemoryLock], and refuses any later growth of +// the guest's memory that cannot be locked. A Client prints its memory state +// ([Client.String]) and logs it ([Client.LogValue]). +// +// Between calls the guest holds the client key and its keyset cache and +// nothing else: data keys are per-call values wiped when the export returns, +// and every buffer staged for a call is wiped before the call's result is +// returned. package encrypt diff --git a/languages/golang/encrypt/example/README.md b/languages/golang/encrypt/example/README.md index aa1f3ca75..0a995caca 100644 --- a/languages/golang/encrypt/example/README.md +++ b/languages/golang/encrypt/example/README.md @@ -1,72 +1,32 @@ -# stack-encrypt Go example +# encrypt example -A runnable tour of the Go binding against real ZeroKMS: seal a value, seal a -record with its index terms, probe those terms with a query, open both again. -It prints what crossed the boundary at each step, including the steps that -are meant to fail. +A runnable tour of the Go SDK against real ZeroKMS: declare a struct with +`stash` tags, generate its encrypted type, seal a batch in one request, derive +a query term, and open the batch again. ## Running it ```bash -stash auth login # once; the example reads ~/.cipherstash -mise run go:encrypt:example # builds both guests, then runs +stash auth login # once; the example reads ~/.cipherstash +mise run go:encrypt:example # builds both guests, then runs ``` Or, if you would rather drive it yourself: ```bash mise run wasm:guest:build wasm:auth-guest:build -cd bindings/go && go run ./encrypt/example +cd languages/golang && go run ./encrypt/example ``` -The guest builds are not optional. This package embeds -`wasm/stack_encrypt_guest.wasm` and `auth` embeds -`wasm/stack_auth_guest.wasm`; both are gitignored, so a fresh checkout has -no guests and `NewClient` / `Resolve` fail until they are built — and the Go -side will not notice a stale one, so rebuild after any change under either -`guest/src/`. +The guest builds are not optional. The `encrypt` package embeds +`wasm/stack_encrypt_guest.wasm` and `auth` embeds `wasm/stack_auth_guest.wasm`; +both are gitignored, so a fresh checkout has no guests and `NewClient` fails +until they are built. ## What it shows -| | | -|---|---| -| **A value** | An arbitrary map sealed under a caller-chosen AAD. A `vcvalue.Plain` field rides alongside in the clear. Opening under the wrong AAD is refused — at the *key retrieval*, not the AEAD, because every data key is bound to its context. | -| **A record** | A struct's `stash` tags drive a plan: each field sealed under its own context, with the index terms it asked for. Three rows, one batched ZeroKMS request. | -| **A query** | An equality term derived from the value being searched for, matched against the stored terms. The same value under another field's context matches nothing — that is what stops a hit in one column being a hit in another. | -| **Order** | ORE terms sorted, recovering the plaintext order without the plaintext. | - -## Credentials - -The example calls `NewClient(ctx)` with no options, so it resolves its -credentials with `encrypt.AutoCredentials`, which is what any -application gets by default. It looks in the environment first and then in -the developer profile, in the order the Rust client uses: - -- **The token.** If `CS_CLIENT_ACCESS_KEY` is set (with `CS_WORKSPACE_CRN`), - the access key is exchanged for a token. Otherwise the current workspace's - stored device session is used. Both run in `auth`'s credential guest. -- **The client key.** `CS_CLIENT_ID` and `CS_CLIENT_KEY` if both are set, - otherwise the current workspace's `secretkey.json`. -- **The endpoint.** `CS_ZEROKMS_HOST` if set, otherwise the token's - `services` claim. - -`CS_CONFIG_PATH` overrides the profile directory, and `CS_CTS_HOST` -overrides the authentication endpoint. - -Nothing in the example spells out the profile's layout. The `stack-profile` -crate reads it inside the credential guest, so the example cannot drift -from it. The crypto guest still sees no environment and no filesystem: -credentials are resolved host-side. - -The device session is **asked on every request and refreshes itself**. A -profile token lasts 45 minutes, so a pinned token would give you a program -that works for a while and then stops; that is why the client takes tokens -only from `auth` strategies and has no way to pass a raw one. The refresh takes the -same cross-process lock as the `stash` CLI. The IdP rotates refresh tokens -and detects replay, so two processes sharing `~/.cipherstash` that both -exchanged the same refresh token would get the whole chain revoked; the lock -prevents that. - -To supply the credentials yourself instead, from a secrets manager and with -no `CS_*` variables or profile, see [`explicit/`](explicit/), which uses -`encrypt.NewCredentials`. +- `model.go`: the struct and its `go:generate` line. +- `user_stash.go`: what `stashgen` wrote from the tags. Regenerate it with + `go generate ./...`; CI fails when that changes a committed file. +- `main.go`: `Encrypt`, `Fields.Email.Equality` and `Decrypt`, the only three + calls a program makes. diff --git a/languages/golang/encrypt/example/explicit/README.md b/languages/golang/encrypt/example/explicit/README.md deleted file mode 100644 index b76e09750..000000000 --- a/languages/golang/encrypt/example/explicit/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Explicit credentials - -A runnable example of `encrypt.NewCredentials`: the application -supplies every credential itself, and nothing is read from `CS_*` variables -or the developer profile. Use this shape when the client key lives in a -secrets manager, when one process talks to more than one workspace, or when -the environment is not yours to set. `../` shows the default, -`AutoCredentials`. - -## Running it - -The secrets come from a directory holding two files, the way a Kubernetes or -Docker secret is mounted: - -| File | Contents | -|---|---| -| `client-key` | The client key: the `CS_CLIENT_KEY` hex form, or the base64 in `secretkey.json`. | -| `access-key` | An access key (`CSAK…`) for the workspace. | - -```bash -mise run go:encrypt:example:explicit -- \ - -secrets-dir /run/secrets \ - -client-id \ - -workspace-crn -``` - -Or build both guests and run it yourself from `bindings/go`: - -```bash -mise run wasm:guest:build wasm:auth-guest:build -cd bindings/go && go run ./encrypt/example/explicit -secrets-dir ... -client-id ... -workspace-crn ... -``` - -Optional flags: `-cts-host` pins the authentication endpoint (default: from -the workspace CRN), and `-require-locked-memory` refuses to run on memory -that cannot be locked in RAM. The ZeroKMS endpoint comes from the token. - -## What it shows - -- **Where secrets come from.** `fileSecrets` reads mounted files. Replace it - with your secrets manager's client; the rest does not change. The client - key goes straight into `NewClientKey`, which takes ownership of the bytes, - and `NewClient` wipes them. -- **Who owns what.** The application opens the credential guest - (`auth.OpenWithoutProfile`) and the access-key strategy, and closes - them after the client. The client asks the strategy for a token on every - request, and the strategy re-exchanges the access key as tokens expire. -- **Locked memory.** With `-require-locked-memory` the credential guest is - opened with `auth.RequireLockedMemory()` and the client with - `WithRequireLockedMemory()`, so both guests stay locked. The client's - printed memory state covers both. -- **A key is for one client.** Passing the same credentials to a second - `NewClient` is refused with `ErrCredentialsConsumed`, before any request. diff --git a/languages/golang/encrypt/example/explicit/main.go b/languages/golang/encrypt/example/explicit/main.go deleted file mode 100644 index 785c3d8e3..000000000 --- a/languages/golang/encrypt/example/explicit/main.go +++ /dev/null @@ -1,170 +0,0 @@ -// Command explicit connects the stack-encrypt Go binding to real ZeroKMS -// with credentials the application supplies itself: the client key and the -// access key come from a secrets source, the client id and workspace from -// configuration, and nothing is read from CS_* variables or the developer -// profile. It is the NewCredentials counterpart of ../example, which uses -// AutoCredentials. -// -// mise run wasm:guest:build wasm:auth-guest:build # both embedded guests -// go run ./encrypt/example/explicit \ -// -secrets-dir /run/secrets \ -// -client-id 6a70bd18-99ac-4650-b104-37eec3a15b09 \ -// -workspace-crn crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY -// -// The secrets directory holds two files, client-key and access-key, the way -// a Kubernetes or Docker secret is mounted. See README.md. -package main - -import ( - "bytes" - "context" - "errors" - "flag" - "fmt" - "os" - "path/filepath" - - "github.com/cipherstash/stack/languages/golang/auth" - "github.com/cipherstash/stack/languages/golang/encrypt" -) - -type user struct { - ID int64 `stash:"-"` - Email string `stash:"label=users/email,index=eq"` -} - -type config struct { - secretsDir string - clientID string - workspaceCRN string - ctsHost string - requireLockedMemory bool -} - -func main() { - var cfg config - flag.StringVar(&cfg.secretsDir, "secrets-dir", "/run/secrets", "directory holding the client-key and access-key secrets") - flag.StringVar(&cfg.clientID, "client-id", "", "the ZeroKMS client id (not a secret)") - flag.StringVar(&cfg.workspaceCRN, "workspace-crn", "", "the workspace the access key belongs to") - flag.StringVar(&cfg.ctsHost, "cts-host", "", "pin the authentication endpoint (default: from the workspace CRN)") - flag.BoolVar(&cfg.requireLockedMemory, "require-locked-memory", false, "refuse to run on memory that cannot be locked in RAM") - flag.Parse() - if cfg.clientID == "" || cfg.workspaceCRN == "" { - fmt.Fprintln(os.Stderr, "usage: explicit -client-id ID -workspace-crn CRN [-secrets-dir DIR]") - os.Exit(2) - } - if err := run(context.Background(), cfg, fileSecrets{dir: cfg.secretsDir}); err != nil { - fmt.Fprintf(os.Stderr, "\nerror: %v\n", err) - os.Exit(1) - } -} - -// secrets is where the application keeps what must not be in its -// configuration. This example reads mounted files; an application would -// call its secrets manager (Vault, AWS Secrets Manager, GCP Secret Manager) -// here instead. The bytes returned are the caller's to wipe or hand on. -type secrets interface { - Get(ctx context.Context, name string) ([]byte, error) -} - -type fileSecrets struct{ dir string } - -func (s fileSecrets) Get(_ context.Context, name string) ([]byte, error) { - b, err := os.ReadFile(filepath.Join(s.dir, name)) //nolint:gosec // example reads the operator's mounted secrets - if err != nil { - return nil, fmt.Errorf("reading secret %q: %w", name, err) - } - // A mounted secret often ends in a newline. Trimming returns a - // sub-slice of the same array, so nothing is copied. - return bytes.TrimSpace(b), nil -} - -func run(ctx context.Context, cfg config, secrets secrets) error { - // The credential guest the token strategy runs in. The caller opens - // it, so the caller chooses its memory policy: under - // -require-locked-memory it is locked from the start and stays locked - // as it grows. NewClient checks it either way, below. - var storeOpts []auth.Option - if cfg.requireLockedMemory { - storeOpts = append(storeOpts, auth.RequireLockedMemory()) - } - // No profile: this application's credentials are all explicit. - store, err := auth.OpenWithoutProfile(ctx, storeOpts...) - if err != nil { - return fmt.Errorf("opening the credential guest: %w", err) - } - // Deferred calls run last-first: the client closes before the strategy - // it asks for tokens, and the strategy before the store it lives in. - defer store.Close() - - accessKey, err := secrets.Get(ctx, "access-key") - if err != nil { - return err - } - var strategyOpts []auth.StrategyOption - if cfg.ctsHost != "" { - strategyOpts = append(strategyOpts, auth.WithAuthBaseURL(cfg.ctsHost)) - } - // The access key crosses as a string, which Go cannot wipe; the bytes - // it was read into can be. - strategy, err := store.AccessKey(ctx, cfg.workspaceCRN, string(accessKey), strategyOpts...) - clear(accessKey) - if err != nil { - return fmt.Errorf("access-key strategy: %w", err) - } - defer strategy.Close() - - keyMaterial, err := secrets.Get(ctx, "client-key") - if err != nil { - return err - } - // NewClientKey takes ownership of the bytes; NewClient wipes them. - creds := encrypt.NewCredentials(cfg.clientID, encrypt.NewClientKey(keyMaterial), strategy) - - opts := []encrypt.ClientOption{encrypt.WithCredentials(creds)} - if cfg.requireLockedMemory { - opts = append(opts, encrypt.WithRequireLockedMemory()) - } - client, err := encrypt.NewClient(ctx, opts...) - if err != nil { - return fmt.Errorf("connecting to ZeroKMS: %w", err) - } - defer client.Close() - // The memory state covers both guests: the crypto guest holding the - // key, and the credential guest the token strategy runs in. - fmt.Printf("connected (%v)\n", client) - - // A key is for one client. These credentials are spent, and a second - // client needs a new key; nothing was sent to find that out. - if _, err := encrypt.NewClient(ctx, encrypt.WithCredentials(creds)); errors.Is(err, encrypt.ErrCredentialsConsumed) { - fmt.Println("reusing the credentials is refused: the key was consumed by the first client") - } else { - return fmt.Errorf("reusing the credentials: got %v, want ErrCredentialsConsumed", err) - } - - cipher := client.DefaultKeyset() - rows := []user{{ID: 1, Email: "alice@example.com"}, {ID: 2, Email: "bob@example.com"}} - records, err := cipher.EncryptRecords(ctx, rows) - if err != nil { - return fmt.Errorf("encrypting records: %w", err) - } - emailCtx, err := encrypt.ParseLabel("users/email") - if err != nil { - return fmt.Errorf("the probe's label: %w", err) - } - probe, err := cipher.Term(ctx, "bob@example.com", emailCtx.Context(), encrypt.Equality) - if err != nil { - return fmt.Errorf("deriving a probe: %w", err) - } - for i, r := range records { - if probe.(encrypt.EqualityTerm).Equal(r["Email"].Equality) { - fmt.Printf("sealed %d rows; the probe for bob@example.com matches row %d\n", len(records), i) - } - } - var back []user - if err := cipher.DecryptRecords(ctx, records, &back); err != nil { - return fmt.Errorf("decrypting records: %w", err) - } - fmt.Printf("opened %v\n", back) - return nil -} diff --git a/languages/golang/encrypt/example/main.go b/languages/golang/encrypt/example/main.go index 1109f167d..2bd2f43bd 100644 --- a/languages/golang/encrypt/example/main.go +++ b/languages/golang/encrypt/example/main.go @@ -1,217 +1,69 @@ -// Command example exercises the stack-encrypt Go binding against real -// ZeroKMS, using the credentials `stash auth login` leaves in the developer -// profile (or the CS_* environment variables, which win). -// -// stash auth login -// mise run wasm:guest:build wasm:auth-guest:build # both embedded guests -// go run ./encrypt/example # from bindings/go -// -// It walks the four things the binding does — seal a value, seal a record -// with its index terms, probe those terms with a query, and open both again -// — and prints what crossed the boundary at each step. package main import ( "context" "fmt" "os" - "sort" "github.com/cipherstash/stack/languages/golang/encrypt" - "github.com/cipherstash/vitaminc/bindings/go/vcvalue" ) -// A record type. The `stash` tag is the Go stand-in for Rust's -// `#[derive(EncryptFrom)]`: `label=` is the field's own encryption -// context, `index=` the terms to derive beside the ciphertext. -type user struct { - ID int64 `stash:"-"` - Email string `stash:"label=users/email,index=eq;match"` - Age uint32 `stash:"label=users/age,index=eq;ore"` -} - func main() { - if err := run(); err != nil { + if err := run(context.Background()); err != nil { fmt.Fprintf(os.Stderr, "\nerror: %v\n", err) os.Exit(1) } } -func run() error { - ctx := context.Background() - // No options: credentials from AutoCredentials, which is the +func run(ctx context.Context) error { + // No options: the credentials come from AutoCredentials, which is the // environment first (CS_CLIENT_ACCESS_KEY + CS_WORKSPACE_CRN, CS_CLIENT_ID - // + CS_CLIENT_KEY), then the developer profile `stash auth login` writes, - // read through auth's credential guest. The token is a refreshing - // device session there, asked on every request, so a long run outlives - // one token. The ZeroKMS endpoint: CS_ZEROKMS_HOST if set, else the token's - // services claim. + // + CS_CLIENT_KEY), then the developer profile `stash auth login` writes. client, err := encrypt.NewClient(ctx) if err != nil { return fmt.Errorf("connecting to ZeroKMS: %w", err) } - // Close runs the guest's own wipe of the client key and every loaded - // index key, and releases the credential guest. The memory's protection - // does not wait on it (see the package docs); this is ordinary resource - // hygiene. defer client.Close() fmt.Printf("connected (%v)\n", client) - cipher := client.DefaultKeyset() - keysetID, err := cipher.KeysetID(ctx) - if err != nil { - return err - } - fmt.Printf("default keyset %s\n\n", keysetID) - - if err := values(ctx, cipher); err != nil { - return err - } - records, err := recordsAndTerms(ctx, cipher) - if err != nil { - return err - } - return ordering(records) -} - -// A whole value, sealed under an AAD of the caller's choosing. The shape of -// the ciphertext mirrors the plaintext, and a field marked Plain rides -// alongside it in the clear. -func values(ctx context.Context, cipher *encrypt.Cipher) error { - section("a value") - - aad := []byte("users/v1") - in := map[string]any{ - "name": "alice", - "age": uint32(34), - "note": vcvalue.Plain{V: "not secret"}, - } - fmt.Printf(" plaintext %v\n", in) - - sealed, err := cipher.Encrypt(ctx, in, aad) - if err != nil { - return fmt.Errorf("encrypting a value: %w", err) - } - for name, node := range sealed.(map[string]any) { - if leaf, ok := node.(encrypt.Sealed); ok { - fmt.Printf(" %-11s %d bytes of ciphertext\n", name, len(leaf)) - } else { - fmt.Printf(" %-11s %v (passthrough — in the clear, and unauthenticated)\n", name, node) - } - } + // One cipher for one tenant: the default keyset, and the tenant as a + // part of every field's context. Every call through it carries both. + cipher := client.DefaultKeyset().Extend("tenant-42") - opened, err := cipher.Decrypt(ctx, sealed, aad) - if err != nil { - return fmt.Errorf("decrypting a value: %w", err) - } - fmt.Printf(" opened %v\n", opened) - - // The AAD is bound into the key as well as the ciphertext, so the wrong - // one does not open the value — it is refused, not silently wrong. - if _, err := cipher.Decrypt(ctx, sealed, []byte("some other context")); err == nil { - return fmt.Errorf("a value opened under an AAD it was not sealed under") - } else { - fmt.Printf(" wrong AAD refused: %v\n", err) - } - return nil -} - -// A record: every field sealed under its own context, with the index terms -// its tag asked for, and all of it from one batched ZeroKMS request. -func recordsAndTerms(ctx context.Context, cipher *encrypt.Cipher) ([]encrypt.EncryptedRecord, error) { - section("records, and the terms that index them") - - users := []user{ + people := []User{ {ID: 1, Email: "alice@example.com", Age: 34}, {ID: 2, Email: "bob@example.com", Age: 29}, - {ID: 3, Email: "carol@example.com", Age: 41}, - } - records, err := cipher.EncryptRecords(ctx, users) - if err != nil { - return nil, fmt.Errorf("encrypting records: %w", err) - } - fmt.Printf(" %d rows sealed in one batched key request\n", len(records)) - for i, r := range records { - fmt.Printf(" row %d Email: %d-byte ciphertext, eq %x…, match %d positions\n", - i, len(r["Email"].Ciphertext.(encrypt.Sealed)), r["Email"].Equality[:6], countPositions(r["Email"].Match)) - fmt.Printf(" Age: %d-byte ciphertext, eq %x…, ore %d bytes\n", - len(r["Age"].Ciphertext.(encrypt.Sealed)), r["Age"].Equality[:6], len(r["Age"].Ore)) } - // A query probe: the same derivation as the stored term, from the value - // being searched for. It never touches the ciphertext — matching is what - // the term is for. - fmt.Println() - emailCtx, err := encrypt.ParseLabel("users/email") + // One call, one ZeroKMS request for both rows. + encrypted, err := Encrypt(ctx, cipher, people) if err != nil { - return nil, fmt.Errorf("the probe's label: %w", err) + return fmt.Errorf("encrypt: %w", err) } - probe, err := cipher.Term(ctx, "bob@example.com", emailCtx.Context(), encrypt.Equality) - if err != nil { - return nil, fmt.Errorf("deriving a probe: %w", err) - } - for i, r := range records { - if probe.(encrypt.EqualityTerm).Equal(r["Email"].Equality) { - fmt.Printf(" probe for bob@example.com matches row %d\n", i) - } + for _, e := range encrypted { + // EncryptedUser prints its passthrough fields and hides the rest. + fmt.Printf("stored: %v (ciphertext %d bytes, equality %d bytes, match %d positions)\n", + e, len(e.Email.Ciphertext), len(e.Email.Equality), len(e.Email.Match)/2) } - // A term is bound to its context. The same value under another field's - // context is a different term, which is what stops a match in one column - // from being a match in another. - nameCtx, err := encrypt.ParseLabel("users/name") + // A query term for one field compares against the stored terms. + term, err := Fields.Email.Equality(ctx, cipher, "bob@example.com") if err != nil { - return nil, fmt.Errorf("the probe's label: %w", err) + return fmt.Errorf("query term: %w", err) } - wrong, err := cipher.Term(ctx, "bob@example.com", nameCtx.Context(), encrypt.Equality) - if err != nil { - return nil, fmt.Errorf("deriving a probe: %w", err) + for _, e := range encrypted { + fmt.Printf("id %d matches bob@example.com: %v\n", e.ID, term.Equal(e.Email.Equality)) } - fmt.Printf(" the same value under users/name matches nothing: %t\n", - !wrong.(encrypt.EqualityTerm).Equal(records[1]["Email"].Equality)) - - var back []user - if err := cipher.DecryptRecords(ctx, records, &back); err != nil { - return nil, fmt.Errorf("decrypting records: %w", err) - } - fmt.Printf("\n opened %v\n", back) - fmt.Printf(" (ID is tagged `-`, so it never crossed the boundary and comes back zero)\n") - return records, nil -} -// ORE terms compare in the plaintext's order without revealing it: sorting -// the rows by their Age term sorts them by age. -func ordering(records []encrypt.EncryptedRecord) error { - section("order, without the values") - - order := []int{0, 1, 2} - sort.Slice(order, func(i, j int) bool { - return records[order[i]]["Age"].Ore.Less(records[order[j]]["Age"].Ore) - }) - fmt.Printf(" rows sorted by their Age ORE terms: %v\n", order) - fmt.Printf(" (ages were 34, 29, 41 — so ascending age is row 1, 0, 2)\n") - return nil -} - -func countPositions(t encrypt.MatchTerm) int { - positions, err := t.Positions() + // Decrypt through the cipher, which refuses another keyset's rows, or + // through the client, which opens each row under the keyset that sealed it. + back, err := Decrypt(ctx, client, encrypted) if err != nil { - return -1 - } - return len(positions) -} - -func section(title string) { - fmt.Printf("── %s %s\n", title, dashes(60-len(title))) -} - -func dashes(n int) string { - if n < 0 { - n = 0 + return fmt.Errorf("decrypt: %w", err) } - out := make([]byte, 0, n*3) - for range n { - out = append(out, "─"...) + // Print ids only: everything else here is plaintext. + for _, u := range back { + fmt.Printf("decrypted id %d\n", u.ID) } - return string(out) + return nil } diff --git a/languages/golang/encrypt/example/model.go b/languages/golang/encrypt/example/model.go new file mode 100644 index 000000000..2982acdba --- /dev/null +++ b/languages/golang/encrypt/example/model.go @@ -0,0 +1,22 @@ +// Command example encrypts a batch of users against real ZeroKMS, using the +// credentials `stash auth login` leaves in the developer profile (or the +// CS_* environment variables, which win), and reads them back. +// +// stash auth login +// mise run go:encrypt:example # builds both guests, then runs +// +// The struct's stash tags are the declaration; user_stash.go beside this +// file is what `go generate` wrote from them, and the program calls only +// the functions in it. +package main + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type User + +// User is one row. Every exported field carries a stash tag, so a new field +// cannot reach the database unencrypted by accident. +type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt,index=equality;match"` + Age uint32 `stash:"age,encrypt,index=equality;ore"` +} diff --git a/languages/golang/encrypt/example/user_stash.go b/languages/golang/encrypt/example/user_stash.go new file mode 100644 index 000000000..f588db10d --- /dev/null +++ b/languages/golang/encrypt/example/user_stash.go @@ -0,0 +1,169 @@ +// Code generated by stashgen. DO NOT EDIT. + +package main + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// User prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedUser struct { + ID int64 + Email EncryptedUserEmail + Age EncryptedUserAge +} + +type EncryptedUserEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} + +type EncryptedUserAge struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Ore encrypt.OreTerm +} + +func (e EncryptedUser) String() string { + return gensupport.Redacted("EncryptedUser", map[string]any{"ID": e.ID}, "Email", "Age") +} + +func (e EncryptedUser) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Email", "Age") +} + +// Stops compiling when User gains, loses, reorders or retypes a field. +var _ = userShape(User{}) + +type userShape struct { + _ struct{} + ID int64 + Email string + Age uint32 +} + +var declaration = gensupport.Declare("users"). + Passthrough("id"). + EncryptIndex("email", gensupport.String, encrypt.Equality, encrypt.Match()). + EncryptIndex("age", gensupport.UInt32, encrypt.Equality, encrypt.Ore) + +var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ + TypeName: "User", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v User) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "email": v.Email, + "age": v.Age, + } + }, + Seal: func(rec gensupport.Record) (EncryptedUser, error) { + var e EncryptedUser + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedUser{}, err + } + e.Email = EncryptedUserEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + Match: rec["email"].Match, + } + e.Age = EncryptedUserAge{ + Ciphertext: rec["age"].Ciphertext, + Equality: rec["age"].Equality, + Ore: rec["age"].Ore, + } + return e, nil + }, + Open: func(e EncryptedUser) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality, Match: e.Email.Match}, + "age": {Ciphertext: e.Age.Ciphertext, Equality: e.Age.Equality, Ore: e.Age.Ore}, + } + }, + Value: func(e EncryptedUser, vals gensupport.Values) (User, error) { + var v User + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return User{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return User{}, err + } + if v.Age, err = gensupport.Get[uint32](vals, "age"); err != nil { + return User{}, err + } + return v, nil + }, +}) + +// Encrypt seals each User in one ZeroKMS request. The result has one element +// for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, main []User) ([]EncryptedUser, error) { + return codec.Encrypt(ctx, cipher, main) +} + +// Decrypt opens each EncryptedUser in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Email EmailField + Age AgeField +}{ + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + Age: AgeField{gensupport.NewField[uint32](declaration, "age")}, +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedUserEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedUserEmail{}, err + } + return EncryptedUserEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil +} + +func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +type AgeField struct { + field gensupport.Field[uint32] +} + +func (f AgeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (EncryptedUserAge, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedUserAge{}, err + } + return EncryptedUserAge{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil +} + +func (f AgeField) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f AgeField) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} diff --git a/languages/golang/encrypt/export_test.go b/languages/golang/encrypt/export_test.go index a7c3a2029..3b2f418ea 100644 --- a/languages/golang/encrypt/export_test.go +++ b/languages/golang/encrypt/export_test.go @@ -2,8 +2,12 @@ package encrypt import ( "context" - "reflect" + "errors" + "fmt" + "net/http" + "testing" + "github.com/cipherstash/stack/languages/golang/internal/guest" "github.com/cipherstash/vitaminc/bindings/go/vcffi" ) @@ -36,18 +40,88 @@ func withZeroKMSURL(url string) ClientOption { return func(o *clientOptions) { o.zerokmsURL = url } } -// GuestPlanInput is the encoded plan object a record call over t sends the -// guest under p, each context extended by ext: what the external tests -// compare byte for byte. Test-only; not part of the package's API. -func GuestPlanInput(p Plan, t reflect.Type, ext ...any) ([]byte, error) { - o := applyOptions([]RecordOption{WithPlan(p), ExtendContext(ext...)}) - bound, err := planFor(t, o) +// The hooks the external tests (package encrypt_test, which can import the +// generated test types this package cannot) reach the internals through. + +// WithZeroKMSURL is withZeroKMSURL for the external tests. +func WithZeroKMSURL(url string) ClientOption { return withZeroKMSURL(url) } + +// Sends is how many ZeroKMS requests the client has made. +func Sends(c *Client) int64 { return c.transport.sends.Load() } + +// ResetSends zeroes the count. +func ResetSends(c *Client) { c.transport.sends.Store(0) } + +// GuestMemory is a copy of the guest's linear memory, for residency scans. +func GuestMemory(t *testing.T, c *Client) []byte { + t.Helper() + mem := c.inst.module.Memory() + view, ok := mem.Read(0, mem.Size()) + if !ok { + t.Fatal("cannot read guest memory") + } + return append([]byte(nil), view...) +} + +// deterministicGuestPath is the deterministic-kms test build, beside the +// real guest; `mise run wasm:guest:build:deterministic` writes it. +const deterministicGuestPath = "wasm/stack_encrypt_guest_deterministic.wasm" + +// ErrDeterministicGuestNotBuilt says the test build is absent. +var ErrDeterministicGuestNotBuilt = errors.New("encrypt: deterministic guest not built; run `mise run wasm:guest:build:deterministic`") + +// NewDeterministicClient is a client over the deterministic-kms test build +// of the guest, seeded: every key derives from the seed and the context, so +// it opens what the Rust record fixture sealed under the same seed and +// needs no ZeroKMS. ErrDeterministicGuestNotBuilt when the build is absent. +func NewDeterministicClient(ctx context.Context, seed [32]byte) (*Client, error) { + wasm, err := guestFS.ReadFile(deterministicGuestPath) + if err != nil { + return nil, ErrDeterministicGuestNotBuilt + } + tr := &transport{rt: refusingTransport{}, token: noToken{}} + inst, err := newInstance(ctx, wasm, tr, guest.BestEffort) if err != nil { return nil, err } - obj, err := planValue(bound, o) + c := newClient(inst, tr) + encoded, err := vcffi.Marshal(seed[:]) if err != nil { + _ = c.Close() return nil, err } - return vcffi.Marshal(obj) + out, err := c.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.cipherInit, buf(encoded)) + }) + if err != nil { + _ = c.Close() + return nil, fmt.Errorf("encrypt: deterministic cipher init: %w", err) + } + if len(out) != len(KeysetID{}) { + _ = c.Close() + return nil, fmt.Errorf("%w: cipher init returned %d bytes for the keyset id", ErrInternal, len(out)) + } + copy(c.def[:], out) + return c, nil +} + +// RawClient is a guest that was never given a client key: every well-formed +// operation is ErrState, every malformed one ErrEncoding, and the checker +// exports work. +func RawClient(t *testing.T, wasm []byte) *Client { + t.Helper() + inst, err := newInstance(context.Background(), wasm, &transport{rt: http.DefaultTransport, token: staticToken("t")}, guest.BestEffort) + if err != nil { + t.Fatal(err) + } + c := newClient(inst, nil) + t.Cleanup(func() { _ = c.Close() }) + return c } + +// EmbeddedGuest is the embedded guest's bytes, or ErrGuestNotBuilt. +func EmbeddedGuest() ([]byte, error) { return embeddedGuest() } + +// LiveClient is liveClient for the external tests: a client against real +// ZeroKMS from the STACK_ENCRYPT_TEST_* variables, or a skip. +func LiveClient(t *testing.T) *Client { return liveClient(t) } diff --git a/languages/golang/encrypt/fixture_test.go b/languages/golang/encrypt/fixture_test.go new file mode 100644 index 000000000..ecb99ee3b --- /dev/null +++ b/languages/golang/encrypt/fixture_test.go @@ -0,0 +1,153 @@ +package encrypt_test + +import ( + "bytes" + "context" + "encoding/hex" + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" +) + +// The record fixture is the proof that generated Go code is a third author +// of one declaration (ADR-0007, amended): the typed Rust chain and the data +// plan lowering each sealed packages/stack-encrypt/tests/fixtures/ +// record_lowering.json under a deterministic key source, and the generated +// testusers package — the same context, identities, kinds and indexes as +// tags — opens both records through the guest built over the same source +// and derives the same term bytes. Skipped when the deterministic test +// build of the guest is absent. + +type recordFixture struct { + KeySource struct { + Kind string `json:"kind"` + Seed string `json:"seed"` + } `json:"key_source"` + KeysetID string `json:"keyset_id"` + Plaintext struct { + Age uint32 `json:"age"` + Email string `json:"email"` + ID uint32 `json:"id"` + Notes string `json:"notes"` + } `json:"plaintext"` + Records map[string]map[string]map[string]json.RawMessage `json:"records"` +} + +func readFixture(t *testing.T) recordFixture { + t.Helper() + raw, err := os.ReadFile(filepath.Join("..", "..", "..", "packages", "stack-encrypt", "tests", "fixtures", "record_lowering.json")) + if err != nil { + t.Fatal(err) + } + var f recordFixture + if err := json.Unmarshal(raw, &f); err != nil { + t.Fatal(err) + } + if f.KeySource.Kind != "deterministic-sha256" || f.KeysetID != "00000000-0000-0000-0000-000000000000" { + t.Fatalf("the fixture's key source changed shape: %+v", f.KeySource) + } + return f +} + +func hexField(t *testing.T, rec map[string]map[string]json.RawMessage, field, output string) []byte { + t.Helper() + raw, ok := rec[field][output] + if !ok { + t.Fatalf("the fixture record has no %s.%s", field, output) + } + var s string + if err := json.Unmarshal(raw, &s); err != nil { + t.Fatal(err) + } + b, err := hex.DecodeString(s) + if err != nil { + t.Fatal(err) + } + return b +} + +func TestGeneratedCodeOpensTheRustRecordFixture(t *testing.T) { + f := readFixture(t) + seedBytes, err := hex.DecodeString(f.KeySource.Seed) + if err != nil || len(seedBytes) != 32 { + t.Fatalf("seed: %v (%d bytes)", err, len(seedBytes)) + } + c, err := encrypt.NewDeterministicClient(context.Background(), [32]byte(seedBytes)) + if err != nil { + t.Skip(err) + } + defer c.Close() + ctx := context.Background() + // The fixture's keyset is the nil UUID: the fake source's default. + cipher := c.DefaultKeyset() + if id, err := cipher.KeysetID(ctx); err != nil || id != (encrypt.KeysetID{}) { + t.Fatalf("default keyset = %v, %v; want the nil id", id, err) + } + + for author, rec := range f.Records { + t.Run(author, func(t *testing.T) { + // The stored record, as a program that read the columns Rust + // wrote would hold it. The passthrough id never crossed the + // binding on either side. + stored := testusers.EncryptedUser{ + ID: int64(f.Plaintext.ID), + Age: testusers.EncryptedUserAge{ + Ciphertext: hexField(t, rec, "age", "c"), + Equality: hexField(t, rec, "age", "eq"), + Ore: hexField(t, rec, "age", "ore"), + }, + Email: testusers.EncryptedUserEmail{ + Ciphertext: hexField(t, rec, "email", "c"), + Equality: hexField(t, rec, "email", "eq"), + Match: hexField(t, rec, "email", "match"), + }, + Notes: testusers.EncryptedUserNotes{Ciphertext: hexField(t, rec, "notes", "c")}, + } + for name, d := range map[string]encrypt.Decrypter{"cipher": cipher, "client": c} { + back, err := testusers.Decrypt(ctx, d, []testusers.EncryptedUser{stored}) + if err != nil { + t.Fatalf("Decrypt through the %s: %v", name, err) + } + want := testusers.User{ID: int64(f.Plaintext.ID), Age: f.Plaintext.Age, Email: f.Plaintext.Email, Notes: f.Plaintext.Notes} + if back[0] != want { + t.Fatalf("Decrypt through the %s = %+v, want %+v", name, back[0], want) + } + } + // The terms Go derives are the bytes Rust stored. + eq, err := testusers.Fields.Age.Equality(ctx, cipher, f.Plaintext.Age) + if err != nil || !eq.Equal(stored.Age.Equality) { + t.Errorf("age equality: %v, equal=%v", err, eq.Equal(stored.Age.Equality)) + } + ore, err := testusers.Fields.Age.Ore(ctx, cipher, f.Plaintext.Age) + if err != nil || !bytes.Equal(ore, stored.Age.Ore) { + t.Errorf("age ore: %v, equal=%v", err, bytes.Equal(ore, stored.Age.Ore)) + } + emailEq, err := testusers.Fields.Email.Equality(ctx, cipher, f.Plaintext.Email) + if err != nil || !emailEq.Equal(stored.Email.Equality) { + t.Errorf("email equality: %v, equal=%v", err, emailEq.Equal(stored.Email.Equality)) + } + match, err := testusers.Fields.Email.Match(ctx, cipher, f.Plaintext.Email) + if err != nil || !bytes.Equal(match, stored.Email.Match) { + t.Errorf("email match: %v, equal=%v", err, bytes.Equal(match, stored.Email.Match)) + } + // A fresh Go record of the same plaintext derives the same terms. + again, err := testusers.Encrypt(ctx, cipher, []testusers.User{{ID: int64(f.Plaintext.ID), Age: f.Plaintext.Age, Email: f.Plaintext.Email, Notes: f.Plaintext.Notes}}) + if err != nil { + t.Fatal(err) + } + if !again[0].Age.Equality.Equal(stored.Age.Equality) || !bytes.Equal(again[0].Email.Match, stored.Email.Match) || !bytes.Equal(again[0].Age.Ore, stored.Age.Ore) { + t.Error("a Go record of the fixture's plaintext derives other terms than Rust did") + } + // A leaf moved under another field's label does not open. + moved := stored + moved.Notes.Ciphertext = stored.Email.Ciphertext + if _, err := testusers.Decrypt(ctx, cipher, []testusers.EncryptedUser{moved}); err == nil { + t.Error("a leaf opened under another field's label") + } + }) + } +} diff --git a/languages/golang/encrypt/gensupport/codec.go b/languages/golang/encrypt/gensupport/codec.go new file mode 100644 index 000000000..5cb90f144 --- /dev/null +++ b/languages/golang/encrypt/gensupport/codec.go @@ -0,0 +1,344 @@ +package gensupport + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "sync" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/internal/record" +) + +// Values is a struct's field values by declared name: what Source gives and +// what Value reads. Passthrough fields are in it too. +type Values map[string]any + +// Output is what one field became: the passthrough value, or the ciphertext +// and each term the field declares. EQL is the EQL value of an encrypt_into +// field, once the engine produces one. +type Output struct { + Value any + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm + Ore encrypt.OreTerm + Ope encrypt.OpeTerm + JSON encrypt.JSONTerm + EQL []byte +} + +// Record is one record's fields by declared name, as Seal reads it and Open +// writes it. +type Record map[string]Output + +// Generated is what a generated file gives the library for one type: the +// declaration, and the four conversions between the plaintext type P, the +// encrypted type E and the data the engine reads and returns. A file writes +// these without reflection: each is a function over named fields. +type Generated[P, E any] struct { + // TypeName is how notices print the type: "User" or "crm.Contact". + TypeName string + // Declaration is the struct's tags as data. + Declaration Declaration + // PrintsPlaintext is true when P has no String and LogValue methods, so + // the running program warns once that P prints its sealed fields. + PrintsPlaintext bool + // Unexported are the unexported fields with no tag, which are neither + // encrypted nor stored; the running program warns once. + Unexported []string + // Source reads every declared field of a value. + Source func(P) Values + // Seal builds the encrypted value from the engine's outputs. + Seal func(Record) (E, error) + // Open reads the engine's inputs from an encrypted value. + Open func(E) Record + // Value builds the plaintext from the opened fields. + Value func(E, Values) (P, error) +} + +// Codec encrypts and decrypts one generated type. +type Codec[P, E any] struct { + g Generated[P, E] + plan *record.Plan + err error + notice sync.Once +} + +// New builds the codec for a generated type. A declaration the engine cannot +// run is reported by the first call, not here: nothing in this package +// panics, and a package-level var cannot return an error. +func New[P, E any](g Generated[P, E]) *Codec[P, E] { + c := &Codec[P, E]{g: g} + c.plan, c.err = g.Declaration.plan() + if c.err == nil && (g.Source == nil || g.Seal == nil || g.Open == nil || g.Value == nil) { + c.err = fmt.Errorf("gensupport: %s: the generated file is incomplete", g.TypeName) + } + return c +} + +func (c *Codec[P, E]) notices() { + c.notice.Do(func() { + NoticeUntagged(c.g.TypeName, c.g.Unexported) + if c.g.PrintsPlaintext { + NoticePrintsPlaintext(c.g.TypeName) + } + }) +} + +// Encrypt seals every value in one request. The result has one element for +// each value, in the same order. +func (c *Codec[P, E]) Encrypt(ctx context.Context, cipher *encrypt.Cipher, values []P) ([]E, error) { + c.notices() + if c.err != nil { + return nil, c.err + } + if cipher == nil { + return nil, fmt.Errorf("gensupport: %s: Encrypt needs a cipher", c.g.TypeName) + } + rows := make([]record.Source, len(values)) + passthrough := make([]map[string]any, len(values)) + for i, v := range values { + vals := c.g.Source(v) + row, keep, err := c.split(vals) + if err != nil { + return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) + } + if c.g.Declaration.opaque { + if row[OpaqueField], err = opaqueBytes(row[OpaqueField]); err != nil { + return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) + } + } + rows[i], passthrough[i] = row, keep + } + sealed, err := cipher.Seal(ctx, c.plan, rows) + if err != nil { + return nil, err + } + out := make([]E, len(values)) + for i, s := range sealed { + rec := make(Record, len(c.g.Declaration.fields)) + for name, v := range passthrough[i] { + rec[name] = Output{Value: v} + } + for name, o := range s { + rec[name] = outputOf(o) + } + if out[i], err = c.g.Seal(rec); err != nil { + return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) + } + } + return out, nil +} + +// Decrypt opens every value in one request. +func (c *Codec[P, E]) Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []E) ([]P, error) { + c.notices() + if c.err != nil { + return nil, c.err + } + if d == nil { + return nil, fmt.Errorf("gensupport: %s: Decrypt needs a Cipher or a Client", c.g.TypeName) + } + records := make([]record.Sealed, len(encrypted)) + passthrough := make([]map[string]any, len(encrypted)) + for i, e := range encrypted { + rec := c.g.Open(e) + records[i] = make(record.Sealed, len(c.plan.Fields)) + passthrough[i] = map[string]any{} + for _, f := range c.g.Declaration.fields { + o, ok := rec[f.name] + switch { + case f.verb == verbOmit: + continue + case !ok: + return nil, fmt.Errorf("gensupport: %s: value %d: Open gave no field %q", c.g.TypeName, i, f.name) + case f.sealed(): + records[i][f.name] = record.Outputs{Ciphertext: o.Ciphertext} + default: + passthrough[i][f.name] = o.Value + } + } + } + sources, err := d.Open(ctx, c.plan, records) + if err != nil { + return nil, err + } + out := make([]P, len(encrypted)) + for i, src := range sources { + vals := make(Values, len(c.g.Declaration.fields)) + for name, v := range passthrough[i] { + vals[name] = v + } + for name, v := range src { + vals[name] = v + } + if c.g.Declaration.opaque { + if vals[OpaqueField], err = opaqueValues(vals[OpaqueField]); err != nil { + return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) + } + } + if out[i], err = c.g.Value(encrypted[i], vals); err != nil { + return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) + } + } + return out, nil +} + +// split sorts a value's fields into what crosses the binding and what stays: +// every declared field must be present and nothing else may be. +func (c *Codec[P, E]) split(vals Values) (record.Source, map[string]any, error) { + row := make(record.Source, len(c.plan.Fields)) + keep := map[string]any{} + for _, f := range c.g.Declaration.fields { + if f.verb == verbOmit { + continue + } + v, ok := vals[f.name] + if !ok { + return nil, nil, fmt.Errorf("the generated Source gave no field %q", f.name) + } + if f.sealed() { + row[f.name] = v + } else { + keep[f.name] = v + } + } + if len(vals) != len(row)+len(keep) { + for name := range vals { + if _, ok := row[name]; ok { + continue + } + if _, ok := keep[name]; ok { + continue + } + return nil, nil, fmt.Errorf("the generated Source gave a field %q the declaration does not name", name) + } + } + return row, keep, nil +} + +func outputOf(o record.Outputs) Output { + out := Output{Ciphertext: o.Ciphertext} + for k, term := range o.Terms { + switch k { + case record.Equality: + out.Equality = term + case record.Match: + out.Match = term + case record.Ore: + out.Ore = term + case record.Ope: + out.Ope = term + } + } + return out +} + +// Passthrough reads a passthrough field from a record, as the Go type the +// struct declares it. +func Passthrough[T any](rec Record, name string) (T, error) { + o, ok := rec[name] + if !ok { + var zero T + return zero, fmt.Errorf("gensupport: no passthrough field %q", name) + } + v, ok := o.Value.(T) + if !ok { + var zero T + return zero, fmt.Errorf("gensupport: passthrough field %q holds a %T, not a %T", name, o.Value, zero) + } + return v, nil +} + +// Get reads one opened field as the Go type the struct declares it. The +// engine returns a value at the field's declared kind; Get converts within +// that kind's family (a uint32 into a uint8 that holds it) and refuses +// anything else, so a value that opens to another type is an error and +// never a silent zero. +func Get[T any](vals Values, name string) (T, error) { + var out T + v, ok := vals[name] + if !ok { + return out, fmt.Errorf("gensupport: the opened value has no field %q", name) + } + if err := convert(v, &out); err != nil { + return out, fmt.Errorf("gensupport: field %q: %w", name, err) + } + return out, nil +} + +// Records wraps a codec with a model's conversions, for separate columns: +// EncryptRows and DecryptRows return and take the model. +func Records[P, E, R any](codec *Codec[P, E], to func(E) R, from func(R) E) *RecordsCodec[P, R] { + return &RecordsCodec[P, R]{ + encrypt: func(ctx context.Context, c *encrypt.Cipher, values []P) ([]R, error) { + es, err := codec.Encrypt(ctx, c, values) + if err != nil { + return nil, err + } + rs := make([]R, len(es)) + for i, e := range es { + rs[i] = to(e) + } + return rs, nil + }, + decrypt: func(ctx context.Context, d encrypt.Decrypter, rows []R) ([]P, error) { + es := make([]E, len(rows)) + for i, r := range rows { + es[i] = from(r) + } + return codec.Decrypt(ctx, d, es) + }, + } +} + +// RecordsCodec encrypts into and decrypts from a model. +type RecordsCodec[P, R any] struct { + encrypt func(context.Context, *encrypt.Cipher, []P) ([]R, error) + decrypt func(context.Context, encrypt.Decrypter, []R) ([]P, error) +} + +// Encrypt seals every value into a model row, in one request. +func (c *RecordsCodec[P, R]) Encrypt(ctx context.Context, cipher *encrypt.Cipher, values []P) ([]R, error) { + return c.encrypt(ctx, cipher, values) +} + +// Decrypt opens every model row, in one request. +func (c *RecordsCodec[P, R]) Decrypt(ctx context.Context, d encrypt.Decrypter, rows []R) ([]P, error) { + return c.decrypt(ctx, d, rows) +} + +// opaqueBytes is an opaque struct's fields as the one value the engine +// seals: a JSON document, so the struct is one column and its fields come +// back as what JSON carries. +func opaqueBytes(fields any) ([]byte, error) { + switch fields.(type) { + case map[string]any, Values: + default: + return nil, fmt.Errorf("the generated Source gave a %T for the opaque value, not a map of its fields", fields) + } + encoded, err := json.Marshal(fields) + if err != nil { + return nil, fmt.Errorf("the opaque value does not encode: %w", err) + } + return encoded, nil +} + +// opaqueValues reads the opened opaque value back into its fields. Numbers +// stay json.Number so Get converts each to the struct's own integer or +// float type without a detour through float64. +func opaqueValues(opened any) (Values, error) { + encoded, ok := opened.([]byte) + if !ok { + return nil, fmt.Errorf("the opaque value opened as %T, not bytes", opened) + } + dec := json.NewDecoder(bytes.NewReader(encoded)) + dec.UseNumber() + var fields map[string]any + if err := dec.Decode(&fields); err != nil { + return nil, fmt.Errorf("the opaque value does not decode: %w", err) + } + return Values(fields), nil +} diff --git a/languages/golang/encrypt/gensupport/convert.go b/languages/golang/encrypt/gensupport/convert.go new file mode 100644 index 000000000..ec6bfce2d --- /dev/null +++ b/languages/golang/encrypt/gensupport/convert.go @@ -0,0 +1,272 @@ +package gensupport + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "math" + "strconv" + + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +// convert writes an opened value into out, a pointer to the Go type the +// struct declares. The engine returns a value at the field's declared kind +// — int32, int64, uint32, uint64, float32, float64, string, []byte, bool, +// or a vcvalue.Object for a composite — and the struct's type is in the +// same family, narrower at most. A value outside the target's range, or of +// another family, is an error. +func convert(v any, out any) error { + // A nil slice or map of the struct went out as JSON null and comes back + // as nil: the zero value it was. + if v == nil { + switch out.(type) { + case *[]byte, *[]any, *[]string, *[]int64, *[]int32, *[]uint32, *[]uint64, *[]float64, *[]bool, *[][]byte, *Values, *map[string]any, *any: + return nil + } + return fmt.Errorf("opened as nothing, and %T holds a value", out) + } + // An opaque struct's fields come back from JSON: a number is a + // json.Number, bytes are a base64 string. Widen them to what the kind + // paths below read. + if n, ok := v.(json.Number); ok { + var err error + if v, err = widenNumber(n, out); err != nil { + return err + } + } + switch out := out.(type) { + case *string: + s, ok := v.(string) + if !ok { + return mismatch(v, *out) + } + *out = s + case *[]byte: + switch b := v.(type) { + case []byte: + *out = append([]byte(nil), b...) + case string: + decoded, err := base64.StdEncoding.DecodeString(b) + if err != nil { + return fmt.Errorf("opened as a string that is not base64 bytes: %w", err) + } + *out = decoded + default: + return mismatch(v, *out) + } + case *bool: + b, ok := v.(bool) + if !ok { + return mismatch(v, *out) + } + *out = b + case *int: + return setInt(v, out, math.MinInt, math.MaxInt) + case *int8: + return setInt(v, out, math.MinInt8, math.MaxInt8) + case *int16: + return setInt(v, out, math.MinInt16, math.MaxInt16) + case *int32: + return setInt(v, out, math.MinInt32, math.MaxInt32) + case *int64: + return setInt(v, out, math.MinInt64, math.MaxInt64) + case *uint: + return setUint(v, out, math.MaxUint) + case *uint8: + return setUint(v, out, math.MaxUint8) + case *uint16: + return setUint(v, out, math.MaxUint16) + case *uint32: + return setUint(v, out, math.MaxUint32) + case *uint64: + return setUint(v, out, math.MaxUint64) + case *float32: + switch f := v.(type) { + case float32: + *out = f + case float64: + if f != 0 && !math.IsInf(f, 0) && !math.IsNaN(f) && (math.Abs(f) > math.MaxFloat32 || math.Abs(f) < math.SmallestNonzeroFloat32) { + return fmt.Errorf("%v does not fit a float32", f) + } + *out = float32(f) + default: + return mismatch(v, *out) + } + case *float64: + switch f := v.(type) { + case float32: + *out = float64(f) + case float64: + *out = f + default: + return mismatch(v, *out) + } + case *Values: + vals, err := valuesOf(v) + if err != nil { + return err + } + *out = vals + case *map[string]any: + vals, err := valuesOf(v) + if err != nil { + return err + } + *out = map[string]any(vals) + case *[]any: + items, ok := v.([]any) + if !ok { + return mismatch(v, *out) + } + *out = append([]any(nil), items...) + case *[]string: + return setSlice(v, out) + case *[]int64: + return setSlice(v, out) + case *[]int32: + return setSlice(v, out) + case *[]uint32: + return setSlice(v, out) + case *[]uint64: + return setSlice(v, out) + case *[]float64: + return setSlice(v, out) + case *[]bool: + return setSlice(v, out) + case *[][]byte: + return setSlice(v, out) + case *any: + *out = v + default: + return fmt.Errorf("the opened value is a %T, which this field's type %T cannot hold", v, out) + } + return nil +} + +func mismatch(v, want any) error { + return fmt.Errorf("opened as %T, not %T", v, want) +} + +// setInt writes an integer of any decoded width into a signed target, within +// its range. +func setInt[T ~int | ~int8 | ~int16 | ~int32 | ~int64](v any, out *T, lo, hi int64) error { + var n int64 + switch i := v.(type) { + case int32: + n = int64(i) + case int64: + n = i + case uint32: + n = int64(i) + case uint64: + if i > math.MaxInt64 { + return fmt.Errorf("%d does not fit a %T", i, *out) + } + n = int64(i) + default: + return mismatch(v, *out) + } + if n < lo || n > hi { + return fmt.Errorf("%d does not fit a %T", n, *out) + } + *out = T(n) + return nil +} + +// setUint writes an integer of any decoded width into an unsigned target, +// within its range. +func setUint[T ~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64](v any, out *T, hi uint64) error { + var n uint64 + switch i := v.(type) { + case uint32: + n = uint64(i) + case uint64: + n = i + case int32: + if i < 0 { + return fmt.Errorf("%d does not fit a %T", i, *out) + } + n = uint64(i) + case int64: + if i < 0 { + return fmt.Errorf("%d does not fit a %T", i, *out) + } + n = uint64(i) + default: + return mismatch(v, *out) + } + if n > hi { + return fmt.Errorf("%d does not fit a %T", n, *out) + } + *out = T(n) + return nil +} + +// setSlice converts each element of an opened array. +func setSlice[E any](v any, out *[]E) error { + items, ok := v.([]any) + if !ok { + return mismatch(v, *out) + } + result := make([]E, len(items)) + for i, item := range items { + if err := convert(item, &result[i]); err != nil { + return fmt.Errorf("element %d: %w", i, err) + } + } + *out = result + return nil +} + +// valuesOf reads an opened composite as Values, nested objects included. +func valuesOf(v any) (Values, error) { + switch obj := v.(type) { + case vcvalue.Object: + vals := make(Values, len(obj)) + for _, f := range obj { + if inner, ok := f.Value.(vcvalue.Object); ok { + nested, err := valuesOf(inner) + if err != nil { + return nil, err + } + vals[f.Key] = nested + continue + } + vals[f.Key] = f.Value + } + return vals, nil + case Values: + return obj, nil + case map[string]any: + return Values(obj), nil + } + return nil, fmt.Errorf("opened as %T, not an object", v) +} + +// widenNumber reads a JSON number as the widest value of the target's +// family: int64 for a signed target, uint64 for an unsigned one, float64 for +// a float. The family conversion then applies its range check. +func widenNumber(n json.Number, out any) (any, error) { + switch out.(type) { + case *int, *int8, *int16, *int32, *int64: + i, err := strconv.ParseInt(string(n), 10, 64) + if err != nil { + return nil, fmt.Errorf("%s is not an integer that fits an int64", n) + } + return i, nil + case *uint, *uint8, *uint16, *uint32, *uint64: + u, err := strconv.ParseUint(string(n), 10, 64) + if err != nil { + return nil, fmt.Errorf("%s is not an integer that fits a uint64", n) + } + return u, nil + case *float32, *float64, *any: + f, err := n.Float64() + if err != nil { + return nil, err + } + return f, nil + } + return n, nil +} diff --git a/languages/golang/encrypt/gensupport/declaration.go b/languages/golang/encrypt/gensupport/declaration.go new file mode 100644 index 000000000..ba835ec62 --- /dev/null +++ b/languages/golang/encrypt/gensupport/declaration.go @@ -0,0 +1,221 @@ +package gensupport + +import ( + "errors" + "fmt" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/internal/record" +) + +// Kind is a field's wire type: the data form of the Rust chain's `::`, +// chosen by stashgen from the field's Go type. A String or UInt32 field +// seals as the typed leaf a Rust record derives; an Untyped field (a struct, +// a slice, a map) seals as one self-describing value. +type Kind string + +// The kinds. int8, int16 and int32 are Int32; int and int64 are Int64; +// uint8, uint16 and uint32 are UInt32; uint and uint64 are UInt64; []byte is +// Bytes. Everything else is Untyped. +const ( + Untyped Kind = "" + Bool Kind = "bool" + Int32 Kind = "int32" + Int64 Kind = "int64" + UInt32 Kind = "uint32" + UInt64 Kind = "uint64" + Float32 Kind = "float32" + Float64 Kind = "float64" + String Kind = "string" + Bytes Kind = "bytes" +) + +// OpaqueField is the one field of an opaque declaration: the whole struct, +// sealed as one value under /value. The struct crosses the binding +// as one JSON document — the engine seals a composite value as a tree of +// leaves, and an opaque struct is one column — so its field types are what +// JSON carries: scalars, []byte (base64), slices and maps of them. +const OpaqueField = "value" + +type verb uint8 + +const ( + verbPassthrough verb = iota + 1 + verbOmit + verbEncrypt + verbEncryptIndex + verbIndex + verbEncryptInto +) + +type field struct { + name string + identity string + kind Kind + verb verb + indexes []encrypt.Index + eqlType string +} + +// Declaration is a generated struct's declaration as data: its context and, +// for each field, what happens to it. Only generated code builds one, from +// the struct's stash tags, and only generated code reads it. A mistake in a +// declaration is found by stashgen; this type still refuses one, since no +// function in this package panics, and the error comes back from the first +// call through the codec. +type Declaration struct { + context []string + opaque bool + fields []field + err error +} + +// Declare starts a declaration with the struct's context: the `context=` +// tag, segments separated by '/'. +func Declare(context string) Declaration { + segments, err := record.ParseContext(context) + if err != nil { + return Declaration{err: fmt.Errorf("gensupport: %v", err)} + } + return Declaration{context: segments} +} + +// DeclareOpaque declares a struct sealed as one value: one field, +// [OpaqueField], encrypted as bytes. +func DeclareOpaque(context string) Declaration { + d := Declare(context) + d.opaque = true + return d.add(field{name: OpaqueField, kind: Bytes, verb: verbEncrypt}) +} + +// Passthrough stores the field as it is. It stays on the host: see the +// record package. +func (d Declaration) Passthrough(name string) Declaration { + return d.add(field{name: name, verb: verbPassthrough}) +} + +// Encrypt seals the field with no index. +func (d Declaration) Encrypt(name string, kind Kind) Declaration { + return d.add(field{name: name, kind: kind, verb: verbEncrypt}) +} + +// EncryptIndex seals the field and derives each index beside it. +func (d Declaration) EncryptIndex(name string, kind Kind, indexes ...encrypt.Index) Declaration { + return d.add(field{name: name, kind: kind, verb: verbEncryptIndex, indexes: indexes}) +} + +// Index derives the indexes alone, with no ciphertext. +func (d Declaration) Index(name string, kind Kind, indexes ...encrypt.Index) Declaration { + return d.add(field{name: name, kind: kind, verb: verbIndex, indexes: indexes}) +} + +// EncryptInto seals the field into one EQL value of the named type. +func (d Declaration) EncryptInto(name string, kind Kind, eqlType string) Declaration { + return d.add(field{name: name, kind: kind, verb: verbEncryptInto, eqlType: eqlType}) +} + +// Omit leaves the field out: it does not cross the binding and is not +// stored. The declaration lists it so a reader sees the choice. +func (d Declaration) Omit(name string) Declaration { + return d.add(field{name: name, verb: verbOmit}) +} + +// Identity gives the named field a context part other than its name: a +// column that was renamed keeps the identity it was first written under, +// so data written before the change still decrypts. +func (d Declaration) Identity(name, identity string) Declaration { + if d.err != nil { + return d + } + for i := range d.fields { + if d.fields[i].name == name { + d.fields[i].identity = identity + return d + } + } + d.err = fmt.Errorf("gensupport: Identity(%q): no such field", name) + return d +} + +func (d Declaration) add(f field) Declaration { + if d.err != nil { + return d + } + if d.opaque && f.name != OpaqueField { + d.err = fmt.Errorf("gensupport: an opaque declaration has one field, not %q", f.name) + return d + } + if f.name == "" { + d.err = errors.New("gensupport: a field has no name") + return d + } + for _, prior := range d.fields { + if prior.name == f.name { + d.err = fmt.Errorf("gensupport: field %q is declared twice", f.name) + return d + } + } + if (f.verb == verbEncryptIndex || f.verb == verbIndex) && len(f.indexes) == 0 { + d.err = fmt.Errorf("gensupport: field %q: an indexed field names at least one index", f.name) + return d + } + d.fields = append(d.fields, f) + return d +} + +// Err is the declaration's mistake, if any. +func (d Declaration) Err() error { return d.err } + +// sealed reports whether a field crosses the binding. +func (f field) sealed() bool { + switch f.verb { + case verbEncrypt, verbEncryptIndex, verbIndex, verbEncryptInto: + return true + } + return false +} + +// plan lowers the declaration to what the engine reads: the sealed fields, +// each with its label, outputs and type. +func (d Declaration) plan() (*record.Plan, error) { + if d.err != nil { + return nil, d.err + } + p := &record.Plan{Context: d.context} + for _, f := range d.fields { + if !f.sealed() { + continue + } + rf := record.Field{Name: f.name, Identity: f.identity, Kind: record.Kind(f.kind)} + switch f.verb { + case verbEncryptInto: + // The next build of the engine lists its EQL types through + // se_targets; until then no declaration can seal into one. + return nil, fmt.Errorf("gensupport: field %q: EQL types are not available yet", f.name) + case verbEncrypt: + rf.Outputs = []record.Output{record.Ciphertext} + case verbEncryptIndex: + rf.Outputs = []record.Output{record.Ciphertext} + } + for _, idx := range f.indexes { + out := idx.Output() + if _, ok := termOutputs[out]; !ok { + return nil, fmt.Errorf("gensupport: field %q: the engine does not derive the %s index yet", f.name, idx) + } + rf.Outputs = append(rf.Outputs, out) + } + p.Fields = append(p.Fields, rf) + } + if len(p.Fields) == 0 { + return nil, errors.New("gensupport: the declaration seals no field") + } + if err := p.Validate(); err != nil { + return nil, fmt.Errorf("gensupport: %v", err) + } + return p, nil +} + +// termOutputs are the indexes the engine derives. +var termOutputs = map[record.Output]struct{}{ + record.Equality: {}, record.Match: {}, record.Ore: {}, record.Ope: {}, +} diff --git a/languages/golang/encrypt/gensupport/field.go b/languages/golang/encrypt/gensupport/field.go new file mode 100644 index 000000000..09a4d1326 --- /dev/null +++ b/languages/golang/encrypt/gensupport/field.go @@ -0,0 +1,104 @@ +package gensupport + +import ( + "context" + "fmt" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/internal/record" +) + +// Field is one sealed field's entry in a generated Fields value: it encrypts +// one value of the field, for an update of one column, and derives the +// terms the field declares, for a query. T is the field's Go type. +type Field[T any] struct { + name string + plan *record.Plan + err error +} + +// NewField makes the entry for one sealed field of a declaration. +func NewField[T any](d Declaration, name string) Field[T] { + f := Field[T]{name: name} + plan, err := d.plan() + if err != nil { + f.err = err + return f + } + rf := plan.Field(name) + if rf == nil { + f.err = fmt.Errorf("gensupport: NewField(%q): the declaration seals no such field", name) + return f + } + // One field, its own plan: the label is the same, so the bytes are the + // same as in a whole record. + f.plan = &record.Plan{Context: plan.Context, Fields: []record.Field{*rf}} + return f +} + +// Encrypt seals one value of the field: its ciphertext and every term it +// declares, in one request. +func (f Field[T]) Encrypt(ctx context.Context, c *encrypt.Cipher, v T) (Output, error) { + if f.err != nil { + return Output{}, f.err + } + if c == nil { + return Output{}, fmt.Errorf("gensupport: %s: Encrypt needs a cipher", f.name) + } + sealed, err := c.Seal(ctx, f.plan, []record.Source{{f.name: v}}) + if err != nil { + return Output{}, err + } + if len(sealed) != 1 { + return Output{}, fmt.Errorf("gensupport: %s: one value came back as %d", f.name, len(sealed)) + } + return outputOf(sealed[0][f.name]), nil +} + +// Query derives the EQL query value for one value of an encrypt_into field. +// No EQL type is available in this build of the engine, so it fails. +func (f Field[T]) Query(context.Context, *encrypt.Cipher, T) (Output, error) { + if f.err != nil { + return Output{}, f.err + } + return Output{}, fmt.Errorf("gensupport: %s: EQL types are not available yet", f.name) +} + +// Equality derives the field's equality term for one value. +func (f Field[T]) Equality(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.EqualityTerm, error) { + return f.term(ctx, c, record.Equality, v) +} + +// Match derives the field's match term for one value. +func (f Field[T]) Match(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.MatchTerm, error) { + return f.term(ctx, c, record.Match, v) +} + +// Ore derives the field's ORE term for one value. +func (f Field[T]) Ore(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.OreTerm, error) { + return f.term(ctx, c, record.Ore, v) +} + +// Ope derives the field's OPE term for one value. +func (f Field[T]) Ope(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.OpeTerm, error) { + return f.term(ctx, c, record.Ope, v) +} + +// JSON derives the field's json index term. The engine does not derive it +// yet, so it fails. +func (f Field[T]) JSON(context.Context, *encrypt.Cipher, T) (encrypt.JSONTerm, error) { + if f.err != nil { + return nil, f.err + } + return nil, fmt.Errorf("gensupport: %s: the engine does not derive the json index yet", f.name) +} + +func (f Field[T]) term(ctx context.Context, c *encrypt.Cipher, output record.Output, v T) ([]byte, error) { + if f.err != nil { + return nil, f.err + } + if c == nil { + return nil, fmt.Errorf("gensupport: %s: a term needs a cipher", f.name) + } + return c.Derive(ctx, f.plan, f.name, output, v) +} diff --git a/languages/golang/encrypt/gensupport/gensupport.go b/languages/golang/encrypt/gensupport/gensupport.go index e50fef63d..3fa87b363 100644 --- a/languages/golang/encrypt/gensupport/gensupport.go +++ b/languages/golang/encrypt/gensupport/gensupport.go @@ -1,9 +1,13 @@ // Package gensupport holds what only code written by stashgen calls. // -// A program never imports this package. The generated file names -// [GeneratedVersion1], prints through [Redacted] and [RedactedLog], and -// reports the two notices that the generator also prints. No function in this -// package panics. +// A program never imports this package: it calls the functions the generated +// file writes into its own package (users.Encrypt, users.Decrypt, +// users.Fields). The generated file names [GeneratedVersion1], builds a +// [Declaration] from the struct's tags, hands the library its conversions in +// a [Generated] value, and prints through [Redacted] and [RedactedLog]. The +// library lowers the declaration to the data plan the engine reads, sends a +// slice of values as one request, and reports the two notices that the +// generator also prints. No function in this package panics. package gensupport import ( diff --git a/languages/golang/encrypt/gensupport/gensupport_internal_test.go b/languages/golang/encrypt/gensupport/gensupport_internal_test.go new file mode 100644 index 000000000..b3d4a45d8 --- /dev/null +++ b/languages/golang/encrypt/gensupport/gensupport_internal_test.go @@ -0,0 +1,261 @@ +package gensupport + +import ( + "encoding/base64" + "encoding/json" + "reflect" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/internal/record" + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +func TestDeclarationLowersToTheEnginesPlan(t *testing.T) { + d := Declare("users"). + Passthrough("id"). + EncryptIndex("age", UInt32, encrypt.Equality, encrypt.Ore). + EncryptIndex("email", String, encrypt.Equality, encrypt.Match()). + Encrypt("notes", String). + Index("score", Int64, encrypt.Ope). + Omit("internal") + plan, err := d.plan() + if err != nil { + t.Fatal(err) + } + want := &record.Plan{Context: []string{"users"}, Fields: []record.Field{ + {Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Ore}}, + {Name: "email", Kind: record.String, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Match}}, + {Name: "notes", Kind: record.String, Outputs: []record.Output{record.Ciphertext}}, + {Name: "score", Kind: record.Int64, Outputs: []record.Output{record.Ope}}, + }} + if !reflect.DeepEqual(plan, want) { + t.Fatalf("plan = %+v", plan) + } + // Passthrough and omitted fields stay on the host; the engine never + // hears of them. + if plan.Field("id") != nil || plan.Field("internal") != nil { + t.Fatal("a passthrough or omitted field reached the plan") + } + // An opaque declaration is one untyped field. + op, err := DeclareOpaque("documents/v2/body").plan() + if err != nil { + t.Fatal(err) + } + if len(op.Fields) != 1 || op.Fields[0].Name != OpaqueField || op.Fields[0].Kind != record.Bytes || op.Descriptor(op.Fields[0]) != "documents/v2/body/value" { + t.Fatalf("opaque plan = %+v", op) + } + // Identity pins the context part. + id, err := Declare("individuals").Encrypt("medicare_number", String).Identity("medicare_number", "medicare_no").plan() + if err != nil { + t.Fatal(err) + } + if id.Descriptor(id.Fields[0]) != "individuals/medicare_no" { + t.Fatalf("identity: %q", id.Descriptor(id.Fields[0])) + } +} + +func TestDeclarationRefusals(t *testing.T) { + cases := map[string]Declaration{ + "empty context": Declare(""), + "context not a label": Declare("users/1x"), + "field twice": Declare("u").Encrypt("a", String).Encrypt("a", String), + "no name": Declare("u").Encrypt("", String), + "indexed with no index": Declare("u").EncryptIndex("a", String), + "opaque with another field": DeclareOpaque("u").Encrypt("a", String), + "identity of no field": Declare("u").Encrypt("a", String).Identity("b", "x"), + "seals nothing": Declare("u").Passthrough("id"), + "EQL not available": Declare("u").EncryptInto("email", String, "TextEq"), + "json index": Declare("u").EncryptIndex("a", String, encrypt.JSON()), + "name not a label": Declare("u").Encrypt("1a", String), + } + for name, d := range cases { + if _, err := d.plan(); err == nil { + t.Errorf("%s: accepted", name) + } + } + if _, err := Declare("u").EncryptInto("email", String, "TextEq").plan(); err == nil || !strings.Contains(err.Error(), "EQL types are not available yet") { + t.Fatalf("encrypt_into: %v", err) + } +} + +func TestConvertStaysWithinAFamily(t *testing.T) { + var u8 uint8 + if err := convert(uint32(7), &u8); err != nil || u8 != 7 { + t.Fatalf("uint32 -> uint8: %v %d", err, u8) + } + if err := convert(uint32(300), &u8); err == nil { + t.Fatal("300 fit a uint8") + } + var i int + if err := convert(int64(-5), &i); err != nil || i != -5 { + t.Fatalf("int64 -> int: %v %d", err, i) + } + if err := convert(uint64(1<<63), &i); err == nil { + t.Fatal("2^63 fit an int") + } + var u uint + if err := convert(int32(-1), &u); err == nil { + t.Fatal("-1 fit a uint") + } + var s string + if err := convert(int32(1), &s); err == nil { + t.Fatal("an integer became a string") + } + var f32 float32 + if err := convert(float64(1.5), &f32); err != nil || f32 != 1.5 { + t.Fatalf("float64 -> float32: %v %v", err, f32) + } + var b []byte + src := []byte{1, 2} + if err := convert(src, &b); err != nil { + t.Fatal(err) + } + src[0] = 9 + if b[0] != 1 { + t.Fatal("convert aliased the decoded slice") + } + var tags []string + if err := convert([]any{"a", "b"}, &tags); err != nil || !reflect.DeepEqual(tags, []string{"a", "b"}) { + t.Fatalf("[]string: %v %v", err, tags) + } + if err := convert([]any{"a", 1}, &tags); err == nil { + t.Fatal("a mixed array became []string") + } + var vals Values + obj := vcvalue.Object{{Key: "title", Value: "x"}, {Key: "inner", Value: vcvalue.Object{{Key: "n", Value: int64(1)}}}} + if err := convert(obj, &vals); err != nil { + t.Fatal(err) + } + if vals["title"] != "x" || vals["inner"].(Values)["n"] != int64(1) { + t.Fatalf("Values = %#v", vals) + } + type unsupported struct{ A int } + var x unsupported + if err := convert(obj, &x); err == nil { + t.Fatal("a struct target was accepted") + } + // An opaque struct's fields come back from JSON. + var n int32 + if err := convert(json.Number("-7"), &n); err != nil || n != -7 { + t.Fatalf("json.Number -> int32: %v %d", err, n) + } + if err := convert(json.Number("3000000000"), &n); err == nil { + t.Fatal("3000000000 fit an int32") + } + var f float64 + if err := convert(json.Number("1.25"), &f); err != nil || f != 1.25 { + t.Fatalf("json.Number -> float64: %v %v", err, f) + } + var raw []byte + if err := convert(base64.StdEncoding.EncodeToString([]byte{1, 2}), &raw); err != nil || !reflect.DeepEqual(raw, []byte{1, 2}) { + t.Fatalf("base64 -> []byte: %v %v", err, raw) + } + fields, err := opaqueValues([]byte(`{"title":"x","n":4,"tags":["a"],"inner":{"k":true}}`)) + if err != nil { + t.Fatal(err) + } + inner, err := Get[Values](fields, "inner") + if err != nil || inner["k"] != true { + t.Fatalf("nested: %v %v", err, inner) + } + if _, err := opaqueBytes("not a map"); err == nil { + t.Fatal("opaqueBytes accepted a string") + } + got, err := Get[uint8](Values{"age": uint32(3)}, "age") + if err != nil || got != 3 { + t.Fatalf("Get = %v %v", got, err) + } + if _, err := Get[uint8](Values{}, "age"); err == nil { + t.Fatal("Get found a missing field") + } + p, err := Passthrough[int64](Record{"id": {Value: int64(4)}}, "id") + if err != nil || p != 4 { + t.Fatalf("Passthrough = %v %v", p, err) + } + if _, err := Passthrough[int64](Record{"id": {Value: "4"}}, "id"); err == nil { + t.Fatal("Passthrough converted a string") + } +} + +type user struct { + ID int64 + Email string + Note string +} + +type encryptedUser struct { + ID int64 + Email Output + Note encrypt.Ciphertext +} + +func userCodec() *Codec[user, encryptedUser] { + return New(Generated[user, encryptedUser]{ + TypeName: "User", + Declaration: Declare("users").Passthrough("id").EncryptIndex("email", String, encrypt.Equality).Encrypt("note", String).Omit("internal"), + Source: func(u user) Values { return Values{"id": u.ID, "email": u.Email, "note": u.Note} }, + Seal: func(rec Record) (encryptedUser, error) { + id, err := Passthrough[int64](rec, "id") + return encryptedUser{ID: id, Email: rec["email"], Note: rec["note"].Ciphertext}, err + }, + Open: func(e encryptedUser) Record { + return Record{"id": {Value: e.ID}, "email": {Ciphertext: e.Email.Ciphertext}, "note": {Ciphertext: e.Note}} + }, + Value: func(e encryptedUser, vals Values) (user, error) { + var u user + var err error + if u.ID, err = Get[int64](vals, "id"); err != nil { + return user{}, err + } + if u.Email, err = Get[string](vals, "email"); err != nil { + return user{}, err + } + if u.Note, err = Get[string](vals, "note"); err != nil { + return user{}, err + } + return u, nil + }, + }) +} + +func TestSplitFailsClosedInBothDirections(t *testing.T) { + c := userCodec() + if c.err != nil { + t.Fatal(c.err) + } + row, keep, err := c.split(Values{"id": int64(1), "email": "a", "note": "b"}) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(row, record.Source{"email": "a", "note": "b"}) || !reflect.DeepEqual(keep, map[string]any{"id": int64(1)}) { + t.Fatalf("split = %v %v", row, keep) + } + if _, _, err := c.split(Values{"id": int64(1), "email": "a"}); err == nil || !strings.Contains(err.Error(), `no field "note"`) { + t.Fatalf("missing field: %v", err) + } + if _, _, err := c.split(Values{"id": int64(1), "email": "a", "note": "b", "stray": 1}); err == nil || !strings.Contains(err.Error(), `"stray"`) { + t.Fatalf("extra field: %v", err) + } +} + +func TestNewReportsAnIncompleteFileAndABadDeclarationOnFirstUse(t *testing.T) { + c := New(Generated[user, encryptedUser]{TypeName: "User", Declaration: Declare("users").Encrypt("a", String)}) + if _, err := c.Encrypt(t.Context(), nil, nil); err == nil || !strings.Contains(err.Error(), "incomplete") { + t.Fatalf("incomplete: %v", err) + } + bad := userCodec() + bad.g.Declaration = Declare("") + bad.plan, bad.err = bad.g.Declaration.plan() + if _, err := bad.Decrypt(t.Context(), nil, nil); err == nil { + t.Fatal("a bad declaration was not reported") + } + // No cipher: a programming error, reported, not a nil dereference. + if _, err := userCodec().Encrypt(t.Context(), nil, []user{{}}); err == nil { + t.Fatal("a nil cipher was accepted") + } + if _, err := userCodec().Decrypt(t.Context(), nil, []encryptedUser{{}}); err == nil { + t.Fatal("a nil decrypter was accepted") + } +} diff --git a/languages/golang/encrypt/guest.go b/languages/golang/encrypt/guest.go index 62ff8f062..59096fb9f 100644 --- a/languages/golang/encrypt/guest.go +++ b/languages/golang/encrypt/guest.go @@ -26,7 +26,7 @@ var guestFS embed.FS const guestPath = "wasm/stack_encrypt_guest.wasm" // ErrGuestNotBuilt is returned by NewClient when no guest module is -// embedded and none was supplied with WithGuest. +// embedded. var ErrGuestNotBuilt = errors.New("encrypt: guest module not built — run `mise run wasm:guest:build`") func embeddedGuest() ([]byte, error) { @@ -61,10 +61,9 @@ type instance struct { exports guest.Exports cipherInit, shutdown, keyset api.Function - encrypt, encryptElement api.Function - decrypt, decryptElement api.Function term api.Function encryptRecord, decryptRecord api.Function + planCheck, targets api.Function } // guestModuleConfig is the module configuration every guest instance runs @@ -143,18 +142,16 @@ func newInstance(ctx context.Context, wasm []byte, t *transport, policy guest.Lo } inst := &instance{runtime: runtime, module: module, mem: mem} exports := map[string]*api.Function{ - "se_alloc": &inst.exports.Alloc, - "se_dealloc": &inst.exports.Dealloc, - "se_cipher_init": &inst.cipherInit, - "se_shutdown": &inst.shutdown, - "se_keyset": &inst.keyset, - "se_encrypt": &inst.encrypt, - "se_encrypt_element": &inst.encryptElement, - "se_decrypt": &inst.decrypt, - "se_decrypt_element": &inst.decryptElement, - "se_term": &inst.term, - "se_encrypt_record": &inst.encryptRecord, - "se_decrypt_record": &inst.decryptRecord, + "se_alloc": &inst.exports.Alloc, + "se_dealloc": &inst.exports.Dealloc, + "se_cipher_init": &inst.cipherInit, + "se_shutdown": &inst.shutdown, + "se_keyset": &inst.keyset, + "se_term": &inst.term, + "se_encrypt_record": &inst.encryptRecord, + "se_decrypt_record": &inst.decryptRecord, + "se_plan_check": &inst.planCheck, + "se_targets": &inst.targets, } for name, slot := range exports { if *slot = module.ExportedFunction(name); *slot == nil { diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index 94aff4e53..ff9ba2331 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -2016,6 +2016,7 @@ dependencies = [ "recipher", "serde", "serde_json", + "sha2 0.10.9", "stack-auth", "stack-encrypt", "stack-guest-abi", diff --git a/languages/golang/encrypt/guest/Cargo.toml b/languages/golang/encrypt/guest/Cargo.toml index e676076a6..8ab7b3117 100644 --- a/languages/golang/encrypt/guest/Cargo.toml +++ b/languages/golang/encrypt/guest/Cargo.toml @@ -22,6 +22,16 @@ publish = false # unit-test natively (`cargo test` here, no wasm toolchain needed). crate-type = ["cdylib", "rlib"] +[features] +# A TEST BUILD of the guest whose key source is the deterministic one the +# record fixture (packages/stack-encrypt/tests/fixtures/record_lowering.json) +# was sealed under: `se_cipher_init` takes the 32-byte seed and nothing +# else, and every data key derives from it. No ZeroKMS, no token, no +# network. The Go tests load it to open the fixture's records and to run +# hermetic round trips; `mise run wasm:guest:build:deterministic` builds it +# beside the real guest. Never the embedded guest. +deterministic-kms = ["stack-kms/test-support", "dep:sha2"] + [dependencies] # The stack crates with default features off: no reqwest, no native TLS — # HTTP comes from the host (see `wasm:wasi-check` in the suite root). @@ -46,6 +56,8 @@ vitaminc-aead-value = "0.5.1" vitaminc-protected = "0.5.1" futures = { version = "0.3", default-features = false, features = ["executor"] } +# Only the deterministic-kms test build derives keys itself. +sha2 = { version = "0.10", optional = true } serde = "1" serde_json = "1" uuid = "1" diff --git a/languages/golang/encrypt/guest/src/abi.rs b/languages/golang/encrypt/guest/src/abi.rs index 74ffeb297..3b5fd1e7d 100644 --- a/languages/golang/encrypt/guest/src/abi.rs +++ b/languages/golang/encrypt/guest/src/abi.rs @@ -6,9 +6,8 @@ //! //! - **Every export handed plaintext wipes that buffer in place before it //! returns**, rather than leaving it for `se_dealloc`: [`se_cipher_init`] -//! (the config carries the client key), and [`se_encrypt`], -//! [`se_encrypt_element`], [`se_term`] and [`se_encrypt_record`] (their -//! value/source buffers). The host's plaintext therefore lives no longer +//! (the config carries the client key), and [`se_term`] and +//! [`se_encrypt_record`] (their value/source buffers). The host's plaintext therefore lives no longer //! than the call, instead of until the host gets round to releasing it. //! **A host must not read a plaintext input buffer back after the call, or //! pass the same buffer to two calls** — it will be zeros. Option, context, @@ -41,14 +40,13 @@ //! / token); calling any other export from inside a host import is //! undefined behaviour of the embedding, not of this module. //! -//! The value exports ([`se_encrypt`] and friends) are the cipher-directed -//! path and take the AAD as `KeysetCipher::encrypt` does: any bytes, none -//! included — a null pointer with zero length is the empty AAD, as a Go -//! `nil` slice is. The record and term exports bind fields, so their -//! contexts must be non-empty (`STATUS_ENCODING` otherwise): each is a -//! [`stack_encrypt::NonEmpty`] from the moment it is parsed, and the sealing -//! and opening sides bind that one value. The asymmetry is the design; see -//! `packages/stack-encrypt/docs/adr/0001-context-optional-cipher-directed-path.md`. +//! There are no whole-value exports: every value crosses as a record under +//! a declaration (ADR-0007, amended). The record and term exports bind +//! fields, so their contexts must be non-empty (`STATUS_ENCODING` +//! otherwise): each is a [`stack_encrypt::NonEmpty`] from the moment it is +//! parsed, and the sealing and opening sides bind that one value. +//! [`se_plan_check`] and [`se_targets`] answer the Go generator's questions +//! and need no cipher. //! //! Every export decodes and validates *all* of its inputs — the operation //! payload, the plan or context, the term kind, the value against the @@ -70,20 +68,25 @@ use futures::executor::block_on; use stack_encrypt::{KeysetCipher, StackCipher}; use stack_guest_abi::abi::{err_status, input, ok_buffer, take_plaintext, wipe_input}; use stack_guest_abi::buffers; -use stack_kms::{ClientOpts, StackKms}; use vitaminc_aead_value::transport as codec; use vitaminc_aead_value::FfiValue; -use crate::config::parse_config; -use crate::host::{HostTokenStrategy, WasiHostConnection}; use crate::ops; use crate::options::{parse_options, parse_selector, scope_for, KeysetSelector, Side}; -use crate::status::{STATUS_ENCODING, STATUS_INTERNAL, STATUS_KMS_TRANSPORT, STATUS_STATE}; +use crate::status::{STATUS_ENCODING, STATUS_INTERNAL, STATUS_STATE}; use stack_encrypt::dynamic::Scope; -/// The instance's cipher: `stack-encrypt` over the host-transport ZeroKMS -/// client with host-supplied tokens. -type GuestCipher = StackCipher>; +/// The key source the instance's cipher runs over: the host-transport +/// ZeroKMS client with host-supplied tokens — or, in the `deterministic-kms` +/// test build, the seeded source the record fixture was sealed under. +#[cfg(not(feature = "deterministic-kms"))] +type GuestKms = + stack_kms::StackKms; +#[cfg(feature = "deterministic-kms")] +type GuestKms = crate::deterministic::DeterministicSource; + +/// The instance's cipher: `stack-encrypt` over [`GuestKms`]. +type GuestCipher = StackCipher; thread_local! { // Wasm is single-threaded, so a thread-local `RefCell` is a plain owner @@ -116,7 +119,7 @@ fn with_cipher(f: impl FnOnce(&GuestCipher) -> Result) -> Result( opts: &[u8], - f: impl FnOnce(&KeysetCipher<'_, StackKms>) -> Result, + f: impl FnOnce(&KeysetCipher<'_, GuestKms>) -> Result, ) -> Result { let options = parse_options(decode(opts)?, Side::Mint)?; with_cipher(|cipher| { @@ -129,7 +132,7 @@ fn with_keyset( /// client for `{"any"}`, one keyset's cipher otherwise. fn with_scope( opts: &[u8], - f: impl FnOnce(Scope<'_, StackKms>) -> Result, + f: impl FnOnce(Scope<'_, GuestKms>) -> Result, ) -> Result { let options = parse_options(decode(opts)?, Side::Open)?; with_cipher(|cipher| { @@ -177,8 +180,13 @@ pub unsafe extern "C" fn se_cipher_init(cfg_ptr: *mut u8, cfg_len: u32) -> u64 { .map_or_else(err_status, ok_buffer) } +#[cfg(not(feature = "deterministic-kms"))] fn cipher_init(decoded: FfiValue) -> Result, u32> { - let config = parse_config(decoded).map_err(|_| STATUS_ENCODING)?; + use crate::host::{HostTokenStrategy, WasiHostConnection}; + use crate::status::STATUS_KMS_TRANSPORT; + use stack_kms::{ClientOpts, StackKms}; + + let config = crate::config::parse_config(decoded).map_err(|_| STATUS_ENCODING)?; if SHUT_DOWN.with(Cell::get) || CIPHER.with(|c| c.borrow().is_some()) { return Err(STATUS_STATE); } @@ -207,7 +215,35 @@ fn cipher_init(decoded: FfiValue) -> Result, u32> { if let Some(size) = config.keyset_cache_size { builder = builder.keyset_cache_size(size); } - let cipher = block_on(builder.init()).map_err(|e| crate::status::status_for_error(&e))?; + install(block_on(builder.init()).map_err(|e| crate::status::status_for_error(&e))?) +} + +/// The `deterministic-kms` build's init: the config buffer is the +/// deterministic source's 32-byte seed and nothing else — no client key, no +/// endpoint, no token, no network. A test artefact, never shipped; see +/// [`crate::deterministic`]. +#[cfg(feature = "deterministic-kms")] +fn cipher_init(decoded: FfiValue) -> Result, u32> { + use vitaminc_protected::Controlled; + + let FfiValue::Bytes(seed) = decoded else { + return Err(STATUS_ENCODING); + }; + let seed: [u8; 32] = seed + .risky_ref() + .as_slice() + .try_into() + .map_err(|_| STATUS_ENCODING)?; + if SHUT_DOWN.with(Cell::get) || CIPHER.with(|c| c.borrow().is_some()) { + return Err(STATUS_STATE); + } + let builder = StackCipher::builder().kms(crate::deterministic::DeterministicSource::new(seed)); + install(block_on(builder.init()).map_err(|e| crate::status::status_for_error(&e))?) +} + +/// Install the built cipher as the instance's and return its default +/// keyset's id. +fn install(cipher: GuestCipher) -> Result, u32> { let default = cipher.default_keyset().keyset_id().as_bytes().to_vec(); CIPHER.with(|c| *c.borrow_mut() = Some(cipher)); Ok(default) @@ -247,7 +283,7 @@ pub extern "C" fn se_shutdown() { /// /// # Safety /// -/// As for [`se_encrypt`]. +/// As for [`se_term`]. #[no_mangle] pub unsafe extern "C" fn se_keyset(sel_ptr: *const u8, sel_len: u32) -> u64 { catch_unwind(AssertUnwindSafe(|| { @@ -266,153 +302,6 @@ pub unsafe extern "C" fn se_keyset(sel_ptr: *const u8, sel_len: u32) -> u64 { .map_or_else(err_status, ok_buffer) } -/// Encrypt an FFI-codec-encoded value tree under the keyset `opts` selects, -/// binding `aad`; every leaf is sealed from one batched key request, -/// dispatched as one `generate-data-key` call per 500 keyed leaves (see -/// `cipher_init` for where that bound comes from). Output: packed pointer -/// to a codec-encoded ciphertext tree whose leaves are the frozen -/// `SealedValue` byte encoding, each carrying the keyset's id. -/// -/// `aad` may be empty (a null pointer with zero length is empty) — see this -/// module's hostile-input notes. `opts` is the options object -/// (`{"keyset": }`, [`crate::options`]); `{"any"}` is refused here. -/// -/// # Safety -/// -/// Pointer/length pairs should name buffers the host wrote via -/// `se_alloc`; each range is bounds-checked against linear memory (a bad -/// pair returns `STATUS_ENCODING` instead of faulting). -#[no_mangle] -pub unsafe extern "C" fn se_encrypt( - val_ptr: *mut u8, - val_len: u32, - aad_ptr: *const u8, - aad_len: u32, - opt_ptr: *const u8, - opt_len: u32, -) -> u64 { - run_encrypt(val_ptr, val_len, aad_ptr, aad_len, opt_ptr, opt_len, false) -} - -/// Like [`se_encrypt`], but seals the value as a *sequence element* — rows -/// written through this export interchange with rows written by encrypting -/// a whole sequence under the same AAD. -/// -/// # Safety -/// -/// As for [`se_encrypt`]. -#[no_mangle] -pub unsafe extern "C" fn se_encrypt_element( - val_ptr: *mut u8, - val_len: u32, - aad_ptr: *const u8, - aad_len: u32, - opt_ptr: *const u8, - opt_len: u32, -) -> u64 { - run_encrypt(val_ptr, val_len, aad_ptr, aad_len, opt_ptr, opt_len, true) -} - -/// Decrypt a codec-encoded ciphertext tree back into a codec-encoded value -/// tree; one batched key request per keyset the leaves were sealed under, -/// dispatched as one `retrieve-data-key` call per 500 keyed leaves. The -/// output buffer contains **plaintext** — the host must copy it out and -/// immediately release it with `se_dealloc` (which wipes it). -/// -/// `aad` must be the one the ciphertext was sealed under, empty included. -/// `opts` constrains which keyset may be opened: `{"any"}` opens leaves from -/// whichever keyset each was sealed under; `{"name"}`, `{"id"}` and -/// `{"default"}` refuse a leaf from any other keyset as -/// `STATUS_FOREIGN_KEYSET`, before any key is retrieved. -/// -/// # Safety -/// -/// As for [`se_encrypt`]. -#[no_mangle] -pub unsafe extern "C" fn se_decrypt( - ct_ptr: *const u8, - ct_len: u32, - aad_ptr: *const u8, - aad_len: u32, - opt_ptr: *const u8, - opt_len: u32, -) -> u64 { - run_decrypt(ct_ptr, ct_len, aad_ptr, aad_len, opt_ptr, opt_len, false) -} - -/// Like [`se_decrypt`], but opens the ciphertext as a *sequence element* — -/// the read-side counterpart of [`se_encrypt_element`], for one row of a -/// batch-encrypted sequence under the batch's AAD. -/// -/// # Safety -/// -/// As for [`se_encrypt`]. -#[no_mangle] -pub unsafe extern "C" fn se_decrypt_element( - ct_ptr: *const u8, - ct_len: u32, - aad_ptr: *const u8, - aad_len: u32, - opt_ptr: *const u8, - opt_len: u32, -) -> u64 { - run_decrypt(ct_ptr, ct_len, aad_ptr, aad_len, opt_ptr, opt_len, true) -} - -/// [`se_encrypt`] / [`se_encrypt_element`]'s shared drive: validate, select -/// the keyset, block on the op. -fn run_encrypt( - val_ptr: *mut u8, - val_len: u32, - aad_ptr: *const u8, - aad_len: u32, - opt_ptr: *const u8, - opt_len: u32, - as_element: bool, -) -> u64 { - catch_unwind(AssertUnwindSafe(|| { - // Plaintext first: the wipe writes through `&mut`, so nothing else - // may be borrowed from linear memory yet. - let value = unsafe { take_plaintext(val_ptr, val_len)? }; - let value = value.as_slice(); - // SAFETY: host-owned ranges the export was handed; the borrows end - // before it returns and before any wipe of an overlapping range. - let aad = unsafe { input(aad_ptr, aad_len)? }; - let opts = unsafe { input(opt_ptr, opt_len)? }; - ops::validate::value(value)?; - with_keyset(opts, |keyset| { - block_on(ops::encrypt_value(keyset, value, aad, as_element)) - }) - })) - .unwrap_or(Err(STATUS_INTERNAL)) - .map_or_else(err_status, ok_buffer) -} - -/// [`se_decrypt`] / [`se_decrypt_element`]'s shared drive. -fn run_decrypt( - ct_ptr: *const u8, - ct_len: u32, - aad_ptr: *const u8, - aad_len: u32, - opt_ptr: *const u8, - opt_len: u32, - as_element: bool, -) -> u64 { - catch_unwind(AssertUnwindSafe(|| { - // SAFETY: host-owned ranges the export was handed; the borrows end - // before it returns and before any wipe of an overlapping range. - let ciphertext = unsafe { input(ct_ptr, ct_len)? }; - let aad = unsafe { input(aad_ptr, aad_len)? }; - let opts = unsafe { input(opt_ptr, opt_len)? }; - ops::validate::tree(ciphertext)?; - with_scope(opts, |scope| { - block_on(ops::decrypt_value(scope, ciphertext, aad, as_element)) - }) - })) - .unwrap_or(Err(STATUS_INTERNAL)) - .map_or_else(err_status, ok_buffer) -} - /// Derive one index term under the keyset `opts` selects: a codec-encoded /// scalar, a codec-encoded context and a term kind ([`ops::TERM_EQUALITY`] /// etc.); the output is the term's frozen byte encoding. Under the local @@ -432,7 +321,9 @@ fn run_decrypt( /// /// # Safety /// -/// As for [`se_encrypt`]. +/// Pointer/length pairs should name buffers the host wrote via +/// `se_alloc`; each range is bounds-checked against linear memory (a bad +/// pair returns `STATUS_ENCODING` instead of faulting). #[no_mangle] pub unsafe extern "C" fn se_term( val_ptr: *mut u8, @@ -468,7 +359,7 @@ pub unsafe extern "C" fn se_term( /// /// # Safety /// -/// As for [`se_encrypt`]. +/// As for [`se_term`]. #[no_mangle] pub unsafe extern "C" fn se_encrypt_record( src_ptr: *mut u8, @@ -498,12 +389,15 @@ pub unsafe extern "C" fn se_encrypt_record( /// the same plan; only the `"c"` outputs participate. One batched key /// request per keyset the leaves were sealed under, dispatched as one /// `retrieve-data-key` call per 500 keyed leaves. `opts` constrains the -/// keyset as for [`se_decrypt`]. The output buffer contains **plaintext** — -/// same host obligations as [`se_decrypt`]. +/// keyset: `{"any"}` opens leaves from whichever keyset each was sealed +/// under; `{"name"}`, `{"id"}` and `{"default"}` refuse a leaf from any +/// other keyset as `STATUS_FOREIGN_KEYSET`, before any key is retrieved. +/// The output buffer contains **plaintext**: the host copies it out and +/// immediately `se_dealloc`s it (which wipes it). /// /// # Safety /// -/// As for [`se_encrypt`]. +/// As for [`se_term`]. #[no_mangle] pub unsafe extern "C" fn se_decrypt_record( rec_ptr: *const u8, @@ -527,3 +421,33 @@ pub unsafe extern "C" fn se_decrypt_record( .unwrap_or(Err(STATUS_INTERNAL)) .map_or_else(err_status, ok_buffer) } + +/// Check a codec-encoded plan without a cipher: `STATUS_ENCODING` for a +/// declaration the engine would refuse ([`ops::plan_check`]), an empty +/// output buffer for one it accepts. Works in every instance state, before +/// [`se_cipher_init`] and after [`se_shutdown`] alike: the generator that +/// asks has no credentials and makes no request. +/// +/// # Safety +/// +/// As for [`se_term`]. +#[no_mangle] +pub unsafe extern "C" fn se_plan_check(plan_ptr: *const u8, plan_len: u32) -> u64 { + catch_unwind(AssertUnwindSafe(|| { + // SAFETY: a host-owned range the export was handed; the borrow ends + // before it returns. + let plan = unsafe { input(plan_ptr, plan_len)? }; + ops::plan_check(plan).map(|()| Vec::new()) + })) + .unwrap_or(Err(STATUS_INTERNAL)) + .map_or_else(err_status, ok_buffer) +} + +/// The EQL types this build produces, as a codec-encoded +/// `{"targets": [...]}` ([`ops::targets`]). Needs no cipher. +#[no_mangle] +pub extern "C" fn se_targets() -> u64 { + catch_unwind(AssertUnwindSafe(ops::targets)) + .unwrap_or(Err(STATUS_INTERNAL)) + .map_or_else(err_status, ok_buffer) +} diff --git a/languages/golang/encrypt/guest/src/deterministic.rs b/languages/golang/encrypt/guest/src/deterministic.rs new file mode 100644 index 000000000..d087dd088 --- /dev/null +++ b/languages/golang/encrypt/guest/src/deterministic.rs @@ -0,0 +1,116 @@ +//! The deterministic key source of the `deterministic-kms` test build: a +//! copy of `DeterministicSource` in stack-encrypt's `tests/common/mod.rs`, +//! which sealed the record fixture (`tests/fixtures/record_lowering.json`). +//! +//! Every data key is `SHA-256(seed ‖ "key" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, its +//! tag `SHA-256(seed ‖ "tag" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, and the IV +//! `SHA-256(seed ‖ "iv" ‖ 0 ‖ descriptor ‖ 0 ‖ counter)[..16]`, where +//! `descriptor` is the context the leaf is sealed under as ZeroKMS renders +//! it. A leaf opens under its own field's descriptor and no other, as under +//! ZeroKMS. The index key is `FakeDataKeySource`'s, the same every test in +//! the repository derives terms under. So a Go test that loads this build +//! with the fixture's seed opens the records Rust sealed and derives the +//! same term bytes. +//! +//! It is a test double. The feature that compiles it is off by default, the +//! Go package never embeds this build, and the fixture README beside the +//! JSON file is the one definition both copies follow. + +use std::borrow::Cow; +use std::sync::atomic::{AtomicU64, Ordering}; + +use sha2::{Digest, Sha256}; +use stack_kms::{ + DataKey, DataKeySource, DataKeyWithTag, FakeDataKeySource, GenerateKeyPayload, IdentifiedBy, + IndexKey, IndexKeySource, RetrieveKeyPayload, UnverifiedContext, +}; +use uuid::Uuid; + +/// The seeded source. See the [module docs](self). +pub struct DeterministicSource { + seed: [u8; 32], + counter: AtomicU64, + index: FakeDataKeySource, +} + +impl DeterministicSource { + /// A source over `seed`. + pub fn new(seed: [u8; 32]) -> Self { + Self { + seed, + counter: AtomicU64::new(0), + index: FakeDataKeySource::new(), + } + } + + fn derive(&self, what: &str, descriptor: &str, salt: &[u8]) -> [u8; 32] { + let mut hasher = Sha256::new(); + hasher.update(self.seed); + hasher.update(what.as_bytes()); + hasher.update([0u8]); + hasher.update(descriptor.as_bytes()); + hasher.update([0u8]); + hasher.update(salt); + hasher.finalize().into() + } +} + +impl DataKeySource for DeterministicSource { + async fn generate_keys( + &self, + payloads: Vec>, + _keyset_id: Option, + _unverified_context: Option>, + ) -> Result, stack_kms::Error> { + Ok(payloads + .into_iter() + .map(|payload| { + let n = self.counter.fetch_add(1, Ordering::SeqCst); + let iv_bytes = self.derive("iv", payload.descriptor, &n.to_le_bytes()); + let mut iv = stack_kms::Iv::default(); + let width = iv.len(); + iv.copy_from_slice(&iv_bytes[..width]); + let key = self.derive("key", payload.descriptor, &iv); + let tag = self.derive("tag", payload.descriptor, &iv); + DataKeyWithTag { + key: DataKey { iv, key }, + tag: tag.to_vec(), + decryption_policy: payload.decryption_policy, + } + }) + .collect()) + } + + async fn retrieve_keys( + &self, + payloads: Vec>, + _keyset_id: Option, + _unverified_context: Option<&UnverifiedContext>, + ) -> Result, stack_kms::Error> { + payloads + .iter() + .map(|payload| { + let iv: stack_kms::Iv = *payload.iv.as_ref(); + let tag = self.derive("tag", payload.descriptor, &iv); + if tag[..] != *payload.tag { + return Err(stack_kms::Error::RetrieveKey( + stack_kms::RetrieveKeyError::FailedRetrieval( + "the tag is not this descriptor's".to_string(), + ), + )); + } + let key = self.derive("key", payload.descriptor, &iv); + Ok(DataKey { iv, key }) + }) + .collect() + } +} + +impl IndexKeySource for DeterministicSource { + async fn load_index_key( + &self, + keyset_id: Option, + ) -> Result<(Uuid, IndexKey), stack_kms::Error> { + self.index.load_index_key(keyset_id).await + } +} diff --git a/languages/golang/encrypt/guest/src/lib.rs b/languages/golang/encrypt/guest/src/lib.rs index 93f6aa365..711473d24 100644 --- a/languages/golang/encrypt/guest/src/lib.rs +++ b/languages/golang/encrypt/guest/src/lib.rs @@ -44,6 +44,12 @@ //! would cross TLS anyway. The client key enters guest memory once at //! `se_cipher_init`; derived data keys and index keys never leave. //! +//! The exports are the record path (`se_encrypt_record`, `se_decrypt_record`), +//! per-field term derivation (`se_term`), keyset resolution (`se_keyset`), +//! the generator's two questions (`se_plan_check`, `se_targets`) and the +//! lifetime pair (`se_cipher_init`, `se_shutdown`). There is no whole-value +//! export: the Go SDK seals every value under a declaration (ADR-0007). +//! //! One instance is one client: `se_cipher_init` runs once per instance and //! the keysets that client uses are selected per call through the options //! object ([`options`]), loaded on first use. There is no cipher handle, @@ -72,6 +78,8 @@ //! out of a tree is exactly what a database column holds. pub mod config; +#[cfg(feature = "deterministic-kms")] +pub mod deterministic; pub mod headers; pub mod ops; pub mod options; diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index e0634a6f3..f240dd5ec 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -19,6 +19,10 @@ //! are attacker-reachable decode/decrypt paths, and the status codes leak //! only the failure class (see `status.rs`). //! +//! There is no whole-value encrypt or decrypt here. Every value the Go SDK +//! seals goes through a declaration (ADR-0007, amended): an opaque struct +//! is a one-field record, so the record path is the one path. +//! //! # What is here, and what is not //! //! The operations themselves live in [`stack_encrypt::dynamic`]: reading a @@ -34,9 +38,7 @@ use stack_encrypt::dynamic::{self, Scalar, Scope}; use stack_encrypt::sem::MatchOptions; use stack_encrypt::target::IndexSpec; -use stack_encrypt::{ - BoxedPassthrough, CipherText, Element, Encrypt, KeysetCipher, SealedValue, StackCipherText, -}; +use stack_encrypt::{BoxedPassthrough, CipherText, KeysetCipher, SealedValue, StackCipherText}; use stack_kms::DataKeySource; use vitaminc_aead_value::{transport as codec, FfiValue}; use vitaminc_protected::Controlled; @@ -57,90 +59,6 @@ pub const TERM_OPE: u32 = 4; /// encoding — the shape that crosses the FFI codec. type BytesTree = CipherText, BoxedPassthrough>; -// ============================================================================= -// Whole-value encrypt / decrypt (the vitaminc guest's vc_encrypt shape) -// ============================================================================= - -/// Encrypt a codec-encoded [`FfiValue`] tree under `aad`, sealing every leaf -/// against a fresh ZeroKMS data key (one batched request; see the module -/// docs for how a batch is chunked). With `as_element`, -/// seal it as a *sequence element* — interchangeable with rows written by -/// encrypting a whole sequence under the same AAD. -/// -/// This is the cipher-directed path, and it takes the AAD as `StackCipher` -/// does: any bytes, including none. An empty `aad` seals under no context — -/// the plain AEAD use `Aes256Cipher` allows, opened symmetrically by -/// [`decrypt_value`] — and is the Go caller's choice to make. The record and -/// term paths ([`encrypt_record`], [`decrypt_record`], [`term`]) are the -/// ones that bind fields: each takes a [`NonEmpty`](stack_encrypt::NonEmpty) context, proven once at -/// the boundary when the plan or the term's context is parsed, and refused -/// as [`STATUS_ENCODING`] when empty. -pub async fn encrypt_value( - cipher: &KeysetCipher<'_, K>, - value: &[u8], - aad: &[u8], - as_element: bool, -) -> Result, u32> -where - K: DataKeySource + Sync, -{ - let value = decode_value(value)?; - let tree = if as_element { - Element(value).encrypt_with_aad(cipher, aad) - } else { - value.encrypt_with_aad(cipher, aad) - } - .map_err(|_| STATUS_INTERNAL)?; - let ct = tree - .seal(cipher, aad) - .await - .map_err(|e| status_for_error(&e))?; - encode_tree(ct) -} - -/// Decrypt a codec-encoded ciphertext tree back into a codec-encoded -/// [`FfiValue`] tree. The output buffer contains plaintext — the ABI -/// layer's ownership rules govern its wiping. -/// -/// One batched `retrieve_keys` per invocation, dispatched as one ZeroKMS -/// call per 500 keyed leaves and, under [`Scope::Client`], per keyset the -/// tree's leaves were sealed under — the same rule [`decrypt_record`] -/// states. A tree small enough and single-keyset enough is the one request -/// that suggests; nothing here promises it in general. -/// -/// Symmetric with [`encrypt_value`]: the AAD is whatever the value was sealed -/// under, empty included. The [`Scope`] says which keysets may be opened: -/// any, or one, refusing the rest before any key is retrieved. -pub async fn decrypt_value( - scope: Scope<'_, K>, - ciphertext: &[u8], - aad: &[u8], - as_element: bool, -) -> Result, u32> -where - K: DataKeySource + Sync, -{ - let tree = decode_tree(ciphertext)?; - // One `decrypt` per arm, not one `decipher` and two drives. The element - // derivation is `Element`'s to apply and naming the type is what asks - // for it; the scope decides whether a foreign leaf is refused before any - // key is retrieved. Only one arm runs, so the retrieve happens once. - let value: FfiValue = match (&scope, as_element) { - (Scope::Client(cipher), true) => cipher - .decrypt::, _>(tree, aad) - .await - .map(Element::into_inner), - (Scope::Client(cipher), false) => cipher.decrypt(tree, aad).await, - (Scope::Keyset(keyset), true) => keyset - .decrypt::, _>(tree, aad) - .await - .map(Element::into_inner), - (Scope::Keyset(keyset), false) => keyset.decrypt(tree, aad).await, - } - .map_err(|e| status_for_error(&e))?; - encode_value(value) -} - // ============================================================================= // Terms // ============================================================================= @@ -268,6 +186,35 @@ where encode_value(value) } +// ============================================================================= +// The generator's questions +// ============================================================================= + +/// Check a codec-encoded plan without a cipher: everything +/// [`dynamic::record::plan`] refuses — a malformed object, an unknown output, +/// a type that does not admit one of its indexes, a context that is not a +/// label of two or more segments, two fields under one identity — is +/// [`STATUS_ENCODING`]. The Go generator (`stashgen`) asks this for every +/// declaration it writes, so it holds no copy of the engine's rules; it +/// asks one field at a time to name the field that failed. +pub fn plan_check(plan: &[u8]) -> Result<(), u32> { + dynamic::record::plan(decode_value(plan)?) + .map(drop) + .map_err(|e| status_for_dynamic(&e)) +} + +/// The EQL types this build of the engine produces, as a codec-encoded +/// `{"targets": [...]}`. Empty until the EQL target dispatch lands: a +/// generator reads an empty list as "no `encrypt_into` type is available +/// yet" and refuses the tag. The shape is fixed here so the next build adds +/// entries to the list rather than a second export. +pub fn targets() -> Result, u32> { + encode_value(FfiValue::Object(vec![( + "targets".to_string(), + FfiValue::Array(Vec::new()), + )])) +} + // ============================================================================= // Boundary validation // ============================================================================= @@ -282,17 +229,6 @@ where pub mod validate { use super::*; - /// A codec-encoded value tree decodes. - pub fn value(bytes: &[u8]) -> Result<(), u32> { - decode_value(bytes).map(drop) - } - - /// A codec-encoded ciphertext tree decodes and its leaves are - /// well-formed `SealedValue` encodings. - pub fn tree(bytes: &[u8]) -> Result<(), u32> { - decode_tree(bytes).map(drop) - } - /// A term's inputs, as [`term`] takes them: the context decodes and is /// non-empty, the kind is one of [`TERM_EQUALITY`] .. [`TERM_OPE`], and /// the value is a scalar the scheme defines that term for diff --git a/languages/golang/encrypt/guest/src/options.rs b/languages/golang/encrypt/guest/src/options.rs index 3fc9b6727..b52fd55ea 100644 --- a/languages/golang/encrypt/guest/src/options.rs +++ b/languages/golang/encrypt/guest/src/options.rs @@ -55,9 +55,9 @@ pub enum KeysetSelector { /// there. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Side { - /// `se_encrypt`, `se_encrypt_element`, `se_encrypt_record`, `se_term`. + /// `se_encrypt_record`, `se_term`. Mint, - /// `se_decrypt`, `se_decrypt_element`, `se_decrypt_record`. + /// `se_decrypt_record`. Open, } diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index 86561438e..1add14e71 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -205,239 +205,6 @@ fn row(age: u32, name: &str) -> FfiValue { // Values // ============================================================================= -#[test] -fn value_round_trips_through_the_guest_ops() { - let cipher = cipher(); - let value = obj(vec![ - ("email", s("alice@example.com")), - ("age", FfiValue::UInt32(34)), - ("id", FfiValue::Passthrough(Box::new(FfiValue::Int64(7)))), - ]); - - let ct = block_on(ops::encrypt_value( - &cipher.default_keyset(), - &encode(value), - b"users/42", - false, - )) - .expect("encrypt"); - let pt = block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"users/42", - false, - )) - .expect("decrypt"); - - let FfiValue::Object(entries) = decode(&pt) else { - panic!("expected an object back"); - }; - assert_eq!(entries.len(), 3); - assert_eq!(text(&entries[0].1), "alice@example.com"); - assert!(matches!(entries[1].1, FfiValue::UInt32(34))); - assert!( - matches!(&entries[2].1, FfiValue::Passthrough(inner) if matches!(**inner, FfiValue::Int64(7))) - ); - - assert_eq!(cipher.kms().generate_calls.load(Ordering::SeqCst), 1); - assert_eq!(cipher.kms().retrieve_calls.load(Ordering::SeqCst), 1); -} - -#[test] -fn element_mode_round_trips() { - let cipher = cipher(); - let ct = block_on(ops::encrypt_value( - &cipher.default_keyset(), - &encode(s("row-0")), - b"users", - true, - )) - .expect("encrypt element"); - let pt = block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"users", - true, - )) - .expect("decrypt element"); - assert_eq!(text(&decode(&pt)), "row-0"); - - // An element is not a plain value: opening it without the element - // derivation must fail authentication. - assert_eq!( - block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"users", - false - )), - Err(STATUS_AUTH) - ); -} - -#[test] -fn guest_leaves_are_the_frozen_storage_encoding() { - // A leaf lifted out of the guest's codec framing is exactly the - // `SealedValue::from_bytes` storage format — a native cipher opens it. - let cipher = cipher(); - let ct = block_on(ops::encrypt_value( - &cipher.default_keyset(), - &encode(s("durable")), - b"ctx", - false, - )) - .expect("encrypt"); - - let CipherText::Single(leaf_bytes) = decode_tree(&ct) else { - panic!("expected a single leaf"); - }; - let leaf = SealedValue::from_bytes(&leaf_bytes).expect("frozen leaf encoding"); - // A guest leaf seals the *value model's* typed payload (`[tag] ++ - // payload`, the vitaminc sealed-leaf format), so the native open goes - // through `FfiValue`'s own `Decrypt` — not a bare `String`. The host's - // AAD is bytes, and a byte context is not a text context (vitaminc 0.5 - // types its leaves), so the native side opens under the byte slice. - let value: FfiValue = block_on(cipher.decrypt(CipherText::Single(leaf), b"ctx".as_slice())) - .expect("native decrypt of a guest leaf"); - assert_eq!(text(&value), "durable"); -} - -#[test] -fn wrong_aad_and_malformed_inputs_map_to_statuses() { - let cipher = cipher(); - let ct = block_on(ops::encrypt_value( - &cipher.default_keyset(), - &encode(s("x")), - b"ctx", - false, - )) - .expect("encrypt"); - - // Wrong AAD: authentication, not encoding. (The fake key source ignores - // descriptors; against ZeroKMS the retrieve is refused first, as - // `STATUS_KMS_FORBIDDEN` — see `status.rs`.) - assert_eq!( - block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"other", - false - )), - Err(STATUS_AUTH) - ); - // Garbage transport bytes on either path: encoding. - assert_eq!( - block_on(ops::encrypt_value( - &cipher.default_keyset(), - b"\xffgarbage", - b"ctx", - false - )), - Err(STATUS_ENCODING) - ); - assert_eq!( - block_on(ops::decrypt_value( - Scope::Client(&cipher), - b"\xffgarbage", - b"ctx", - false - )), - Err(STATUS_ENCODING) - ); - // A truncated leaf inside a well-formed tree: encoding (structural), - // never a parse of the wrong layout. - let CipherText::Single(leaf_bytes) = decode_tree(&ct) else { - panic!("expected a single leaf"); - }; - let mut out = Vec::new(); - codec::encode_ciphertext::, FfiValue>( - &CipherText::Single(leaf_bytes[..10].to_vec()), - &mut out, - ) - .expect("encode truncated"); - assert_eq!( - block_on(ops::decrypt_value( - Scope::Client(&cipher), - &out, - b"ctx", - false - )), - Err(STATUS_ENCODING) - ); -} - -/// The value paths are the cipher-directed path, and take the AAD as -/// `StackCipher::encrypt` does — any bytes, none included. An empty AAD -/// seals under no context and opens under the same, and a null pointer with -/// zero length is the same empty AAD (the ABI's `input` maps it so). Binding -/// a value to a field is the record and term paths' job, where the context is -/// a `NonEmpty`. -#[test] -fn an_empty_aad_round_trips_on_the_value_paths() { - let cipher = cipher(); - let value = encode(s("x")); - - for as_element in [false, true] { - let ct = block_on(ops::encrypt_value( - &cipher.default_keyset(), - &value, - b"", - as_element, - )) - .expect("encrypt under an empty aad"); - let out = block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"", - as_element, - )) - .expect("decrypt under an empty aad"); - assert_eq!(out, value, "element: {as_element}"); - - // Empty is a context like any other: not interchangeable with one - // that carries bytes. - assert_eq!( - block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"ctx", - as_element - )), - Err(STATUS_AUTH), - "element: {as_element}" - ); - } - - // Odd-looking but non-empty bytes are a context too, and bind. - let zeros = &[0u8; 8][..]; - let ct = block_on(ops::encrypt_value( - &cipher.default_keyset(), - &value, - zeros, - false, - )) - .expect("encrypt"); - let opened = decode( - &block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - zeros, - false, - )) - .expect("decrypt"), - ); - assert_eq!(text(&opened), "x"); - assert_eq!( - block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"ctx", - false - )), - Err(STATUS_AUTH) - ); -} - // ============================================================================= // Terms // ============================================================================= @@ -1153,51 +920,6 @@ fn keyset_named<'c>( block_on(cipher.keyset(IdentifiedBy::Name(name.to_string().into()))).expect("select keyset") } -/// A value sealed under a tenant's keyset opens through that keyset, through -/// `{"any"}`, and not through another tenant's — and the refusal costs no -/// ZeroKMS call. -#[test] -fn a_value_opens_under_its_own_keyset_or_any_but_not_another() { - let cipher = cipher(); - let acme = keyset_named(&cipher, "acme"); - let globex = keyset_named(&cipher, "globex"); - let ct = block_on(ops::encrypt_value(&acme, &encode(s("x")), b"ctx", false)).expect("encrypt"); - - let pt = block_on(ops::decrypt_value( - Scope::Keyset(acme.clone()), - &ct, - b"ctx", - false, - )) - .expect("own keyset opens"); - assert_eq!(text(&decode(&pt)), "x"); - let pt = block_on(ops::decrypt_value( - Scope::Client(&cipher), - &ct, - b"ctx", - false, - )) - .expect("any opens"); - assert_eq!(text(&decode(&pt)), "x"); - let retrieves = cipher.kms().retrieve_calls.load(Ordering::SeqCst); - - assert_eq!( - block_on(ops::decrypt_value( - Scope::Keyset(globex), - &ct, - b"ctx", - false - )), - Err(STATUS_FOREIGN_KEYSET), - "another tenant's keyset must refuse the leaf" - ); - assert_eq!( - cipher.kms().retrieve_calls.load(Ordering::SeqCst), - retrieves, - "the refusal happens before any key is retrieved" - ); -} - /// A record batch whose rows were sealed under different keysets opens /// through `{"any"}` in one call per keyset, and not through one keyset. #[test] @@ -1526,3 +1248,54 @@ fn record_tree_validation_refuses_what_decrypt_record_refuses() { } assert_eq!(cipher.kms().retrieve_calls.load(Ordering::SeqCst), 0); } + +// ============================================================================= +// The generator's questions +// ============================================================================= + +/// `plan_check` accepts what `encrypt_record` would run and refuses what it +/// would refuse, with no cipher in hand: the Go generator asks this for every +/// declaration it writes. +#[test] +fn plan_check_answers_without_a_cipher() { + assert_eq!(ops::plan_check(&plan()), Ok(())); + assert_eq!(ops::plan_check(&extended_plan()), Ok(())); + + // match over an integer: a type that does not admit one of its indexes. + let bad_index = encode(obj(vec![( + "age", + obj(vec![ + ("context", label("age")), + ("outputs", FfiValue::Array(vec![s("c"), s("match")])), + ("type", s("uint32")), + ]), + )])); + assert_eq!(ops::plan_check(&bad_index), Err(STATUS_ENCODING)); + + // a one-segment context is not a label a fields plan can seal under. + let one_segment = encode(obj(vec![( + "age", + obj(vec![ + ("context", FfiValue::Array(vec![s("users")])), + ("outputs", FfiValue::Array(vec![s("c")])), + ]), + )])); + assert_eq!(ops::plan_check(&one_segment), Err(STATUS_ENCODING)); + + // not a plan at all. + assert_eq!(ops::plan_check(&encode(s("users"))), Err(STATUS_ENCODING)); + assert_eq!(ops::plan_check(b"\xff\xff"), Err(STATUS_ENCODING)); +} + +/// `targets` is the fixed wire shape with no entries until the EQL target +/// dispatch lands. +#[test] +fn targets_is_an_empty_list_for_now() { + let out = ops::targets().expect("targets encode"); + let FfiValue::Object(entries) = decode(&out) else { + panic!("targets is an object"); + }; + assert_eq!(entries.len(), 1); + assert_eq!(entries[0].0, "targets"); + assert!(matches!(&entries[0].1, FfiValue::Array(items) if items.is_empty())); +} diff --git a/languages/golang/encrypt/guest_test.go b/languages/golang/encrypt/guest_test.go index dd830860b..ae3190e4c 100644 --- a/languages/golang/encrypt/guest_test.go +++ b/languages/golang/encrypt/guest_test.go @@ -19,7 +19,7 @@ import ( "github.com/cipherstash/stack/languages/golang/auth" "github.com/cipherstash/stack/languages/golang/internal/guest" - "github.com/cipherstash/vitaminc/bindings/go/vcvalue" + "github.com/cipherstash/stack/languages/golang/internal/record" "github.com/tetratelabs/wazero" "github.com/tetratelabs/wazero/sys" ) @@ -145,7 +145,7 @@ func TestImportSurfaceIsWASIPlusTransport(t *testing.T) { if !reflect.DeepEqual(transportImports, want) { t.Fatalf("transport imports = %v, want %v", transportImports, want) } - for name := range map[string]bool{"se_alloc": true, "se_dealloc": true, "se_cipher_init": true, "se_shutdown": true, "se_keyset": true, "se_encrypt": true, "se_decrypt": true, "se_encrypt_element": true, "se_decrypt_element": true, "se_term": true, "se_encrypt_record": true, "se_decrypt_record": true} { + for name := range map[string]bool{"se_alloc": true, "se_dealloc": true, "se_cipher_init": true, "se_shutdown": true, "se_keyset": true, "se_term": true, "se_encrypt_record": true, "se_decrypt_record": true, "se_plan_check": true, "se_targets": true} { if _, ok := compiled.ExportedFunctions()[name]; !ok { t.Errorf("guest does not export %s", name) } @@ -351,42 +351,6 @@ func TestOutOfRangeStatusIsTransport(t *testing.T) { } } -// A bare empty part is an empty context, which the guest refuses at the -// boundary — so the constructor refuses it first, rather than handing back -// a Context that fails every call it is used in. A list is empty only when -// every part is, so With may still carry one. -func TestEmptyContextIsRefusedAtTheRoot(t *testing.T) { - for name, part := range map[string]any{"string": "", "bytes": []byte{}} { - t.Run(name, func(t *testing.T) { - if _, err := NewContext(part); err == nil { - t.Fatal("NewContext accepted an empty part") - } - func() { - defer func() { - if recover() == nil { - t.Error("MustContext did not panic on an empty part") - } - }() - _ = MustContext(part) - }() - }) - } - // The rule is the tree's: an empty part beside a non-empty one is a - // context the guest takes, so With must not inherit the root's check. - mixed, err := MustContext("users/age").With("") - if err != nil { - t.Fatalf("With(empty): %v", err) - } - guestOrSkip(t) - ctx := context.Background() - // No cipher on a raw instance, so a context the guest accepts reaches - // the state check — ErrState here means the context itself passed, - // where a refused one is ErrEncoding before it. - if _, err := rawInstance(t).DefaultKeyset().Term(ctx, uint32(34), mixed, Equality); !errors.Is(err, ErrState) { - t.Fatalf("Term under [non-empty, empty]: %v, want ErrState (the context accepted)", err) - } -} - // A body that fails partway through is a transport failure, not a partial // response the guest is handed. io.ReadAll returns the bytes it managed to // read alongside the error; those bytes are a fragment of a ZeroKMS reply @@ -610,26 +574,48 @@ func mustHex(s string) []byte { return b } -type recordRow struct { - Age uint32 `stash:"label=users/age,index=eq;ore"` - Email string `stash:"label=users/email,index=eq;match"` +// usersPlan is the declaration the record tests send: what a generated +// users package lowers its tags to. +func usersPlan() *record.Plan { + return &record.Plan{Context: []string{"users"}, Fields: []record.Field{ + {Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Ore}}, + {Name: "email", Kind: record.String, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Match}}, + }} +} + +func usersRow(age uint32, email string) record.Source { + return record.Source{"age": age, "email": email} +} + +// fixtureRecord is a stored record of usersPlan with no real key behind it. +func fixtureRecord() record.Sealed { + return record.Sealed{ + "age": {Ciphertext: fixtureLeaf, Terms: map[record.Output][]byte{record.Equality: {1}, record.Ore: {2}}}, + "email": {Ciphertext: fixtureLeaf}, + } } -// A record decrypted under a plan that names a field it does not carry is +// A record opened under a plan that seals a field it does not carry is // refused on the host, with the field named, before the guest is asked. func TestMismatchedPlanIsRefusedBeforeTheGuest(t *testing.T) { ctx := context.Background() c := rawInstance(t) - record := EncryptedRecord{"Age": {Ciphertext: Sealed(fixtureLeaf)}} - plan, err := NewPlan(FieldPlan{Field: "Email", Name: "email", Context: label(t, "users/email").Context()}) - if err != nil { - t.Fatal(err) - } - var out []struct{ Email string } - err = c.DecryptRecords(ctx, []EncryptedRecord{record}, &out, WithPlan(plan)) - if err == nil || errors.Is(err, ErrState) || !strings.Contains(err.Error(), `no ciphertext for field "email"`) { + partial := record.Sealed{"age": {Ciphertext: fixtureLeaf}} + _, err := c.DefaultKeyset().Open(ctx, usersPlan(), []record.Sealed{partial}) + if !errors.Is(err, errNoCiphertext) || errors.Is(err, ErrState) || !strings.Contains(err.Error(), `field "email"`) { t.Fatalf("mismatched plan: %v, want the host's refusal naming the field", err) } + // A source row missing a field, or carrying one the plan does not name, + // is refused the same way. + for name, row := range map[string]record.Source{ + "missing": {"age": uint32(1)}, + "extra": {"age": uint32(1), "email": "a@b.c", "stray": "x"}, + } { + _, err := c.DefaultKeyset().Seal(ctx, usersPlan(), []record.Source{row}) + if !errors.Is(err, ErrEncoding) || errors.Is(err, ErrState) { + t.Errorf("%s row: %v, want ErrEncoding before the guest", name, err) + } + } } // Every encoding the package builds reaches the guest's own parsers and @@ -641,60 +627,33 @@ func TestGuestAcceptsEveryEncodingThisPackageBuilds(t *testing.T) { def := c.DefaultKeyset() named := c.Keyset(KeysetName("acme")) byID := c.Keyset(KeysetID{9}) - ct := map[string]any{"name": Sealed(fixtureLeaf), "note": vcvalue.Plain{V: "clear"}} - record := EncryptedRecord{ - "Age": {Ciphertext: Sealed(fixtureLeaf), Equality: EqualityTerm{1}, Ore: OreTerm{2}}, - "Email": {Ciphertext: Sealed(fixtureLeaf)}, - } - rows := []recordRow{{Age: 1, Email: "a@b.c"}} - var out []recordRow - var one recordRow - type untaggedRow struct { - Age uint32 - Email string - } - plan, err := NewPlan( - FieldPlan{Field: "Age", Context: label(t, "users/age").Context(), Terms: []TermKind{Equality, Ore}}, - FieldPlan{Field: "Email", Context: label(t, "users/email").Context(), Terms: []TermKind{Equality, Match}}, - ) - if err != nil { - t.Fatal(err) - } - var planned []untaggedRow + tenant := def.Extend(uint64(7), "eu", []byte{1}) + rows := []record.Source{usersRow(1, "a@b.c"), usersRow(2, "b@c.d")} + records := []record.Sealed{fixtureRecord(), fixtureRecord()} calls := map[string]func() error{ - "KeysetID by name": func() error { _, err := named.KeysetID(ctx); return err }, - "KeysetID by id": func() error { _, err := byID.KeysetID(ctx); return err }, - "KeysetID default": func() error { _, err := def.KeysetID(ctx); return err }, - "Encrypt": func() error { _, err := def.Encrypt(ctx, map[string]any{"a": 1}, []byte("aad")); return err }, - "EncryptElement": func() error { _, err := named.EncryptElement(ctx, "row", nil); return err }, - "Decrypt bound": func() error { _, err := byID.Decrypt(ctx, ct, nil); return err }, - "Decrypt any": func() error { _, err := c.Decrypt(ctx, ct, []byte("aad")); return err }, - "DecryptElement any": func() error { _, err := c.DecryptElement(ctx, Sealed(fixtureLeaf), nil); return err }, - "Term equality": func() error { _, err := def.Term(ctx, uint32(34), MustContext("users/age"), Equality); return err }, - "Term match": func() error { _, err := named.Term(ctx, "alice", MustContext("users/email"), Match); return err }, - "Term ore extended": func() error { - c, _ := MustContext("users/age").With(uint64(7)) - _, err := byID.Term(ctx, 1.5, c, Ore) + "KeysetID by name": func() error { _, err := named.KeysetID(ctx); return err }, + "KeysetID by id": func() error { _, err := byID.KeysetID(ctx); return err }, + "KeysetID default": func() error { _, err := def.KeysetID(ctx); return err }, + "Seal default": func() error { _, err := def.Seal(ctx, usersPlan(), rows); return err }, + "Seal named": func() error { _, err := named.Seal(ctx, usersPlan(), rows[:1]); return err }, + "Seal extended": func() error { _, err := tenant.Seal(ctx, usersPlan(), rows); return err }, + "Open bound": func() error { _, err := byID.Open(ctx, usersPlan(), records); return err }, + "Open extended": func() error { _, err := tenant.Open(ctx, usersPlan(), records); return err }, + "Open any": func() error { _, err := c.Open(ctx, usersPlan(), records); return err }, + "Derive equality": func() error { _, err := def.Derive(ctx, usersPlan(), "age", record.Equality, uint32(34)); return err }, + "Derive match": func() error { _, err := named.Derive(ctx, usersPlan(), "email", record.Match, "alice"); return err }, + "Derive ore ext": func() error { _, err := tenant.Derive(ctx, usersPlan(), "age", record.Ore, uint32(1)); return err }, + "Derive ope": func() error { + p := usersPlan() + p.Fields[0].Outputs = append(p.Fields[0].Outputs, record.Ope) + _, err := byID.Derive(ctx, p, "age", record.Ope, uint32(1)) return err }, - "Term ope bytes": func() error { _, err := def.Term(ctx, []byte{1}, MustContext("k"), Ope); return err }, - "Term ext option": func() error { - _, err := byID.Term(ctx, 1.5, MustContext("users/age"), Ore, ExtendContext(uint64(7), "eu")) + "Seal untyped": func() error { + p := &record.Plan{Context: []string{"documents", "v2"}, Fields: []record.Field{{Name: "value", Outputs: []record.Output{record.Ciphertext}}}} + _, err := def.Seal(ctx, p, []record.Source{{"value": map[string]any{"title": "x", "tags": []string{"a"}}}}) return err }, - "EncryptRecords": func() error { _, err := def.EncryptRecords(ctx, rows); return err }, - "EncryptRecords ext": func() error { _, err := named.EncryptRecords(ctx, &rows, ExtendContext(uint64(7), "eu")); return err }, - "EncryptRecord": func() error { _, err := byID.EncryptRecord(ctx, rows[0]); return err }, - "DecryptRecords bound": func() error { return def.DecryptRecords(ctx, []EncryptedRecord{record}, &out) }, - "DecryptRecords any": func() error { return c.DecryptRecords(ctx, []EncryptedRecord{record, record}, &out) }, - "DecryptRecord any": func() error { return c.DecryptRecord(ctx, record, &one, ExtendContext("x")) }, - "EncryptRecords plan": func() error { - _, err := def.EncryptRecords(ctx, []untaggedRow{{Age: 1, Email: "a@b.c"}}, WithPlan(plan)) - return err - }, - "DecryptRecords plan": func() error { - return c.DecryptRecords(ctx, []EncryptedRecord{record}, &planned, WithPlan(plan), ExtendContext(uint64(7))) - }, } for name, call := range calls { if err := call(); !errors.Is(err, ErrState) { @@ -709,125 +668,57 @@ func TestGuestRefusesMalformedInputsBeforeState(t *testing.T) { ctx := context.Background() c := rawInstance(t) def := c.DefaultKeyset() - type badRow struct { - Age float64 `stash:"label=users/age,index=eq"` - } + plan := usersPlan() + floatAge := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Float64, Outputs: []record.Output{record.Ciphertext, record.Equality}}}} + matchInt := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Match}}}} calls := map[string]func() error{ - "float under equality": func() error { _, err := def.Term(ctx, 1.5, MustContext("k"), Equality); return err }, - "integer under match": func() error { _, err := def.Term(ctx, 1, MustContext("k"), Match); return err }, - "container as term value": func() error { _, err := def.Term(ctx, []any{1}, MustContext("k"), Ore); return err }, - // NewContext refuses this one now (see - // TestEmptyContextIsRefusedAtTheRoot); built by hand so the guest's - // own boundary check stays covered from this side too. - "empty context part": func() error { - _, err := def.Term(ctx, 1, Context{node: ""}, Equality) - return err - }, - "unknown term kind": func() error { _, err := def.Term(ctx, 1, MustContext("k"), TermKind(9)); return err }, - "name with spaces": func() error { _, err := c.Keyset(KeysetName("not a name")).KeysetID(ctx); return err }, - "empty name": func() error { _, err := c.Keyset(KeysetName("")).KeysetID(ctx); return err }, - "any as a keyset": func() error { _, err := c.Keyset(anyKeyset{}).KeysetID(ctx); return err }, - "float under eq in plan": func() error { _, err := def.EncryptRecords(ctx, []badRow{{1.5}}); return err }, - "malformed leaf": func() error { - _, err := c.Decrypt(ctx, Sealed{1, 2, 3}, nil) + "float under equality": func() error { _, err := def.Derive(ctx, floatAge, "age", record.Equality, 1.5); return err }, + "integer under match": func() error { _, err := def.Derive(ctx, matchInt, "age", record.Match, uint32(1)); return err }, + "container as term value": func() error { _, err := def.Derive(ctx, plan, "age", record.Equality, []any{1}); return err }, + "a kind the type refuses": func() error { _, err := def.Seal(ctx, matchInt, []record.Source{usersRow(1, "x")}); return err }, + "a value of another kind": func() error { + _, err := def.Seal(ctx, plan, []record.Source{{"age": "thirty", "email": "x"}}) return err }, - "record without c": func() error { - return c.DecryptRecord(ctx, EncryptedRecord{"Age": {Equality: EqualityTerm{1}}, "Email": {Ciphertext: Sealed(fixtureLeaf)}}, new(recordRow)) - }, - // ExtendContext checks nothing when it is built; Context.With - // refuses the part when a call applies it. A call that dropped - // that error would run under a context missing the extension: - // a probe that matches no rows, or rows no probe matches. - "bad ext part in a term": func() error { - _, err := def.Term(ctx, 1, MustContext("k"), Equality, ExtendContext(1.5)) + "a non-leaf under c": func() error { + _, err := def.Open(ctx, plan, []record.Sealed{{"age": {Ciphertext: []byte{0xff}}, "email": {Ciphertext: fixtureLeaf}}}) return err }, - "bad ext part in a record write": func() error { - _, err := def.EncryptRecords(ctx, []recordRow{{Age: 1, Email: "a@b.c"}}, ExtendContext(1.5)) - return err - }, - "bad ext part in a record read": func() error { - record := EncryptedRecord{"Age": {Ciphertext: Sealed(fixtureLeaf)}, "Email": {Ciphertext: Sealed(fixtureLeaf)}} - return c.DecryptRecord(ctx, record, new(recordRow), ExtendContext(1.5)) - }, + "a term the field lacks": func() error { _, err := def.Derive(ctx, plan, "age", record.Match, uint32(1)); return err }, + "a field the plan lacks": func() error { _, err := def.Derive(ctx, plan, "name", record.Equality, "x"); return err }, + "an empty context segment": func() error { p := usersPlan(); p.Context = []string{""}; _, err := def.Seal(ctx, p, nil); return err }, + "an extension the codec cannot carry": func() error { _, err := def.Extend(1.5).Seal(ctx, plan, nil); return err }, + "a nil keyset selector": func() error { _, err := c.Keyset(nil).Seal(ctx, plan, nil); return err }, } for name, call := range calls { - err := call() - if errors.Is(err, ErrState) { - t.Errorf("%s: reached the cipher (ErrState); must be refused at parse", name) - } else if err == nil { - t.Errorf("%s: accepted", name) + if err := call(); !errors.Is(err, ErrEncoding) { + t.Errorf("%s: %v, want ErrEncoding", name, err) } } } -// NewPlan leaves two rules to the record call: every field under one table -// (the label's leading segments), and no two fields under one label. The -// guest's lowering refuses both at parse, before it looks for a cipher, as -// the caller's input (ErrEncoding), on the write and on the read. -func TestGuestRefusesAPlanNewPlanLeavesToTheCall(t *testing.T) { +// se_plan_check answers with no cipher: a plan the engine runs passes, a +// plan it refuses is ErrEncoding, never ErrState. +func TestPlanCheckAnswersWithoutACipher(t *testing.T) { ctx := context.Background() c := rawInstance(t) - type row struct { - Age uint32 - Email string - } - record := EncryptedRecord{ - "Age": {Ciphertext: Sealed(fixtureLeaf)}, - "Email": {Ciphertext: Sealed(fixtureLeaf)}, - } - for name, fields := range map[string][]FieldPlan{ - "two tables": { - {Field: "Age", Context: label(t, "users/age").Context()}, - {Field: "Email", Context: label(t, "accounts/email").Context()}, - }, - "one label twice": { - {Field: "Age", Context: label(t, "users/age").Context(), Terms: []TermKind{Equality}}, - {Field: "Email", Context: label(t, "users/age").Context(), Terms: []TermKind{Equality}}, - }, - } { - p, err := NewPlan(fields...) - if err != nil { - t.Fatalf("%s: NewPlan refused it: %v", name, err) - } - _, err = c.DefaultKeyset().EncryptRecords(ctx, []row{{Age: 1, Email: "a@b.c"}}, WithPlan(p)) - if !errors.Is(err, ErrEncoding) { - t.Errorf("%s: EncryptRecords err = %v, want ErrEncoding", name, err) - } - var out row - if err := c.DecryptRecord(ctx, record, &out, WithPlan(p)); !errors.Is(err, ErrEncoding) { - t.Errorf("%s: DecryptRecord err = %v, want ErrEncoding", name, err) - } + k := &Checker{c: c} + if err := k.Check(ctx, usersPlan()); err != nil { + t.Fatalf("a good plan: %v", err) } -} - -// A bad ExtendContext part fails the call for that reason, on the probe and -// on both record directions. TestGuestRefusesMalformedInputsBeforeState -// shows the call never reaches the cipher, but a call that ignored the -// error and went on with an empty context would be refused too, by the -// guest, for another reason; only the error's own words tell the two apart. -func TestBadExtensionPartFailsTheCall(t *testing.T) { - ctx := context.Background() - c := rawInstance(t) - def := c.DefaultKeyset() - bad := ExtendContext(uint64(7), 1.5) - record := EncryptedRecord{"Age": {Ciphertext: Sealed(fixtureLeaf)}, "Email": {Ciphertext: Sealed(fixtureLeaf)}} - for name, call := range map[string]func() error{ - "Term": func() error { - _, err := def.Term(ctx, 1, MustContext("k"), Equality, bad) - return err - }, - "EncryptRecords": func() error { - _, err := def.EncryptRecords(ctx, []recordRow{{Age: 1, Email: "a@b.c"}}, bad) - return err - }, - "DecryptRecord": func() error { return c.DecryptRecord(ctx, record, new(recordRow), bad) }, - } { - if err := call(); err == nil || !strings.Contains(err.Error(), "float64 is not a context part") { - t.Errorf("%s with a float64 extension part: %v, want the part refused", name, err) + refused := []*record.Plan{ + {Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Match}}}}, + {Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Float64, Outputs: []record.Output{record.Equality}}}}, + } + for i, p := range refused { + if err := k.Check(ctx, p); !errors.Is(err, ErrEncoding) { + t.Errorf("plan %d: %v, want ErrEncoding", i, err) } } + targets, err := k.Targets(ctx) + if err != nil || len(targets) != 0 { + t.Fatalf("Targets = %v, %v; want none in this build", targets, err) + } } func TestClosedClientIsState(t *testing.T) { @@ -839,8 +730,8 @@ func TestClosedClientIsState(t *testing.T) { if err := c.Close(); err != nil { t.Fatalf("second Close: %v", err) } - if _, err := c.DefaultKeyset().Encrypt(ctx, "x", nil); !errors.Is(err, ErrState) { - t.Fatalf("Encrypt after Close: %v", err) + if _, err := c.DefaultKeyset().Seal(ctx, usersPlan(), []record.Source{usersRow(1, "x")}); !errors.Is(err, ErrState) { + t.Fatalf("Seal after Close: %v", err) } } @@ -865,8 +756,9 @@ func TestHostilePointerLengthPairsAreStatusesNotTraps(t *testing.T) { for name, fn := range map[string]func() ([]uint64, error){ "se_cipher_init": func() ([]uint64, error) { return inst.cipherInit.Call(ctx, uint64(staged.Ptr), hostile) }, "se_keyset": func() ([]uint64, error) { return inst.keyset.Call(ctx, uint64(staged.Ptr), hostile) }, - "se_encrypt": func() ([]uint64, error) { - return inst.encrypt.Call(ctx, uint64(staged.Ptr), hostile, 0, 0, uint64(staged.Ptr), 4) + "se_plan_check": func() ([]uint64, error) { return inst.planCheck.Call(ctx, uint64(staged.Ptr), hostile) }, + "se_encrypt_record": func() ([]uint64, error) { + return inst.encryptRecord.Call(ctx, uint64(staged.Ptr), hostile, uint64(staged.Ptr), 4, uint64(staged.Ptr), 4) }, } { res, err := fn() @@ -886,7 +778,7 @@ func TestHostilePointerLengthPairsAreStatusesNotTraps(t *testing.T) { t.Fatalf("dealloc with a mismatched length trapped: %v", err) } // The instance still works. - if _, err := c.DefaultKeyset().Encrypt(ctx, "alive", nil); !errors.Is(err, ErrState) { + if _, err := c.DefaultKeyset().Seal(ctx, usersPlan(), []record.Source{usersRow(1, "alive")}); !errors.Is(err, ErrState) { t.Fatalf("instance poisoned: %v", err) } } diff --git a/languages/golang/encrypt/internal/testusers/document_stash.go b/languages/golang/encrypt/internal/testusers/document_stash.go new file mode 100644 index 000000000..b55d48b56 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/document_stash.go @@ -0,0 +1,88 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Document prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedDocument struct { + Sealed encrypt.Ciphertext +} + +func (e EncryptedDocument) String() string { + return gensupport.Redacted("EncryptedDocument", nil, "Sealed") +} + +func (e EncryptedDocument) LogValue() slog.Value { + return gensupport.RedactedLog(nil, "Sealed") +} + +// Stops compiling when Document gains, loses, reorders or retypes a field. +var _ = documentShape(Document{}) + +type documentShape struct { + _ struct{} + Title string + Body string + Tags []string +} + +var documentDeclaration = gensupport.DeclareOpaque("documents/v2/body") + +var documentCodec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ + TypeName: "Document", + Declaration: documentDeclaration, + PrintsPlaintext: true, + Source: func(v Document) gensupport.Values { + return gensupport.Values{gensupport.OpaqueField: map[string]any{ + "title": v.Title, + "body": v.Body, + "tags": v.Tags, + }} + }, + Seal: func(rec gensupport.Record) (EncryptedDocument, error) { + return EncryptedDocument{Sealed: rec[gensupport.OpaqueField].Ciphertext}, nil + }, + Open: func(e EncryptedDocument) gensupport.Record { + return gensupport.Record{gensupport.OpaqueField: {Ciphertext: e.Sealed}} + }, + Value: func(e EncryptedDocument, vals gensupport.Values) (Document, error) { + fields, err := gensupport.Get[gensupport.Values](vals, gensupport.OpaqueField) + if err != nil { + return Document{}, err + } + var v Document + if v.Title, err = gensupport.Get[string](fields, "title"); err != nil { + return Document{}, err + } + if v.Body, err = gensupport.Get[string](fields, "body"); err != nil { + return Document{}, err + } + if v.Tags, err = gensupport.Get[[]string](fields, "tags"); err != nil { + return Document{}, err + } + return v, nil + }, +}) + +// EncryptDocument seals each Document in one ZeroKMS request. The result has +// one element for each input, in the same order. +func EncryptDocument(ctx context.Context, cipher *encrypt.Cipher, values []Document) ([]EncryptedDocument, error) { + return documentCodec.Encrypt(ctx, cipher, values) +} + +// DecryptDocument opens each EncryptedDocument in one ZeroKMS request. +func DecryptDocument(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedDocument) ([]Document, error) { + return documentCodec.Decrypt(ctx, d, encrypted) +} diff --git a/languages/golang/encrypt/internal/testusers/probe_stash.go b/languages/golang/encrypt/internal/testusers/probe_stash.go new file mode 100644 index 000000000..b17727cb9 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/probe_stash.go @@ -0,0 +1,280 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Probe prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedProbe struct { + U32 EncryptedProbeU32 + U64 EncryptedProbeU64 + I64 EncryptedProbeI64 + S EncryptedProbeS + B EncryptedProbeB +} + +type EncryptedProbeU32 struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm + Ope encrypt.OpeTerm +} + +type EncryptedProbeU64 struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm + Ope encrypt.OpeTerm +} + +type EncryptedProbeI64 struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm + Ope encrypt.OpeTerm +} + +type EncryptedProbeS struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm + Ope encrypt.OpeTerm +} + +type EncryptedProbeB struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm + Ope encrypt.OpeTerm +} + +func (e EncryptedProbe) String() string { + return gensupport.Redacted("EncryptedProbe", nil, "U32", "U64", "I64", "S", "B") +} + +func (e EncryptedProbe) LogValue() slog.Value { + return gensupport.RedactedLog(nil, "U32", "U64", "I64", "S", "B") +} + +// Stops compiling when Probe gains, loses, reorders or retypes a field. +var _ = probeShape(Probe{}) + +type probeShape struct { + _ struct{} + U32 uint32 + U64 uint64 + I64 int64 + S string + B []byte +} + +var probeDeclaration = gensupport.Declare("prop"). + EncryptIndex("u32", gensupport.UInt32, encrypt.Ore, encrypt.Ope). + EncryptIndex("u64", gensupport.UInt64, encrypt.Ore, encrypt.Ope). + EncryptIndex("i64", gensupport.Int64, encrypt.Ore, encrypt.Ope). + EncryptIndex("s", gensupport.String, encrypt.Ore, encrypt.Ope). + EncryptIndex("b", gensupport.Bytes, encrypt.Ore, encrypt.Ope) + +var probeCodec = gensupport.New(gensupport.Generated[Probe, EncryptedProbe]{ + TypeName: "Probe", + Declaration: probeDeclaration, + PrintsPlaintext: true, + Source: func(v Probe) gensupport.Values { + return gensupport.Values{ + "u32": v.U32, + "u64": v.U64, + "i64": v.I64, + "s": v.S, + "b": v.B, + } + }, + Seal: func(rec gensupport.Record) (EncryptedProbe, error) { + var e EncryptedProbe + e.U32 = EncryptedProbeU32{ + Ciphertext: rec["u32"].Ciphertext, + Ore: rec["u32"].Ore, + Ope: rec["u32"].Ope, + } + e.U64 = EncryptedProbeU64{ + Ciphertext: rec["u64"].Ciphertext, + Ore: rec["u64"].Ore, + Ope: rec["u64"].Ope, + } + e.I64 = EncryptedProbeI64{ + Ciphertext: rec["i64"].Ciphertext, + Ore: rec["i64"].Ore, + Ope: rec["i64"].Ope, + } + e.S = EncryptedProbeS{ + Ciphertext: rec["s"].Ciphertext, + Ore: rec["s"].Ore, + Ope: rec["s"].Ope, + } + e.B = EncryptedProbeB{ + Ciphertext: rec["b"].Ciphertext, + Ore: rec["b"].Ore, + Ope: rec["b"].Ope, + } + return e, nil + }, + Open: func(e EncryptedProbe) gensupport.Record { + return gensupport.Record{ + "u32": {Ciphertext: e.U32.Ciphertext, Ore: e.U32.Ore, Ope: e.U32.Ope}, + "u64": {Ciphertext: e.U64.Ciphertext, Ore: e.U64.Ore, Ope: e.U64.Ope}, + "i64": {Ciphertext: e.I64.Ciphertext, Ore: e.I64.Ore, Ope: e.I64.Ope}, + "s": {Ciphertext: e.S.Ciphertext, Ore: e.S.Ore, Ope: e.S.Ope}, + "b": {Ciphertext: e.B.Ciphertext, Ore: e.B.Ore, Ope: e.B.Ope}, + } + }, + Value: func(e EncryptedProbe, vals gensupport.Values) (Probe, error) { + var v Probe + var err error + if v.U32, err = gensupport.Get[uint32](vals, "u32"); err != nil { + return Probe{}, err + } + if v.U64, err = gensupport.Get[uint64](vals, "u64"); err != nil { + return Probe{}, err + } + if v.I64, err = gensupport.Get[int64](vals, "i64"); err != nil { + return Probe{}, err + } + if v.S, err = gensupport.Get[string](vals, "s"); err != nil { + return Probe{}, err + } + if v.B, err = gensupport.Get[[]byte](vals, "b"); err != nil { + return Probe{}, err + } + return v, nil + }, +}) + +// EncryptProbe seals each Probe in one ZeroKMS request. The result has one +// element for each input, in the same order. +func EncryptProbe(ctx context.Context, cipher *encrypt.Cipher, values []Probe) ([]EncryptedProbe, error) { + return probeCodec.Encrypt(ctx, cipher, values) +} + +// DecryptProbe opens each EncryptedProbe in one ZeroKMS request. +func DecryptProbe(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedProbe) ([]Probe, error) { + return probeCodec.Decrypt(ctx, d, encrypted) +} + +var ProbeFields = struct { + U32 ProbeU32Field + U64 ProbeU64Field + I64 ProbeI64Field + S ProbeSField + B ProbeBField +}{ + U32: ProbeU32Field{gensupport.NewField[uint32](probeDeclaration, "u32")}, + U64: ProbeU64Field{gensupport.NewField[uint64](probeDeclaration, "u64")}, + I64: ProbeI64Field{gensupport.NewField[int64](probeDeclaration, "i64")}, + S: ProbeSField{gensupport.NewField[string](probeDeclaration, "s")}, + B: ProbeBField{gensupport.NewField[[]byte](probeDeclaration, "b")}, +} + +type ProbeU32Field struct { + field gensupport.Field[uint32] +} + +func (f ProbeU32Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (EncryptedProbeU32, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProbeU32{}, err + } + return EncryptedProbeU32{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil +} + +func (f ProbeU32Field) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +func (f ProbeU32Field) Ope(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} + +type ProbeU64Field struct { + field gensupport.Field[uint64] +} + +func (f ProbeU64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint64) (EncryptedProbeU64, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProbeU64{}, err + } + return EncryptedProbeU64{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil +} + +func (f ProbeU64Field) Ore(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +func (f ProbeU64Field) Ope(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} + +type ProbeI64Field struct { + field gensupport.Field[int64] +} + +func (f ProbeI64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int64) (EncryptedProbeI64, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProbeI64{}, err + } + return EncryptedProbeI64{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil +} + +func (f ProbeI64Field) Ore(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +func (f ProbeI64Field) Ope(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} + +type ProbeSField struct { + field gensupport.Field[string] +} + +func (f ProbeSField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedProbeS, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProbeS{}, err + } + return EncryptedProbeS{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil +} + +func (f ProbeSField) Ore(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +func (f ProbeSField) Ope(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} + +type ProbeBField struct { + field gensupport.Field[[]byte] +} + +func (f ProbeBField) Encrypt(ctx context.Context, c *encrypt.Cipher, v []byte) (EncryptedProbeB, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProbeB{}, err + } + return EncryptedProbeB{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil +} + +func (f ProbeBField) Ore(ctx context.Context, c *encrypt.Cipher, v []byte) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +func (f ProbeBField) Ope(ctx context.Context, c *encrypt.Cipher, v []byte) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} diff --git a/languages/golang/encrypt/internal/testusers/user_stash.go b/languages/golang/encrypt/internal/testusers/user_stash.go new file mode 100644 index 000000000..14b0a3567 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/user_stash.go @@ -0,0 +1,195 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// User prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedUser struct { + ID int64 + Age EncryptedUserAge + Email EncryptedUserEmail + Notes EncryptedUserNotes +} + +type EncryptedUserAge struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Ore encrypt.OreTerm +} + +type EncryptedUserEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} + +type EncryptedUserNotes struct { + Ciphertext encrypt.Ciphertext +} + +func (e EncryptedUser) String() string { + return gensupport.Redacted("EncryptedUser", map[string]any{"ID": e.ID}, "Age", "Email", "Notes") +} + +func (e EncryptedUser) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Age", "Email", "Notes") +} + +// Stops compiling when User gains, loses, reorders or retypes a field. +var _ = userShape(User{}) + +type userShape struct { + _ struct{} + ID int64 + Age uint32 + Email string + Notes string + Internal string +} + +var declaration = gensupport.Declare("users"). + Passthrough("id"). + EncryptIndex("age", gensupport.UInt32, encrypt.Equality, encrypt.Ore). + EncryptIndex("email", gensupport.String, encrypt.Equality, encrypt.Match()). + Encrypt("notes", gensupport.String). + Omit("internal") + +var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ + TypeName: "User", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v User) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "age": v.Age, + "email": v.Email, + "notes": v.Notes, + } + }, + Seal: func(rec gensupport.Record) (EncryptedUser, error) { + var e EncryptedUser + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedUser{}, err + } + e.Age = EncryptedUserAge{ + Ciphertext: rec["age"].Ciphertext, + Equality: rec["age"].Equality, + Ore: rec["age"].Ore, + } + e.Email = EncryptedUserEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + Match: rec["email"].Match, + } + e.Notes = EncryptedUserNotes{Ciphertext: rec["notes"].Ciphertext} + return e, nil + }, + Open: func(e EncryptedUser) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "age": {Ciphertext: e.Age.Ciphertext, Equality: e.Age.Equality, Ore: e.Age.Ore}, + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality, Match: e.Email.Match}, + "notes": {Ciphertext: e.Notes.Ciphertext}, + } + }, + Value: func(e EncryptedUser, vals gensupport.Values) (User, error) { + var v User + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return User{}, err + } + if v.Age, err = gensupport.Get[uint32](vals, "age"); err != nil { + return User{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return User{}, err + } + if v.Notes, err = gensupport.Get[string](vals, "notes"); err != nil { + return User{}, err + } + return v, nil + }, +}) + +// Encrypt seals each User in one ZeroKMS request. The result has one element +// for each input, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, testusers []User) ([]EncryptedUser, error) { + return codec.Encrypt(ctx, cipher, testusers) +} + +// Decrypt opens each EncryptedUser in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { + return codec.Decrypt(ctx, d, encrypted) +} + +var Fields = struct { + Age AgeField + Email EmailField + Notes NotesField +}{ + Age: AgeField{gensupport.NewField[uint32](declaration, "age")}, + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + Notes: NotesField{gensupport.NewField[string](declaration, "notes")}, +} + +type AgeField struct { + field gensupport.Field[uint32] +} + +func (f AgeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (EncryptedUserAge, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedUserAge{}, err + } + return EncryptedUserAge{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil +} + +func (f AgeField) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f AgeField) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type EmailField struct { + field gensupport.Field[string] +} + +func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedUserEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedUserEmail{}, err + } + return EncryptedUserEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil +} + +func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +type NotesField struct { + field gensupport.Field[string] +} + +func (f NotesField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedUserNotes, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedUserNotes{Ciphertext: out.Ciphertext}, err +} diff --git a/languages/golang/encrypt/internal/testusers/users.go b/languages/golang/encrypt/internal/testusers/users.go new file mode 100644 index 000000000..063e97f07 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/users.go @@ -0,0 +1,45 @@ +// Package testusers holds the tagged structs the encrypt tests encrypt +// through the generated API: a record in separate columns, an opaque struct, +// and a probe struct with one field of each scalar type for the term +// ordering tests. The *_stash.go files beside them are what stashgen writes +// from the tags; `go generate ./...` rewrites them, and CI fails when that +// changes a file. +package testusers + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type User + +// User is the record of the Rust record fixture +// (packages/stack-encrypt/tests/fixtures/record_lowering.json): the same +// context, identities, kinds and indexes, so the fixture's records open +// through the generated code and its terms compare. +type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough"` + Age uint32 `stash:"age,encrypt,index=equality;ore"` + Email string `stash:"email,encrypt,index=equality;match"` + Notes string `stash:"notes,encrypt"` + Internal string `stash:"-"` +} + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Document -name Document + +// Document is sealed as one value. +type Document struct { + _ struct{} `stash:"context=documents/v2/body,opaque"` + Title string + Body string + Tags []string +} + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Probe -name Probe + +// Probe has one field of each scalar type the ORE and OPE indexes order, for +// the property tests of term ordering. +type Probe struct { + _ struct{} `stash:"context=prop"` + U32 uint32 `stash:"u32,encrypt,index=ore;ope"` + U64 uint64 `stash:"u64,encrypt,index=ore;ope"` + I64 int64 `stash:"i64,encrypt,index=ore;ope"` + S string `stash:"s,encrypt,index=ore;ope"` + B []byte `stash:"b,encrypt,index=ore;ope"` +} diff --git a/languages/golang/encrypt/label.go b/languages/golang/encrypt/label.go deleted file mode 100644 index c48944f3f..000000000 --- a/languages/golang/encrypt/label.go +++ /dev/null @@ -1,142 +0,0 @@ -package encrypt - -import ( - "errors" - "fmt" - "slices" - "strings" - "unicode" -) - -// Label names the data a field or a probe binds: a table and a column -// ("users/email"), a document path ("documents/v2/body"), any name a direct -// consumer chooses. ZeroKMS binds the data key to that name and writes it in -// its log, spelled exactly as given. It is the Go form of Rust's -// stack_encrypt::Label; EQL's identifier, a table and a column, is a Label of -// two segments ([plan.Identifier]). -// -// # Naming and scoping -// -// A context carries two kinds of information, and each has one spelling: -// -// - A name says WHAT the data is. Spell it as a Label. -// - A scope says WHICH slice of that data: a tenant, a row. Spell it by -// extending the name's context with [Context.With], or with the -// [ExtendContext] option on a record call. -// -// In practice: -// -// What you mean Spelling ZeroKMS log -// the users.email column label, _ := ParseLabel("users/email") users/email -// that column, tenant 7 label.Context().With(uint64(7)) (users/email)/7u64 -// a deeper name ParseLabel("documents/v2/body") documents/v2/body -// a one-part name ParseLabel("users"), the same as NewContext("users") users -// -// A one-part name is a probe's: a planned field ([FieldPlan.Context]) binds -// a label of at least two segments, a table and a column, since the guest -// seals every field of a record under one context and the field's own -// identity. -// -// Do not build a name with With, and do not put a scope into a Label. The -// renderer keeps the two apart: a name is one flat list, a scope nests. So -// (users/email)/7u64 is never read as a three-segment name, and -// documents/v2/body is never read as a scoped column. -// -// A two-segment Label binds the same context a Rust -// `#[stash(struct = .., context = "
")]` derive gives a field. That is -// what lets a Go label open a row a Rust derive wrote, and a probe built from -// the label match the terms the derive produced. -// -// # Segments -// -// Every segment is plain — non-empty, no control or invisible format -// characters (zero-width and bidirectional marks), none of '/', '(' or ')', -// not beginning with "b64:", a digit or '-' — which is exactly -// the text the descriptor renders verbatim. So a Label's [Label.String] is -// its descriptor, [ParseLabel] reads that string back losslessly (no -// segment can contain the separator), and a string that is not a label is -// refused with a [LabelError] naming the segment, never escaped silently. -type Label struct { - segments []string -} - -// labelSeparator joins a label's segments: the descriptor's own separator. -const labelSeparator = "/" - -// NewLabel makes a label from its segments, each checked to be plain. -func NewLabel(segments ...string) (Label, error) { - if len(segments) == 0 { - return Label{}, ErrEmptyLabel - } - for i, s := range segments { - if err := checkSegment(i, s); err != nil { - return Label{}, err - } - } - return Label{segments: slices.Clone(segments)}, nil -} - -// ParseLabel reads a label from its rendered form, segments separated by -// '/': the inverse of [Label.String]. "users//email" and "users/" are -// refused (an empty segment), as is "" (one empty segment). -func ParseLabel(s string) (Label, error) { - return NewLabel(strings.Split(s, labelSeparator)...) -} - -// Segments returns the label's segments, in order; at least one. -func (l Label) Segments() []string { return slices.Clone(l.segments) } - -// String renders the label as its descriptor: the segments joined by '/'. -func (l Label) String() string { return strings.Join(l.segments, labelSeparator) } - -// Context is the label as the context a field or probe binds. A zero -// Label gives the zero Context, which every call refuses as "needs a -// context". -func (l Label) Context() Context { return flatContext(l.segments) } - -// ErrEmptyLabel is [NewLabel]'s refusal of no segments at all. -var ErrEmptyLabel = errors.New("encrypt: a label needs at least one segment") - -// LabelError says why a string is not a [Label] segment. Index is the -// segment's position, counting from zero. -type LabelError struct { - Index int - Reason string -} - -func (e *LabelError) Error() string { - return fmt.Sprintf("encrypt: label segment %d %s", e.Index, e.Reason) -} - -// checkSegment is the one definition of plain text, the same as Rust's -// Label::check_segment: what passes here is what the descriptor renders -// verbatim. -func checkSegment(index int, s string) error { - if s == "" { - return &LabelError{Index: index, Reason: "is empty"} - } - if strings.HasPrefix(s, "b64:") || s[0] == '-' || (s[0] >= '0' && s[0] <= '9') { - return &LabelError{Index: index, Reason: "begins like another descriptor form (b64:, a digit or -)"} - } - for _, r := range s { - if r == '/' { - return &LabelError{Index: index, Reason: "contains '/', the separator"} - } - if unicode.IsControl(r) || r == '(' || r == ')' { - return &LabelError{Index: index, Reason: fmt.Sprintf("contains %q, which the descriptor reserves", r)} - } - if strings.ContainsRune(invisible, r) { - return &LabelError{Index: index, Reason: fmt.Sprintf("contains %q, an invisible format character", r)} - } - } - return nil -} - -// invisible is the format characters with no glyph of their own: the soft -// hyphen, the Arabic letter mark, the Mongolian vowel separator, the -// zero-width characters, the bidirectional embeddings, overrides and -// isolates, and the byte-order mark. unicode.IsControl covers only Cc; -// these are Cf. A name containing one prints like another name in the -// ZeroKMS log, so they are refused beside the control characters. The same -// list as Rust's Label::INVISIBLE; the shared fixture holds the two together. -const invisible = "\u00ad\u061c\u180e\u200b\u200c\u200d\u200e\u200f\u202a\u202b\u202c\u202d\u202e\u2060\u2061\u2062\u2063\u2064\u2066\u2067\u2068\u2069\ufeff" diff --git a/languages/golang/encrypt/label_test.go b/languages/golang/encrypt/label_test.go deleted file mode 100644 index b64104243..000000000 --- a/languages/golang/encrypt/label_test.go +++ /dev/null @@ -1,252 +0,0 @@ -package encrypt - -import ( - "encoding/json" - "errors" - "os" - "path/filepath" - "reflect" - "strings" - "testing" -) - -// A label's string is its descriptor and reads back losslessly; as a -// context, one segment is the bare part, two are the pair With builds, and -// more are a flat list — the same shapes Rust's Label takes. -func TestLabelRendersAsItsDisplayAndBindsTheMatchingContext(t *testing.T) { - pair, err := NewContext("users") - if err != nil { - t.Fatal(err) - } - if pair, err = pair.With("age"); err != nil { - t.Fatal(err) - } - for text, want := range map[string]any{ - "users": "users", - "users/age": []any{"users", "age"}, - "documents/v2/body": []any{"documents", "v2", "body"}, - "naïve/with space": []any{"naïve", "with space"}, - } { - l, err := ParseLabel(text) - if err != nil { - t.Fatalf("ParseLabel(%q): %v", text, err) - } - if l.String() != text { - t.Errorf("ParseLabel(%q).String() = %q", text, l.String()) - } - if got := l.Segments(); strings.Join(got, "/") != text { - t.Errorf("ParseLabel(%q).Segments() = %q", text, got) - } - if got := l.Context().value(); !reflect.DeepEqual(got, want) { - t.Errorf("ParseLabel(%q).Context() = %#v, want %#v", text, got, want) - } - if again, err := NewLabel(l.Segments()...); err != nil || !reflect.DeepEqual(again, l) { - t.Errorf("NewLabel(segments of %q) = %#v, %v", text, again, err) - } - } - if got := label(t, "users/age").Context().value(); !reflect.DeepEqual(got, pair.value()) { - t.Errorf("a two-segment label is not the With pair: %#v vs %#v", got, pair.value()) - } - if got := label(t, "users").Context().value(); !reflect.DeepEqual(got, MustContext("users").value()) { - t.Errorf("a one-segment label is not the bare part: %#v", got) - } - // A literal containing '/' is one part, not the pair: the two spell - // different contexts, as in Rust. - if reflect.DeepEqual(MustContext("users/age").value(), label(t, "users/age").Context().value()) { - t.Error(`NewContext("users/age") and label(t, "users/age") bind the same context`) - } - if got := (Label{}).Context(); !got.isZero() { - t.Errorf("zero Label's Context = %#v, want the zero Context", got) - } -} - -// Every way a segment is not plain is refused and named, matching Rust's -// LabelError variants: a label never renders escaped. -func TestLabelRefusesSegmentsThatWouldNotRenderVerbatim(t *testing.T) { - if _, err := NewLabel(); !errors.Is(err, ErrEmptyLabel) { - t.Errorf("NewLabel() = %v, want ErrEmptyLabel", err) - } - for _, tc := range []struct { - segments []string - index int - reason string - }{ - {[]string{""}, 0, "is empty"}, - {[]string{"users", ""}, 1, "is empty"}, - {[]string{"users", "a/b"}, 1, "separator"}, - {[]string{"b64:x"}, 0, "another descriptor form"}, - {[]string{"users", "7"}, 1, "another descriptor form"}, - {[]string{"-x"}, 0, "another descriptor form"}, - {[]string{"a(b"}, 0, "reserves"}, - {[]string{"a)b"}, 0, "reserves"}, - {[]string{"a\tb"}, 0, "reserves"}, - {[]string{"a\u0085b"}, 0, "reserves"}, - } { - _, err := NewLabel(tc.segments...) - var le *LabelError - if !errors.As(err, &le) { - t.Errorf("NewLabel(%q) = %v, want a LabelError", tc.segments, err) - continue - } - if le.Index != tc.index || !strings.Contains(le.Reason, tc.reason) { - t.Errorf("NewLabel(%q) = %v, want segment %d %q", tc.segments, err, tc.index, tc.reason) - } - } - for _, text := range []string{"", "/", "users/", "/age", "users//age"} { - if _, err := ParseLabel(text); err == nil { - t.Errorf("ParseLabel(%q) succeeded; want a refusal", text) - } - } -} - -// The segment rule is one rule in two languages. Rust's -// plain_text_and_label_segments_are_one_rule reads the same fixture, so a -// change to either implementation that the other does not follow fails here -// or there. -func TestLabelSegmentRuleMatchesTheSharedFixture(t *testing.T) { - raw, err := os.ReadFile(filepath.Join("..", "..", "..", "packages", "stack-encrypt", "tests", "fixtures", "label_segments.json")) - if err != nil { - t.Fatal(err) - } - var fixture struct { - Plain []string `json:"plain"` - NotPlain []string `json:"not_plain"` - } - if err := json.Unmarshal(raw, &fixture); err != nil { - t.Fatal(err) - } - if len(fixture.Plain) == 0 || len(fixture.NotPlain) == 0 { - t.Fatalf("fixture is empty: %+v", fixture) - } - for _, ok := range fixture.Plain { - if _, err := NewLabel(ok); err != nil { - t.Errorf("NewLabel(%q) = %v, want ok", ok, err) - } - } - for _, bad := range fixture.NotPlain { - if _, err := NewLabel(bad); err == nil { - t.Errorf("NewLabel(%q) succeeded; want a refusal", bad) - } - } -} - -// label is ParseLabel for a label the test knows to be valid: the fixture -// form of the error-returning constructor, since there is no panicking one. -func label(t testing.TB, s string) Label { - t.Helper() - l, err := ParseLabel(s) - if err != nil { - t.Fatal(err) - } - return l -} - -// A struct tag names a field's label, and only a label: the one-part -// context= form, which the guest's lowering cannot seal a field under, is -// refused with the tag to use instead, and a plan built by hand needs a -// non-zero Context. -func TestTagsSpellALabel(t *testing.T) { - type tagged struct { - Email string `stash:"label=users/email"` - } - p, err := PlanFromTags(reflect.TypeOf(tagged{})) - if err != nil { - t.Fatal(err) - } - fields := p.Fields() - if got := fields[0].Context.value(); !reflect.DeepEqual(got, []any{"users", "email"}) { - t.Errorf("label=users/email bound %#v", got) - } - // context= is one part, and a planned field binds a label. - _, err = PlanFromTags(reflect.TypeOf(struct { - Notes string `stash:"context=notes/v1"` - }{})) - if err == nil || !strings.Contains(err.Error(), `context="notes/v1" names one text part`) || !strings.Contains(err.Error(), "use label=") { - t.Errorf("context=notes/v1: err = %v; want the one part refused and label= named", err) - } - // A one-segment label is one part too. - _, err = PlanFromTags(reflect.TypeOf(struct { - Notes string `stash:"label=notes"` - }{})) - if err == nil || !strings.Contains(err.Error(), "one part, not a label") { - t.Errorf("label=notes: err = %v; want the one segment refused", err) - } - for name, typ := range map[string]reflect.Type{ - "both": reflect.TypeOf(struct { - A string `stash:"label=t/a,context=a"` - }{}), - "label twice": reflect.TypeOf(struct { - A string `stash:"label=t/a,label=t/b"` - }{}), - "bad label": reflect.TypeOf(struct { - A string `stash:"label=t/a/"` - }{}), - "empty context": reflect.TypeOf(struct { - A string `stash:"context="` - }{}), - "no context": reflect.TypeOf(struct { - A string `stash:"index=eq"` - }{}), - } { - if _, err := PlanFromTags(typ); err == nil { - t.Errorf("%s: PlanFromTags succeeded; want a refusal", name) - } - } - if _, err := NewPlan(FieldPlan{Field: "A"}); err == nil || !strings.Contains(err.Error(), "needs a context") { - t.Errorf("NewPlan without a context = %v", err) - } -} - -// A repeated option is reported as what the author wrote, not as a mix of -// the two keys. -func TestARepeatedOwnContextOptionNamesItself(t *testing.T) { - _, err := PlanFromTags(reflect.TypeOf(struct { - A string `stash:"context=a,context=b"` - }{})) - if err == nil || !strings.Contains(err.Error(), "context= and context= both given") { - t.Errorf("err = %v, want the repeated option named", err) - } -} - -// A Label owns its segments, as a Context owns its parts: NewLabel copies -// the slice in and Segments copies it out, so neither side can change the -// name the data is keyed under through a shared array. -func TestLabelOwnsItsSegments(t *testing.T) { - segments := []string{"users", "email"} - l, err := NewLabel(segments...) - if err != nil { - t.Fatal(err) - } - segments[1] = "phone" - if l.String() != "users/email" { - t.Errorf("label followed the caller's slice: %s", l) - } - out := l.Segments() - out[0] = "accounts" - if l.String() != "users/email" { - t.Errorf("label followed the returned slice: %s", l) - } -} - -// Equal is the supported comparison: == on two list contexts panics. -func TestContextEqualComparesParts(t *testing.T) { - pair := label(t, "users/email").Context() - if !pair.Equal(label(t, "users/email").Context()) { - t.Error("equal labels compare unequal") - } - if pair.Equal(label(t, "users/phone").Context()) || pair.Equal(MustContext("users/email")) { - t.Error("different contexts compare equal") - } - if !MustContext("users").Equal(label(t, "users").Context()) { - t.Error("a one-segment label is not the bare part") - } - if (Context{}).Equal(pair) || !(Context{}).Equal(Context{}) { - t.Error("the zero Context compares wrongly") - } - defer func() { - if recover() == nil { - t.Error("== on list contexts did not panic; Equal's reason to exist is gone, revisit its doc") - } - }() - _ = pair == label(t, "users/email").Context() -} diff --git a/languages/golang/encrypt/leaf.go b/languages/golang/encrypt/leaf.go deleted file mode 100644 index 37437e322..000000000 --- a/languages/golang/encrypt/leaf.go +++ /dev/null @@ -1,122 +0,0 @@ -package encrypt - -import ( - "database/sql/driver" - "fmt" - - "github.com/cipherstash/vitaminc/bindings/go/vcffi" -) - -// Sealed is one encrypted leaf: the frozen stack-encrypt storage encoding -// (version, keyset id, IV, ZeroKMS tag, ciphertext), exactly what a -// database column holds. It is a distinct type from vcvalue.Sealed on -// purpose: a stack-encrypt leaf is not decryptable by vitaminc-encrypt and -// must never scan or marshal where one belongs. -type Sealed []byte - -// SealedNone is the authenticated marker for an absent value (a nil -// pointer, a Null) inside a ciphertext. -type SealedNone []byte - -// SealedEmptySeq is the authenticated marker for an empty sequence. -type SealedEmptySeq []byte - -// SealedEmptyMap is the authenticated marker for an empty map. -type SealedEmptyMap []byte - -// Value implements driver.Valuer, binding the leaf as a byte column. -func (s Sealed) Value() (driver.Value, error) { return []byte(s), nil } - -// Value implements driver.Valuer. -func (s SealedNone) Value() (driver.Value, error) { return []byte(s), nil } - -// Value implements driver.Valuer. -func (s SealedEmptySeq) Value() (driver.Value, error) { return []byte(s), nil } - -// Value implements driver.Valuer. -func (s SealedEmptyMap) Value() (driver.Value, error) { return []byte(s), nil } - -// Scan implements sql.Scanner, loading a leaf from a byte column. -func (s *Sealed) Scan(src any) error { - b, err := scanBytes("Sealed", src) - *s = b - return err -} - -// Scan implements sql.Scanner. -func (s *SealedNone) Scan(src any) error { - b, err := scanBytes("SealedNone", src) - *s = b - return err -} - -// Scan implements sql.Scanner. -func (s *SealedEmptySeq) Scan(src any) error { - b, err := scanBytes("SealedEmptySeq", src) - *s = b - return err -} - -// Scan implements sql.Scanner. -func (s *SealedEmptyMap) Scan(src any) error { - b, err := scanBytes("SealedEmptyMap", src) - *s = b - return err -} - -// scanBytes copies a driver byte value: drivers may reuse the source slice -// after Scan returns. -func scanBytes(kind string, src any) ([]byte, error) { - switch v := src.(type) { - case []byte: - out := make([]byte, len(v)) - copy(out, v) - return out, nil - case string: - return []byte(v), nil - case nil: - return nil, fmt.Errorf("encrypt: cannot scan NULL into %s", kind) - default: - return nil, fmt.Errorf("encrypt: cannot scan %T into %s", src, kind) - } -} - -// leaves is the vcffi.LeafSet of this binding's leaf types. -var leaves = vcffi.LeafSet{ - Classify: func(v any) (vcffi.LeafKind, []byte, bool) { - switch n := v.(type) { - case Sealed: - return vcffi.LeafSingle, n, true - case SealedNone: - return vcffi.LeafNone, n, true - case SealedEmptySeq: - return vcffi.LeafEmptySeq, n, true - case SealedEmptyMap: - return vcffi.LeafEmptyMap, n, true - default: - return 0, nil, false - } - }, - Make: func(kind vcffi.LeafKind, bytes []byte) any { - switch kind { - case vcffi.LeafSingle: - return Sealed(bytes) - case vcffi.LeafNone: - return SealedNone(bytes) - case vcffi.LeafEmptySeq: - return SealedEmptySeq(bytes) - case vcffi.LeafEmptyMap: - return SealedEmptyMap(bytes) - default: - return nil - } - }, -} - -func marshalCipherText(v any) ([]byte, error) { - return vcffi.MarshalCipherText(leaves, v) -} - -func unmarshalCipherText(buf []byte) (any, error) { - return vcffi.UnmarshalCipherText(leaves, buf) -} diff --git a/languages/golang/encrypt/live_internal_test.go b/languages/golang/encrypt/live_internal_test.go new file mode 100644 index 000000000..504a0423e --- /dev/null +++ b/languages/golang/encrypt/live_internal_test.go @@ -0,0 +1,175 @@ +package encrypt + +import ( + "context" + "errors" + "os" + "testing" + + "github.com/cipherstash/stack/languages/golang/auth" + "github.com/cipherstash/stack/languages/golang/internal/record" +) + +// Round trips through real ZeroKMS key material run when these variables +// are set (CI's `live` job exports them; locally a gitignored mise.local.toml +// can), and each test is skipped unless its own are: +// +// - STACK_ENCRYPT_TEST_CLIENT_ID, STACK_ENCRYPT_TEST_CLIENT_KEY: the +// seeded client (every live test); +// - STACK_ENCRYPT_TEST_CLIENT_ACCESS_KEY, STACK_ENCRYPT_TEST_WORKSPACE_CRN: +// an access key and its workspace, exchanged for a token (every live +// test) — by an auth access-key strategy given to NewCredentials +// (liveClient), and by AutoCredentials from the environment +// (TestLiveAutoCredentialsFromTheEnvironment). There is no raw-token +// variable: the client takes tokens only from auth strategies; +// - STACK_ENCRYPT_TEST_ZEROKMS_URL (optional): the ZeroKMS endpoint, else +// the token's services claim; +// - STACK_ENCRYPT_TEST_CTS_HOST (optional): the authentication endpoint +// the access key is exchanged at, else discovery from the workspace CRN; +// - STACK_ENCRYPT_TEST_OTHER_KEYSET (optional): a second keyset's name. +// +// The round trips through generated code are in live_test.go (package +// encrypt_test, which can import the generated test types this package +// cannot); this file holds the client they share and the one test that +// needs the package's internals. + +func liveClient(t *testing.T) *Client { + t.Helper() + clientID, clientKey := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ID"), os.Getenv("STACK_ENCRYPT_TEST_CLIENT_KEY") + accessKey, crn := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ACCESS_KEY"), os.Getenv("STACK_ENCRYPT_TEST_WORKSPACE_CRN") + url := os.Getenv("STACK_ENCRYPT_TEST_ZEROKMS_URL") + if clientID == "" || clientKey == "" || accessKey == "" || crn == "" { + t.Skip("STACK_ENCRYPT_TEST_{CLIENT_ID,CLIENT_KEY,CLIENT_ACCESS_KEY,WORKSPACE_CRN} not set") + } + guestOrSkip(t) + authGuestOrSkip(t) + // The explicit path: the caller opens the store and the strategy, and + // closes them after the client (cleanups run last-registered first). + store, err := auth.OpenWithoutProfile(t.Context()) + if err != nil { + t.Fatalf("auth.OpenWithoutProfile: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + var strategyOpts []auth.StrategyOption + if cts := os.Getenv("STACK_ENCRYPT_TEST_CTS_HOST"); cts != "" { + strategyOpts = append(strategyOpts, auth.WithAuthBaseURL(cts)) + } + strategy, err := store.AccessKey(t.Context(), crn, accessKey, strategyOpts...) + if err != nil { + t.Fatalf("auth access-key strategy: %v", err) + } + t.Cleanup(func() { + if err := strategy.Close(); err != nil { + t.Errorf("strategy.Close: %v", err) + } + }) + material := []byte(clientKey) + key := NewClientKey(material) + // The credentials a successful NewClient resolved are released by the + // client's Close, once: the wiring only a real load-keyset response can + // reach. NewCredentials itself holds nothing to release — the strategy + // is the caller's — so the spy adds a Close to count. + var released int + creds := credentialsFunc(func(ctx context.Context, opts resolveOptions) (*resolvedCredentials, error) { + r, err := NewCredentials(clientID, key, strategy).resolve(ctx, opts) + if err == nil { + r.Close = func() error { released++; return nil } + } + return r, err + }) + c, err := NewClient(t.Context(), WithCredentials(creds), withZeroKMSURL(url)) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + t.Cleanup(func() { + _ = c.Close() + _ = c.Close() + if released != 1 { + t.Errorf("Close released the credentials %d times, want once", released) + } + // The client never closes the caller's strategy. + if _, err := strategy.Token(context.Background()); err != nil { + t.Errorf("the strategy after Client.Close: %v, want still usable", err) + } + }) + if released != 0 { + t.Fatalf("a successful NewClient released the credentials %d times, want 0", released) + } + // The successful outcome of the consumption contract, which only a + // real load-keyset response can reach: the key is empty and the bytes + // it was built from are zero once the client exists. + if !key.IsZero() { + t.Error("the key still holds material after NewClient succeeded") + } + for i, b := range material { + if b != 0 { + t.Fatalf("byte %d of the key material was not wiped by a successful NewClient", i) + } + } + return c +} + +// The zero-configuration path end to end: NewClient with no options, its +// credentials from AutoCredentials, the token from a real access-key +// exchange — the CI shape of a deployment, with the variables the Rust +// client reads and no developer profile. It drives the record path directly +// because this package's tests cannot import the generated test types. +func TestLiveAutoCredentialsFromTheEnvironment(t *testing.T) { + clientID, clientKey := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ID"), os.Getenv("STACK_ENCRYPT_TEST_CLIENT_KEY") + accessKey, crn := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ACCESS_KEY"), os.Getenv("STACK_ENCRYPT_TEST_WORKSPACE_CRN") + if clientID == "" || clientKey == "" || accessKey == "" || crn == "" { + t.Skip("STACK_ENCRYPT_TEST_{CLIENT_ID,CLIENT_KEY,CLIENT_ACCESS_KEY,WORKSPACE_CRN} not set") + } + guestOrSkip(t) + authGuestOrSkip(t) + // An empty profile directory, so only the environment can answer, and + // none of the developer's own CS_* variables. + cleanEnv(t, t.TempDir()) + if cts := os.Getenv("STACK_ENCRYPT_TEST_CTS_HOST"); cts != "" { + t.Setenv("CS_CTS_HOST", cts) + } else if err := os.Unsetenv("CS_CTS_HOST"); err != nil { // cleanEnv's placeholder + t.Fatal(err) + } + if url := os.Getenv("STACK_ENCRYPT_TEST_ZEROKMS_URL"); url != "" { + t.Setenv("CS_ZEROKMS_HOST", url) + } + t.Setenv(envAccessKey, accessKey) + t.Setenv(envWorkspaceCRN, crn) + t.Setenv(envClientID, clientID) + t.Setenv(envClientKey, clientKey) + + ctx := t.Context() + c, err := NewClient(ctx) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + defer func() { + if err := c.Close(); err != nil { + t.Errorf("Close: %v", err) + } + }() + // Whether memory locks depends on the host; that it is reported, and + // consistently, does not. + if err := c.MemoryLockError(); err != nil && !errors.Is(err, ErrMemoryLock) { + t.Errorf("MemoryLockError = %v, want nil or ErrMemoryLock", err) + } + if c.MemoryLocked() != (c.MemoryLockError() == nil) { + t.Error("MemoryLocked disagrees with MemoryLockError") + } + + plan := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "name", Kind: record.String, Outputs: []record.Output{record.Ciphertext}}}} + sealed, err := c.DefaultKeyset().Seal(ctx, plan, []record.Source{{"name": "alice"}}) + if err != nil { + t.Fatalf("Seal: %v", err) + } + if len(sealed) != 1 || len(sealed[0]["name"].Ciphertext) == 0 { + t.Fatalf("sealed = %v", sealed) + } + back, err := c.Open(ctx, plan, sealed) + if err != nil { + t.Fatalf("Open: %v", err) + } + if len(back) != 1 || back[0]["name"] != "alice" { + t.Fatalf("Open = %#v, want alice", back) + } +} diff --git a/languages/golang/encrypt/live_test.go b/languages/golang/encrypt/live_test.go index 14280958e..fdeb4ffaf 100644 --- a/languages/golang/encrypt/live_test.go +++ b/languages/golang/encrypt/live_test.go @@ -1,329 +1,119 @@ -package encrypt +package encrypt_test import ( "bytes" - "context" "errors" "os" "reflect" - "strings" "testing" - "github.com/cipherstash/stack/languages/golang/auth" - "github.com/cipherstash/vitaminc/bindings/go/vcvalue" + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" ) -// Round trips through real ZeroKMS key material. No CI harness runs these -// yet: one that boots zerokms-server and exports the variables below is -// tracked in CIP-4024. Until then they run locally when the variables are -// set (from a gitignored mise.local.toml, say), and each test is skipped -// unless its own are: -// -// - STACK_ENCRYPT_TEST_CLIENT_ID, STACK_ENCRYPT_TEST_CLIENT_KEY: the -// seeded client (every live test); -// - STACK_ENCRYPT_TEST_CLIENT_ACCESS_KEY, STACK_ENCRYPT_TEST_WORKSPACE_CRN: -// an access key and its workspace, exchanged for a token (every live -// test) — by a auth access-key strategy given to NewCredentials -// (liveClient), and by AutoCredentials from the environment -// (TestLiveAutoCredentialsFromTheEnvironment). There is no raw-token -// variable: the client takes tokens only from auth strategies; -// - STACK_ENCRYPT_TEST_ZEROKMS_URL (optional): the ZeroKMS endpoint, else -// the token's services claim; -// - STACK_ENCRYPT_TEST_CTS_HOST (optional): the authentication endpoint -// the access key is exchanged at, else discovery from the workspace CRN; -// - STACK_ENCRYPT_TEST_OTHER_KEYSET (optional): a second keyset's name. +// Round trips through real ZeroKMS, through the generated API. See +// live_internal_test.go for the variables that enable them. -func liveClient(t *testing.T) *Client { - t.Helper() - clientID, clientKey := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ID"), os.Getenv("STACK_ENCRYPT_TEST_CLIENT_KEY") - accessKey, crn := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ACCESS_KEY"), os.Getenv("STACK_ENCRYPT_TEST_WORKSPACE_CRN") - url := os.Getenv("STACK_ENCRYPT_TEST_ZEROKMS_URL") - if clientID == "" || clientKey == "" || accessKey == "" || crn == "" { - t.Skip("STACK_ENCRYPT_TEST_{CLIENT_ID,CLIENT_KEY,CLIENT_ACCESS_KEY,WORKSPACE_CRN} not set") - } - guestOrSkip(t) - authGuestOrSkip(t) - // The explicit path: the caller opens the store and the strategy, and - // closes them after the client (cleanups run last-registered first). - store, err := auth.OpenWithoutProfile(t.Context()) - if err != nil { - t.Fatalf("auth.OpenWithoutProfile: %v", err) - } - t.Cleanup(func() { _ = store.Close() }) - var strategyOpts []auth.StrategyOption - if cts := os.Getenv("STACK_ENCRYPT_TEST_CTS_HOST"); cts != "" { - strategyOpts = append(strategyOpts, auth.WithAuthBaseURL(cts)) - } - strategy, err := store.AccessKey(t.Context(), crn, accessKey, strategyOpts...) - if err != nil { - t.Fatalf("auth access-key strategy: %v", err) - } - t.Cleanup(func() { - if err := strategy.Close(); err != nil { - t.Errorf("strategy.Close: %v", err) - } - }) - material := []byte(clientKey) - key := NewClientKey(material) - // The credentials a successful NewClient resolved are released by the - // client's Close, once: the wiring only a real load-keyset response can - // reach. NewCredentials itself holds nothing to release — the strategy - // is the caller's — so the spy adds a Close to count. - var released int - creds := credentialsFunc(func(ctx context.Context, opts resolveOptions) (*resolvedCredentials, error) { - r, err := NewCredentials(clientID, key, strategy).resolve(ctx, opts) - if err == nil { - r.Close = func() error { released++; return nil } - } - return r, err - }) - c, err := NewClient(t.Context(), WithCredentials(creds), withZeroKMSURL(url)) - if err != nil { - t.Fatalf("NewClient: %v", err) - } - t.Cleanup(func() { - _ = c.Close() - _ = c.Close() - if released != 1 { - t.Errorf("Close released the credentials %d times, want once", released) - } - // The client never closes the caller's strategy. - if _, err := strategy.Token(context.Background()); err != nil { - t.Errorf("the strategy after Client.Close: %v, want still usable", err) - } - }) - if released != 0 { - t.Fatalf("a successful NewClient released the credentials %d times, want 0", released) - } - // The successful outcome of the consumption contract, which only a - // real load-keyset response can reach: the key is empty and the bytes - // it was built from are zero once the client exists. - if !key.IsZero() { - t.Error("the key still holds material after NewClient succeeded") - } - for i, b := range material { - if b != 0 { - t.Fatalf("byte %d of the key material was not wiped by a successful NewClient", i) - } - } - return c -} - -type liveUser struct { - ID int64 `stash:"-"` - Age uint32 `stash:"label=users/age,index=eq;ore"` - Email string `stash:"label=users/email,index=eq;match"` -} - -func TestLiveValueRoundTrip(t *testing.T) { - c := liveClient(t) - ctx := t.Context() - cipher := c.DefaultKeyset() - aad := []byte("users/v1") - in := map[string]any{"name": "alice", "age": uint32(34), "note": vcvalue.Plain{V: "clear"}} - - c.transport.sends.Store(0) - ct, err := cipher.Encrypt(ctx, in, aad) - if err != nil { - t.Fatal(err) - } - if n := c.transport.sends.Load(); n != 1 { - t.Errorf("encrypt made %d ZeroKMS calls, want 1", n) - } - fields := ct.(map[string]any) - if _, ok := fields["name"].(Sealed); !ok { - t.Fatalf("name sealed as %T", fields["name"]) - } - if fields["note"] != (vcvalue.Plain{V: "clear"}) { - t.Fatalf("passthrough came back as %v", fields["note"]) - } - - for name, open := range map[string]func() (any, error){ - "bound": func() (any, error) { return cipher.Decrypt(ctx, ct, aad) }, - "client": func() (any, error) { return c.Decrypt(ctx, ct, aad) }, - } { - pt, err := open() - if err != nil { - t.Fatalf("%s decrypt: %v", name, err) - } - want := vcvalue.Object{{Key: "age", Value: uint32(34)}, {Key: "name", Value: "alice"}, {Key: "note", Value: vcvalue.Plain{V: "clear"}}} - if !reflect.DeepEqual(pt, want) { - t.Fatalf("%s decrypt = %#v", name, pt) - } - } - if _, err := cipher.Decrypt(ctx, ct, []byte("wrong")); err == nil { - t.Fatal("wrong AAD decrypted") - } - // The default keyset's id is what the leaves carry: the bound cipher of - // that id opens them too. - defID, err := cipher.KeysetID(ctx) - if err != nil { - t.Fatalf("resolve the default keyset: %v", err) - } - if _, err := c.Keyset(defID).Decrypt(ctx, ct, aad); err != nil { - t.Fatalf("decrypt under the default keyset by id: %v", err) - } +var livePeople = []testusers.User{ + {ID: 1, Age: 34, Email: "alice@example.com", Notes: "likes cats", Internal: "never stored"}, + {ID: 2, Age: 29, Email: "bob@example.com", Notes: "likes dogs"}, } func TestLiveRecordsAndTerms(t *testing.T) { - c := liveClient(t) + c := encrypt.LiveClient(t) ctx := t.Context() cipher := c.DefaultKeyset() - users := []liveUser{{1, 34, "alice@example.com"}, {2, 29, "bob@example.com"}} - c.transport.sends.Store(0) - records, err := cipher.EncryptRecords(ctx, users) + encrypt.ResetSends(c) + encrypted, err := testusers.Encrypt(ctx, cipher, livePeople) if err != nil { t.Fatal(err) } - if n := c.transport.sends.Load(); n != 1 { - t.Errorf("EncryptRecords made %d ZeroKMS calls for %d rows, want 1", n, len(users)) + if n := encrypt.Sends(c); n != 1 { + t.Errorf("Encrypt made %d ZeroKMS calls for %d rows, want 1", n, len(livePeople)) } - if len(records) != 2 || len(records[0]["Age"].Equality) != 32 || records[0]["Email"].Match == nil || records[0]["Age"].Ore == nil { - t.Fatalf("records = %+v", records) + if len(encrypted) != 2 || len(encrypted[0].Age.Equality) != 32 || encrypted[0].Email.Match == nil || encrypted[0].Age.Ore == nil || encrypted[0].ID != 1 { + t.Fatalf("encrypted = %+v", encrypted) } - probe, err := cipher.Term(ctx, uint32(34), label(t, "users/age").Context(), Equality) + probe, err := testusers.Fields.Age.Equality(ctx, cipher, 34) if err != nil { t.Fatal(err) } - if !probe.(EqualityTerm).Equal(records[0]["Age"].Equality) { + if !probe.Equal(encrypted[0].Age.Equality) { t.Error("probe does not equal the stored equality term") } - if probe.(EqualityTerm).Equal(records[1]["Age"].Equality) { + if probe.Equal(encrypted[1].Age.Equality) { t.Error("probe equals another value's term") } - var back []liveUser - if err := cipher.DecryptRecords(ctx, records, &back); err != nil { + want := append([]testusers.User(nil), livePeople...) + want[0].Internal = "" // left out: never stored + encrypt.ResetSends(c) + back, err := testusers.Decrypt(ctx, cipher, encrypted) + if err != nil { t.Fatal(err) } - for i := range users { - users[i].ID = 0 // not part of the record + if n := encrypt.Sends(c); n != 1 { + t.Errorf("Decrypt made %d ZeroKMS calls, want 1", n) } - if !reflect.DeepEqual(back, users) { - t.Fatalf("decrypted %+v, want %+v", back, users) + if !reflect.DeepEqual(back, want) { + t.Fatalf("decrypted %+v, want %+v", back, want) } - var one liveUser - if err := c.DecryptRecord(ctx, records[1], &one); err != nil || one.Email != "bob@example.com" { - t.Fatalf("DecryptRecord: %v %+v", err, one) + // The client opens too, under whichever keyset sealed each row. + if back, err := testusers.Decrypt(ctx, c, encrypted[1:]); err != nil || back[0].Email != "bob@example.com" { + t.Fatalf("Decrypt through the client: %v %+v", err, back) } - // A context extension is part of the identity. - ext, err := cipher.EncryptRecords(ctx, users, ExtendContext(uint64(7))) + // A context extension is part of the identity: the write, the query + // and the read all go through the extended cipher. + tenant7, tenant8 := cipher.Extend(uint64(7)), cipher.Extend(uint64(8)) + ext, err := testusers.Encrypt(ctx, tenant7, livePeople) if err != nil { t.Fatal(err) } - if err := cipher.DecryptRecords(ctx, ext, &back); !errors.Is(err, ErrForbidden) && !errors.Is(err, ErrAuthentication) { + if _, err := testusers.Decrypt(ctx, cipher, ext); !errors.Is(err, encrypt.ErrForbidden) && !errors.Is(err, encrypt.ErrAuthentication) { t.Fatalf("extended record opened without its extension: %v", err) } - if err := cipher.DecryptRecords(ctx, ext, &back, ExtendContext(uint64(7))); err != nil { + if _, err := testusers.Decrypt(ctx, tenant7, ext); err != nil { t.Fatalf("extended record with its extension: %v", err) } - - // A probe takes the same option, and matches only the rows written - // under it: not another tenant's, and not the unextended ones. - tenant7, tenant8 := ExtendContext(uint64(7)), ExtendContext(uint64(8)) - other, err := cipher.EncryptRecords(ctx, users, tenant8) + other, err := testusers.Encrypt(ctx, tenant8, livePeople) if err != nil { t.Fatal(err) } - scoped, err := cipher.Term(ctx, "bob@example.com", label(t, "users/email").Context(), Equality, tenant7) + scoped, err := testusers.Fields.Email.Equality(ctx, tenant7, "bob@example.com") if err != nil { t.Fatal(err) } - if !scoped.(EqualityTerm).Equal(ext[1]["Email"].Equality) { + if !scoped.Equal(ext[1].Email.Equality) { t.Error("tenant probe does not equal the term written under the same extension") } - if scoped.(EqualityTerm).Equal(other[1]["Email"].Equality) { - t.Error("tenant probe equals another tenant's term") - } - if scoped.(EqualityTerm).Equal(records[1]["Email"].Equality) { - t.Error("tenant probe equals the unextended term") - } - unscoped, err := cipher.Term(ctx, "bob@example.com", label(t, "users/email").Context(), Equality) - if err != nil { - t.Fatal(err) - } - if unscoped.(EqualityTerm).Equal(ext[1]["Email"].Equality) { - t.Error("an unextended probe equals a tenant's term") - } -} - -// An explicit plan round-trips a struct that carries no tags, and a record -// is only readable under the plan it was written under. -func TestLiveExplicitPlanRoundTrip(t *testing.T) { - c := liveClient(t) - ctx := t.Context() - cipher := c.DefaultKeyset() - type generated struct { // no tags, as protobuf output has none - Age uint32 - Email string - } - plan, err := NewPlan( - FieldPlan{Field: "Age", Context: label(t, "users/age").Context(), Terms: []TermKind{Equality, Ore}}, - FieldPlan{Field: "Email", Context: label(t, "users/email").Context(), Terms: []TermKind{Equality, Match}}, - ) - if err != nil { - t.Fatal(err) - } - users := []generated{{34, "alice@example.com"}, {29, "bob@example.com"}} - - records, err := cipher.EncryptRecords(ctx, users, WithPlan(plan)) - if err != nil { - t.Fatal(err) - } - if len(records) != 2 || len(records[0]["Age"].Equality) != 32 || records[0]["Email"].Match == nil { - t.Fatalf("records = %+v", records) - } - var back []generated - if err := cipher.DecryptRecords(ctx, records, &back, WithPlan(plan)); err != nil { - t.Fatal(err) - } - if !reflect.DeepEqual(back, users) { - t.Fatalf("decrypted %+v, want %+v", back, users) - } - var one generated - if err := c.DecryptRecord(ctx, records[1], &one, WithPlan(plan)); err != nil || one.Email != "bob@example.com" { - t.Fatalf("DecryptRecord: %v %+v", err, one) - } - - // A plan naming a field the record does not carry is refused before - // any key is requested. - other, err := NewPlan(FieldPlan{Field: "Email", Name: "email", Context: label(t, "users/email").Context()}) - if err != nil { - t.Fatal(err) - } - c.transport.sends.Store(0) - err = cipher.DecryptRecords(ctx, records, &back, WithPlan(other)) - if err == nil || !strings.Contains(err.Error(), `no ciphertext for field "email"`) { - t.Fatalf("mismatched plan: %v", err) - } - if n := c.transport.sends.Load(); n != 0 { - t.Errorf("mismatched plan made %d ZeroKMS calls, want 0", n) + if scoped.Equal(other[1].Email.Equality) || scoped.Equal(encrypted[1].Email.Equality) { + t.Error("tenant probe equals another tenant's or the unextended term") } } func TestLiveForeignKeysetIsRefusedBeforeRetrieval(t *testing.T) { - c := liveClient(t) + c := encrypt.LiveClient(t) ctx := t.Context() other := os.Getenv("STACK_ENCRYPT_TEST_OTHER_KEYSET") if other == "" { t.Skip("STACK_ENCRYPT_TEST_OTHER_KEYSET not set") } - ct, err := c.Keyset(KeysetName(other)).Encrypt(ctx, "tenant b", nil) + encrypted, err := testusers.EncryptDocument(ctx, c.Keyset(encrypt.KeysetName(other)), []testusers.Document{{Title: "tenant b", Body: "x"}}) if err != nil { t.Fatal(err) } - c.transport.sends.Store(0) - if _, err := c.DefaultKeyset().Decrypt(ctx, ct, nil); !errors.Is(err, ErrForeignKeyset) { - t.Fatalf("default cipher opened another keyset's leaf: %v", err) + encrypt.ResetSends(c) + if _, err := testusers.DecryptDocument(ctx, c.DefaultKeyset(), encrypted); !errors.Is(err, encrypt.ErrForeignKeyset) { + t.Fatalf("default cipher opened another keyset's row: %v", err) } - if n := c.transport.sends.Load(); n != 0 { - t.Errorf("a foreign leaf cost %d ZeroKMS calls before refusal", n) + if n := encrypt.Sends(c); n != 0 { + t.Errorf("a foreign row cost %d ZeroKMS calls before refusal", n) } - if pt, err := c.Decrypt(ctx, ct, nil); err != nil || pt != "tenant b" { - t.Fatalf("client decrypt of the other keyset: %v %v", pt, err) + if docs, err := testusers.DecryptDocument(ctx, c, encrypted); err != nil || docs[0].Title != "tenant b" { + t.Fatalf("client decrypt of the other keyset: %v %+v", err, docs) } } @@ -332,82 +122,12 @@ func TestLiveForeignKeysetIsRefusedBeforeRetrieval(t *testing.T) { // wiped by se_dealloc — so between calls the guest holds only the client // key and its keyset cache. func TestPlaintextDoesNotRemainInGuestMemoryAfterEncrypt(t *testing.T) { - c := liveClient(t) - ctx := t.Context() + c := encrypt.LiveClient(t) const plaintext = "residency-probe-4111-b1c2d3e4f5" - if _, err := c.DefaultKeyset().Encrypt(ctx, plaintext, nil); err != nil { + if _, err := testusers.Encrypt(t.Context(), c.DefaultKeyset(), []testusers.User{{Notes: plaintext}}); err != nil { t.Fatalf("Encrypt: %v", err) } - mem := c.inst.module.Memory() - view, ok := mem.Read(0, mem.Size()) - if !ok { - t.Fatal("cannot read guest memory") - } - if n := bytes.Count(view, []byte(plaintext)); n != 0 { + if n := bytes.Count(encrypt.GuestMemory(t, c), []byte(plaintext)); n != 0 { t.Fatalf("plaintext found %d times in guest memory after Encrypt returned", n) } } - -// The zero-configuration path end to end: NewClient with no options, its -// credentials from AutoCredentials, the token from a real access-key -// exchange — the CI shape of a deployment, with the variables the Rust -// client reads and no developer profile. -func TestLiveAutoCredentialsFromTheEnvironment(t *testing.T) { - clientID, clientKey := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ID"), os.Getenv("STACK_ENCRYPT_TEST_CLIENT_KEY") - accessKey, crn := os.Getenv("STACK_ENCRYPT_TEST_CLIENT_ACCESS_KEY"), os.Getenv("STACK_ENCRYPT_TEST_WORKSPACE_CRN") - if clientID == "" || clientKey == "" || accessKey == "" || crn == "" { - t.Skip("STACK_ENCRYPT_TEST_{CLIENT_ID,CLIENT_KEY,CLIENT_ACCESS_KEY,WORKSPACE_CRN} not set") - } - guestOrSkip(t) - authGuestOrSkip(t) - // An empty profile directory, so only the environment can answer, and - // none of the developer's own CS_* variables. - cleanEnv(t, t.TempDir()) - if cts := os.Getenv("STACK_ENCRYPT_TEST_CTS_HOST"); cts != "" { - t.Setenv("CS_CTS_HOST", cts) - } else if err := os.Unsetenv("CS_CTS_HOST"); err != nil { // cleanEnv's placeholder - t.Fatal(err) - } - if url := os.Getenv("STACK_ENCRYPT_TEST_ZEROKMS_URL"); url != "" { - t.Setenv("CS_ZEROKMS_HOST", url) - } - t.Setenv(envAccessKey, accessKey) - t.Setenv(envWorkspaceCRN, crn) - t.Setenv(envClientID, clientID) - t.Setenv(envClientKey, clientKey) - - ctx := t.Context() - c, err := NewClient(ctx) - if err != nil { - t.Fatalf("NewClient: %v", err) - } - defer func() { - if err := c.Close(); err != nil { - t.Errorf("Close: %v", err) - } - }() - // Whether memory locks depends on the host; that it is reported, and - // consistently, does not. - if err := c.MemoryLockError(); err != nil && !errors.Is(err, ErrMemoryLock) { - t.Errorf("MemoryLockError = %v, want nil or ErrMemoryLock", err) - } - if c.MemoryLocked() != (c.MemoryLockError() == nil) { - t.Error("MemoryLocked disagrees with MemoryLockError") - } - - aad := []byte("users/v1") - ct, err := c.DefaultKeyset().Encrypt(ctx, "alice", aad) - if err != nil { - t.Fatalf("Encrypt: %v", err) - } - if _, ok := ct.(Sealed); !ok { - t.Fatalf("sealed as %T", ct) - } - pt, err := c.Decrypt(ctx, ct, aad) - if err != nil { - t.Fatalf("Decrypt: %v", err) - } - if pt != "alice" { - t.Fatalf("Decrypt = %#v, want %q", pt, "alice") - } -} diff --git a/languages/golang/encrypt/memory_test.go b/languages/golang/encrypt/memory_test.go index ba4da6c30..9a0d7e598 100644 --- a/languages/golang/encrypt/memory_test.go +++ b/languages/golang/encrypt/memory_test.go @@ -89,7 +89,7 @@ func TestRequireLockedMemoryRefusesAnUnlockableGuest(t *testing.T) { limited := !probe.IsFallback() _, err := NewClient(context.Background(), WithCredentials(newTestCredentials("6a70bd18-99ac-4650-b104-37eec3a15b09", NewClientKey([]byte("00")), staticToken("t"))), - WithGuest(wasiProbe), + withGuest(wasiProbe), WithRequireLockedMemory(), ) if !errors.Is(err, ErrMemoryLock) { @@ -159,7 +159,7 @@ func TestRequireLockedMemoryRefusesACallerStoreUnlocked(t *testing.T) { key := NewClientKey([]byte("00")) _, err = NewClient(ctx, WithCredentials(NewCredentials("6a70bd18-99ac-4650-b104-37eec3a15b09", key, strategy)), - WithGuest(wasiProbe), + withGuest(wasiProbe), withZeroKMSURL("https://zerokms.invalid"), WithRequireLockedMemory(), ) diff --git a/languages/golang/encrypt/options.go b/languages/golang/encrypt/options.go index fe725f103..7a6bd19c5 100644 --- a/languages/golang/encrypt/options.go +++ b/languages/golang/encrypt/options.go @@ -63,9 +63,10 @@ func WithTransport(rt http.RoundTripper) ClientOption { return func(o *clientOptions) { o.transport = rt } } -// WithGuest overrides the embedded wasm module. Nil means the embedded one, -// the default. -func WithGuest(wasm []byte) ClientOption { +// withGuest overrides the embedded wasm module, for the tests that drive a +// probe module or the deterministic test build. The package embeds the one +// build a program runs; a program cannot swap it. +func withGuest(wasm []byte) ClientOption { return func(o *clientOptions) { o.guest = wasm } } diff --git a/languages/golang/encrypt/options_test.go b/languages/golang/encrypt/options_test.go index 571ea1401..c4b066258 100644 --- a/languages/golang/encrypt/options_test.go +++ b/languages/golang/encrypt/options_test.go @@ -28,7 +28,7 @@ func TestClientOptionsSetTheirField(t *testing.T) { withZeroKMSURL("https://second.example"), WithKeysetCacheSize(4096), WithTransport(rt), - WithGuest(wasm), + withGuest(wasm), WithRequireLockedMemory(), } { opt(&got) diff --git a/languages/golang/encrypt/order_live_test.go b/languages/golang/encrypt/order_live_test.go deleted file mode 100644 index 7747ac37c..000000000 --- a/languages/golang/encrypt/order_live_test.go +++ /dev/null @@ -1,169 +0,0 @@ -package encrypt - -import ( - "bytes" - "cmp" - "context" - "fmt" - "math/rand" - "reflect" - "testing" - "testing/quick" -) - -// Property tests of term ordering: random plaintexts, terms derived by the -// guest, comparison in Go. ORE is probabilistic — a wrong comparator or a -// wrong derivation still agrees with plaintext order on many pairs — so a -// fixed vector set says little; hundreds of random pairs per type say -// more. The terms come from the guest's index key, which needs a loaded -// keyset, so these run under the live harness (skipped without -// credentials) — see live_test.go. - -// orderProperty checks, for random pairs of T, that the Go comparison of -// their terms agrees with the plaintext order and that a term compares -// equal to itself (derivation is deterministic). gen, when given, replaces -// quick's generator for T. -func orderProperty[T any](t *testing.T, cipher *Cipher, kind TermKind, less func(a, b T) int, gen ...func(*rand.Rand) T) { - t.Helper() - ctx := context.Background() - context := MustContext(fmt.Sprintf("prop/%s/%T", kind, *new(T))) - term := func(v T) []byte { - t.Helper() - out, err := cipher.Term(ctx, v, context, kind) - if err != nil { - t.Fatalf("Term(%v): %v", v, err) - } - switch tt := out.(type) { - case OreTerm: - return tt - case OpeTerm: - return tt - default: - t.Fatalf("Term returned %T", out) - return nil - } - } - compare := func(a, b []byte) int { - if kind == Ore { - return OreTerm(a).Compare(OreTerm(b)) - } - return OpeTerm(a).Compare(OpeTerm(b)) - } - sign := func(n int) int { - return cmp.Compare(n, 0) - } - holds := func(a, b T) bool { - ta, tb := term(a), term(b) - if compare(ta, ta) != 0 || compare(tb, tb) != 0 { - t.Logf("a term does not compare equal to itself: %v", a) - return false - } - if !bytes.Equal(ta, term(a)) { - t.Logf("derivation is not deterministic for %v", a) - return false - } - want, got := sign(less(a, b)), sign(compare(ta, tb)) - if got != want { - t.Logf("%v vs %v: plaintext order %d, term order %d", a, b, want, got) - return false - } - return sign(compare(tb, ta)) == -want - } - cfg := &quick.Config{MaxCount: 300, Rand: rand.New(rand.NewSource(int64(kind)))} - if len(gen) > 0 { - cfg.Values = func(args []reflect.Value, r *rand.Rand) { - for i := range args { - args[i] = reflect.ValueOf(gen[0](r)) - } - } - } - if err := quick.Check(holds, cfg); err != nil { - t.Fatal(err) - } -} - -// Neighbouring values are where a comparator that mishandles the last -// differing bit shows; quick's uniform generator almost never produces -// them, so they are checked explicitly alongside. -func adjacentProperty[T any](t *testing.T, cipher *Cipher, kind TermKind, values []T, less func(a, b T) int) { - t.Helper() - ctx := context.Background() - context := MustContext(fmt.Sprintf("prop/%s/%T", kind, *new(T))) - terms := make([][]byte, len(values)) - for i, v := range values { - out, err := cipher.Term(ctx, v, context, kind) - if err != nil { - t.Fatalf("Term(%v): %v", v, err) - } - terms[i] = reflect.ValueOf(out).Bytes() - } - for i := range values { - for j := range values { - var got int - if kind == Ore { - got = OreTerm(terms[i]).Compare(OreTerm(terms[j])) - } else { - got = OpeTerm(terms[i]).Compare(OpeTerm(terms[j])) - } - if want := cmp.Compare(less(values[i], values[j]), 0); got != want { - t.Errorf("%v vs %v: plaintext order %d, term order %d", values[i], values[j], want, got) - } - } - } -} - -// collatedAlphabet holds characters that the ORE and OPE string encodings -// keep as they are. Before deriving a term, cllw-ore's orderize_string -// decomposes each character canonically and drops anything that is not -// alphanumeric, whitespace or ASCII punctuation. So two strings order by -// their UTF-8 bytes only when that collation leaves both unchanged: a -// private-use character is dropped, and a precomposed Hangul syllable or -// an accented letter decomposes. The non-ASCII letters here have no -// canonical decomposition, so multi-byte UTF-8 ordering is still covered. -var collatedAlphabet = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~ßжω中") - -func collatedString(r *rand.Rand) string { - out := make([]rune, r.Intn(24)) - for i := range out { - out[i] = collatedAlphabet[r.Intn(len(collatedAlphabet))] - } - return string(out) -} - -func TestLiveTermOrderIsPlaintextOrder(t *testing.T) { - c := liveClient(t) - cipher := c.DefaultKeyset() - for _, kind := range []TermKind{Ore, Ope} { - t.Run(kind.String(), func(t *testing.T) { - t.Run("uint32", func(t *testing.T) { - orderProperty(t, cipher, kind, cmp.Compare[uint32]) - adjacentProperty(t, cipher, kind, []uint32{0, 1, 2, 255, 256, 257, 65535, 65536, 1<<31 - 1, 1 << 31, 1<<32 - 2, 1<<32 - 1}, cmp.Compare[uint32]) - }) - t.Run("uint64", func(t *testing.T) { - orderProperty(t, cipher, kind, cmp.Compare[uint64]) - adjacentProperty(t, cipher, kind, []uint64{0, 1, 1<<32 - 1, 1 << 32, 1<<63 - 1, 1 << 63, 1<<64 - 1}, cmp.Compare[uint64]) - }) - t.Run("int64", func(t *testing.T) { - orderProperty(t, cipher, kind, cmp.Compare[int64]) - adjacentProperty(t, cipher, kind, []int64{-1 << 63, -1<<63 + 1, -2, -1, 0, 1, 2, 1<<63 - 1}, cmp.Compare[int64]) - }) - t.Run("string", func(t *testing.T) { - // Strings order by the UTF-8 bytes of their collated form; a - // prefix orders before its extensions. See collatedAlphabet. - orderProperty(t, cipher, kind, func(a, b string) int { return bytes.Compare([]byte(a), []byte(b)) }, collatedString) - adjacentProperty(t, cipher, kind, []string{"", "a", "aa", "ab", "b", "ba", "ß", "ßa", "中"}, func(a, b string) int { return bytes.Compare([]byte(a), []byte(b)) }) - // Collation drops a control or private-use character, and - // strips the accent from a decomposed letter, so the terms - // cannot tell these pairs apart. - same := func(a, b string) int { return 0 } - adjacentProperty(t, cipher, kind, []string{"", "\x7f"}, same) - adjacentProperty(t, cipher, kind, []string{"ab", "a\ue000b"}, same) - adjacentProperty(t, cipher, kind, []string{"e", "é"}, same) - }) - t.Run("bytes", func(t *testing.T) { - orderProperty(t, cipher, kind, bytes.Compare) - adjacentProperty(t, cipher, kind, [][]byte{{}, {0}, {0, 0}, {0, 1}, {1}, {255}, {255, 0}}, bytes.Compare) - }) - }) - } -} diff --git a/languages/golang/encrypt/order_test.go b/languages/golang/encrypt/order_test.go new file mode 100644 index 000000000..d1b9b8131 --- /dev/null +++ b/languages/golang/encrypt/order_test.go @@ -0,0 +1,194 @@ +package encrypt_test + +import ( + "bytes" + "cmp" + "context" + "math/rand" + "reflect" + "testing" + "testing/quick" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" +) + +// Property tests of term ordering: random plaintexts, terms derived by the +// guest through the generated Probe fields, comparison in Go. ORE is +// probabilistic — a wrong comparator or a wrong derivation still agrees +// with plaintext order on many pairs — so a fixed vector set says little; +// hundreds of random pairs per type say more. The index key is the +// deterministic build's, so these run hermetically. + +// termOf derives one ORE or OPE term for v. +type termOf[T any] func(ctx context.Context, c *encrypt.Cipher, v T) ([]byte, error) + +func oreOf[T any](f func(context.Context, *encrypt.Cipher, T) (encrypt.OreTerm, error)) termOf[T] { + return func(ctx context.Context, c *encrypt.Cipher, v T) ([]byte, error) { return f(ctx, c, v) } +} + +func opeOf[T any](f func(context.Context, *encrypt.Cipher, T) (encrypt.OpeTerm, error)) termOf[T] { + return func(ctx context.Context, c *encrypt.Cipher, v T) ([]byte, error) { return f(ctx, c, v) } +} + +func compareTerms(ope bool, a, b []byte) int { + if ope { + return encrypt.OpeTerm(a).Compare(encrypt.OpeTerm(b)) + } + return encrypt.OreTerm(a).Compare(encrypt.OreTerm(b)) +} + +// orderProperty checks, for random pairs of T, that the Go comparison of +// their terms agrees with the plaintext order and that a term compares +// equal to itself (derivation is deterministic). gen, when given, replaces +// quick's generator for T. +func orderProperty[T any](t *testing.T, cipher *encrypt.Cipher, ope bool, derive termOf[T], less func(a, b T) int, gen ...func(*rand.Rand) T) { + t.Helper() + ctx := context.Background() + term := func(v T) []byte { + t.Helper() + out, err := derive(ctx, cipher, v) + if err != nil { + t.Fatalf("term(%v): %v", v, err) + } + return out + } + sign := func(n int) int { return cmp.Compare(n, 0) } + holds := func(a, b T) bool { + ta, tb := term(a), term(b) + if compareTerms(ope, ta, ta) != 0 || compareTerms(ope, tb, tb) != 0 { + t.Logf("a term does not compare equal to itself: %v", a) + return false + } + if !bytes.Equal(ta, term(a)) { + t.Logf("derivation is not deterministic for %v", a) + return false + } + want, got := sign(less(a, b)), sign(compareTerms(ope, ta, tb)) + if got != want { + t.Logf("%v vs %v: plaintext order %d, term order %d", a, b, want, got) + return false + } + return sign(compareTerms(ope, tb, ta)) == -want + } + seed := int64(3) + if ope { + seed = 4 + } + cfg := &quick.Config{MaxCount: 300, Rand: rand.New(rand.NewSource(seed))} + if len(gen) > 0 { + cfg.Values = func(args []reflect.Value, r *rand.Rand) { + for i := range args { + args[i] = reflect.ValueOf(gen[0](r)) + } + } + } + if err := quick.Check(holds, cfg); err != nil { + t.Fatal(err) + } +} + +// Neighbouring values are where a comparator that mishandles the last +// differing bit shows; quick's uniform generator almost never produces +// them, so they are checked explicitly alongside. +func adjacentProperty[T any](t *testing.T, cipher *encrypt.Cipher, ope bool, derive termOf[T], values []T, less func(a, b T) int) { + t.Helper() + ctx := context.Background() + terms := make([][]byte, len(values)) + for i, v := range values { + out, err := derive(ctx, cipher, v) + if err != nil { + t.Fatalf("term(%v): %v", v, err) + } + terms[i] = out + } + for i := range values { + for j := range values { + if got, want := compareTerms(ope, terms[i], terms[j]), cmp.Compare(less(values[i], values[j]), 0); got != want { + t.Errorf("%v vs %v: plaintext order %d, term order %d", values[i], values[j], want, got) + } + } + } +} + +// collatedAlphabet holds characters that the ORE and OPE string encodings +// keep as they are. Before deriving a term, cllw-ore's orderize_string +// decomposes each character canonically and drops anything that is not +// alphanumeric, whitespace or ASCII punctuation. So two strings order by +// their UTF-8 bytes only when that collation leaves both unchanged: a +// private-use character is dropped, and a precomposed Hangul syllable or +// an accented letter decomposes. The non-ASCII letters here have no +// canonical decomposition, so multi-byte UTF-8 ordering is still covered. +var collatedAlphabet = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~ßжω中") + +func collatedString(r *rand.Rand) string { + out := make([]rune, r.Intn(24)) + for i := range out { + out[i] = collatedAlphabet[r.Intn(len(collatedAlphabet))] + } + return string(out) +} + +func TestTermOrderIsPlaintextOrder(t *testing.T) { + c := deterministicClient(t) + cipher := c.DefaultKeyset() + f := testusers.ProbeFields + byteOrder := func(a, b string) int { return bytes.Compare([]byte(a), []byte(b)) } + same := func(a, b string) int { return 0 } + for _, ope := range []bool{false, true} { + name := "ore" + if ope { + name = "ope" + } + t.Run(name, func(t *testing.T) { + t.Run("uint32", func(t *testing.T) { + d := oreOf(f.U32.Ore) + if ope { + d = opeOf(f.U32.Ope) + } + orderProperty(t, cipher, ope, d, cmp.Compare[uint32]) + adjacentProperty(t, cipher, ope, d, []uint32{0, 1, 2, 255, 256, 257, 65535, 65536, 1<<31 - 1, 1 << 31, 1<<32 - 2, 1<<32 - 1}, cmp.Compare[uint32]) + }) + t.Run("uint64", func(t *testing.T) { + d := oreOf(f.U64.Ore) + if ope { + d = opeOf(f.U64.Ope) + } + orderProperty(t, cipher, ope, d, cmp.Compare[uint64]) + adjacentProperty(t, cipher, ope, d, []uint64{0, 1, 1<<32 - 1, 1 << 32, 1<<63 - 1, 1 << 63, 1<<64 - 1}, cmp.Compare[uint64]) + }) + t.Run("int64", func(t *testing.T) { + d := oreOf(f.I64.Ore) + if ope { + d = opeOf(f.I64.Ope) + } + orderProperty(t, cipher, ope, d, cmp.Compare[int64]) + adjacentProperty(t, cipher, ope, d, []int64{-1 << 63, -1<<63 + 1, -2, -1, 0, 1, 2, 1<<63 - 1}, cmp.Compare[int64]) + }) + t.Run("string", func(t *testing.T) { + d := oreOf(f.S.Ore) + if ope { + d = opeOf(f.S.Ope) + } + // Strings order by the UTF-8 bytes of their collated form; a + // prefix orders before its extensions. See collatedAlphabet. + orderProperty(t, cipher, ope, d, byteOrder, collatedString) + adjacentProperty(t, cipher, ope, d, []string{"", "a", "aa", "ab", "b", "ba", "ß", "ßa", "中"}, byteOrder) + // Collation drops a control or private-use character, and + // strips the accent from a decomposed letter, so the terms + // cannot tell these pairs apart. + adjacentProperty(t, cipher, ope, d, []string{"", "\x7f"}, same) + adjacentProperty(t, cipher, ope, d, []string{"ab", "ab"}, same) + adjacentProperty(t, cipher, ope, d, []string{"e", "é"}, same) + }) + t.Run("bytes", func(t *testing.T) { + d := oreOf(f.B.Ore) + if ope { + d = opeOf(f.B.Ope) + } + orderProperty(t, cipher, ope, d, bytes.Compare) + adjacentProperty(t, cipher, ope, d, [][]byte{{}, {0}, {0, 0}, {0, 1}, {1}, {255}, {255, 0}}, bytes.Compare) + }) + }) + } +} diff --git a/languages/golang/encrypt/plan/doc.go b/languages/golang/encrypt/plan/doc.go deleted file mode 100644 index 6cd6b86cd..000000000 --- a/languages/golang/encrypt/plan/doc.go +++ /dev/null @@ -1,91 +0,0 @@ -// Package plan builds a record [encrypt.Plan] from what a domain -// schema already says about its fields, through a policy written in Go. -// -// Storage decisions do not belong in the schema. The schema carries facts — -// a field's name, its kind, and its annotations, such as Fideslang -// `data_categories` — and a [Policy], a pure function of a field's [Fact], -// decides what becomes of it: [Encrypt] into a [Target], [Plaintext], or -// [Fail]. Policies are ordinary values and compose: [When] is a rule, -// [FirstOf] takes the first rule that matches, and [Policy.OrElse] refines a -// shared base per message. -// -// var category = plan.Key("fides.data_categories") -// -// var Base = plan.FirstOf( -// plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(encrypt.Equality))), -// plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(encrypt.Equality, encrypt.Match))), -// plan.When(category.Under("user"), plan.Encrypt(plan.EQL())), -// ) -// -// var Individuals = plan.ForMessage(&Individual{}, plan.Table("individuals"), -// plan.FirstOf( -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), -// plan.Column("medicare_number")), -// ).OrElse(Base), -// ) -// -// var individuals = plan.MustPlanFor(source, Individuals) -// // cipher.EncryptRecords(ctx, rows, encrypt.WithPlan(individuals)) -// -// # Facts -// -// A [Source] makes the facts for a message. A protobuf source (planned in -// CIP-4088) reads descriptors and their custom options, and needs nothing -// from this package beyond [Fact], [Source] and [Key]. Any function -// returning facts is one: -// -// var source = plan.SourceFunc(func(msg any) ([]plan.Fact, error) { -// return []plan.Fact{ -// {Field: "id", GoField: "ID"}, -// {Field: "email", GoField: "Email", Annotations: []plan.Annotation{ -// {Key: "fides.data_categories", Values: []string{"user.contact.email"}}}}, -// {Field: "medicare_no", GoField: "MedicareNo", Annotations: []plan.Annotation{ -// {Key: "fides.data_categories", Values: []string{"user.government_id"}}}}, -// }, nil -// }) -// -// [Message.Build] takes facts directly, so a generator or a test can build -// a plan without a source at all; [PlanFor] also checks the plan binds to -// the message's Go type. -// -// # Contexts -// -// A field's context is its AAD, its ZeroKMS data-key binding and its terms' -// PRF context, fixed when data is first written. For an [EQL] target the -// context is the column identity, "
/" ([Identifier]): the -// table is the message's [Table], which is required and never derived from -// the message's name, and the column is the column the field is first -// stored in. A rule sets that column with [Column] (the field's schema name -// by default), which for an EQL target sets the identity too. Once data is -// written the identity must never change, so after a database rename -// (ALTER TABLE ... RENAME COLUMN) the rule stores into the new column and -// pins the old identity with [Identity]: -// -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), -// plan.Column("medicare_num"), plan.Identity("medicare_number")) -// -// A [Custom] target supplies its context itself; [Column] names only its -// record key, and [Identity] is refused. -// -// Nothing on the write path notices a context that changed: a rename with -// no pin simply writes new rows under a new context. Check them in with a -// golden test ([github.com/cipherstash/stack/languages/golang/encrypt/plan/plantest.Golden]), -// which snapshots what the policy stores each field as and fails, naming -// the pin, when a context changes: -// -// func TestIndividualsPolicy(t *testing.T) { -// plantest.Golden(t, source, Individuals) -// } -// -// # Failing closed -// -// A field with annotations that no rule decides is an error when the plan -// is built ([ErrUnmatched]), naming the field and its annotations; there is -// no built-in default, so a catch-all, Plaintext included, is written in the -// policy. A field with no annotations that no rule names is not the -// policy's concern: it is left out of the plan and stored as it is. A -// message the policy encrypts nothing of has no plan to build -// ([ErrNothingEncrypted]): its records are stored without one. Build plans -// at startup with [MustPlanFor], so a gap stops the process before it -// writes anything. -package plan diff --git a/languages/golang/encrypt/plan/fact.go b/languages/golang/encrypt/plan/fact.go deleted file mode 100644 index 6955da6fa..000000000 --- a/languages/golang/encrypt/plan/fact.go +++ /dev/null @@ -1,114 +0,0 @@ -package plan - -import ( - "fmt" - "slices" - "strings" -) - -// Fact is what the SDK knows about one field of a message: where it is, -// what it is, and the annotations the domain schema put on it. It is -// source-agnostic. A [Source] makes facts from something that describes a -// message, such as a protobuf descriptor; a policy reads them and never -// learns where they came from. -type Fact struct { - // Message is the message's (or struct's) full name, for errors. - Message string - // Field is the field's schema name: the proto field name, or, for a - // struct, the Go field name in snake_case. It is the column a field - // encrypts into unless a rule pins another ([Column]). - Field string - // GoField is the Go struct field the plan binds to. Field when empty. - GoField string - // Number is the proto field number; 0 when the source has none. - Number int32 - // Kind is the field's kind in the source's own spelling ("string", - // "int64", ...); informational, for rules that match on it. - Kind string - // Annotations are the facts proper: the domain schema's classification - // of the field, such as its Fideslang data categories. A field with no - // annotations is not the policy's concern unless a rule names it. - Annotations []Annotation -} - -// Annotation is one annotation on a field: a key and its values. For a -// protobuf extension the key is the extension's full name and the values -// are its (repeated) string values. -type Annotation struct { - Key string - Values []string -} - -// Values returns the values of every annotation on the field under key, -// in order. -func (f Fact) Values(key string) []string { - var out []string - for _, a := range f.Annotations { - if a.Key == key { - out = append(out, a.Values...) - } - } - return out -} - -// hasValue reports whether any value under key satisfies pred, without -// collecting the values: the matchers run once per rule per field. -func (f Fact) hasValue(key string, pred func(string) bool) bool { - for _, a := range f.Annotations { - if a.Key != key { - continue - } - if slices.ContainsFunc(a.Values, pred) { - return true - } - } - return false -} - -// goField is the Go struct field the fact binds to. -func (f Fact) goField() string { - if f.GoField != "" { - return f.GoField - } - return f.Field -} - -// String names the field — and the Go field it binds to, when that is -// spelled differently — and its annotations, the way errors name them. -func (f Fact) String() string { - var b strings.Builder - if f.Message != "" { - b.WriteString(f.Message) - b.WriteByte('.') - } - b.WriteString(f.Field) - if f.GoField != "" && f.GoField != f.Field { - b.WriteString(" (") - b.WriteString(f.GoField) - b.WriteByte(')') - } - if len(f.Annotations) > 0 { - b.WriteString(" [") - for i, a := range f.Annotations { - if i > 0 { - b.WriteString("; ") - } - fmt.Fprintf(&b, "%s=%s", a.Key, strings.Join(a.Values, ",")) - } - b.WriteByte(']') - } - return b.String() -} - -// Source makes the facts for a message. msg is whatever [ForMessage] was -// given, such as a proto.Message for the protobuf source planned in -// CIP-4088. Facts are returned in field order. -type Source interface { - Facts(msg any) ([]Fact, error) -} - -// SourceFunc adapts a function to a [Source]. -type SourceFunc func(msg any) ([]Fact, error) - -// Facts calls f. -func (f SourceFunc) Facts(msg any) ([]Fact, error) { return f(msg) } diff --git a/languages/golang/encrypt/plan/message.go b/languages/golang/encrypt/plan/message.go deleted file mode 100644 index 2ebbf9e9d..000000000 --- a/languages/golang/encrypt/plan/message.go +++ /dev/null @@ -1,240 +0,0 @@ -package plan - -import ( - "errors" - "fmt" - "reflect" - - "github.com/cipherstash/stack/languages/golang/encrypt" -) - -var ( - // ErrUnmatched is a field that carries annotations no rule of the - // policy decides. There is no built-in default: a catch-all, even - // Plaintext, must be written in the policy. - ErrUnmatched = errors.New("no rule decides the field") - // ErrRefused is a field the policy decided to [Fail]. - ErrRefused = errors.New("the policy refuses the field") - // ErrInvalid is a decision that cannot become a plan field: no target, - // a pinned column or identity on a field that is not encrypted, an - // identity on a Custom target, an empty context, a '/' in a column - // identity, or two EQL fields sharing one identity. - ErrInvalid = errors.New("invalid decision") - // ErrNothingEncrypted is a message the policy encrypts no field of: - // every classified field decided Plaintext, or none classified. Such a - // message has no plan to build, and its records are stored without - // one — a [encrypt.Plan] always seals at least one field. - ErrNothingEncrypted = errors.New("the policy encrypts no field of the message") -) - -// Table names the table a message's records are stored in: the first half -// of every EQL field's column identity. It is required, and never derived -// from the message's name, because it is part of every context and a -// guess (a pluralisation) would be permanent. -type Table string - -// Message is a policy for one message type: the message, its table, and -// the rules its fields are decided by. A Message is a value; build it once -// (a package-level var) and build its plan at startup with [MustPlanFor]. -type Message struct { - msg any - table Table - policy Policy -} - -// ForMessage scopes policy to msg, stored in table. msg is what the -// [Source] reads facts from, such as a proto.Message for a protobuf -// source. Refine a shared base per message with OrElse: -// -// var Individuals = plan.ForMessage(&Individual{}, plan.Table("individuals"), -// plan.FirstOf( -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), -// plan.Column("medicare_number")), -// ).OrElse(Base), -// ) -func ForMessage(msg any, table Table, policy Policy) Message { - return Message{msg: msg, table: table, policy: policy} -} - -// Msg returns the message the policy is for. -func (m Message) Msg() any { return m.msg } - -// Table returns the message's table. -func (m Message) Table() Table { return m.table } - -// Decide runs the message's policy on one field. -func (m Message) Decide(f Fact) (Decision, bool) { return m.policy.Decide(f) } - -// Build decides every field of facts and returns the plan: one field per -// Encrypt decision, in fact order. A field no rule matches is plaintext -// when it has no annotations (not the policy's concern) and ErrUnmatched -// when it has any; a Fail decision is ErrRefused. Every failing field is -// reported, not only the first, each naming the field and its facts. -// -// Pure: no I/O, no client. Build is what [PlanFor] runs after reading the -// facts, and what a generator or a golden test runs on facts it holds. -func (m Message) Build(facts []Fact) (encrypt.Plan, error) { - if m.table == "" { - return encrypt.Plan{}, fmt.Errorf("plan: %s: a message needs a Table", messageName(m, facts)) - } - var fields []encrypt.FieldPlan - var errs []error - // An EQL identity is one column's context: two fields sharing one would - // bind each other's ciphertexts and terms. NewPlan refuses a shared - // record key, but with Identity the identity can be shared without one. - identities := map[string]Fact{} - for _, f := range facts { - fp, id, planned, err := m.field(f) - if err == nil && id != "" { - if prev, dup := identities[id]; dup { - err = fmt.Errorf("%w: identity %q is already field %s's", ErrInvalid, id, prev.Field) - } else { - identities[id] = f - } - } - if err != nil { - errs = append(errs, fmt.Errorf("plan: %s: %w", f, err)) - continue - } - if planned { - fields = append(fields, fp) - } - } - if len(errs) > 0 { - return encrypt.Plan{}, errors.Join(errs...) - } - if len(fields) == 0 { - return encrypt.Plan{}, fmt.Errorf("plan: %s: %w; a message with nothing to encrypt needs no plan", messageName(m, facts), ErrNothingEncrypted) - } - p, err := encrypt.NewPlan(fields...) - if err != nil { - return encrypt.Plan{}, fmt.Errorf("plan: %s: %w", messageName(m, facts), err) - } - return p, nil -} - -// field decides one field: its plan field, its EQL identity ("" for a -// Custom target), and whether it is planned. -func (m Message) field(f Fact) (encrypt.FieldPlan, string, bool, error) { - none := func(err error) (encrypt.FieldPlan, string, bool, error) { - return encrypt.FieldPlan{}, "", false, err - } - d, ok := m.policy.Decide(f) - if !ok { - if len(f.Annotations) == 0 { - return none(nil) - } - return none(ErrUnmatched) - } - switch d.verdict { - case sealed: - case plaintext: - if d.column != "" { - return none(fmt.Errorf("%w: Column(%q) pinned on a Plaintext field", ErrInvalid, d.column)) - } - if d.identity != "" { - return none(fmt.Errorf("%w: Identity(%q) pinned on a Plaintext field", ErrInvalid, d.identity)) - } - return none(nil) - case fail: - return none(fmt.Errorf("%w: %s", ErrRefused, d.reason)) - default: - return none(fmt.Errorf("%w: the zero Decision", ErrInvalid)) - } - if d.target == nil { - return none(fmt.Errorf("%w: Encrypt with no target", ErrInvalid)) - } - column := f.Field - if d.column != "" { - column = d.column - } - if column == "" { - return none(fmt.Errorf("%w: the field has no name to store it under", ErrInvalid)) - } - _, custom := d.target.(customTarget) - identity := column - if custom { - // A Custom target's context is its own: there is no identity to - // pin, and a pin would read as if it took effect. - if d.identity != "" { - return none(fmt.Errorf("%w: Identity(%q) on %v, whose context is fixed", ErrInvalid, d.identity, d.target)) - } - identity = "" - } else if d.identity != "" { - identity = d.identity - } - // An EQL target's context is Identifier.Label(), which refuses a table - // or column that would not render as itself — a '/' among them, since it - // would read as two names. A Custom target's column is only the record - // key, so it may contain anything. - context, err := d.target.Context(Identifier{Table: string(m.table), Column: identity}) - if err != nil { - // Both errors stay reachable: ErrInvalid for the policy's caller, and - // the target's own (a *encrypt.LabelError, say) for one that - // wants to know which name was wrong. - return none(fmt.Errorf("%w: target %v: %w", ErrInvalid, d.target, err)) - } - return encrypt.FieldPlan{ - Field: f.goField(), - Name: column, - Context: context, - Terms: d.target.Terms(), - }, identity, true, nil -} - -func messageName(m Message, facts []Fact) string { - if len(facts) > 0 && facts[0].Message != "" { - return facts[0].Message - } - return fmt.Sprintf("%T", m.msg) -} - -// PlanFor reads m's facts from src, builds its plan (see [Message.Build]) -// and, when m's message is a struct or a pointer to one, checks the plan -// binds to it: every planned field is an exported, direct field of the -// type. A fact whose GoField the type does not have — a typo in a source, -// a generated field renamed — is then an error here, not at the first -// record call. -func PlanFor(src Source, m Message) (encrypt.Plan, error) { - if src == nil { - return encrypt.Plan{}, errors.New("plan: PlanFor needs a Source") - } - facts, err := src.Facts(m.msg) - if err != nil { - return encrypt.Plan{}, err - } - p, err := m.Build(facts) - if err != nil { - return encrypt.Plan{}, err - } - if t := structType(m.msg); t != nil { - if err := p.Validate(t); err != nil { - return encrypt.Plan{}, fmt.Errorf("plan: %s: %w", messageName(m, facts), err) - } - } - return p, nil -} - -// structType is msg's struct type, through one pointer, or nil when msg is -// not a struct: nothing to bind a plan to. -func structType(msg any) reflect.Type { - t := reflect.TypeOf(msg) - if t != nil && t.Kind() == reflect.Pointer { - t = t.Elem() - } - if t == nil || t.Kind() != reflect.Struct { - return nil - } - return t -} - -// MustPlanFor is [PlanFor] for startup: it panics when the policy does not -// decide every classified field, or the plan does not bind to the message, -// so a policy gap stops the process before it writes anything. -func MustPlanFor(src Source, m Message) encrypt.Plan { - p, err := PlanFor(src, m) - if err != nil { - panic(err) - } - return p -} diff --git a/languages/golang/encrypt/plan/plan_test.go b/languages/golang/encrypt/plan/plan_test.go deleted file mode 100644 index 2abb03e85..000000000 --- a/languages/golang/encrypt/plan/plan_test.go +++ /dev/null @@ -1,541 +0,0 @@ -package plan_test - -import ( - "errors" - "fmt" - "reflect" - "strings" - "testing" - - se "github.com/cipherstash/stack/languages/golang/encrypt" - "github.com/cipherstash/stack/languages/golang/encrypt/plan" - "github.com/cipherstash/stack/languages/golang/internal/factstest" -) - -var category = plan.Key("fides.data_categories") - -var base = plan.FirstOf( - plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(se.Equality))), - plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(se.Equality, se.Match))), - plan.When(category.Under("user"), plan.Encrypt(plan.EQL())), -) - -type individual struct { - ID int64 - Email string `facts:"fides.data_categories=user.contact.email"` - Name string `facts:"fides.data_categories=user.name"` - MedicareNo string `facts:"fides.data_categories=user.government_id"` - Country string `facts:"fides.data_categories=system.operations"` -} - -var individuals = plan.ForMessage(&individual{}, plan.Table("individuals"), - plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(se.Equality, se.Ore)), plan.Column("medicare_number")), - plan.When(category.Under("system"), plan.Plaintext()), - ).OrElse(base), -) - -func TestPolicyBuildsThePlan(t *testing.T) { - p, err := plan.PlanFor(factstest.StructTags, individuals) - if err != nil { - t.Fatal(err) - } - // Columns are the schema's spelling of the Go field: what the Rust - // derive binds and the database names. - want := []se.FieldPlan{ - {Field: "Email", Name: "email", Context: label(t, "individuals/email").Context(), Terms: []se.TermKind{se.Equality, se.Match}}, - {Field: "Name", Name: "name", Context: label(t, "individuals/name").Context()}, - // The per-message rule wins over the base's government_id rule. - {Field: "MedicareNo", Name: "medicare_number", Context: label(t, "individuals/medicare_number").Context(), Terms: []se.TermKind{se.Equality, se.Ore}}, - } - if got := p.Fields(); !reflect.DeepEqual(got, want) { - t.Fatalf("fields =\n%+v\nwant\n%+v", got, want) - } -} - -// A classified field no rule decides fails the build, naming the field and -// its facts, and every such field is reported. -func TestUnmatchedFactFailsTheBuild(t *testing.T) { - type patient struct { - ID int64 - Fingerprint []byte `facts:"fides.data_categories=user.biometric.fingerprint"` - Diagnosis string `facts:"fides.data_categories=user.health"` - Email string `facts:"fides.data_categories=user.contact.email"` - } - narrow := plan.FirstOf( - plan.When(category.Under("user.contact"), plan.Encrypt(plan.EQL(se.Equality))), - ) - m := plan.ForMessage(patient{}, "patients", narrow) - _, err := plan.PlanFor(factstest.StructTags, m) - if !errors.Is(err, plan.ErrUnmatched) { - t.Fatalf("err = %v, want ErrUnmatched", err) - } - for _, want := range []string{ - "fingerprint (Fingerprint)", "user.biometric.fingerprint", - "diagnosis (Diagnosis)", "user.health", - } { - if !strings.Contains(err.Error(), want) { - t.Errorf("error %q does not name %q", err, want) - } - } - if strings.Contains(err.Error(), "Email") || strings.Contains(err.Error(), ".id") { - t.Errorf("error %q names a field that was decided or unclassified", err) - } - func() { - defer func() { - if r := recover(); r == nil { - t.Error("MustPlanFor did not panic on an unmatched fact") - } - }() - plan.MustPlanFor(factstest.StructTags, m) - }() - - // A catch-all written in the policy closes the gap; Plaintext counts. - closed := plan.ForMessage(patient{}, "patients", narrow.OrElse( - plan.When(category.Present(), plan.Plaintext()), - )) - p := plan.MustPlanFor(factstest.StructTags, closed) - if got := p.Fields(); len(got) != 1 || got[0].Field != "Email" { - t.Fatalf("fields = %+v, want Email only", got) - } -} - -// A message the policy encrypts nothing of has no plan to build: the error -// says so, rather than encrypt's "at least one field". -func TestNothingEncryptedIsItsOwnError(t *testing.T) { - type audit struct { - ID int64 - Kind string `facts:"fides.data_categories=system.operations"` - } - m := plan.ForMessage(audit{}, "audits", plan.When(category.Under("system"), plan.Plaintext())) - _, err := plan.PlanFor(factstest.StructTags, m) - if !errors.Is(err, plan.ErrNothingEncrypted) { - t.Fatalf("err = %v, want ErrNothingEncrypted", err) - } - if !strings.Contains(err.Error(), "audit") || !strings.Contains(err.Error(), "needs no plan") { - t.Errorf("error %q does not name the message and say what to do", err) - } -} - -// PlanFor binds the plan to the message's type: a fact naming a Go field -// the type does not have fails at build, not at the first record call. -func TestPlanForRefusesAFieldTheMessageDoesNotHave(t *testing.T) { - facts := []plan.Fact{{Message: "acme.v1.Individual", Field: "email", GoField: "EMail", - Annotations: []plan.Annotation{{Key: "k", Values: []string{"v"}}}}} - src := plan.SourceFunc(func(any) ([]plan.Fact, error) { return facts, nil }) - m := plan.ForMessage(&individual{}, "individuals", plan.When(plan.Key("k").Present(), plan.Encrypt(plan.EQL()))) - _, err := plan.PlanFor(src, m) - if err == nil || !strings.Contains(err.Error(), "EMail") || !strings.Contains(err.Error(), "not an exported field") { - t.Fatalf("err = %v, want the unbound field named", err) - } - // With no message to bind to, Build alone cannot know, and does not try. - if _, err := plan.ForMessage(nil, "individuals", plan.When(plan.Key("k").Present(), plan.Encrypt(plan.EQL()))).Build(facts); err != nil { - t.Fatalf("Build with no message: %v", err) - } -} - -// An unclassified field is not the policy's concern, but a rule may still -// name it. -func TestUnclassifiedFieldsAreLeftOutUnlessNamed(t *testing.T) { - facts := []plan.Fact{ - {Message: "acme.v1.Individual", Field: "id", GoField: "Id", Number: 1, Kind: "int64"}, - {Message: "acme.v1.Individual", Field: "notes", GoField: "Notes", Number: 2, Kind: "string"}, - } - m := plan.ForMessage(nil, "individuals", plan.When(plan.Field("notes"), plan.Encrypt(plan.EQL()))) - p, err := m.Build(facts) - if err != nil { - t.Fatal(err) - } - want := []se.FieldPlan{{Field: "Notes", Name: "notes", Context: label(t, "individuals/notes").Context()}} - if got := p.Fields(); !reflect.DeepEqual(got, want) { - t.Fatalf("fields = %+v, want %+v", got, want) - } -} - -// A context is fixed at first write. Pinning the column keeps it, and the -// record key, through a field rename (proto or Go): on a column never -// renamed in the database, Column sets the identity too. -func TestColumnPinSurvivesRenames(t *testing.T) { - gov := []plan.Annotation{{Key: "fides.data_categories", Values: []string{"user.government_id"}}} - before := []plan.Fact{{Message: "acme.v1.Individual", Field: "medicare_number", GoField: "MedicareNumber", Annotations: gov}} - after := []plan.Fact{{Message: "acme.v1.Individual", Field: "medicare_no", GoField: "MedicareNo", Annotations: gov}} - - v1 := plan.ForMessage(nil, "individuals", base) - v2 := plan.ForMessage(nil, "individuals", plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(se.Equality)), plan.Column("medicare_number")), - ).OrElse(base)) - - p1, err := v1.Build(before) - if err != nil { - t.Fatal(err) - } - p2, err := v2.Build(after) - if err != nil { - t.Fatal(err) - } - f1, f2 := p1.Fields()[0], p2.Fields()[0] - want := label(t, "individuals/medicare_number").Context() - if !f1.Context.Equal(want) || !f2.Context.Equal(f1.Context) { - t.Fatalf("contexts %v, %v: want both individuals/medicare_number", f1.Context, f2.Context) - } - if f2.Name != "medicare_number" || f2.Field != "MedicareNo" { - t.Fatalf("pinned field = %+v", f2) - } - // Without the pin, the rename would have changed the context. - unpinned, err := v1.Build(after) - if err != nil { - t.Fatal(err) - } - if got := unpinned.Fields()[0].Context; !got.Equal(label(t, "individuals/medicare_no").Context()) { - t.Fatalf("unpinned context = %v", got) - } -} - -// A database rename moves the record key, never the identity: new writes go -// to the new column, under the context existing rows were written with. -func TestIdentityKeepsTheContextThroughAColumnRename(t *testing.T) { - gov := []plan.Annotation{{Key: "fides.data_categories", Values: []string{"user.government_id"}}} - facts := []plan.Fact{{Message: "acme.v1.Individual", Field: "medicare_no", GoField: "MedicareNo", Annotations: gov}} - eq := plan.Encrypt(plan.EQL(se.Equality)) - for name, tc := range map[string]struct { - opts []plan.RuleOption - key, context string - }{ - // Neither: both are the field's schema name. - "defaults": {nil, "medicare_no", "individuals/medicare_no"}, - // Column alone, on a field never renamed in the database: both. - "column": {[]plan.RuleOption{plan.Column("medicare_number")}, "medicare_number", "individuals/medicare_number"}, - // ALTER TABLE individuals RENAME COLUMN medicare_number TO medicare_num. - "renamed column": {[]plan.RuleOption{plan.Column("medicare_num"), plan.Identity("medicare_number")}, "medicare_num", "individuals/medicare_number"}, - // The column renamed to the field's own name. - "identity alone": {[]plan.RuleOption{plan.Identity("medicare_number")}, "medicare_no", "individuals/medicare_number"}, - } { - p, err := plan.ForMessage(nil, "individuals", plan.When(plan.Field("medicare_no"), eq, tc.opts...)).Build(facts) - if err != nil { - t.Errorf("%s: %v", name, err) - continue - } - want := []se.FieldPlan{{Field: "MedicareNo", Name: tc.key, Context: label(t, tc.context).Context(), Terms: []se.TermKind{se.Equality}}} - if got := p.Fields(); !reflect.DeepEqual(got, want) { - t.Errorf("%s: fields = %+v, want %+v", name, got, want) - } - } -} - -func TestContextsByTarget(t *testing.T) { - facts := []plan.Fact{ - {Field: "email", GoField: "Email", Annotations: []plan.Annotation{{Key: "k", Values: []string{"eql"}}}}, - {Field: "blob", GoField: "Blob", Annotations: []plan.Annotation{{Key: "k", Values: []string{"custom"}}}}, - } - k := plan.Key("k") - m := plan.ForMessage(nil, "users", plan.FirstOf( - plan.When(k.Is("eql"), plan.Encrypt(plan.EQL(se.Equality))), - plan.When(k.Is("custom"), plan.Encrypt(plan.Custom("tenant-blobs/v1", se.Ope)), plan.Column("blob_v1")), - )) - p, err := m.Build(facts) - if err != nil { - t.Fatal(err) - } - want := []se.FieldPlan{ - {Field: "Email", Name: "email", Context: label(t, "users/email").Context(), Terms: []se.TermKind{se.Equality}}, - // A custom target's context is its own; the pin names the record key only. - {Field: "Blob", Name: "blob_v1", Context: label(t, "tenant-blobs/v1").Context(), Terms: []se.TermKind{se.Ope}}, - } - if got := p.Fields(); !reflect.DeepEqual(got, want) { - t.Fatalf("fields =\n%+v\nwant\n%+v", got, want) - } - // A '/' in a custom target's column is only a '/' in a record key: no - // identity to make ambiguous. - slashed := plan.ForMessage(nil, "users", plan.When(k.Is("custom"), plan.Encrypt(plan.Custom("tenant-blobs/v1")), plan.Column("blob/v1"))) - p, err = slashed.Build(facts[1:]) - if err != nil { - t.Fatal(err) - } - if got := p.Fields()[0].Name; got != "blob/v1" { - t.Errorf("custom record key = %q, want blob/v1", got) - } -} - -func TestBuildRefusesMalformedDecisions(t *testing.T) { - classified := []plan.Annotation{{Key: "k", Values: []string{"v"}}} - fact := []plan.Fact{{Field: "a", Annotations: classified}} - // Each case fails for its own reason: a sentinel, or the message the - // reason is spelled by where encrypt reports it. - for name, tc := range map[string]struct { - table plan.Table - policy plan.Policy - want error - msg string - }{ - "no table": {"", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL())), nil, "needs a Table"}, - "slash in table": {"a/b", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL())), nil, "contains '/'"}, - "slash in column": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL()), plan.Column("x/y")), plan.ErrInvalid, "contains '/'"}, - "empty field name": {"t", plan.When(plan.Kind(""), plan.Encrypt(plan.EQL())), plan.ErrInvalid, "no name"}, - "fail": {"t", plan.When(plan.Field("a"), plan.Fail("biometrics are never stored")), plan.ErrRefused, ""}, - "nil target": {"t", plan.When(plan.Field("a"), plan.Encrypt(nil)), plan.ErrInvalid, "no target"}, - "column on plain": {"t", plan.When(plan.Field("a"), plan.Plaintext(), plan.Column("c")), plan.ErrInvalid, "Plaintext"}, - "identity on plain": {"t", plan.When(plan.Field("a"), plan.Plaintext(), plan.Identity("c")), plan.ErrInvalid, "Plaintext"}, - "identity on custom": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.Custom("t/ctx")), plan.Identity("c")), plan.ErrInvalid, "context is fixed"}, - "one-part custom": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.Custom("ctx"))), nil, "one part, not a label"}, - "unplain custom": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.Custom("t/7up"))), plan.ErrInvalid, "is not a label"}, - "slash in identity": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL()), plan.Column("c"), plan.Identity("x/y")), plan.ErrInvalid, "contains '/'"}, - // Identifier.Label() refuses more than '/': every reason a segment is - // not plain, named as the table or the column identity it came from. - "digit in table": {"2024_events", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL())), plan.ErrInvalid, `table "2024_events"`}, - "digit in column": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL()), plan.Column("2fa_secret")), plan.ErrInvalid, `column identity "2fa_secret"`}, - "b64 in column": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL()), plan.Column("b64:x")), plan.ErrInvalid, "another descriptor form"}, - "paren in column": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL()), plan.Column("a(b")), plan.ErrInvalid, "reserves"}, - "invisible in table": {"users\u200b", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL())), plan.ErrInvalid, "invisible"}, - "zero decision": {"t", plan.When(plan.Field("a"), plan.Decision{}), plan.ErrInvalid, "zero Decision"}, - "empty context": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.Custom(""))), plan.ErrInvalid, "empty context"}, - "nil policy": {"t", nil, plan.ErrUnmatched, ""}, - "nothing encrypted": {"t", plan.When(plan.Field("a"), plan.Plaintext()), plan.ErrNothingEncrypted, "needs no plan"}, - "term kind unknown": {"t", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL(se.TermKind(9)))), nil, "unknown term kind"}, - "nil policies skip": {"t", plan.FirstOf(nil, nil), plan.ErrUnmatched, ""}, - "fail names reason": {"t", plan.When(plan.Field("a"), plan.Fail("no biometrics")), plan.ErrRefused, "no biometrics"}, - "matcher composites": {"t", plan.When(plan.All(plan.Field("a"), plan.Not(plan.Kind("string"))), plan.Fail("x")), plan.ErrRefused, ""}, - } { - facts := fact - if name == "empty field name" { - facts = []plan.Fact{{Field: "", Annotations: classified}} - } - _, err := plan.ForMessage(nil, tc.table, tc.policy).Build(facts) - if err == nil { - t.Errorf("%s: built", name) - continue - } - if tc.want != nil && !errors.Is(err, tc.want) { - t.Errorf("%s: err = %v, want %v", name, err, tc.want) - } - if tc.msg != "" && !strings.Contains(err.Error(), tc.msg) { - t.Errorf("%s: err = %q, want it to say %q", name, err, tc.msg) - } - } - _, err := plan.ForMessage(nil, "t", plan.When(plan.Field("a"), plan.Fail("no biometrics"))).Build(fact) - if !strings.Contains(err.Error(), "no biometrics") { - t.Errorf("fail error %q does not carry its reason", err) - } - // Two fields pinned to one column are one record name twice. - two := []plan.Fact{{Field: "a", Annotations: classified}, {Field: "b", Annotations: classified}} - if _, err := plan.ForMessage(nil, "t", plan.When(plan.Any(plan.Field("a"), plan.Field("b")), plan.Encrypt(plan.EQL()), plan.Column("c"))).Build(two); err == nil { - t.Error("two fields pinned to one column built") - } - // Two columns with one identity would bind each other's ciphertexts. - _, err = plan.ForMessage(nil, "t", plan.FirstOf( - plan.When(plan.Field("a"), plan.Encrypt(plan.EQL())), - plan.When(plan.Field("b"), plan.Encrypt(plan.EQL()), plan.Identity("a")), - )).Build(two) - if !errors.Is(err, plan.ErrInvalid) || !strings.Contains(err.Error(), `identity "a" is already field a's`) { - t.Errorf("two fields sharing an identity: err = %v", err) - } - // Custom targets may share a context: it is the policy's to choose, and - // the guest, not NewPlan, refuses two fields under one label when a - // record call runs. - if _, err := plan.ForMessage(nil, "t", plan.When(plan.Any(plan.Field("a"), plan.Field("b")), plan.Encrypt(plan.Custom("blobs/ctx")))).Build(two); err != nil { - t.Errorf("two custom fields sharing a context: %v", err) - } -} - -func TestKeyMatchers(t *testing.T) { - f := plan.Fact{Annotations: []plan.Annotation{ - {Key: "fides.data_categories", Values: []string{"user.contactless", "user.contact.email"}}, - {Key: "other", Values: []string{"x"}}, - }} - for m, want := range map[string]bool{ - "user": true, - "user.contact": true, - "user.contact.email": true, - "user.contact.email.work": false, - "user.contac": false, - "system": false, - } { - if got := category.Under(m)(f); got != want { - t.Errorf("Under(%q) = %v, want %v", m, got, want) - } - } - if !category.Is("user.contactless")(f) || category.Is("user")(f) { - t.Error("Is matches other than exactly") - } - if !plan.Key("other").Present()(f) || plan.Key("absent").Present()(f) { - t.Error("Present") - } - if got := f.Values("other"); !reflect.DeepEqual(got, []string{"x"}) { - t.Errorf("Values = %v", got) - } - // A prefix that could read as a catch-all but match nothing is refused - // when the rule is written, not silently dead. - for _, prefix := range []string{"", "user.", ".user", "a..b"} { - func() { - defer func() { - if recover() == nil { - t.Errorf("Under(%q) did not panic", prefix) - } - }() - category.Under(prefix) - }() - } -} - -// The combinators refuse a nil matcher when the rule is written, as When -// does, naming the combinator; and they hold their own copy of the -// matchers, so a later write to the caller's slice changes nothing. -func TestCombinatorsRefuseNilAndCopyTheirMatchers(t *testing.T) { - var unset plan.Matcher - for name, build := range map[string]func(){ - "Any": func() { plan.Any(plan.Field("a"), unset) }, - "All": func() { plan.All(unset) }, - "Not": func() { plan.Not(unset) }, - // With no matchers, All would match every field and Any none: a - // rule built from an empty slice would silently decide everything - // or nothing. - "Any()": func() { plan.Any() }, - "All()": func() { plan.All([]plan.Matcher{}...) }, - } { - func() { - defer func() { - r := recover() - if r == nil { - t.Errorf("%s did not panic", name) - } else if !strings.Contains(fmt.Sprint(r), "plan."+strings.TrimSuffix(name, "()")) { - t.Errorf("%s panicked with %v, which does not name it", name, r) - } - }() - build() - }() - } - ms := []plan.Matcher{plan.Field("a")} - any, all := plan.Any(ms...), plan.All(ms...) - ms[0] = plan.Field("b") - a, b := plan.Fact{Field: "a"}, plan.Fact{Field: "b"} - if !any(a) || any(b) || !all(a) || all(b) { - t.Error("a write to the caller's slice changed the matcher") - } -} - -func TestPinsRefuseAnEmptyName(t *testing.T) { - for name, pin := range map[string]func(string) plan.RuleOption{"Column": plan.Column, "Identity": plan.Identity} { - func() { - defer func() { - if recover() == nil { - t.Errorf("%s(\"\") did not panic", name) - } - }() - pin("") - }() - } -} - -func TestDecisionsSpellThemselves(t *testing.T) { - d, ok := plan.When(plan.Field("a"), plan.Encrypt(plan.EQL(se.Equality, se.Match)), plan.Column("c"), plan.Identity("old_c")).Decide(plan.Fact{Field: "a"}) - if !ok { - t.Fatal("no match") - } - if got, want := d.String(), `Encrypt(EQL(eq, match)) Column("c") Identity("old_c")`; got != want { - t.Errorf("String = %s, want %s", got, want) - } - if target, ok := d.Target(); !ok || target == nil || d.Column() != "c" || d.Identity() != "old_c" { - t.Errorf("accessors: %v %v %q %q", target, ok, d.Column(), d.Identity()) - } - if d := plan.Encrypt(plan.EQL()); d.Column() != "" || d.Identity() != "" { - t.Errorf("unpinned accessors: %q %q", d.Column(), d.Identity()) - } - for d, want := range map[string]string{ - plan.Plaintext().String(): "Plaintext()", - plan.Fail("no").String(): `Fail("no")`, - plan.Encrypt(plan.Custom("ctx", se.Ope)).String(): `Encrypt(Custom("ctx", ope))`, - plan.Encrypt(plan.Custom("ctx")).String(): `Encrypt(Custom("ctx"))`, - plan.Decision{}.String(): "Decision{}", - } { - if d != want { - t.Errorf("String = %s, want %s", d, want) - } - } - if _, ok := plan.Plaintext().Target(); ok { - t.Error("Plaintext has a target") - } -} - -func TestFactAndSource(t *testing.T) { - f := plan.Fact{Message: "row", Field: "email", GoField: "Email", Annotations: []plan.Annotation{ - {Key: "a", Values: []string{"x", "y"}}, {Key: "b", Values: []string{"z"}}, {Key: "a", Values: []string{"w"}}, - }} - if got := f.String(); got != "row.email (Email) [a=x,y; b=z; a=w]" { - t.Errorf("String = %s", got) - } - if got := f.Values("a"); !reflect.DeepEqual(got, []string{"x", "y", "w"}) { - t.Errorf("Values = %v", got) - } - src := plan.SourceFunc(func(any) ([]plan.Fact, error) { return []plan.Fact{f}, nil }) - if got, err := src.Facts(nil); err != nil || len(got) != 1 { - t.Errorf("SourceFunc.Facts = %v, %v", got, err) - } - if _, err := plan.PlanFor(nil, individuals); err == nil { - t.Error("PlanFor without a source") - } - if individuals.Table() != "individuals" || individuals.Msg() == nil { - t.Error("Message accessors") - } -} - -func TestWhenRefusesANilMatcher(t *testing.T) { - defer func() { - if recover() == nil { - t.Error("When(nil, ...) did not panic") - } - }() - plan.When(nil, plan.Plaintext()) -} - -// label is se.ParseLabel for a label the test knows to be valid. -func label(t testing.TB, s string) se.Label { - t.Helper() - l, err := se.ParseLabel(s) - if err != nil { - t.Fatal(err) - } - return l -} - -// With the identity pinned, the storage column is only the record key, as a -// Custom target's is: a '/' in it names a database column, not a context, so -// it is accepted and the context stays the pinned identity's label. -func TestASlashInARenamedStorageColumnIsOnlyARecordKey(t *testing.T) { - facts := []plan.Fact{{Field: "blob", GoField: "Blob", Annotations: []plan.Annotation{{Key: "k", Values: []string{"v"}}}}} - p, err := plan.ForMessage(nil, "t", plan.When(plan.Field("blob"), plan.Encrypt(plan.EQL()), plan.Column("blob/v1"), plan.Identity("blob"))).Build(facts) - if err != nil { - t.Fatal(err) - } - f := p.Fields()[0] - if f.Name != "blob/v1" || !f.Context.Equal(label(t, "t/blob").Context()) { - t.Fatalf("field = %+v, want record key blob/v1 under t/blob", f) - } -} - -// The label error survives the wrapping, so a caller can learn which half -// of the identifier was wrong rather than only that the decision is invalid. -func TestABadIdentifierKeepsItsLabelError(t *testing.T) { - facts := []plan.Fact{{Field: "a", Annotations: []plan.Annotation{{Key: "k", Values: []string{"v"}}}}} - _, err := plan.ForMessage(nil, "a/b", plan.When(plan.Field("a"), plan.Encrypt(plan.EQL()))).Build(facts) - var le *se.LabelError - if !errors.Is(err, plan.ErrInvalid) || !errors.As(err, &le) || le.Index != 0 { - t.Fatalf("err = %v; want ErrInvalid wrapping a LabelError for segment 0", err) - } - if !strings.Contains(err.Error(), `table "a/b"`) { - t.Errorf("err = %v; want the table named", err) - } -} - -// Only an EQL target's context is built from the table, so a message whose -// every encrypted field has a Custom target builds with a table name that -// would not be a plain segment. Recorded, not endorsed: the table is unused -// by such a field's context. -func TestACustomOnlyMessageTakesAnyTableName(t *testing.T) { - facts := []plan.Fact{{Field: "a", Annotations: []plan.Annotation{{Key: "k", Values: []string{"v"}}}}} - p, err := plan.ForMessage(nil, "a/b", plan.When(plan.Field("a"), plan.Encrypt(plan.Custom("blobs/ctx")))).Build(facts) - if err != nil { - t.Fatal(err) - } - if got := p.Fields()[0].Context; !got.Equal(label(t, "blobs/ctx").Context()) { - t.Errorf("context = %v, want the custom label", got) - } -} diff --git a/languages/golang/encrypt/plan/plantest/compare.go b/languages/golang/encrypt/plan/plantest/compare.go deleted file mode 100644 index 93a48fcec..000000000 --- a/languages/golang/encrypt/plan/plantest/compare.go +++ /dev/null @@ -1,450 +0,0 @@ -package plantest - -import ( - "fmt" - "path/filepath" - "slices" - "strings" - - "github.com/cipherstash/stack/languages/golang/encrypt/plan" -) - -// changes is what differs between a snapshot and the policy now, sorted -// by what each costs. -type changes struct { - // contexts lose data: a context no field writes any more. - contexts []string - // migrations need rows rewritten: a target that changed under a - // context that did not. - migrations []string - // other changes cost nothing already written: a new field, a changed - // fact, a field the policy no longer decides. - other []string -} - -// explain says what changed between the snapshot want and the policy now. -func explain(want []byte, cur snapshot, facts []plan.Fact, m plan.Message) string { - old, err := parse(want) - if err != nil { - return fmt.Sprintf("The snapshot does not parse (%v), so its changes cannot be sorted; the diff below shows them.\n\n", err) - } - return compare(old, cur, facts, m).String() -} - -func (c changes) String() string { - var b strings.Builder - section := func(title string, items []string) { - if len(items) == 0 { - return - } - b.WriteString(title) - b.WriteString("\n") - for _, it := range items { - fmt.Fprintf(&b, " - %s\n", it) - } - b.WriteString("\n") - } - section("CONTEXT CHANGES. These lose data: a context is bound into every ciphertext and query term written under it, so rows already written under the old one no longer decrypt, and their terms no longer match queries. Keep the old context by pinning the field's rule with plan.Column (and plan.Identity after a database column rename).", c.contexts) - section("TARGET CHANGES. These need a migration: rows already written keep what they were written with until they are rewritten.", c.migrations) - section("OTHER CHANGES.", c.other) - if b.Len() == 0 { - return "Nothing the policy stores changed, but the snapshot's text did: an edit by hand, or a newer plantest.\n\n" - } - return b.String() -} - -// compare sorts the differences between the snapshot old and the policy -// now (cur, built from facts by m). -func compare(old, cur snapshot, facts []plan.Fact, m plan.Message) changes { - var c changes - if old.table != cur.table { - // Only an EQL column's context is under the table; a Custom one is - // the target's own, and a plaintext field has none. - if slices.ContainsFunc(old.columns, func(o column) bool { return o.kind == kindEQL }) { - c.contexts = append(c.contexts, fmt.Sprintf("the message's table is %s, was %s. The table is the first half of every EQL column's context; restore plan.Table(%q).", - token(cur.table), token(old.table), old.table)) - } else { - c.other = append(c.other, fmt.Sprintf("the message's table is %s, was %s. No context moves with it: the message has no EQL column, whose context is the only one under the table.", - token(cur.table), token(old.table))) - } - } - - oldCols, curCols := byName(old.columns), byName(cur.columns) - oldPlain, curPlain := byField(old.plaintext), byField(cur.plaintext) - oldContexts, curContexts := map[string]bool{}, map[string][]string{} - for _, o := range old.columns { - oldContexts[contextKey(o)] = true - } - for _, n := range cur.columns { - curContexts[contextKey(n)] = append(curContexts[contextKey(n)], token(n.name)) - } - // The entries now that an old one accounts for, and the old plaintext - // fields an entry now accounts for. - seenCol, seenPlain, seenOldPlain := map[string]bool{}, map[string]bool{}, map[string]bool{} - - for _, o := range old.columns { - if n, ok := curCols[o.name]; ok { - seenCol[n.name] = true - if !sameContext(o, n) { - c.contexts = append(c.contexts, fmt.Sprintf("column %s: its context is %s, was %s. %s", - token(o.name), n.context, o.context, pinAdvice(m, facts, n.from, o, old.table))) - } - c.stored(o, n) - continue - } - // A database column rename: a new column under the same context. - // Only an EQL context is evidence of one, being the column's own - // identity; Custom columns may share a context with no relation - // between them. - if n, ok := only(cur.columns, func(n column) bool { - _, before := oldCols[n.name] - return !before && !seenCol[n.name] && sameContext(n, o) && o.kind == kindEQL && n.kind == kindEQL - }); ok { - seenCol[n.name] = true - c.migrations = append(c.migrations, fmt.Sprintf("column %s is now stored in column %s, under the same context: the database column is renamed with it (ALTER TABLE ... RENAME COLUMN).", - token(o.name), token(n.name))) - c.stored(o, n) - continue - } - // Decided Plaintext now: under the column's name, or the only new - // plaintext field with its facts. - if p, ok := only(cur.plaintext, func(p plain) bool { - _, before := oldPlain[p.field] - return !before && !seenPlain[p.field] && (p.field == o.name || len(o.facts) > 0 && slices.Equal(p.facts, o.facts)) - }); ok { - seenPlain[p.field] = true - c.migrations = append(c.migrations, fmt.Sprintf("column %s is now field %s, decided Plaintext. Rows already written hold ciphertexts under %s, which a migration must decrypt before the field is read as plaintext.", - token(o.name), token(p.field), o.context)) - continue - } - // The context is still written, by a column sharing a Custom - // context: nothing already written stops decrypting, but nothing - // reads this column. - if writers := curContexts[contextKey(o)]; len(writers) > 0 { - still := "column " + writers[0] + " still writes it" - if len(writers) > 1 { - still = "columns " + strings.Join(writers, ", ") + " still write it" - } - c.other = append(c.other, fmt.Sprintf("column %s is no longer written. Its context %s is not lost, since %s, but no field reads column %s now: if its field was renamed, pin the renamed field's rule with plan.Column(%q).", - token(o.name), o.context, still, token(o.name), o.name)) - continue - } - // The context is lost. The only new column with the same facts, - // under a context new to the snapshot, may be the same field - // renamed. Matching facts are no proof: an unrelated new field can - // carry the same ones, and pinning it would store two fields in one - // column. So the guess states both readings, and the column is - // still listed as new. - msg := fmt.Sprintf("column %s: no field writes its context %s any more.", token(o.name), o.context) - if n, ok := only(cur.columns, func(n column) bool { - _, before := oldCols[n.name] - return !before && !seenCol[n.name] && !oldContexts[contextKey(n)] && len(o.facts) > 0 && slices.Equal(n.facts, o.facts) - }); ok { - seenCol[n.name] = true - msg += fmt.Sprintf(" Field %s now writes column %s under %s with the same facts, so it may be the same field renamed: %s"+ - " If %s is instead a new field and %s was removed, do not pin it: that would store two fields in column %s under one context.", - fieldName(n.from), token(n.name), n.context, pinAdvice(m, facts, n.from, o, old.table), - token(n.name), token(o.name), token(o.name)) - c.other = append(c.other, fmt.Sprintf("new column %s, under %s, with terms [%s]. It is also named above as a possible rename of column %s.", - token(n.name), n.context, termList(n.terms), token(o.name))) - c.stored(o, n) - } else { - msg += fmt.Sprintf(" If its field was renamed, pin the renamed field's rule with plan.Column(%q); if it was removed, rows written under it can no longer be read through this policy.", o.name) - } - c.contexts = append(c.contexts, msg) - } - - for _, n := range cur.columns { - if seenCol[n.name] { - continue - } - if n.from != nil { - if p, ok := oldPlain[n.from.fact.Field]; ok { - if _, still := curPlain[p.field]; !still { - seenOldPlain[p.field] = true - c.migrations = append(c.migrations, fmt.Sprintf("field %s, decided Plaintext before, is now encrypted into column %s under %s. Rows already written hold its plaintext, which a migration must encrypt.", - token(p.field), token(n.name), n.context)) - continue - } - } - } - c.other = append(c.other, fmt.Sprintf("new column %s, under %s, with terms [%s].", token(n.name), n.context, termList(n.terms))) - } - - for _, o := range old.plaintext { - if n, ok := curPlain[o.field]; ok { - if !slices.Equal(o.facts, n.facts) { - c.other = append(c.other, fmt.Sprintf("plaintext field %s: its facts are [%s], were [%s].", token(o.field), factList(n.facts), factList(o.facts))) - } - continue - } - if !seenOldPlain[o.field] { - c.other = append(c.other, fmt.Sprintf("plaintext field %s is no longer decided by the policy (renamed, removed, or no longer classified).", token(o.field))) - } - } - for _, n := range cur.plaintext { - if _, before := oldPlain[n.field]; !before && !seenPlain[n.field] { - c.other = append(c.other, fmt.Sprintf("new plaintext field %s.", token(n.field))) - } - } - return c -} - -// stored compares what one column stores, the context aside. -func (c *changes) stored(o, n column) { - if !slices.Equal(o.terms, n.terms) { - c.migrations = append(c.migrations, fmt.Sprintf("column %s: its terms are [%s], were [%s]. Rows already written carry the terms they were written with until they are re-encrypted, so a query on a new term misses them.", - token(n.name), termList(n.terms), termList(o.terms))) - } - if o.kind != n.kind { - // A changed context is reported on its own, under CONTEXT CHANGES. - where := ", under the same context" - if !sameContext(o, n) { - where = "" - } - c.other = append(c.other, fmt.Sprintf("column %s: its target is %s, was %s%s.", token(n.name), n.kind, o.kind, where)) - } - if !slices.Equal(o.facts, n.facts) { - c.other = append(c.other, fmt.Sprintf("column %s: its facts are [%s], were [%s].", token(n.name), factList(n.facts), factList(o.facts))) - } -} - -// contextKey names a column's context exactly: its segments. An EQL -// context and a Custom one are both labels, so the two kinds share a -// context when their segments match; the kind is how the context was -// chosen, not part of it. -func contextKey(c column) string { return c.context } - -// sameContext reports whether a and b bind the same context. -func sameContext(a, b column) bool { return contextKey(a) == contextKey(b) } - -// pinAdvice says how to store the field from in the old column under the -// old context again, each answer checked by building the plan with it: the -// old table, when the table moved, with a pin if it needs one; a pin; or -// why no pin can. -func pinAdvice(m plan.Message, facts []plan.Fact, from *decided, old column, oldTable string) string { - if from == nil { - return "" - } - if oldTable != string(m.Table()) { - if pin, ok := pin(m, plan.Table(oldTable), facts, from, old); ok { - if pin == "" { - return fmt.Sprintf("Restoring plan.Table(%q) brings it back.", oldTable) - } - return fmt.Sprintf("Restoring plan.Table(%q) and pinning the rule that decides field %s with %s brings it back.", oldTable, fieldName(from), pin) - } - } - if pin, ok := pin(m, m.Table(), facts, from, old); ok { - return fmt.Sprintf("Pinning the rule that decides field %s with %s keeps it.", fieldName(from), pin) - } - return fmt.Sprintf("No plan.Column or plan.Identity pin on the rule that decides field %s brings it back: the context comes from the target itself (a plan.Custom context, or a change of target), so restore that.", fieldName(from)) -} - -// pin is the rule options, spelled as Go, that store the field from as -// column under context again in table, checked by building the plan with -// them, and whether any does. With m's own table it tries only pins; with -// another, no pin first (""). -func pin(m plan.Message, table plan.Table, facts []plan.Fact, from *decided, old column) (string, bool) { - column, context := old.name, old.context - if column == "" { - return "", false - } - want, err := contextOf(old) - if err != nil { - return "", false - } - type try struct { - spelled string - opts []plan.RuleOption - } - var tries []try - if table != m.Table() { - tries = append(tries, try{}) - } - tries = append(tries, try{fmt.Sprintf("plan.Column(%q)", column), []plan.RuleOption{plan.Column(column)}}) - if segments, err := segmentsOf(context); err == nil && len(segments) == 2 && segments[0] == string(table) { - id := segments[1] - tries = append(tries, - try{fmt.Sprintf("plan.Identity(%q)", id), []plan.RuleOption{plan.Identity(id)}}, - try{fmt.Sprintf("plan.Column(%q), plan.Identity(%q)", column, id), []plan.RuleOption{plan.Column(column), plan.Identity(id)}}, - ) - } - for _, t := range tries { - rule := plan.When(plan.Field(from.fact.Field), from.decision, t.opts...) - p, err := plan.ForMessage(m.Msg(), table, rule.OrElse(m.Decide)).Build(facts) - if err != nil { - continue - } - for _, fp := range p.Fields() { - if fp.Field == goField(from.fact) && fp.Name == column && fp.Context.Equal(want) { - return t.spelled, true - } - } - } - return "", false -} - -// only is the one element of s that keep accepts, if exactly one does. -func only[T any](s []T, keep func(T) bool) (T, bool) { - var found T - n := 0 - for _, v := range s { - if keep(v) { - found = v - n++ - } - } - if n != 1 { - var zero T - return zero, false - } - return found, true -} - -func byName(cs []column) map[string]column { - out := make(map[string]column, len(cs)) - for _, c := range cs { - out[c.name] = c - } - return out -} - -func byField(ps []plain) map[string]plain { - out := make(map[string]plain, len(ps)) - for _, p := range ps { - out[p.field] = p - } - return out -} - -// fieldName names a field the way a rule matches it, and the Go field it -// binds to when that is spelled differently. -func fieldName(d *decided) string { - if d == nil { - return "?" - } - name := token(d.fact.Field) - if g := goField(d.fact); g != d.fact.Field { - name += " (" + g + ")" - } - return name -} - -func factList(fs []fact) string { - if len(fs) == 0 { - return "none" - } - parts := make([]string, len(fs)) - for i, f := range fs { - parts[i] = token(f.key) + "=" + token(f.value) - } - return strings.Join(parts, " ") -} - -// diff is a unified diff of the snapshot want and the policy now, with -// two lines of context. -func diff(path string, want, got []byte) string { - es := edits(lines(want), lines(got)) - const context = 2 - keep := make([]bool, len(es)) - for i, e := range es { - if e.op == ' ' { - continue - } - for j := max(0, i-context); j <= min(len(es)-1, i+context); j++ { - keep[j] = true - } - } - var b strings.Builder - fmt.Fprintf(&b, "--- %s\n+++ the policy now\n", filepath.ToSlash(path)) - for i := 0; i < len(es); { - if !keep[i] { - i++ - continue - } - end := i - for end < len(es) && keep[end] { - end++ - } - var olds, news int - for _, e := range es[i:end] { - if e.op != '+' { - olds++ - } - if e.op != '-' { - news++ - } - } - fmt.Fprintf(&b, "@@ -%d,%d +%d,%d @@\n", es[i].a+1, olds, es[i].b+1, news) - for _, e := range es[i:end] { - b.WriteByte(e.op) - b.WriteString(e.line) - b.WriteByte('\n') - } - i = end - } - return b.String() -} - -// edit is one line of a diff: kept (' '), removed ('-') or added ('+'), -// with the index each side has reached. -type edit struct { - op byte - line string - a, b int -} - -// edits is a shortest edit script from a to b, by longest common -// subsequence. Snapshots are small; past a few million cells it gives up -// on alignment and replaces a with b wholesale. -func edits(a, b []string) []edit { - var out []edit - if len(a)*len(b) > 1<<22 { - for i, l := range a { - out = append(out, edit{'-', l, i, 0}) - } - for j, l := range b { - out = append(out, edit{'+', l, len(a), j}) - } - return out - } - lcs := make([][]int32, len(a)+1) - for i := range lcs { - lcs[i] = make([]int32, len(b)+1) - } - for i := len(a) - 1; i >= 0; i-- { - for j := len(b) - 1; j >= 0; j-- { - if a[i] == b[j] { - lcs[i][j] = lcs[i+1][j+1] + 1 - } else { - lcs[i][j] = max(lcs[i+1][j], lcs[i][j+1]) - } - } - } - i, j := 0, 0 - for i < len(a) || j < len(b) { - switch { - case i < len(a) && j < len(b) && a[i] == b[j]: - out = append(out, edit{' ', a[i], i, j}) - i++ - j++ - case j == len(b) || i < len(a) && lcs[i+1][j] >= lcs[i][j+1]: - out = append(out, edit{'-', a[i], i, j}) - i++ - default: - out = append(out, edit{'+', b[j], i, j}) - j++ - } - } - return out -} - -// lines splits text into its lines, without the final newline's empty -// one. -func lines(text []byte) []string { - s := strings.TrimSuffix(string(text), "\n") - if s == "" { - return nil - } - return strings.Split(s, "\n") -} diff --git a/languages/golang/encrypt/plan/plantest/golden_test.go b/languages/golang/encrypt/plan/plantest/golden_test.go deleted file mode 100644 index 861676cf9..000000000 --- a/languages/golang/encrypt/plan/plantest/golden_test.go +++ /dev/null @@ -1,47 +0,0 @@ -package plantest_test - -import ( - "testing" - - se "github.com/cipherstash/stack/languages/golang/encrypt" - "github.com/cipherstash/stack/languages/golang/encrypt/plan" - "github.com/cipherstash/stack/languages/golang/encrypt/plan/plantest" - "github.com/cipherstash/stack/languages/golang/internal/factstest" -) - -var category = plan.Key("fides.data_categories") - -type individual struct { - ID int64 - Email string `facts:"fides.data_categories=user.contact.email"` - Name string `facts:"fides.data_categories=user.name"` - MedicareNo string `facts:"fides.data_categories=user.government_id"` - Notes []byte `facts:"fides.data_categories=user.content"` - Country string `facts:"fides.data_categories=system.operations"` -} - -// individuals has a field whose database column was renamed (stored in -// medicare_num, under its first identity), a Custom target, and a field -// left in plaintext. -var individuals = plan.ForMessage(&individual{}, plan.Table("individuals"), plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(se.Equality)), - plan.Column("medicare_num"), plan.Identity("medicare_number")), - plan.When(category.Under("user.content"), plan.Encrypt(plan.Custom("individuals-notes/v1"))), - plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(se.Equality, se.Match))), - plan.When(category.Under("user"), plan.Encrypt(plan.EQL(se.Ore))), - plan.When(category.Under("system"), plan.Plaintext()), -)) - -type audit struct { - ID int64 - Kind string `facts:"fides.data_categories=system.operations"` -} - -// audits encrypts nothing: its snapshot is the fields it leaves plaintext. -var audits = plan.ForMessage(audit{}, plan.Table("audits"), plan.When(category.Under("system"), plan.Plaintext())) - -// Each message's snapshot is checked in at testdata/TestPolicies/.golden. -func TestPolicies(t *testing.T) { - t.Run("individuals", func(t *testing.T) { plantest.Golden(t, factstest.StructTags, individuals) }) - t.Run("audits", func(t *testing.T) { plantest.Golden(t, factstest.StructTags, audits) }) -} diff --git a/languages/golang/encrypt/plan/plantest/plantest.go b/languages/golang/encrypt/plan/plantest/plantest.go deleted file mode 100644 index 1dfd06101..000000000 --- a/languages/golang/encrypt/plan/plantest/plantest.go +++ /dev/null @@ -1,222 +0,0 @@ -// Package plantest checks in what a [plan.Policy] stores each field as, so -// a change to a field's encryption context fails a test instead of the -// data. -// -// A field's context is bound into every ciphertext and query term written -// under it. A policy derives contexts from names, so an ordinary rename — -// a proto field, a Go struct field, a table — can change one with no error -// on the write path: rows already written stop decrypting, and their terms -// stop matching queries. [Golden] makes that a test failure: -// -// func TestIndividualsPolicy(t *testing.T) { -// plantest.Golden(t, source, policy.Individuals) -// } -// -// The first run with -update writes the snapshot to -// testdata/TestIndividualsPolicy.golden; check it in. From then on Golden -// builds the plan the way [plan.MustPlanFor] does at startup, and fails -// when the plan no longer matches the snapshot, naming each change and -// calling out a changed context apart from the rest: a changed context is -// data loss, while a changed target (its index terms, or encrypting a -// field that was plaintext) is a migration. When a rename changed a -// context, the failure names the [plan.Column] (or [plan.Identity]) pin -// that keeps it, checked by building the plan with that pin. -// -// # The snapshot -// -// The snapshot records what a policy stores, not what the schema calls it: -// per message its [plan.Table]; per encrypted field its column (the record -// key), its context, whether its target is EQL (the context is the column -// identity) or Custom (the target supplies it), its index terms and its -// facts; per field decided -// [plan.Plaintext], its name and its facts. Fields with no facts that no -// rule names are not the policy's concern and are left out. Encrypted -// fields are named by column, so a schema rename that the policy pins -// leaves the snapshot byte-for-byte unchanged, which is how a reviewer -// tells a safe rename from one that loses data. -// -// The text is line-oriented and deterministic across runs and platforms: -// fields are sorted, facts are sorted, line endings are "\n" (a checkout -// that converted them to "\r\n" still compares equal), and any value that -// is not a plain identifier is quoted as a Go string literal. It is meant -// to be read by people as well as compared: the list of classified fields, -// what protects each, and the identifier it is bound to. -// -// # The -update flag -// -// plantest registers the conventional -update test flag, as a bool on the -// default flag set, when no package it imports has done so already. A test -// package that defines its own -update panics with "flag redefined"; -// read plantest's instead, with flag.Lookup("update"). -package plantest - -import ( - "errors" - "flag" - "fmt" - "hash/fnv" - "io/fs" - "os" - "path/filepath" - "regexp" - "strings" - "testing" - - "github.com/cipherstash/stack/languages/golang/encrypt/plan" -) - -func init() { - if flag.Lookup("update") == nil { - flag.Bool("update", false, "rewrite plantest golden snapshots (testdata/*.golden) from the policy") - } -} - -// lookupFlag finds a flag on the default flag set. It is a variable so a -// test can read -update from a flag set of its own instead of setting the -// one the whole test binary shares. -var lookupFlag = flag.Lookup - -// updating reports whether the test binary was run with -update. -func updating() bool { - f := lookupFlag("update") - if f == nil { - return false - } - g, ok := f.Value.(flag.Getter) - if !ok { - return false - } - on, ok := g.Get().(bool) - return ok && on -} - -// Golden builds m's plan over src's facts, as [plan.PlanFor] does, and -// checks what it stores each field as against the snapshot checked in at -// testdata/.golden (a subtest's name is a path below -// testdata). With -update it writes the snapshot instead, logging what -// changed. -// -// The test fails when the policy does not build (a classified field no -// rule decides, a [plan.Fail], a plan that does not bind to the message), -// when there is no snapshot yet, and when the plan differs from it. A -// message the policy encrypts nothing of is not a failure: its snapshot -// lists the fields it decided Plaintext. -// -// A [plan.Source] is expected to be pure, as a policy is: Golden reads the -// facts more than once. -func Golden(t testing.TB, src plan.Source, m plan.Message) { - t.Helper() - logs, err := check(goldenPath(t.Name()), src, m, updating(), rerun(t.Name())) - if logs != "" { - t.Log(logs) - } - if err != nil { - t.Error(err) - } -} - -// goldenPath is the snapshot file for a test: testdata/.golden, a -// subtest's name a path below it. A character a file system could refuse -// is spelled '_', and so is a trailing dot, which Windows drops. A name -// spelled differently from the test's, or one Windows reserves as a device -// (CON, NUL.x, COM1), takes '~' and a hash of the test's spelling before -// its first dot, so two tests never share a snapshot: "a:b" and "a?b" are -// a_b~.golden with different hashes, and no name spelled as it is -// contains '~'. Every platform spells a name the same way, so a snapshot -// written on one is found on another. -func goldenPath(name string) string { - parts := strings.Split(name, "/") - for i, p := range parts { - parts[i] = strings.Map(func(r rune) rune { - if r < 0x80 && (r == '-' || r == '_' || r == '.' || r == '+' || - 'a' <= r && r <= 'z' || 'A' <= r && r <= 'Z' || '0' <= r && r <= '9') { - return r - } - return '_' - }, p) - if strings.Trim(parts[i], ".") == "" { - parts[i] = strings.Repeat("_", len(parts[i])+1) - } - trimmed := strings.TrimRight(parts[i], ".") - parts[i] = trimmed + strings.Repeat("_", len(parts[i])-len(trimmed)) - stem, ext, dotted := strings.Cut(parts[i], ".") - if parts[i] != p || windowsDevice(stem) { - h := fnv.New32a() - _, _ = h.Write([]byte(p)) // a hash.Hash never returns an error - parts[i] = fmt.Sprintf("%s~%08x", stem, h.Sum32()) - if dotted { - parts[i] += "." + ext - } - } - } - return filepath.Join(append([]string{"testdata"}, parts...)...) + ".golden" -} - -// windowsDevice reports whether Windows reserves name as a device, with or -// without an extension. -func windowsDevice(name string) bool { - switch strings.ToUpper(name) { - case "CON", "PRN", "AUX", "NUL", - "COM0", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", - "LPT0", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9": - return true - } - return false -} - -// rerun is the command that runs just this test with -update. -func rerun(name string) string { - parts := strings.Split(name, "/") - for i, p := range parts { - parts[i] = "^" + regexp.QuoteMeta(p) + "$" - } - return fmt.Sprintf("go test -run '%s' -update", strings.Join(parts, "/")) -} - -// check is Golden without the testing.TB: it compares (or, updating, -// writes) the snapshot at path, and returns what to log and what failed. -func check(path string, src plan.Source, m plan.Message, update bool, rerun string) (string, error) { - cur, facts, err := take(src, m) - if err != nil { - return "", fmt.Errorf("plantest: the policy does not build, so there is nothing to snapshot: %w", err) - } - got := cur.render() - path = filepath.Clean(path) - want, readErr := os.ReadFile(path) - if readErr != nil && !errors.Is(readErr, fs.ErrNotExist) { - return "", fmt.Errorf("plantest: %w", readErr) - } - same := readErr == nil && string(normalize(want)) == string(got) - - if update { - if same { - return "", nil - } - if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil { - return "", fmt.Errorf("plantest: %w", err) - } - // A snapshot is checked in, not secret; git records no mode but - // the executable bit. - if err := os.WriteFile(path, got, 0o644); err != nil { //nolint:gosec // see above - return "", fmt.Errorf("plantest: %w", err) - } - if readErr != nil { - return fmt.Sprintf("plantest: wrote %s; review it and check it in", path), nil - } - return fmt.Sprintf("plantest: updated %s, recording:\n\n%s", path, strings.TrimRight(explain(want, cur, facts, m), "\n")), nil - } - - if readErr != nil { - return "", fmt.Errorf("plantest: no snapshot at %s. Write it with\n\n\t%s\n\nthen review it and check it in", path, rerun) - } - if same { - return "", nil - } - return "", fmt.Errorf("plantest: %s no longer matches the policy.\n\n%sIf every change is intended (no row has been written under a changed context yet, or a migration ships with it), record them with\n\n\t%s\n\n%s", - path, explain(want, cur, facts, m), rerun, diff(path, normalize(want), got)) -} - -// normalize undoes a checkout's "\r\n" line endings. -func normalize(b []byte) []byte { - return []byte(strings.ReplaceAll(string(b), "\r\n", "\n")) -} diff --git a/languages/golang/encrypt/plan/plantest/plantest_internal_test.go b/languages/golang/encrypt/plan/plantest/plantest_internal_test.go deleted file mode 100644 index d945754e0..000000000 --- a/languages/golang/encrypt/plan/plantest/plantest_internal_test.go +++ /dev/null @@ -1,813 +0,0 @@ -package plantest - -import ( - "bytes" - "errors" - "flag" - "fmt" - "io/fs" - "os" - "path/filepath" - "runtime" - "slices" - "strings" - "testing" - - se "github.com/cipherstash/stack/languages/golang/encrypt" - "github.com/cipherstash/stack/languages/golang/encrypt/plan" - "github.com/cipherstash/stack/languages/golang/internal/factstest" -) - -var category = plan.Key("fides.data_categories") - -var base = plan.FirstOf( - plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(se.Equality))), - plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(se.Equality, se.Match))), - plan.When(category.Under("user"), plan.Encrypt(plan.EQL())), - plan.When(category.Under("system"), plan.Plaintext()), -) - -func classified(values ...string) []plan.Annotation { - return []plan.Annotation{{Key: "fides.data_categories", Values: values}} -} - -// proto is a protobuf-shaped source: the facts are given, and a field -// keeps its number through a rename. -func proto(facts ...plan.Fact) plan.Source { - for i := range facts { - facts[i].Message = "acme.v1.Individual" - } - return plan.SourceFunc(func(any) ([]plan.Fact, error) { return facts, nil }) -} - -// individualV1 is the schema before a rename. -func individualV1() plan.Source { - return proto( - plan.Fact{Field: "id", GoField: "Id", Number: 1, Kind: "int64"}, - plan.Fact{Field: "email", GoField: "Email", Number: 2, Kind: "string", Annotations: classified("user.contact.email")}, - plan.Fact{Field: "medicare_number", GoField: "MedicareNumber", Number: 3, Kind: "string", Annotations: classified("user.government_id")}, - plan.Fact{Field: "country", GoField: "Country", Number: 4, Kind: "string", Annotations: classified("system.operations")}, - ) -} - -// individualV2 is individualV1 with field 3 renamed. -func individualV2() plan.Source { - return proto( - plan.Fact{Field: "id", GoField: "Id", Number: 1, Kind: "int64"}, - plan.Fact{Field: "email", GoField: "Email", Number: 2, Kind: "string", Annotations: classified("user.contact.email")}, - plan.Fact{Field: "medicare_no", GoField: "MedicareNo", Number: 3, Kind: "string", Annotations: classified("user.government_id")}, - plan.Fact{Field: "country", GoField: "Country", Number: 4, Kind: "string", Annotations: classified("system.operations")}, - ) -} - -// record writes the snapshot of m over src to a fresh path and returns -// the path and what it wrote. -func record(t *testing.T, src plan.Source, m plan.Message) (string, []byte) { - t.Helper() - path := filepath.Join(t.TempDir(), "testdata", "TestPolicy.golden") - logs, err := check(path, src, m, true, "RERUN") - if err != nil { - t.Fatal(err) - } - if !strings.Contains(logs, "wrote "+path) { - t.Errorf("logs = %q, want the path written", logs) - } - written, err := os.ReadFile(path) - if err != nil { - t.Fatal(err) - } - return path, written -} - -func mustContain(t *testing.T, err error, wants ...string) { - t.Helper() - if err == nil { - t.Fatalf("passed, want a failure saying %q", wants) - } - for _, want := range wants { - if !strings.Contains(err.Error(), want) { - t.Errorf("failure does not say %q:\n%s", want, err) - } - } -} - -// The acceptance case: a renamed proto field with no pin fails with a -// context change, naming the pin; with the pin it passes, and the -// snapshot is the one checked in, byte for byte. -func TestRenameWithoutAPinIsAContextChange(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - path, before := record(t, individualV1(), m) - - _, err := check(path, individualV2(), m, false, "RERUN") - mustContain(t, err, - "CONTEXT CHANGES", - `column medicare_number: no field writes its context ["individuals", "medicare_number"] any more.`, - `Field medicare_no (MedicareNo) now writes column medicare_no under ["individuals", "medicare_no"]`, - `Pinning the rule that decides field medicare_no (MedicareNo) with plan.Column("medicare_number") keeps it.`, - "RERUN", - "- context [\"individuals\", \"medicare_number\"]", - "+ context [\"individuals\", \"medicare_no\"]", - ) - // A context change is data loss, not a migration. The new column is - // still listed, since the guess may be wrong. - if strings.Contains(err.Error(), "TARGET CHANGES") { - t.Errorf("a rename is reported as a migration:\n%s", err) - } - mustContain(t, err, "OTHER CHANGES", `new column medicare_no, under ["individuals", "medicare_no"], with terms [eq]. It is also named above as a possible rename of column medicare_number.`) - - pinned := plan.ForMessage(nil, "individuals", plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(se.Equality)), plan.Column("medicare_number")), - ).OrElse(base)) - if _, err := check(path, individualV2(), pinned, false, "RERUN"); err != nil { - t.Fatalf("the pinned rename fails: %v", err) - } - after, _, err := take(individualV2(), pinned) - if err != nil { - t.Fatal(err) - } - if !bytes.Equal(after.render(), before) { - t.Fatalf("the pinned rename changed the snapshot:\n%s\nwas\n%s", after.render(), before) - } -} - -// The same holds for a Go struct field renamed, where a fact source has -// no field number and the plan binds to the struct. -func TestStructFieldRename(t *testing.T) { - type v1 struct { - ID int64 - MedicareNumber string `facts:"fides.data_categories=user.government_id"` - } - type v2 struct { - ID int64 - MedicareNo string `facts:"fides.data_categories=user.government_id"` - } - path, _ := record(t, factstest.StructTags, plan.ForMessage(v1{}, "individuals", base)) - _, err := check(path, factstest.StructTags, plan.ForMessage(v2{}, "individuals", base), false, "RERUN") - mustContain(t, err, "CONTEXT CHANGES", `plan.Column("medicare_number")`) - pinned := plan.ForMessage(v2{}, "individuals", plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(se.Equality)), plan.Column("medicare_number")), - ).OrElse(base)) - if _, err := check(path, factstest.StructTags, pinned, false, "RERUN"); err != nil { - t.Fatalf("the pinned rename fails: %v", err) - } -} - -// Each kind of change lands in its own section, with the advice that -// fits it. -func TestChangesAreSortedByWhatTheyCost(t *testing.T) { - gov := plan.Encrypt(plan.EQL(se.Equality)) - for name, tc := range map[string]struct { - before, after plan.Message - src plan.Source // individualV1 when nil - section string // the section reported; "" when it does not build - also []string // further sections reported - says []string - never []string // advice that would be wrong here - }{ - "identity pin dropped": { - before: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), gov, plan.Identity("medicare_no"))).OrElse(base)), - after: plan.ForMessage(nil, "individuals", base), - section: "CONTEXT CHANGES", - says: []string{`column medicare_number: its context is ["individuals", "medicare_number"], was ["individuals", "medicare_no"].`, `with plan.Identity("medicare_no") keeps it.`}, - }, - "table changed": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "people", base), - section: "CONTEXT CHANGES", - says: []string{`the message's table is people, was individuals.`, `restore plan.Table("individuals")`, `its context is ["people", "email"], was ["individuals", "email"]. Restoring plan.Table("individuals") brings it back.`}, - }, - "table changed, nothing encrypted": { - before: plan.ForMessage(nil, "individuals", plan.When(category.Present(), plan.Plaintext())), - after: plan.ForMessage(nil, "people", plan.When(category.Present(), plan.Plaintext())), - section: "OTHER CHANGES", - says: []string{"the message's table is people, was individuals. No context moves with it"}, - }, - "table changed, only Custom targets": { - before: plan.ForMessage(nil, "individuals", plan.When(category.Under("user"), plan.Encrypt(plan.Custom("pii/v1"))).OrElse(base)), - after: plan.ForMessage(nil, "people", plan.When(category.Under("user"), plan.Encrypt(plan.Custom("pii/v1"))).OrElse(base)), - section: "OTHER CHANGES", - says: []string{"the message's table is people, was individuals. No context moves with it"}, - }, - "table and a table-shaped Custom context changed": { - before: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), plan.Encrypt(plan.Custom("individuals/medicare_number")))).OrElse(base)), - after: plan.ForMessage(nil, "people", plan.FirstOf(plan.When(plan.Field("medicare_number"), plan.Encrypt(plan.Custom("people/medicare_number")))).OrElse(base)), - section: "CONTEXT CHANGES", - says: []string{`column medicare_number: its context is ["people", "medicare_number"], was ["individuals", "medicare_number"]. No plan.Column or plan.Identity pin`}, - // Restoring the table leaves the Custom context as it is. - never: []string{`was ["individuals", "medicare_number"]. Restoring`}, - }, - "table changed and field renamed": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "people", base), - src: individualV2(), - section: "CONTEXT CHANGES", - also: []string{"OTHER CHANGES"}, - says: []string{`Restoring plan.Table("individuals") and pinning the rule that decides field medicare_no (MedicareNo) with plan.Column("medicare_number") brings it back.`}, - }, - // The Custom label reads like the EQL identity, and it is the same - // context: both bind the label (table, column). Only the target - // changed, and nothing written is lost. - "target kind changed, the context the same": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), plan.Encrypt(plan.Custom("individuals/medicare_number", se.Equality)))).OrElse(base)), - section: "OTHER CHANGES", - says: []string{"column medicare_number: its target is Custom, was EQL, under the same context."}, - }, - "target kind and context changed": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), plan.Encrypt(plan.Custom("elsewhere/v1", se.Equality)))).OrElse(base)), - section: "CONTEXT CHANGES", - also: []string{"OTHER CHANGES"}, - says: []string{ - `column medicare_number: its context is ["elsewhere", "v1"], was ["individuals", "medicare_number"].`, - "column medicare_number: its target is Custom, was EQL.", - }, - never: []string{"under the same context"}, - }, - "custom context changed": { - before: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), plan.Encrypt(plan.Custom("gov/v1")))).OrElse(base)), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), plan.Encrypt(plan.Custom("gov/v2")))).OrElse(base)), - section: "CONTEXT CHANGES", - says: []string{`its context is ["gov", "v2"], was ["gov", "v1"]. No plan.Column or plan.Identity pin`, "a plan.Custom context"}, - }, - "encrypted now plaintext": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), plan.Plaintext())).OrElse(base)), - section: "TARGET CHANGES", - says: []string{"column medicare_number is now field medicare_number, decided Plaintext.", `ciphertexts under ["individuals", "medicare_number"]`}, - }, - "terms changed": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("email"), plan.Encrypt(plan.EQL(se.Equality)))).OrElse(base)), - section: "TARGET CHANGES", - says: []string{"column email: its terms are [eq], were [eq match]."}, - }, - "plaintext now encrypted": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("country"), plan.Encrypt(plan.EQL()))).OrElse(base)), - section: "TARGET CHANGES", - says: []string{`field country, decided Plaintext before, is now encrypted into column country under ["individuals", "country"].`}, - }, - "database column renamed": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("medicare_number"), gov, plan.Column("medicare_num"), plan.Identity("medicare_number"))).OrElse(base)), - section: "TARGET CHANGES", - says: []string{"column medicare_number is now stored in column medicare_num, under the same context"}, - }, - "database column renamed, then the field renamed": { - before: plan.ForMessage(nil, "individuals", plan.FirstOf( - plan.When(plan.Field("medicare_number"), gov, plan.Column("medicare_num"), plan.Identity("medicare_id")), - ).OrElse(base)), - after: plan.ForMessage(nil, "individuals", base), - src: individualV2(), - section: "CONTEXT CHANGES", - also: []string{"OTHER CHANGES"}, - says: []string{ - `column medicare_num: no field writes its context ["individuals", "medicare_id"] any more.`, - `with plan.Column("medicare_num"), plan.Identity("medicare_id") keeps it.`, - }, - }, - "field newly decided Plaintext": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf( - plan.When(plan.Field("id"), plan.Plaintext()), - ).OrElse(base)), - section: "OTHER CHANGES", - says: []string{"new plaintext field id."}, - }, - "field newly classified": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("id"), plan.Encrypt(plan.EQL(se.Ore)))).OrElse(base)), - section: "OTHER CHANGES", - says: []string{`new column id, under ["individuals", "id"], with terms [ore].`}, - }, - "facts changed": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", base), - src: proto( - plan.Fact{Field: "email", GoField: "Email", Number: 2, Annotations: classified("user.contact.email", "user.contact.email.work")}, - plan.Fact{Field: "medicare_number", GoField: "MedicareNumber", Number: 3, Annotations: classified("user.government_id")}, - plan.Fact{Field: "country", GoField: "Country", Number: 4, Annotations: classified("system.operations", "system.location")}, - ), - section: "OTHER CHANGES", - says: []string{ - "column email: its facts are [fides.data_categories=user.contact.email fides.data_categories=user.contact.email.work], were [fides.data_categories=user.contact.email].", - "plaintext field country: its facts are", - }, - }, - "plaintext field gone": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", base), - src: proto( - plan.Fact{Field: "email", GoField: "Email", Number: 2, Annotations: classified("user.contact.email")}, - plan.Fact{Field: "medicare_number", GoField: "MedicareNumber", Number: 3, Annotations: classified("user.government_id")}, - ), - section: "OTHER CHANGES", - says: []string{"plaintext field country is no longer decided by the policy"}, - }, - "refused": { - before: plan.ForMessage(nil, "individuals", base), - after: plan.ForMessage(nil, "individuals", plan.FirstOf(plan.When(plan.Field("country"), plan.Fail("never stored"))).OrElse(base)), - says: []string{"the policy does not build", "never stored"}, - }, - } { - t.Run(name, func(t *testing.T) { - path, _ := record(t, individualV1(), tc.before) - src := tc.src - if src == nil { - src = individualV1() - } - _, err := check(path, src, tc.after, false, "RERUN") - mustContain(t, err, tc.says...) - for _, never := range tc.never { - if strings.Contains(err.Error(), never) { - t.Errorf("failure says %q:\n%s", never, err) - } - } - if tc.section == "" { - return - } - for _, s := range []string{"CONTEXT CHANGES", "TARGET CHANGES", "OTHER CHANGES"} { - want := s == tc.section || slices.Contains(tc.also, s) - if got := strings.Contains(err.Error(), s); got != want { - t.Errorf("reports %s = %v, want %s and %v only:\n%s", s, got, tc.section, tc.also, err) - } - } - }) - } -} - -// A field removed from the schema without a trace is a lost context: the -// failure says what to do either way. -func TestALostContextWithNoCandidate(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - path, _ := record(t, individualV1(), m) - gone := proto( - plan.Fact{Field: "email", GoField: "Email", Number: 2, Annotations: classified("user.contact.email")}, - plan.Fact{Field: "country", GoField: "Country", Number: 4, Annotations: classified("system.operations")}, - ) - _, err := check(path, gone, m, false, "RERUN") - mustContain(t, err, "CONTEXT CHANGES", `column medicare_number: no field writes its context ["individuals", "medicare_number"] any more. If its field was renamed, pin the renamed field's rule with plan.Column("medicare_number")`) -} - -// Custom columns may share a context, so a new one under the context of -// one that disappeared is no evidence of a database rename, and the -// context is not lost while another column still writes it. -func TestSharedCustomContextIsNotARename(t *testing.T) { - m := plan.ForMessage(nil, "individuals", plan.When(category.Under("user"), plan.Encrypt(plan.Custom("pii/v1")))) - path, _ := record(t, proto( - plan.Fact{Field: "a", GoField: "A", Number: 1, Annotations: classified("user.name")}, - plan.Fact{Field: "b", GoField: "B", Number: 2, Annotations: classified("user.name")}, - ), m) - _, err := check(path, proto( - plan.Fact{Field: "b", GoField: "B", Number: 2, Annotations: classified("user.name")}, - plan.Fact{Field: "c", GoField: "C", Number: 3, Annotations: classified("user.content")}, - ), m, false, "RERUN") - mustContain(t, err, "OTHER CHANGES", - `column a is no longer written. Its context ["pii", "v1"] is not lost, since columns b, c still write it`, - `new column c, under ["pii", "v1"]`) - for _, never := range []string{"RENAME COLUMN", "CONTEXT CHANGES"} { - if strings.Contains(err.Error(), never) { - t.Errorf("failure says %q:\n%s", never, err) - } - } -} - -// An unrelated field added as another is removed, with the same facts, -// looks like a rename. The guess says it may be one, warns against the -// pin if it is not, and still lists the new column, so the developer can -// tell which it is. -func TestARenameGuessStatesTheOtherReading(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - path, _ := record(t, proto( - plan.Fact{Field: "id", GoField: "Id", Number: 1}, - plan.Fact{Field: "home_phone", GoField: "HomePhone", Number: 2, Annotations: classified("user.contact.phone")}, - ), m) - _, err := check(path, proto( - plan.Fact{Field: "id", GoField: "Id", Number: 1}, - plan.Fact{Field: "work_phone", GoField: "WorkPhone", Number: 3, Annotations: classified("user.contact.phone")}, - ), m, false, "RERUN") - mustContain(t, err, - "CONTEXT CHANGES", - `column home_phone: no field writes its context ["individuals", "home_phone"] any more.`, - `Field work_phone (WorkPhone) now writes column work_phone under ["individuals", "work_phone"] with the same facts, so it may be the same field renamed:`, - `with plan.Column("home_phone") keeps it.`, - "If work_phone is instead a new field and home_phone was removed, do not pin it: that would store two fields in column home_phone under one context.", - "OTHER CHANGES", - `new column work_phone, under ["individuals", "work_phone"], with terms [none]. It is also named above as a possible rename of column home_phone.`, - ) - if strings.Contains(err.Error(), "likely") { - t.Errorf("the guess is stated as likely:\n%s", err) - } -} - -// A Custom context that happens to be spelled like an identity is still -// Custom: the kind is not read off one sample. -func TestTargetKind(t *testing.T) { - for _, tc := range []struct { - target plan.Target - want string - }{ - {plan.EQL(), kindEQL}, - {plan.EQL(se.Equality, se.Match), kindEQL}, - {plan.Custom("pii/v1"), kindCustom}, - {plan.Custom("plantest/a"), kindCustom}, - {plan.Custom("plantest/b"), kindCustom}, - {plan.Custom("plantest/probe"), kindCustom}, - {plan.Custom("individuals/email"), kindCustom}, - } { - got, err := targetKind(tc.target) - if err != nil { - t.Errorf("targetKind(%v): %v", tc.target, err) - continue - } - if got != tc.want { - t.Errorf("targetKind(%v) = %s, want %s", tc.target, got, tc.want) - } - } -} - -// refusingTarget refuses every identity, as a target with a bad context -// would. -type refusingTarget struct{} - -func (refusingTarget) Terms() []se.TermKind { return nil } -func (refusingTarget) Context(plan.Identifier) (se.Context, error) { - return se.Context{}, errors.New("no context here") -} - -// A target that refuses the probe identities is an error, not a kind: a -// Custom guess would hide it. -func TestTargetKindReportsARefusal(t *testing.T) { - kind, err := targetKind(refusingTarget{}) - if err == nil || !strings.Contains(err.Error(), "no context here") { - t.Fatalf("targetKind = %q, %v; want the target's error", kind, err) - } -} - -// Two new columns with the facts of the one that disappeared: no guess, -// and the generic advice. -func TestAmbiguousRenameIsNotGuessed(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - path, _ := record(t, individualV1(), m) - split := proto( - plan.Fact{Field: "email", GoField: "Email", Number: 2, Annotations: classified("user.contact.email")}, - plan.Fact{Field: "medicare_a", GoField: "MedicareA", Number: 5, Annotations: classified("user.government_id")}, - plan.Fact{Field: "medicare_b", GoField: "MedicareB", Number: 6, Annotations: classified("user.government_id")}, - plan.Fact{Field: "country", GoField: "Country", Number: 4, Annotations: classified("system.operations")}, - ) - _, err := check(path, split, m, false, "RERUN") - mustContain(t, err, "If its field was renamed", "new column medicare_a", "new column medicare_b") - if strings.Contains(err.Error(), "may be the same field renamed") { - t.Errorf("guessed a rename between two candidates:\n%s", err) - } -} - -// The snapshot is the same whatever order the source gives fields and -// facts in, and from run to run. -func TestSnapshotIsDeterministic(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - a, _, err := take(individualV1(), m) - if err != nil { - t.Fatal(err) - } - shuffled := proto( - plan.Fact{Field: "country", GoField: "Country", Number: 4, Annotations: classified("system.operations")}, - plan.Fact{Field: "medicare_number", GoField: "MedicareNumber", Number: 3, Annotations: classified("user.government_id")}, - plan.Fact{Field: "email", GoField: "Email", Number: 2, Annotations: []plan.Annotation{ - {Key: "fides.data_categories", Values: []string{"user.contact.email"}}, - {Key: "fides.data_categories", Values: []string{"user.contact.email"}}, - }}, - plan.Fact{Field: "id", GoField: "Id", Number: 1}, - ) - b, _, err := take(shuffled, m) - if err != nil { - t.Fatal(err) - } - if !bytes.Equal(a.render(), b.render()) { - t.Fatalf("field or fact order changed the snapshot:\n%s\nvs\n%s", a.render(), b.render()) - } - want := header + ` -table individuals - -column email - context ["individuals", "email"] - target EQL - terms eq match - fact fides.data_categories user.contact.email - -column medicare_number - context ["individuals", "medicare_number"] - target EQL - terms eq - fact fides.data_categories user.government_id - -plaintext country - fact fides.data_categories system.operations -` - if got := string(a.render()); got != want { - t.Fatalf("snapshot =\n%s\nwant\n%s", got, want) - } -} - -// Whatever a name, a context or a fact holds, the snapshot reads back as -// what was written. -func TestSnapshotRoundTrips(t *testing.T) { - odd := []string{"", "with space", `"quoted"`, "new\nline", "tab\there", "naïve", "#hash", "none", "a=b,c"} - s := snapshot{table: "odd table"} - for i, v := range odd { - kind := kindEQL - if i%2 == 1 { - kind = kindCustom - } - // Each segment is quoted, whatever it holds; a label has at least - // two. - s.columns = append(s.columns, column{name: v + string(rune('a'+i)), context: shape([]string{"t", v}), kind: kind, terms: []string{"eq", "ore", "odd term"}, facts: []fact{{v, v}, {"k", v}}}) - s.plaintext = append(s.plaintext, plain{field: v + string(rune('a'+i)), facts: []fact{{v, "x"}}}) - } - s.columns = append(s.columns, column{name: "bare", context: shape([]string{"t", "bare"}), kind: kindEQL}) - for i := range s.columns { - sortFacts(s.columns[i].facts) - } - s.sort() - text := s.render() - back, err := parse(text) - if err != nil { - t.Fatalf("%v\n%s", err, text) - } - if again := back.render(); !bytes.Equal(again, text) { - t.Fatalf("round trip changed the snapshot:\n%s\nwas\n%s", again, text) - } - if !strings.Contains(string(text), " terms none\n") { - t.Errorf("an unindexed column does not say so:\n%s", text) - } -} - -func TestMissingSnapshot(t *testing.T) { - path := filepath.Join(t.TempDir(), "testdata", "TestPolicy.golden") - _, err := check(path, individualV1(), plan.ForMessage(nil, "individuals", base), false, "go test -run '^TestPolicy$' -update") - mustContain(t, err, "no snapshot at "+path, "go test -run '^TestPolicy$' -update") -} - -func TestUpdateRewritesAndSaysWhatItRecorded(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - path, _ := record(t, individualV1(), m) - logs, err := check(path, individualV2(), m, true, "RERUN") - if err != nil { - t.Fatal(err) - } - if !strings.Contains(logs, "updated "+path) || !strings.Contains(logs, "CONTEXT CHANGES") { - t.Errorf("logs = %q, want the update and the context change it recorded", logs) - } - if _, err := check(path, individualV2(), m, false, "RERUN"); err != nil { - t.Fatalf("after -update: %v", err) - } - // Updating an unchanged snapshot is silent and leaves it alone. - if logs, err := check(path, individualV2(), m, true, "RERUN"); err != nil || logs != "" { - t.Errorf("no-op update: logs %q, err %v", logs, err) - } -} - -// A checkout that converted line endings still matches. -func TestCRLFCheckoutMatches(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - path, written := record(t, individualV1(), m) - if err := os.WriteFile(path, bytes.ReplaceAll(written, []byte("\n"), []byte("\r\n")), 0o600); err != nil { - t.Fatal(err) - } - if _, err := check(path, individualV1(), m, false, "RERUN"); err != nil { - t.Fatal(err) - } -} - -func TestTextOnlyAndUnreadableSnapshots(t *testing.T) { - m := plan.ForMessage(nil, "individuals", base) - path, written := record(t, individualV1(), m) - - // Same content, different text: still a failure, said as such. - edited := bytes.Replace(written, []byte("table individuals"), []byte(`table "individuals"`), 1) - if err := os.WriteFile(path, edited, 0o600); err != nil { - t.Fatal(err) - } - _, err := check(path, individualV1(), m, false, "RERUN") - mustContain(t, err, "Nothing the policy stores changed", `-table "individuals"`, "+table individuals") - - if err := os.WriteFile(path, append(written, []byte("garbage line\n")...), 0o600); err != nil { - t.Fatal(err) - } - _, err = check(path, individualV1(), m, false, "RERUN") - mustContain(t, err, "does not parse", `unexpected "garbage line"`, "+++ the policy now") -} - -// parse refuses what render never writes, so a damaged snapshot falls back -// to the plain diff rather than a summary built on a misreading. -func TestParseRejects(t *testing.T) { - const col = "\ncolumn email\n context [\"individuals\", \"email\"]\n target EQL\n terms eq\n" - for name, tc := range map[string]struct{ text, says string }{ - "empty file": {"", "no table line"}, - "header only": {header, "no table line"}, - "no table line": {header + col, "no table line"}, - "second table line": {header + "\ntable individuals\ntable people\n" + col, `unexpected "table people"`}, - "unterminated quote": {header + "\ntable \"individuals\n", "bad quoted value"}, - "unknown target": {header + "\ntable individuals\n" + strings.Replace(col, "target EQL", "target Mystery", 1), `unexpected " target Mystery"`}, - "unknown line": {header + "\ntable individuals\n" + col + "garbage line\n", `unexpected "garbage line"`}, - "column without a target": { - header + "\ntable individuals\n" + strings.Replace(col, " target EQL\n", "", 1), - `column "email" has no context or target line`, - }, - // The spelling before contexts were segments: "notes/v1" there was - // one text part for a Custom column, so reading it as a label - // would guess at the context. - "context as joined text": { - header + "\ntable individuals\n" + strings.Replace(col, `["individuals", "email"]`, "individuals/email", 1), - "is not a list of segments", - }, - "context with a stray separator": { - header + "\ntable individuals\n" + strings.Replace(col, `"individuals", "email"`, `"individuals","email"`, 1), - `expected ", " or "]"`, - }, - "context with trailing text": { - header + "\ntable individuals\n" + strings.Replace(col, `"email"]`, `"email"] x`, 1), - `expected ", " or "]"`, - }, - "column without a context": { - header + "\ntable individuals\n" + strings.Replace(col, " context [\"individuals\", \"email\"]\n", "", 1), - `column "email" has no context or target line`, - }, - } { - t.Run(name, func(t *testing.T) { - _, err := parse([]byte(tc.text)) - if err == nil || !strings.Contains(err.Error(), tc.says) { - t.Errorf("parse error = %v, want one saying %q", err, tc.says) - } - }) - } - // The complete column parses, so each case above fails for its own - // reason. - if _, err := parse([]byte(header + "\ntable individuals\n" + col)); err != nil { - t.Errorf("the complete snapshot does not parse: %v", err) - } -} - -// A snapshot that is there but cannot be read is reported as that, not as -// a missing one: running -update would hide the cause. -func TestUnreadableSnapshotIsNotMissing(t *testing.T) { - path := filepath.Join(t.TempDir(), "TestPolicy.golden") - if err := os.Mkdir(path, 0o750); err != nil { - t.Fatal(err) - } - m := plan.ForMessage(nil, "individuals", base) - for _, update := range []bool{false, true} { - _, err := check(path, individualV1(), m, update, "RERUN") - mustContain(t, err, "plantest: ", path) - if strings.Contains(err.Error(), "no snapshot") || strings.Contains(err.Error(), "RERUN") { - t.Errorf("update=%v: an unreadable snapshot is reported as missing:\n%s", update, err) - } - } -} - -// -update fails when it cannot write the snapshot, rather than reporting -// one written. -func TestUpdateFailsWhenItCannotWrite(t *testing.T) { - if runtime.GOOS == "windows" || os.Geteuid() == 0 { - t.Skip("needs a directory the test cannot write to") - } - m := plan.ForMessage(nil, "individuals", base) - locked := t.TempDir() - if err := os.Chmod(locked, 0o500); err != nil { - t.Fatal(err) - } - t.Cleanup(func() { _ = os.Chmod(locked, 0o700) }) - for name, path := range map[string]string{ - "its directory cannot be made": filepath.Join(locked, "testdata", "TestPolicy.golden"), - "the file cannot be written": filepath.Join(locked, "TestPolicy.golden"), - } { - t.Run(name, func(t *testing.T) { - logs, err := check(path, individualV1(), m, true, "RERUN") - if !errors.Is(err, fs.ErrPermission) { - t.Fatalf("err = %v (logs %q), want a permission error", err, logs) - } - if logs != "" { - t.Errorf("logs = %q, want nothing claimed written", logs) - } - }) - } -} - -// A policy that does not build fails the test with the build's error. -func TestPolicyThatDoesNotBuild(t *testing.T) { - narrow := plan.ForMessage(nil, "individuals", plan.When(category.Under("user.contact"), plan.Encrypt(plan.EQL()))) - _, err := check(filepath.Join(t.TempDir(), "x.golden"), individualV1(), narrow, true, "RERUN") - if !errors.Is(err, plan.ErrUnmatched) { - t.Fatalf("err = %v, want ErrUnmatched", err) - } - mustContain(t, err, "the policy does not build", "medicare_number") - if _, err := check("x.golden", nil, narrow, true, "RERUN"); err == nil || !strings.Contains(err.Error(), "needs a Source") { - t.Errorf("nil source: %v", err) - } -} - -// A message the policy encrypts nothing of has a snapshot: the fields it -// decided Plaintext. -func TestNothingEncrypted(t *testing.T) { - m := plan.ForMessage(nil, "individuals", plan.When(category.Present(), plan.Plaintext())) - _, written := record(t, individualV1(), m) - if strings.Contains(string(written), "column ") || !strings.Contains(string(written), "plaintext medicare_number") { - t.Fatalf("snapshot:\n%s", written) - } -} - -func TestGoldenPathAndRerun(t *testing.T) { - for name, want := range map[string]string{ - "TestPolicy": filepath.Join("testdata", "TestPolicy.golden"), - "TestPolicy/individuals": filepath.Join("testdata", "TestPolicy", "individuals.golden"), - "TestPolicy/v1.2+build_name-x": filepath.Join("testdata", "TestPolicy", "v1.2+build_name-x.golden"), - "TestPolicy/CONSOLE": filepath.Join("testdata", "TestPolicy", "CONSOLE.golden"), - // Spelled differently, so hashed. - "TestPolicy/a:b*c?": filepath.Join("testdata", "TestPolicy", "a_b_c_~e7b9b9a8.golden"), - "TestPolicy/..": filepath.Join("testdata", "TestPolicy", "___~a3d4a70d.golden"), - "TestPolicy/name.": filepath.Join("testdata", "TestPolicy", "name_~19e5c1d8.golden"), - "TestPolicy/name../x": filepath.Join("testdata", "TestPolicy", "name__~bab05642", "x.golden"), - // Windows reserves device names, with any extension: the hash goes - // before the first dot, so the stem is no longer one. - "TestPolicy/CON": filepath.Join("testdata", "TestPolicy", "CON~3367e86b.golden"), - "TestPolicy/nul.golden": filepath.Join("testdata", "TestPolicy", "nul~80e138eb.golden.golden"), - "TestPolicy/Com1.a.b": filepath.Join("testdata", "TestPolicy", "Com1~14b3c4c0.a.b.golden"), - "TestPolicy/LPT9": filepath.Join("testdata", "TestPolicy", "LPT9~891f37a2.golden"), - "AUX/individuals": filepath.Join("testdata", "AUX~679e8439", "individuals.golden"), - } { - if got := goldenPath(name); got != want { - t.Errorf("goldenPath(%q) = %q, want %q", name, got, want) - } - } - // Names that differ only in what is respelled, or in a respelling and - // what it is respelled to, keep their own snapshots. - for _, pair := range [][2]string{{"T/a:b", "T/a?b"}, {"T/a:b", "T/a_b"}, {"T/CON", "T/CON_"}, {"T/x.", "T/x_"}, {"T/..", "T/___"}} { - if a, b := goldenPath(pair[0]), goldenPath(pair[1]); a == b { - t.Errorf("goldenPath(%q) and goldenPath(%q) are both %q", pair[0], pair[1], a) - } - } - for name, want := range map[string]string{ - "TestPolicy": `go test -run '^TestPolicy$' -update`, - "TestPolicy/a.b": `go test -run '^TestPolicy$/^a\.b$' -update`, - } { - if got := rerun(name); got != want { - t.Errorf("rerun(%q) = %s, want %s", name, got, want) - } - } -} - -// updateFlag points lookupFlag at a flag set of the test's own holding -// -update at on, so no test sets the flag the whole binary shares. -func updateFlag(t *testing.T, on bool) { - t.Helper() - fs := flag.NewFlagSet(t.Name(), flag.ContinueOnError) - fs.Bool("update", on, "") - saved := lookupFlag - lookupFlag = fs.Lookup - t.Cleanup(func() { lookupFlag = saved }) -} - -func TestUpdateFlag(t *testing.T) { - if flag.Lookup("update") == nil { - t.Fatal("-update is not registered") - } - updateFlag(t, true) - if !updating() { - t.Error("updating() does not follow -update") - } - updateFlag(t, false) - if updating() { - t.Error("updating() is on without -update") - } - lookupFlag = func(string) *flag.Flag { return nil } - if updating() { - t.Error("updating() is on with no -update flag") - } -} - -// recorder stands in for *testing.T to read what Golden reports. -type recorder struct { - *testing.T - failed bool - msg string -} - -func (r *recorder) Error(args ...any) { r.failed = true; r.msg = fmt.Sprint(args...) } - -// Golden itself fails the test, naming the snapshot it looked for, and -// not only check: a Golden that logged instead would pass every caller. -func TestGoldenReportsAMissingSnapshot(t *testing.T) { - updateFlag(t, false) // never write into testdata, even under -update - r := &recorder{T: t} - Golden(r, individualV1(), plan.ForMessage(nil, "individuals", base)) - if !r.failed { - t.Fatal("Golden passed with no snapshot checked in") - } - if !strings.Contains(r.msg, filepath.Join("testdata", t.Name()+".golden")) { - t.Errorf("message does not name the snapshot path: %s", r.msg) - } - if !strings.Contains(r.msg, rerun(t.Name())) { - t.Errorf("message does not give the command that writes it: %s", r.msg) - } -} diff --git a/languages/golang/encrypt/plan/plantest/snapshot.go b/languages/golang/encrypt/plan/plantest/snapshot.go deleted file mode 100644 index 8c9a3580f..000000000 --- a/languages/golang/encrypt/plan/plantest/snapshot.go +++ /dev/null @@ -1,454 +0,0 @@ -package plantest - -import ( - "bytes" - "errors" - "fmt" - "slices" - "strconv" - "strings" - - "github.com/cipherstash/stack/languages/golang/encrypt" - "github.com/cipherstash/stack/languages/golang/encrypt/plan" -) - -// header opens every snapshot. One sentence per line: the file is read in -// diffs, where a rewrapped paragraph would read as a change. -const header = `# Written by plantest.Golden: what the policy stores each field it decides as. -# Regenerate it with go test -update; do not edit it by hand. -# A column's context is bound into every ciphertext and query term written under it, so once a row is written it must never change. -` - -// snapshot is what a message's policy stores each field it decides as: -// the persistent half of a plan, which a rename the policy pins leaves -// alone. -type snapshot struct { - table string - columns []column // encrypted fields, by column - plaintext []plain // fields decided Plaintext, by field -} - -// column is one encrypted field, named by its record key. -type column struct { - name string - // context is the label the column binds, spelled as its segments by - // [shape]: ["individuals", "email"]. The segments, not a joined text, - // are what a context is, so the file spells them: "notes/v1" as one - // text part and as two segments are different contexts that read alike. - context string - // kind is how the context is chosen: kindEQL when it is the column - // identity, ["
", ""], so the table and a column rename - // move it; kindCustom when the target supplies it whatever the column, - // so several columns may share it. Both bind a label, so an EQL column - // and a Custom one with the same segments bind the same context. - kind string - terms []string - facts []fact - // from is the field it is decided from now; nil when read from a file. - from *decided -} - -// plain is one field decided Plaintext, named by its schema name. -type plain struct { - field string - facts []fact - from *decided -} - -// decided is a field of the message as the policy sees it now. -type decided struct { - fact plan.Fact - decision plan.Decision -} - -// The kinds of target a column is stored as. -const ( - kindEQL = "EQL" - kindCustom = "Custom" -) - -// targetKind is kindEQL when t binds the column identity as its context, -// as [plan.EQL] does, and kindCustom when it binds anything else, as -// [plan.Custom] does. It asks the target rather than its type, so a typed -// EQL target counts as EQL. It asks twice, with two identities: a Custom -// context is fixed, so it may equal one of them but never both. A target -// that refuses either identity is an error, not a kind. -func targetKind(t plan.Target) (string, error) { - for _, probe := range []plan.Identifier{{Table: "plantest", Column: "a"}, {Table: "plantest", Column: "b"}} { - got, err := t.Context(probe) - if err != nil { - return "", fmt.Errorf("plantest: target %v refuses the identity %s: %w", t, probe, err) - } - label, err := probe.Label() - if err != nil { - return "", fmt.Errorf("plantest: the probe identity %s: %w", probe, err) - } - if !got.Equal(label.Context()) { - return kindCustom, nil - } - } - return kindEQL, nil -} - -// contextText is the [shape] a snapshot stores for a column's context. An -// EQL column's context is its identity's label, (table, column identity); -// a Custom column's is the label [plan.Custom] parses from its argument. -// The shape is checked against the context the plan actually binds, so -// the file cannot name a context the plan does not use. -func contextText(table string, kind string, d plan.Decision, fp encrypt.FieldPlan) (string, error) { - var label encrypt.Label - switch kind { - case kindEQL: - identity := d.Identity() - if identity == "" { - identity = fp.Name - } - var err error - if label, err = (plan.Identifier{Table: table, Column: identity}).Label(); err != nil { - return "", fmt.Errorf("plantest: column %q: %w", fp.Name, err) - } - default: - // A Custom context has no accessor: plan.Custom renders its - // argument quoted, as Custom("", ...), so read it back. - target, _ := d.Target() - spelled, ok := strings.CutPrefix(fmt.Sprint(target), "Custom(") - var text string - var err error - if ok { - if text, err = strconv.QuotedPrefix(spelled); err == nil { - text, err = strconv.Unquote(text) - } - } - if !ok || err != nil { - return "", fmt.Errorf("plantest: column %q: cannot spell the context of target %v; a target that does not bind its column identity must be plan.Custom", fp.Name, target) - } - if label, err = encrypt.ParseLabel(text); err != nil { - return "", fmt.Errorf("plantest: column %q: %w", fp.Name, err) - } - } - if !label.Context().Equal(fp.Context) { - return "", fmt.Errorf("plantest: column %q: the plan binds a context other than %s", fp.Name, shape(label.Segments())) - } - return shape(label.Segments()), nil -} - -// shape spells a label's segments as a snapshot stores them: a bracketed, -// comma-separated list of Go string literals, ["individuals", "email"]. -// Every segment is quoted, so the spelling does not depend on which -// characters a segment holds. -func shape(segments []string) string { - quoted := make([]string, len(segments)) - for i, s := range segments { - quoted[i] = strconv.Quote(s) - } - return "[" + strings.Join(quoted, ", ") + "]" -} - -// segmentsOf reads a [shape] back into its segments. It accepts only what -// shape writes, so a context in any other spelling (a file written before -// contexts were spelled as segments) is refused rather than guessed at. -func segmentsOf(spelled string) ([]string, error) { - rest, ok := strings.CutPrefix(spelled, "[") - if !ok { - return nil, fmt.Errorf("context %s is not a list of segments such as [\"table\", \"column\"]; a file written by an older plantest spells contexts differently, so read what the new one writes before recording it", spelled) - } - var out []string - for { - q, err := strconv.QuotedPrefix(rest) - if err != nil { - return nil, fmt.Errorf("context %s: bad segment", spelled) - } - seg, _ := strconv.Unquote(q) - out = append(out, seg) - rest = rest[len(q):] - if r, ok := strings.CutPrefix(rest, ", "); ok { - rest = r - continue - } - if rest != "]" { - return nil, fmt.Errorf("context %s: expected \", \" or \"]\" after a segment", spelled) - } - if shape(out) != spelled { - return nil, fmt.Errorf("context %s is not spelled as plantest writes it", spelled) - } - return out, nil - } -} - -// contextOf is the context a snapshot's column names, rebuilt from its -// segments: what [contextText] wrote. -func contextOf(c column) (encrypt.Context, error) { - segments, err := segmentsOf(c.context) - if err != nil { - return encrypt.Context{}, err - } - label, err := encrypt.NewLabel(segments...) - if err != nil { - return encrypt.Context{}, err - } - return label.Context(), nil -} - -// fact is one annotation value. -type fact struct{ key, value string } - -// take builds m's plan from src, as plan.PlanFor does at startup, and -// snapshots it. It returns the facts as well, for the hints a comparison -// builds. -func take(src plan.Source, m plan.Message) (snapshot, []plan.Fact, error) { - if src == nil { - return snapshot{}, nil, errors.New("plantest: Golden needs a Source") - } - p, err := plan.PlanFor(src, m) - if err != nil && !errors.Is(err, plan.ErrNothingEncrypted) { - return snapshot{}, nil, err - } - facts, err := src.Facts(m.Msg()) - if err != nil { - return snapshot{}, nil, err - } - planned := map[string]encrypt.FieldPlan{} - for _, fp := range p.Fields() { - planned[fp.Field] = fp - } - s := snapshot{table: string(m.Table())} - seen := map[string]bool{} - for _, f := range facts { - d, ok := m.Decide(f) - if !ok { - // Unclassified and unnamed: PlanFor has already refused a - // classified field no rule decides. - continue - } - if seen[f.Field] { - return snapshot{}, nil, fmt.Errorf("plantest: the source gives field %q twice", f.Field) - } - seen[f.Field] = true - from := &decided{fact: f, decision: d} - if _, encrypted := d.Target(); !encrypted { - s.plaintext = append(s.plaintext, plain{field: f.Field, facts: factsOf(f), from: from}) - continue - } - target, _ := d.Target() - fp, ok := planned[goField(f)] - if !ok { - return snapshot{}, nil, fmt.Errorf("plantest: %s is decided %v but is not in the plan", f, d) - } - terms := make([]string, len(fp.Terms)) - for i, k := range fp.Terms { - terms[i] = k.String() - } - kind, err := targetKind(target) - if err != nil { - return snapshot{}, nil, err - } - context, err := contextText(s.table, kind, d, fp) - if err != nil { - return snapshot{}, nil, err - } - s.columns = append(s.columns, column{name: fp.Name, context: context, kind: kind, terms: terms, facts: factsOf(f), from: from}) - } - s.sort() - return s, facts, nil -} - -// goField is the Go field a fact binds to, as the plan package spells it. -func goField(f plan.Fact) string { - if f.GoField != "" { - return f.GoField - } - return f.Field -} - -// factsOf is a field's annotations as sorted, distinct (key, value) pairs: -// their order in the schema means nothing to a policy. -func factsOf(f plan.Fact) []fact { - var out []fact - for _, a := range f.Annotations { - for _, v := range a.Values { - out = append(out, fact{a.Key, v}) - } - } - sortFacts(out) - return slices.Compact(out) -} - -func sortFacts(fs []fact) { - slices.SortFunc(fs, func(a, b fact) int { - if c := strings.Compare(a.key, b.key); c != 0 { - return c - } - return strings.Compare(a.value, b.value) - }) -} - -func (s *snapshot) sort() { - slices.SortFunc(s.columns, func(a, b column) int { return strings.Compare(a.name, b.name) }) - slices.SortFunc(s.plaintext, func(a, b plain) int { return strings.Compare(a.field, b.field) }) -} - -// render spells the snapshot. parse reads it back. -func (s snapshot) render() []byte { - var b bytes.Buffer - b.WriteString(header) - fmt.Fprintf(&b, "\ntable %s\n", token(s.table)) - for _, c := range s.columns { - fmt.Fprintf(&b, "\ncolumn %s\n", token(c.name)) - fmt.Fprintf(&b, " context %s\n", c.context) - fmt.Fprintf(&b, " target %s\n", c.kind) - fmt.Fprintf(&b, " terms %s\n", termList(c.terms)) - writeFacts(&b, c.facts) - } - for _, p := range s.plaintext { - fmt.Fprintf(&b, "\nplaintext %s\n", token(p.field)) - writeFacts(&b, p.facts) - } - return b.Bytes() -} - -// termList spells a column's terms, "none" for none: an unindexed column -// is a decision too, and says nothing about the value but its ciphertext. -// Each term goes through token like every other value, so a term name -// with a space could not split into two. Term names are TermKind.String() -// values, none of them "none". -func termList(terms []string) string { - if len(terms) == 0 { - return "none" - } - parts := make([]string, len(terms)) - for i, t := range terms { - parts[i] = token(t) - } - return strings.Join(parts, " ") -} - -func writeFacts(b *bytes.Buffer, facts []fact) { - for _, f := range facts { - fmt.Fprintf(b, " fact %s %s\n", token(f.key), token(f.value)) - } -} - -// token spells a value: bare when it is a plain identifier, a Go string -// literal otherwise, so a space, a quote or a newline in a name or a -// context cannot change how the line reads. -func token(s string) string { - if s == "" { - return strconv.Quote(s) - } - for i := 0; i < len(s); i++ { - if !bare(s[i]) { - return strconv.Quote(s) - } - } - return s -} - -// bare reports whether c may appear in an unquoted value. -func bare(c byte) bool { - return 'a' <= c && c <= 'z' || 'A' <= c && c <= 'Z' || '0' <= c && c <= '9' || - c == '_' || c == '.' || c == '/' || c == '-' || c == ':' || c == '@' || c == '+' -} - -// parse reads a snapshot render wrote. It is strict about structure and -// lenient about nothing else: any byte difference fails the comparison -// anyway, and parse only lets the failure say what changed. -func parse(data []byte) (snapshot, error) { - var s snapshot - var table bool - // The entry the indented lines below belong to: a column (>= 0), a - // plaintext field (plainAt >= 0), or neither. - colAt, plainAt := -1, -1 - for n, line := range strings.Split(string(normalize(data)), "\n") { - if strings.TrimSpace(line) == "" || strings.HasPrefix(line, "#") { - continue - } - indented := strings.HasPrefix(line, " ") - // A context line holds a shape, not tokens: its segments are - // quoted and separated by ", ". - if spelled, ok := strings.CutPrefix(line, " context "); ok && colAt >= 0 { - if _, err := segmentsOf(spelled); err != nil { - return snapshot{}, fmt.Errorf("line %d: %w", n+1, err) - } - s.columns[colAt].context = spelled - continue - } - toks, err := tokens(strings.TrimSpace(line)) - if err != nil { - return snapshot{}, fmt.Errorf("line %d: %w", n+1, err) - } - switch { - case !indented && toks[0] == "table" && len(toks) == 2 && !table: - s.table, table = toks[1], true - case !indented && toks[0] == "column" && len(toks) == 2: - s.columns = append(s.columns, column{name: toks[1]}) - colAt, plainAt = len(s.columns)-1, -1 - case !indented && toks[0] == "plaintext" && len(toks) == 2: - s.plaintext = append(s.plaintext, plain{field: toks[1]}) - colAt, plainAt = -1, len(s.plaintext)-1 - case indented && toks[0] == "target" && len(toks) == 2 && colAt >= 0 && (toks[1] == kindEQL || toks[1] == kindCustom): - s.columns[colAt].kind = toks[1] - case indented && toks[0] == "terms" && len(toks) >= 2 && colAt >= 0: - if len(toks) == 2 && toks[1] == "none" { - s.columns[colAt].terms = nil - } else { - s.columns[colAt].terms = toks[1:] - } - case indented && toks[0] == "fact" && len(toks) == 3 && colAt >= 0: - s.columns[colAt].facts = append(s.columns[colAt].facts, fact{toks[1], toks[2]}) - case indented && toks[0] == "fact" && len(toks) == 3 && plainAt >= 0: - s.plaintext[plainAt].facts = append(s.plaintext[plainAt].facts, fact{toks[1], toks[2]}) - default: - return snapshot{}, fmt.Errorf("line %d: unexpected %q", n+1, line) - } - } - if !table { - return snapshot{}, errors.New("no table line") - } - // compare reads a column's context and target kind: one without them - // would read as a Custom-free, contextless column, and a table rename - // as harmless. - for _, c := range s.columns { - if c.kind == "" || c.context == "" { - return snapshot{}, fmt.Errorf("column %q has no context or target line", c.name) - } - } - for i := range s.columns { - sortFacts(s.columns[i].facts) - } - for i := range s.plaintext { - sortFacts(s.plaintext[i].facts) - } - s.sort() - return s, nil -} - -// tokens splits a line into its values, unquoting the quoted ones. -func tokens(line string) ([]string, error) { - var out []string - for line != "" { - if line[0] == '"' { - q, err := strconv.QuotedPrefix(line) - if err != nil { - return nil, fmt.Errorf("bad quoted value in %q", line) - } - v, err := strconv.Unquote(q) - if err != nil { - return nil, fmt.Errorf("bad quoted value in %q", line) - } - out = append(out, v) - line = line[len(q):] - } else { - end := strings.IndexByte(line, ' ') - if end < 0 { - end = len(line) - } - out = append(out, line[:end]) - line = line[end:] - } - line = strings.TrimLeft(line, " ") - } - if len(out) == 0 { - return nil, errors.New("empty line") - } - return out, nil -} diff --git a/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/audits.golden b/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/audits.golden deleted file mode 100644 index 7c0ccce9f..000000000 --- a/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/audits.golden +++ /dev/null @@ -1,8 +0,0 @@ -# Written by plantest.Golden: what the policy stores each field it decides as. -# Regenerate it with go test -update; do not edit it by hand. -# A column's context is bound into every ciphertext and query term written under it, so once a row is written it must never change. - -table audits - -plaintext kind - fact fides.data_categories system.operations diff --git a/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/individuals.golden b/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/individuals.golden deleted file mode 100644 index fa5a7bbff..000000000 --- a/languages/golang/encrypt/plan/plantest/testdata/TestPolicies/individuals.golden +++ /dev/null @@ -1,32 +0,0 @@ -# Written by plantest.Golden: what the policy stores each field it decides as. -# Regenerate it with go test -update; do not edit it by hand. -# A column's context is bound into every ciphertext and query term written under it, so once a row is written it must never change. - -table individuals - -column email - context ["individuals", "email"] - target EQL - terms eq match - fact fides.data_categories user.contact.email - -column medicare_num - context ["individuals", "medicare_number"] - target EQL - terms eq - fact fides.data_categories user.government_id - -column name - context ["individuals", "name"] - target EQL - terms ore - fact fides.data_categories user.name - -column notes - context ["individuals-notes", "v1"] - target Custom - terms none - fact fides.data_categories user.content - -plaintext country - fact fides.data_categories system.operations diff --git a/languages/golang/encrypt/plan/policy.go b/languages/golang/encrypt/plan/policy.go deleted file mode 100644 index 2811b3fd2..000000000 --- a/languages/golang/encrypt/plan/policy.go +++ /dev/null @@ -1,374 +0,0 @@ -package plan - -import ( - "errors" - "fmt" - "slices" - "strings" - - "github.com/cipherstash/stack/languages/golang/encrypt" -) - -// Identifier is a field's column identity: the table its message is -// stored in and the column its data was first written to. For an EQL -// target it is the field's encryption context, so it is fixed at first -// write and must never change — which is why the table is given, never -// derived from a message name, and why a rule can pin the column half -// ([Identity]) apart from the column the value is stored in ([Column]) -// once the database column is renamed. -type Identifier struct { - Table string - Column string -} - -// String joins the table and the column with '/', for messages. The context -// an EQL target binds, and the descriptor ZeroKMS logs for it, is -// [Identifier.Label], which refuses a name this joining would misrender. -func (id Identifier) String() string { return id.Table + "/" + id.Column } - -// Label is the identifier as the two-segment [encrypt.Label] an EQL -// target binds: the shape the Rust derive gives a -// `#[stash(struct = T, context = "
")]` field, and what EQL's own -// Identifier describes. It is refused when either half is not plain — -// contains '/', '(' or ')', a control character, or begins with "b64:", a -// digit or '-' — since such a name would not render as itself. -func (id Identifier) Label() (encrypt.Label, error) { - return encrypt.NewLabel(id.Table, id.Column) -} - -// Target is what an encrypted field is stored as: the index terms derived -// beside its ciphertext, and the context it binds. The context is the -// AAD, the ZeroKMS data-key binding and the terms' PRF context at once. -type Target interface { - // Terms lists the index terms to derive, in order. - Terms() []encrypt.TermKind - // Context returns the field's context given its column identity. An - // EQL target binds id.Label(); a custom target returns its own. - Context(id Identifier) (encrypt.Context, error) -} - -// EQL is an EQL column target: the field binds its column identity -// ([Identifier.Label]) as its context and derives the given terms. Typed -// EQL targets (a text-with-equality column, say) are this with the terms -// filled in, and implement [Target] the same way. -func EQL(terms ...encrypt.TermKind) Target { - return eqlTarget{terms: slices.Clone(terms)} -} - -type eqlTarget struct{ terms []encrypt.TermKind } - -func (t eqlTarget) Terms() []encrypt.TermKind { return slices.Clone(t.terms) } -func (t eqlTarget) Context(id Identifier) (encrypt.Context, error) { - l, err := id.Label() - if err != nil { - // The label error names a segment index; the caller gave a table and - // a column identity, so say which of those it was. - half, name := "table", id.Table - var le *encrypt.LabelError - if errors.As(err, &le) && le.Index == 1 { - half, name = "column identity", id.Column - } - return encrypt.Context{}, fmt.Errorf("%s %q cannot name a context: %w", half, name, err) - } - return l.Context(), nil -} -func (t eqlTarget) String() string { return "EQL(" + termList(t.terms) + ")" } - -// Custom is a non-EQL target: the field binds context, whatever its -// column, and derives the given terms. The context is the policy's to -// choose and, like any context, must never change once data is written -// under it. It is a label of at least two plain segments, written as -// [encrypt.ParseLabel] reads it ("notes/v1": the segments notes and -// v1, rendered as written in the ZeroKMS log), since that is the one shape -// of context a planned field binds; the message's table plays no part in -// it. A table and a column are an [EQL] target. -func Custom(context string, terms ...encrypt.TermKind) Target { - return customTarget{context: context, terms: slices.Clone(terms)} -} - -type customTarget struct { - context string - terms []encrypt.TermKind -} - -func (t customTarget) Terms() []encrypt.TermKind { return slices.Clone(t.terms) } -func (t customTarget) Context(Identifier) (encrypt.Context, error) { - if t.context == "" { - return encrypt.Context{}, errors.New("an empty string is an empty context") - } - l, err := encrypt.ParseLabel(t.context) - if err != nil { - return encrypt.Context{}, fmt.Errorf("context %q is not a label: %w", t.context, err) - } - return l.Context(), nil -} -func (t customTarget) String() string { - return fmt.Sprintf("Custom(%q%s)", t.context, prefixed(termList(t.terms))) -} - -func termList(terms []encrypt.TermKind) string { - names := make([]string, len(terms)) - for i, k := range terms { - names[i] = k.String() - } - return strings.Join(names, ", ") -} - -func prefixed(s string) string { - if s == "" { - return "" - } - return ", " + s -} - -type verdict uint8 - -const ( - sealed verdict = iota + 1 - plaintext - fail -) - -// Decision is what a policy decides for one field: encrypt it into a -// target, leave it plaintext, or refuse it. Build one with [Encrypt], -// [Plaintext] or [Fail]. -type Decision struct { - verdict verdict - target Target - reason string - column string - identity string -} - -// Encrypt decides that the field is encrypted into target. -func Encrypt(target Target) Decision { return Decision{verdict: sealed, target: target} } - -// Plaintext decides that the field is stored as it is: not part of the -// plan, never sent to the guest. -func Plaintext() Decision { return Decision{verdict: plaintext} } - -// Fail decides that the field must not be planned at all: building the -// plan fails, naming the field and reason. -func Fail(reason string) Decision { return Decision{verdict: fail, reason: reason} } - -// Target returns the decision's target, and whether it encrypts at all. -func (d Decision) Target() (Target, bool) { return d.target, d.verdict == sealed } - -// Column returns the column the decision stores the field in, or "" for -// the field's own name. -func (d Decision) Column() string { return d.column } - -// Identity returns the column half of the identity the decision pins, or -// "" for the effective column's (see [Identity]). -func (d Decision) Identity() string { return d.identity } - -// String spells the decision for tests and errors. -func (d Decision) String() string { - var s string - switch d.verdict { - case sealed: - s = fmt.Sprintf("Encrypt(%v)", d.target) - case plaintext: - s = "Plaintext()" - case fail: - s = fmt.Sprintf("Fail(%q)", d.reason) - default: - return "Decision{}" - } - if d.column != "" { - s += fmt.Sprintf(" Column(%q)", d.column) - } - if d.identity != "" { - s += fmt.Sprintf(" Identity(%q)", d.identity) - } - return s -} - -// Policy maps a field's facts to a decision, or reports no match (false). -// It is a pure function — no I/O, no client — so a policy is tested by -// calling it on hand-built facts. Policies compose with [FirstOf] and -// [Policy.OrElse]; [When] is the leaf. -type Policy func(Fact) (Decision, bool) - -// Decide runs the policy on one field. A nil policy matches nothing. -func (p Policy) Decide(f Fact) (Decision, bool) { - if p == nil { - return Decision{}, false - } - return p(f) -} - -// OrElse is p, falling back to q for the fields p does not match: the -// per-message refinement over a base policy. -func (p Policy) OrElse(q Policy) Policy { return FirstOf(p, q) } - -// FirstOf is the first of policies that matches a field, in order. Nil -// policies are skipped. -func FirstOf(policies ...Policy) Policy { - policies = slices.Clone(policies) - return func(f Fact) (Decision, bool) { - for _, p := range policies { - if d, ok := p.Decide(f); ok { - return d, true - } - } - return Decision{}, false - } -} - -// RuleOption adjusts the decision a [When] rule makes. -type RuleOption func(*Decision) - -// Column names the column an encrypted field is stored in: its record -// key ([encrypt.FieldPlan.Name]). It defaults to the field's schema -// name, so a rule needs it only when the two differ — after the field is -// renamed in the schema, say. For an EQL target, the column also sets the -// field's identity unless [Identity] pins another: on a field never -// renamed in the database, Column alone is enough. Only meaningful with -// [Encrypt]; building a plan refuses it elsewhere. An empty name is a -// programming error and panics: a pin that is not there would silently -// store the field under its own name instead. -func Column(name string) RuleOption { - if name == "" { - panic("plan.Column: empty column name") - } - return func(d *Decision) { d.column = name } -} - -// Identity pins the column half of an EQL field's identity -// ([Identifier]), and so its context "
/": the AAD bound to -// every stored ciphertext, its ZeroKMS data-key binding and its terms' PRF -// context. It defaults to the effective [Column], so a field whose -// database column has never been renamed needs no Identity. -// -// Once data is written, a field's identity must never change: rows -// written under the old one would no longer decrypt, and their terms would -// no longer match queries. A database rename (ALTER TABLE ... RENAME -// COLUMN) is therefore spelled as the new column and the old identity: -// -// plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(encrypt.Equality)), -// plan.Column("medicare_no"), plan.Identity("medicare_number")) -// -// Only meaningful with an EQL [Encrypt]: a [Custom] target's context is -// its own, and building a plan refuses Identity there and on [Plaintext]. -// An empty name is a programming error and panics, as for [Column]. -func Identity(name string) RuleOption { - if name == "" { - panic("plan.Identity: empty column name") - } - return func(d *Decision) { d.identity = name } -} - -// When decides d for the fields m matches, and matches nothing else. -func When(m Matcher, d Decision, opts ...RuleOption) Policy { - if m == nil { - panic("plan.When: nil matcher") - } - for _, opt := range opts { - opt(&d) - } - return func(f Fact) (Decision, bool) { - if !m(f) { - return Decision{}, false - } - return d, true - } -} - -// Matcher is a predicate over a field's facts. -type Matcher func(Fact) bool - -// Field matches the field whose schema name ([Fact.Field]) is name. -func Field(name string) Matcher { - return func(f Fact) bool { return f.Field == name } -} - -// Kind matches fields of the given kind, in the source's spelling. -func Kind(kind string) Matcher { - return func(f Fact) bool { return f.Kind == kind } -} - -// Any matches when at least one of ms does. A nil matcher, or none at -// all, is a programming error and panics here, as it does in [When]. -func Any(ms ...Matcher) Matcher { - ms = matchers("plan.Any", ms) - return func(f Fact) bool { - return slices.ContainsFunc(ms, func(m Matcher) bool { return m(f) }) - } -} - -// All matches when every one of ms does. A nil matcher, or none at all, -// is a programming error and panics here, as it does in [When]: with no -// matchers All would match every field, so a rule built from a slice that -// came back empty would decide every field below it. A policy that needs -// a catch-all writes a final rule whose matcher says so. -func All(ms ...Matcher) Matcher { - ms = matchers("plan.All", ms) - return func(f Fact) bool { - for _, m := range ms { - if !m(f) { - return false - } - } - return true - } -} - -// Not matches when m does not. A nil matcher is a programming error and -// panics here, as it does in [When]. -func Not(m Matcher) Matcher { - if m == nil { - panic("plan.Not: nil matcher") - } - return func(f Fact) bool { return !m(f) } -} - -// matchers is a combinator's own copy of its matchers, none of them nil: -// a later write to the caller's slice must not change the matcher, and a -// nil found now names the combinator instead of crashing a build. -func matchers(combinator string, ms []Matcher) []Matcher { - if len(ms) == 0 { - panic(combinator + ": no matchers; All() would match every field and Any() none") - } - for i, m := range ms { - if m == nil { - panic(fmt.Sprintf("%s: nil matcher at index %d", combinator, i)) - } - } - return slices.Clone(ms) -} - -// Key is an annotation key, the handle rules match annotations through. A -// protobuf fact source hands out a Key per extension; for struct tags it -// is the tag's key: -// -// var category = plan.Key("fides.data_categories") -type Key string - -// Present matches fields with any value under the key. -func (k Key) Present() Matcher { - return func(f Fact) bool { return f.hasValue(string(k), func(string) bool { return true }) } -} - -// Is matches fields with value among their values under the key. -func (k Key) Is(value string) Matcher { - return func(f Fact) bool { return f.hasValue(string(k), func(v string) bool { return v == value }) } -} - -// Under matches fields with a value under the key at or below prefix in a -// dot-separated taxonomy (Fideslang's): "user.contact" matches -// "user.contact" and "user.contact.email", not "user.contactless". The -// prefix is one or more non-empty dot-separated segments; anything else -// ("", "user.", ".user", "a..b") could match nothing while reading as a -// catch-all, so it is a programming error and panics. -func (k Key) Under(prefix string) Matcher { - if prefix == "" || slices.Contains(strings.Split(prefix, "."), "") { - panic(fmt.Sprintf("plan.Key(%q).Under(%q): a prefix is one or more non-empty dot-separated segments", string(k), prefix)) - } - below := prefix + "." - return func(f Fact) bool { - return f.hasValue(string(k), func(v string) bool { - return v == prefix || strings.HasPrefix(v, below) - }) - } -} diff --git a/languages/golang/encrypt/policy_plan_test.go b/languages/golang/encrypt/policy_plan_test.go deleted file mode 100644 index 006142f3d..000000000 --- a/languages/golang/encrypt/policy_plan_test.go +++ /dev/null @@ -1,98 +0,0 @@ -package encrypt_test - -import ( - "bytes" - "reflect" - "strings" - "testing" - - se "github.com/cipherstash/stack/languages/golang/encrypt" - "github.com/cipherstash/stack/languages/golang/encrypt/plan" - "github.com/cipherstash/stack/languages/golang/internal/factstest" -) - -// Validate refuses a nil type as PlanFromTags does, for the zero plan -// and a built one alike, rather than dereferencing it. -func TestValidateRefusesANilType(t *testing.T) { - built, err := se.NewPlan(se.FieldPlan{Field: "A", Context: label(t, "t/a").Context()}) - if err != nil { - t.Fatal(err) - } - for name, p := range map[string]se.Plan{"zero": {}, "built": built} { - if err := p.Validate(nil); err == nil || !strings.Contains(err.Error(), "nil type") { - t.Errorf("%s plan: Validate(nil) = %v, want an error naming the nil type", name, err) - } - } -} - -// A plan a policy builds is a Plan like any other: the guest receives -// byte-identical input to the equivalent plan built by hand. This test is -// here, not in package plan, because the guest encoding is unexported; -// plan imports encrypt, so only an external test can hold both. -func TestPolicyPlanIsTheHandBuiltPlan(t *testing.T) { - type individual struct { - ID int64 - Email string `facts:"fides.data_categories=user.contact.email"` - Name string `facts:"fides.data_categories=user.name"` - MedicareNo string `facts:"fides.data_categories=user.government_id"` - Country string `facts:"fides.data_categories=system.operations"` - } - category := plan.Key("fides.data_categories") - base := plan.FirstOf( - plan.When(category.Under("user.government_id"), plan.Encrypt(plan.EQL(se.Equality))), - plan.When(category.Under("user.contact.email"), plan.Encrypt(plan.EQL(se.Equality, se.Match))), - plan.When(category.Under("user"), plan.Encrypt(plan.EQL())), - plan.When(category.Present(), plan.Plaintext()), - ) - email := se.FieldPlan{Field: "Email", Name: "email", Context: label(t, "individuals/email").Context(), Terms: []se.TermKind{se.Equality, se.Match}} - name := se.FieldPlan{Field: "Name", Name: "name", Context: label(t, "individuals/name").Context()} - for label, tc := range map[string]struct { - pins []plan.RuleOption - medicare se.FieldPlan - }{ - // The schema spelling of each column, as a Rust derive or a - // database would have it; Column alone sets the identity too. - "column": { - []plan.RuleOption{plan.Column("medicare_number")}, - se.FieldPlan{Field: "MedicareNo", Name: "medicare_number", Context: label(t, "individuals/medicare_number").Context(), Terms: []se.TermKind{se.Equality, se.Ore}}, - }, - // After a database rename: the new column, the old identity. - "renamed column": { - []plan.RuleOption{plan.Column("medicare_num"), plan.Identity("medicare_number")}, - se.FieldPlan{Field: "MedicareNo", Name: "medicare_num", Context: label(t, "individuals/medicare_number").Context(), Terms: []se.TermKind{se.Equality, se.Ore}}, - }, - } { - individuals := plan.ForMessage(&individual{}, "individuals", plan.FirstOf( - plan.When(plan.Field("medicare_no"), plan.Encrypt(plan.EQL(se.Equality, se.Ore)), tc.pins...), - ).OrElse(base)) - fromPolicy := plan.MustPlanFor(factstest.StructTags, individuals) - byHand, err := se.NewPlan(email, name, tc.medicare) - if err != nil { - t.Fatal(err) - } - typ := reflect.TypeOf(individual{}) - for _, ext := range [][]any{nil, {uint64(7)}} { - a, err := se.GuestPlanInput(fromPolicy, typ, ext...) - if err != nil { - t.Fatal(err) - } - b, err := se.GuestPlanInput(byHand, typ, ext...) - if err != nil { - t.Fatal(err) - } - if !bytes.Equal(a, b) { - t.Fatalf("%s, extension %v: guest input differs:\npolicy %x\nhand %x", label, ext, a, b) - } - } - } -} - -// label is se.ParseLabel for a label the test knows to be valid. -func label(t testing.TB, s string) se.Label { - t.Helper() - l, err := se.ParseLabel(s) - if err != nil { - t.Fatal(err) - } - return l -} diff --git a/languages/golang/encrypt/record.go b/languages/golang/encrypt/record.go deleted file mode 100644 index ac8651607..000000000 --- a/languages/golang/encrypt/record.go +++ /dev/null @@ -1,924 +0,0 @@ -package encrypt - -import ( - "context" - "errors" - "fmt" - "reflect" - "slices" - "strings" - "sync" - - "github.com/cipherstash/vitaminc/bindings/go/vcffi" - "github.com/cipherstash/vitaminc/bindings/go/vcvalue" -) - -// Record plans: per field, which context to bind and which outputs to -// derive. A plan is a value ([Plan]) with two sources: `stash` struct tags -// ([PlanFromTags], the default, and the Go stand-in for the Rust derive), or -// an explicit plan built with [NewPlan] and passed through [WithPlan] — for -// structs whose source cannot carry a tag, such as generated code. -// -// A struct field's `stash` tag says what to do with it: -// -// type User struct { -// ID int64 `stash:"-"` // not sent to the guest -// Age uint32 `stash:"label=users/age,index=eq;ore"` // sealed + equality and ORE terms -// Email string `stash:"label=users/email,index=eq;match"` // sealed + equality and match terms -// Notes string `stash:"label=users/notes"` // sealed only -// } -// -// Options are comma-separated: the field's label, `label=
/` -// (a [Label] of at least two plain segments, parsed with [ParseLabel]), -// required for a planned field — see [FieldPlan.Context]; `index=[;]` -// (eq, match, ore, ope); and `name=` (the record key; the Go -// field name otherwise). The `context=` option, which named one text -// part as the field's whole context, is refused: the guest seals every -// field under the record's one context and the field's own identity, and a -// one-part context has neither. A field tagged `-` or `plain`, or not -// tagged at all, is not part of the record: it never crosses the boundary, -// and stays the caller's to store. Unexported fields are ignored. -// -// The same plan, built by hand. Each field's context is a [Label], the -// table and the column; [ParseLabel] refuses a name that would not render -// as itself, so check its error: -// -// age, err := encrypt.ParseLabel("users/age") -// email, err := encrypt.ParseLabel("users/email") -// notes, err := encrypt.ParseLabel("users/notes") -// plan, err := encrypt.NewPlan( -// encrypt.FieldPlan{ -// Field: "Age", -// Context: age.Context(), -// Terms: []encrypt.TermKind{ -// encrypt.Equality, encrypt.Ore, -// }, -// }, -// encrypt.FieldPlan{ -// Field: "Email", -// Context: email.Context(), -// Terms: []encrypt.TermKind{ -// encrypt.Equality, encrypt.Match, -// }, -// }, -// encrypt.FieldPlan{Field: "Notes", Context: notes.Context()}, -// ) -// records, err := cipher.EncryptRecords( -// ctx, users, encrypt.WithPlan(plan), -// ) -// -// Every planned field is sealed (the `"c"` output). What the guest receives -// is the same object whichever way the plan was built. The context each -// field binds is its label, extended by [ExtendContext] parts exactly as a -// Rust chain's .extend(..) extends a field's label: -// label.Context().With(p1).With(p2). - -// EncryptedField is one field's outputs from EncryptRecords: the sealed -// ciphertext and whichever index terms the plan asked for (nil otherwise). -type EncryptedField struct { - // Ciphertext is the field's sealed value: a Sealed leaf for a scalar, - // or the same nested shape Cipher.Encrypt returns for a composite. - Ciphertext any - Equality EqualityTerm - Match MatchTerm - Ore OreTerm - Ope OpeTerm -} - -// EncryptedRecord is one record's planned fields, by wire name. -type EncryptedRecord map[string]EncryptedField - -// RecordOption adjusts one record call: [Cipher.EncryptRecords], -// [Cipher.EncryptRecord], [Cipher.DecryptRecords], [Cipher.DecryptRecord] -// and the Client forms of the last two. An option is a value built by one -// of the functions below, and a call applies the options it is given in -// order. -// -// A RecordOption that is not also an [Option], such as [WithPlan], means -// something only on a record call, so handing it to [Cipher.Term] does not -// compile. -type RecordOption interface { - applyRecord(*recordOptions) -} - -// Option is a [RecordOption] that the probe call, [Cipher.Term], accepts -// as well: what a record and the probe that matches it must agree on. -// Every Option is a RecordOption, so one value serves the encrypt, decrypt -// and probe calls alike, and the three cannot drift apart: -// -// tenant := encrypt.ExtendContext(uint64(tenantID)) -// rows, err := cipher.EncryptRecords(ctx, users, tenant) -// probe, err := cipher.Term(ctx, "bob@example.com", email, encrypt.Equality, tenant) -// err = cipher.DecryptRecords(ctx, rows, &back, tenant) -type Option interface { - RecordOption - applyTerm(*termOptions) -} - -type recordOptions struct { - extension []any - plan Plan -} - -type termOptions struct { - extension []any -} - -// contextExtension is what [ExtendContext] returns. -type contextExtension struct{ parts []any } - -// appendTo adds the extension's parts after any an earlier option gave: -// the one rule for combining extensions, shared by the record calls and -// the probe so the two cannot combine them differently. -func (e contextExtension) appendTo(ext *[]any) { - *ext = append(*ext, e.parts...) -} - -func (e contextExtension) applyRecord(o *recordOptions) { e.appendTo(&o.extension) } - -func (e contextExtension) applyTerm(o *termOptions) { e.appendTo(&o.extension) } - -// ExtendContext extends every field's context by parts, in order, the way -// the Rust derive extends a field's context by the caller's -// (encrypt_into_with_context): a field tagged label=users/age with -// ExtendContext(uint64(7)) binds [["users", "age"], 7]. On [Cipher.Term] it -// extends the probe's context the same way, so a probe built under the -// extension a record was written under compares against that record's -// terms, and under any other extension, or none, against nothing. -// -// The same extension must be given to decrypt the records. A part's type -// is part of the context (an int crosses as int64, so uint64(7) and 7 are -// different contexts), which is why an extension is best held in one value -// and passed to every call rather than spelled afresh at each. The option -// owns its parts: a byte-slice part is copied, so a caller's buffer reused -// after the call does not change what the option extends by. -// -// Several ExtendContext options on one call join in order: -// ExtendContext(a), ExtendContext(b) is the same context as -// ExtendContext(a, b), on a record call and on Term alike. So each call -// must receive a given extension once. A helper that always adds the -// tenant, called by code that adds the tenant as well, writes records -// under [field, tenant, tenant], and a probe built with the tenant once -// matches none of them, with no error. -// -// A part is checked when a call applies it, not here: a part that is not -// a string, a byte slice or an integer fails the call it is given to. -func ExtendContext(parts ...any) Option { - owned := make([]any, len(parts)) - for i, part := range parts { - owned[i] = ownPart(part) - } - return contextExtension{parts: owned} -} - -// extend is c extended by every part of ext, in order: the one definition -// of how an extension applies, shared by the record plan and the probe so -// the two cannot disagree. -func extend(c Context, ext []any) (Context, error) { - for _, part := range ext { - var err error - if c, err = c.With(part); err != nil { - return Context{}, err - } - } - return c, nil -} - -// planOption is what [WithPlan] returns. -type planOption struct{ plan Plan } - -func (p planOption) applyRecord(o *recordOptions) { o.plan = p.plan } - -// WithPlan encrypts or decrypts records under an explicit plan instead of -// the struct's `stash` tags. -// -// What decryption needs from the encrypting plan is what names and keys -// the ciphertext: each field's record Name, its Context (extended by the -// same [ExtendContext] parts), and the set of fields that carry a -// ciphertext. Field only selects which Go field the plaintext is written -// to, so it may differ between the two sides: a record encrypted from a -// generated struct may be decrypted into a domain struct under a plan -// with the same Names and Contexts. Terms are one-way outputs, derived on -// encryption and never sent to decrypt, so they need not match either. -func WithPlan(p Plan) RecordOption { - return planOption{plan: p} -} - -// FieldPlan is one planned field of a record. -type FieldPlan struct { - // Field is the Go struct field name. It must be exported. - Field string - // Name is the record key the field's outputs are stored under: the - // column name, in EQL terms. Field when empty. - Name string - // Context is the field's label: the context it is sealed and indexed - // under, before the record call extends it by any ExtendContext parts. - // Required. - // - // It is a [Label] of at least two plain segments — a table and a - // column, ParseLabel("users/age") then .Context(): the pair ["users", "age"] - // a Rust `#[derive(EncryptFrom)]` with `struct = .., context = "users"` - // binds its `age` field under, rendering the ZeroKMS descriptor - // users/age. Nothing else: the guest lowers a record plan into one - // context per record (the label's leading segments) with one identity - // per field (its last), so a one-part context ([NewContext]), a - // one-segment label, a byte or integer part, and a Context already - // extended with [Context.With] are refused by [NewPlan], naming the - // field. A probe for the field ([Cipher.Term]) takes the same Context, - // so the two cannot drift. - Context Context - // Terms lists the terms to derive beside the ciphertext, in order. - Terms []TermKind -} - -// Plan is a record plan: which fields of a struct to seal, under which -// context, with which terms. It is an immutable value; the zero Plan -// means "the struct's tags". Build one with [NewPlan] or [PlanFromTags]. -type Plan struct { - d *planData -} - -// planData is the validated, shared body of a Plan. Every copy of the Plan -// points at the same body, so it is never mutated after NewPlan returns. -type planData struct { - fields []planField -} - -// planField is a validated FieldPlan: Name filled in, every term kind -// known and named once. -type planField struct { - field string - name string - context Context - terms []TermKind -} - -// outputs spells the field's outputs the way the guest reads them: the -// ciphertext first, then each term. -func (f planField) outputs() []string { - out := make([]string, 1, 1+len(f.terms)) - out[0] = "c" - for _, k := range f.terms { - out = append(out, k.String()) - } - return out -} - -// NewPlan validates the fields and returns the plan. Every field needs a -// Field and a Context that is a label of at least two plain segments (see -// [FieldPlan.Context]); Go field names must be unique, and so must record -// names (Name, or Field); Terms must be kinds this package defines, each -// at most once per field. These are the rules the guest holds a plan to -// when a record call runs, checked here so a mistake is found when the plan -// is built. Two rules stay with the call: every field of a plan must sit -// under one table (the label's leading segments), as a Rust chain's -// `Plan::context(table).fields()` does, and no two fields may bind one -// label, since they would share a context and their terms would be -// interchangeable; a plan that breaks either is refused by the record call -// that runs it. A plan is built once and reused across calls, like the type -// it describes. -func NewPlan(fields ...FieldPlan) (Plan, error) { - p, err := newPlan(fields) - if err != nil { - return Plan{}, fmt.Errorf("encrypt: %w", err) - } - return p, nil -} - -// newPlan is the one validation both constructors go through; its errors -// name the field, and the caller adds the prefix and, for tags, the type. -func newPlan(fields []FieldPlan) (Plan, error) { - if len(fields) == 0 { - return Plan{}, errors.New("a plan needs at least one field") - } - d := &planData{fields: make([]planField, 0, len(fields))} - seenField := make(map[string]bool, len(fields)) - seenName := make(map[string]bool, len(fields)) - for _, f := range fields { - if f.Field == "" { - return Plan{}, errors.New("plan field without a Field name") - } - if seenField[f.Field] { - return Plan{}, fmt.Errorf("plan field %s: the Go field is planned twice", f.Field) - } - seenField[f.Field] = true - if f.Context.isZero() { - return Plan{}, fmt.Errorf("plan field %s: a planned field needs a context", f.Field) - } - if _, err := f.Context.fieldLabel(); err != nil { - return Plan{}, fmt.Errorf("plan field %s: context %w", f.Field, err) - } - pf := planField{field: f.Field, name: f.Field, context: f.Context} - if f.Name != "" { - pf.name = f.Name - } - for _, k := range f.Terms { - if !k.valid() { - return Plan{}, fmt.Errorf("plan field %s: unknown term kind %s", f.Field, k) - } - if slices.Contains(pf.terms, k) { - return Plan{}, fmt.Errorf("plan field %s: term kind %s given twice", f.Field, k) - } - pf.terms = append(pf.terms, k) - } - if seenName[pf.name] { - return Plan{}, fmt.Errorf("two plan fields share the record name %q", pf.name) - } - seenName[pf.name] = true - d.fields = append(d.fields, pf) - } - return Plan{d: d}, nil -} - -// Fields returns the plan's fields, in order, as they were given to NewPlan -// (Name filled in). A copy: mutating it does not touch the plan. -func (p Plan) Fields() []FieldPlan { - if p.d == nil { - return nil - } - out := make([]FieldPlan, len(p.d.fields)) - for i, f := range p.d.fields { - out[i] = FieldPlan{Field: f.field, Name: f.name, Context: f.context, Terms: slices.Clone(f.terms)} - } - return out -} - -var tagPlans sync.Map // reflect.Type → Plan - -// PlanFromTags parses (and caches) the plan a struct type's `stash` tags -// describe. This is the plan the record calls use when no WithPlan option -// is given. The tag grammar is checked here; what the fields mean is -// checked by the same validation NewPlan runs. -func PlanFromTags(t reflect.Type) (Plan, error) { - if t == nil { - return Plan{}, errors.New("encrypt: records must be structs, not a nil type") - } - if cached, ok := tagPlans.Load(t); ok { - return cached.(Plan), nil - } - if t.Kind() != reflect.Struct { - return Plan{}, fmt.Errorf("encrypt: records must be structs, not %s", t) - } - var fields []FieldPlan - for i := 0; i < t.NumField(); i++ { - f := t.Field(i) - if !f.IsExported() { - continue - } - tag, ok := f.Tag.Lookup("stash") - if !ok || tag == "-" || tag == "plain" { - continue - } - pf := FieldPlan{Field: f.Name, Name: f.Name} - var ownKey, ownValue string // the option that set pf.Context, for the message on a second one - for _, opt := range strings.Split(tag, ",") { - key, value, _ := strings.Cut(opt, "=") - switch key { - case "label", "context": - if ownKey != "" { - return Plan{}, fmt.Errorf("encrypt: field %s.%s: %s= and %s= both given; a field has one own context", t, f.Name, ownKey, key) - } - ownKey, ownValue = key, value - var err error - if key == "label" { - var l Label - if l, err = ParseLabel(value); err == nil { - pf.Context = l.Context() - } - } else { - pf.Context, err = NewContext(value) - } - if err != nil { - return Plan{}, fmt.Errorf("encrypt: field %s.%s: %s=%q: %w", t, f.Name, key, value, err) - } - case "name": - if value == "" { - return Plan{}, fmt.Errorf("encrypt: field %s.%s: name must not be empty", t, f.Name) - } - pf.Name = value - case "index": - for _, k := range strings.Split(value, ";") { - kind, ok := parseTermKind(k) - if !ok { - return Plan{}, fmt.Errorf("encrypt: field %s.%s: unknown term kind %q", t, f.Name, k) - } - pf.Terms = append(pf.Terms, kind) - } - default: - return Plan{}, fmt.Errorf("encrypt: field %s.%s: unknown stash tag option %q", t, f.Name, opt) - } - } - // Checked after the options so a repeated or doubled option is - // reported as what the author wrote; NewPlan would refuse the one - // part too, but this names the tag to use instead. - if ownKey == "context" { - return Plan{}, fmt.Errorf("stackencrypt: field %s.%s: context=%q names one text part, which a planned field cannot bind; use label=
/", t, f.Name, ownValue) - } - fields = append(fields, pf) - } - if len(fields) == 0 { - return Plan{}, fmt.Errorf("encrypt: %s has no fields tagged for encryption", t) - } - plan, err := newPlan(fields) - if err != nil { - return Plan{}, fmt.Errorf("encrypt: %s: %w", t, err) - } - tagPlans.Store(t, plan) - return plan, nil -} - -// fieldPlan is one planned field bound to a struct type: the plan's field -// resolved to its index. -type fieldPlan struct { - index int // struct field index - name string // wire name - context Context // the field's own context - outputs []string -} - -// Validate checks that the plan binds to t: every planned field is an -// exported, direct field of the struct type. The record calls make the -// same check on every call and fail with the same error; this is for a -// caller that builds a plan at startup for a type it knows, so a field the -// type does not have is reported then rather than at the first write. The -// zero Plan is the type's own tags, and PlanFromTags validates those. -func (p Plan) Validate(t reflect.Type) error { - if p.d == nil { - _, err := PlanFromTags(t) - return err - } - if t == nil { - return errors.New("encrypt: records must be structs, not a nil type") - } - _, err := p.bind(t) - return err -} - -// bind resolves the plan's fields against a struct type. Not cached: a -// name lookup per field is far below the cost of the call it precedes, and -// a cache keyed by plan would grow with every plan a caller ever built. -func (p Plan) bind(t reflect.Type) ([]fieldPlan, error) { - if t.Kind() != reflect.Struct { - return nil, fmt.Errorf("encrypt: records must be structs, not %s", t) - } - bound := make([]fieldPlan, len(p.d.fields)) - for i, f := range p.d.fields { - sf, ok := t.FieldByName(f.field) - if !ok || !sf.IsExported() || len(sf.Index) != 1 { - return nil, fmt.Errorf("encrypt: plan field %s is not an exported field of %s", f.field, t) - } - bound[i] = fieldPlan{index: sf.Index[0], name: f.name, context: f.context, outputs: f.outputs()} - } - return bound, nil -} - -// planFor binds the plan a record call runs under: the option's, or the -// struct's tags. -func planFor(t reflect.Type, o recordOptions) ([]fieldPlan, error) { - p := o.plan - if p.d == nil { - var err error - if p, err = PlanFromTags(t); err != nil { - return nil, err - } - } - return p.bind(t) -} - -// planValue renders the plan object for the guest, each field's context -// extended by the options. -func planValue(plan []fieldPlan, opts recordOptions) (vcvalue.Object, error) { - out := make(vcvalue.Object, 0, len(plan)) - for _, f := range plan { - ctx, err := extend(f.context, opts.extension) - if err != nil { - return nil, err - } - outputs := make([]any, len(f.outputs)) - for i, o := range f.outputs { - outputs[i] = o - } - out = append(out, vcvalue.Field{Key: f.name, Value: vcvalue.Object{ - {Key: "context", Value: ctx.value()}, - {Key: "outputs", Value: outputs}, - }}) - } - return out, nil -} - -func applyOptions(opts []RecordOption) recordOptions { - var o recordOptions - for _, opt := range opts { - opt.applyRecord(&o) - } - return o -} - -// EncryptRecords seals every row of a slice of structs (or a pointer to -// one) per the struct's `stash` tags, or per [WithPlan]: all rows and -// fields from batched ZeroKMS key requests (one per 500 sealed fields), -// terms derived under this keyset's index key. One EncryptedRecord per -// row, in order. -func (cph *Cipher) EncryptRecords(ctx context.Context, rows any, opts ...RecordOption) ([]EncryptedRecord, error) { - v := reflect.Indirect(reflect.ValueOf(rows)) - if !v.IsValid() || v.Kind() != reflect.Slice { - return nil, fmt.Errorf("encrypt: EncryptRecords takes a slice of structs, not %T", rows) - } - o := applyOptions(opts) - plan, err := planFor(v.Type().Elem(), o) - if err != nil { - return nil, err - } - source := make([]any, v.Len()) - for i := range source { - source[i] = sourceRow(v.Index(i), plan) - } - tree, err := cph.encryptRecords(ctx, plan, source, o) - if err != nil { - return nil, err - } - items, ok := tree.([]any) - if !ok || len(items) != v.Len() { - return nil, fmt.Errorf("%w: record batch came back as %T", ErrInternal, tree) - } - out := make([]EncryptedRecord, len(items)) - for i, item := range items { - if out[i], err = encryptedRecordOf(item); err != nil { - return nil, err - } - } - return out, nil -} - -// EncryptRecord seals one struct (or a pointer to one) per its `stash` -// tags, or per [WithPlan]; see EncryptRecords. -func (cph *Cipher) EncryptRecord(ctx context.Context, row any, opts ...RecordOption) (EncryptedRecord, error) { - v := reflect.Indirect(reflect.ValueOf(row)) - if !v.IsValid() { - return nil, fmt.Errorf("encrypt: EncryptRecord takes a struct, not %T", row) - } - o := applyOptions(opts) - plan, err := planFor(v.Type(), o) - if err != nil { - return nil, err - } - tree, err := cph.encryptRecords(ctx, plan, sourceRow(v, plan), o) - if err != nil { - return nil, err - } - return encryptedRecordOf(tree) -} - -func sourceRow(row reflect.Value, plan []fieldPlan) vcvalue.Object { - out := make(vcvalue.Object, 0, len(plan)) - for _, f := range plan { - out = append(out, vcvalue.Field{Key: f.name, Value: row.Field(f.index).Interface()}) - } - return out -} - -func (cph *Cipher) encryptRecords(ctx context.Context, plan []fieldPlan, source any, o recordOptions) (any, error) { - planObj, err := planValue(plan, o) - if err != nil { - return nil, err - } - encodedSource, err := vcffi.Marshal(source) - if err != nil { - return nil, err - } - defer wipe(encodedSource) - encodedPlan, err := vcffi.Marshal(planObj) - if err != nil { - return nil, err - } - opts, err := vcffi.Marshal(options(cph.keyset)) - if err != nil { - return nil, err - } - out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { - return inst.call(ctx, inst.encryptRecord, buf(encodedSource), buf(encodedPlan), buf(opts)) - }) - if err != nil { - return nil, err - } - defer wipe(out) - return unmarshalCipherText(out) -} - -// encryptedRecordOf lifts one decoded record node into an EncryptedRecord. -func encryptedRecordOf(node any) (EncryptedRecord, error) { - fields, ok := node.(map[string]any) - if !ok { - return nil, fmt.Errorf("%w: record came back as %T", ErrInternal, node) - } - rec := make(EncryptedRecord, len(fields)) - for name, outputs := range fields { - om, ok := outputs.(map[string]any) - if !ok { - return nil, fmt.Errorf("%w: field %q came back as %T", ErrInternal, name, outputs) - } - var f EncryptedField - for key, out := range om { - if key == "c" { - f.Ciphertext = out - continue - } - term, err := termBytes(out) - if err != nil { - return nil, fmt.Errorf("%w: field %q output %q: %v", ErrInternal, name, key, err) - } - switch key { - case "eq": - f.Equality = term - case "match": - f.Match = term - case "ore": - f.Ore = term - case "ope": - f.Ope = term - default: - return nil, fmt.Errorf("%w: field %q has unknown output %q", ErrInternal, name, key) - } - } - rec[name] = f - } - return rec, nil -} - -// termBytes unwraps a term node: a passthrough carrying the term's bytes. -func termBytes(node any) ([]byte, error) { - plain, ok := node.(vcvalue.Plain) - if !ok { - return nil, fmt.Errorf("term node is %T, not a passthrough", node) - } - b, ok := plain.V.([]byte) - if !ok { - return nil, fmt.Errorf("term payload is %T, not bytes", plain.V) - } - return b, nil -} - -// DecryptRecords opens records produced by EncryptRecords under this keyset -// (a record from another keyset is ErrForeignKeyset) into out, a pointer to -// a slice of the same struct type, one element per record. Only the sealed -// outputs participate; terms are one-way. Fields the plan does not name are -// left as they are: when the slice already holds one row per record, each -// row keeps its other fields; otherwise it is replaced by a fresh slice. -// Nothing is written unless every record decodes. -func (cph *Cipher) DecryptRecords(ctx context.Context, records []EncryptedRecord, out any, opts ...RecordOption) error { - return cph.client.decryptRecords(ctx, cph.keyset, records, out, opts) -} - -// DecryptRecord opens one record into out, a pointer to a struct; see -// DecryptRecords. Fields the plan does not name keep their values, and -// nothing is written unless every planned field decodes. -func (cph *Cipher) DecryptRecord(ctx context.Context, record EncryptedRecord, out any, opts ...RecordOption) error { - return cph.client.decryptRecord(ctx, cph.keyset, record, out, opts) -} - -func (c *Client) decryptRecords(ctx context.Context, sel KeysetSelector, records []EncryptedRecord, out any, opts []RecordOption) error { - ptr := reflect.ValueOf(out) - if ptr.Kind() != reflect.Pointer || ptr.IsNil() || ptr.Elem().Kind() != reflect.Slice { - return fmt.Errorf("encrypt: DecryptRecords writes into a pointer to a slice of structs, not %T", out) - } - o := applyOptions(opts) - plan, err := planFor(ptr.Elem().Type().Elem(), o) - if err != nil { - return err - } - tree := make([]any, len(records)) - for i, rec := range records { - if tree[i], err = recordTree(rec, plan); err != nil { - return err - } - } - values, err := c.decryptRecordTree(ctx, sel, plan, tree, o) - if err != nil { - return err - } - items, ok := values.([]any) - if !ok || len(items) != len(records) { - return fmt.Errorf("%w: record batch decrypted as %T", ErrInternal, values) - } - return commitRecords(ptr.Elem(), items, plan) -} - -// commitRecords writes decrypted records into a slice value, atomically: -// the rows are assembled in a scratch slice — copies of the existing rows -// when there is one per record, zero rows otherwise — and stored only once -// every record has been assigned. -func commitRecords(slice reflect.Value, items []any, plan []fieldPlan) error { - scratch := reflect.MakeSlice(slice.Type(), len(items), len(items)) - if slice.Len() == len(items) { - reflect.Copy(scratch, slice) - } - for i, item := range items { - if err := assignRecord(scratch.Index(i), item, plan); err != nil { - return err - } - } - slice.Set(scratch) - return nil -} - -// commitRecord writes one decrypted record into a struct value, atomically: -// a copy takes the planned fields and replaces the original only once -// every one of them has been assigned. -func commitRecord(target reflect.Value, item any, plan []fieldPlan) error { - scratch := reflect.New(target.Type()).Elem() - scratch.Set(target) - if err := assignRecord(scratch, item, plan); err != nil { - return err - } - target.Set(scratch) - return nil -} - -func (c *Client) decryptRecord(ctx context.Context, sel KeysetSelector, record EncryptedRecord, out any, opts []RecordOption) error { - ptr := reflect.ValueOf(out) - if ptr.Kind() != reflect.Pointer || ptr.IsNil() || ptr.Elem().Kind() != reflect.Struct { - return fmt.Errorf("encrypt: DecryptRecord writes into a pointer to a struct, not %T", out) - } - o := applyOptions(opts) - plan, err := planFor(ptr.Elem().Type(), o) - if err != nil { - return err - } - tree, err := recordTree(record, plan) - if err != nil { - return err - } - value, err := c.decryptRecordTree(ctx, sel, plan, tree, o) - if err != nil { - return err - } - return commitRecord(ptr.Elem(), value, plan) -} - -// recordTree renders the ciphertext tree the guest opens: per planned -// field, its "c" output. Terms are not sent. -func recordTree(rec EncryptedRecord, plan []fieldPlan) (map[string]any, error) { - tree := make(map[string]any, len(plan)) - for _, f := range plan { - field, ok := rec[f.name] - if !ok || field.Ciphertext == nil { - return nil, fmt.Errorf("encrypt: record has no ciphertext for field %q", f.name) - } - tree[f.name] = map[string]any{"c": field.Ciphertext} - } - return tree, nil -} - -func (c *Client) decryptRecordTree(ctx context.Context, sel KeysetSelector, plan []fieldPlan, tree any, o recordOptions) (any, error) { - planObj, err := planValue(plan, o) - if err != nil { - return nil, err - } - encodedTree, err := marshalCipherText(tree) - if err != nil { - return nil, err - } - encodedPlan, err := vcffi.Marshal(planObj) - if err != nil { - return nil, err - } - opts, err := vcffi.Marshal(options(sel)) - if err != nil { - return nil, err - } - out, err := c.call(ctx, func(inst *instance) ([]byte, error) { - return inst.call(ctx, inst.decryptRecord, buf(encodedTree), buf(encodedPlan), buf(opts)) - }) - if err != nil { - return nil, err - } - defer wipe(out) - return vcffi.Unmarshal(out) -} - -// assignRecord writes a decrypted record (a vcvalue.Object of the plan's -// fields) into a struct value. -func assignRecord(target reflect.Value, value any, plan []fieldPlan) error { - obj, ok := value.(vcvalue.Object) - if !ok { - return fmt.Errorf("%w: record decrypted as %T", ErrInternal, value) - } - byName := make(map[string]any, len(obj)) - for _, f := range obj { - byName[f.Key] = f.Value - } - for _, f := range plan { - v, ok := byName[f.name] - if !ok { - return fmt.Errorf("%w: decrypted record lacks field %q", ErrInternal, f.name) - } - if err := assignField(target.Field(f.index), v); err != nil { - return fmt.Errorf("encrypt: field %q: %w", f.name, err) - } - } - return nil -} - -var errUnassignable = errors.New("cannot assign decrypted value") - -// assignField sets a struct field from a decoded value, converting within -// a numeric family when the value fits and refusing anything lossy. A -// decoded nil (a sealed none) is accepted only by a field that can hold -// one — a pointer, slice, map or interface — never as a zero scalar. -func assignField(field reflect.Value, v any) error { - if v == nil { - switch field.Kind() { - case reflect.Pointer, reflect.Slice, reflect.Map, reflect.Interface: - field.Set(reflect.Zero(field.Type())) - return nil - default: - return fmt.Errorf("%w: nil into %s", errUnassignable, field.Type()) - } - } - if field.Kind() == reflect.Pointer { - elem := reflect.New(field.Type().Elem()) - if err := assignField(elem.Elem(), v); err != nil { - return err - } - field.Set(elem) - return nil - } - rv := reflect.ValueOf(v) - if rv.Type().AssignableTo(field.Type()) { - field.Set(rv) - return nil - } - // A defined type over the same kind (type Flag bool, type Raw []byte) - // converts without loss. - if rv.Kind() == field.Kind() && rv.Type().ConvertibleTo(field.Type()) { - switch field.Kind() { - case reflect.Bool, reflect.String, reflect.Slice: - field.Set(rv.Convert(field.Type())) - return nil - } - } - switch field.Kind() { - case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: - var n int64 - switch x := v.(type) { - case int32: - n = int64(x) - case int64: - n = x - default: - return fmt.Errorf("%w: %T into %s", errUnassignable, v, field.Type()) - } - if field.OverflowInt(n) { - return fmt.Errorf("%w: %d overflows %s", errUnassignable, n, field.Type()) - } - field.SetInt(n) - case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: - var n uint64 - switch x := v.(type) { - case uint32: - n = uint64(x) - case uint64: - n = x - default: - return fmt.Errorf("%w: %T into %s", errUnassignable, v, field.Type()) - } - if field.OverflowUint(n) { - return fmt.Errorf("%w: %d overflows %s", errUnassignable, n, field.Type()) - } - field.SetUint(n) - case reflect.Float32, reflect.Float64: - var f float64 - switch x := v.(type) { - case float32: - f = float64(x) - case float64: - f = x - default: - return fmt.Errorf("%w: %T into %s", errUnassignable, v, field.Type()) - } - if field.OverflowFloat(f) { - return fmt.Errorf("%w: %v overflows %s", errUnassignable, f, field.Type()) - } - // Narrowing must be exact: a float64 that float32 cannot represent - // would silently round. NaN is its own case, never equal to itself. - if field.Kind() == reflect.Float32 && float64(float32(f)) != f && f == f { - return fmt.Errorf("%w: %v is not representable as %s", errUnassignable, f, field.Type()) - } - field.SetFloat(f) - case reflect.String: - s, ok := v.(string) - if !ok { - return fmt.Errorf("%w: %T into %s", errUnassignable, v, field.Type()) - } - field.SetString(s) - case reflect.Bool: - b, ok := v.(bool) - if !ok { - return fmt.Errorf("%w: %T into %s", errUnassignable, v, field.Type()) - } - field.SetBool(b) - default: - return fmt.Errorf("%w: %T into %s", errUnassignable, v, field.Type()) - } - return nil -} diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go new file mode 100644 index 000000000..cbda5144a --- /dev/null +++ b/languages/golang/encrypt/records.go @@ -0,0 +1,316 @@ +package encrypt + +import ( + "context" + "errors" + "fmt" + + "github.com/cipherstash/stack/languages/golang/internal/record" + "github.com/cipherstash/vitaminc/bindings/go/vcffi" + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +// The record path: what generated code calls through encrypt/gensupport. +// The three methods take and return the internal record types, so a program +// cannot call them with anything but what a generated declaration lowered +// to; the generated functions (users.Encrypt, users.Decrypt, users.Fields) +// are the API. Every call is one guest call, and one batched ZeroKMS request +// however many rows it carries (one request per 500 sealed leaves). + +// Decrypter opens records: a *Cipher, which refuses a record sealed under +// another keyset before any key is retrieved, or a *Client, which opens each +// record under the keyset that sealed it. Generated Decrypt functions take +// one. Its method is for generated code; a program does not call it. +type Decrypter interface { + Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) +} + +// Seal encrypts rows under the plan, through this cipher's keyset and with +// its extension: the sealed fields of every row in one request, in order. +// For generated code. +func (cph *Cipher) Seal(ctx context.Context, plan *record.Plan, rows []record.Source) ([]record.Sealed, error) { + p, err := cph.plan(plan) + if err != nil { + return nil, err + } + source := make([]any, len(rows)) + for i, row := range rows { + source[i], err = sourceRow(p, row) + if err != nil { + return nil, fmt.Errorf("%w: row %d: %v", ErrEncoding, i, err) + } + } + encodedSource, err := vcffi.Marshal(source) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + // The source is plaintext: its transport copy is wiped once it is in + // the guest, and the guest wipes its own copy before it returns. + defer wipe(encodedSource) + encodedPlan, err := vcffi.Marshal(p.Wire()) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + opts, err := vcffi.Marshal(options(cph.keyset)) + if err != nil { + return nil, err + } + out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.encryptRecord, buf(encodedSource), buf(encodedPlan), buf(opts)) + }) + if err != nil { + return nil, err + } + defer wipe(out) + tree, err := unmarshalCipherText(out) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrInternal, err) + } + items, ok := tree.([]any) + if !ok || len(items) != len(rows) { + return nil, fmt.Errorf("%w: %d rows came back as %T", ErrInternal, len(rows), tree) + } + sealed := make([]record.Sealed, len(items)) + for i, item := range items { + if sealed[i], err = sealedOf(p, item); err != nil { + return nil, fmt.Errorf("%w: row %d: %v", ErrInternal, i, err) + } + } + return sealed, nil +} + +// Open decrypts records sealed under the plan: every row in one request, +// in order. A record from another keyset is [ErrForeignKeyset]. For +// generated code. +func (cph *Cipher) Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) { + p, err := cph.plan(plan) + if err != nil { + return nil, err + } + return cph.client.open(ctx, cph.keyset, p, records) +} + +// Derive derives one index term for one value of a field, under the plan's +// context for that field and this cipher's extension: the term a query +// compares against the stored one. For generated code. +func (cph *Cipher) Derive(ctx context.Context, plan *record.Plan, field string, output record.Output, value any) ([]byte, error) { + p, err := cph.plan(plan) + if err != nil { + return nil, err + } + f := p.Field(field) + if f == nil { + return nil, fmt.Errorf("%w: the plan has no field %q", ErrEncoding, field) + } + code, ok := termKindCode(output) + if !ok { + return nil, fmt.Errorf("%w: field %q: no term %q", ErrEncoding, field, output) + } + declared := false + for _, o := range f.Outputs { + declared = declared || o == output + } + if !declared { + return nil, fmt.Errorf("%w: field %q declares no %q index", ErrEncoding, field, output) + } + encodedValue, err := vcffi.Marshal(value) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + // The probe value is plaintext: its transport copy is wiped once it is + // in the guest. + defer wipe(encodedValue) + encodedContext, err := vcffi.Marshal(p.FieldContext(*f)) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + opts, err := vcffi.Marshal(options(cph.keyset)) + if err != nil { + return nil, err + } + return cph.client.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.term, buf(encodedValue), buf(encodedContext), scalar(uint64(code)), buf(opts)) + }) +} + +// Open decrypts records sealed under the plan by any keyset of this client: +// each record is opened under the keyset that sealed it, with one request +// per keyset. For generated code. +func (c *Client) Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) { + if err := plan.Validate(); err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + return c.open(ctx, anyKeyset{}, plan, records) +} + +// plan is the declaration's plan with this cipher's extension, checked. +func (cph *Cipher) plan(plan *record.Plan) (*record.Plan, error) { + if cph.err != nil { + return nil, cph.err + } + if plan == nil { + return nil, fmt.Errorf("%w: no plan", ErrEncoding) + } + p := *plan + p.Extension = append(append([]any(nil), plan.Extension...), cph.extension...) + if err := p.Validate(); err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + return &p, nil +} + +// sourceRow renders one row for the guest: the plan's fields, in plan order, +// each with its value. Fail closed in both directions: a field with no value +// and a value with no field are both refused before anything is sent. +func sourceRow(p *record.Plan, row record.Source) (vcvalue.Object, error) { + if len(row) != len(p.Fields) { + for name := range row { + if p.Field(name) == nil { + return nil, fmt.Errorf("the plan has no field %q", name) + } + } + } + out := make(vcvalue.Object, 0, len(p.Fields)) + for _, f := range p.Fields { + v, ok := row[f.Name] + if !ok { + return nil, fmt.Errorf("no value for field %q", f.Name) + } + out = append(out, vcvalue.Field{Key: f.Name, Value: v}) + } + return out, nil +} + +// sealedOf lifts one decoded record node into the per-field outputs. +func sealedOf(p *record.Plan, node any) (record.Sealed, error) { + fields, ok := node.(map[string]any) + if !ok { + return nil, fmt.Errorf("record came back as %T", node) + } + sealed := make(record.Sealed, len(fields)) + for name, outputs := range fields { + if p.Field(name) == nil { + return nil, fmt.Errorf("the engine returned a field %q the plan does not name", name) + } + om, ok := outputs.(map[string]any) + if !ok { + return nil, fmt.Errorf("field %q came back as %T", name, outputs) + } + var o record.Outputs + for key, out := range om { + if key == string(record.Ciphertext) { + leaf, ok := out.(Ciphertext) + if !ok { + return nil, fmt.Errorf("field %q: the ciphertext is a %T, not one leaf", name, out) + } + o.Ciphertext = leaf + continue + } + term, err := termBytes(out) + if err != nil { + return nil, fmt.Errorf("field %q output %q: %v", name, key, err) + } + if o.Terms == nil { + o.Terms = map[record.Output][]byte{} + } + o.Terms[record.Output(key)] = term + } + sealed[name] = o + } + for _, f := range p.Fields { + if _, ok := sealed[f.Name]; !ok { + return nil, fmt.Errorf("the engine returned no outputs for field %q", f.Name) + } + } + return sealed, nil +} + +// termBytes unwraps a term node: a passthrough carrying the term's bytes. +func termBytes(node any) ([]byte, error) { + plain, ok := node.(vcvalue.Plain) + if !ok { + return nil, fmt.Errorf("term node is %T, not a passthrough", node) + } + b, ok := plain.V.([]byte) + if !ok { + return nil, fmt.Errorf("term payload is %T, not bytes", plain.V) + } + return b, nil +} + +// errNoCiphertext is a stored record missing a field the plan seals. +var errNoCiphertext = errors.New("encrypt: the record has no ciphertext for a sealed field") + +func (c *Client) open(ctx context.Context, sel KeysetSelector, p *record.Plan, records []record.Sealed) ([]record.Source, error) { + trees := make([]any, len(records)) + for i, rec := range records { + tree := make(map[string]any, len(p.Fields)) + for _, f := range p.Fields { + hasCiphertext := false + for _, o := range f.Outputs { + hasCiphertext = hasCiphertext || o == record.Ciphertext + } + if !hasCiphertext { + continue + } + outputs, ok := rec[f.Name] + if !ok || outputs.Ciphertext == nil { + return nil, fmt.Errorf("%w: row %d, field %q", errNoCiphertext, i, f.Name) + } + tree[f.Name] = map[string]any{string(record.Ciphertext): Ciphertext(outputs.Ciphertext)} + } + trees[i] = tree + } + encodedTree, err := marshalCipherText(trees) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + encodedPlan, err := vcffi.Marshal(p.Wire()) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + opts, err := vcffi.Marshal(options(sel)) + if err != nil { + return nil, err + } + out, err := c.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.decryptRecord, buf(encodedTree), buf(encodedPlan), buf(opts)) + }) + if err != nil { + return nil, err + } + // The output is plaintext: decode, then wipe the transport copy. + defer wipe(out) + decoded, err := vcffi.Unmarshal(out) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrInternal, err) + } + items, ok := decoded.([]any) + if !ok || len(items) != len(records) { + return nil, fmt.Errorf("%w: %d records came back as %T", ErrInternal, len(records), decoded) + } + sources := make([]record.Source, len(items)) + for i, item := range items { + obj, ok := item.(vcvalue.Object) + if !ok { + return nil, fmt.Errorf("%w: record %d decrypted as %T", ErrInternal, i, item) + } + src := make(record.Source, len(obj)) + for _, f := range obj { + src[f.Key] = f.Value + } + for _, f := range p.Fields { + if _, ok := src[f.Name]; !ok { + // An index-only field has no ciphertext to open and comes + // back as nothing; every sealed field must. + for _, o := range f.Outputs { + if o == record.Ciphertext { + return nil, fmt.Errorf("%w: record %d lacks field %q", ErrInternal, i, f.Name) + } + } + } + } + sources[i] = src + } + return sources, nil +} diff --git a/languages/golang/encrypt/roundtrip_test.go b/languages/golang/encrypt/roundtrip_test.go new file mode 100644 index 000000000..57ee85efb --- /dev/null +++ b/languages/golang/encrypt/roundtrip_test.go @@ -0,0 +1,258 @@ +package encrypt_test + +import ( + "bytes" + "context" + "errors" + "reflect" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" +) + +// Hermetic round trips through the real guest over the deterministic-kms +// test build: every key derives from a seed, so these need no ZeroKMS and +// run wherever `mise run wasm:guest:build:deterministic` has run. + +var testSeed = [32]byte([]byte("encrypt round-trip tests seed v1")) + +func deterministicClient(t *testing.T) *encrypt.Client { + t.Helper() + c, err := encrypt.NewDeterministicClient(context.Background(), testSeed) + if errors.Is(err, encrypt.ErrDeterministicGuestNotBuilt) || errors.Is(err, encrypt.ErrGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = c.Close() }) + return c +} + +var people = []testusers.User{ + {ID: 1, Age: 34, Email: "alice@example.com", Notes: "likes cats", Internal: "never stored"}, + {ID: 2, Age: 29, Email: "bob@example.com", Notes: "likes dogs"}, + {ID: 3, Age: 29, Email: "carol@example.com"}, +} + +func TestEncryptDecryptRoundTrip(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + + encrypted, err := testusers.Encrypt(ctx, cipher, people) + if err != nil { + t.Fatal(err) + } + if len(encrypted) != len(people) { + t.Fatalf("%d rows for %d values", len(encrypted), len(people)) + } + for i, e := range encrypted { + if e.ID != people[i].ID { + t.Errorf("row %d: passthrough id %d, want %d", i, e.ID, people[i].ID) + } + if len(e.Age.Ciphertext) == 0 || len(e.Age.Equality) != 32 || len(e.Age.Ore) == 0 || len(e.Email.Ciphertext) == 0 || len(e.Email.Match) == 0 || len(e.Notes.Ciphertext) == 0 { + t.Errorf("row %d: outputs missing: %+v", i, e) + } + } + // Equal plaintexts give equal terms, under one keyset and context; + // different ones do not. + if !encrypted[1].Age.Equality.Equal(encrypted[2].Age.Equality) || encrypted[0].Age.Equality.Equal(encrypted[1].Age.Equality) { + t.Error("equality terms do not follow the plaintext") + } + if !encrypted[1].Age.Ore.Less(encrypted[0].Age.Ore) || encrypted[1].Age.Ore.Compare(encrypted[2].Age.Ore) != 0 { + t.Error("ORE terms do not order as the plaintexts") + } + // Ciphertexts are fresh each time. + if bytes.Equal(encrypted[1].Age.Ciphertext, encrypted[2].Age.Ciphertext) { + t.Error("two rows share a ciphertext") + } + + want := append([]testusers.User(nil), people...) + want[0].Internal = "" + for _, d := range []struct { + name string + by encrypt.Decrypter + }{{"cipher", cipher}, {"client", c}} { + back, err := testusers.Decrypt(ctx, d.by, encrypted) + if err != nil { + t.Fatalf("Decrypt through the %s: %v", d.name, err) + } + if !reflect.DeepEqual(back, want) { + t.Fatalf("Decrypt through the %s = %+v, want %+v", d.name, back, want) + } + } + + // One field: the Fields entry seals and probes it. + one, err := testusers.Fields.Email.Encrypt(ctx, cipher, "bob@example.com") + if err != nil { + t.Fatal(err) + } + if !one.Equality.Equal(encrypted[1].Email.Equality) || !bytes.Equal(one.Match, encrypted[1].Email.Match) { + t.Error("a field's own Encrypt derives other terms than the record's") + } + probe, err := testusers.Fields.Email.Equality(ctx, cipher, "bob@example.com") + if err != nil { + t.Fatal(err) + } + if !probe.Equal(encrypted[1].Email.Equality) || probe.Equal(encrypted[0].Email.Equality) { + t.Error("the probe does not single out its row") + } + match, err := testusers.Fields.Email.Match(ctx, cipher, "bob@example.com") + if err != nil || !bytes.Equal(match, encrypted[1].Email.Match) { + t.Errorf("match probe: %v", err) + } + // A column update: one field's ciphertext opens in a row. + encrypted[0].Email = testusers.EncryptedUserEmail(one) + back, err := testusers.Decrypt(ctx, cipher, encrypted[:1]) + if err != nil || back[0].Email != "bob@example.com" { + t.Fatalf("after a column update: %v %+v", err, back) + } +} + +func TestEncryptedTypesHideSealedFields(t *testing.T) { + e := testusers.EncryptedUser{ID: 7} + e.Email.Ciphertext = []byte("secret bytes") + s := e.String() + if !strings.Contains(s, "ID: 7") || strings.Contains(s, "secret") || !strings.Contains(s, "Email: [sealed]") { + t.Fatalf("String = %q", s) + } +} + +func TestExtendIsPartOfTheIdentity(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + tenant7, tenant8 := cipher.Extend(uint64(7)), cipher.Extend(uint64(8)) + + plain, err := testusers.Encrypt(ctx, cipher, people[:2]) + if err != nil { + t.Fatal(err) + } + ext, err := testusers.Encrypt(ctx, tenant7, people[:2]) + if err != nil { + t.Fatal(err) + } + other, err := testusers.Encrypt(ctx, tenant8, people[:2]) + if err != nil { + t.Fatal(err) + } + if ext[1].Email.Equality.Equal(plain[1].Email.Equality) || ext[1].Email.Equality.Equal(other[1].Email.Equality) { + t.Error("terms under different extensions are equal") + } + if _, err := testusers.Decrypt(ctx, cipher, ext); err == nil { + t.Fatal("an extended record opened without its extension") + } + if _, err := testusers.Decrypt(ctx, tenant8, ext); err == nil { + t.Fatal("an extended record opened under another extension") + } + back, err := testusers.Decrypt(ctx, tenant7, ext) + if err != nil || back[1].Email != "bob@example.com" { + t.Fatalf("with its extension: %v %+v", err, back) + } + // The probe takes the same cipher. + probe, err := testusers.Fields.Email.Equality(ctx, tenant7, "bob@example.com") + if err != nil || !probe.Equal(ext[1].Email.Equality) || probe.Equal(plain[1].Email.Equality) { + t.Fatalf("tenant probe: %v", err) + } + // Extend(a, b) is Extend(a).Extend(b); a byte part is copied. + part := []byte("eu") + ab, err := testusers.Encrypt(ctx, cipher.Extend(uint64(7), part), people[:1]) + if err != nil { + t.Fatal(err) + } + part[0] = 'X' + ba, err := testusers.Encrypt(ctx, tenant7.Extend([]byte("eu")), people[:1]) + if err != nil { + t.Fatal(err) + } + if !ab[0].Email.Equality.Equal(ba[0].Email.Equality) { + t.Error("Extend(a, b) and Extend(a).Extend(b) derive different terms") + } + // A part the codec cannot carry fails the first call, as a value. + if _, err := testusers.Encrypt(ctx, cipher.Extend(1.5), people[:1]); !errors.Is(err, encrypt.ErrEncoding) { + t.Fatalf("Extend(1.5): %v, want ErrEncoding", err) + } +} + +func TestOpaqueStructSealsAsOneValue(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + docs := []testusers.Document{ + {Title: "Handbook", Body: "Welcome aboard.", Tags: []string{"hr", "onboarding"}}, + {Title: "Empty"}, + } + encrypted, err := testusers.EncryptDocument(ctx, cipher, docs) + if err != nil { + t.Fatal(err) + } + if len(encrypted) != 2 || len(encrypted[0].Sealed) == 0 { + t.Fatalf("encrypted = %+v", encrypted) + } + back, err := testusers.DecryptDocument(ctx, c, encrypted) + if err != nil { + t.Fatal(err) + } + if back[0].Title != "Handbook" || back[0].Body != "Welcome aboard." || !reflect.DeepEqual(back[0].Tags, []string{"hr", "onboarding"}) || back[1].Title != "Empty" { + t.Fatalf("DecryptDocument = %+v", back) + } + // A document is one column: its String hides it. + if s := encrypted[0].String(); strings.Contains(s, "Handbook") || !strings.Contains(s, "[sealed]") { + t.Fatalf("String = %q", s) + } +} + +func TestForeignKeysetIsRefusedBeforeAnyKey(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + other := c.Keyset(encrypt.KeysetName("tenant-b")) + encrypted, err := testusers.Encrypt(ctx, other, people[:1]) + if err != nil { + t.Fatal(err) + } + if _, err := testusers.Decrypt(ctx, c.DefaultKeyset(), encrypted); !errors.Is(err, encrypt.ErrForeignKeyset) { + t.Fatalf("the default cipher opened another keyset's row: %v", err) + } + id, err := other.KeysetID(ctx) + if err != nil { + t.Fatal(err) + } + for name, d := range map[string]encrypt.Decrypter{"client": c, "the keyset by id": c.Keyset(id), "the keyset by name": other} { + back, err := testusers.Decrypt(ctx, d, encrypted) + if err != nil || back[0].Email != "alice@example.com" { + t.Fatalf("%s: %v %+v", name, err, back) + } + } +} + +func TestATamperedRecordDoesNotOpen(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + encrypted, err := testusers.Encrypt(ctx, cipher, people[:1]) + if err != nil { + t.Fatal(err) + } + // A ciphertext moved under another field's label does not open: the key + // is bound to the descriptor. + swapped := encrypted + swapped[0].Notes.Ciphertext = encrypted[0].Email.Ciphertext + if _, err := testusers.Decrypt(ctx, cipher, swapped); err == nil { + t.Fatal("a leaf opened under another field's context") + } + // A flipped byte does not open. + flipped, _ := testusers.Encrypt(ctx, cipher, people[:1]) + flipped[0].Email.Ciphertext[len(flipped[0].Email.Ciphertext)-1] ^= 1 + if _, err := testusers.Decrypt(ctx, cipher, flipped); err == nil { + t.Fatal("a tampered leaf opened") + } + // A record with no ciphertext for a sealed field is refused on the host. + missing, _ := testusers.Encrypt(ctx, cipher, people[:1]) + missing[0].Age.Ciphertext = nil + if _, err := testusers.Decrypt(ctx, cipher, missing); err == nil || !strings.Contains(err.Error(), `"age"`) { + t.Fatalf("a missing ciphertext: %v", err) + } +} diff --git a/languages/golang/encrypt/term.go b/languages/golang/encrypt/term.go index d9dc40ee3..4d54115e6 100644 --- a/languages/golang/encrypt/term.go +++ b/languages/golang/encrypt/term.go @@ -6,53 +6,89 @@ import ( "database/sql/driver" "encoding/binary" "fmt" + + "github.com/cipherstash/stack/languages/golang/internal/record" ) -// TermKind selects which index term a probe or a plan field derives. The -// values are the guest's term-kind codes. -type TermKind uint32 +// Index is one index on a field, as a generated declaration names it: +// [Equality], [Ore], [Ope], [Match] or [JSON]. The words are the Rust API's +// words for the same behaviour, and a declaration in any language spells +// them the same way. +type Index interface { + // Output is the index's wire key. + Output() record.Output + // String is the index's tag word. + String() string +} + +type index record.Output + +func (i index) Output() record.Output { return record.Output(i) } + +func (i index) String() string { + switch record.Output(i) { + case record.Equality: + return "equality" + case record.Match: + return "match" + case record.Ore: + return "ore" + case record.Ope: + return "ope" + } + return string(i) +} -const ( +// The indexes that take no options. +var ( // Equality is a PRF equality term: 32 bytes, compared with // [EqualityTerm.Equal]. Defined for integers, strings and bytes. - Equality TermKind = 1 - // Match is a full-text match term: the tokenized positions of a string, - // as little-endian uint16s. Strings only. - Match TermKind = 2 + Equality Index = index(record.Equality) // Ore is an order-revealing (CLLW ORE) term over any scalar. - Ore TermKind = 3 + Ore Index = index(record.Ore) // Ope is an order-preserving (CLLW OPE) term over any scalar. - Ope TermKind = 4 + Ope Index = index(record.Ope) ) -// termKindNames is the one table of plan-tag spellings: TermKind.String, -// parseTermKind and TermKind.valid all read it. -var termKindNames = map[TermKind]string{ - Equality: "eq", - Match: "match", - Ore: "ore", - Ope: "ope", +// MatchOption is an option of the match index. None is defined yet: the +// engine's data plan carries the match index under its default options, and +// a non-default option has no wire form (stack-encrypt plan builder, +// Additions item 7). +type MatchOption interface { + matchOption() } -func (k TermKind) String() string { - if name, ok := termKindNames[k]; ok { - return name - } - return fmt.Sprintf("TermKind(%d)", uint32(k)) +// Match is a full-text match index: the tokenized positions of a string. +// Strings only. +func Match(options ...MatchOption) Index { + _ = options // none exist; the type is declared so a declaration reads as the tag does + return index(record.Match) } -// valid reports whether k is a kind this package defines. -func (k TermKind) valid() bool { - _, ok := termKindNames[k] - return ok +// JSONOption is an option of the json index. +type JSONOption interface { + jsonOption() } -// parseTermKind maps a plan-tag spelling to its kind. -func parseTermKind(s string) (TermKind, bool) { - for k, name := range termKindNames { - if name == s { - return k, true - } +// JSON is the index over a JSON document. The engine does not derive it +// yet: a declaration that names it is refused by stashgen, and at run time +// by the engine. +func JSON(options ...JSONOption) Index { + _ = options + return index("json") +} + +// termKindCode is the guest's se_term code for an index. +func termKindCode(o record.Output) (uint32, bool) { + switch o { + case record.Equality: + return 1, true + case record.Match: + return 2, true + case record.Ore: + return 3, true + case record.Ope: + return 4, true } return 0, false } @@ -112,6 +148,11 @@ func (t OpeTerm) Compare(other OpeTerm) int { return bytes.Compare(t, other) } // Less reports whether t's plaintext orders before other's. func (t OpeTerm) Less(other OpeTerm) bool { return t.Compare(other) < 0 } +// JSONTerm is the term of the json index. The engine does not derive it +// yet; the type exists so a generated declaration that names [JSON] has a +// Go type to fail into. +type JSONTerm []byte + // compareCLLW is cllw-ore's compare_lex: compare_slice over the common // prefix, then by length. For equal-length terms (integers) that is // compare_slice alone. @@ -169,6 +210,9 @@ func (t OreTerm) Value() (driver.Value, error) { return []byte(t), nil } // Value implements driver.Valuer. func (t OpeTerm) Value() (driver.Value, error) { return []byte(t), nil } +// Value implements driver.Valuer. +func (t JSONTerm) Value() (driver.Value, error) { return []byte(t), nil } + // Scan implements sql.Scanner. func (t *EqualityTerm) Scan(src any) error { b, err := scanBytes("EqualityTerm", src) @@ -196,3 +240,10 @@ func (t *OpeTerm) Scan(src any) error { *t = b return err } + +// Scan implements sql.Scanner. +func (t *JSONTerm) Scan(src any) error { + b, err := scanBytes("JSONTerm", src) + *t = b + return err +} diff --git a/languages/golang/encrypt/unit_test.go b/languages/golang/encrypt/unit_test.go index 50d0c9ecc..bc33f2918 100644 --- a/languages/golang/encrypt/unit_test.go +++ b/languages/golang/encrypt/unit_test.go @@ -3,7 +3,6 @@ package encrypt import ( "bufio" "bytes" - "context" "encoding/hex" "errors" "io" @@ -21,76 +20,6 @@ import ( // Pure Go: no guest needed. -func TestCommitRecordsPreservesRowsAndIsAtomic(t *testing.T) { - type row struct { - ID int64 `stash:"-"` - Age uint8 `stash:"label=users/age"` - Email string `stash:"label=users/email"` - } - plan, err := planFor(reflect.TypeOf(row{}), recordOptions{}) - if err != nil { - t.Fatal(err) - } - decoded := func(age any, email string) vcvalue.Object { - return vcvalue.Object{{Key: "Age", Value: age}, {Key: "Email", Value: email}} - } - - // One row per record: unplanned fields survive. - rows := []row{{ID: 1, Age: 9}, {ID: 2, Age: 9}} - if err := commitRecords(reflect.ValueOf(&rows).Elem(), []any{decoded(uint32(30), "a"), decoded(uint32(40), "b")}, plan); err != nil { - t.Fatal(err) - } - if want := []row{{1, 30, "a"}, {2, 40, "b"}}; !reflect.DeepEqual(rows, want) { - t.Fatalf("rows = %+v, want %+v", rows, want) - } - - // A failing record leaves the slice untouched. - before := append([]row(nil), rows...) - err = commitRecords(reflect.ValueOf(&rows).Elem(), []any{decoded(uint32(31), "c"), decoded(uint32(300), "d")}, plan) - if !errors.Is(err, errUnassignable) { - t.Fatalf("overflowing batch: %v", err) - } - if !reflect.DeepEqual(rows, before) { - t.Fatalf("partial write: %+v", rows) - } - - // A different length replaces the slice. - if err := commitRecords(reflect.ValueOf(&rows).Elem(), []any{decoded(uint32(1), "z")}, plan); err != nil { - t.Fatal(err) - } - if want := []row{{0, 1, "z"}}; !reflect.DeepEqual(rows, want) { - t.Fatalf("rows = %+v, want %+v", rows, want) - } - - // One record: same contract on a struct. - one := row{ID: 7, Age: 1, Email: "keep"} - if err := commitRecord(reflect.ValueOf(&one).Elem(), decoded(uint32(300), "new"), plan); !errors.Is(err, errUnassignable) { - t.Fatalf("overflowing record: %v", err) - } - if one != (row{7, 1, "keep"}) { - t.Fatalf("partial write: %+v", one) - } - if err := commitRecord(reflect.ValueOf(&one).Elem(), decoded(uint32(2), "new"), plan); err != nil { - t.Fatal(err) - } - if one != (row{7, 2, "new"}) { - t.Fatalf("record = %+v", one) - } -} - -func TestEncryptRecordRejectsNil(t *testing.T) { - c := &Client{closed: true} - cph := c.DefaultKeyset() - for name, in := range map[string]any{"nil": nil, "nil pointer": (*taggedUser)(nil)} { - if _, err := cph.EncryptRecord(context.Background(), in); err == nil || errors.Is(err, ErrState) { - t.Errorf("EncryptRecord(%s): %v, want a record error", name, err) - } - if _, err := cph.EncryptRecords(context.Background(), in); err == nil || errors.Is(err, ErrState) { - t.Errorf("EncryptRecords(%s): %v, want a record error", name, err) - } - } -} - func TestKeysetIDRoundTripsCanonicalForm(t *testing.T) { const s = "6a70bd18-99ac-4650-b104-37eec3a15b09" id, err := ParseKeysetID(s) @@ -128,526 +57,6 @@ func TestSelectorsSpellEveryVariant(t *testing.T) { } } -// A Context owns its parts, as an option does: NewContext and With copy a -// byte-slice part in, so a caller's buffer reused once the context is -// built does not change it. -func TestContextOwnsItsByteParts(t *testing.T) { - root, ext := []byte("users/email"), []byte("eu") - c, err := NewContext(root) - if err != nil { - t.Fatal(err) - } - if c, err = c.With(ext); err != nil { - t.Fatal(err) - } - copy(root, "users/phone") - copy(ext, "us") - parts, ok := c.value().([]any) - if !ok || len(parts) != 2 { - t.Fatalf("context value is %#v, want a two-part list", c.value()) - } - if got := string(parts[0].([]byte)); got != "users/email" { - t.Errorf("root part is %q after the caller's buffer changed, want \"users/email\"", got) - } - if got := string(parts[1].([]byte)); got != "eu" { - t.Errorf("extension part is %q after the caller's buffer changed, want \"eu\"", got) - } -} - -func TestContextNestsToTheLeft(t *testing.T) { - c := MustContext("users/age") - if got := c.value(); got != "users/age" { - t.Fatalf("bare part = %v", got) - } - c, err := c.With(uint64(7)) - if err != nil { - t.Fatal(err) - } - c, err = c.With("eu") - if err != nil { - t.Fatal(err) - } - want := []any{[]any{"users/age", uint64(7)}, "eu"} - if got := c.value(); !reflect.DeepEqual(got, want) { - t.Fatalf("With chain = %v, want %v", got, want) - } - for _, bad := range []any{1.5, true, nil, []any{"x"}, map[string]any{}} { - if _, err := NewContext(bad); err == nil { - t.Errorf("NewContext(%T) accepted", bad) - } - } - if _, err := (Context{}).With("x"); err == nil { - t.Error("an empty context extended") - } -} - -type taggedUser struct { - ID int64 `stash:"-"` - Age uint32 `stash:"label=users/age,index=eq;ore"` - Email string `stash:"label=users/email,index=eq;match,name=email"` - Notes string `stash:"label=users/notes"` - Plain string `stash:"plain"` - NoTag string - hidden string `stash:"context=x"` //nolint:unused // proves unexported fields are skipped -} - -func TestPlanFromTags(t *testing.T) { - plan, err := planFor(reflect.TypeOf(taggedUser{}), recordOptions{}) - if err != nil { - t.Fatal(err) - } - want := []fieldPlan{ - {index: 1, name: "Age", context: label(t, "users/age").Context(), outputs: []string{"c", "eq", "ore"}}, - {index: 2, name: "email", context: label(t, "users/email").Context(), outputs: []string{"c", "eq", "match"}}, - {index: 3, name: "Notes", context: label(t, "users/notes").Context(), outputs: []string{"c"}}, - } - if !reflect.DeepEqual(plan, want) { - t.Fatalf("plan = %+v\nwant %+v", plan, want) - } - - obj, err := planValue(plan, recordOptions{extension: []any{uint64(7)}}) - if err != nil { - t.Fatal(err) - } - age := obj[0].Value.(vcvalue.Object) - if got := age[0].Value; !reflect.DeepEqual(got, []any{[]any{"users", "age"}, uint64(7)}) { - t.Fatalf("extended context = %v", got) - } - if _, err := vcffi.Marshal(obj); err != nil { - t.Fatalf("plan does not marshal: %v", err) - } - - for name, bad := range map[string]any{ - "no context": struct { - A int `stash:"index=eq"` - }{}, - "unknown kind": struct { - A int `stash:"label=t/c,index=fuzzy"` - }{}, - "unknown option": struct { - A int `stash:"label=t/c,store=true"` - }{}, - "empty context": struct { - A int `stash:"context="` - }{}, - "nothing tagged": struct{ A int }{}, - "not a struct": 42, - "nil type": nil, - "term twice": struct { - A int `stash:"label=t/c,index=eq;eq"` - }{}, - "duplicate name": struct { - A int `stash:"label=t/c,name=x"` - B int `stash:"label=t/d,name=x"` - }{}, - "one-part context": struct { - A int `stash:"context=c"` - }{}, - } { - if _, err := PlanFromTags(reflect.TypeOf(bad)); err == nil { - t.Errorf("%s: plan accepted", name) - } - } -} - -// An explicit plan is the tag plan by another route: the same fields give -// the guest the same bytes, and WithPlan's zero value is the tag path. -func TestExplicitPlanIsTheTagPlan(t *testing.T) { - typ := reflect.TypeOf(taggedUser{}) - explicit, err := NewPlan( - FieldPlan{Field: "Age", Context: label(t, "users/age").Context(), Terms: []TermKind{Equality, Ore}}, - FieldPlan{Field: "Email", Name: "email", Context: label(t, "users/email").Context(), Terms: []TermKind{Equality, Match}}, - FieldPlan{Field: "Notes", Context: label(t, "users/notes").Context()}, - ) - if err != nil { - t.Fatal(err) - } - tagged, err := PlanFromTags(typ) - if err != nil { - t.Fatal(err) - } - if !reflect.DeepEqual(explicit.Fields(), tagged.Fields()) { - t.Fatalf("fields differ:\n%+v\n%+v", explicit.Fields(), tagged.Fields()) - } - encode := func(p Plan) []byte { - bound, err := p.bind(typ) - if err != nil { - t.Fatal(err) - } - obj, err := planValue(bound, recordOptions{extension: []any{uint64(7)}}) - if err != nil { - t.Fatal(err) - } - b, err := vcffi.Marshal(obj) - if err != nil { - t.Fatal(err) - } - return b - } - if a, b := encode(explicit), encode(tagged); !bytes.Equal(a, b) { - t.Fatalf("guest input differs:\n%x\n%x", a, b) - } - viaOption, err := planFor(typ, applyOptions([]RecordOption{WithPlan(explicit)})) - if err != nil { - t.Fatal(err) - } - viaTags, err := planFor(typ, applyOptions([]RecordOption{WithPlan(Plan{})})) - if err != nil { - t.Fatal(err) - } - if !reflect.DeepEqual(viaOption, viaTags) { - t.Fatalf("bound plans differ:\n%+v\n%+v", viaOption, viaTags) - } - // Fields returns a copy. - explicit.Fields()[0].Context = MustContext("changed") - if !explicit.Fields()[0].Context.Equal(label(t, "users/age").Context()) { - t.Fatal("Fields exposed the plan's own slice") - } -} - -// A probe's context under ExtendContext is, byte for byte, the context the -// record plan sends for a field with the same own context under the same -// extension — the one place the probe and the stored term could silently -// disagree. And it differs from the unextended context and from another -// extension's, which is what makes the match tenant-specific. -func TestTermExtensionMatchesRecordFieldContext(t *testing.T) { - type row struct { - Email string `stash:"label=users/email,index=eq"` - } - ext := []any{uint64(7), "eu"} - o := applyOptions([]RecordOption{ExtendContext(ext...)}) - bound, err := planFor(reflect.TypeOf(row{}), o) - if err != nil { - t.Fatal(err) - } - obj, err := planValue(bound, o) - if err != nil { - t.Fatal(err) - } - spec, ok := obj[0].Value.(vcvalue.Object) - if !ok || spec[0].Key != "context" { - t.Fatalf("plan field encodes as %+v", obj[0].Value) - } - fieldContext := spec[0].Value - - var to termOptions - ExtendContext(ext...).applyTerm(&to) - probe, err := extend(label(t, "users/email").Context(), to.extension) - if err != nil { - t.Fatal(err) - } - if !reflect.DeepEqual(probe.value(), fieldContext) { - t.Fatalf("probe context %#v, record field context %#v", probe.value(), fieldContext) - } - if reflect.DeepEqual(label(t, "users/email").Context().value(), fieldContext) { - t.Fatal("the unextended probe context equals the extended field's") - } - other, err := extend(label(t, "users/email").Context(), []any{uint64(8), "eu"}) - if err != nil { - t.Fatal(err) - } - if reflect.DeepEqual(other.value(), fieldContext) { - t.Fatal("another tenant's probe context equals the field's") - } - // The same option value, held once and passed to both calls, is how - // the two sides are kept in step; it applies identically through - // either interface. An option that means something only on a record - // call is not an Option, so Cipher.Term cannot accept it and ignore it. - var opt RecordOption = ExtendContext(ext...) - if _, ok := opt.(Option); !ok { - t.Fatal("ExtendContext is not an Option through its RecordOption interface") - } - if _, ok := WithPlan(Plan{}).(Option); ok { - t.Fatal("WithPlan is an Option; Cipher.Term must not accept it") - } -} - -// Several ExtendContext options on one call join in order, and the record -// calls and the probe join them by the same rule: two options a and b are -// the context ExtendContext(a, b) gives, on both sides. A rule that let a -// later option replace an earlier one on one side only would put records -// and probes under different contexts with no error. -func TestSeveralExtensionsJoinInOrder(t *testing.T) { - type row struct { - Email string `stash:"label=users/email,index=eq"` - } - typ := reflect.TypeOf(row{}) - fieldContext := func(opts ...RecordOption) any { - t.Helper() - o := applyOptions(opts) - bound, err := planFor(typ, o) - if err != nil { - t.Fatal(err) - } - obj, err := planValue(bound, o) - if err != nil { - t.Fatal(err) - } - return obj[0].Value.(vcvalue.Object)[0].Value - } - probeContext := func(opts ...Option) any { - t.Helper() - var to termOptions - for _, opt := range opts { - opt.applyTerm(&to) - } - c, err := extend(label(t, "users/email").Context(), to.extension) - if err != nil { - t.Fatal(err) - } - return c.value() - } - - tenant, region := ExtendContext(uint64(7)), ExtendContext("eu") - want := fieldContext(ExtendContext(uint64(7), "eu")) - if got := fieldContext(tenant, region); !reflect.DeepEqual(got, want) { - t.Errorf("record: two options give %#v, one option with both parts %#v", got, want) - } - if got := probeContext(tenant, region); !reflect.DeepEqual(got, want) { - t.Errorf("probe: two options give %#v, the record's one-option context %#v", got, want) - } - if got := probeContext(ExtendContext(uint64(7), "eu")); !reflect.DeepEqual(got, want) { - t.Errorf("probe: one option gives %#v, the record's %#v", got, want) - } - // Order is part of the context: the same parts the other way round are - // another context, on both sides. - if got := fieldContext(region, tenant); reflect.DeepEqual(got, want) { - t.Error("record: options in the other order give the same context") - } - if got := probeContext(region, tenant); reflect.DeepEqual(got, want) { - t.Error("probe: options in the other order give the same context") - } - // Joining is not deduplication: the same extension given twice extends - // twice, which is why a call must receive it once. - if got := fieldContext(tenant, tenant); reflect.DeepEqual(got, fieldContext(tenant)) { - t.Error("record: the same extension given twice is the single-extension context") - } -} - -// An option owns its parts. A byte-slice part is copied when the option -// is built, so a caller's buffer reused between the write and the probe -// does not move the context the saved option extends by, on either side. -func TestExtendContextOwnsItsByteParts(t *testing.T) { - region := []byte("eu") - opt := ExtendContext(uint64(7), region) - first := applyOptions([]RecordOption{opt}) - var firstProbe termOptions - opt.applyTerm(&firstProbe) - - copy(region, "us") - - second := applyOptions([]RecordOption{opt}) - var secondProbe termOptions - opt.applyTerm(&secondProbe) - for name, ext := range map[string][]any{ - "record, before": first.extension, "record, after": second.extension, - "probe, before": firstProbe.extension, "probe, after": secondProbe.extension, - } { - if got := string(ext[1].([]byte)); got != "eu" { - t.Errorf("%s: byte part is %q after the caller's buffer changed, want \"eu\"", name, got) - } - } - if !reflect.DeepEqual(first.extension, second.extension) || !reflect.DeepEqual(firstProbe.extension, secondProbe.extension) { - t.Error("the same option applied twice gave different extensions") - } -} - -// A plan can name only exported, direct fields of the struct it binds to, -// and only fields that exist; an untagged struct binds fine under it. -func TestPlanBindsByFieldName(t *testing.T) { - type embedded struct{ Inner string } - type untagged struct { - embedded - Email string - hidden string //nolint:unused // proves unexported fields are refused - } - typ := reflect.TypeOf(untagged{}) - if _, err := PlanFromTags(typ); err == nil { - t.Fatal("untagged struct has a tag plan") - } - ok, err := NewPlan(FieldPlan{Field: "Email", Context: label(t, "t/c").Context()}) - if err != nil { - t.Fatal(err) - } - bound, err := planFor(typ, applyOptions([]RecordOption{WithPlan(ok)})) - if err != nil { - t.Fatal(err) - } - if len(bound) != 1 || bound[0].index != 1 || bound[0].name != "Email" { - t.Fatalf("bound = %+v", bound) - } - for name, field := range map[string]string{ - "missing": "Nope", - "unexported": "hidden", - "promoted": "Inner", - } { - p, err := NewPlan(FieldPlan{Field: field, Context: label(t, "t/c").Context()}) - if err != nil { - t.Fatal(err) - } - if _, err := p.bind(typ); err == nil || !strings.Contains(err.Error(), field) { - t.Errorf("%s: bind error = %v, want one naming %q", name, err, field) - } - } - if _, err := ok.bind(reflect.TypeOf(42)); err == nil { - t.Error("bound to a non-struct") - } -} - -func TestNewPlanRefusesMalformedFields(t *testing.T) { - c := label(t, "t/c").Context() - d := label(t, "t/d").Context() - for name, fields := range map[string][]FieldPlan{ - "no fields": nil, - "no field name": {{Context: c}}, - "no context": {{Field: "A"}}, - "unknown kind": {{Field: "A", Context: c, Terms: []TermKind{TermKind(9)}}}, - "duplicate name": {{Field: "A", Context: c, Name: "x"}, {Field: "B", Context: d, Name: "x"}}, - "field twice": {{Field: "A", Name: "x", Context: c}, {Field: "A", Name: "y", Context: d}}, - "term twice": {{Field: "A", Context: c, Terms: []TermKind{Equality, Equality}}}, - } { - if _, err := NewPlan(fields...); err == nil { - t.Errorf("%s: plan accepted", name) - } - } - if (Plan{}).Fields() != nil { - t.Error("zero plan has fields") - } -} - -// A planned field's context is a label of at least two plain segments and -// nothing else: the one shape the guest lowers a plan field under. Every -// other Context the type can spell is refused when the plan is built, with -// the field named and the accepted form in the message, rather than by the -// guest at the first record call. -func TestNewPlanRefusesAContextThatIsNotAFieldLabel(t *testing.T) { - extended, err := label(t, "users/age").Context().With(uint64(7)) - if err != nil { - t.Fatal(err) - } - barePair, err := MustContext("c").With(uint64(7)) - if err != nil { - t.Fatal(err) - } - listPart, err := label(t, "users/age").Context().With("eu") - if err != nil { - t.Fatal(err) - } - for name, tc := range map[string]struct { - context Context - says string - }{ - "one text part": {MustContext("c"), "one part, not a label"}, - "one-segment label": {label(t, "users").Context(), "one part, not a label"}, - "an integer": {MustContext(uint64(7)), "one part, not a label"}, - "bytes": {MustContext([]byte{1, 2}), "one part, not a label"}, - "a label extended": {extended, "is extended"}, - "a bare part extended by an integer": {barePair, "is extended"}, - "a label extended by text": {listPart, "is extended"}, - } { - _, err := NewPlan(FieldPlan{Field: "Age", Context: tc.context}) - if err == nil { - t.Errorf("%s: NewPlan accepted it", name) - continue - } - if !strings.Contains(err.Error(), "plan field Age") || !strings.Contains(err.Error(), tc.says) { - t.Errorf("%s: err = %v; want the field named and %q", name, err, tc.says) - } - } - // A flat list of plain text parts is the label it spells, whichever - // constructor built it: the bytes are the label's. - pair, err := MustContext("users").With("age") - if err != nil { - t.Fatal(err) - } - if _, err := NewPlan(FieldPlan{Field: "Age", Context: pair}); err != nil { - t.Errorf("NewContext(\"users\").With(\"age\") refused: %v", err) - } - // A segment that is not plain is refused for the same reason a Label is. - if _, err := NewPlan(FieldPlan{Field: "Age", Context: Context{node: []any{"users", "7up"}}}); err == nil || !strings.Contains(err.Error(), "not a plain label") { - t.Errorf("a non-plain segment: err = %v", err) - } - // Validate on an explicit plan is NewPlan's check: the plan never built. - if _, err := NewPlan(FieldPlan{Field: "Age", Context: MustContext("c")}); err == nil { - t.Error("a one-part context built a plan Validate could be asked about") - } -} - -func TestAssignFieldConvertsWithinFamiliesOnly(t *testing.T) { - type flag bool - type name string - type raw []byte - type row struct { - I int - U8 uint8 - F float32 - F64 float64 - S string - B []byte - P *int64 - Bool bool - Flag flag - Name name - Raw raw - M map[string]int - Any any - } - var r row - rv := reflect.ValueOf(&r).Elem() - must := func(field string, v any) { - t.Helper() - if err := assignField(rv.FieldByName(field), v); err != nil { - t.Fatalf("%s <- %T: %v", field, v, err) - } - } - must("I", int64(-5)) - must("U8", uint32(200)) - must("F", float32(1.5)) - must("F", float64(0.5)) // exactly representable: narrows - must("F64", float32(0.1)) - must("S", "s") - must("B", []byte{1}) - must("P", int64(9)) - must("Bool", true) - must("Flag", true) - must("Name", "n") - must("Raw", []byte{2}) - if r.I != -5 || r.U8 != 200 || r.F != 0.5 || r.F64 != float64(float32(0.1)) || r.S != "s" || string(r.B) != "\x01" || *r.P != 9 || - !r.Bool || !bool(r.Flag) || r.Name != "n" || string(r.Raw) != "\x02" { - t.Fatalf("assigned %+v", r) - } - // A sealed none decodes as nil: only a field that can hold one takes it. - r.P, r.B, r.M, r.Any = new(int64), []byte{1}, map[string]int{"a": 1}, 1 - must("P", nil) - must("B", nil) - must("M", nil) - must("Any", nil) - if r.P != nil || r.B != nil || r.M != nil || r.Any != nil { - t.Fatalf("nil did not clear: %+v", r) - } - for _, bad := range []struct { - field string - v any - }{ - {"U8", uint32(300)}, // overflow - {"I", uint64(1)}, // family - {"S", int64(1)}, // kind - {"Bool", "true"}, // kind - {"I", float64(1)}, // family - {"F", float64(0.1)}, // not representable as float32 - {"F", float64(16777217)}, // in range, not representable - {"I", nil}, // a none into a scalar - {"S", nil}, // a none into a scalar - {"Bool", nil}, // a none into a scalar - {"Name", []byte("n")}, // kind - {"Raw", "r"}, // kind - } { - if err := assignField(rv.FieldByName(bad.field), bad.v); !errors.Is(err, errUnassignable) { - t.Errorf("%s <- %v: got %v, want errUnassignable", bad.field, bad.v, err) - } - } -} - // The request body wipes its buffer when closed, not before: reads up to // Close see the bytes, Close zeroes them, a read after Close is an error // rather than zeros, and a second Close is harmless. @@ -717,9 +126,9 @@ func TestTermsAndLeavesScanAndValue(t *testing.T) { if _, err := (MatchTerm{1}).Positions(); err == nil { t.Fatal("odd match term accepted") } - var s Sealed + var s Ciphertext if err := s.Scan(nil); err == nil { - t.Fatal("NULL scanned into Sealed") + t.Fatal("NULL scanned into Ciphertext") } if err := s.Scan("ab"); err != nil || string(s) != "ab" { t.Fatalf("string scan: %v %q", err, s) @@ -740,10 +149,10 @@ func TestTermsAndLeavesScanAndValue(t *testing.T) { func TestLeafSetKeepsStackEncryptLeavesDistinct(t *testing.T) { ct := map[string]any{ - "a": Sealed{1}, - "b": SealedNone{2}, - "c": SealedEmptySeq{3}, - "d": SealedEmptyMap{4}, + "a": Ciphertext{1}, + "b": otherLeaf{kind: vcffi.LeafNone, bytes: []byte{2}}, + "c": otherLeaf{kind: vcffi.LeafEmptySeq, bytes: []byte{3}}, + "d": otherLeaf{kind: vcffi.LeafEmptyMap, bytes: []byte{4}}, "p": vcvalue.Plain{V: "clear"}, } encoded, err := marshalCipherText(ct) @@ -761,8 +170,8 @@ func TestLeafSetKeepsStackEncryptLeavesDistinct(t *testing.T) { if _, err := marshalCipherText(map[string]any{"a": vcvalue.Sealed{1}}); err == nil { t.Fatal("vcvalue.Sealed accepted as a stack-encrypt leaf") } - if _, err := vcffi.MarshalCipherText(vcffi.VCValueLeaves(), Sealed{1}); err == nil { - t.Fatal("encrypt.Sealed accepted as a vitaminc leaf") + if _, err := vcffi.MarshalCipherText(vcffi.VCValueLeaves(), Ciphertext{1}); err == nil { + t.Fatal("encrypt.Ciphertext accepted as a vitaminc leaf") } } diff --git a/languages/golang/encrypt/wasm/README.md b/languages/golang/encrypt/wasm/README.md index 9cd0785f8..f6afc39d6 100644 --- a/languages/golang/encrypt/wasm/README.md +++ b/languages/golang/encrypt/wasm/README.md @@ -1,6 +1,13 @@ -# Guest module +# Guest modules `stack_encrypt_guest.wasm` is a build artefact of the Rust crate in `../guest`, copied here by `mise run wasm:guest:build`. It is not committed; the Go package embeds this directory and reports `ErrGuestNotBuilt` from `NewClient` when the module is absent, and its tests skip. + +`stack_encrypt_guest_deterministic.wasm` is the same crate built with the +`deterministic-kms` feature by `mise run wasm:guest:build:deterministic`: a +TEST build whose keys derive from a seed, so the tests open the Rust record +fixture and run round trips with no ZeroKMS. The package never embeds it as +the guest a program runs; only the tests load it, and they skip when it is +absent. diff --git a/languages/golang/internal/factstest/factstest.go b/languages/golang/internal/factstest/factstest.go deleted file mode 100644 index d00406cb7..000000000 --- a/languages/golang/internal/factstest/factstest.go +++ /dev/null @@ -1,166 +0,0 @@ -// Package factstest is a test-only fact source for the plan package: Go -// struct fields annotated with a `facts` tag. The tag syntax is not API; -// real facts come from a schema, such as the protobuf source planned in -// CIP-4088. Internal, and imported only by _test files. -package factstest - -import ( - "errors" - "fmt" - "reflect" - "slices" - "strings" - "unicode" - - "github.com/cipherstash/stack/languages/golang/encrypt/plan" -) - -// StructTags is a Go-struct fact source for tests: one fact per exported, -// direct field of a struct (msg is a struct value or a pointer to one), -// with the annotations its `facts` tag lists: -// -// type Individual struct { -// ID int64 -// Email string `facts:"fides.data_categories=user.contact.email"` -// MedicareNo string `facts:"fides.data_categories=user.government_id,user.financial"` -// } -// -// The tag is `key=value[,value...]`, repeated with `;` for more keys. -// GoField is the Go field name and Field is its snake_case ("MedicareNo" -// is "medicare_no", "ID" is "id", "HTTPPort" is "http_port"): the name a -// proto field or a database column would have, so the column identity a -// field binds by default is the one the Rust derive and the schema spell. -// Number is 0 and Kind is the field's reflect.Kind (through one pointer). -// -// Unexported and embedded fields are not facts: a plan binds exported, -// direct fields only. A `facts` tag on one — or on any field of an -// embedded struct — is an error, not a field quietly left in plaintext. -var StructTags plan.Source = plan.SourceFunc(structFacts) - -func structFacts(msg any) ([]plan.Fact, error) { - t := reflect.TypeOf(msg) - if t != nil && t.Kind() == reflect.Pointer { - t = t.Elem() - } - if t == nil || t.Kind() != reflect.Struct { - return nil, fmt.Errorf("factstest: StructTags reads structs, not %T", msg) - } - facts := make([]plan.Fact, 0, t.NumField()) - for i := 0; i < t.NumField(); i++ { - sf := t.Field(i) - if !sf.IsExported() || sf.Anonymous { - if err := refuseUnbindableTag(t, sf); err != nil { - return nil, fmt.Errorf("factstest: %s.%s: %w", t, sf.Name, err) - } - continue - } - kind := sf.Type - if kind.Kind() == reflect.Pointer { - kind = kind.Elem() - } - annotations, err := parseFactsTag(sf.Tag.Get("facts")) - if err != nil { - return nil, fmt.Errorf("factstest: %s.%s: %w", t, sf.Name, err) - } - facts = append(facts, plan.Fact{ - Message: t.String(), - Field: snakeCase(sf.Name), - GoField: sf.Name, - Kind: kind.Kind().String(), - Annotations: annotations, - }) - } - return facts, nil -} - -// refuseUnbindableTag is the error for a `facts` tag on a field sf of -// outer that a plan cannot bind: an unexported or embedded field, or any -// field of an embedded struct, however deep. The tag says the field is -// classified; dropping it would store the field in plaintext with no rule -// ever asked. A struct that embeds itself (`type Node struct { *Node; ... -// }`) is not searched again: its fields are outer's own, already read. -func refuseUnbindableTag(outer reflect.Type, sf reflect.StructField) error { - if sf.Tag.Get("facts") != "" { - if sf.Anonymous { - return errors.New("a facts tag on an embedded field, which a plan cannot bind") - } - return errors.New("a facts tag on an unexported field, which a plan cannot bind") - } - if !sf.Anonymous { - return nil - } - if tagged := firstFactsTag(sf.Type, []reflect.Type{outer}); tagged != "" { - return fmt.Errorf("embedded %s has a facts tag on %s, which a plan cannot bind; make it a direct field", sf.Type, tagged) - } - return nil -} - -// firstFactsTag names the first field of t (a struct, through one -// pointer), or of a struct embedded in it, that carries a facts tag; "" -// when none does. seen is the structs already on the path down: an -// embedding can be recursive (`type Node struct { *Node; ... }`), and a -// struct already being searched has nothing new to find. -func firstFactsTag(t reflect.Type, seen []reflect.Type) string { - if t.Kind() == reflect.Pointer { - t = t.Elem() - } - if t.Kind() != reflect.Struct || slices.Contains(seen, t) { - return "" - } - seen = append(seen, t) - for i := 0; i < t.NumField(); i++ { - sf := t.Field(i) - if sf.Tag.Get("facts") != "" { - return sf.Name - } - if sf.Anonymous { - if name := firstFactsTag(sf.Type, seen); name != "" { - return sf.Name + "." + name - } - } - } - return "" -} - -// snakeCase is a Go field name as a schema would spell it: a lower-case -// word per hump, joined by underscores, with an initialism kept as one -// word ("HTTPPort" is "http_port", "ID" is "id"). Digits stay with the -// word before them ("Line2" is "line2"). -func snakeCase(name string) string { - runes := []rune(name) - var b strings.Builder - b.Grow(len(name) + 4) - for i, r := range runes { - if i > 0 && unicode.IsUpper(r) { - prev := runes[i-1] - nextLower := i+1 < len(runes) && unicode.IsLower(runes[i+1]) - if unicode.IsLower(prev) || unicode.IsDigit(prev) || (unicode.IsUpper(prev) && nextLower) { - b.WriteByte('_') - } - } - b.WriteRune(unicode.ToLower(r)) - } - return b.String() -} - -func parseFactsTag(tag string) ([]plan.Annotation, error) { - if tag == "" { - return nil, nil - } - var out []plan.Annotation - for _, part := range strings.Split(tag, ";") { - key, values, ok := strings.Cut(part, "=") - if !ok || key == "" || values == "" { - return nil, fmt.Errorf("facts tag %q: want key=value[,value...]", part) - } - if slices.ContainsFunc(out, func(a plan.Annotation) bool { return a.Key == key }) { - return nil, fmt.Errorf("facts tag: key %q given twice", key) - } - vs := strings.Split(values, ",") - if slices.Contains(vs, "") { - return nil, errors.New("facts tag: empty value for " + key) - } - out = append(out, plan.Annotation{Key: key, Values: vs}) - } - return out, nil -} diff --git a/languages/golang/internal/factstest/factstest_test.go b/languages/golang/internal/factstest/factstest_test.go deleted file mode 100644 index 1c1d9017c..000000000 --- a/languages/golang/internal/factstest/factstest_test.go +++ /dev/null @@ -1,118 +0,0 @@ -package factstest_test - -import ( - "reflect" - "strings" - "testing" - - "github.com/cipherstash/stack/languages/golang/encrypt/plan" - "github.com/cipherstash/stack/languages/golang/internal/factstest" -) - -// Recursive embeddings, legal in Go, which a scan of embedded structs must -// not follow forever. -type ( - list struct { - *list //nolint:unused // the recursion is the point - Value string - } - node struct { - *node //nolint:unused // the recursion is the point - Secret string `facts:"a=x"` - } -) - -func TestStructTags(t *testing.T) { - type embedded struct{ Inner string } - type row struct { - embedded - ID *int64 - Email string `facts:"a=x,y;b=z"` - hidden string //nolint:unused // proves untagged unexported fields are skipped - } - facts, err := factstest.StructTags.Facts(&row{}) - if err != nil { - t.Fatal(err) - } - want := []plan.Fact{ - {Message: "factstest_test.row", Field: "id", GoField: "ID", Kind: "int64"}, - {Message: "factstest_test.row", Field: "email", GoField: "Email", Kind: "string", Annotations: []plan.Annotation{ - {Key: "a", Values: []string{"x", "y"}}, {Key: "b", Values: []string{"z"}}, - }}, - } - if !reflect.DeepEqual(facts, want) { - t.Fatalf("facts =\n%+v\nwant\n%+v", facts, want) - } - if got := facts[1].String(); got != "factstest_test.row.email (Email) [a=x,y; b=z]" { - t.Errorf("String = %s", got) - } - type taggedInner struct { - Secret string `facts:"a=x"` - } - type deeper struct{ taggedInner } - for name, bad := range map[string]struct { - msg any - say string - }{ - "not a struct": {42, "reads structs"}, - "nil": {nil, "reads structs"}, - "no value": {struct { - A string `facts:"a="` - }{}, "key=value"}, - "no key": {struct { - A string `facts:"=x"` - }{}, "key=value"}, - "empty value": {struct { - A string `facts:"a=x,"` - }{}, "empty value"}, - "key twice": {struct { - A string `facts:"a=x;a=y"` - }{}, "given twice"}, - // A tag the plan cannot bind is refused, never quietly plaintext. - "tagged unexported field": {struct { - medicareNo string `facts:"a=x"` //nolint:unused // the tag is the point - }{}, "unexported field"}, - "tagged embedded field": {struct { - embedded `facts:"a=x"` - }{}, "embedded field"}, - "tag inside an embedded struct": {struct{ taggedInner }{}, "Secret"}, - "tag two embeddings deep": {struct{ deeper }{}, "taggedInner.Secret"}, - "tag inside an embedded pointer": {struct{ *taggedInner }{}, "Secret"}, - } { - _, err := factstest.StructTags.Facts(bad.msg) - if err == nil { - t.Errorf("%s: facts read", name) - } else if !strings.Contains(err.Error(), bad.say) { - t.Errorf("%s: err = %q, want it to say %q", name, err, bad.say) - } - } - // A recursive embedding terminates, and is not a fact: the embedded - // copy's fields are the struct's own, tagged or not. - if facts, err := factstest.StructTags.Facts(list{}); err != nil || len(facts) != 1 || facts[0].GoField != "Value" { - t.Errorf("recursive embedding: facts %+v, %v; want Value alone", facts, err) - } - if facts, err := factstest.StructTags.Facts(node{}); err != nil || len(facts) != 1 || facts[0].GoField != "Secret" || len(facts[0].Annotations) != 1 { - t.Errorf("recursive embedding beside a tag: facts %+v, %v; want Secret alone, classified", facts, err) - } - // The schema spelling of a Go field name. - type spelled struct { - ID int64 - Email string - HTTPPort int - MedicareNo string - Line2 string - UserID string - OAuth2Key string - } - got, err := factstest.StructTags.Facts(spelled{}) - if err != nil { - t.Fatal(err) - } - names := make([]string, len(got)) - for i, f := range got { - names[i] = f.Field - } - if want := []string{"id", "email", "http_port", "medicare_no", "line2", "user_id", "o_auth2_key"}; !reflect.DeepEqual(names, want) { - t.Errorf("schema names = %v, want %v", names, want) - } -} diff --git a/languages/golang/internal/record/fixture_test.go b/languages/golang/internal/record/fixture_test.go new file mode 100644 index 000000000..5ccd3a3d7 --- /dev/null +++ b/languages/golang/internal/record/fixture_test.go @@ -0,0 +1,39 @@ +package record + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" +) + +// The segment rule is one rule in two languages. Rust's +// plain_text_and_label_segments_are_one_rule reads the same fixture, so a +// change to either implementation that the other does not follow fails here +// or there. +func TestSegmentRuleMatchesTheSharedFixture(t *testing.T) { + raw, err := os.ReadFile(filepath.Join("..", "..", "..", "..", "packages", "stack-encrypt", "tests", "fixtures", "label_segments.json")) + if err != nil { + t.Fatal(err) + } + var fixture struct { + Plain []string `json:"plain"` + NotPlain []string `json:"not_plain"` + } + if err := json.Unmarshal(raw, &fixture); err != nil { + t.Fatal(err) + } + if len(fixture.Plain) == 0 || len(fixture.NotPlain) == 0 { + t.Fatalf("fixture is empty: %+v", fixture) + } + for _, ok := range fixture.Plain { + if err := CheckSegment(ok); err != nil { + t.Errorf("CheckSegment(%q) = %v, want ok", ok, err) + } + } + for _, bad := range fixture.NotPlain { + if err := CheckSegment(bad); err == nil { + t.Errorf("CheckSegment(%q) succeeded; want a refusal", bad) + } + } +} diff --git a/languages/golang/internal/record/record.go b/languages/golang/internal/record/record.go new file mode 100644 index 000000000..9dacb251a --- /dev/null +++ b/languages/golang/internal/record/record.go @@ -0,0 +1,316 @@ +// Package record is the data form of a declaration as the engine reads it: +// the plan object the guest's se_encrypt_record, se_decrypt_record and +// se_plan_check parse, and the per-field outputs they return. It is shared by +// the encrypt package, which runs plans, by encrypt/gensupport, which lowers +// a generated declaration into one, and by stashgen, which asks the engine +// to check one. +// +// The spellings here are wire format, fixed by stack-encrypt's +// dynamic::record: the output keys "c", "eq", "match", "ore", "ope" and +// "passthrough", and the type names "bool", "int32", "int64", "uint32", +// "uint64", "float32", "float64", "string" and "bytes". A field's context is +// its label — the plan's context segments and the field's identity — nested +// to the left under each part the caller extends it by. +// +// Passthrough fields do not appear in a Plan. The Go SDK keeps them on the +// host: the FFI codec cannot carry every Go type a program stores beside a +// ciphertext (a time.Time, a driver.Valuer), and nothing the engine does to +// a passthrough value could be observed. +package record + +import ( + "errors" + "fmt" + "strings" + "unicode" + + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +// Kind is a plan field's declared type: the data form of the Rust chain's +// `::`. A field typed uint32 or string lowers to a real u32 or String +// leaf, the same bytes a Rust record derives; every other kind, and Untyped, +// seals in vitaminc's self-describing tagged encoding as one leaf. +type Kind string + +// The kinds. The names are vitaminc's ValueKind names, frozen. +const ( + Untyped Kind = "" + Bool Kind = "bool" + Int32 Kind = "int32" + Int64 Kind = "int64" + UInt32 Kind = "uint32" + UInt64 Kind = "uint64" + Float32 Kind = "float32" + Float64 Kind = "float64" + String Kind = "string" + Bytes Kind = "bytes" +) + +// Output is one output of a sealed field, by its wire key. +type Output string + +// The outputs. Passthrough is not one a Plan asks for; see the package doc. +const ( + Ciphertext Output = "c" + Equality Output = "eq" + Match Output = "match" + Ore Output = "ore" + Ope Output = "ope" +) + +// IsTerm reports whether the output is an index term. +func (o Output) IsTerm() bool { return o != Ciphertext && o != "" } + +// Field is one sealed field of a plan. +type Field struct { + // Name is the field's name in the declaration: the key of its value in a + // source and of its outputs in a result. + Name string + // Identity is the last segment of the field's label, under the plan's + // context. It is the name unless a policy pinned another. + Identity string + // Kind is the declared type, or Untyped. + Kind Kind + // Outputs are the field's outputs: Ciphertext and/or terms, at least one. + Outputs []Output +} + +// Plan is a declaration as the engine reads it: one context, any extension, +// and the sealed fields. +type Plan struct { + // Context is the label every field's label extends: at least one plain + // segment. The guest requires a field's label to have two or more + // segments, which the identity supplies. + Context []string + // Extension is the caller's parts, nested to the left of every field's + // label, in order: a tenant, a region. + Extension []any + // Fields are the sealed fields, in declared order. + Fields []Field +} + +// ParseContext splits a `context=` value on '/' and checks each segment is +// plain: what a label is, so the ZeroKMS descriptor renders it verbatim. +func ParseContext(s string) ([]string, error) { + if s == "" { + return nil, errors.New("the context is empty") + } + segments := strings.Split(s, "/") + for i, seg := range segments { + if err := CheckSegment(seg); err != nil { + return nil, fmt.Errorf("context %q: segment %d %v", s, i, err) + } + } + return segments, nil +} + +// CheckSegment is the one rule for a plain label segment, the same as +// stack-encrypt's Label::check_segment: non-empty, no control or invisible +// format character, none of '/', '(' and ')', and not beginning like another +// descriptor form ("b64:", a digit or '-'). The shared fixture +// packages/stack-encrypt/tests/fixtures/label_segments.json holds the two +// implementations together. +func CheckSegment(s string) error { + if s == "" { + return errors.New("is empty") + } + if strings.HasPrefix(s, "b64:") || s[0] == '-' || (s[0] >= '0' && s[0] <= '9') { + return errors.New("begins like another descriptor form (b64:, a digit or -)") + } + for _, r := range s { + if r == '/' { + return errors.New("contains '/', the separator") + } + if unicode.IsControl(r) || r == '(' || r == ')' { + return fmt.Errorf("contains %q, which the descriptor reserves", r) + } + if strings.ContainsRune(invisible, r) { + return fmt.Errorf("contains %q, an invisible format character", r) + } + } + return nil +} + +// invisible is the format characters with no glyph of their own: the soft +// hyphen, the Arabic letter mark, the Mongolian vowel separator, the +// zero-width characters, the bidirectional embeddings, overrides and +// isolates, and the byte-order mark. unicode.IsControl covers only Cc; +// these are Cf. A name containing one prints like another name in the +// ZeroKMS log, so they are refused beside the control characters. The same +// list as Rust's Label::INVISIBLE. +const invisible = "\u00ad\u061c\u180e\u200b\u200c\u200d\u200e\u200f\u202a\u202b\u202c\u202d\u202e\u2060\u2061\u2062\u2063\u2064\u2066\u2067\u2068\u2069\ufeff" + +// CheckPart refuses a context extension part that is not a string, a byte +// slice or an integer the codec carries. +func CheckPart(part any) error { + switch part.(type) { + case string, []byte, int32, int64, uint32, uint64, int: + return nil + default: + return fmt.Errorf("%T is not a context part (string, []byte or integer)", part) + } +} + +// Validate checks what the host can check before the engine sees the plan: +// a context, plain segments, at least one field, names and identities once, +// outputs once and at least one, known kinds, and extension parts the codec +// carries. The engine's own rules — which index a kind admits, what the +// builder refuses — are the engine's, asked through se_plan_check. +func (p *Plan) Validate() error { + if len(p.Context) == 0 { + return errors.New("record: the plan has no context") + } + for i, seg := range p.Context { + if err := CheckSegment(seg); err != nil { + return fmt.Errorf("record: context segment %d %v", i, err) + } + } + for _, part := range p.Extension { + if err := CheckPart(part); err != nil { + return fmt.Errorf("record: context extension: %v", err) + } + } + if len(p.Fields) == 0 { + return errors.New("record: the plan has no sealed field") + } + names, identities := map[string]bool{}, map[string]bool{} + for _, f := range p.Fields { + if f.Name == "" { + return errors.New("record: a field has no name") + } + if names[f.Name] { + return fmt.Errorf("record: field %q is declared twice", f.Name) + } + names[f.Name] = true + identity := f.identity() + if err := CheckSegment(identity); err != nil { + return fmt.Errorf("record: field %q: identity %q %v", f.Name, identity, err) + } + if identities[identity] { + return fmt.Errorf("record: field %q: identity %q is used twice", f.Name, identity) + } + identities[identity] = true + if !f.Kind.known() { + return fmt.Errorf("record: field %q: unknown kind %q", f.Name, f.Kind) + } + if len(f.Outputs) == 0 { + return fmt.Errorf("record: field %q has no output", f.Name) + } + seen := map[Output]bool{} + for _, o := range f.Outputs { + switch o { + case Ciphertext, Equality, Match, Ore, Ope: + default: + return fmt.Errorf("record: field %q: unknown output %q", f.Name, o) + } + if seen[o] { + return fmt.Errorf("record: field %q: output %q is asked for twice", f.Name, o) + } + seen[o] = true + } + } + return nil +} + +func (k Kind) known() bool { + switch k { + case Untyped, Bool, Int32, Int64, UInt32, UInt64, Float32, Float64, String, Bytes: + return true + } + return false +} + +func (f Field) identity() string { + if f.Identity != "" { + return f.Identity + } + return f.Name +} + +// Field returns the field named, or nil. +func (p *Plan) Field(name string) *Field { + for i := range p.Fields { + if p.Fields[i].Name == name { + return &p.Fields[i] + } + } + return nil +} + +// Wire renders the plan as the guest parses it: per field, its context +// (the label, extended), its outputs and its type. Validate first. +func (p *Plan) Wire() vcvalue.Object { + out := make(vcvalue.Object, 0, len(p.Fields)) + for _, f := range p.Fields { + outputs := make([]any, len(f.Outputs)) + for i, o := range f.Outputs { + outputs[i] = string(o) + } + spec := vcvalue.Object{ + {Key: "context", Value: p.FieldContext(f)}, + {Key: "outputs", Value: outputs}, + } + if f.Kind != Untyped { + spec = append(spec, vcvalue.Field{Key: "type", Value: string(f.Kind)}) + } + out = append(out, vcvalue.Field{Key: f.Name, Value: spec}) + } + return out +} + +// FieldContext is one field's context as the guest reads it: the label as a +// flat list of its segments, then each extension part nested to the left — +// [[["users", "age"], 7], "eu"] — exactly the shape se_term takes for a +// probe of that field. +func (p *Plan) FieldContext(f Field) any { + label := make([]any, 0, len(p.Context)+1) + for _, s := range p.Context { + label = append(label, s) + } + label = append(label, f.identity()) + var node any = label + for _, part := range p.Extension { + node = []any{node, ownPart(part)} + } + return node +} + +// ownPart copies a byte-slice part so a caller's buffer reused later does +// not change a context already built. +func ownPart(part any) any { + if b, ok := part.([]byte); ok { + out := make([]byte, len(b)) + copy(out, b) + return out + } + return part +} + +// Descriptor renders a field's label as ZeroKMS logs it, for messages. +func (p *Plan) Descriptor(f Field) string { + return strings.Join(append(append([]string{}, p.Context...), f.identity()), "/") +} + +// Source is one record's plaintext for the engine: each sealed field's value +// by name. Passthrough and omitted fields are not in it. +type Source = map[string]any + +// Outputs is what the engine produced for one sealed field. +type Outputs struct { + // Ciphertext is the frozen leaf bytes, or nil for an index-only field. + Ciphertext []byte + // Terms are the index terms by output, each its frozen bytes. + Terms map[Output][]byte +} + +// Sealed is one record as stored: each sealed field's outputs by name. +type Sealed = map[string]Outputs + +// Target is one EQL type the engine produces, as se_targets lists it. +type Target struct { + Name string + Kind Kind + Terms []Output + Query string +} diff --git a/languages/golang/internal/record/record_test.go b/languages/golang/internal/record/record_test.go new file mode 100644 index 000000000..8b00cd51d --- /dev/null +++ b/languages/golang/internal/record/record_test.go @@ -0,0 +1,115 @@ +package record + +import ( + "reflect" + "strings" + "testing" + + "github.com/cipherstash/vitaminc/bindings/go/vcffi" + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +func users() *Plan { + return &Plan{Context: []string{"users"}, Fields: []Field{ + {Name: "age", Kind: UInt32, Outputs: []Output{Ciphertext, Equality, Ore}}, + {Name: "email", Kind: String, Outputs: []Output{Ciphertext, Equality, Match}}, + {Name: "notes", Kind: String, Outputs: []Output{Ciphertext}}, + }} +} + +func TestWireIsTheFixturesPlan(t *testing.T) { + p := users() + if err := p.Validate(); err != nil { + t.Fatal(err) + } + wire := p.Wire() + want := vcvalue.Object{ + {Key: "age", Value: vcvalue.Object{ + {Key: "context", Value: []any{"users", "age"}}, + {Key: "outputs", Value: []any{"c", "eq", "ore"}}, + {Key: "type", Value: "uint32"}, + }}, + {Key: "email", Value: vcvalue.Object{ + {Key: "context", Value: []any{"users", "email"}}, + {Key: "outputs", Value: []any{"c", "eq", "match"}}, + {Key: "type", Value: "string"}, + }}, + {Key: "notes", Value: vcvalue.Object{ + {Key: "context", Value: []any{"users", "notes"}}, + {Key: "outputs", Value: []any{"c"}}, + {Key: "type", Value: "string"}, + }}, + } + if !reflect.DeepEqual(wire, want) { + t.Fatalf("Wire = %#v", wire) + } + if _, err := vcffi.Marshal(wire); err != nil { + t.Fatal(err) + } +} + +func TestExtensionNestsToTheLeftAndIdentityReplacesTheName(t *testing.T) { + p := &Plan{Context: []string{"documents", "v2"}, Extension: []any{uint64(7), "eu"}, Fields: []Field{{Name: "medicare_number", Identity: "medicare_no", Outputs: []Output{Ciphertext}}}} + if err := p.Validate(); err != nil { + t.Fatal(err) + } + got := p.FieldContext(p.Fields[0]) + want := []any{[]any{[]any{"documents", "v2", "medicare_no"}, uint64(7)}, "eu"} + if !reflect.DeepEqual(got, want) { + t.Fatalf("FieldContext = %#v", got) + } + if p.Descriptor(p.Fields[0]) != "documents/v2/medicare_no" { + t.Fatalf("Descriptor = %q", p.Descriptor(p.Fields[0])) + } + // An untyped field carries no "type" key. + if spec := p.Wire()[0].Value.(vcvalue.Object); len(spec) != 2 { + t.Fatalf("untyped field spec = %#v", spec) + } + // A byte part is copied. + buf := []byte("eu") + q := &Plan{Context: []string{"a"}, Extension: []any{buf}, Fields: []Field{{Name: "f", Outputs: []Output{Ciphertext}}}} + ctx := q.FieldContext(q.Fields[0]) + buf[0] = 'X' + if string(ctx.([]any)[1].([]byte)) != "eu" { + t.Fatal("the extension aliases the caller's buffer") + } +} + +func TestValidateRefusals(t *testing.T) { + one := func(f Field) *Plan { return &Plan{Context: []string{"users"}, Fields: []Field{f}} } + cases := map[string]*Plan{ + "no context": {Fields: []Field{{Name: "a", Outputs: []Output{Ciphertext}}}}, + "bad segment": {Context: []string{"users/x"}, Fields: []Field{{Name: "a", Outputs: []Output{Ciphertext}}}}, + "no field": {Context: []string{"users"}}, + "no name": one(Field{Outputs: []Output{Ciphertext}}), + "name twice": {Context: []string{"users"}, Fields: []Field{{Name: "a", Outputs: []Output{Ciphertext}}, {Name: "a", Outputs: []Output{Ciphertext}}}}, + "identity twice": {Context: []string{"users"}, Fields: []Field{{Name: "a", Identity: "x", Outputs: []Output{Ciphertext}}, {Name: "b", Identity: "x", Outputs: []Output{Ciphertext}}}}, + "identity not plain": one(Field{Name: "a", Identity: "1x", Outputs: []Output{Ciphertext}}), + "unknown kind": one(Field{Name: "a", Kind: "integer", Outputs: []Output{Ciphertext}}), + "no output": one(Field{Name: "a"}), + "unknown output": one(Field{Name: "a", Outputs: []Output{"json"}}), + "output twice": one(Field{Name: "a", Outputs: []Output{Equality, Equality}}), + "bad extension part": {Context: []string{"users"}, Extension: []any{1.5}, Fields: []Field{{Name: "a", Outputs: []Output{Ciphertext}}}}, + "name is not a label": one(Field{Name: "b64:x", Outputs: []Output{Ciphertext}}), + } + for name, p := range cases { + if err := p.Validate(); err == nil { + t.Errorf("%s: accepted", name) + } + } +} + +func TestParseContext(t *testing.T) { + got, err := ParseContext("documents/v2/body") + if err != nil || !reflect.DeepEqual(got, []string{"documents", "v2", "body"}) { + t.Fatalf("ParseContext = %v, %v", got, err) + } + for _, bad := range []string{"", "users/", "/users", "users//email", "b64:x", "1users", "-x", "a(b)", "a\u200bb"} { + if _, err := ParseContext(bad); err == nil { + t.Errorf("ParseContext(%q) accepted", bad) + } + } + if err := CheckSegment("x/y"); err == nil || !strings.Contains(err.Error(), "separator") { + t.Fatalf("CheckSegment: %v", err) + } +} diff --git a/languages/golang/stashgen/declaration.go b/languages/golang/stashgen/declaration.go index 8bca7c2ea..2d990a25e 100644 --- a/languages/golang/stashgen/declaration.go +++ b/languages/golang/stashgen/declaration.go @@ -168,6 +168,9 @@ type GoType struct { Name string // Kind is the type's underlying kind. Kind Kind + // Basic is the underlying basic type's name for a scalar kind — "uint8", + // "int", "string" — which decides the wire type the field seals as. + Basic string // Elem is the element type of a slice, map or pointer, and nil otherwise. Elem *GoType // Fields are the fields of a struct kind, in declared order. diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index 5b778a451..dc8b899d2 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -174,6 +174,39 @@ func (w *writer) shape(f *genFile) { w.p("}") } +// kindExpr is the gensupport kind of a Go type: the wire type the field +// seals as. The mapping is vcffi's: int8, int16 and int32 travel as int32, +// int and int64 as int64, the unsigned ones likewise, []byte as bytes. A +// composite is untyped and seals as one self-describing value. +func kindExpr(t GoType) string { + switch t.Kind { + case KindString: + return "gensupport.String" + case KindBool: + return "gensupport.Bool" + case KindBytes: + return "gensupport.Bytes" + case KindInt: + switch t.Basic { + case "int8", "int16", "int32": + return "gensupport.Int32" + } + return "gensupport.Int64" + case KindUint: + switch t.Basic { + case "uint8", "uint16", "uint32", "byte": + return "gensupport.UInt32" + } + return "gensupport.UInt64" + case KindFloat: + if t.Basic == "float32" { + return "gensupport.Float32" + } + return "gensupport.Float64" + } + return "gensupport.Untyped" +} + func indexExpr(idx Index) string { switch idx.Name { case IndexMatch, IndexJSON: @@ -200,9 +233,9 @@ func (w *writer) declaration(f *genFile) { case VerbPassthrough: w.p("\tPassthrough(%q)%s", fld.Name, end) case VerbEncrypt: - w.p("\tEncrypt(%q)%s", fld.Name, end) + w.p("\tEncrypt(%q, %s)%s", fld.Name, kindExpr(fld.GoType), end) case VerbEncryptInto: - w.p("\tEncryptInto(%q, %q)%s", fld.Name, fld.EQLType, end) + w.p("\tEncryptInto(%q, %s, %q)%s", fld.Name, kindExpr(fld.GoType), fld.EQLType, end) case VerbEncryptIndex, VerbIndex: call := "EncryptIndex" if fld.Verb == VerbIndex { @@ -212,7 +245,7 @@ func (w *writer) declaration(f *genFile) { for n, idx := range fld.Indexes { args[n] = indexExpr(idx) } - w.p("\t%s(%q, %s)%s", call, fld.Name, strings.Join(args, ", "), end) + w.p("\t%s(%q, %s, %s)%s", call, fld.Name, kindExpr(fld.GoType), strings.Join(args, ", "), end) } if fld.Identity != "" { end = "." diff --git a/languages/golang/stashgen/engine.go b/languages/golang/stashgen/engine.go index 8817d8dea..7d7d10f7c 100644 --- a/languages/golang/stashgen/engine.go +++ b/languages/golang/stashgen/engine.go @@ -2,15 +2,19 @@ package stashgen import ( "context" - "errors" + "fmt" + "strings" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/internal/record" ) // Engine answers the generator's questions about what the Rust engine can do. // The generator holds no copy of the engine's rules: every refusal about an // index, an EQL type or a field type comes from here. // -// The SDK's implementation runs the WASI guest that the SDK embeds. See -// [GuestEngine]. +// [GuestEngine] is the SDK's implementation: the WASI guest that package +// encrypt embeds, asked through se_plan_check and se_targets. type Engine interface { // EQLTypes lists the EQL types the engine can produce: each name, its // plaintext kind, its indexes and its query type. @@ -35,17 +39,199 @@ type EQLType struct { Query string } -// ErrEngineUnavailable is returned by [GuestEngine] until the SDK's guest -// is wired to the generator. -var ErrEngineUnavailable = errors.New("stashgen: the embedded engine is not available in this build") +// GuestEngine is the engine the SDK embeds: the WASI guest in package +// encrypt, asked through its se_plan_check and se_targets exports. It holds +// no credentials and makes no request. Close it when done. +func GuestEngine(ctx context.Context) (Engine, error) { + checker, err := encrypt.NewChecker(ctx) + if err != nil { + return nil, err + } + return &guestEngine{checker: checker}, nil +} -// GuestEngine returns the engine the SDK embeds. -// -// TODO(stack#1046 integration): run the WASI guest that package encrypt -// embeds (the build with the EQL types), ask it for its EQL types and have -// it check each declaration. Until then this returns [ErrEngineUnavailable], -// and stashgen stops before it reads any package. The static -// engine in package enginetest is the reference for what an Engine answers. -func GuestEngine(context.Context) (Engine, error) { - return nil, ErrEngineUnavailable +type guestEngine struct { + checker *encrypt.Checker +} + +// Close releases the guest. The Engine interface does not name it, so a +// caller that holds the engine as an Engine asserts to io.Closer. +func (e *guestEngine) Close() error { return e.checker.Close() } + +// EQLTypes asks the guest which EQL types it produces. +func (e *guestEngine) EQLTypes(ctx context.Context) ([]EQLType, error) { + targets, err := e.checker.Targets(ctx) + if err != nil { + return nil, err + } + out := make([]EQLType, 0, len(targets)) + for _, t := range targets { + eqlType := EQLType{Name: t.Name, Plaintext: kindOfWire(t.Kind), Query: t.Query} + for _, term := range t.Terms { + if name, ok := indexOfOutput[term]; ok { + eqlType.Indexes = append(eqlType.Indexes, name) + } + } + out = append(out, eqlType) + } + return out, nil +} + +// Check asks the guest about the declaration one field at a time, so the +// error names the field, then about the whole: the engine's rules across +// fields (two fields under one identity) show only there. +func (e *guestEngine) Check(ctx context.Context, d Declaration) error { + eqlTypes, err := e.EQLTypes(ctx) + if err != nil { + return err + } + plan, err := lowerDeclaration(d, eqlTypes) + if err != nil { + return err + } + if len(plan.Fields) == 0 { + // Nothing crosses the binding: an all-passthrough struct, which the + // generator refuses before it gets here. + return nil + } + for _, f := range plan.Fields { + one := &record.Plan{Context: plan.Context, Fields: []record.Field{f}} + if err := e.checker.Check(ctx, one); err != nil { + field := d.field(f.Name) + return &FieldError{Type: d.Type, Field: field.GoName, Reason: fmt.Sprintf( + "the engine refuses the declaration: %s over a %s value under %q (%v)", + describeOutputs(f.Outputs), kindWord(f.Kind), plan.Descriptor(f), err)} + } + } + if err := e.checker.Check(ctx, plan); err != nil { + return &FieldError{Type: d.Type, Reason: fmt.Sprintf("the engine refuses the declaration as a whole: %v", err)} + } + return nil +} + +// lowerDeclaration is the generator's lowering of a declaration to the plan +// the engine reads: the same shape gensupport lowers the generated +// declaration to. Passthrough and omitted fields stay on the host. +func lowerDeclaration(d Declaration, eqlTypes []EQLType) (*record.Plan, error) { + segments, err := record.ParseContext(d.Context) + if err != nil { + return nil, &FieldError{Type: d.Type, Reason: err.Error()} + } + plan := &record.Plan{Context: segments} + if d.Opaque { + // gensupport seals an opaque struct as one JSON document under + // /value; see gensupport.OpaqueField. + plan.Fields = []record.Field{{Name: "value", Kind: record.Bytes, Outputs: []record.Output{record.Ciphertext}}} + return plan, nil + } + for _, f := range d.Fields { + if !f.Sealed() { + continue + } + rf := record.Field{Name: f.Name, Identity: f.Identity, Kind: wireKind(f.GoType)} + switch f.Verb { + case VerbEncryptInto: + if len(eqlTypes) == 0 { + return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: "EQL types are not available yet; the engine produces none in this build"} + } + return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine cannot produce the EQL type %s yet", f.EQLType)} + case VerbEncrypt, VerbEncryptIndex: + rf.Outputs = append(rf.Outputs, record.Ciphertext) + } + for _, idx := range f.Indexes { + if len(idx.Options) > 0 { + return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("index %s: the engine cannot carry index options yet", idx)} + } + out, ok := outputOfIndex[idx.Name] + if !ok { + return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine does not derive the %s index yet", idx.Name)} + } + rf.Outputs = append(rf.Outputs, out) + } + plan.Fields = append(plan.Fields, rf) + } + return plan, nil +} + +var outputOfIndex = map[IndexName]record.Output{ + IndexEquality: record.Equality, IndexMatch: record.Match, IndexOre: record.Ore, IndexOpe: record.Ope, +} + +var indexOfOutput = map[record.Output]IndexName{ + record.Equality: IndexEquality, record.Match: IndexMatch, record.Ore: IndexOre, record.Ope: IndexOpe, +} + +// wireKind is the record kind of a Go type: the same mapping the emitter +// writes into the generated declaration (kindExpr). +func wireKind(t GoType) record.Kind { + switch kindExpr(t) { + case "gensupport.String": + return record.String + case "gensupport.Bool": + return record.Bool + case "gensupport.Bytes": + return record.Bytes + case "gensupport.Int32": + return record.Int32 + case "gensupport.Int64": + return record.Int64 + case "gensupport.UInt32": + return record.UInt32 + case "gensupport.UInt64": + return record.UInt64 + case "gensupport.Float32": + return record.Float32 + case "gensupport.Float64": + return record.Float64 + } + return record.Untyped +} + +// kindOfWire is the generator's kind for a wire kind, for an EQL type's +// plaintext. +func kindOfWire(k record.Kind) Kind { + switch k { + case record.String: + return KindString + case record.Bool: + return KindBool + case record.Bytes: + return KindBytes + case record.Int32, record.Int64: + return KindInt + case record.UInt32, record.UInt64: + return KindUint + case record.Float32, record.Float64: + return KindFloat + } + return KindOther +} + +func kindWord(k record.Kind) string { + if k == record.Untyped { + return "composite" + } + return string(k) +} + +func describeOutputs(outputs []record.Output) string { + words := make([]string, 0, len(outputs)) + for _, o := range outputs { + if o == record.Ciphertext { + words = append(words, "a ciphertext") + continue + } + words = append(words, string(indexOfOutput[o])+" index") + } + return strings.Join(words, ", ") +} + +// field finds a declared field by its declaration name. +func (d Declaration) field(name string) Field { + for _, f := range d.Fields { + if f.Name == name { + return f + } + } + return Field{} } diff --git a/languages/golang/stashgen/policy_test.go b/languages/golang/stashgen/policy_test.go index 7703def47..8bcf53515 100644 --- a/languages/golang/stashgen/policy_test.go +++ b/languages/golang/stashgen/policy_test.go @@ -181,8 +181,10 @@ func TestGenerateNeedsOutputAndAMessage(t *testing.T) { if _, _, err := stashgen.MessageType(42); err == nil { t.Fatal("an int accepted as a message") } + // With no WithEngine the embedded guest answers; the output directory + // is no module, so the run stops there or, with no guest built, before. err = stashgen.Generate(policy.SourceFunc(individualFacts), policy.ForMessage(&local{}, "c", base), stashgen.Output(filepath.Join(t.TempDir(), "x_stash.go"))) - if err == nil || !strings.Contains(err.Error(), "not available in this build") { - t.Fatalf("without an engine: %v", err) + if err == nil { + t.Fatal("a message in no module generated") } } diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index ebf7ba500..4148dacd5 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -526,6 +526,9 @@ func (r *reader) buildFields(c *collected) error { continue } g := genField{Field: Field{Name: cf.tag.Name, GoName: cf.goName, GoType: r.goType(cf.typ), Verb: VerbEncrypt}, typeExpr: r.typeExpr(cf.typ)} + if !readableInOpaque(g.GoType) { + return fieldErr(typeName, cf.goName, "the SDK cannot read a %s back out of an opaque struct yet; it reads scalars, []byte, slices of scalars and maps of scalars", g.typeExpr) + } if prev, dup := seen[g.Name]; dup { return fieldErr(typeName, cf.goName, "two fields write the name %q: %s and %s", g.Name, prev, cf.goName) } @@ -568,6 +571,9 @@ func (r *reader) buildFields(c *collected) error { } } if eqlType == nil { + if len(r.eql) == 0 { + return fieldErr(typeName, cf.goName, "EQL types are not available yet; this build of the engine produces none") + } return fieldErr(typeName, cf.goName, "the engine cannot produce the EQL type %s", field.EQLType) } f.imports.add(eqlPath, "eql") @@ -628,6 +634,7 @@ func classify(t types.Type, typeExpr func(types.Type) string, seen map[types.Typ defer delete(seen, t) switch u := t.Underlying().(type) { case *types.Basic: + g.Basic = u.Name() switch { case u.Info()&types.IsString != 0: g.Kind = KindString @@ -642,7 +649,7 @@ func classify(t types.Type, typeExpr func(types.Type) string, seen map[types.Typ } case *types.Slice: if b, ok := u.Elem().Underlying().(*types.Basic); ok && b.Kind() == types.Byte { - g.Kind = KindBytes + g.Kind, g.Basic = KindBytes, "[]byte" } else { elem := classify(u.Elem(), typeExpr, seen) g.Kind, g.Elem = KindSlice, &elem @@ -766,3 +773,16 @@ func itOrEach(fields []string) string { } return "each" } + +// readableInOpaque reports whether gensupport.Get reads the type back out of +// an opened opaque value: a scalar, bytes, a slice of scalars, or a map from +// strings to scalars. A nested struct is not, yet. +func readableInOpaque(t GoType) bool { + switch t.Kind { + case KindString, KindBool, KindInt, KindUint, KindFloat, KindBytes: + return true + case KindSlice, KindMap: + return t.Elem != nil && t.Elem.Kind.Scalar() + } + return false +} diff --git a/languages/golang/stashgen/stub_test.go b/languages/golang/stashgen/stub_test.go index 7bc09d9c8..e933ffc34 100644 --- a/languages/golang/stashgen/stub_test.go +++ b/languages/golang/stashgen/stub_test.go @@ -4,6 +4,7 @@ import ( "context" "go/types" "path/filepath" + "strings" "testing" "golang.org/x/tools/go/packages" @@ -15,8 +16,20 @@ import ( // what ships. A symbol that is only in the stub is the integration step's // work, and a symbol only in the real package is fine. func TestStubAgreesWithGensupport(t *testing.T) { - real := loadScope(t, filepath.Join("..", "encrypt", "gensupport"), ".") - stub := loadScope(t, filepath.Join("testdata", "stubsdk"), "./encrypt/gensupport") + compareStub(t, "gensupport", filepath.Join("..", "encrypt", "gensupport"), "./encrypt/gensupport", true) +} + +// The encrypt package is wider than what generated code names; only the +// symbols the stub declares are compared, and an interface by presence (the +// real one's methods take the module's internal types). +func TestStubAgreesWithEncrypt(t *testing.T) { + compareStub(t, "encrypt", filepath.Join("..", "encrypt"), "./encrypt", false) +} + +func compareStub(t *testing.T, pkg, realDir, stubPattern string, every bool) { + t.Helper() + real := loadScope(t, realDir, ".") + stub := loadScope(t, filepath.Join("testdata", "stubsdk"), stubPattern) shared := 0 for _, name := range real.Names() { obj := real.Lookup(name) @@ -25,14 +38,27 @@ func TestStubAgreesWithGensupport(t *testing.T) { } stubObj := stub.Lookup(name) if stubObj == nil { - t.Errorf("gensupport.%s is not in the stub; generated code may not name it", name) + if every { + t.Errorf("%s.%s is not in the stub; generated code may not name it", pkg, name) + } continue } shared++ - got := types.TypeString(stubObj.Type(), stripPkg) - want := types.TypeString(obj.Type(), stripPkg) + if _, isInterface := obj.Type().Underlying().(*types.Interface); isInterface { + if _, stubInterface := stubObj.Type().Underlying().(*types.Interface); !stubInterface { + t.Errorf("%s.%s: real package has an interface, stub has %s", pkg, name, stubObj.Type()) + } + continue + } + got := shape(stubObj.Type()) + want := shape(obj.Type()) if got != want { - t.Errorf("gensupport.%s: stub has %s, real package has %s", name, got, want) + t.Errorf("%s.%s: stub has %s, real package has %s", pkg, name, got, want) + } + } + for _, name := range stub.Names() { + if stub.Lookup(name).Exported() && real.Lookup(name) == nil { + t.Errorf("%s.%s is in the stub and not in the real package: the contract is unmet", pkg, name) } } if shared == 0 { @@ -44,6 +70,53 @@ func TestStubAgreesWithGensupport(t *testing.T) { // name alone. func stripPkg(*types.Package) string { return "" } +// shape renders a type without parameter names, which a stub need not +// repeat: a signature's type parameters, parameter types and result types. +func shape(t types.Type) string { + sig, ok := t.(*types.Signature) + if !ok { + return types.TypeString(t, stripPkg) + } + var b strings.Builder + b.WriteString("func") + if tp := sig.TypeParams(); tp != nil && tp.Len() > 0 { + b.WriteString("[") + for i := range tp.Len() { + if i > 0 { + b.WriteString(", ") + } + b.WriteString(tp.At(i).Obj().Name()) + b.WriteString(" ") + b.WriteString(types.TypeString(tp.At(i).Constraint(), stripPkg)) + } + b.WriteString("]") + } + b.WriteString("(") + for i := range sig.Params().Len() { + if i > 0 { + b.WriteString(", ") + } + if sig.Variadic() && i == sig.Params().Len()-1 { + b.WriteString("...") + b.WriteString(types.TypeString(sig.Params().At(i).Type().(*types.Slice).Elem(), stripPkg)) + continue + } + b.WriteString(types.TypeString(sig.Params().At(i).Type(), stripPkg)) + } + b.WriteString(")") + if sig.Results().Len() > 0 { + b.WriteString(" (") + for i := range sig.Results().Len() { + if i > 0 { + b.WriteString(", ") + } + b.WriteString(types.TypeString(sig.Results().At(i).Type(), stripPkg)) + } + b.WriteString(")") + } + return b.String() +} + func loadScope(t *testing.T, dir, pattern string) *types.Scope { t.Helper() cfg := &packages.Config{Context: context.Background(), Dir: dir, Mode: packages.NeedName | packages.NeedTypes} diff --git a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden index 3a1c3abd4..0d08a56fd 100644 --- a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden @@ -57,7 +57,7 @@ var declaration = gensupport.Declare("accounts"). Passthrough("created_at"). Passthrough("updated_at"). Passthrough("deleted_at"). - EncryptInto("email", "TextEq"). + EncryptInto("email", gensupport.String, "TextEq"). Omit("token") var codec = gensupport.New(gensupport.Generated[Account, EncryptedAccount]{ diff --git a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden index fc73074a3..facaff393 100644 --- a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden @@ -54,8 +54,8 @@ type contactShape struct { var declaration = gensupport.Declare("contacts"). Passthrough("id"). - EncryptIndex("email", encrypt.Equality, encrypt.Match()). - EncryptIndex("phone_number", encrypt.Equality). + EncryptIndex("email", gensupport.String, encrypt.Equality, encrypt.Match()). + EncryptIndex("phone_number", gensupport.String, encrypt.Equality). Omit("internal") var codec = gensupport.New(gensupport.Generated[crm.Contact, EncryptedContact]{ diff --git a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden index a63fe9fbd..ce3945fce 100644 --- a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden @@ -55,10 +55,10 @@ type patientShape struct { } var declaration = gensupport.Declare("patients"). - EncryptInto("name", "TextEq"). - EncryptIndex("email", encrypt.Equality). + EncryptInto("name", gensupport.String, "TextEq"). + EncryptIndex("email", gensupport.String, encrypt.Equality). Passthrough("mrn"). - Encrypt("chart") + Encrypt("chart", gensupport.Bytes) var codec = gensupport.New(gensupport.Generated[Patient, EncryptedPatient]{ TypeName: "Patient", diff --git a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden index 1fb87876b..aaca5c3fd 100644 --- a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden @@ -50,9 +50,9 @@ func (e EncryptedIndividual) LogValue() slog.Value { var declaration = gensupport.Declare("individuals"). Passthrough("id"). - Encrypt("name"). - EncryptIndex("email", encrypt.Equality, encrypt.Match()). - EncryptInto("medicare_number", "TextEq"). + Encrypt("name", gensupport.String). + EncryptIndex("email", gensupport.String, encrypt.Equality, encrypt.Match()). + EncryptInto("medicare_number", gensupport.String, "TextEq"). Passthrough("nickname") var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndividual]{ diff --git a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden index 57893c933..7effe1280 100644 --- a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden @@ -60,9 +60,9 @@ type orderShape struct { var orderDeclaration = gensupport.Declare("orders"). Passthrough("id"). - EncryptIndex("amount", encrypt.Equality, encrypt.Ore). - Encrypt("note"). - EncryptIndex("customer", encrypt.Match(), encrypt.Ope) + EncryptIndex("amount", gensupport.Int64, encrypt.Equality, encrypt.Ore). + Encrypt("note", gensupport.String). + EncryptIndex("customer", gensupport.String, encrypt.Match(), encrypt.Ope) var orderCodec = gensupport.New(gensupport.Generated[Order, EncryptedOrder]{ TypeName: "Order", diff --git a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden index 4f3c53b71..cdd36738c 100644 --- a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden @@ -38,7 +38,7 @@ type refundShape struct { var declaration = gensupport.Declare("refunds"). Passthrough("id"). - EncryptInto("reason", "TextEq") + EncryptInto("reason", gensupport.String, "TextEq") var codec = gensupport.New(gensupport.Generated[Refund, EncryptedRefund]{ TypeName: "Refund", diff --git a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden index a2af9c6a4..5224ae7a4 100644 --- a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden @@ -44,8 +44,8 @@ type userShape struct { var declaration = gensupport.Declare("users"). Passthrough("id"). - EncryptInto("email", "TextEq"). - EncryptInto("name", "TextEq"). + EncryptInto("email", gensupport.String, "TextEq"). + EncryptInto("name", gensupport.String, "TextEq"). Omit("internal") var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ diff --git a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden index 1fb87876b..aaca5c3fd 100644 --- a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden +++ b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden @@ -50,9 +50,9 @@ func (e EncryptedIndividual) LogValue() slog.Value { var declaration = gensupport.Declare("individuals"). Passthrough("id"). - Encrypt("name"). - EncryptIndex("email", encrypt.Equality, encrypt.Match()). - EncryptInto("medicare_number", "TextEq"). + Encrypt("name", gensupport.String). + EncryptIndex("email", gensupport.String, encrypt.Equality, encrypt.Match()). + EncryptInto("medicare_number", gensupport.String, "TextEq"). Passthrough("nickname") var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndividual]{ diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go index dbb7b5df3..216bbde6f 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go @@ -14,7 +14,9 @@ type Cipher struct{ _ struct{} } // Client decrypts under the keyset that sealed each value. type Client struct{ _ struct{} } -// Decrypter is what Decrypt takes: a *Client or a *Cipher. +// Decrypter is what Decrypt takes: a *Client or a *Cipher. The real +// interface's method takes the module's internal record types; the stub's +// is a placeholder, and the agreement test compares interfaces by presence. type Decrypter interface{ decrypter() } func (*Cipher) decrypter() {} @@ -33,11 +35,15 @@ type ( ) // Index is one index on a field, as generated code names it in a declaration. -type Index interface{ index() } +type Index interface { + Output() string + String() string +} type namedIndex string -func (namedIndex) index() {} +func (n namedIndex) Output() string { return string(n) } +func (n namedIndex) String() string { return string(n) } // The indexes. Equality, Ore and Ope take no options; Match and JSON do. var ( @@ -68,10 +74,12 @@ func (*Client) Keyset(KeysetName) *Cipher { return nil } func (c *Cipher) Extend(string) *Cipher { return c } // NewClient opens a client. -func NewClient(context.Context, ...Option) (*Client, error) { return nil, nil } +func NewClient(context.Context, ...ClientOption) (*Client, error) { return nil, nil } -// Option configures a client. -type Option interface{ option() } +// ClientOption configures a client. +type ClientOption func(*clientOptions) + +type clientOptions struct{} // Close closes the client. func (*Client) Close() error { return nil } diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go index 2cf9b1673..0e1aa24a2 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go @@ -18,7 +18,24 @@ type generatedVersion uint8 const GeneratedVersion1 generatedVersion = 1 // OpaqueField is the one field of an opaque declaration. -const OpaqueField = "." +const OpaqueField = "value" + +// Kind is a field's wire type. +type Kind string + +// The kinds. +const ( + Untyped Kind = "" + Bool Kind = "bool" + Int32 Kind = "int32" + Int64 Kind = "int64" + UInt32 Kind = "uint32" + UInt64 Kind = "uint64" + Float32 Kind = "float32" + Float64 Kind = "float64" + String Kind = "string" + Bytes Kind = "bytes" +) // Redacted formats a value with its sealed fields hidden. func Redacted(typeName string, shown map[string]any, hidden ...string) string { return "" } @@ -53,12 +70,13 @@ func Declare(context string) Declaration { return Declaration{} } // DeclareOpaque declares a struct sealed as one value. func DeclareOpaque(context string) Declaration { return Declaration{} } -func (d Declaration) Passthrough(name string) Declaration { return d } -func (d Declaration) Encrypt(name string) Declaration { return d } -func (d Declaration) EncryptIndex(name string, idx ...encrypt.Index) Declaration { return d } -func (d Declaration) Index(name string, idx ...encrypt.Index) Declaration { return d } -func (d Declaration) EncryptInto(name, eqlType string) Declaration { return d } -func (d Declaration) Omit(name string) Declaration { return d } +func (d Declaration) Passthrough(name string) Declaration { return d } +func (d Declaration) Encrypt(name string, kind Kind) Declaration { return d } +func (d Declaration) EncryptIndex(name string, kind Kind, idx ...encrypt.Index) Declaration { return d } +func (d Declaration) Index(name string, kind Kind, idx ...encrypt.Index) Declaration { return d } +func (d Declaration) EncryptInto(name string, kind Kind, eqlType string) Declaration { return d } +func (d Declaration) Omit(name string) Declaration { return d } +func (d Declaration) Err() error { return nil } // Generated is what a generated file gives the library for one type. type Generated[P, E any] struct { diff --git a/mise.toml b/mise.toml index c20ee3766..86a8e1c15 100644 --- a/mise.toml +++ b/mise.toml @@ -183,6 +183,36 @@ cp "$module" ../wasm/stack_encrypt_guest.wasm echo "embedded into languages/golang/encrypt/wasm/stack_encrypt_guest.wasm" """ +# The deterministic-kms TEST build of the same crate: `se_cipher_init` takes +# the record fixture's 32-byte seed and derives every key from it, so the Go +# tests open the records Rust sealed (packages/stack-encrypt/tests/fixtures/ +# record_lowering.json) and run hermetic round trips with no ZeroKMS. The Go +# package never embeds it: its tests look for it beside the real guest and +# skip when it is absent. The import-surface gate denies what the real +# build's does; it requires no transport import, because a build with no +# ZeroKMS client has none. +[tasks."wasm:guest:build:deterministic"] +description = "Build the deterministic-kms TEST build of the stack-encrypt WASI guest (wasm32-wasip1, release), for the Go fixture and round-trip tests" +shell = "bash -c" +run = """ +set -euo pipefail +rustup target add wasm32-wasip1 +root=$(pwd) +cd languages/golang/encrypt/guest +cargo build --target wasm32-wasip1 --release --features deterministic-kms +module=target/wasm32-wasip1/release/stack_encrypt_guest.wasm +python3 "$root/scripts/check-wasm-imports.py" "$module" \\ + --allow-module wasi_snapshot_preview1 \\ + --deny-prefix wasi_snapshot_preview1:path_ \\ + --deny-prefix wasi_snapshot_preview1:sock_ \\ + --deny-prefix wasi_snapshot_preview1:fd_prestat \\ + --require wasi_snapshot_preview1:random_get +# No transport imports to require: with no ZeroKMS client in the build the +# linker drops the host module altogether, which is the point of the build. +cp "$module" ../wasm/stack_encrypt_guest_deterministic.wasm +echo "test build at languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm" +""" + [tasks."wasm:guest:test"] description = "Lint and natively test the stack-encrypt WASI guest (ops/config/status modules run on the host target)" shell = "bash -c" @@ -198,9 +228,11 @@ RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p stack-guest-abi --target wasm3 cd languages/golang/encrypt/guest cargo fmt --check cargo clippy --all-targets -- -D warnings +cargo clippy --all-targets --features deterministic-kms -- -D warnings # The wasm32-only modules (abi, host) only compile for the target; lint # them there so a broken export surface can't hide behind native-only CI. cargo clippy --target wasm32-wasip1 -- -D warnings +cargo clippy --target wasm32-wasip1 --features deterministic-kms -- -D warnings # Intra-doc links, on the target the crate is written for (the wasm32-only # modules are part of the crate docs). rustdoc only warns on a broken link # and exits 0, so without -D warnings a stale link ships silently. @@ -267,7 +299,7 @@ dir = "languages/golang" run = "golangci-lint run ./..." [tasks."go:encrypt:example"] -description = "Run the stack-encrypt Go example (languages/golang/encrypt/example) against real ZeroKMS; needs `stash auth login` first" +description = "Run the Go SDK example (languages/golang/encrypt/example) against real ZeroKMS; needs `stash auth login` first" shell = "bash -c" # Both guests: the example reads the profile through auth. depends = ["wasm:guest:build", "wasm:auth-guest:build"] @@ -278,15 +310,3 @@ cd languages/golang # if set, else the developer profile. See encrypt/example/README.md. CGO_ENABLED=0 go run ./encrypt/example """ - -[tasks."go:encrypt:example:explicit"] -description = "Run the explicit-credentials Go example (languages/golang/encrypt/example/explicit) against real ZeroKMS; pass -secrets-dir, -client-id and -workspace-crn after --" -shell = "bash -c" -depends = ["wasm:guest:build", "wasm:auth-guest:build"] -run = """ -set -euo pipefail -cd languages/golang -# Credentials come only from the flags and the secrets directory: no CS_* -# variables, no profile. See encrypt/example/explicit/README.md. -CGO_ENABLED=0 go run ./encrypt/example/explicit "$@" -""" diff --git a/scripts/__tests__/crates-ci.test.mjs b/scripts/__tests__/crates-ci.test.mjs index 5cb9371f6..8b3a64bb3 100644 --- a/scripts/__tests__/crates-ci.test.mjs +++ b/scripts/__tests__/crates-ci.test.mjs @@ -225,10 +225,6 @@ const CI_EXEMPT_TASKS = new Map([ 'go:encrypt:example', 'A walkthrough against real ZeroKMS for a developer who has run `stash auth login`. CI exercises the same client through the Go live tests in tests-golang.yml `live`.', ], - [ - 'go:encrypt:example:explicit', - 'The same walkthrough, with credentials passed as flags after `--`. Nothing for CI to pass; the live tests cover explicit credentials (`liveClient`).', - ], ]) describe('the scan sees every task mise sees', () => { From 3b53ce95cc2f0ce4f1d956956b392c8df3fc15a4 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:22:59 -0700 Subject: [PATCH 06/30] fix(golang): every field type the generator accepts round-trips MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two Codex findings on PR #1094, both real, both the same shape: a value the generator accepted and Encrypt sealed could never be decrypted. Get sent every opened value through convert, whose switch knows the wire kinds and a few slice types. A passthrough field is the Go value the struct held, kept on the host, so a time.Time or a sql.NullTime — the plan's own accounts example — reached convert and was refused. Get now returns a value that already is a T before it converts anything. An opaque struct came back from JSON into Values and each field went through Get, so a map[string]string (JSON gives map[string]any), a type defined over a scalar, a nested struct or a pointer could not be read, although classify and readableInOpaque had accepted them. The opaque value now crosses as a JSON document of a generated shape struct (documentOpaque, with json tags for the declared names), and the generated Value decodes it straight into that struct with gensupport. Opaque, so every type encoding/json carries both ways comes back as it was. The generator's rule is that: a field of an opaque struct may be a scalar or a type defined over one, []byte, a slice or array, a map with string or integer keys, a pointer, a struct whose fields are all exported, or a type with its own MarshalJSON/UnmarshalJSON (time.Time); a channel, an interface, a map keyed by a struct, or a struct JSON would truncate is refused at generate time with the field named. Outside an opaque struct the engine seals a composite as a tree of leaves and a column holds one, so stashgen now refuses encrypt or index= on a struct, slice or map field ("seals only as part of an opaque struct") instead of letting Encrypt fail at run time. A type defined over a scalar (type Email string) is accepted: the engine returns the underlying type, and the generated Value reads it at that type and converts, since Go has no way to do that generically without reflection. The proof is in encrypt/internal/testusers: Account (time.Time and sql.NullTime passthrough beside an indexed field), Kinds (one sealed field of every scalar kind, defined types among them, and passthrough fields of every type the codec cannot carry) and Everything (an opaque struct with a field of every type JSON carries), each generated by the real stashgen against the real guest and walked through Encrypt then Decrypt over the deterministic build with reflect.DeepEqual. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/cmd/stashgen/README.md | 9 +- languages/golang/encrypt/gensupport/codec.go | 57 +- .../golang/encrypt/gensupport/convert.go | 67 +- .../gensupport/gensupport_internal_test.go | 51 +- .../internal/testusers/account_stash.go | 146 ++++ .../internal/testusers/document_stash.go | 32 +- .../internal/testusers/everything_stash.go | 176 +++++ .../encrypt/internal/testusers/kinds.go | 100 +++ .../encrypt/internal/testusers/kinds_stash.go | 744 ++++++++++++++++++ languages/golang/encrypt/kinds_test.go | 116 +++ languages/golang/stashgen/emit.go | 45 +- languages/golang/stashgen/read.go | 94 ++- languages/golang/stashgen/refusal_test.go | 13 +- .../cases/documents/document_stash.go.golden | 32 +- .../stubsdk/encrypt/gensupport/gensupport.go | 3 + 15 files changed, 1515 insertions(+), 170 deletions(-) create mode 100644 languages/golang/encrypt/internal/testusers/account_stash.go create mode 100644 languages/golang/encrypt/internal/testusers/everything_stash.go create mode 100644 languages/golang/encrypt/internal/testusers/kinds.go create mode 100644 languages/golang/encrypt/internal/testusers/kinds_stash.go create mode 100644 languages/golang/encrypt/kinds_test.go diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index a5397f03f..79b49a637 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -73,6 +73,8 @@ The first part of a tag is the field's name, which is the column name in a datab | `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value | The index names are `equality`, `match`, `ore`, `ope` and `json`. +A `match` index needs text with at least one token: the engine derives no match term for an empty or separator-only string, or one shorter than the n-gram, because an empty term would match every row. +`Encrypt` then returns an error naming the row, the field and the index; give the field a value, or drop the index. An index takes its options in parentheses after its name, separated by commas: `index=equality;match(k=3)`. These words are the same as the Rust API's words for the same behaviour. @@ -136,7 +138,9 @@ The error names the type and the field, and never a value. - a model with a field that has no tag, or with no field for an output; - two structs in one package that would both write `Encrypt`; - an embedded struct from another package with no tag; -- a struct whose every field is left out. +- a struct whose every field is left out; +- a struct, slice or map field with `encrypt` or `index=` outside an opaque struct; +- a field of an opaque struct whose type JSON cannot carry both ways. ## Declarations from a policy @@ -180,7 +184,8 @@ No warning, error or log line holds a plaintext value. Generated code sends the engine every sealed field with its value, under the declaration lowered to data: each field's label (`/`), its outputs and its wire type (`int64`, `string`, `bytes`, ...), which `stashgen` chose from the field's Go type. A passthrough field stays on the host: the engine does nothing to it a program could observe, and the FFI codec cannot carry every Go type a program stores beside a ciphertext. -An `opaque` struct crosses as one JSON document and is one column; its fields are what JSON carries: scalars, `[]byte`, slices and maps of them. +An `opaque` struct crosses as one JSON document and is one column, decoded back into the exact Go types the struct declares; a field may be any type `encoding/json` carries both ways (a scalar or a type defined over one, `[]byte`, slices, maps with string or integer keys, pointers, structs whose fields are all exported, `time.Time`). +A sealed field outside an opaque struct is one scalar, or a type defined over one; a struct, slice or map field seals only inside an opaque struct. ## Status diff --git a/languages/golang/encrypt/gensupport/codec.go b/languages/golang/encrypt/gensupport/codec.go index 5cb90f144..de9db8bbc 100644 --- a/languages/golang/encrypt/gensupport/codec.go +++ b/languages/golang/encrypt/gensupport/codec.go @@ -1,7 +1,6 @@ package gensupport import ( - "bytes" "context" "encoding/json" "fmt" @@ -174,11 +173,6 @@ func (c *Codec[P, E]) Decrypt(ctx context.Context, d encrypt.Decrypter, encrypte for name, v := range src { vals[name] = v } - if c.g.Declaration.opaque { - if vals[OpaqueField], err = opaqueValues(vals[OpaqueField]); err != nil { - return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) - } - } if out[i], err = c.g.Value(encrypted[i], vals); err != nil { return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) } @@ -252,17 +246,23 @@ func Passthrough[T any](rec Record, name string) (T, error) { return v, nil } -// Get reads one opened field as the Go type the struct declares it. The -// engine returns a value at the field's declared kind; Get converts within -// that kind's family (a uint32 into a uint8 that holds it) and refuses -// anything else, so a value that opens to another type is an error and -// never a silent zero. +// Get reads one opened field as the Go type the struct declares it. A +// passthrough field is the Go value the struct held, whatever its type, and +// comes back as it is. A sealed field comes back from the engine at its +// declared wire kind; Get converts within that kind's family (a uint32 into +// a uint8 that holds it) and refuses anything else, so a value that opens to +// another type is an error and never a silent zero. A defined type over a +// scalar (type Email string) is read at its underlying type and converted by +// the generated code. func Get[T any](vals Values, name string) (T, error) { var out T v, ok := vals[name] if !ok { return out, fmt.Errorf("gensupport: the opened value has no field %q", name) } + if exact, ok := v.(T); ok { + return exact, nil + } if err := convert(v, &out); err != nil { return out, fmt.Errorf("gensupport: field %q: %w", name, err) } @@ -311,14 +311,10 @@ func (c *RecordsCodec[P, R]) Decrypt(ctx context.Context, d encrypt.Decrypter, r } // opaqueBytes is an opaque struct's fields as the one value the engine -// seals: a JSON document, so the struct is one column and its fields come -// back as what JSON carries. +// seals: a JSON document of the generated shape struct, so the struct is one +// column and every field type encoding/json round-trips comes back as it +// was. func opaqueBytes(fields any) ([]byte, error) { - switch fields.(type) { - case map[string]any, Values: - default: - return nil, fmt.Errorf("the generated Source gave a %T for the opaque value, not a map of its fields", fields) - } encoded, err := json.Marshal(fields) if err != nil { return nil, fmt.Errorf("the opaque value does not encode: %w", err) @@ -326,19 +322,20 @@ func opaqueBytes(fields any) ([]byte, error) { return encoded, nil } -// opaqueValues reads the opened opaque value back into its fields. Numbers -// stay json.Number so Get converts each to the struct's own integer or -// float type without a detour through float64. -func opaqueValues(opened any) (Values, error) { - encoded, ok := opened.([]byte) +// Opaque reads an opened opaque value into the generated shape struct: the +// JSON document the engine returned, decoded into the exact Go types the +// struct declares. Generated code calls it from Value. +func Opaque[T any](vals Values, out *T) error { + v, ok := vals[OpaqueField] + if !ok { + return fmt.Errorf("gensupport: the opened value has no field %q", OpaqueField) + } + encoded, ok := v.([]byte) if !ok { - return nil, fmt.Errorf("the opaque value opened as %T, not bytes", opened) + return fmt.Errorf("gensupport: the opaque value opened as %T, not bytes", v) } - dec := json.NewDecoder(bytes.NewReader(encoded)) - dec.UseNumber() - var fields map[string]any - if err := dec.Decode(&fields); err != nil { - return nil, fmt.Errorf("the opaque value does not decode: %w", err) + if err := json.Unmarshal(encoded, out); err != nil { + return fmt.Errorf("gensupport: the opaque value does not decode: %w", err) } - return Values(fields), nil + return nil } diff --git a/languages/golang/encrypt/gensupport/convert.go b/languages/golang/encrypt/gensupport/convert.go index ec6bfce2d..4a4de4748 100644 --- a/languages/golang/encrypt/gensupport/convert.go +++ b/languages/golang/encrypt/gensupport/convert.go @@ -1,24 +1,20 @@ package gensupport import ( - "encoding/base64" - "encoding/json" "fmt" "math" - "strconv" "github.com/cipherstash/vitaminc/bindings/go/vcvalue" ) -// convert writes an opened value into out, a pointer to the Go type the -// struct declares. The engine returns a value at the field's declared kind -// — int32, int64, uint32, uint64, float32, float64, string, []byte, bool, -// or a vcvalue.Object for a composite — and the struct's type is in the -// same family, narrower at most. A value outside the target's range, or of -// another family, is an error. +// convert writes an opened sealed value into out, a pointer to the Go type +// the struct declares. The engine returns a value at the field's declared +// kind — int32, int64, uint32, uint64, float32, float64, string, []byte, +// bool, or a vcvalue.Object for a composite — and the struct's type is in +// the same family, narrower at most. A value outside the target's range, or +// of another family, is an error. func convert(v any, out any) error { - // A nil slice or map of the struct went out as JSON null and comes back - // as nil: the zero value it was. + // A nil slice or map comes back as nil: the zero value it was. if v == nil { switch out.(type) { case *[]byte, *[]any, *[]string, *[]int64, *[]int32, *[]uint32, *[]uint64, *[]float64, *[]bool, *[][]byte, *Values, *map[string]any, *any: @@ -26,15 +22,6 @@ func convert(v any, out any) error { } return fmt.Errorf("opened as nothing, and %T holds a value", out) } - // An opaque struct's fields come back from JSON: a number is a - // json.Number, bytes are a base64 string. Widen them to what the kind - // paths below read. - if n, ok := v.(json.Number); ok { - var err error - if v, err = widenNumber(n, out); err != nil { - return err - } - } switch out := out.(type) { case *string: s, ok := v.(string) @@ -43,18 +30,11 @@ func convert(v any, out any) error { } *out = s case *[]byte: - switch b := v.(type) { - case []byte: - *out = append([]byte(nil), b...) - case string: - decoded, err := base64.StdEncoding.DecodeString(b) - if err != nil { - return fmt.Errorf("opened as a string that is not base64 bytes: %w", err) - } - *out = decoded - default: + b, ok := v.([]byte) + if !ok { return mismatch(v, *out) } + *out = append([]byte(nil), b...) case *bool: b, ok := v.(bool) if !ok { @@ -243,30 +223,3 @@ func valuesOf(v any) (Values, error) { } return nil, fmt.Errorf("opened as %T, not an object", v) } - -// widenNumber reads a JSON number as the widest value of the target's -// family: int64 for a signed target, uint64 for an unsigned one, float64 for -// a float. The family conversion then applies its range check. -func widenNumber(n json.Number, out any) (any, error) { - switch out.(type) { - case *int, *int8, *int16, *int32, *int64: - i, err := strconv.ParseInt(string(n), 10, 64) - if err != nil { - return nil, fmt.Errorf("%s is not an integer that fits an int64", n) - } - return i, nil - case *uint, *uint8, *uint16, *uint32, *uint64: - u, err := strconv.ParseUint(string(n), 10, 64) - if err != nil { - return nil, fmt.Errorf("%s is not an integer that fits a uint64", n) - } - return u, nil - case *float32, *float64, *any: - f, err := n.Float64() - if err != nil { - return nil, err - } - return f, nil - } - return n, nil -} diff --git a/languages/golang/encrypt/gensupport/gensupport_internal_test.go b/languages/golang/encrypt/gensupport/gensupport_internal_test.go index b3d4a45d8..53fcbe9de 100644 --- a/languages/golang/encrypt/gensupport/gensupport_internal_test.go +++ b/languages/golang/encrypt/gensupport/gensupport_internal_test.go @@ -1,11 +1,10 @@ package gensupport import ( - "encoding/base64" - "encoding/json" "reflect" "strings" "testing" + "time" "github.com/cipherstash/stack/languages/golang/encrypt" "github.com/cipherstash/stack/languages/golang/internal/record" @@ -136,32 +135,40 @@ func TestConvertStaysWithinAFamily(t *testing.T) { if err := convert(obj, &x); err == nil { t.Fatal("a struct target was accepted") } - // An opaque struct's fields come back from JSON. - var n int32 - if err := convert(json.Number("-7"), &n); err != nil || n != -7 { - t.Fatalf("json.Number -> int32: %v %d", err, n) + // An opaque value decodes straight into the generated shape type. + type shape struct { + Title string `json:"title"` + N int32 `json:"n"` + Tags []string `json:"tags"` + Inner map[string]string `json:"inner"` } - if err := convert(json.Number("3000000000"), &n); err == nil { - t.Fatal("3000000000 fit an int32") + encoded, err := opaqueBytes(shape{Title: "x", N: 4, Tags: []string{"a"}, Inner: map[string]string{"k": "v"}}) + if err != nil { + t.Fatal(err) } - var f float64 - if err := convert(json.Number("1.25"), &f); err != nil || f != 1.25 { - t.Fatalf("json.Number -> float64: %v %v", err, f) + var back shape + if err := Opaque(Values{OpaqueField: encoded}, &back); err != nil || back.Title != "x" || back.N != 4 || back.Inner["k"] != "v" { + t.Fatalf("Opaque: %v %+v", err, back) } - var raw []byte - if err := convert(base64.StdEncoding.EncodeToString([]byte{1, 2}), &raw); err != nil || !reflect.DeepEqual(raw, []byte{1, 2}) { - t.Fatalf("base64 -> []byte: %v %v", err, raw) + if err := Opaque(Values{OpaqueField: "not bytes"}, &back); err == nil { + t.Fatal("Opaque accepted a string") } - fields, err := opaqueValues([]byte(`{"title":"x","n":4,"tags":["a"],"inner":{"k":true}}`)) - if err != nil { - t.Fatal(err) + if err := Opaque(Values{}, &back); err == nil { + t.Fatal("Opaque found a missing field") + } + // A passthrough value of any type comes back as it is. + when := time.Date(2026, 10, 6, 1, 2, 3, 0, time.UTC) + gotTime, err := Get[time.Time](Values{"at": when}, "at") + if err != nil || !gotTime.Equal(when) { + t.Fatalf("Get[time.Time] = %v %v", gotTime, err) } - inner, err := Get[Values](fields, "inner") - if err != nil || inner["k"] != true { - t.Fatalf("nested: %v %v", err, inner) + type defined string + gotDefined, err := Get[defined](Values{"d": defined("x")}, "d") + if err != nil || gotDefined != "x" { + t.Fatalf("Get[defined] = %v %v", gotDefined, err) } - if _, err := opaqueBytes("not a map"); err == nil { - t.Fatal("opaqueBytes accepted a string") + if _, err := Get[defined](Values{"d": "x"}, "d"); err == nil { + t.Fatal("Get converted a wire string into a defined type; the generated code does that") } got, err := Get[uint8](Values{"age": uint32(3)}, "age") if err != nil || got != 3 { diff --git a/languages/golang/encrypt/internal/testusers/account_stash.go b/languages/golang/encrypt/internal/testusers/account_stash.go new file mode 100644 index 000000000..92f4c3f47 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/account_stash.go @@ -0,0 +1,146 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "database/sql" + "log/slog" + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Account prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedAccount struct { + ID uint + CreatedAt time.Time + DeletedAt sql.NullTime + Email EncryptedAccountEmail +} + +type EncryptedAccountEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +func (e EncryptedAccount) String() string { + return gensupport.Redacted("EncryptedAccount", map[string]any{"ID": e.ID, "CreatedAt": e.CreatedAt, "DeletedAt": e.DeletedAt}, "Email") +} + +func (e EncryptedAccount) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID, "CreatedAt": e.CreatedAt, "DeletedAt": e.DeletedAt}, "Email") +} + +// Stops compiling when Account gains, loses, reorders or retypes a field. +var _ = accountShape(Account{}) + +type accountShape struct { + _ struct{} + ID uint + CreatedAt time.Time + DeletedAt sql.NullTime + Email string +} + +var accountDeclaration = gensupport.Declare("accounts"). + Passthrough("id"). + Passthrough("created_at"). + Passthrough("deleted_at"). + EncryptIndex("email", gensupport.String, encrypt.Equality) + +var accountCodec = gensupport.New(gensupport.Generated[Account, EncryptedAccount]{ + TypeName: "Account", + Declaration: accountDeclaration, + PrintsPlaintext: true, + Source: func(v Account) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "created_at": v.CreatedAt, + "deleted_at": v.DeletedAt, + "email": v.Email, + } + }, + Seal: func(rec gensupport.Record) (EncryptedAccount, error) { + var e EncryptedAccount + var err error + if e.ID, err = gensupport.Passthrough[uint](rec, "id"); err != nil { + return EncryptedAccount{}, err + } + if e.CreatedAt, err = gensupport.Passthrough[time.Time](rec, "created_at"); err != nil { + return EncryptedAccount{}, err + } + if e.DeletedAt, err = gensupport.Passthrough[sql.NullTime](rec, "deleted_at"); err != nil { + return EncryptedAccount{}, err + } + e.Email = EncryptedAccountEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + } + return e, nil + }, + Open: func(e EncryptedAccount) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "created_at": {Value: e.CreatedAt}, + "deleted_at": {Value: e.DeletedAt}, + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality}, + } + }, + Value: func(e EncryptedAccount, vals gensupport.Values) (Account, error) { + var v Account + var err error + if v.ID, err = gensupport.Get[uint](vals, "id"); err != nil { + return Account{}, err + } + if v.CreatedAt, err = gensupport.Get[time.Time](vals, "created_at"); err != nil { + return Account{}, err + } + if v.DeletedAt, err = gensupport.Get[sql.NullTime](vals, "deleted_at"); err != nil { + return Account{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return Account{}, err + } + return v, nil + }, +}) + +// EncryptAccount seals each Account in one ZeroKMS request. The result has one +// element for each input, in the same order. +func EncryptAccount(ctx context.Context, cipher *encrypt.Cipher, values []Account) ([]EncryptedAccount, error) { + return accountCodec.Encrypt(ctx, cipher, values) +} + +// DecryptAccount opens each EncryptedAccount in one ZeroKMS request. +func DecryptAccount(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedAccount) ([]Account, error) { + return accountCodec.Decrypt(ctx, d, encrypted) +} + +var AccountFields = struct { + Email AccountEmailField +}{ + Email: AccountEmailField{gensupport.NewField[string](accountDeclaration, "email")}, +} + +type AccountEmailField struct { + field gensupport.Field[string] +} + +func (f AccountEmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedAccountEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedAccountEmail{}, err + } + return EncryptedAccountEmail{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f AccountEmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} diff --git a/languages/golang/encrypt/internal/testusers/document_stash.go b/languages/golang/encrypt/internal/testusers/document_stash.go index b55d48b56..351c19547 100644 --- a/languages/golang/encrypt/internal/testusers/document_stash.go +++ b/languages/golang/encrypt/internal/testusers/document_stash.go @@ -38,6 +38,14 @@ type documentShape struct { Tags []string } +// The opaque value as it crosses the binding: one JSON document of the +// struct's fields, by their declared names. +type documentOpaque struct { + Title string `json:"title"` + Body string `json:"body"` + Tags []string `json:"tags"` +} + var documentDeclaration = gensupport.DeclareOpaque("documents/v2/body") var documentCodec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ @@ -45,10 +53,10 @@ var documentCodec = gensupport.New(gensupport.Generated[Document, EncryptedDocum Declaration: documentDeclaration, PrintsPlaintext: true, Source: func(v Document) gensupport.Values { - return gensupport.Values{gensupport.OpaqueField: map[string]any{ - "title": v.Title, - "body": v.Body, - "tags": v.Tags, + return gensupport.Values{gensupport.OpaqueField: documentOpaque{ + Title: v.Title, + Body: v.Body, + Tags: v.Tags, }} }, Seal: func(rec gensupport.Record) (EncryptedDocument, error) { @@ -58,20 +66,14 @@ var documentCodec = gensupport.New(gensupport.Generated[Document, EncryptedDocum return gensupport.Record{gensupport.OpaqueField: {Ciphertext: e.Sealed}} }, Value: func(e EncryptedDocument, vals gensupport.Values) (Document, error) { - fields, err := gensupport.Get[gensupport.Values](vals, gensupport.OpaqueField) - if err != nil { + var o documentOpaque + if err := gensupport.Opaque(vals, &o); err != nil { return Document{}, err } var v Document - if v.Title, err = gensupport.Get[string](fields, "title"); err != nil { - return Document{}, err - } - if v.Body, err = gensupport.Get[string](fields, "body"); err != nil { - return Document{}, err - } - if v.Tags, err = gensupport.Get[[]string](fields, "tags"); err != nil { - return Document{}, err - } + v.Title = o.Title + v.Body = o.Body + v.Tags = o.Tags return v, nil }, }) diff --git a/languages/golang/encrypt/internal/testusers/everything_stash.go b/languages/golang/encrypt/internal/testusers/everything_stash.go new file mode 100644 index 000000000..deed845dd --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/everything_stash.go @@ -0,0 +1,176 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "database/sql" + "log/slog" + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Everything prints its sealed fields in the clear: it has no String or +// LogValue method. Write them, or run stashgen with -redact. + +type EncryptedEverything struct { + Sealed encrypt.Ciphertext +} + +func (e EncryptedEverything) String() string { + return gensupport.Redacted("EncryptedEverything", nil, "Sealed") +} + +func (e EncryptedEverything) LogValue() slog.Value { + return gensupport.RedactedLog(nil, "Sealed") +} + +// Stops compiling when Everything gains, loses, reorders or retypes a field. +var _ = everythingShape(Everything{}) + +type everythingShape struct { + _ struct{} + S string + E Email + Bo bool + I8 int8 + I int + U64 uint64 + F32 float32 + F64 float64 + By []byte + Bl Blob + Sc Score + Tags []string + Emails []Email + Counts map[string]int + Labels map[string]string + ByKey map[int]string + In Inner + Ins []Inner + P *string + PI *Inner + T time.Time + N sql.NullTime + Nested [][]byte + Arr [2]uint8 +} + +// The opaque value as it crosses the binding: one JSON document of the +// struct's fields, by their declared names. +type everythingOpaque struct { + S string `json:"s"` + E Email `json:"e"` + Bo bool `json:"bo"` + I8 int8 `json:"i8"` + I int `json:"i"` + U64 uint64 `json:"u64"` + F32 float32 `json:"f32"` + F64 float64 `json:"f64"` + By []byte `json:"by"` + Bl Blob `json:"bl"` + Sc Score `json:"sc"` + Tags []string `json:"tags"` + Emails []Email `json:"emails"` + Counts map[string]int `json:"counts"` + Labels map[string]string `json:"labels"` + ByKey map[int]string `json:"by_key"` + In Inner `json:"in"` + Ins []Inner `json:"ins"` + P *string `json:"p"` + PI *Inner `json:"pi"` + T time.Time `json:"t"` + N sql.NullTime `json:"n"` + Nested [][]byte `json:"nested"` + Arr [2]uint8 `json:"arr"` +} + +var everythingDeclaration = gensupport.DeclareOpaque("everything") + +var everythingCodec = gensupport.New(gensupport.Generated[Everything, EncryptedEverything]{ + TypeName: "Everything", + Declaration: everythingDeclaration, + PrintsPlaintext: true, + Source: func(v Everything) gensupport.Values { + return gensupport.Values{gensupport.OpaqueField: everythingOpaque{ + S: v.S, + E: v.E, + Bo: v.Bo, + I8: v.I8, + I: v.I, + U64: v.U64, + F32: v.F32, + F64: v.F64, + By: v.By, + Bl: v.Bl, + Sc: v.Sc, + Tags: v.Tags, + Emails: v.Emails, + Counts: v.Counts, + Labels: v.Labels, + ByKey: v.ByKey, + In: v.In, + Ins: v.Ins, + P: v.P, + PI: v.PI, + T: v.T, + N: v.N, + Nested: v.Nested, + Arr: v.Arr, + }} + }, + Seal: func(rec gensupport.Record) (EncryptedEverything, error) { + return EncryptedEverything{Sealed: rec[gensupport.OpaqueField].Ciphertext}, nil + }, + Open: func(e EncryptedEverything) gensupport.Record { + return gensupport.Record{gensupport.OpaqueField: {Ciphertext: e.Sealed}} + }, + Value: func(e EncryptedEverything, vals gensupport.Values) (Everything, error) { + var o everythingOpaque + if err := gensupport.Opaque(vals, &o); err != nil { + return Everything{}, err + } + var v Everything + v.S = o.S + v.E = o.E + v.Bo = o.Bo + v.I8 = o.I8 + v.I = o.I + v.U64 = o.U64 + v.F32 = o.F32 + v.F64 = o.F64 + v.By = o.By + v.Bl = o.Bl + v.Sc = o.Sc + v.Tags = o.Tags + v.Emails = o.Emails + v.Counts = o.Counts + v.Labels = o.Labels + v.ByKey = o.ByKey + v.In = o.In + v.Ins = o.Ins + v.P = o.P + v.PI = o.PI + v.T = o.T + v.N = o.N + v.Nested = o.Nested + v.Arr = o.Arr + return v, nil + }, +}) + +// EncryptEverything seals each Everything in one ZeroKMS request. The result +// has one element for each input, in the same order. +func EncryptEverything(ctx context.Context, cipher *encrypt.Cipher, values []Everything) ([]EncryptedEverything, error) { + return everythingCodec.Encrypt(ctx, cipher, values) +} + +// DecryptEverything opens each EncryptedEverything in one ZeroKMS request. +func DecryptEverything(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedEverything) ([]Everything, error) { + return everythingCodec.Decrypt(ctx, d, encrypted) +} diff --git a/languages/golang/encrypt/internal/testusers/kinds.go b/languages/golang/encrypt/internal/testusers/kinds.go new file mode 100644 index 000000000..5827ebde6 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/kinds.go @@ -0,0 +1,100 @@ +package testusers + +import ( + "database/sql" + "time" +) + +// Defined types over scalars: the engine returns the underlying type and the +// generated code converts. +type ( + Email string + Blob []byte + Score int16 +) + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Account -name Account + +// Account is the shape of the plan's accounts example: passthrough fields +// whose types the FFI codec cannot carry (a time.Time, a driver.Valuer), kept +// on the host and returned as they are. +type Account struct { + _ struct{} `stash:"context=accounts"` + ID uint `stash:"id,passthrough"` + CreatedAt time.Time `stash:"created_at,passthrough"` + DeletedAt sql.NullTime `stash:"deleted_at,passthrough"` + Email string `stash:"email,encrypt,index=equality"` +} + +// Inner is a struct carried through a passthrough field and inside an +// opaque struct. +type Inner struct { + Name string + N int32 +} + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Kinds -name Kinds + +// Kinds has one sealed field of every scalar kind stashgen accepts outside +// an opaque struct, with defined types among them, and passthrough fields of +// types the codec cannot carry. Everything it accepts round-trips. +type Kinds struct { + _ struct{} `stash:"context=kinds"` + S string `stash:"s,encrypt,index=equality;match"` + E Email `stash:"e,encrypt,index=equality;match;ore"` + Bo bool `stash:"bo,encrypt"` + I8 int8 `stash:"i8,encrypt,index=equality;ore"` + I16 int16 `stash:"i16,encrypt,index=ope"` + I32 int32 `stash:"i32,encrypt,index=equality"` + I int `stash:"i,encrypt,index=ore"` + I64 int64 `stash:"i64,encrypt"` + U8 uint8 `stash:"u8,encrypt,index=equality"` + U16 uint16 `stash:"u16,encrypt"` + U32 uint32 `stash:"u32,encrypt,index=equality;ore"` + U uint `stash:"u,encrypt"` + U64 uint64 `stash:"u64,encrypt,index=ope"` + F32 float32 `stash:"f32,encrypt"` + F64 float64 `stash:"f64,encrypt,index=ore"` + By []byte `stash:"by,encrypt,index=equality"` + Bl Blob `stash:"bl,encrypt"` + Sc Score `stash:"sc,encrypt,index=ore"` + + P *int `stash:"p,passthrough"` + M map[string]string `stash:"m,passthrough"` + T time.Time `stash:"t,passthrough"` + N sql.NullTime `stash:"n,passthrough"` + In Inner `stash:"in,passthrough"` + Sk []Inner `stash:"sk,passthrough"` +} + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Everything -name Everything + +// Everything is an opaque struct with a field of every type JSON carries +// both ways. It crosses as one document and comes back as it was. +type Everything struct { + _ struct{} `stash:"context=everything,opaque"` + S string + E Email + Bo bool + I8 int8 + I int + U64 uint64 + F32 float32 + F64 float64 + By []byte + Bl Blob + Sc Score + Tags []string + Emails []Email + Counts map[string]int + Labels map[string]string + ByKey map[int]string + In Inner + Ins []Inner + P *string + PI *Inner + T time.Time + N sql.NullTime + Nested [][]byte + Arr [2]uint8 +} diff --git a/languages/golang/encrypt/internal/testusers/kinds_stash.go b/languages/golang/encrypt/internal/testusers/kinds_stash.go new file mode 100644 index 000000000..9270e3316 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/kinds_stash.go @@ -0,0 +1,744 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "database/sql" + "log/slog" + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Kinds prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedKinds struct { + S EncryptedKindsS + E EncryptedKindsE + Bo EncryptedKindsBo + I8 EncryptedKindsI8 + I16 EncryptedKindsI16 + I32 EncryptedKindsI32 + I EncryptedKindsI + I64 EncryptedKindsI64 + U8 EncryptedKindsU8 + U16 EncryptedKindsU16 + U32 EncryptedKindsU32 + U EncryptedKindsU + U64 EncryptedKindsU64 + F32 EncryptedKindsF32 + F64 EncryptedKindsF64 + By EncryptedKindsBy + Bl EncryptedKindsBl + Sc EncryptedKindsSc + P *int + M map[string]string + T time.Time + N sql.NullTime + In Inner + Sk []Inner +} + +type EncryptedKindsS struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} + +type EncryptedKindsE struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm + Ore encrypt.OreTerm +} + +type EncryptedKindsBo struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedKindsI8 struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Ore encrypt.OreTerm +} + +type EncryptedKindsI16 struct { + Ciphertext encrypt.Ciphertext + Ope encrypt.OpeTerm +} + +type EncryptedKindsI32 struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +type EncryptedKindsI struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm +} + +type EncryptedKindsI64 struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedKindsU8 struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +type EncryptedKindsU16 struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedKindsU32 struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Ore encrypt.OreTerm +} + +type EncryptedKindsU struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedKindsU64 struct { + Ciphertext encrypt.Ciphertext + Ope encrypt.OpeTerm +} + +type EncryptedKindsF32 struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedKindsF64 struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm +} + +type EncryptedKindsBy struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +type EncryptedKindsBl struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedKindsSc struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm +} + +func (e EncryptedKinds) String() string { + return gensupport.Redacted("EncryptedKinds", map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc") +} + +func (e EncryptedKinds) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc") +} + +// Stops compiling when Kinds gains, loses, reorders or retypes a field. +var _ = kindsShape(Kinds{}) + +type kindsShape struct { + _ struct{} + S string + E Email + Bo bool + I8 int8 + I16 int16 + I32 int32 + I int + I64 int64 + U8 uint8 + U16 uint16 + U32 uint32 + U uint + U64 uint64 + F32 float32 + F64 float64 + By []byte + Bl Blob + Sc Score + P *int + M map[string]string + T time.Time + N sql.NullTime + In Inner + Sk []Inner +} + +var kindsDeclaration = gensupport.Declare("kinds"). + EncryptIndex("s", gensupport.String, encrypt.Equality, encrypt.Match()). + EncryptIndex("e", gensupport.String, encrypt.Equality, encrypt.Match(), encrypt.Ore). + Encrypt("bo", gensupport.Bool). + EncryptIndex("i8", gensupport.Int32, encrypt.Equality, encrypt.Ore). + EncryptIndex("i16", gensupport.Int32, encrypt.Ope). + EncryptIndex("i32", gensupport.Int32, encrypt.Equality). + EncryptIndex("i", gensupport.Int64, encrypt.Ore). + Encrypt("i64", gensupport.Int64). + EncryptIndex("u8", gensupport.UInt32, encrypt.Equality). + Encrypt("u16", gensupport.UInt32). + EncryptIndex("u32", gensupport.UInt32, encrypt.Equality, encrypt.Ore). + Encrypt("u", gensupport.UInt64). + EncryptIndex("u64", gensupport.UInt64, encrypt.Ope). + Encrypt("f32", gensupport.Float32). + EncryptIndex("f64", gensupport.Float64, encrypt.Ore). + EncryptIndex("by", gensupport.Bytes, encrypt.Equality). + Encrypt("bl", gensupport.Bytes). + EncryptIndex("sc", gensupport.Int32, encrypt.Ore). + Passthrough("p"). + Passthrough("m"). + Passthrough("t"). + Passthrough("n"). + Passthrough("in"). + Passthrough("sk") + +var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ + TypeName: "Kinds", + Declaration: kindsDeclaration, + PrintsPlaintext: true, + Source: func(v Kinds) gensupport.Values { + return gensupport.Values{ + "s": v.S, + "e": v.E, + "bo": v.Bo, + "i8": v.I8, + "i16": v.I16, + "i32": v.I32, + "i": v.I, + "i64": v.I64, + "u8": v.U8, + "u16": v.U16, + "u32": v.U32, + "u": v.U, + "u64": v.U64, + "f32": v.F32, + "f64": v.F64, + "by": v.By, + "bl": v.Bl, + "sc": v.Sc, + "p": v.P, + "m": v.M, + "t": v.T, + "n": v.N, + "in": v.In, + "sk": v.Sk, + } + }, + Seal: func(rec gensupport.Record) (EncryptedKinds, error) { + var e EncryptedKinds + var err error + if e.P, err = gensupport.Passthrough[*int](rec, "p"); err != nil { + return EncryptedKinds{}, err + } + if e.M, err = gensupport.Passthrough[map[string]string](rec, "m"); err != nil { + return EncryptedKinds{}, err + } + if e.T, err = gensupport.Passthrough[time.Time](rec, "t"); err != nil { + return EncryptedKinds{}, err + } + if e.N, err = gensupport.Passthrough[sql.NullTime](rec, "n"); err != nil { + return EncryptedKinds{}, err + } + if e.In, err = gensupport.Passthrough[Inner](rec, "in"); err != nil { + return EncryptedKinds{}, err + } + if e.Sk, err = gensupport.Passthrough[[]Inner](rec, "sk"); err != nil { + return EncryptedKinds{}, err + } + e.S = EncryptedKindsS{ + Ciphertext: rec["s"].Ciphertext, + Equality: rec["s"].Equality, + Match: rec["s"].Match, + } + e.E = EncryptedKindsE{ + Ciphertext: rec["e"].Ciphertext, + Equality: rec["e"].Equality, + Match: rec["e"].Match, + Ore: rec["e"].Ore, + } + e.Bo = EncryptedKindsBo{Ciphertext: rec["bo"].Ciphertext} + e.I8 = EncryptedKindsI8{ + Ciphertext: rec["i8"].Ciphertext, + Equality: rec["i8"].Equality, + Ore: rec["i8"].Ore, + } + e.I16 = EncryptedKindsI16{ + Ciphertext: rec["i16"].Ciphertext, + Ope: rec["i16"].Ope, + } + e.I32 = EncryptedKindsI32{ + Ciphertext: rec["i32"].Ciphertext, + Equality: rec["i32"].Equality, + } + e.I = EncryptedKindsI{ + Ciphertext: rec["i"].Ciphertext, + Ore: rec["i"].Ore, + } + e.I64 = EncryptedKindsI64{Ciphertext: rec["i64"].Ciphertext} + e.U8 = EncryptedKindsU8{ + Ciphertext: rec["u8"].Ciphertext, + Equality: rec["u8"].Equality, + } + e.U16 = EncryptedKindsU16{Ciphertext: rec["u16"].Ciphertext} + e.U32 = EncryptedKindsU32{ + Ciphertext: rec["u32"].Ciphertext, + Equality: rec["u32"].Equality, + Ore: rec["u32"].Ore, + } + e.U = EncryptedKindsU{Ciphertext: rec["u"].Ciphertext} + e.U64 = EncryptedKindsU64{ + Ciphertext: rec["u64"].Ciphertext, + Ope: rec["u64"].Ope, + } + e.F32 = EncryptedKindsF32{Ciphertext: rec["f32"].Ciphertext} + e.F64 = EncryptedKindsF64{ + Ciphertext: rec["f64"].Ciphertext, + Ore: rec["f64"].Ore, + } + e.By = EncryptedKindsBy{ + Ciphertext: rec["by"].Ciphertext, + Equality: rec["by"].Equality, + } + e.Bl = EncryptedKindsBl{Ciphertext: rec["bl"].Ciphertext} + e.Sc = EncryptedKindsSc{ + Ciphertext: rec["sc"].Ciphertext, + Ore: rec["sc"].Ore, + } + return e, nil + }, + Open: func(e EncryptedKinds) gensupport.Record { + return gensupport.Record{ + "s": {Ciphertext: e.S.Ciphertext, Equality: e.S.Equality, Match: e.S.Match}, + "e": {Ciphertext: e.E.Ciphertext, Equality: e.E.Equality, Match: e.E.Match, Ore: e.E.Ore}, + "bo": {Ciphertext: e.Bo.Ciphertext}, + "i8": {Ciphertext: e.I8.Ciphertext, Equality: e.I8.Equality, Ore: e.I8.Ore}, + "i16": {Ciphertext: e.I16.Ciphertext, Ope: e.I16.Ope}, + "i32": {Ciphertext: e.I32.Ciphertext, Equality: e.I32.Equality}, + "i": {Ciphertext: e.I.Ciphertext, Ore: e.I.Ore}, + "i64": {Ciphertext: e.I64.Ciphertext}, + "u8": {Ciphertext: e.U8.Ciphertext, Equality: e.U8.Equality}, + "u16": {Ciphertext: e.U16.Ciphertext}, + "u32": {Ciphertext: e.U32.Ciphertext, Equality: e.U32.Equality, Ore: e.U32.Ore}, + "u": {Ciphertext: e.U.Ciphertext}, + "u64": {Ciphertext: e.U64.Ciphertext, Ope: e.U64.Ope}, + "f32": {Ciphertext: e.F32.Ciphertext}, + "f64": {Ciphertext: e.F64.Ciphertext, Ore: e.F64.Ore}, + "by": {Ciphertext: e.By.Ciphertext, Equality: e.By.Equality}, + "bl": {Ciphertext: e.Bl.Ciphertext}, + "sc": {Ciphertext: e.Sc.Ciphertext, Ore: e.Sc.Ore}, + "p": {Value: e.P}, + "m": {Value: e.M}, + "t": {Value: e.T}, + "n": {Value: e.N}, + "in": {Value: e.In}, + "sk": {Value: e.Sk}, + } + }, + Value: func(e EncryptedKinds, vals gensupport.Values) (Kinds, error) { + var v Kinds + var err error + if v.S, err = gensupport.Get[string](vals, "s"); err != nil { + return Kinds{}, err + } + rawE, err := gensupport.Get[string](vals, "e") + if err != nil { + return Kinds{}, err + } + v.E = Email(rawE) + if v.Bo, err = gensupport.Get[bool](vals, "bo"); err != nil { + return Kinds{}, err + } + if v.I8, err = gensupport.Get[int8](vals, "i8"); err != nil { + return Kinds{}, err + } + if v.I16, err = gensupport.Get[int16](vals, "i16"); err != nil { + return Kinds{}, err + } + if v.I32, err = gensupport.Get[int32](vals, "i32"); err != nil { + return Kinds{}, err + } + if v.I, err = gensupport.Get[int](vals, "i"); err != nil { + return Kinds{}, err + } + if v.I64, err = gensupport.Get[int64](vals, "i64"); err != nil { + return Kinds{}, err + } + if v.U8, err = gensupport.Get[uint8](vals, "u8"); err != nil { + return Kinds{}, err + } + if v.U16, err = gensupport.Get[uint16](vals, "u16"); err != nil { + return Kinds{}, err + } + if v.U32, err = gensupport.Get[uint32](vals, "u32"); err != nil { + return Kinds{}, err + } + if v.U, err = gensupport.Get[uint](vals, "u"); err != nil { + return Kinds{}, err + } + if v.U64, err = gensupport.Get[uint64](vals, "u64"); err != nil { + return Kinds{}, err + } + if v.F32, err = gensupport.Get[float32](vals, "f32"); err != nil { + return Kinds{}, err + } + if v.F64, err = gensupport.Get[float64](vals, "f64"); err != nil { + return Kinds{}, err + } + if v.By, err = gensupport.Get[[]byte](vals, "by"); err != nil { + return Kinds{}, err + } + rawBl, err := gensupport.Get[[]byte](vals, "bl") + if err != nil { + return Kinds{}, err + } + v.Bl = Blob(rawBl) + rawSc, err := gensupport.Get[int16](vals, "sc") + if err != nil { + return Kinds{}, err + } + v.Sc = Score(rawSc) + if v.P, err = gensupport.Get[*int](vals, "p"); err != nil { + return Kinds{}, err + } + if v.M, err = gensupport.Get[map[string]string](vals, "m"); err != nil { + return Kinds{}, err + } + if v.T, err = gensupport.Get[time.Time](vals, "t"); err != nil { + return Kinds{}, err + } + if v.N, err = gensupport.Get[sql.NullTime](vals, "n"); err != nil { + return Kinds{}, err + } + if v.In, err = gensupport.Get[Inner](vals, "in"); err != nil { + return Kinds{}, err + } + if v.Sk, err = gensupport.Get[[]Inner](vals, "sk"); err != nil { + return Kinds{}, err + } + return v, nil + }, +}) + +// EncryptKinds seals each Kinds in one ZeroKMS request. The result has one +// element for each input, in the same order. +func EncryptKinds(ctx context.Context, cipher *encrypt.Cipher, values []Kinds) ([]EncryptedKinds, error) { + return kindsCodec.Encrypt(ctx, cipher, values) +} + +// DecryptKinds opens each EncryptedKinds in one ZeroKMS request. +func DecryptKinds(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedKinds) ([]Kinds, error) { + return kindsCodec.Decrypt(ctx, d, encrypted) +} + +var KindsFields = struct { + S KindsSField + E KindsEField + Bo KindsBoField + I8 KindsI8Field + I16 KindsI16Field + I32 KindsI32Field + I KindsIField + I64 KindsI64Field + U8 KindsU8Field + U16 KindsU16Field + U32 KindsU32Field + U KindsUField + U64 KindsU64Field + F32 KindsF32Field + F64 KindsF64Field + By KindsByField + Bl KindsBlField + Sc KindsScField +}{ + S: KindsSField{gensupport.NewField[string](kindsDeclaration, "s")}, + E: KindsEField{gensupport.NewField[Email](kindsDeclaration, "e")}, + Bo: KindsBoField{gensupport.NewField[bool](kindsDeclaration, "bo")}, + I8: KindsI8Field{gensupport.NewField[int8](kindsDeclaration, "i8")}, + I16: KindsI16Field{gensupport.NewField[int16](kindsDeclaration, "i16")}, + I32: KindsI32Field{gensupport.NewField[int32](kindsDeclaration, "i32")}, + I: KindsIField{gensupport.NewField[int](kindsDeclaration, "i")}, + I64: KindsI64Field{gensupport.NewField[int64](kindsDeclaration, "i64")}, + U8: KindsU8Field{gensupport.NewField[uint8](kindsDeclaration, "u8")}, + U16: KindsU16Field{gensupport.NewField[uint16](kindsDeclaration, "u16")}, + U32: KindsU32Field{gensupport.NewField[uint32](kindsDeclaration, "u32")}, + U: KindsUField{gensupport.NewField[uint](kindsDeclaration, "u")}, + U64: KindsU64Field{gensupport.NewField[uint64](kindsDeclaration, "u64")}, + F32: KindsF32Field{gensupport.NewField[float32](kindsDeclaration, "f32")}, + F64: KindsF64Field{gensupport.NewField[float64](kindsDeclaration, "f64")}, + By: KindsByField{gensupport.NewField[[]byte](kindsDeclaration, "by")}, + Bl: KindsBlField{gensupport.NewField[Blob](kindsDeclaration, "bl")}, + Sc: KindsScField{gensupport.NewField[Score](kindsDeclaration, "sc")}, +} + +type KindsSField struct { + field gensupport.Field[string] +} + +func (f KindsSField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedKindsS, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsS{}, err + } + return EncryptedKindsS{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil +} + +func (f KindsSField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f KindsSField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +type KindsEField struct { + field gensupport.Field[Email] +} + +func (f KindsEField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Email) (EncryptedKindsE, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsE{}, err + } + return EncryptedKindsE{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match, Ore: out.Ore}, nil +} + +func (f KindsEField) Equality(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f KindsEField) Match(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.MatchTerm, error) { + return f.field.Match(ctx, c, v) +} + +func (f KindsEField) Ore(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type KindsBoField struct { + field gensupport.Field[bool] +} + +func (f KindsBoField) Encrypt(ctx context.Context, c *encrypt.Cipher, v bool) (EncryptedKindsBo, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedKindsBo{Ciphertext: out.Ciphertext}, err +} + +type KindsI8Field struct { + field gensupport.Field[int8] +} + +func (f KindsI8Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int8) (EncryptedKindsI8, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsI8{}, err + } + return EncryptedKindsI8{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil +} + +func (f KindsI8Field) Equality(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f KindsI8Field) Ore(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type KindsI16Field struct { + field gensupport.Field[int16] +} + +func (f KindsI16Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int16) (EncryptedKindsI16, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsI16{}, err + } + return EncryptedKindsI16{Ciphertext: out.Ciphertext, Ope: out.Ope}, nil +} + +func (f KindsI16Field) Ope(ctx context.Context, c *encrypt.Cipher, v int16) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} + +type KindsI32Field struct { + field gensupport.Field[int32] +} + +func (f KindsI32Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int32) (EncryptedKindsI32, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsI32{}, err + } + return EncryptedKindsI32{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f KindsI32Field) Equality(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +type KindsIField struct { + field gensupport.Field[int] +} + +func (f KindsIField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int) (EncryptedKindsI, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsI{}, err + } + return EncryptedKindsI{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil +} + +func (f KindsIField) Ore(ctx context.Context, c *encrypt.Cipher, v int) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type KindsI64Field struct { + field gensupport.Field[int64] +} + +func (f KindsI64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int64) (EncryptedKindsI64, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedKindsI64{Ciphertext: out.Ciphertext}, err +} + +type KindsU8Field struct { + field gensupport.Field[uint8] +} + +func (f KindsU8Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint8) (EncryptedKindsU8, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsU8{}, err + } + return EncryptedKindsU8{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f KindsU8Field) Equality(ctx context.Context, c *encrypt.Cipher, v uint8) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +type KindsU16Field struct { + field gensupport.Field[uint16] +} + +func (f KindsU16Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint16) (EncryptedKindsU16, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedKindsU16{Ciphertext: out.Ciphertext}, err +} + +type KindsU32Field struct { + field gensupport.Field[uint32] +} + +func (f KindsU32Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (EncryptedKindsU32, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsU32{}, err + } + return EncryptedKindsU32{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil +} + +func (f KindsU32Field) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +func (f KindsU32Field) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type KindsUField struct { + field gensupport.Field[uint] +} + +func (f KindsUField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint) (EncryptedKindsU, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedKindsU{Ciphertext: out.Ciphertext}, err +} + +type KindsU64Field struct { + field gensupport.Field[uint64] +} + +func (f KindsU64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint64) (EncryptedKindsU64, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsU64{}, err + } + return EncryptedKindsU64{Ciphertext: out.Ciphertext, Ope: out.Ope}, nil +} + +func (f KindsU64Field) Ope(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { + return f.field.Ope(ctx, c, v) +} + +type KindsF32Field struct { + field gensupport.Field[float32] +} + +func (f KindsF32Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v float32) (EncryptedKindsF32, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedKindsF32{Ciphertext: out.Ciphertext}, err +} + +type KindsF64Field struct { + field gensupport.Field[float64] +} + +func (f KindsF64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v float64) (EncryptedKindsF64, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsF64{}, err + } + return EncryptedKindsF64{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil +} + +func (f KindsF64Field) Ore(ctx context.Context, c *encrypt.Cipher, v float64) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type KindsByField struct { + field gensupport.Field[[]byte] +} + +func (f KindsByField) Encrypt(ctx context.Context, c *encrypt.Cipher, v []byte) (EncryptedKindsBy, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsBy{}, err + } + return EncryptedKindsBy{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f KindsByField) Equality(ctx context.Context, c *encrypt.Cipher, v []byte) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +type KindsBlField struct { + field gensupport.Field[Blob] +} + +func (f KindsBlField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Blob) (EncryptedKindsBl, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedKindsBl{Ciphertext: out.Ciphertext}, err +} + +type KindsScField struct { + field gensupport.Field[Score] +} + +func (f KindsScField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Score) (EncryptedKindsSc, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsSc{}, err + } + return EncryptedKindsSc{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil +} + +func (f KindsScField) Ore(ctx context.Context, c *encrypt.Cipher, v Score) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} diff --git a/languages/golang/encrypt/kinds_test.go b/languages/golang/encrypt/kinds_test.go new file mode 100644 index 000000000..8718b48a3 --- /dev/null +++ b/languages/golang/encrypt/kinds_test.go @@ -0,0 +1,116 @@ +package encrypt_test + +import ( + "context" + "database/sql" + "reflect" + "testing" + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" +) + +// Everything the generator accepts round-trips: one sealed field of every +// scalar kind, defined types among them, passthrough fields of types the FFI +// codec cannot carry, and an opaque struct with a field of every type JSON +// carries. Hermetic, over the deterministic guest. + +func TestPassthroughKeepsEveryGoType(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + when := time.Date(2026, 10, 6, 1, 2, 3, 4, time.FixedZone("AEDT", 11*3600)) + accounts := []testusers.Account{ + {ID: 1, CreatedAt: when, DeletedAt: sql.NullTime{Time: when.Add(time.Hour), Valid: true}, Email: "alice@example.com"}, + {ID: 2, CreatedAt: when.UTC(), Email: "bob@example.com"}, + } + encrypted, err := testusers.EncryptAccount(ctx, cipher, accounts) + if err != nil { + t.Fatal(err) + } + if !encrypted[0].CreatedAt.Equal(when) || encrypted[0].DeletedAt != accounts[0].DeletedAt || encrypted[1].ID != 2 { + t.Fatalf("passthrough fields: %+v", encrypted) + } + back, err := testusers.DecryptAccount(ctx, c, encrypted) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(back, accounts) { + t.Fatalf("DecryptAccount = %+v, want %+v", back, accounts) + } +} + +func TestEverySealedKindRoundTrips(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + seven := 7 + when := time.Date(2026, 1, 2, 3, 4, 5, 6, time.UTC) + in := []testusers.Kinds{{ + S: "alice", E: "alice@example.com", Bo: true, + I8: -8, I16: -16, I32: -32, I: -64, I64: -1 << 40, + U8: 8, U16: 16, U32: 32, U: 64, U64: 1 << 40, + F32: 1.5, F64: -2.25, + By: []byte{1, 2, 3}, Bl: testusers.Blob{4, 5}, Sc: -3, + P: &seven, M: map[string]string{"k": "v"}, T: when, N: sql.NullTime{Time: when, Valid: true}, + In: testusers.Inner{Name: "in", N: 1}, Sk: []testusers.Inner{{Name: "a", N: 2}}, + }, { + S: "bob", E: "bob@example.com", I8: 127, I16: 32767, I32: 1<<31 - 1, I: 1 << 30, I64: 1<<63 - 1, + U8: 255, U16: 65535, U32: 1<<32 - 1, U: 1 << 30, U64: 1<<64 - 1, + F32: -0.5, F64: 1e300, By: []byte{}, Bl: testusers.Blob{}, Sc: 32767, + M: map[string]string{}, Sk: []testusers.Inner{}, + }} + encrypted, err := testusers.EncryptKinds(ctx, cipher, in) + if err != nil { + t.Fatal(err) + } + for _, e := range encrypted { + if len(e.E.Equality) != 32 || len(e.E.Match) == 0 || len(e.E.Ore) == 0 || len(e.Sc.Ore) == 0 || len(e.F64.Ore) == 0 || len(e.U64.Ope) == 0 { + t.Fatalf("terms missing: %+v", e) + } + } + back, err := testusers.DecryptKinds(ctx, c, encrypted) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(back, in) { + t.Fatalf("DecryptKinds =\n%+v\nwant\n%+v", back, in) + } + // A defined type's field entry takes and derives at the defined type. + term, err := testusers.KindsFields.E.Equality(ctx, cipher, testusers.Email("alice@example.com")) + if err != nil || !term.Equal(encrypted[0].E.Equality) { + t.Fatalf("defined-type probe: %v", err) + } + one, err := testusers.KindsFields.Sc.Encrypt(ctx, cipher, -3) + if err != nil || len(one.Ciphertext) == 0 { + t.Fatalf("Score field: %v", err) + } +} + +func TestEveryOpaqueFieldTypeRoundTrips(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + hello := "hello" + when := time.Date(2026, 1, 2, 3, 4, 5, 6, time.UTC) + in := []testusers.Everything{{ + S: "s", E: "e@example.com", Bo: true, I8: -1, I: 42, U64: 1 << 40, F32: 0.25, F64: 9.75, + By: []byte("bytes"), Bl: testusers.Blob("blob"), Sc: -7, + Tags: []string{"a", "b"}, Emails: []testusers.Email{"x@y"}, Counts: map[string]int{"a": 1}, Labels: map[string]string{"k": "v"}, ByKey: map[int]string{3: "three"}, + In: testusers.Inner{Name: "in", N: 5}, Ins: []testusers.Inner{{Name: "i", N: 6}}, P: &hello, PI: &testusers.Inner{Name: "pi", N: 7}, + T: when, N: sql.NullTime{Time: when, Valid: true}, Nested: [][]byte{{1}, {2, 3}}, Arr: [2]uint8{9, 8}, + }, { + // Zero values, with the slices and maps nil: JSON null comes back as nil. + }} + encrypted, err := testusers.EncryptEverything(ctx, cipher, in) + if err != nil { + t.Fatal(err) + } + back, err := testusers.DecryptEverything(ctx, c, encrypted) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(back, in) { + t.Fatalf("DecryptEverything =\n%+v\nwant\n%+v", back, in) + } +} diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index dc8b899d2..32e41dee6 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -25,6 +25,7 @@ func emit(f *genFile) ([]byte, error) { w.encryptedType(f) w.printMethods(f) w.shape(f) + w.opaqueShape(f) w.declaration(f) w.codec(f) w.functions(f) @@ -155,6 +156,22 @@ func (w *writer) printMethods(f *genFile) { w.p("}") } +// opaqueShape writes the struct an opaque value crosses the binding as: the +// struct's fields by their declared names, as one JSON document. +func (w *writer) opaqueShape(f *genFile) { + if !f.decl.Opaque { + return + } + w.nl() + w.p("// The opaque value as it crosses the binding: one JSON document of the") + w.p("// struct's fields, by their declared names.") + w.p("type %s struct {", f.opaqueShape) + for _, g := range f.opaque { + w.p("\t%s %s `json:%q`", g.GoName, g.typeExpr, g.Name) + } + w.p("}") +} + func (w *writer) shape(f *genFile) { if f.shapeName == "" { return @@ -298,9 +315,9 @@ func (w *writer) source(f *genFile) { if f.decl.Opaque { entries := make([]string, len(f.opaque)) for i, g := range f.opaque { - entries[i] = fmt.Sprintf("%q: v.%s", g.Name, g.GoName) + entries[i] = fmt.Sprintf("%s: v.%s", g.GoName, g.GoName) } - w.literal("\t\t", "return gensupport.Values{gensupport.OpaqueField: map[string]any{", entries, "}}") + w.literal("\t\t", "return gensupport.Values{gensupport.OpaqueField: "+f.opaqueShape+"{", entries, "}}") } else { entries := make([]string, len(f.fields)) for i, g := range f.fields { @@ -386,15 +403,17 @@ func (w *writer) value(f *genFile) { fail := func() { w.p("\t\t\treturn %s, err", f.zeroExpr) } switch { case f.decl.Opaque: - w.p("\t\tfields, err := gensupport.Get[gensupport.Values](vals, gensupport.OpaqueField)") - w.p("\t\tif err != nil {") + w.p("\t\tvar o %s", f.opaqueShape) + w.p("\t\tif err := gensupport.Opaque(vals, &o); err != nil {") fail() w.p("\t\t}") - w.p("\t\tvar v %s", f.typeExpr) + if f.isPointer { + w.p("\t\tv := &%s{}", strings.TrimPrefix(f.typeExpr, "*")) + } else { + w.p("\t\tvar v %s", f.typeExpr) + } for _, g := range f.opaque { - w.p("\t\tif v.%s, err = gensupport.Get[%s](fields, %q); err != nil {", g.GoName, g.typeExpr, g.Name) - fail() - w.p("\t\t}") + w.p("\t\tv.%s = o.%s", g.GoName, g.GoName) } default: if f.isPointer { @@ -404,6 +423,16 @@ func (w *writer) value(f *genFile) { } w.p("\t\tvar err error") for _, g := range f.fields { + if g.Sealed() && g.definedScalar() { + // The engine returns the underlying type; the conversion + // to the defined type is the struct's own. + w.p("\t\traw%s, err := gensupport.Get[%s](vals, %q)", g.GoName, g.GoType.Basic, g.Name) + w.p("\t\tif err != nil {") + fail() + w.p("\t\t}") + w.p("\t\tv.%s = %s(raw%s)", g.GoName, g.typeExpr, g.GoName) + continue + } w.p("\t\tif v.%s, err = gensupport.Get[%s](vals, %q); err != nil {", g.GoName, g.typeExpr, g.Name) fail() w.p("\t\t}") diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index 4148dacd5..8c2d3fe0a 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -49,11 +49,12 @@ type genFile struct { redact bool redactRecv string - decl Declaration - members []encMember // the fields of the encrypted type - fields []genField // every stored field, in declared order - opaque []genField // the fields of an opaque struct - models []genModel + decl Declaration + members []encMember // the fields of the encrypted type + fields []genField // every stored field, in declared order + opaque []genField // the fields of an opaque struct + opaqueShape string // the generated struct the opaque value crosses as + models []genModel } // shapeField is one field of the shape struct, which mirrors the plaintext @@ -86,6 +87,13 @@ type genField struct { pathType string // the type with its package path, for model checks } +// definedScalar reports whether the field's type is a named type over a +// scalar (type Email string): the engine returns the underlying type, and +// the generated code converts. +func (g genField) definedScalar() bool { + return g.GoType.Kind.Scalar() && g.GoType.Basic != "" && g.typeExpr != g.GoType.Basic +} + // output is one output of a sealed field in separate columns. type output struct { name string // "Ciphertext", "Equality" @@ -526,8 +534,8 @@ func (r *reader) buildFields(c *collected) error { continue } g := genField{Field: Field{Name: cf.tag.Name, GoName: cf.goName, GoType: r.goType(cf.typ), Verb: VerbEncrypt}, typeExpr: r.typeExpr(cf.typ)} - if !readableInOpaque(g.GoType) { - return fieldErr(typeName, cf.goName, "the SDK cannot read a %s back out of an opaque struct yet; it reads scalars, []byte, slices of scalars and maps of scalars", g.typeExpr) + if reason := notJSONEncodable(cf.typ, map[types.Type]bool{}); reason != "" { + return fieldErr(typeName, cf.goName, "an opaque struct crosses as one JSON document, and %s %s", g.typeExpr, reason) } if prev, dup := seen[g.Name]; dup { return fieldErr(typeName, cf.goName, "two fields write the name %q: %s and %s", g.Name, prev, cf.goName) @@ -537,6 +545,7 @@ func (r *reader) buildFields(c *collected) error { f.decl.Fields = append(f.decl.Fields, g.Field) } f.members = []encMember{{name: "Sealed", typeExpr: "encrypt.Ciphertext"}} + f.opaqueShape = lowerFirst(valueBaseName(f)) + "Opaque" return nil } @@ -559,6 +568,11 @@ func (r *reader) buildFields(c *collected) error { continue } g := genField{Field: field, typeExpr: r.typeExpr(cf.typ), via: cf.via, tags: cf.tags, pathType: pathType(cf.typ)} + if g.Sealed() && !g.GoType.Kind.Scalar() { + // The engine seals a composite as a tree of leaves, and a column + // holds one leaf; only an opaque struct seals a whole value. + return fieldErr(typeName, cf.goName, "a sealed field is one scalar (a string, number, bool or []byte, or a type defined over one); a %s seals only as part of an opaque struct", g.typeExpr) + } switch field.Verb { case VerbPassthrough: g.outputType = g.typeExpr @@ -774,15 +788,61 @@ func itOrEach(fields []string) string { return "each" } -// readableInOpaque reports whether gensupport.Get reads the type back out of -// an opened opaque value: a scalar, bytes, a slice of scalars, or a map from -// strings to scalars. A nested struct is not, yet. -func readableInOpaque(t GoType) bool { - switch t.Kind { - case KindString, KindBool, KindInt, KindUint, KindFloat, KindBytes: - return true - case KindSlice, KindMap: - return t.Elem != nil && t.Elem.Kind.Scalar() +// notJSONEncodable says why encoding/json cannot carry a type both ways, or +// "" when it can: every field of an opaque struct must, because the struct +// crosses the binding as one JSON document and comes back by decoding it +// into the generated shape struct. A type with its own MarshalJSON and +// UnmarshalJSON (time.Time) carries itself; otherwise a scalar, a slice or +// array of an encodable type, a map with string or integer keys, a pointer, +// or a struct whose every field is exported and encodable. A struct with an +// unexported field would come back without it, an interface or a channel +// not at all. +func notJSONEncodable(t types.Type, seen map[types.Type]bool) string { + if seen[t] { + return "" } - return false + seen[t] = true + defer delete(seen, t) + if hasMethod(types.NewPointer(t), "UnmarshalJSON") && hasMethod(t, "MarshalJSON") { + return "" + } + switch u := t.Underlying().(type) { + case *types.Basic: + if u.Info()&(types.IsBoolean|types.IsInteger|types.IsFloat|types.IsString) != 0 && u.Info()&types.IsUntyped == 0 { + return "" + } + return "is not a bool, integer, float or string" + case *types.Slice: + return notJSONEncodable(u.Elem(), seen) + case *types.Array: + return notJSONEncodable(u.Elem(), seen) + case *types.Pointer: + return notJSONEncodable(u.Elem(), seen) + case *types.Map: + if k, ok := u.Key().Underlying().(*types.Basic); !ok || k.Info()&(types.IsString|types.IsInteger) == 0 { + return "has a map key that is not a string or an integer" + } + return notJSONEncodable(u.Elem(), seen) + case *types.Struct: + for i := range u.NumFields() { + fld := u.Field(i) + if !fld.Exported() { + return fmt.Sprintf("has an unexported field %s that JSON would drop", fld.Name()) + } + if reason := notJSONEncodable(fld.Type(), seen); reason != "" { + return "has a field " + fld.Name() + " that " + reason + } + } + return "" + } + return "is not a type JSON carries" +} + +// valueBaseName is the plaintext type's bare name: User for User, *pb.Individual, crm.Contact. +func valueBaseName(f *genFile) string { + name := strings.TrimPrefix(f.typeExpr, "*") + if i := strings.LastIndexByte(name, '.'); i >= 0 { + name = name[i+1:] + } + return name } diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index a26d44c60..773dc011a 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -70,11 +70,16 @@ func TestRefusals(t *testing.T) { {"a _ field with no context", user("\t_ struct{}\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "_", "carries the context tag"}, {"an index that does not apply to the field type", user(ctx + "\tAge int32 `stash:\"age,encrypt,index=match\"`"), stashgen.Request{Type: "User"}, "Age", "match applies to a string, not to int32"}, {"an EQL type that does not apply to the field type", user(ctx + "\tAge int64 `stash:\"age,encrypt_into=TextEq\"`"), stashgen.Request{Type: "User"}, "Age", "TextEq seals a string, and int64 is int"}, - {"a field type the engine cannot seal", user(ctx + "\tDone chan int `stash:\"done,encrypt\"`"), stashgen.Request{Type: "User"}, "Done", "cannot seal a value of type chan int"}, - {"a struct with unexported fields the engine cannot seal", user(ctx + "\tAt time.Time `stash:\"at,encrypt\"`"), stashgen.Request{Type: "User"}, "At", "cannot seal a value of type time.Time"}, + {"a field type the engine cannot seal", user(ctx + "\tDone chan int `stash:\"done,encrypt\"`"), stashgen.Request{Type: "User"}, "Done", "seals only as part of an opaque struct"}, + {"a composite sealed outside an opaque struct", user(ctx + "\tAt time.Time `stash:\"at,encrypt\"`"), stashgen.Request{Type: "User"}, "At", "a time.Time seals only as part of an opaque struct"}, + {"a slice sealed outside an opaque struct", user(ctx + "\tTags []string `stash:\"tags,encrypt\"`"), stashgen.Request{Type: "User"}, "Tags", "seals only as part of an opaque struct"}, + {"an opaque struct with a channel", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tDone chan int"), stashgen.Request{Type: "User"}, "Done", "is not a type JSON carries"}, + {"an opaque struct with an interface", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tAny any"), stashgen.Request{Type: "User"}, "Any", "is not a type JSON carries"}, + {"an opaque struct with a struct JSON would truncate", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tTx hidden") + "\ntype hidden struct {\n\tA int\n\tb int\n}\n", stashgen.Request{Type: "User"}, "Tx", "JSON would drop"}, + {"an opaque struct with a map keyed by a struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tM map[hidden]int") + "\ntype hidden struct{ A int }\n", stashgen.Request{Type: "User"}, "M", "map key that is not a string or an integer"}, {"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "cannot produce the EQL type TextMatch"}, - {"an index on an equality-only kind", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "equality does not apply to map[string]string"}, - {"the json index, not in the engine yet", user(ctx + "\tAttrs map[string]string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"}, + {"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"}, + {"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"}, {"an index option the engine cannot carry", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "cannot carry index options"}, {"a passthrough field that has an index", user(ctx + "\tID int64 `stash:\"id,passthrough,index=equality\"`"), stashgen.Request{Type: "User"}, "ID", "passthrough field has no index"}, {"an embedded struct from another package with no tag", user(ctx + "\tgorm.Model\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Model", "cannot carry tags; tag the field"}, diff --git a/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden index e8f9267d8..e34e0e77c 100644 --- a/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden @@ -38,6 +38,14 @@ type documentShape struct { Tags []string } +// The opaque value as it crosses the binding: one JSON document of the +// struct's fields, by their declared names. +type documentOpaque struct { + Title string `json:"title"` + Body string `json:"body"` + Tags []string `json:"tags"` +} + var declaration = gensupport.DeclareOpaque("documents/v2/body") var codec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ @@ -45,10 +53,10 @@ var codec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ Declaration: declaration, PrintsPlaintext: true, Source: func(v Document) gensupport.Values { - return gensupport.Values{gensupport.OpaqueField: map[string]any{ - "title": v.Title, - "body": v.Body, - "tags": v.Tags, + return gensupport.Values{gensupport.OpaqueField: documentOpaque{ + Title: v.Title, + Body: v.Body, + Tags: v.Tags, }} }, Seal: func(rec gensupport.Record) (EncryptedDocument, error) { @@ -58,20 +66,14 @@ var codec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ return gensupport.Record{gensupport.OpaqueField: {Ciphertext: e.Sealed}} }, Value: func(e EncryptedDocument, vals gensupport.Values) (Document, error) { - fields, err := gensupport.Get[gensupport.Values](vals, gensupport.OpaqueField) - if err != nil { + var o documentOpaque + if err := gensupport.Opaque(vals, &o); err != nil { return Document{}, err } var v Document - if v.Title, err = gensupport.Get[string](fields, "title"); err != nil { - return Document{}, err - } - if v.Body, err = gensupport.Get[string](fields, "body"); err != nil { - return Document{}, err - } - if v.Tags, err = gensupport.Get[[]string](fields, "tags"); err != nil { - return Document{}, err - } + v.Title = o.Title + v.Body = o.Body + v.Tags = o.Tags return v, nil }, }) diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go index 0e1aa24a2..1df868db6 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go @@ -107,6 +107,9 @@ func Passthrough[T any](Record, string) (T, error) { var z T; return z, nil } // Get reads one value. func Get[T any](Values, string) (T, error) { var z T; return z, nil } +// Opaque reads an opened opaque value into the generated shape struct. +func Opaque[T any](Values, *T) error { return nil } + // Field is one sealed field's entry. type Field[T any] struct{ _ struct{} } From 711c08a23e8eaa5a8a7ccb14bdd93720c2a1b34a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:22:59 -0700 Subject: [PATCH 07/30] fix(golang): a term the engine cannot derive names its field and index MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's live job failed TestPlaintextDoesNotRemainInGuestMemoryAfterEncrypt with "term derivation failed": it encrypted a User with every field but Notes at its zero value, and the empty Email sits under a match index. Reproduced hermetically over the deterministic guest: the empty string and a separator-only string fail, a zero Age under ore and an empty Notes with no index do not. The engine is right to refuse. sem::match_terms defines no match term for text that yields no token — empty, separator-only, or shorter than the n-gram — because an empty term would match every row, and the typed Rust path refuses it the same way. Not a lowering bug; nothing changes under packages/stack-encrypt. What was wrong on the Go side is the message: the guest reports a status and nothing else. Cipher.Seal now asks the engine again on ErrTerm, one term at a time (a term derives locally, so this costs no key request), and wraps the error with the row, the field and the index: `value 1, field "email": the engine derives no match term for this value (a match index needs text with at least one token; ...)`. TestAMatchIndexNeedsText pins it for the record call and for the field entry, and pins that the other zero values seal and open. The live residency test encrypts a fully populated record. cmd/stashgen's README says what a match index needs (in the previous commit, with the README's other changes). Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/encrypt/live_test.go | 7 +++- languages/golang/encrypt/records.go | 40 ++++++++++++++++++++ languages/golang/encrypt/roundtrip_test.go | 43 ++++++++++++++++++++++ 3 files changed, 88 insertions(+), 2 deletions(-) diff --git a/languages/golang/encrypt/live_test.go b/languages/golang/encrypt/live_test.go index fdeb4ffaf..39c850369 100644 --- a/languages/golang/encrypt/live_test.go +++ b/languages/golang/encrypt/live_test.go @@ -120,11 +120,14 @@ func TestLiveForeignKeysetIsRefusedBeforeRetrieval(t *testing.T) { // Per-call hygiene on a real round trip: once Encrypt has returned, the // plaintext it was given is nowhere in guest memory — the staged input was // wiped by se_dealloc — so between calls the guest holds only the client -// key and its keyset cache. +// key and its keyset cache. The record is fully populated: a match index +// over an empty string has no token and the engine refuses it (see +// TestAMatchIndexNeedsText). func TestPlaintextDoesNotRemainInGuestMemoryAfterEncrypt(t *testing.T) { c := encrypt.LiveClient(t) const plaintext = "residency-probe-4111-b1c2d3e4f5" - if _, err := testusers.Encrypt(t.Context(), c.DefaultKeyset(), []testusers.User{{Notes: plaintext}}); err != nil { + probe := testusers.User{ID: 1, Age: 34, Email: "probe@example.com", Notes: plaintext} + if _, err := testusers.Encrypt(t.Context(), c.DefaultKeyset(), []testusers.User{probe}); err != nil { t.Fatalf("Encrypt: %v", err) } if n := bytes.Count(encrypt.GuestMemory(t, c), []byte(plaintext)); n != 0 { diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go index cbda5144a..1970c581d 100644 --- a/languages/golang/encrypt/records.go +++ b/languages/golang/encrypt/records.go @@ -58,6 +58,9 @@ func (cph *Cipher) Seal(ctx context.Context, plan *record.Plan, rows []record.So out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { return inst.call(ctx, inst.encryptRecord, buf(encodedSource), buf(encodedPlan), buf(opts)) }) + if errors.Is(err, ErrTerm) { + return nil, cph.locateTermFailure(ctx, p, rows, err) + } if err != nil { return nil, err } @@ -314,3 +317,40 @@ func (c *Client) open(ctx context.Context, sel KeysetSelector, p *record.Plan, r } return sources, nil } + +// locateTermFailure names the row, field and index behind an ErrTerm from a +// record call. The guest reports a status and nothing else, so the host asks +// the engine again, one term at a time, which costs no key request: a term +// derives locally. The engine defines no match term for text that yields no +// token (empty, separator-only, or shorter than the n-gram), because an +// empty term would match every row; a program hands such a field a value or +// drops the match index. +func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows []record.Source, err error) error { + for i, row := range rows { + for _, f := range p.Fields { + for _, o := range f.Outputs { + if !o.IsTerm() { + continue + } + if _, derr := cph.Derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) { + return fmt.Errorf("%w: value %d, field %q: the engine derives no %s term for this value (a match index needs text with at least one token; an empty or separator-only string has none)", err, i, f.Name, indexWord(o)) + } + } + } + } + return err +} + +func indexWord(o record.Output) string { + switch o { + case record.Equality: + return "equality" + case record.Match: + return "match" + case record.Ore: + return "ore" + case record.Ope: + return "ope" + } + return string(o) +} diff --git a/languages/golang/encrypt/roundtrip_test.go b/languages/golang/encrypt/roundtrip_test.go index 57ee85efb..5909dff02 100644 --- a/languages/golang/encrypt/roundtrip_test.go +++ b/languages/golang/encrypt/roundtrip_test.go @@ -256,3 +256,46 @@ func TestATamperedRecordDoesNotOpen(t *testing.T) { t.Fatalf("a missing ciphertext: %v", err) } } + +// A match index needs text that yields a token: the engine defines no match +// term for an empty or separator-only string, because an empty term would +// match every row, and the typed Rust path refuses it the same way. The Go +// error names the row, the field and the index. Every other zero value +// seals: a zero integer under ore and equality, an empty string with no +// match index. +func TestAMatchIndexNeedsText(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + for name, u := range map[string]testusers.User{ + "the zero value": {}, + "an empty email": {ID: 1, Age: 34, Notes: "x"}, + "a separator email": {ID: 1, Age: 34, Email: " \t", Notes: "x"}, + } { + _, err := testusers.Encrypt(ctx, cipher, []testusers.User{{ID: 9, Age: 1, Email: "ok@example.com", Notes: "x"}, u}) + if !errors.Is(err, encrypt.ErrTerm) { + t.Fatalf("%s: %v, want ErrTerm", name, err) + } + for _, want := range []string{`value 1`, `field "email"`, "no match term"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("%s: error %q does not name %s", name, err, want) + } + } + } + // The field entry says the same for one value. + if _, err := testusers.Fields.Email.Encrypt(ctx, cipher, ""); !errors.Is(err, encrypt.ErrTerm) || !strings.Contains(err.Error(), `field "email"`) { + t.Fatalf("Fields.Email.Encrypt(\"\"): %v", err) + } + if _, err := testusers.Fields.Email.Match(ctx, cipher, ""); !errors.Is(err, encrypt.ErrTerm) { + t.Fatalf("Fields.Email.Match(\"\"): %v", err) + } + // Zero values the engine does seal. + encrypted, err := testusers.Encrypt(ctx, cipher, []testusers.User{{Email: "zero@example.com"}}) + if err != nil { + t.Fatalf("a zero age and empty notes: %v", err) + } + back, err := testusers.Decrypt(ctx, cipher, encrypted) + if err != nil || back[0].Age != 0 || back[0].Notes != "" || back[0].ID != 0 { + t.Fatalf("zero values round trip: %v %+v", err, back) + } +} From 21995e6cfb3da70d21d08eaff3af53e6127d6aee Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:50:45 -0700 Subject: [PATCH 08/30] fix(golang): the deterministic test guest lives under testdata, out of the embed `//go:embed wasm` embeds the whole directory, so after `mise run wasm:guest:build:deterministic` every binary built from the tree carried the seed-only test build beside the real guest. Nothing loaded it, but a later read of guestFS by pattern could have. The task now writes it to encrypt/testdata, which go build and the embed both ignore; the tests read it from disk with os.ReadFile and skip when it is absent. The .gitignore entry, the CI artifact and checksum paths and the wasm README follow it. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- .github/workflows/tests-golang.yml | 20 ++++++++++---------- .gitignore | 4 ++++ languages/golang/encrypt/export_test.go | 11 +++++++---- languages/golang/encrypt/wasm/README.md | 9 +++------ mise.toml | 11 ++++++----- 5 files changed, 30 insertions(+), 25 deletions(-) diff --git a/.github/workflows/tests-golang.yml b/.github/workflows/tests-golang.yml index 73b8d8de8..db3025508 100644 --- a/.github/workflows/tests-golang.yml +++ b/.github/workflows/tests-golang.yml @@ -148,11 +148,11 @@ jobs: - name: stack-encrypt guest release build and import-surface gate run: mise run wasm:guest:build - # The deterministic-kms TEST build beside it: the Go tests open the - # record fixture Rust sealed through it, and run round trips with no - # ZeroKMS. The Go package never embeds it; its tests skip when it is - # absent, so a build that forgets this step would pass with less - # coverage, which is why the job builds it unconditionally. + # The deterministic-kms TEST build, under encrypt/testdata where no + # build or embed sees it: the Go tests open the record fixture Rust + # sealed through it, and run round trips with no ZeroKMS. The tests + # skip when it is absent, so a build that forgets this step would pass + # with less coverage, which is why the job builds it unconditionally. - name: stack-encrypt guest deterministic test build run: mise run wasm:guest:build:deterministic @@ -167,7 +167,7 @@ jobs: # same bytes rather than a stale or rebuilt guest. - name: Record the guests' checksums run: | - for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/testdata/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do (cd languages/golang && openssl dgst -sha256 "$guest" | awk '{print $NF}' > "$guest.sha256" && echo "$guest sha256 $(cat "$guest.sha256")") done @@ -178,8 +178,8 @@ jobs: path: | languages/golang/encrypt/wasm/stack_encrypt_guest.wasm languages/golang/encrypt/wasm/stack_encrypt_guest.wasm.sha256 - languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm - languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm.sha256 + languages/golang/encrypt/testdata/stack_encrypt_guest_deterministic.wasm + languages/golang/encrypt/testdata/stack_encrypt_guest_deterministic.wasm.sha256 languages/golang/auth/wasm/stack_auth_guest.wasm languages/golang/auth/wasm/stack_auth_guest.wasm.sha256 if-no-files-found: error @@ -283,7 +283,7 @@ jobs: - name: The guests are the ones Linux built and checked run: | cd languages/golang - for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/testdata/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do want=$(cat "$guest.sha256") got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}') if [ "$want" != "$got" ]; then @@ -349,7 +349,7 @@ jobs: - name: The guests are the ones the wasi-check job built and checked run: | cd languages/golang - for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/wasm/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do + for guest in encrypt/wasm/stack_encrypt_guest.wasm encrypt/testdata/stack_encrypt_guest_deterministic.wasm auth/wasm/stack_auth_guest.wasm; do want=$(cat "$guest.sha256") got=$(openssl dgst -sha256 "$guest" | awk '{print $NF}') if [ "$want" != "$got" ]; then diff --git a/.gitignore b/.gitignore index eccbed69f..b638c7cbb 100644 --- a/.gitignore +++ b/.gitignore @@ -112,3 +112,7 @@ languages/golang/encrypt/wasm/*.wasm languages/golang/auth/wasm/*.wasm languages/golang/encrypt/wasm/*.sha256 languages/golang/auth/wasm/*.sha256 +# The deterministic-kms TEST build of the stack-encrypt guest, from `mise run +# wasm:guest:build:deterministic`; under testdata so no binary embeds it. +languages/golang/encrypt/testdata/*.wasm +languages/golang/encrypt/testdata/*.sha256 diff --git a/languages/golang/encrypt/export_test.go b/languages/golang/encrypt/export_test.go index 3b2f418ea..e3322e275 100644 --- a/languages/golang/encrypt/export_test.go +++ b/languages/golang/encrypt/export_test.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "net/http" + "os" "testing" "github.com/cipherstash/stack/languages/golang/internal/guest" @@ -63,9 +64,11 @@ func GuestMemory(t *testing.T, c *Client) []byte { return append([]byte(nil), view...) } -// deterministicGuestPath is the deterministic-kms test build, beside the -// real guest; `mise run wasm:guest:build:deterministic` writes it. -const deterministicGuestPath = "wasm/stack_encrypt_guest_deterministic.wasm" +// deterministicGuestPath is the deterministic-kms test build, which `mise +// run wasm:guest:build:deterministic` writes under testdata: a directory +// `go build` and the package's `//go:embed wasm` ignore, so the test build +// is read from disk here and embedded in no binary. +const deterministicGuestPath = "testdata/stack_encrypt_guest_deterministic.wasm" // ErrDeterministicGuestNotBuilt says the test build is absent. var ErrDeterministicGuestNotBuilt = errors.New("encrypt: deterministic guest not built; run `mise run wasm:guest:build:deterministic`") @@ -75,7 +78,7 @@ var ErrDeterministicGuestNotBuilt = errors.New("encrypt: deterministic guest not // it opens what the Rust record fixture sealed under the same seed and // needs no ZeroKMS. ErrDeterministicGuestNotBuilt when the build is absent. func NewDeterministicClient(ctx context.Context, seed [32]byte) (*Client, error) { - wasm, err := guestFS.ReadFile(deterministicGuestPath) + wasm, err := os.ReadFile(deterministicGuestPath) if err != nil { return nil, ErrDeterministicGuestNotBuilt } diff --git a/languages/golang/encrypt/wasm/README.md b/languages/golang/encrypt/wasm/README.md index f6afc39d6..fa91f617d 100644 --- a/languages/golang/encrypt/wasm/README.md +++ b/languages/golang/encrypt/wasm/README.md @@ -5,9 +5,6 @@ the Go package embeds this directory and reports `ErrGuestNotBuilt` from `NewClient` when the module is absent, and its tests skip. -`stack_encrypt_guest_deterministic.wasm` is the same crate built with the -`deterministic-kms` feature by `mise run wasm:guest:build:deterministic`: a -TEST build whose keys derive from a seed, so the tests open the Rust record -fixture and run round trips with no ZeroKMS. The package never embeds it as -the guest a program runs; only the tests load it, and they skip when it is -absent. +The deterministic-kms TEST build of the same crate lives in `../testdata/` +(`mise run wasm:guest:build:deterministic`), not here: this directory is what +`//go:embed wasm` ships in every binary, and `go build` ignores `testdata`. diff --git a/mise.toml b/mise.toml index 86a8e1c15..80ccee9ab 100644 --- a/mise.toml +++ b/mise.toml @@ -186,9 +186,10 @@ echo "embedded into languages/golang/encrypt/wasm/stack_encrypt_guest.wasm" # The deterministic-kms TEST build of the same crate: `se_cipher_init` takes # the record fixture's 32-byte seed and derives every key from it, so the Go # tests open the records Rust sealed (packages/stack-encrypt/tests/fixtures/ -# record_lowering.json) and run hermetic round trips with no ZeroKMS. The Go -# package never embeds it: its tests look for it beside the real guest and -# skip when it is absent. The import-surface gate denies what the real +# record_lowering.json) and run hermetic round trips with no ZeroKMS. It goes +# under encrypt/testdata, which `go build` and the package's `//go:embed +# wasm` both ignore, so no binary ever carries it; the tests read it from +# disk and skip when it is absent. The import-surface gate denies what the real # build's does; it requires no transport import, because a build with no # ZeroKMS client has none. [tasks."wasm:guest:build:deterministic"] @@ -209,8 +210,8 @@ python3 "$root/scripts/check-wasm-imports.py" "$module" \\ --require wasi_snapshot_preview1:random_get # No transport imports to require: with no ZeroKMS client in the build the # linker drops the host module altogether, which is the point of the build. -cp "$module" ../wasm/stack_encrypt_guest_deterministic.wasm -echo "test build at languages/golang/encrypt/wasm/stack_encrypt_guest_deterministic.wasm" +cp "$module" ../testdata/stack_encrypt_guest_deterministic.wasm +echo "test build at languages/golang/encrypt/testdata/stack_encrypt_guest_deterministic.wasm" """ [tasks."wasm:guest:test"] From 4573a055d20161b12b0a8e02b5477b4bef5f637a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:50:46 -0700 Subject: [PATCH 09/30] fix(golang): Get reads every type the generator accepts, and a model round-trips A reviewer generated a field of a defined type (type Status string) and could not decrypt it: convert named only the built-in types, so a *Status, a time.Duration, a []int or a map[string]string reached its default arm. Get now falls through to convertVia, the one place this package uses reflection: a defined scalar type is read as its underlying type and converted, a slice or array element by element, a map with string keys entry by entry, each through convert and its range checks. Generated code stays reflection-free. A JSON number widens to its family's widest type first, so the same reader serves an opaque document's fields. TestGetReadsEveryTypeTheGeneratorAccepts holds the reviewer's cases and more; testusers.Kinds gains a Status and a time.Duration field, Everything gains []int, []float32, Status and time.Duration, and both round-trip through generated code over the deterministic guest. testusers.User gains -model Rows=UserRow and TestModelRowsRoundTrip proves each term lands in its own column and the rows decrypt back; nothing ran Records before. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- .../golang/encrypt/gensupport/convert.go | 159 +++++++++++++++++- .../golang/encrypt/gensupport/declaration.go | 8 +- .../gensupport/gensupport_internal_test.go | 82 ++++++++- .../internal/testusers/everything_stash.go | 16 ++ .../encrypt/internal/testusers/kinds.go | 51 +++--- .../encrypt/internal/testusers/kinds_stash.go | 78 ++++++++- .../encrypt/internal/testusers/user_stash.go | 45 +++++ .../encrypt/internal/testusers/users.go | 18 +- languages/golang/encrypt/kinds_test.go | 44 ++++- 9 files changed, 467 insertions(+), 34 deletions(-) diff --git a/languages/golang/encrypt/gensupport/convert.go b/languages/golang/encrypt/gensupport/convert.go index 4a4de4748..dfc3819d0 100644 --- a/languages/golang/encrypt/gensupport/convert.go +++ b/languages/golang/encrypt/gensupport/convert.go @@ -1,8 +1,11 @@ package gensupport import ( + "encoding/json" "fmt" "math" + "reflect" + "strconv" "github.com/cipherstash/vitaminc/bindings/go/vcvalue" ) @@ -13,7 +16,21 @@ import ( // bool, or a vcvalue.Object for a composite — and the struct's type is in // the same family, narrower at most. A value outside the target's range, or // of another family, is an error. +// +// The switch names the built-in types; a type defined over one (type Status +// string, time.Duration), or a slice or map of any readable type, is read +// through its underlying type by [convertVia], the one place this package +// uses reflection. Generated code uses none: it hands Get the field's type +// and the engine's value, and reads a value back. func convert(v any, out any) error { + // A JSON number — what an opaque document carries — widens to its + // family's widest type, and the family's range check applies below. + if n, ok := v.(json.Number); ok { + var err error + if v, err = widenNumber(n, out); err != nil { + return err + } + } // A nil slice or map comes back as nil: the zero value it was. if v == nil { switch out.(type) { @@ -119,11 +136,151 @@ func convert(v any, out any) error { case *any: *out = v default: - return fmt.Errorf("the opened value is a %T, which this field's type %T cannot hold", v, out) + return convertVia(v, out) } return nil } +// convertVia reads a value into a type the switch in convert does not name: +// a type defined over a scalar is read as its underlying type and converted; +// a slice or array element by element; a map with string keys entry by +// entry, each value through convert. Anything else is a mismatch. +func convertVia(v any, out any) error { + target := reflect.ValueOf(out) + if target.Kind() != reflect.Pointer || target.IsNil() { + return fmt.Errorf("the opened value is a %T, which %T cannot hold", v, out) + } + elem := target.Elem() + t := elem.Type() + switch t.Kind() { + case reflect.Bool, reflect.String, + reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, + reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, + reflect.Float32, reflect.Float64: + // A defined type: read the underlying type, then convert. + under := reflect.New(underlying(t)) + if err := convert(v, under.Interface()); err != nil { + return err + } + elem.Set(under.Elem().Convert(t)) + return nil + case reflect.Slice: + if t.Elem().Kind() == reflect.Uint8 { + var b []byte + if err := convert(v, &b); err != nil { + return err + } + elem.Set(reflect.ValueOf(b).Convert(t)) + return nil + } + items, ok := v.([]any) + if !ok { + return fmt.Errorf("opened as %T, not %s", v, t) + } + result := reflect.MakeSlice(t, len(items), len(items)) + for i, item := range items { + if err := convert(item, result.Index(i).Addr().Interface()); err != nil { + return fmt.Errorf("element %d: %w", i, err) + } + } + elem.Set(result) + return nil + case reflect.Array: + items, ok := v.([]any) + if !ok || len(items) != t.Len() { + return fmt.Errorf("opened as %T with %d elements, not %s", v, len(items), t) + } + result := reflect.New(t).Elem() + for i, item := range items { + if err := convert(item, result.Index(i).Addr().Interface()); err != nil { + return fmt.Errorf("element %d: %w", i, err) + } + } + elem.Set(result) + return nil + case reflect.Map: + if t.Key().Kind() != reflect.String { + return fmt.Errorf("%s has a key that is not a string", t) + } + vals, err := valuesOf(v) + if err != nil { + return err + } + result := reflect.MakeMapWithSize(t, len(vals)) + for key, item := range vals { + slot := reflect.New(t.Elem()) + if err := convert(item, slot.Interface()); err != nil { + return fmt.Errorf("entry %q: %w", key, err) + } + result.SetMapIndex(reflect.ValueOf(key).Convert(t.Key()), slot.Elem()) + } + elem.Set(result) + return nil + } + return fmt.Errorf("the opened value is a %T, which this field's type %s cannot hold", v, t) +} + +// underlying is the built-in type a defined scalar type is declared over. +func underlying(t reflect.Type) reflect.Type { + switch t.Kind() { + case reflect.Bool: + return reflect.TypeFor[bool]() + case reflect.String: + return reflect.TypeFor[string]() + case reflect.Int: + return reflect.TypeFor[int]() + case reflect.Int8: + return reflect.TypeFor[int8]() + case reflect.Int16: + return reflect.TypeFor[int16]() + case reflect.Int32: + return reflect.TypeFor[int32]() + case reflect.Int64: + return reflect.TypeFor[int64]() + case reflect.Uint: + return reflect.TypeFor[uint]() + case reflect.Uint8: + return reflect.TypeFor[uint8]() + case reflect.Uint16: + return reflect.TypeFor[uint16]() + case reflect.Uint32: + return reflect.TypeFor[uint32]() + case reflect.Uint64: + return reflect.TypeFor[uint64]() + case reflect.Float32: + return reflect.TypeFor[float32]() + } + return reflect.TypeFor[float64]() +} + +// widenNumber reads a JSON number as the widest value of the target's +// family: int64 for a signed target, uint64 for an unsigned one, float64 for +// a float. The family conversion then applies its range check. +func widenNumber(n json.Number, out any) (any, error) { + kind := reflect.TypeOf(out).Elem().Kind() + switch kind { + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + i, err := strconv.ParseInt(string(n), 10, 64) + if err != nil { + return nil, fmt.Errorf("%s is not an integer that fits an int64", n) + } + return i, nil + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + u, err := strconv.ParseUint(string(n), 10, 64) + if err != nil { + return nil, fmt.Errorf("%s is not an integer that fits a uint64", n) + } + return u, nil + case reflect.Float32, reflect.Float64, reflect.Interface: + f, err := n.Float64() + if err != nil { + return nil, err + } + return f, nil + } + return n, nil +} + func mismatch(v, want any) error { return fmt.Errorf("opened as %T, not %T", v, want) } diff --git a/languages/golang/encrypt/gensupport/declaration.go b/languages/golang/encrypt/gensupport/declaration.go index ba835ec62..be71e6e6d 100644 --- a/languages/golang/encrypt/gensupport/declaration.go +++ b/languages/golang/encrypt/gensupport/declaration.go @@ -10,13 +10,15 @@ import ( // Kind is a field's wire type: the data form of the Rust chain's `::`, // chosen by stashgen from the field's Go type. A String or UInt32 field -// seals as the typed leaf a Rust record derives; an Untyped field (a struct, -// a slice, a map) seals as one self-describing value. +// seals as the typed leaf a Rust record derives. Every sealed field has a +// scalar kind: stashgen refuses a struct, slice or map outside an opaque +// struct, and an opaque struct seals as Bytes (one JSON document). Untyped +// names a field with no declared type and is not what generated code writes. type Kind string // The kinds. int8, int16 and int32 are Int32; int and int64 are Int64; // uint8, uint16 and uint32 are UInt32; uint and uint64 are UInt64; []byte is -// Bytes. Everything else is Untyped. +// Bytes. A type defined over one of these has its underlying kind. const ( Untyped Kind = "" Bool Kind = "bool" diff --git a/languages/golang/encrypt/gensupport/gensupport_internal_test.go b/languages/golang/encrypt/gensupport/gensupport_internal_test.go index 53fcbe9de..40b028f3d 100644 --- a/languages/golang/encrypt/gensupport/gensupport_internal_test.go +++ b/languages/golang/encrypt/gensupport/gensupport_internal_test.go @@ -1,6 +1,8 @@ package gensupport import ( + "encoding/json" + "fmt" "reflect" "strings" "testing" @@ -167,8 +169,10 @@ func TestConvertStaysWithinAFamily(t *testing.T) { if err != nil || gotDefined != "x" { t.Fatalf("Get[defined] = %v %v", gotDefined, err) } - if _, err := Get[defined](Values{"d": "x"}, "d"); err == nil { - t.Fatal("Get converted a wire string into a defined type; the generated code does that") + // A defined type is read at its underlying type when the value is the + // engine's wire value. + if got, err := Get[defined](Values{"d": "x"}, "d"); err != nil || got != "x" { + t.Fatalf("Get[defined] from the wire = %v %v", got, err) } got, err := Get[uint8](Values{"age": uint32(3)}, "age") if err != nil || got != 3 { @@ -266,3 +270,77 @@ func TestNewReportsAnIncompleteFileAndABadDeclarationOnFirstUse(t *testing.T) { t.Fatal("a nil decrypter was accepted") } } + +type status string + +// Every type the generator accepts on a sealed field, and every type JSON +// hands back for an opaque one, reads through Get. The reviewer's cases. +func TestGetReadsEveryTypeTheGeneratorAccepts(t *testing.T) { + type labels map[string]string + cases := map[string]func() error{ + "named string": func() error { v, err := Get[status](Values{"v": "active"}, "v"); return check(err, v == "active") }, + "named int64": func() error { v, err := Get[time.Duration](Values{"v": int64(5)}, "v"); return check(err, v == 5) }, + "named int16": func() error { v, err := Get[Score](Values{"v": int32(-3)}, "v"); return check(err, v == -3) }, + "named []byte": func() error { v, err := Get[Blob](Values{"v": []byte{1}}, "v"); return check(err, len(v) == 1) }, + "[]int": func() error { + v, err := Get[[]int](Values{"v": []any{json.Number("1")}}, "v") + return check(err, len(v) == 1 && v[0] == 1) + }, + "[]int from wire": func() error { v, err := Get[[]int](Values{"v": []any{int64(2)}}, "v"); return check(err, v[0] == 2) }, + "[]float32": func() error { + v, err := Get[[]float32](Values{"v": []any{json.Number("1.5")}}, "v") + return check(err, v[0] == 1.5) + }, + "[]status": func() error { v, err := Get[[]status](Values{"v": []any{"a"}}, "v"); return check(err, v[0] == "a") }, + "[2]uint8": func() error { + v, err := Get[[2]uint8](Values{"v": []any{json.Number("9"), json.Number("8")}}, "v") + return check(err, v == [2]uint8{9, 8}) + }, + "map[string]string": func() error { + v, err := Get[map[string]string](Values{"v": map[string]any{"a": "b"}}, "v") + return check(err, v["a"] == "b") + }, + "named map": func() error { + v, err := Get[labels](Values{"v": map[string]any{"a": "b"}}, "v") + return check(err, v["a"] == "b") + }, + "map[string]int": func() error { + v, err := Get[map[string]int](Values{"v": vcvalue.Object{{Key: "a", Value: int64(3)}}}, "v") + return check(err, v["a"] == 3) + }, + "exact passthrough": func() error { + v, err := Get[time.Time](Values{"v": time.Unix(1, 0)}, "v") + return check(err, v.Unix() == 1) + }, + } + for name, get := range cases { + if err := get(); err != nil { + t.Errorf("%s: %v", name, err) + } + } + // And the refusals stay refusals. + if _, err := Get[status](Values{"v": int64(1)}, "v"); err == nil { + t.Error("an integer became a named string") + } + if _, err := Get[[]int8](Values{"v": []any{int64(300)}}, "v"); err == nil { + t.Error("300 fit an int8 element") + } + if _, err := Get[map[int]string](Values{"v": map[string]any{"a": "b"}}, "v"); err == nil { + t.Error("a map with integer keys was read from a wire object") + } +} + +type ( + Score int16 + Blob []byte +) + +func check(err error, ok bool) error { + if err != nil { + return err + } + if !ok { + return fmt.Errorf("wrong value") + } + return nil +} diff --git a/languages/golang/encrypt/internal/testusers/everything_stash.go b/languages/golang/encrypt/internal/testusers/everything_stash.go index deed845dd..2caf2f81a 100644 --- a/languages/golang/encrypt/internal/testusers/everything_stash.go +++ b/languages/golang/encrypt/internal/testusers/everything_stash.go @@ -47,6 +47,10 @@ type everythingShape struct { Bl Blob Sc Score Tags []string + Ints []int + F32s []float32 + St Status + D time.Duration Emails []Email Counts map[string]int Labels map[string]string @@ -76,6 +80,10 @@ type everythingOpaque struct { Bl Blob `json:"bl"` Sc Score `json:"sc"` Tags []string `json:"tags"` + Ints []int `json:"ints"` + F32s []float32 `json:"f32s"` + St Status `json:"st"` + D time.Duration `json:"d"` Emails []Email `json:"emails"` Counts map[string]int `json:"counts"` Labels map[string]string `json:"labels"` @@ -110,6 +118,10 @@ var everythingCodec = gensupport.New(gensupport.Generated[Everything, EncryptedE Bl: v.Bl, Sc: v.Sc, Tags: v.Tags, + Ints: v.Ints, + F32s: v.F32s, + St: v.St, + D: v.D, Emails: v.Emails, Counts: v.Counts, Labels: v.Labels, @@ -148,6 +160,10 @@ var everythingCodec = gensupport.New(gensupport.Generated[Everything, EncryptedE v.Bl = o.Bl v.Sc = o.Sc v.Tags = o.Tags + v.Ints = o.Ints + v.F32s = o.F32s + v.St = o.St + v.D = o.D v.Emails = o.Emails v.Counts = o.Counts v.Labels = o.Labels diff --git a/languages/golang/encrypt/internal/testusers/kinds.go b/languages/golang/encrypt/internal/testusers/kinds.go index 5827ebde6..e2429366a 100644 --- a/languages/golang/encrypt/internal/testusers/kinds.go +++ b/languages/golang/encrypt/internal/testusers/kinds.go @@ -8,9 +8,10 @@ import ( // Defined types over scalars: the engine returns the underlying type and the // generated code converts. type ( - Email string - Blob []byte - Score int16 + Email string + Blob []byte + Score int16 + Status string ) //go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Account -name Account @@ -39,25 +40,27 @@ type Inner struct { // an opaque struct, with defined types among them, and passthrough fields of // types the codec cannot carry. Everything it accepts round-trips. type Kinds struct { - _ struct{} `stash:"context=kinds"` - S string `stash:"s,encrypt,index=equality;match"` - E Email `stash:"e,encrypt,index=equality;match;ore"` - Bo bool `stash:"bo,encrypt"` - I8 int8 `stash:"i8,encrypt,index=equality;ore"` - I16 int16 `stash:"i16,encrypt,index=ope"` - I32 int32 `stash:"i32,encrypt,index=equality"` - I int `stash:"i,encrypt,index=ore"` - I64 int64 `stash:"i64,encrypt"` - U8 uint8 `stash:"u8,encrypt,index=equality"` - U16 uint16 `stash:"u16,encrypt"` - U32 uint32 `stash:"u32,encrypt,index=equality;ore"` - U uint `stash:"u,encrypt"` - U64 uint64 `stash:"u64,encrypt,index=ope"` - F32 float32 `stash:"f32,encrypt"` - F64 float64 `stash:"f64,encrypt,index=ore"` - By []byte `stash:"by,encrypt,index=equality"` - Bl Blob `stash:"bl,encrypt"` - Sc Score `stash:"sc,encrypt,index=ore"` + _ struct{} `stash:"context=kinds"` + S string `stash:"s,encrypt,index=equality;match"` + E Email `stash:"e,encrypt,index=equality;match;ore"` + Bo bool `stash:"bo,encrypt"` + I8 int8 `stash:"i8,encrypt,index=equality;ore"` + I16 int16 `stash:"i16,encrypt,index=ope"` + I32 int32 `stash:"i32,encrypt,index=equality"` + I int `stash:"i,encrypt,index=ore"` + I64 int64 `stash:"i64,encrypt"` + U8 uint8 `stash:"u8,encrypt,index=equality"` + U16 uint16 `stash:"u16,encrypt"` + U32 uint32 `stash:"u32,encrypt,index=equality;ore"` + U uint `stash:"u,encrypt"` + U64 uint64 `stash:"u64,encrypt,index=ope"` + F32 float32 `stash:"f32,encrypt"` + F64 float64 `stash:"f64,encrypt,index=ore"` + By []byte `stash:"by,encrypt,index=equality"` + Bl Blob `stash:"bl,encrypt"` + Sc Score `stash:"sc,encrypt,index=ore"` + St Status `stash:"st,encrypt,index=equality"` + D time.Duration `stash:"d,encrypt,index=ore"` P *int `stash:"p,passthrough"` M map[string]string `stash:"m,passthrough"` @@ -85,6 +88,10 @@ type Everything struct { Bl Blob Sc Score Tags []string + Ints []int + F32s []float32 + St Status + D time.Duration Emails []Email Counts map[string]int Labels map[string]string diff --git a/languages/golang/encrypt/internal/testusers/kinds_stash.go b/languages/golang/encrypt/internal/testusers/kinds_stash.go index 9270e3316..228d7cb84 100644 --- a/languages/golang/encrypt/internal/testusers/kinds_stash.go +++ b/languages/golang/encrypt/internal/testusers/kinds_stash.go @@ -37,6 +37,8 @@ type EncryptedKinds struct { By EncryptedKindsBy Bl EncryptedKindsBl Sc EncryptedKindsSc + St EncryptedKindsSt + D EncryptedKindsD P *int M map[string]string T time.Time @@ -134,12 +136,22 @@ type EncryptedKindsSc struct { Ore encrypt.OreTerm } +type EncryptedKindsSt struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +type EncryptedKindsD struct { + Ciphertext encrypt.Ciphertext + Ore encrypt.OreTerm +} + func (e EncryptedKinds) String() string { - return gensupport.Redacted("EncryptedKinds", map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc") + return gensupport.Redacted("EncryptedKinds", map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc", "St", "D") } func (e EncryptedKinds) LogValue() slog.Value { - return gensupport.RedactedLog(map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc") + return gensupport.RedactedLog(map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc", "St", "D") } // Stops compiling when Kinds gains, loses, reorders or retypes a field. @@ -165,6 +177,8 @@ type kindsShape struct { By []byte Bl Blob Sc Score + St Status + D time.Duration P *int M map[string]string T time.Time @@ -192,6 +206,8 @@ var kindsDeclaration = gensupport.Declare("kinds"). EncryptIndex("by", gensupport.Bytes, encrypt.Equality). Encrypt("bl", gensupport.Bytes). EncryptIndex("sc", gensupport.Int32, encrypt.Ore). + EncryptIndex("st", gensupport.String, encrypt.Equality). + EncryptIndex("d", gensupport.Int64, encrypt.Ore). Passthrough("p"). Passthrough("m"). Passthrough("t"). @@ -223,6 +239,8 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ "by": v.By, "bl": v.Bl, "sc": v.Sc, + "st": v.St, + "d": v.D, "p": v.P, "m": v.M, "t": v.T, @@ -311,6 +329,14 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ Ciphertext: rec["sc"].Ciphertext, Ore: rec["sc"].Ore, } + e.St = EncryptedKindsSt{ + Ciphertext: rec["st"].Ciphertext, + Equality: rec["st"].Equality, + } + e.D = EncryptedKindsD{ + Ciphertext: rec["d"].Ciphertext, + Ore: rec["d"].Ore, + } return e, nil }, Open: func(e EncryptedKinds) gensupport.Record { @@ -333,6 +359,8 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ "by": {Ciphertext: e.By.Ciphertext, Equality: e.By.Equality}, "bl": {Ciphertext: e.Bl.Ciphertext}, "sc": {Ciphertext: e.Sc.Ciphertext, Ore: e.Sc.Ore}, + "st": {Ciphertext: e.St.Ciphertext, Equality: e.St.Equality}, + "d": {Ciphertext: e.D.Ciphertext, Ore: e.D.Ore}, "p": {Value: e.P}, "m": {Value: e.M}, "t": {Value: e.T}, @@ -404,6 +432,16 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ return Kinds{}, err } v.Sc = Score(rawSc) + rawSt, err := gensupport.Get[string](vals, "st") + if err != nil { + return Kinds{}, err + } + v.St = Status(rawSt) + rawD, err := gensupport.Get[int64](vals, "d") + if err != nil { + return Kinds{}, err + } + v.D = time.Duration(rawD) if v.P, err = gensupport.Get[*int](vals, "p"); err != nil { return Kinds{}, err } @@ -456,6 +494,8 @@ var KindsFields = struct { By KindsByField Bl KindsBlField Sc KindsScField + St KindsStField + D KindsDField }{ S: KindsSField{gensupport.NewField[string](kindsDeclaration, "s")}, E: KindsEField{gensupport.NewField[Email](kindsDeclaration, "e")}, @@ -475,6 +515,8 @@ var KindsFields = struct { By: KindsByField{gensupport.NewField[[]byte](kindsDeclaration, "by")}, Bl: KindsBlField{gensupport.NewField[Blob](kindsDeclaration, "bl")}, Sc: KindsScField{gensupport.NewField[Score](kindsDeclaration, "sc")}, + St: KindsStField{gensupport.NewField[Status](kindsDeclaration, "st")}, + D: KindsDField{gensupport.NewField[time.Duration](kindsDeclaration, "d")}, } type KindsSField struct { @@ -742,3 +784,35 @@ func (f KindsScField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Score) ( func (f KindsScField) Ore(ctx context.Context, c *encrypt.Cipher, v Score) (encrypt.OreTerm, error) { return f.field.Ore(ctx, c, v) } + +type KindsStField struct { + field gensupport.Field[Status] +} + +func (f KindsStField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Status) (EncryptedKindsSt, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsSt{}, err + } + return EncryptedKindsSt{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f KindsStField) Equality(ctx context.Context, c *encrypt.Cipher, v Status) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} + +type KindsDField struct { + field gensupport.Field[time.Duration] +} + +func (f KindsDField) Encrypt(ctx context.Context, c *encrypt.Cipher, v time.Duration) (EncryptedKindsD, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedKindsD{}, err + } + return EncryptedKindsD{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil +} + +func (f KindsDField) Ore(ctx context.Context, c *encrypt.Cipher, v time.Duration) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} diff --git a/languages/golang/encrypt/internal/testusers/user_stash.go b/languages/golang/encrypt/internal/testusers/user_stash.go index 14b0a3567..ea5f8d14a 100644 --- a/languages/golang/encrypt/internal/testusers/user_stash.go +++ b/languages/golang/encrypt/internal/testusers/user_stash.go @@ -193,3 +193,48 @@ func (f NotesField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E out, err := f.field.Encrypt(ctx, c, v) return EncryptedUserNotes{Ciphertext: out.Ciphertext}, err } + +// The -model flag: UserRow converts to and from its shape only while the +// two have the same fields, with the same types, in the same order. +type rowsShape struct { + ID int64 + Age encrypt.Ciphertext + AgeEq encrypt.EqualityTerm + AgeOre encrypt.OreTerm + Email encrypt.Ciphertext + EmailEq encrypt.EqualityTerm + EmailMatch encrypt.MatchTerm + Notes encrypt.Ciphertext +} + +var rowsCodec = gensupport.Records(codec, + func(e EncryptedUser) UserRow { + return UserRow(rowsShape{ + ID: e.ID, + Age: e.Age.Ciphertext, + AgeEq: e.Age.Equality, + AgeOre: e.Age.Ore, + Email: e.Email.Ciphertext, + EmailEq: e.Email.Equality, + EmailMatch: e.Email.Match, + Notes: e.Notes.Ciphertext, + }) + }, + func(r UserRow) EncryptedUser { + s := rowsShape(r) + return EncryptedUser{ + ID: s.ID, + Age: EncryptedUserAge{Ciphertext: s.Age, Equality: s.AgeEq, Ore: s.AgeOre}, + Email: EncryptedUserEmail{Ciphertext: s.Email, Equality: s.EmailEq, Match: s.EmailMatch}, + Notes: EncryptedUserNotes{Ciphertext: s.Notes}, + } + }, +) + +func EncryptRows(ctx context.Context, cipher *encrypt.Cipher, testusers []User) ([]UserRow, error) { + return rowsCodec.Encrypt(ctx, cipher, testusers) +} + +func DecryptRows(ctx context.Context, d encrypt.Decrypter, rows []UserRow) ([]User, error) { + return rowsCodec.Decrypt(ctx, d, rows) +} diff --git a/languages/golang/encrypt/internal/testusers/users.go b/languages/golang/encrypt/internal/testusers/users.go index 063e97f07..2ab1d2780 100644 --- a/languages/golang/encrypt/internal/testusers/users.go +++ b/languages/golang/encrypt/internal/testusers/users.go @@ -6,7 +6,9 @@ // changes a file. package testusers -//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type User +import "github.com/cipherstash/stack/languages/golang/encrypt" + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type User -model Rows=UserRow // User is the record of the Rust record fixture // (packages/stack-encrypt/tests/fixtures/record_lowering.json): the same @@ -21,6 +23,20 @@ type User struct { Internal string `stash:"-"` } +// UserRow is a model for User in separate columns: one field for each +// output, each tagged with the output it holds, for a library that maps one +// struct field to one column. +type UserRow struct { + ID int64 `stash:"id"` + Age encrypt.Ciphertext `stash:"age"` + AgeEq encrypt.EqualityTerm `stash:"age,equality"` + AgeOre encrypt.OreTerm `stash:"age,ore"` + Email encrypt.Ciphertext `stash:"email"` + EmailEq encrypt.EqualityTerm `stash:"email,equality"` + EmailMatch encrypt.MatchTerm `stash:"email,match"` + Notes encrypt.Ciphertext `stash:"notes"` +} + //go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Document -name Document // Document is sealed as one value. diff --git a/languages/golang/encrypt/kinds_test.go b/languages/golang/encrypt/kinds_test.go index 8718b48a3..5b610bef0 100644 --- a/languages/golang/encrypt/kinds_test.go +++ b/languages/golang/encrypt/kinds_test.go @@ -1,6 +1,7 @@ package encrypt_test import ( + "bytes" "context" "database/sql" "reflect" @@ -51,13 +52,13 @@ func TestEverySealedKindRoundTrips(t *testing.T) { I8: -8, I16: -16, I32: -32, I: -64, I64: -1 << 40, U8: 8, U16: 16, U32: 32, U: 64, U64: 1 << 40, F32: 1.5, F64: -2.25, - By: []byte{1, 2, 3}, Bl: testusers.Blob{4, 5}, Sc: -3, + By: []byte{1, 2, 3}, Bl: testusers.Blob{4, 5}, Sc: -3, St: "active", D: 90 * time.Second, P: &seven, M: map[string]string{"k": "v"}, T: when, N: sql.NullTime{Time: when, Valid: true}, In: testusers.Inner{Name: "in", N: 1}, Sk: []testusers.Inner{{Name: "a", N: 2}}, }, { S: "bob", E: "bob@example.com", I8: 127, I16: 32767, I32: 1<<31 - 1, I: 1 << 30, I64: 1<<63 - 1, U8: 255, U16: 65535, U32: 1<<32 - 1, U: 1 << 30, U64: 1<<64 - 1, - F32: -0.5, F64: 1e300, By: []byte{}, Bl: testusers.Blob{}, Sc: 32767, + F32: -0.5, F64: 1e300, By: []byte{}, Bl: testusers.Blob{}, Sc: 32767, St: "x", D: -time.Minute, M: map[string]string{}, Sk: []testusers.Inner{}, }} encrypted, err := testusers.EncryptKinds(ctx, cipher, in) @@ -85,6 +86,43 @@ func TestEverySealedKindRoundTrips(t *testing.T) { if err != nil || len(one.Ciphertext) == 0 { t.Fatalf("Score field: %v", err) } + if probe, err := testusers.KindsFields.St.Equality(ctx, cipher, "active"); err != nil || !probe.Equal(encrypted[0].St.Equality) { + t.Fatalf("Status probe: %v", err) + } +} + +// A model in separate columns: each term is in its own column, and the rows +// decrypt back to the input. The reviewer's test. +func TestModelRowsRoundTrip(t *testing.T) { + c := deterministicClient(t) + cipher := c.DefaultKeyset() + rows, err := testusers.EncryptRows(t.Context(), cipher, people) + if err != nil { + t.Fatal(err) + } + probe, err := testusers.Fields.Email.Equality(t.Context(), cipher, people[1].Email) + if err != nil || !probe.Equal(rows[1].EmailEq) || probe.Equal(rows[0].EmailEq) { + t.Fatalf("EmailEq does not hold the email's equality term: %v", err) + } + ageProbe, err := testusers.Fields.Age.Ore(t.Context(), cipher, people[1].Age) + if err != nil || ageProbe.Compare(rows[1].AgeOre) != 0 || !rows[1].AgeOre.Less(rows[0].AgeOre) { + t.Fatalf("AgeOre does not hold the age's ORE term: %v", err) + } + match, err := testusers.Fields.Email.Match(t.Context(), cipher, people[1].Email) + if err != nil || !bytes.Equal(match, rows[1].EmailMatch) || len(rows[1].Notes) == 0 || rows[1].ID != people[1].ID { + t.Fatalf("the other columns: %v %+v", err, rows[1]) + } + want := append([]testusers.User(nil), people...) + want[0].Internal = "" + back, err := testusers.DecryptRows(t.Context(), c, rows) + if err != nil || !reflect.DeepEqual(back, want) { + t.Fatalf("DecryptRows = %+v, %v", back, err) + } + // A row whose columns were swapped does not open as the input. + rows[0].Age, rows[0].Notes = rows[0].Notes, rows[0].Age + if _, err := testusers.DecryptRows(t.Context(), c, rows[:1]); err == nil { + t.Fatal("swapped columns opened") + } } func TestEveryOpaqueFieldTypeRoundTrips(t *testing.T) { @@ -96,7 +134,7 @@ func TestEveryOpaqueFieldTypeRoundTrips(t *testing.T) { in := []testusers.Everything{{ S: "s", E: "e@example.com", Bo: true, I8: -1, I: 42, U64: 1 << 40, F32: 0.25, F64: 9.75, By: []byte("bytes"), Bl: testusers.Blob("blob"), Sc: -7, - Tags: []string{"a", "b"}, Emails: []testusers.Email{"x@y"}, Counts: map[string]int{"a": 1}, Labels: map[string]string{"k": "v"}, ByKey: map[int]string{3: "three"}, + Tags: []string{"a", "b"}, Ints: []int{1, -2}, F32s: []float32{1.5, -0.25}, St: "active", D: time.Hour, Emails: []testusers.Email{"x@y"}, Counts: map[string]int{"a": 1}, Labels: map[string]string{"k": "v"}, ByKey: map[int]string{3: "three"}, In: testusers.Inner{Name: "in", N: 5}, Ins: []testusers.Inner{{Name: "i", N: 6}}, P: &hello, PI: &testusers.Inner{Name: "pi", N: 7}, T: when, N: sql.NullTime{Time: when, Valid: true}, Nested: [][]byte{{1}, {2, 3}}, Arr: [2]uint8{9, 8}, }, { From cf04a94e5c4c4230ba171b0f31f45a22313920d8 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:50:46 -0700 Subject: [PATCH 10/30] fix(golang): se_targets entries are read whole against the agreed wire keys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Go decoder was the only definition of an se_targets entry and kept a zero value for an unknown key or a mistyped value. The entry shape is now the one eql-bindings serialises — name, family, suffix, plaintext (a ValueKind name or null), sql_domain, indexes, query, query_sql_domain, producible, reason — documented as wire format in the guest's targets rustdoc and read by parseTargets, a function that returns ErrInternal on an unknown key, a repeated key, a missing name, indexes or producible, or a value of the wrong type. record.Target carries those fields; stashgen's EQLType carries Producible and Reason and the reader refuses a type the engine lists but does not produce, with the engine's reason. TestParseTargetsReadsEachEntry drives it with a two-entry list and every malformed shape. The fake engine's equality rule now matches dynamic::admits: integers, text and bytes, not floats or booleans. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/encrypt/checker.go | 116 ++++++++++++++---- languages/golang/encrypt/guest/src/ops.rs | 17 +++ languages/golang/encrypt/targets_test.go | 61 +++++++++ languages/golang/internal/record/record.go | 28 ++++- languages/golang/stashgen/engine.go | 16 ++- .../golang/stashgen/enginetest/enginetest.go | 13 +- languages/golang/stashgen/read.go | 9 +- 7 files changed, 221 insertions(+), 39 deletions(-) create mode 100644 languages/golang/encrypt/targets_test.go diff --git a/languages/golang/encrypt/checker.go b/languages/golang/encrypt/checker.go index 8041daacc..3677a5c9c 100644 --- a/languages/golang/encrypt/checker.go +++ b/languages/golang/encrypt/checker.go @@ -69,44 +69,116 @@ func (k *Checker) Targets(ctx context.Context) ([]record.Target, error) { if err != nil { return nil, fmt.Errorf("%w: %v", ErrInternal, err) } + return parseTargets(decoded) +} + +// parseTargets reads se_targets's decoded value: {"targets": [entry, ...]}, +// each entry an object with exactly the keys record.Target documents. The +// shape is a contract between eql-bindings' serialiser and this reader, so +// an unknown key, a missing name, or a value of the wrong type is +// ErrInternal: a decoder that kept a zero value would let stashgen decide an +// encrypt_into on a type with no kind or no indexes. +func parseTargets(decoded any) ([]record.Target, error) { obj, ok := decoded.(vcvalue.Object) if !ok || len(obj) != 1 || obj[0].Key != "targets" { - return nil, fmt.Errorf("%w: se_targets returned %T", ErrInternal, decoded) + return nil, fmt.Errorf("%w: se_targets returned %T, not {\"targets\": [...]}", ErrInternal, decoded) } items, ok := obj[0].Value.([]any) if !ok { return nil, fmt.Errorf("%w: se_targets returned %T for the list", ErrInternal, obj[0].Value) } targets := make([]record.Target, 0, len(items)) - for _, item := range items { + for i, item := range items { entry, ok := item.(vcvalue.Object) if !ok { - return nil, fmt.Errorf("%w: a target came back as %T", ErrInternal, item) + return nil, fmt.Errorf("%w: target %d came back as %T", ErrInternal, i, item) + } + t, err := parseTarget(entry) + if err != nil { + return nil, fmt.Errorf("%w: target %d: %v", ErrInternal, i, err) + } + targets = append(targets, t) + } + return targets, nil +} + +func parseTarget(entry vcvalue.Object) (record.Target, error) { + var t record.Target + seen := map[string]bool{} + for _, f := range entry { + if seen[f.Key] { + return t, fmt.Errorf("key %q twice", f.Key) } - var t record.Target - for _, f := range entry { - switch f.Key { - case "name": - t.Name, _ = f.Value.(string) - case "kind": - kind, _ := f.Value.(string) - t.Kind = record.Kind(kind) - case "query": - t.Query, _ = f.Value.(string) - case "terms": - terms, _ := f.Value.([]any) - for _, term := range terms { - s, _ := term.(string) - t.Terms = append(t.Terms, record.Output(s)) + seen[f.Key] = true + var err error + switch f.Key { + case "name": + t.Name, err = targetString(f) + case "family": + t.Family, err = targetString(f) + case "suffix": + t.Suffix, err = targetString(f) + case "plaintext": + var kind string + kind, err = targetOptionalString(f) + t.Plaintext = record.Kind(kind) + case "sql_domain": + t.SQLDomain, err = targetString(f) + case "indexes": + list, ok := f.Value.([]any) + if !ok { + return t, fmt.Errorf("indexes is %T, not a list", f.Value) + } + for _, item := range list { + s, ok := item.(string) + if !ok { + return t, fmt.Errorf("an index is %T, not a string", item) } + t.Indexes = append(t.Indexes, record.Output(s)) + } + case "query": + t.Query, err = targetOptionalString(f) + case "query_sql_domain": + t.QuerySQLDomain, err = targetOptionalString(f) + case "producible": + b, ok := f.Value.(bool) + if !ok { + return t, fmt.Errorf("producible is %T, not a bool", f.Value) } + t.Producible = b + case "reason": + t.Reason, err = targetOptionalString(f) + default: + return t, fmt.Errorf("unknown key %q", f.Key) } - if t.Name == "" { - return nil, fmt.Errorf("%w: a target has no name", ErrInternal) + if err != nil { + return t, err } - targets = append(targets, t) } - return targets, nil + if t.Name == "" { + return t, errors.New("no name") + } + for _, key := range []string{"indexes", "producible"} { + if !seen[key] { + return t, fmt.Errorf("no %s", key) + } + } + return t, nil +} + +func targetString(f vcvalue.Field) (string, error) { + s, ok := f.Value.(string) + if !ok { + return "", fmt.Errorf("%s is %T, not a string", f.Key, f.Value) + } + return s, nil +} + +func targetOptionalString(f vcvalue.Field) (string, error) { + if f.Value == nil { + return "", nil + } + return targetString(f) } // refusingTransport fails every request: the checker makes none. diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index f240dd5ec..a233eaf1b 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -208,6 +208,23 @@ pub fn plan_check(plan: &[u8]) -> Result<(), u32> { /// generator reads an empty list as "no `encrypt_into` type is available /// yet" and refuses the tag. The shape is fixed here so the next build adds /// entries to the list rather than a second export. +/// +/// **Each entry is wire format**, the serialisation of eql-bindings' target +/// record, and the Go reader (`encrypt.parseTargets`) refuses an entry with +/// any other key or a value of another type: +/// +/// | key | type | meaning | +/// |--------------------|-------------------|---------| +/// | `name` | string | the Go type name and the `encrypt_into` value: `TextEq` | +/// | `family` | string | `Text`, `Integer`, … | +/// | `suffix` | string | `Eq`, `Ord`, `Match`, … or `""` | +/// | `plaintext` | string or null | the vitaminc `ValueKind` name the type seals: `"string"` | +/// | `sql_domain` | string | the Postgres domain of the stored value | +/// | `indexes` | list of string | `eq`, `match`, `ore`, `ope`, `json` | +/// | `query` | string or null | the Go type name of the query value: `TextEqQuery` | +/// | `query_sql_domain` | string or null | the Postgres domain of the query value | +/// | `producible` | bool | whether this build produces the type | +/// | `reason` | string or null | why not, when `producible` is false | pub fn targets() -> Result, u32> { encode_value(FfiValue::Object(vec![( "targets".to_string(), diff --git a/languages/golang/encrypt/targets_test.go b/languages/golang/encrypt/targets_test.go new file mode 100644 index 000000000..ec4a91b5f --- /dev/null +++ b/languages/golang/encrypt/targets_test.go @@ -0,0 +1,61 @@ +package encrypt + +import ( + "errors" + "reflect" + "testing" + + "github.com/cipherstash/stack/languages/golang/internal/record" + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +// The se_targets entry shape is shared with eql-bindings' serialiser; this +// reader is strict about it, so the next build's entries are read whole or +// refused, never half-read. +func TestParseTargetsReadsEachEntry(t *testing.T) { + entry := func(fields ...vcvalue.Field) vcvalue.Object { return vcvalue.Object(fields) } + textEq := entry( + vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "family", Value: "Text"}, vcvalue.Field{Key: "suffix", Value: "Eq"}, + vcvalue.Field{Key: "plaintext", Value: "string"}, vcvalue.Field{Key: "sql_domain", Value: "eql_v3_text_eq"}, + vcvalue.Field{Key: "indexes", Value: []any{"eq"}}, vcvalue.Field{Key: "query", Value: "TextEqQuery"}, + vcvalue.Field{Key: "query_sql_domain", Value: "eql_v3.query_text_eq"}, vcvalue.Field{Key: "producible", Value: true}, vcvalue.Field{Key: "reason", Value: nil}, + ) + notYet := entry( + vcvalue.Field{Key: "name", Value: "JSON"}, vcvalue.Field{Key: "family", Value: "JSON"}, vcvalue.Field{Key: "suffix", Value: ""}, + vcvalue.Field{Key: "plaintext", Value: nil}, vcvalue.Field{Key: "sql_domain", Value: "eql_v3_json"}, + vcvalue.Field{Key: "indexes", Value: []any{"json"}}, vcvalue.Field{Key: "query", Value: nil}, vcvalue.Field{Key: "query_sql_domain", Value: nil}, + vcvalue.Field{Key: "producible", Value: false}, vcvalue.Field{Key: "reason", Value: "the JSON index is a new operation in the engine"}, + ) + got, err := parseTargets(vcvalue.Object{{Key: "targets", Value: []any{textEq, notYet}}}) + if err != nil { + t.Fatal(err) + } + want := []record.Target{ + {Name: "TextEq", Family: "Text", Suffix: "Eq", Plaintext: record.String, SQLDomain: "eql_v3_text_eq", Indexes: []record.Output{record.Equality}, Query: "TextEqQuery", QuerySQLDomain: "eql_v3.query_text_eq", Producible: true}, + {Name: "JSON", Family: "JSON", Plaintext: record.Untyped, SQLDomain: "eql_v3_json", Indexes: []record.Output{"json"}, Reason: "the JSON index is a new operation in the engine"}, + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("parseTargets =\n%+v\nwant\n%+v", got, want) + } + empty, err := parseTargets(vcvalue.Object{{Key: "targets", Value: []any{}}}) + if err != nil || len(empty) != 0 { + t.Fatalf("empty list: %v %v", empty, err) + } + for name, bad := range map[string]any{ + "not an object": "targets", + "another key": vcvalue.Object{{Key: "types", Value: []any{}}}, + "list not a list": vcvalue.Object{{Key: "targets", Value: "x"}}, + "entry not object": vcvalue.Object{{Key: "targets", Value: []any{"TextEq"}}}, + "numeric plaintext": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "plaintext", Value: int64(1)}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "unknown key": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "kind", Value: "string"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "no name": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "no producible": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}})}}}, + "index not string": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{int64(1)}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "producible string": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: "yes"})}}}, + "key twice": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "A"}, vcvalue.Field{Key: "name", Value: "B"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + } { + if _, err := parseTargets(bad); !errors.Is(err, ErrInternal) { + t.Errorf("%s: %v, want ErrInternal", name, err) + } + } +} diff --git a/languages/golang/internal/record/record.go b/languages/golang/internal/record/record.go index 9dacb251a..36c7659c4 100644 --- a/languages/golang/internal/record/record.go +++ b/languages/golang/internal/record/record.go @@ -307,10 +307,30 @@ type Outputs struct { // Sealed is one record as stored: each sealed field's outputs by name. type Sealed = map[string]Outputs -// Target is one EQL type the engine produces, as se_targets lists it. +// Target is one EQL type as se_targets lists it. The entry's keys are wire +// format, written by eql-bindings' serialiser and read by [ParseTargets]: +// name, family, suffix, plaintext (a ValueKind name, or null), sql_domain, +// indexes (eq | match | ore | ope | json), query (or null), query_sql_domain +// (or null), producible and reason (or null). type Target struct { - Name string - Kind Kind - Terms []Output + // Name is the Go type name, and the value of encrypt_into: TextEq. + Name string + // Family and Suffix are the two halves of the name: Text, Eq. + Family string + Suffix string + // Plaintext is the kind the type seals, or Untyped when the entry says + // null. + Plaintext Kind + // SQLDomain is the Postgres domain of the stored value. + SQLDomain string + // Indexes are the terms the type carries. + Indexes []Output + // Query is the Go type name of the query value, or "" for none. Query string + // QuerySQLDomain is the Postgres domain of the query value, or "". + QuerySQLDomain string + // Producible says whether this build of the engine produces the type; + // Reason says why not when it does not. + Producible bool + Reason string } diff --git a/languages/golang/stashgen/engine.go b/languages/golang/stashgen/engine.go index 7d7d10f7c..279132247 100644 --- a/languages/golang/stashgen/engine.go +++ b/languages/golang/stashgen/engine.go @@ -37,6 +37,10 @@ type EQLType struct { // Query is the Go type name of the type's query value, such as // TextEqQuery, or "" for a type with no terms. Query string + // Producible says whether this build of the engine produces the type; + // Reason says why not when it does not. + Producible bool + Reason string } // GuestEngine is the engine the SDK embeds: the WASI guest in package @@ -66,8 +70,8 @@ func (e *guestEngine) EQLTypes(ctx context.Context) ([]EQLType, error) { } out := make([]EQLType, 0, len(targets)) for _, t := range targets { - eqlType := EQLType{Name: t.Name, Plaintext: kindOfWire(t.Kind), Query: t.Query} - for _, term := range t.Terms { + eqlType := EQLType{Name: t.Name, Plaintext: kindOfWire(t.Plaintext), Query: t.Query, Producible: t.Producible, Reason: t.Reason} + for _, term := range t.Indexes { if name, ok := indexOfOutput[term]; ok { eqlType.Indexes = append(eqlType.Indexes, name) } @@ -131,10 +135,10 @@ func lowerDeclaration(d Declaration, eqlTypes []EQLType) (*record.Plan, error) { rf := record.Field{Name: f.Name, Identity: f.Identity, Kind: wireKind(f.GoType)} switch f.Verb { case VerbEncryptInto: - if len(eqlTypes) == 0 { - return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: "EQL types are not available yet; the engine produces none in this build"} - } - return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine cannot produce the EQL type %s yet", f.EQLType)} + // The reader refused anything the engine does not produce; a + // producible type reaches the engine's check in the next build, + // which lowers it. Until then no type is producible. + return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine cannot seal into the EQL type %s in this build", f.EQLType)} case VerbEncrypt, VerbEncryptIndex: rf.Outputs = append(rf.Outputs, record.Ciphertext) } diff --git a/languages/golang/stashgen/enginetest/enginetest.go b/languages/golang/stashgen/enginetest/enginetest.go index e3361d1ee..10de7a3c0 100644 --- a/languages/golang/stashgen/enginetest/enginetest.go +++ b/languages/golang/stashgen/enginetest/enginetest.go @@ -22,7 +22,7 @@ var _ stashgen.Engine = Static{} // EQLTypes returns TextEq, the one EQL type the engine produces today. func (Static) EQLTypes(context.Context) ([]stashgen.EQLType, error) { return []stashgen.EQLType{ - {Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery"}, + {Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery", Producible: true}, }, nil } @@ -64,14 +64,15 @@ func (e Static) Check(ctx context.Context, d stashgen.Declaration) error { return nil } -// indexApplies mirrors the Index impls in stack-encrypt's target/index.rs: -// Equality for every PRF value, Match for text, Ore and Ope for what cllw-ore -// implements (text, bytes, bool, integers and floats). The JSON index is not -// in the engine yet. +// indexApplies mirrors dynamic::admits in stack-encrypt: equality over +// integers, text and bytes (no floats, no booleans: they have no PRF +// encoding), match over text alone, ORE and OPE over every scalar. The JSON +// index is not in the engine yet. func indexApplies(name stashgen.IndexName, t stashgen.GoType) error { switch name { case stashgen.IndexEquality: - if t.Kind.Scalar() { + switch t.Kind { + case stashgen.KindString, stashgen.KindBytes, stashgen.KindInt, stashgen.KindUint: return nil } case stashgen.IndexMatch: diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index 8c2d3fe0a..bc6032741 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -588,7 +588,14 @@ func (r *reader) buildFields(c *collected) error { if len(r.eql) == 0 { return fieldErr(typeName, cf.goName, "EQL types are not available yet; this build of the engine produces none") } - return fieldErr(typeName, cf.goName, "the engine cannot produce the EQL type %s", field.EQLType) + return fieldErr(typeName, cf.goName, "the engine has no EQL type %s", field.EQLType) + } + if !eqlType.Producible { + reason := eqlType.Reason + if reason == "" { + reason = "not in this build of the engine" + } + return fieldErr(typeName, cf.goName, "the engine cannot produce the EQL type %s yet: %s", field.EQLType, reason) } f.imports.add(eqlPath, "eql") g.outputType = "eql." + eqlType.Name From 60c28217d232cb64e2814778794ace401f420b72 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:50:46 -0700 Subject: [PATCH 11/30] test(golang): the refusals hold against the embedded engine, not only the fake MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TestRefusalsHoldAgainstTheEmbeddedEngine runs stashgen with GuestEngine — the engine it ships with — over the cases a reviewer found the fake and the real engine disagreeing on (time.Time, a channel, equality on a map, equality on a float) and the composites a reviewer sealed and could not open ([]string, map[string]int64); each is refused naming the field, and a float equality is refused by both engines alike. The sealed-field rule (one scalar, or a type defined over one; a composite only inside an opaque struct) is what refuses the first three before either engine is asked. TestTheWholeDeclarationIsCheckedAgainstTheEmbeddedEngine makes the whole-plan check fail — two fields a policy pinned to one identity — and pins its FieldError with no field name. The two doc comments that still said a composite seals as one value say what the code does. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/encrypt/ciphertext.go | 9 ++- languages/golang/stashgen/refusal_test.go | 83 ++++++++++++++++++++++- 2 files changed, 88 insertions(+), 4 deletions(-) diff --git a/languages/golang/encrypt/ciphertext.go b/languages/golang/encrypt/ciphertext.go index 7aa4b4fef..41b39a05e 100644 --- a/languages/golang/encrypt/ciphertext.go +++ b/languages/golang/encrypt/ciphertext.go @@ -14,9 +14,12 @@ import ( // own leaf on purpose: a stack-encrypt leaf is not decryptable by // vitaminc-encrypt and must never scan or marshal where one belongs. // -// A field of any Go type seals to one leaf. A scalar seals as the typed leaf -// a Rust record derives; a struct, slice or map seals as one self-describing -// value, which only a dynamic reader opens. +// A sealed field is one scalar — a string, a number, a bool or a []byte, or +// a type defined over one — and seals as the typed leaf a Rust record +// derives. A struct, slice or map seals only as part of an opaque struct, +// which crosses as one JSON document and is one leaf; stashgen refuses it +// anywhere else, because the engine would seal it as a tree of leaves and a +// column holds one. type Ciphertext []byte // Value implements driver.Valuer, binding the leaf as a byte column. diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index 773dc011a..77db9649f 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -3,11 +3,13 @@ package stashgen_test import ( "context" "errors" + "io" "os" "path/filepath" "strings" "testing" + "github.com/cipherstash/stack/languages/golang/encrypt" "github.com/cipherstash/stack/languages/golang/stashgen" "github.com/cipherstash/stack/languages/golang/stashgen/enginetest" ) @@ -73,11 +75,13 @@ func TestRefusals(t *testing.T) { {"a field type the engine cannot seal", user(ctx + "\tDone chan int `stash:\"done,encrypt\"`"), stashgen.Request{Type: "User"}, "Done", "seals only as part of an opaque struct"}, {"a composite sealed outside an opaque struct", user(ctx + "\tAt time.Time `stash:\"at,encrypt\"`"), stashgen.Request{Type: "User"}, "At", "a time.Time seals only as part of an opaque struct"}, {"a slice sealed outside an opaque struct", user(ctx + "\tTags []string `stash:\"tags,encrypt\"`"), stashgen.Request{Type: "User"}, "Tags", "seals only as part of an opaque struct"}, + {"a map sealed outside an opaque struct", user(ctx + "\tCounts map[string]int64 `stash:\"counts,encrypt\"`"), stashgen.Request{Type: "User"}, "Counts", "seals only as part of an opaque struct"}, + {"a slice of ints sealed outside an opaque struct", user(ctx + "\tNs []int64 `stash:\"ns,encrypt\"`"), stashgen.Request{Type: "User"}, "Ns", "seals only as part of an opaque struct"}, {"an opaque struct with a channel", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tDone chan int"), stashgen.Request{Type: "User"}, "Done", "is not a type JSON carries"}, {"an opaque struct with an interface", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tAny any"), stashgen.Request{Type: "User"}, "Any", "is not a type JSON carries"}, {"an opaque struct with a struct JSON would truncate", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tTx hidden") + "\ntype hidden struct {\n\tA int\n\tb int\n}\n", stashgen.Request{Type: "User"}, "Tx", "JSON would drop"}, {"an opaque struct with a map keyed by a struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tM map[hidden]int") + "\ntype hidden struct{ A int }\n", stashgen.Request{Type: "User"}, "M", "map key that is not a string or an integer"}, - {"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "cannot produce the EQL type TextMatch"}, + {"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "has no EQL type TextMatch"}, {"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"}, {"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"}, {"an index option the engine cannot carry", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "cannot carry index options"}, @@ -206,3 +210,80 @@ func TestAStructWithPrintMethodsGetsNoNotice(t *testing.T) { t.Fatal("PrintsPlaintext set for a type with String and LogValue") } } + +// The refusals hold against the engine stashgen ships with, not only the +// fake: the embedded guest, through GuestEngine. Skips only when the guest +// is not built. The cases are the ones a reviewer found the two engines +// disagreeing on, and the composites a reviewer sealed and could not open. +func TestRefusalsHoldAgainstTheEmbeddedEngine(t *testing.T) { + engine, err := stashgen.GuestEngine(context.Background()) + if errors.Is(err, encrypt.ErrGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + defer engine.(io.Closer).Close() + for _, c := range []struct{ name, field, line, want string }{ + {"time.Time", "At", "\tAt time.Time `stash:\"at,encrypt\"`", "seals only as part of an opaque struct"}, + {"a channel", "Done", "\tDone chan int `stash:\"done,encrypt\"`", "seals only as part of an opaque struct"}, + {"equality on a map", "Attrs", "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`", "seals only as part of an opaque struct"}, + {"a sealed []string", "Tags", "\tTags []string `stash:\"tags,encrypt\"`", "seals only as part of an opaque struct"}, + {"a sealed map[string]int64", "Counts", "\tCounts map[string]int64 `stash:\"counts,encrypt\"`", "seals only as part of an opaque struct"}, + {"equality on a float", "Score", "\tScore float64 `stash:\"score,encrypt,index=equality\"`", "the engine refuses the declaration"}, + {"equality on a bool", "Done", "\tDone bool `stash:\"done,encrypt,index=equality\"`", "the engine refuses the declaration"}, + {"match on an integer", "Age", "\tAge int32 `stash:\"age,encrypt,index=match\"`", "the engine refuses the declaration"}, + {"encrypt_into, no EQL type in this build", "Email", "\tEmail string `stash:\"email,encrypt_into=TextEq\"`", "EQL types are not available yet"}, + } { + t.Run(c.name, func(t *testing.T) { + dir := writeModule(t, map[string]string{"model.go": user(ctx + c.line), "crm/contact.go": crmPackage}) + _, err := stashgen.FromTags(context.Background(), engine, stashgen.Request{Dir: dir, Type: "User"}) + var fe *stashgen.FieldError + if !errors.As(err, &fe) || fe.Field != c.field { + t.Fatalf("err = %v, want a *FieldError naming %s", err, c.field) + } + if !strings.Contains(err.Error(), c.want) { + t.Fatalf("err = %v, want %q", err, c.want) + } + }) + } + // What the fake refuses, the engine refuses: the same cases against both. + for _, c := range []struct{ name, line string }{ + {"equality on a float", "\tScore float64 `stash:\"score,encrypt,index=equality\"`"}, + {"a declaration the engine runs", "\tAge uint32 `stash:\"age,encrypt,index=equality;ore\"`\n\tName string `stash:\"name,encrypt,index=match\"`"}, + } { + dir := writeModule(t, map[string]string{"model.go": user(ctx + c.line), "crm/contact.go": crmPackage}) + _, real := stashgen.FromTags(context.Background(), engine, stashgen.Request{Dir: dir, Type: "User"}) + _, fake := stashgen.FromTags(context.Background(), enginetest.Static{}, stashgen.Request{Dir: dir, Type: "User"}) + if (real == nil) != (fake == nil) { + t.Errorf("%s: the engine says %v, the fake says %v", c.name, real, fake) + } + } +} + +// The whole-declaration check: two fields that pass one at a time and fail +// together, because a policy pinned both to one identity. The error is a +// *FieldError with no field. +func TestTheWholeDeclarationIsCheckedAgainstTheEmbeddedEngine(t *testing.T) { + engine, err := stashgen.GuestEngine(context.Background()) + if errors.Is(err, encrypt.ErrGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + defer engine.(io.Closer).Close() + decl := stashgen.Declaration{Type: "Individual", Context: "individuals", Fields: []stashgen.Field{ + {Name: "medicare_number", GoName: "MedicareNo", GoType: stashgen.GoType{Name: "string", Kind: stashgen.KindString, Basic: "string"}, Verb: stashgen.VerbEncrypt, Identity: "id"}, + {Name: "legacy_number", GoName: "LegacyNo", GoType: stashgen.GoType{Name: "string", Kind: stashgen.KindString, Basic: "string"}, Verb: stashgen.VerbEncrypt, Identity: "id"}, + }} + err = engine.Check(context.Background(), decl) + var fe *stashgen.FieldError + if !errors.As(err, &fe) || fe.Field != "" || !strings.Contains(err.Error(), "as a whole") { + t.Fatalf("err = %v, want the whole-declaration refusal naming no field", err) + } + decl.Fields[1].Identity = "" + if err := engine.Check(context.Background(), decl); err != nil { + t.Fatalf("distinct identities: %v", err) + } +} From 2695b38a5af1ace98756bbdd05c344f70d89719f Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:50:47 -0700 Subject: [PATCH 12/30] test(golang): residency on every pull request, and the fixture test fails on any other error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The residency check runs hermetically over the deterministic guest with a populated record (a match index refuses an empty Email) and checks the guest's memory for the plaintext and the email after Encrypt. The same check after Decrypt is its own test and is skipped with the reason: one unwiped copy of each opened string remains, made by vitaminc-aead-value's FfiValue decoder, which copies the decrypted leaf into a new Protected and leaves the AEAD output it copied from to drop unwiped — the host's output buffer is wiped through se_dealloc and the Protected payloads on drop, so the copy is vitaminc's to remove, and the test runs again once it is. The live variant keeps the Encrypt check across a real ZeroKMS request. The fixture test skips only when the guest is not built and fails on every other error from NewDeterministicClient. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/encrypt/fixture_test.go | 6 ++- languages/golang/encrypt/live_test.go | 23 ++++++----- languages/golang/encrypt/roundtrip_test.go | 45 ++++++++++++++++++++++ 3 files changed, 64 insertions(+), 10 deletions(-) diff --git a/languages/golang/encrypt/fixture_test.go b/languages/golang/encrypt/fixture_test.go index ecb99ee3b..e992a9679 100644 --- a/languages/golang/encrypt/fixture_test.go +++ b/languages/golang/encrypt/fixture_test.go @@ -5,6 +5,7 @@ import ( "context" "encoding/hex" "encoding/json" + "errors" "os" "path/filepath" "testing" @@ -77,9 +78,12 @@ func TestGeneratedCodeOpensTheRustRecordFixture(t *testing.T) { t.Fatalf("seed: %v (%d bytes)", err, len(seedBytes)) } c, err := encrypt.NewDeterministicClient(context.Background(), [32]byte(seedBytes)) - if err != nil { + if errors.Is(err, encrypt.ErrDeterministicGuestNotBuilt) || errors.Is(err, encrypt.ErrGuestNotBuilt) { t.Skip(err) } + if err != nil { + t.Fatal(err) + } defer c.Close() ctx := context.Background() // The fixture's keyset is the nil UUID: the fake source's default. diff --git a/languages/golang/encrypt/live_test.go b/languages/golang/encrypt/live_test.go index 39c850369..b479240c4 100644 --- a/languages/golang/encrypt/live_test.go +++ b/languages/golang/encrypt/live_test.go @@ -117,20 +117,25 @@ func TestLiveForeignKeysetIsRefusedBeforeRetrieval(t *testing.T) { } } -// Per-call hygiene on a real round trip: once Encrypt has returned, the -// plaintext it was given is nowhere in guest memory — the staged input was -// wiped by se_dealloc — so between calls the guest holds only the client -// key and its keyset cache. The record is fully populated: a match index -// over an empty string has no token and the engine refuses it (see -// TestAMatchIndexNeedsText). -func TestPlaintextDoesNotRemainInGuestMemoryAfterEncrypt(t *testing.T) { +// The hermetic residency test (TestPlaintextDoesNotRemainInGuestMemory) +// runs on every pull request; this is the same check across a real ZeroKMS +// request, where the guest also stages the service's responses — wrapped +// data keys — and must wipe those too. +func TestPlaintextDoesNotRemainInGuestMemoryAfterLiveRoundTrip(t *testing.T) { c := encrypt.LiveClient(t) + cipher := c.DefaultKeyset() const plaintext = "residency-probe-4111-b1c2d3e4f5" probe := testusers.User{ID: 1, Age: 34, Email: "probe@example.com", Notes: plaintext} - if _, err := testusers.Encrypt(t.Context(), c.DefaultKeyset(), []testusers.User{probe}); err != nil { + encrypted, err := testusers.Encrypt(t.Context(), cipher, []testusers.User{probe}) + if err != nil { t.Fatalf("Encrypt: %v", err) } if n := bytes.Count(encrypt.GuestMemory(t, c), []byte(plaintext)); n != 0 { - t.Fatalf("plaintext found %d times in guest memory after Encrypt returned", n) + t.Fatalf("plaintext found %d times in guest memory after Encrypt", n) + } + // After Decrypt one copy remains; see the hermetic + // TestPlaintextDoesNotRemainInGuestMemoryAfterDecrypt for the cause. + if _, err := testusers.Decrypt(t.Context(), cipher, encrypted); err != nil { + t.Fatalf("Decrypt: %v", err) } } diff --git a/languages/golang/encrypt/roundtrip_test.go b/languages/golang/encrypt/roundtrip_test.go index 5909dff02..ae88fd562 100644 --- a/languages/golang/encrypt/roundtrip_test.go +++ b/languages/golang/encrypt/roundtrip_test.go @@ -299,3 +299,48 @@ func TestAMatchIndexNeedsText(t *testing.T) { t.Fatalf("zero values round trip: %v %+v", err, back) } } + +// Per-call hygiene, on every pull request: once Encrypt has returned, the +// plaintext it carried is nowhere in guest memory — every buffer staged for a +// call is wiped before its result comes back — so between calls the guest +// holds the keyset cache and nothing of the program's data. +func TestPlaintextDoesNotRemainInGuestMemory(t *testing.T) { + c := deterministicClient(t) + cipher := c.DefaultKeyset() + const plaintext = "residency-probe-4111-b1c2d3e4f5" + const email = "residency-probe@example.com" + if _, err := testusers.Encrypt(t.Context(), cipher, []testusers.User{{ID: 1, Age: 34, Email: email, Notes: plaintext}}); err != nil { + t.Fatal(err) + } + for _, needle := range []string{plaintext, email} { + if n := bytes.Count(encrypt.GuestMemory(t, c), []byte(needle)); n != 0 { + t.Fatalf("%q found %d times in guest memory after Encrypt", needle, n) + } + } +} + +// The same check after Decrypt, which is the call that puts plaintext back +// into the guest. It fails today: exactly one copy of each opened string +// stays in the guest's freed heap. The host's copy is wiped (the output +// buffer goes through se_dealloc) and the FfiValue's Protected payloads are +// wiped on drop; the copy that stays is made inside vitaminc-aead-value's +// FfiValue decoder, which copies the decrypted leaf's bytes into a new +// Protected (value.rs, the `tags::STRING` arm) and leaves the AEAD output +// it copied from to drop unwiped. That is vitaminc's to fix; the skip +// records it, and the test runs again once it is. +func TestPlaintextDoesNotRemainInGuestMemoryAfterDecrypt(t *testing.T) { + t.Skip("known: one unwiped copy of each opened string remains after Decrypt; see the comment above (vitaminc-aead-value FfiValue decode)") + c := deterministicClient(t) + cipher := c.DefaultKeyset() + const plaintext = "residency-probe-4111-b1c2d3e4f5" + encrypted, err := testusers.Encrypt(t.Context(), cipher, []testusers.User{{ID: 1, Age: 34, Email: "probe@example.com", Notes: plaintext}}) + if err != nil { + t.Fatal(err) + } + if _, err := testusers.Decrypt(t.Context(), cipher, encrypted); err != nil { + t.Fatal(err) + } + if n := bytes.Count(encrypt.GuestMemory(t, c), []byte(plaintext)); n != 0 { + t.Fatalf("plaintext found %d times in guest memory after Decrypt", n) + } +} From 772be38d7ce7eaa9aca9621b6cd1612879e43a37 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 02:51:04 -0700 Subject: [PATCH 13/30] refactor(golang): one DeterministicSource, protoc-gen-go's GoName, and the review's small findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DeterministicSource existed twice — stack-encrypt's tests/common and the guest's deterministic.rs — and the Go fixture test depends on the two deriving the same bytes. It now lives once in stack-kms behind its test-support feature, beside FakeDataKeySource; the stack-encrypt tests and the guest import it, and the guest's deterministic-kms feature no longer carries its own sha2. The rebuilt test guest still opens the fixture, which is the proof the move changed no bytes. protosource.GoName is protoc-gen-go's GoCamelCase word for word: a digit ends a word (foo_1bar is Foo_1Bar, sha256sum is Sha256Sum), a leading underscore is X, a dot is an underscore unless a lower-case letter follows; TestGoName held the wrong value for foo_1bar. scalar's enum branch has a test through a dynamic descriptor, since testpb declares no enum. stack-guest-abi's crate doc names the guest's real path. The three documents say what a match index needs — text with at least one token, three characters for the n-gram — that the whole batch fails otherwise, and that an optional or short value does not belong under match. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/cmd/stashgen/README.md | 5 +- languages/golang/encrypt/README.md | 7 + languages/golang/encrypt/doc.go | 7 + languages/golang/encrypt/guest/Cargo.lock | 1 - languages/golang/encrypt/guest/Cargo.toml | 4 +- .../golang/encrypt/guest/src/deterministic.rs | 125 ++--------------- .../encrypt/policy/protosource/protosource.go | 46 ++++--- .../policy/protosource/protosource_test.go | 49 ++++++- packages/stack-encrypt/tests/common/mod.rs | 105 +-------------- packages/stack-guest-abi/src/lib.rs | 2 +- packages/stack-kms/src/key_source.rs | 126 ++++++++++++++++++ packages/stack-kms/src/lib.rs | 2 + 12 files changed, 240 insertions(+), 239 deletions(-) diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 79b49a637..39dbc1538 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -73,8 +73,9 @@ The first part of a tag is the field's name, which is the column name in a datab | `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value | The index names are `equality`, `match`, `ore`, `ope` and `json`. -A `match` index needs text with at least one token: the engine derives no match term for an empty or separator-only string, or one shorter than the n-gram, because an empty term would match every row. -`Encrypt` then returns an error naming the row, the field and the index; give the field a value, or drop the index. +A `match` index needs text with at least one token: the engine derives no match term for an empty string, separator-only text, or text shorter than the n-gram length (3 characters), because an empty term would match every row. +`Encrypt` then fails for the whole batch, with an error naming the row, the field and the index. +So an optional or short value does not belong under `match`: give the field `equality` alone, or make the value required. An index takes its options in parentheses after its name, separated by commas: `index=equality;match(k=3)`. These words are the same as the Rust API's words for the same behaviour. diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index ba790af47..00caee33d 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -143,6 +143,13 @@ and `Equality`, `Match`, `Ore` or `Ope` (the term types). A field with An `opaque` struct is one `Sealed` field. Every stored type implements `driver.Valuer` and `sql.Scanner`. +A `match` index needs text with at least one token. The engine derives no +match term for an empty string, separator-only text, or text shorter than the +n-gram length (3 characters), because an empty term would match every row. +`Encrypt` then fails for the whole batch with `ErrTerm`, naming the row, the +field and the index. So an optional or short value does not belong under +`match`: give such a field `equality` alone, or make the value required. + Terms are byte-equal to the ones the Rust crate derives, so a term from `users.Fields.Email.Equality` compares against a stored term written from any language. `EqualityTerm.Equal` compares in constant time; `OreTerm.Compare` diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index c1ce88202..9679ce799 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -93,6 +93,13 @@ // [Match], [Ore] and [Ope]; [JSON] is declared and refused until the engine // derives it. Every stored type implements driver.Valuer and sql.Scanner. // +// A match index needs text with at least one token: the engine derives no +// match term for an empty string, separator-only text, or text shorter than +// the n-gram length (3 characters), because an empty term would match every +// row. Encrypt then fails for the whole batch with [ErrTerm], naming the row, +// the field and the index, so an optional or short value does not belong +// under match. +// // # Transport and auth // // The guest imports exactly two host functions: an HTTP send, served by any diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index ff9ba2331..94aff4e53 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -2016,7 +2016,6 @@ dependencies = [ "recipher", "serde", "serde_json", - "sha2 0.10.9", "stack-auth", "stack-encrypt", "stack-guest-abi", diff --git a/languages/golang/encrypt/guest/Cargo.toml b/languages/golang/encrypt/guest/Cargo.toml index 8ab7b3117..3d47e469d 100644 --- a/languages/golang/encrypt/guest/Cargo.toml +++ b/languages/golang/encrypt/guest/Cargo.toml @@ -30,7 +30,7 @@ crate-type = ["cdylib", "rlib"] # network. The Go tests load it to open the fixture's records and to run # hermetic round trips; `mise run wasm:guest:build:deterministic` builds it # beside the real guest. Never the embedded guest. -deterministic-kms = ["stack-kms/test-support", "dep:sha2"] +deterministic-kms = ["stack-kms/test-support"] [dependencies] # The stack crates with default features off: no reqwest, no native TLS — @@ -56,8 +56,6 @@ vitaminc-aead-value = "0.5.1" vitaminc-protected = "0.5.1" futures = { version = "0.3", default-features = false, features = ["executor"] } -# Only the deterministic-kms test build derives keys itself. -sha2 = { version = "0.10", optional = true } serde = "1" serde_json = "1" uuid = "1" diff --git a/languages/golang/encrypt/guest/src/deterministic.rs b/languages/golang/encrypt/guest/src/deterministic.rs index d087dd088..e58a26e74 100644 --- a/languages/golang/encrypt/guest/src/deterministic.rs +++ b/languages/golang/encrypt/guest/src/deterministic.rs @@ -1,116 +1,13 @@ -//! The deterministic key source of the `deterministic-kms` test build: a -//! copy of `DeterministicSource` in stack-encrypt's `tests/common/mod.rs`, -//! which sealed the record fixture (`tests/fixtures/record_lowering.json`). +//! The deterministic key source of the `deterministic-kms` test build: +//! `stack-kms`'s [`DeterministicSource`] (behind its `test-support` feature), +//! the one definition that also sealed stack-encrypt's record fixture +//! (`tests/fixtures/record_lowering.json`). A Go test that loads this build +//! with the fixture's seed opens the records Rust sealed and derives the same +//! term bytes, because both run the same code. //! -//! Every data key is `SHA-256(seed ‖ "key" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, its -//! tag `SHA-256(seed ‖ "tag" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, and the IV -//! `SHA-256(seed ‖ "iv" ‖ 0 ‖ descriptor ‖ 0 ‖ counter)[..16]`, where -//! `descriptor` is the context the leaf is sealed under as ZeroKMS renders -//! it. A leaf opens under its own field's descriptor and no other, as under -//! ZeroKMS. The index key is `FakeDataKeySource`'s, the same every test in -//! the repository derives terms under. So a Go test that loads this build -//! with the fixture's seed opens the records Rust sealed and derives the -//! same term bytes. -//! -//! It is a test double. The feature that compiles it is off by default, the -//! Go package never embeds this build, and the fixture README beside the -//! JSON file is the one definition both copies follow. - -use std::borrow::Cow; -use std::sync::atomic::{AtomicU64, Ordering}; - -use sha2::{Digest, Sha256}; -use stack_kms::{ - DataKey, DataKeySource, DataKeyWithTag, FakeDataKeySource, GenerateKeyPayload, IdentifiedBy, - IndexKey, IndexKeySource, RetrieveKeyPayload, UnverifiedContext, -}; -use uuid::Uuid; - -/// The seeded source. See the [module docs](self). -pub struct DeterministicSource { - seed: [u8; 32], - counter: AtomicU64, - index: FakeDataKeySource, -} - -impl DeterministicSource { - /// A source over `seed`. - pub fn new(seed: [u8; 32]) -> Self { - Self { - seed, - counter: AtomicU64::new(0), - index: FakeDataKeySource::new(), - } - } - - fn derive(&self, what: &str, descriptor: &str, salt: &[u8]) -> [u8; 32] { - let mut hasher = Sha256::new(); - hasher.update(self.seed); - hasher.update(what.as_bytes()); - hasher.update([0u8]); - hasher.update(descriptor.as_bytes()); - hasher.update([0u8]); - hasher.update(salt); - hasher.finalize().into() - } -} - -impl DataKeySource for DeterministicSource { - async fn generate_keys( - &self, - payloads: Vec>, - _keyset_id: Option, - _unverified_context: Option>, - ) -> Result, stack_kms::Error> { - Ok(payloads - .into_iter() - .map(|payload| { - let n = self.counter.fetch_add(1, Ordering::SeqCst); - let iv_bytes = self.derive("iv", payload.descriptor, &n.to_le_bytes()); - let mut iv = stack_kms::Iv::default(); - let width = iv.len(); - iv.copy_from_slice(&iv_bytes[..width]); - let key = self.derive("key", payload.descriptor, &iv); - let tag = self.derive("tag", payload.descriptor, &iv); - DataKeyWithTag { - key: DataKey { iv, key }, - tag: tag.to_vec(), - decryption_policy: payload.decryption_policy, - } - }) - .collect()) - } - - async fn retrieve_keys( - &self, - payloads: Vec>, - _keyset_id: Option, - _unverified_context: Option<&UnverifiedContext>, - ) -> Result, stack_kms::Error> { - payloads - .iter() - .map(|payload| { - let iv: stack_kms::Iv = *payload.iv.as_ref(); - let tag = self.derive("tag", payload.descriptor, &iv); - if tag[..] != *payload.tag { - return Err(stack_kms::Error::RetrieveKey( - stack_kms::RetrieveKeyError::FailedRetrieval( - "the tag is not this descriptor's".to_string(), - ), - )); - } - let key = self.derive("key", payload.descriptor, &iv); - Ok(DataKey { iv, key }) - }) - .collect() - } -} +//! It is a test double. The feature that compiles it is off by default, and +//! the build it produces goes under languages/golang/encrypt/testdata, which +//! `go build` and the package's `//go:embed wasm` both ignore, so no Go +//! binary carries it; the tests read it from disk. -impl IndexKeySource for DeterministicSource { - async fn load_index_key( - &self, - keyset_id: Option, - ) -> Result<(Uuid, IndexKey), stack_kms::Error> { - self.index.load_index_key(keyset_id).await - } -} +pub use stack_kms::DeterministicSource; diff --git a/languages/golang/encrypt/policy/protosource/protosource.go b/languages/golang/encrypt/policy/protosource/protosource.go index 8f2901505..4190419ee 100644 --- a/languages/golang/encrypt/policy/protosource/protosource.go +++ b/languages/golang/encrypt/policy/protosource/protosource.go @@ -8,7 +8,6 @@ package protosource import ( "fmt" - "strings" "google.golang.org/protobuf/proto" "google.golang.org/protobuf/reflect/protoreflect" @@ -82,27 +81,42 @@ func scalar(fd protoreflect.FieldDescriptor, v protoreflect.Value) string { return v.String() } -// GoName is the Go field name protoc-gen-go gives a proto field: the first -// letter capitalised, and an underscore before a lower-case letter dropped -// with that letter capitalised. medicare_no becomes MedicareNo; foo_1bar -// keeps its underscore, as protoc-gen-go does. +// GoName is the Go field name protoc-gen-go gives a proto field: its +// GoCamelCase, word for word. A word starts at an underscore or a capital; +// a digit is a word of its own, so the letter after it starts a new word +// (foo_1bar is Foo_1Bar, sha256sum is Sha256Sum); an underscore before a +// lower-case letter is dropped and the letter capitalised; a leading +// underscore becomes X (_x is XX); a dot becomes an underscore. func GoName(protoName string) string { - var b strings.Builder - upper := true + var b []byte for i := 0; i < len(protoName); i++ { c := protoName[i] switch { - case c == '_' && i+1 < len(protoName) && protoName[i+1] >= 'a' && protoName[i+1] <= 'z': - upper = true + case c == '.' && i+1 < len(protoName) && isASCIILower(protoName[i+1]): + // Skip over '.' in ".{{lowercase}}". case c == '.': - b.WriteByte('_') - case upper && c >= 'a' && c <= 'z': - b.WriteByte(c - 'a' + 'A') - upper = false + b = append(b, '_') + case c == '_' && (i == 0 || protoName[i-1] == '.'): + // An initial '_' (or one after '.') becomes 'X', so the name + // starts with a capital. + b = append(b, 'X') + case c == '_' && i+1 < len(protoName) && isASCIILower(protoName[i+1]): + // Skip over '_' in "_{{lowercase}}". + case c >= '0' && c <= '9': + b = append(b, c) default: - b.WriteByte(c) - upper = false + // A letter starts a word, capitalised; the lower-case run after + // it is the rest of the word. + if isASCIILower(c) { + c -= 'a' - 'A' + } + b = append(b, c) + for ; i+1 < len(protoName) && isASCIILower(protoName[i+1]); i++ { + b = append(b, protoName[i+1]) + } } } - return b.String() + return string(b) } + +func isASCIILower(c byte) bool { return 'a' <= c && c <= 'z' } diff --git a/languages/golang/encrypt/policy/protosource/protosource_test.go b/languages/golang/encrypt/policy/protosource/protosource_test.go index c0492451b..df01118dd 100644 --- a/languages/golang/encrypt/policy/protosource/protosource_test.go +++ b/languages/golang/encrypt/policy/protosource/protosource_test.go @@ -4,6 +4,11 @@ import ( "strings" "testing" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/reflect/protodesc" + "google.golang.org/protobuf/reflect/protoreflect" + "google.golang.org/protobuf/types/descriptorpb" + "github.com/cipherstash/stack/languages/golang/encrypt/policy" "github.com/cipherstash/stack/languages/golang/encrypt/policy/protosource/internal/testpb" ) @@ -62,10 +67,52 @@ func TestNotAMessage(t *testing.T) { } func TestGoName(t *testing.T) { - cases := map[string]string{"id": "Id", "medicare_no": "MedicareNo", "foo_1bar": "Foo_1bar", "Already": "Already", "a_b_c": "ABC", "x__y": "X_Y"} + // protoc-gen-go's GoCamelCase, including the cases a reviewer found the + // first version wrong on: a digit ends a word, a leading underscore is X. + cases := map[string]string{ + "id": "Id", "medicare_no": "MedicareNo", "foo_1bar": "Foo_1Bar", "sha256sum": "Sha256Sum", "_x": "XX", + "Already": "Already", "a_b_c": "ABC", "x__y": "X_Y", "a.b": "AB", "a.B": "A_B", "x_Y": "X_Y", "ab1": "Ab1", + } for in, want := range cases { if got := GoName(in); got != want { t.Errorf("GoName(%q) = %q, want %q", in, got, want) } } } + +// scalar spells an enum option by its value name: a dynamic enum field +// stands in for a generated one, since testpb declares no enum. +func TestScalarSpellsAnEnumByName(t *testing.T) { + file, err := protodesc.NewFile(&descriptorpb.FileDescriptorProto{ + Name: proto.String("enum_test.proto"), + Package: proto.String("enumtest"), + Syntax: proto.String("proto3"), + EnumType: []*descriptorpb.EnumDescriptorProto{{ + Name: proto.String("Level"), + Value: []*descriptorpb.EnumValueDescriptorProto{ + {Name: proto.String("LEVEL_UNSPECIFIED"), Number: proto.Int32(0)}, + {Name: proto.String("LEVEL_HIGH"), Number: proto.Int32(2)}, + }, + }}, + MessageType: []*descriptorpb.DescriptorProto{{ + Name: proto.String("Holder"), + Field: []*descriptorpb.FieldDescriptorProto{{ + Name: proto.String("level"), Number: proto.Int32(1), + Type: descriptorpb.FieldDescriptorProto_TYPE_ENUM.Enum(), + TypeName: proto.String(".enumtest.Level"), + JsonName: proto.String("level"), + }}, + }}, + }, nil) + if err != nil { + t.Fatal(err) + } + fd := file.Messages().Get(0).Fields().Get(0) + if got := scalar(fd, protoreflect.ValueOfEnum(2)); got != "LEVEL_HIGH" { + t.Fatalf("scalar(enum 2) = %q, want LEVEL_HIGH", got) + } + // A number with no name falls back to the number. + if got := scalar(fd, protoreflect.ValueOfEnum(7)); got != "7" { + t.Fatalf("scalar(enum 7) = %q", got) + } +} diff --git a/packages/stack-encrypt/tests/common/mod.rs b/packages/stack-encrypt/tests/common/mod.rs index fcf2a984e..8f0af0ea9 100644 --- a/packages/stack-encrypt/tests/common/mod.rs +++ b/packages/stack-encrypt/tests/common/mod.rs @@ -220,107 +220,10 @@ pub async fn recording_cipher() -> (StackCipher, Arc Self { - Self { - seed, - counter: std::sync::atomic::AtomicU64::new(0), - index: FakeDataKeySource::new(), - } - } - - fn derive(&self, what: &str, descriptor: &str, salt: &[u8]) -> [u8; 32] { - use sha2::{Digest, Sha256}; - let mut hasher = Sha256::new(); - hasher.update(self.seed); - hasher.update(what.as_bytes()); - hasher.update([0u8]); - hasher.update(descriptor.as_bytes()); - hasher.update([0u8]); - hasher.update(salt); - hasher.finalize().into() - } -} - -impl DataKeySource for DeterministicSource { - async fn generate_keys( - &self, - payloads: Vec>, - _keyset_id: Option, - _unverified_context: Option>, - ) -> Result, stack_kms::Error> { - Ok(payloads - .into_iter() - .map(|payload| { - let n = self.counter.fetch_add(1, AtomicOrdering::SeqCst); - let iv_bytes = self.derive("iv", payload.descriptor, &n.to_le_bytes()); - let mut iv = stack_kms::Iv::default(); - let width = iv.len(); - iv.copy_from_slice(&iv_bytes[..width]); - let key = self.derive("key", payload.descriptor, &iv); - let tag = self.derive("tag", payload.descriptor, &iv); - DataKeyWithTag { - key: DataKey { iv, key }, - tag: tag.to_vec(), - decryption_policy: payload.decryption_policy, - } - }) - .collect()) - } - - async fn retrieve_keys( - &self, - payloads: Vec>, - _keyset_id: Option, - _unverified_context: Option<&UnverifiedContext>, - ) -> Result, stack_kms::Error> { - payloads - .iter() - .map(|payload| { - let iv: stack_kms::Iv = *payload.iv.as_ref(); - let tag = self.derive("tag", payload.descriptor, &iv); - if tag[..] != *payload.tag { - return Err(stack_kms::Error::RetrieveKey( - stack_kms::RetrieveKeyError::FailedRetrieval( - "the tag is not this descriptor's".to_string(), - ), - )); - } - let key = self.derive("key", payload.descriptor, &iv); - Ok(DataKey { iv, key }) - }) - .collect() - } -} - -impl IndexKeySource for DeterministicSource { - async fn load_index_key( - &self, - keyset_id: Option, - ) -> Result<(Uuid, IndexKey), stack_kms::Error> { - self.index.load_index_key(keyset_id).await - } -} +/// The deterministic source `stack-kms` ships behind `test-support`: the one +/// definition the record fixture, this suite and the Go guest's +/// `deterministic-kms` test build all read, so they cannot drift apart. +pub use stack_kms::DeterministicSource; /// A cipher over [`DeterministicSource`] with `seed`. pub async fn deterministic_cipher(seed: [u8; 32]) -> StackCipher { diff --git a/packages/stack-guest-abi/src/lib.rs b/packages/stack-guest-abi/src/lib.rs index 7374c78f7..48f9b9966 100644 --- a/packages/stack-guest-abi/src/lib.rs +++ b/packages/stack-guest-abi/src/lib.rs @@ -31,7 +31,7 @@ //! # The guest ABI the Go binding's WASI guests share //! //! The Go binding reaches Rust through WASI modules run by wazero: the -//! crypto guest (`bindings/go/encrypt/guest`, `stack-encrypt` over a +//! crypto guest (`languages/golang/encrypt/guest`, `stack-encrypt` over a //! host-provided transport) and, per ADR-0005, the credential guest //! (`stack-profile` and `stack-auth`). Everything a guest needs that is //! *not* about what it does — how the host gets bytes in and out, how a diff --git a/packages/stack-kms/src/key_source.rs b/packages/stack-kms/src/key_source.rs index 70e7f00ed..30462d697 100644 --- a/packages/stack-kms/src/key_source.rs +++ b/packages/stack-kms/src/key_source.rs @@ -272,6 +272,132 @@ mod fake { #[cfg(feature = "test-support")] pub use fake::FakeDataKeySource; +/// The deterministic test source: every key derives from a seed and the +/// descriptor, so a record sealed in one process opens in another built from +/// the same seed. `stack-encrypt`'s record fixture +/// (`tests/fixtures/record_lowering.json`) is sealed under it, and the Go +/// guest's `deterministic-kms` test build runs over it, so both read one +/// definition and cannot drift apart. +#[cfg(feature = "test-support")] +mod deterministic { + use std::borrow::Cow; + use std::sync::atomic::{AtomicU64, Ordering}; + + use sha2::{Digest, Sha256}; + use uuid::Uuid; + use zerokms_protocol::{IdentifiedBy, UnverifiedContext}; + + use super::fake::FakeDataKeySource; + use super::{DataKeySource, IndexKeySource}; + use crate::errors::{Error, RetrieveKeyError}; + use crate::key::{DataKey, DataKeyWithTag, IndexKey}; + use crate::payload::{GenerateKeyPayload, RetrieveKeyPayload}; + + /// A [`DataKeySource`] whose keys are a function of a seed and the + /// descriptor, for fixtures that hold real sealed bytes. + /// + /// Every data key is `SHA-256(seed ‖ "key" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, + /// its tag `SHA-256(seed ‖ "tag" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, and the IV + /// `SHA-256(seed ‖ "iv" ‖ 0 ‖ descriptor ‖ 0 ‖ counter)[..16]`, where + /// `descriptor` is the context the leaf is sealed under as ZeroKMS + /// renders it. `retrieve_keys` re-derives the key and the tag from what + /// the leaf stores and refuses a tag that is not this descriptor's, so a + /// leaf opened under another field's label is refused as ZeroKMS would + /// refuse it. The index key is [`FakeDataKeySource`]'s, deterministic per + /// keyset, so terms are the terms every other test derives. + /// + /// A test double, not a cipher: the derivation is SHA-256 over + /// concatenated parts and models nothing of ZeroKMS beyond determinism. + pub struct DeterministicSource { + seed: [u8; 32], + counter: AtomicU64, + index: FakeDataKeySource, + } + + impl DeterministicSource { + /// A source over `seed`. + pub fn new(seed: [u8; 32]) -> Self { + Self { + seed, + counter: AtomicU64::new(0), + index: FakeDataKeySource::new(), + } + } + + fn derive(&self, what: &str, descriptor: &str, salt: &[u8]) -> [u8; 32] { + let mut hasher = Sha256::new(); + hasher.update(self.seed); + hasher.update(what.as_bytes()); + hasher.update([0u8]); + hasher.update(descriptor.as_bytes()); + hasher.update([0u8]); + hasher.update(salt); + hasher.finalize().into() + } + } + + impl DataKeySource for DeterministicSource { + async fn generate_keys( + &self, + payloads: Vec>, + _keyset_id: Option, + _unverified_context: Option>, + ) -> Result, Error> { + Ok(payloads + .into_iter() + .map(|payload| { + let n = self.counter.fetch_add(1, Ordering::SeqCst); + let iv_bytes = self.derive("iv", payload.descriptor, &n.to_le_bytes()); + let mut iv = crate::Iv::default(); + let width = iv.len(); + iv.copy_from_slice(&iv_bytes[..width]); + let key = self.derive("key", payload.descriptor, &iv); + let tag = self.derive("tag", payload.descriptor, &iv); + DataKeyWithTag { + key: DataKey { iv, key }, + tag: tag.to_vec(), + decryption_policy: payload.decryption_policy, + } + }) + .collect()) + } + + async fn retrieve_keys( + &self, + payloads: Vec>, + _keyset_id: Option, + _unverified_context: Option<&UnverifiedContext>, + ) -> Result, Error> { + payloads + .iter() + .map(|payload| { + let iv: crate::Iv = *payload.iv.as_ref(); + let tag = self.derive("tag", payload.descriptor, &iv); + if tag[..] != *payload.tag { + return Err(Error::RetrieveKey(RetrieveKeyError::FailedRetrieval( + "the tag is not this descriptor's".to_string(), + ))); + } + let key = self.derive("key", payload.descriptor, &iv); + Ok(DataKey { iv, key }) + }) + .collect() + } + } + + impl IndexKeySource for DeterministicSource { + async fn load_index_key( + &self, + keyset_id: Option, + ) -> Result<(Uuid, IndexKey), Error> { + self.index.load_index_key(keyset_id).await + } + } +} + +#[cfg(feature = "test-support")] +pub use deterministic::DeterministicSource; + #[cfg(all(test, feature = "test-support"))] mod tests { use super::*; diff --git a/packages/stack-kms/src/lib.rs b/packages/stack-kms/src/lib.rs index c653e5bf2..eda5e5839 100644 --- a/packages/stack-kms/src/lib.rs +++ b/packages/stack-kms/src/lib.rs @@ -128,6 +128,8 @@ pub use key::{ClientKey, DataKey, DataKeyWithTag, IndexKey, V1KeySet}; // Key source abstractions (production = `StackKms`; tests = `FakeDataKeySource`) #[cfg(feature = "test-support")] +pub use key_source::DeterministicSource; +#[cfg(feature = "test-support")] pub use key_source::FakeDataKeySource; pub use key_source::{DataKeySource, IndexKeySource}; From e739fdfabeb7b515097eb6cda64287432e8dd7c6 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 17:16:49 -0700 Subject: [PATCH 14/30] chore(stack-kms): DeterministicSource debugs opaquely; stack-encrypt drops sha2 DeterministicSource prints without its seed, as DataKey does. stack-encrypt no longer uses sha2 now that the deterministic source lives in stack-kms, and the label_segments fixture names the Go test that actually reads it. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- Cargo.lock | 1 - packages/stack-encrypt/Cargo.toml | 5 ----- packages/stack-encrypt/tests/fixtures/label_segments.json | 2 +- packages/stack-kms/Cargo.toml | 4 +++- packages/stack-kms/src/key_source.rs | 3 +++ 5 files changed, 7 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 94e2dcaeb..60687ec98 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3281,7 +3281,6 @@ dependencies = [ "cllw-ore", "serde", "serde_json", - "sha2 0.10.9", "stack-auth", "stack-encrypt-derive", "stack-kms", diff --git a/packages/stack-encrypt/Cargo.toml b/packages/stack-encrypt/Cargo.toml index 64100be44..ea9c805d6 100644 --- a/packages/stack-encrypt/Cargo.toml +++ b/packages/stack-encrypt/Cargo.toml @@ -64,11 +64,6 @@ test-support = ["stack-kms/test-support"] [dev-dependencies] serde_json = { workspace = true } -# The deterministic key source the record fixture is sealed under -# (`tests/common`): keys and tags derived from a seed, the descriptor and the -# IV, so a committed record opens in another process. Same version stack-kms -# uses for its fake. -sha2 = "0.10.6" # `default-features = false` here too: dev-dependency features unify into the # `cargo test -p stack-encrypt --no-default-features` graph, so leaving the # default on would silently pull `stack-kms/http` -> `stack-auth/http` -> diff --git a/packages/stack-encrypt/tests/fixtures/label_segments.json b/packages/stack-encrypt/tests/fixtures/label_segments.json index 5eef3aceb..de6cdbe48 100644 --- a/packages/stack-encrypt/tests/fixtures/label_segments.json +++ b/packages/stack-encrypt/tests/fixtures/label_segments.json @@ -1,5 +1,5 @@ { - "_comment": "One rule, two suites. A Label segment is plain exactly when the descriptor renders it verbatim. The Rust unit test in src/descriptor.rs and the Go test in languages/golang/encrypt/label_test.go both read this file, so the two implementations of the rule cannot drift apart silently.", + "_comment": "One rule, two suites. A Label segment is plain exactly when the descriptor renders it verbatim. The Rust unit test in src/descriptor.rs and the Go test in languages/golang/internal/record/fixture_test.go both read this file, so the two implementations of the rule cannot drift apart silently.", "plain": [ "users", "email_address", diff --git a/packages/stack-kms/Cargo.toml b/packages/stack-kms/Cargo.toml index 9c9d7fbb5..105a38230 100644 --- a/packages/stack-kms/Cargo.toml +++ b/packages/stack-kms/Cargo.toml @@ -27,7 +27,9 @@ http = ["dep:reqwest", "dep:lazy_static", "stack-auth/http"] profile = ["dep:stack-profile"] # Exposes `FakeDataKeySource`, a deterministic in-memory `DataKeySource` for # downstream crates (e.g. `stack-encrypt`) to unit-test encrypt/decrypt without -# ZeroKMS credentials or network access. +# ZeroKMS credentials or network access, and `DeterministicSource`, whose keys +# are a function of a seed and the descriptor, for fixtures that hold real +# sealed bytes (the stack-encrypt record fixture and the Go test guest). test-support = [] # Enables HTTP/2 in the reqwest client. ZeroKMS endpoints negotiate h2 via # ALPN; without it reqwest is HTTP/1.1-only and opens a connection per diff --git a/packages/stack-kms/src/key_source.rs b/packages/stack-kms/src/key_source.rs index 30462d697..7856123d2 100644 --- a/packages/stack-kms/src/key_source.rs +++ b/packages/stack-kms/src/key_source.rs @@ -314,6 +314,9 @@ mod deterministic { index: FakeDataKeySource, } + // Debug without the seed, which derives every key. + opaque_debug::implement!(DeterministicSource); + impl DeterministicSource { /// A source over `seed`. pub fn new(seed: [u8; 32]) -> Self { From fa106c49793960d7ae23f58271d840c14b336946 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 17:16:50 -0700 Subject: [PATCH 15/30] fix(golang)!: index-only fields and nil interface passthroughs round-trip The second cipherstash-bot review on #1094: - Decrypt of a type with an index-only field (`index=` without `encrypt`) failed for every row: nothing opens for it, and Value asked for it. The generated Value now skips it, so it keeps its zero value. - A nil interface in a passthrough field (`any`, `error`) failed Encrypt and Decrypt: a nil interface asserts to no type. Passthrough and Get return the zero T for it. - Open on a nil *Cipher or *Client returns ErrEncoding instead of panicking. - A nil element of a pointer message type is an error, not a panic. - An opaque struct with NaN or an infinity is ErrEncoding, and the README says floats there must be finite. - Generate takes ctx first (WithContext is gone) and closes the guest engine it starts. - stashgen refuses a file that would redeclare a name the package declares. - protosource refuses a message with a oneof; a proto3 optional stays a fact. - Declaration.Identity and add no longer write into a slice an earlier Declaration shares. - Seal passes the unextended plan to locateTermFailure, so a probe is not extended twice. New tests: an index-only type and nil any/error passthroughs round-trip over the deterministic guest; a renamed field opens under its Identity and not without it; empty batches, nil []byte, NaN, nil Decrypter; refusal cases for model double-binding, a model naming no field, two opaque fields writing one JSON key, a package name clash, duplicate facts and an unparsable decision. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/cmd/stashgen/README.md | 7 +- languages/golang/encrypt/gensupport/codec.go | 18 +- .../golang/encrypt/gensupport/declaration.go | 7 +- .../encrypt/internal/testusers/kinds.go | 14 ++ .../encrypt/internal/testusers/kinds_stash.go | 28 ++- .../internal/testusers/lookup_stash.go | 161 ++++++++++++++++++ languages/golang/encrypt/kinds_test.go | 141 +++++++++++++++ .../encrypt/policy/protosource/protosource.go | 5 + .../policy/protosource/protosource_test.go | 50 ++++++ languages/golang/encrypt/records.go | 14 +- languages/golang/stashgen/emit.go | 11 ++ languages/golang/stashgen/export_test.go | 2 +- languages/golang/stashgen/generate.go | 48 ++++++ languages/golang/stashgen/policy.go | 27 +-- languages/golang/stashgen/policy_test.go | 11 +- languages/golang/stashgen/refusal_test.go | 4 + .../foreign/individualstash_stash.go.golden | 3 + .../policy_individual_stash.go.golden | 3 + 18 files changed, 529 insertions(+), 25 deletions(-) create mode 100644 languages/golang/encrypt/internal/testusers/lookup_stash.go diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 39dbc1538..fe26a9dee 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -67,7 +67,7 @@ The first part of a tag is the field's name, which is the column name in a datab | `stash:"email,encrypt_into=TextEq"` | seal the field into one EQL value, with the terms that EQL type has | | `stash:"notes,encrypt"` | seal the field, with no index | | `stash:"email,encrypt,index=equality;match"` | seal the field, and derive each index beside it | -| `stash:"attrs,index=json"` | derive the index alone | +| `stash:"attrs,index=json"` | derive the index alone; no ciphertext is stored, so `Decrypt` leaves the field at its zero value | | `stash:"id,passthrough"` | store the field as it is | | `stash:"-"` | leave the field out | | `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value | @@ -163,7 +163,7 @@ var Individuals = policy.ForMessage(&pb.Individual{}, policy.Context("individual //go:generate go run ../cmd/genencrypt func main() { - err := stashgen.Generate(protosource.New(), rules.Individuals, + err := stashgen.Generate(context.Background(), protosource.New(), rules.Individuals, stashgen.Output("../individuals/individual_stash.go")) ... } @@ -186,7 +186,10 @@ No warning, error or log line holds a plaintext value. Generated code sends the engine every sealed field with its value, under the declaration lowered to data: each field's label (`/`), its outputs and its wire type (`int64`, `string`, `bytes`, ...), which `stashgen` chose from the field's Go type. A passthrough field stays on the host: the engine does nothing to it a program could observe, and the FFI codec cannot carry every Go type a program stores beside a ciphertext. An `opaque` struct crosses as one JSON document and is one column, decoded back into the exact Go types the struct declares; a field may be any type `encoding/json` carries both ways (a scalar or a type defined over one, `[]byte`, slices, maps with string or integer keys, pointers, structs whose fields are all exported, `time.Time`). +A float in an opaque struct must be finite: `encoding/json` refuses NaN and the infinities, so `Encrypt` fails for the batch with `encrypt.ErrEncoding`. A sealed float field outside an opaque struct carries them. A sealed field outside an opaque struct is one scalar, or a type defined over one; a struct, slice or map field seals only inside an opaque struct. +A nil `[]byte` in a sealed field, or a nil slice of a type defined over `[]byte`, comes back from `Decrypt` as an empty, non-nil slice; inside an opaque struct, nil comes back as nil. +A protobuf message with a `oneof` cannot be generated from a policy: protoc-gen-go puts its members in wrapper types, not in the message struct, and `Generate` refuses it. ## Status diff --git a/languages/golang/encrypt/gensupport/codec.go b/languages/golang/encrypt/gensupport/codec.go index de9db8bbc..df9255ae6 100644 --- a/languages/golang/encrypt/gensupport/codec.go +++ b/languages/golang/encrypt/gensupport/codec.go @@ -100,6 +100,9 @@ func (c *Codec[P, E]) Encrypt(ctx context.Context, cipher *encrypt.Cipher, value passthrough := make([]map[string]any, len(values)) for i, v := range values { vals := c.g.Source(v) + if vals == nil { + return nil, fmt.Errorf("gensupport: %s: value %d is nil", c.g.TypeName, i) + } row, keep, err := c.split(vals) if err != nil { return nil, fmt.Errorf("gensupport: %s: value %d: %w", c.g.TypeName, i, err) @@ -241,6 +244,11 @@ func Passthrough[T any](rec Record, name string) (T, error) { v, ok := o.Value.(T) if !ok { var zero T + if o.Value == nil && any(zero) == nil { + // A nil interface value asserts to no type. For an interface T + // it is the value the struct held. + return zero, nil + } return zero, fmt.Errorf("gensupport: passthrough field %q holds a %T, not a %T", name, o.Value, zero) } return v, nil @@ -263,6 +271,12 @@ func Get[T any](vals Values, name string) (T, error) { if exact, ok := v.(T); ok { return exact, nil } + if v == nil && any(out) == nil { + // A nil interface value asserts to no type. For an interface T, + // such as a passthrough error or any, it is the value the struct + // held. + return out, nil + } if err := convert(v, &out); err != nil { return out, fmt.Errorf("gensupport: field %q: %w", name, err) } @@ -317,7 +331,9 @@ func (c *RecordsCodec[P, R]) Decrypt(ctx context.Context, d encrypt.Decrypter, r func opaqueBytes(fields any) ([]byte, error) { encoded, err := json.Marshal(fields) if err != nil { - return nil, fmt.Errorf("the opaque value does not encode: %w", err) + // encoding/json refuses NaN and the infinities, which a sealed float + // field outside an opaque struct accepts. + return nil, fmt.Errorf("%w: the opaque value does not encode: %v", encrypt.ErrEncoding, err) } return encoded, nil } diff --git a/languages/golang/encrypt/gensupport/declaration.go b/languages/golang/encrypt/gensupport/declaration.go index be71e6e6d..8228890e8 100644 --- a/languages/golang/encrypt/gensupport/declaration.go +++ b/languages/golang/encrypt/gensupport/declaration.go @@ -3,6 +3,7 @@ package gensupport import ( "errors" "fmt" + "slices" "github.com/cipherstash/stack/languages/golang/encrypt" "github.com/cipherstash/stack/languages/golang/internal/record" @@ -131,6 +132,8 @@ func (d Declaration) Identity(name, identity string) Declaration { } for i := range d.fields { if d.fields[i].name == name { + // A copy, so the receiver's fields are not changed under it. + d.fields = slices.Clone(d.fields) d.fields[i].identity = identity return d } @@ -161,7 +164,9 @@ func (d Declaration) add(f field) Declaration { d.err = fmt.Errorf("gensupport: field %q: an indexed field names at least one index", f.name) return d } - d.fields = append(d.fields, f) + // Clip, so the append copies and never writes into an array that an + // earlier Declaration shares. + d.fields = append(slices.Clip(d.fields), f) return d } diff --git a/languages/golang/encrypt/internal/testusers/kinds.go b/languages/golang/encrypt/internal/testusers/kinds.go index e2429366a..e302f04d1 100644 --- a/languages/golang/encrypt/internal/testusers/kinds.go +++ b/languages/golang/encrypt/internal/testusers/kinds.go @@ -68,6 +68,20 @@ type Kinds struct { N sql.NullTime `stash:"n,passthrough"` In Inner `stash:"in,passthrough"` Sk []Inner `stash:"sk,passthrough"` + A any `stash:"a,passthrough"` + Er error `stash:"er,passthrough"` +} + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Lookup -name Lookup + +// Lookup has index-only fields: each stores its terms and no ciphertext, so +// Decrypt leaves it at its zero value and still opens the sealed field. +type Lookup struct { + _ struct{} `stash:"context=lookups"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt"` + Score int32 `stash:"score,index=ore"` + Code Status `stash:"code,index=equality"` } //go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Everything -name Everything diff --git a/languages/golang/encrypt/internal/testusers/kinds_stash.go b/languages/golang/encrypt/internal/testusers/kinds_stash.go index 228d7cb84..d052e08d8 100644 --- a/languages/golang/encrypt/internal/testusers/kinds_stash.go +++ b/languages/golang/encrypt/internal/testusers/kinds_stash.go @@ -45,6 +45,8 @@ type EncryptedKinds struct { N sql.NullTime In Inner Sk []Inner + A any + Er error } type EncryptedKindsS struct { @@ -147,11 +149,11 @@ type EncryptedKindsD struct { } func (e EncryptedKinds) String() string { - return gensupport.Redacted("EncryptedKinds", map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc", "St", "D") + return gensupport.Redacted("EncryptedKinds", map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk, "A": e.A, "Er": e.Er}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc", "St", "D") } func (e EncryptedKinds) LogValue() slog.Value { - return gensupport.RedactedLog(map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc", "St", "D") + return gensupport.RedactedLog(map[string]any{"P": e.P, "M": e.M, "T": e.T, "N": e.N, "In": e.In, "Sk": e.Sk, "A": e.A, "Er": e.Er}, "S", "E", "Bo", "I8", "I16", "I32", "I", "I64", "U8", "U16", "U32", "U", "U64", "F32", "F64", "By", "Bl", "Sc", "St", "D") } // Stops compiling when Kinds gains, loses, reorders or retypes a field. @@ -185,6 +187,8 @@ type kindsShape struct { N sql.NullTime In Inner Sk []Inner + A any + Er error } var kindsDeclaration = gensupport.Declare("kinds"). @@ -213,7 +217,9 @@ var kindsDeclaration = gensupport.Declare("kinds"). Passthrough("t"). Passthrough("n"). Passthrough("in"). - Passthrough("sk") + Passthrough("sk"). + Passthrough("a"). + Passthrough("er") var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ TypeName: "Kinds", @@ -247,6 +253,8 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ "n": v.N, "in": v.In, "sk": v.Sk, + "a": v.A, + "er": v.Er, } }, Seal: func(rec gensupport.Record) (EncryptedKinds, error) { @@ -270,6 +278,12 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ if e.Sk, err = gensupport.Passthrough[[]Inner](rec, "sk"); err != nil { return EncryptedKinds{}, err } + if e.A, err = gensupport.Passthrough[any](rec, "a"); err != nil { + return EncryptedKinds{}, err + } + if e.Er, err = gensupport.Passthrough[error](rec, "er"); err != nil { + return EncryptedKinds{}, err + } e.S = EncryptedKindsS{ Ciphertext: rec["s"].Ciphertext, Equality: rec["s"].Equality, @@ -367,6 +381,8 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ "n": {Value: e.N}, "in": {Value: e.In}, "sk": {Value: e.Sk}, + "a": {Value: e.A}, + "er": {Value: e.Er}, } }, Value: func(e EncryptedKinds, vals gensupport.Values) (Kinds, error) { @@ -460,6 +476,12 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ if v.Sk, err = gensupport.Get[[]Inner](vals, "sk"); err != nil { return Kinds{}, err } + if v.A, err = gensupport.Get[any](vals, "a"); err != nil { + return Kinds{}, err + } + if v.Er, err = gensupport.Get[error](vals, "er"); err != nil { + return Kinds{}, err + } return v, nil }, }) diff --git a/languages/golang/encrypt/internal/testusers/lookup_stash.go b/languages/golang/encrypt/internal/testusers/lookup_stash.go new file mode 100644 index 000000000..ee461c65e --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/lookup_stash.go @@ -0,0 +1,161 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Lookup prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedLookup struct { + ID int64 + Email EncryptedLookupEmail + Score EncryptedLookupScore + Code EncryptedLookupCode +} + +type EncryptedLookupEmail struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedLookupScore struct { + Ore encrypt.OreTerm +} + +type EncryptedLookupCode struct { + Equality encrypt.EqualityTerm +} + +func (e EncryptedLookup) String() string { + return gensupport.Redacted("EncryptedLookup", map[string]any{"ID": e.ID}, "Email", "Score", "Code") +} + +func (e EncryptedLookup) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Email", "Score", "Code") +} + +// Stops compiling when Lookup gains, loses, reorders or retypes a field. +var _ = lookupShape(Lookup{}) + +type lookupShape struct { + _ struct{} + ID int64 + Email string + Score int32 + Code Status +} + +var lookupDeclaration = gensupport.Declare("lookups"). + Passthrough("id"). + Encrypt("email", gensupport.String). + Index("score", gensupport.Int32, encrypt.Ore). + Index("code", gensupport.String, encrypt.Equality) + +var lookupCodec = gensupport.New(gensupport.Generated[Lookup, EncryptedLookup]{ + TypeName: "Lookup", + Declaration: lookupDeclaration, + PrintsPlaintext: true, + Source: func(v Lookup) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "email": v.Email, + "score": v.Score, + "code": v.Code, + } + }, + Seal: func(rec gensupport.Record) (EncryptedLookup, error) { + var e EncryptedLookup + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedLookup{}, err + } + e.Email = EncryptedLookupEmail{Ciphertext: rec["email"].Ciphertext} + e.Score = EncryptedLookupScore{Ore: rec["score"].Ore} + e.Code = EncryptedLookupCode{Equality: rec["code"].Equality} + return e, nil + }, + Open: func(e EncryptedLookup) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "email": {Ciphertext: e.Email.Ciphertext}, + "score": {Ore: e.Score.Ore}, + "code": {Equality: e.Code.Equality}, + } + }, + Value: func(e EncryptedLookup, vals gensupport.Values) (Lookup, error) { + var v Lookup + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return Lookup{}, err + } + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return Lookup{}, err + } + return v, nil + }, +}) + +// EncryptLookup seals each Lookup in one ZeroKMS request. The result has one +// element for each input, in the same order. +func EncryptLookup(ctx context.Context, cipher *encrypt.Cipher, values []Lookup) ([]EncryptedLookup, error) { + return lookupCodec.Encrypt(ctx, cipher, values) +} + +// DecryptLookup opens each EncryptedLookup in one ZeroKMS request. +func DecryptLookup(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedLookup) ([]Lookup, error) { + return lookupCodec.Decrypt(ctx, d, encrypted) +} + +var LookupFields = struct { + Email LookupEmailField + Score LookupScoreField + Code LookupCodeField +}{ + Email: LookupEmailField{gensupport.NewField[string](lookupDeclaration, "email")}, + Score: LookupScoreField{gensupport.NewField[int32](lookupDeclaration, "score")}, + Code: LookupCodeField{gensupport.NewField[Status](lookupDeclaration, "code")}, +} + +type LookupEmailField struct { + field gensupport.Field[string] +} + +func (f LookupEmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedLookupEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedLookupEmail{Ciphertext: out.Ciphertext}, err +} + +type LookupScoreField struct { + field gensupport.Field[int32] +} + +func (f LookupScoreField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int32) (EncryptedLookupScore, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedLookupScore{Ore: out.Ore}, err +} + +func (f LookupScoreField) Ore(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.OreTerm, error) { + return f.field.Ore(ctx, c, v) +} + +type LookupCodeField struct { + field gensupport.Field[Status] +} + +func (f LookupCodeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Status) (EncryptedLookupCode, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedLookupCode{Equality: out.Equality}, err +} + +func (f LookupCodeField) Equality(ctx context.Context, c *encrypt.Cipher, v Status) (encrypt.EqualityTerm, error) { + return f.field.Equality(ctx, c, v) +} diff --git a/languages/golang/encrypt/kinds_test.go b/languages/golang/encrypt/kinds_test.go index 5b610bef0..fcade46d3 100644 --- a/languages/golang/encrypt/kinds_test.go +++ b/languages/golang/encrypt/kinds_test.go @@ -4,10 +4,15 @@ import ( "bytes" "context" "database/sql" + "errors" + "math" "reflect" + "strings" "testing" "time" + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" ) @@ -55,7 +60,10 @@ func TestEverySealedKindRoundTrips(t *testing.T) { By: []byte{1, 2, 3}, Bl: testusers.Blob{4, 5}, Sc: -3, St: "active", D: 90 * time.Second, P: &seven, M: map[string]string{"k": "v"}, T: when, N: sql.NullTime{Time: when, Valid: true}, In: testusers.Inner{Name: "in", N: 1}, Sk: []testusers.Inner{{Name: "a", N: 2}}, + A: "any", Er: errKept, }, { + // A and Er are nil interfaces: a nil interface asserts to no type, + // and the passthrough still comes back as nil. S: "bob", E: "bob@example.com", I8: 127, I16: 32767, I32: 1<<31 - 1, I: 1 << 30, I64: 1<<63 - 1, U8: 255, U16: 65535, U32: 1<<32 - 1, U: 1 << 30, U64: 1<<64 - 1, F32: -0.5, F64: 1e300, By: []byte{}, Bl: testusers.Blob{}, Sc: 32767, St: "x", D: -time.Minute, @@ -152,3 +160,136 @@ func TestEveryOpaqueFieldTypeRoundTrips(t *testing.T) { t.Fatalf("DecryptEverything =\n%+v\nwant\n%+v", back, in) } } + +var errKept = errors.New("kept as it is") + +// An index-only field stores its terms and no ciphertext: Decrypt opens the +// sealed fields and leaves it at its zero value. +func TestIndexOnlyFieldsDecryptToTheirZeroValue(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + in := []testusers.Lookup{{ID: 1, Email: "alice@example.com", Score: 30, Code: "gold"}, {ID: 2, Email: "bob@example.com", Score: 10, Code: "grey"}} + encrypted, err := testusers.EncryptLookup(ctx, cipher, in) + if err != nil { + t.Fatal(err) + } + if !encrypted[1].Score.Ore.Less(encrypted[0].Score.Ore) || len(encrypted[0].Code.Equality) == 0 { + t.Fatalf("index-only terms: %+v", encrypted) + } + probe, err := testusers.LookupFields.Code.Equality(ctx, cipher, "gold") + if err != nil || !probe.Equal(encrypted[0].Code.Equality) { + t.Fatalf("Code probe: %v", err) + } + back, err := testusers.DecryptLookup(ctx, c, encrypted) + if err != nil { + t.Fatal(err) + } + want := []testusers.Lookup{{ID: 1, Email: "alice@example.com"}, {ID: 2, Email: "bob@example.com"}} + if !reflect.DeepEqual(back, want) { + t.Fatalf("DecryptLookup = %+v, want %+v", back, want) + } +} + +// A nil []byte in a sealed field comes back as an empty, non-nil slice: the +// engine carries bytes, not their absence. cmd/stashgen/README.md says so. +func TestANilSealedByteSliceComesBackEmpty(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + encrypted, err := testusers.EncryptKinds(ctx, c.DefaultKeyset(), []testusers.Kinds{{S: "carol", E: "carol@example.com"}}) + if err != nil { + t.Fatal(err) + } + back, err := testusers.DecryptKinds(ctx, c, encrypted) + if err != nil { + t.Fatal(err) + } + if back[0].By == nil || len(back[0].By) != 0 || back[0].Bl == nil || len(back[0].Bl) != 0 { + t.Fatalf("By = %#v, Bl = %#v; want empty and non-nil", back[0].By, back[0].Bl) + } +} + +// An empty batch, nil or zero-length, encrypts and decrypts to nothing. +func TestAnEmptyBatchRoundTrips(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + for _, in := range [][]testusers.User{nil, {}} { + encrypted, err := testusers.Encrypt(ctx, c.DefaultKeyset(), in) + if err != nil || len(encrypted) != 0 { + t.Fatalf("Encrypt(%#v) = %v, %v", in, encrypted, err) + } + back, err := testusers.Decrypt(ctx, c, encrypted) + if err != nil || len(back) != 0 { + t.Fatalf("Decrypt(%#v) = %v, %v", encrypted, back, err) + } + } +} + +// encoding/json refuses NaN and the infinities, so an opaque struct holding +// one is an ErrEncoding that names the value, not an unclassified error. +func TestANonFiniteFloatInAnOpaqueStructIsErrEncoding(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + for _, v := range []testusers.Everything{{F64: math.NaN()}, {F32: float32(math.Inf(1))}} { + _, err := testusers.EncryptEverything(ctx, c.DefaultKeyset(), []testusers.Everything{v}) + if !errors.Is(err, encrypt.ErrEncoding) || !strings.Contains(err.Error(), "value 0") { + t.Fatalf("err = %v, want ErrEncoding naming value 0", err) + } + } +} + +// A nil *Cipher or *Client held in a Decrypter is not a nil interface, so +// Decrypt reaches Open, which returns an error instead of panicking. +func TestDecryptThroughANilCipherOrClientIsAnError(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + encrypted, err := testusers.Encrypt(ctx, c.DefaultKeyset(), people) + if err != nil { + t.Fatal(err) + } + var cipher *encrypt.Cipher + var client *encrypt.Client + for _, d := range []encrypt.Decrypter{cipher, client} { + if _, err := testusers.Decrypt(ctx, d, encrypted); !errors.Is(err, encrypt.ErrEncoding) { + t.Fatalf("Decrypt through %T(nil): %v", d, err) + } + } +} + +// Identity keeps a renamed field's context: data sealed under the old name +// opens under the new one when the new one declares the old as its +// identity, and does not open without it. +func TestARenamedFieldOpensUnderItsIdentity(t *testing.T) { + c := deterministicClient(t) + ctx := context.Background() + type value struct{ Email string } + type sealed struct{ Email encrypt.Ciphertext } + codec := func(name string, decl gensupport.Declaration) *gensupport.Codec[value, sealed] { + return gensupport.New(gensupport.Generated[value, sealed]{ + TypeName: "value", + Declaration: decl, + Source: func(v value) gensupport.Values { return gensupport.Values{name: v.Email} }, + Seal: func(rec gensupport.Record) (sealed, error) { return sealed{rec[name].Ciphertext}, nil }, + Open: func(e sealed) gensupport.Record { return gensupport.Record{name: {Ciphertext: e.Email}} }, + Value: func(_ sealed, vals gensupport.Values) (value, error) { + s, err := gensupport.Get[string](vals, name) + return value{s}, err + }, + }) + } + before := codec("email", gensupport.Declare("users").Encrypt("email", gensupport.String)) + renamed := codec("mail", gensupport.Declare("users").Encrypt("mail", gensupport.String).Identity("mail", "email")) + unkept := codec("mail", gensupport.Declare("users").Encrypt("mail", gensupport.String)) + + encrypted, err := before.Encrypt(ctx, c.DefaultKeyset(), []value{{"alice@example.com"}}) + if err != nil { + t.Fatal(err) + } + back, err := renamed.Decrypt(ctx, c, encrypted) + if err != nil || back[0].Email != "alice@example.com" { + t.Fatalf("renamed with its identity: %v, %v", back, err) + } + if _, err := unkept.Decrypt(ctx, c, encrypted); err == nil { + t.Fatal("renamed without its identity opened the old data") + } +} diff --git a/languages/golang/encrypt/policy/protosource/protosource.go b/languages/golang/encrypt/policy/protosource/protosource.go index 4190419ee..66850fb0b 100644 --- a/languages/golang/encrypt/policy/protosource/protosource.go +++ b/languages/golang/encrypt/policy/protosource/protosource.go @@ -33,6 +33,11 @@ func (source) Facts(message any) ([]policy.Fact, error) { facts := make([]policy.Fact, 0, fields.Len()) for i := range fields.Len() { fd := fields.Get(i) + if od := fd.ContainingOneof(); od != nil && !od.IsSynthetic() { + // protoc-gen-go puts a oneof's members in wrapper types behind + // one interface field, so the struct has no field for them. + return nil, fmt.Errorf("protosource: %s: field %s is in the oneof %s, and a oneof cannot be generated", md.FullName(), fd.Name(), od.Name()) + } facts = append(facts, policy.Fact{ Message: string(md.FullName()), Name: string(fd.Name()), diff --git a/languages/golang/encrypt/policy/protosource/protosource_test.go b/languages/golang/encrypt/policy/protosource/protosource_test.go index df01118dd..a14e15fb3 100644 --- a/languages/golang/encrypt/policy/protosource/protosource_test.go +++ b/languages/golang/encrypt/policy/protosource/protosource_test.go @@ -8,6 +8,7 @@ import ( "google.golang.org/protobuf/reflect/protodesc" "google.golang.org/protobuf/reflect/protoreflect" "google.golang.org/protobuf/types/descriptorpb" + "google.golang.org/protobuf/types/dynamicpb" "github.com/cipherstash/stack/languages/golang/encrypt/policy" "github.com/cipherstash/stack/languages/golang/encrypt/policy/protosource/internal/testpb" @@ -116,3 +117,52 @@ func TestScalarSpellsAnEnumByName(t *testing.T) { t.Fatalf("scalar(enum 7) = %q", got) } } + +// A oneof's members live in wrapper types, not in the message struct, so +// Facts refuses the message by name. A proto3 optional field is a synthetic +// oneof, which protoc-gen-go makes a pointer field, and stays a fact. +func TestFactsRefuseAOneof(t *testing.T) { + str := descriptorpb.FieldDescriptorProto_TYPE_STRING.Enum() + file, err := protodesc.NewFile(&descriptorpb.FileDescriptorProto{ + Name: proto.String("oneof_test.proto"), + Package: proto.String("oneoftest"), + Syntax: proto.String("proto3"), + MessageType: []*descriptorpb.DescriptorProto{{ + Name: proto.String("Contact"), + Field: []*descriptorpb.FieldDescriptorProto{ + {Name: proto.String("nickname"), Number: proto.Int32(1), Type: str, JsonName: proto.String("nickname"), + OneofIndex: proto.Int32(1), Proto3Optional: proto.Bool(true)}, + {Name: proto.String("email"), Number: proto.Int32(2), Type: str, JsonName: proto.String("email"), OneofIndex: proto.Int32(0)}, + {Name: proto.String("phone"), Number: proto.Int32(3), Type: str, JsonName: proto.String("phone"), OneofIndex: proto.Int32(0)}, + }, + OneofDecl: []*descriptorpb.OneofDescriptorProto{{Name: proto.String("reach")}, {Name: proto.String("_nickname")}}, + }}, + }, nil) + if err != nil { + t.Fatal(err) + } + _, err = New().Facts(dynamicpb.NewMessage(file.Messages().Get(0))) + if err == nil || !strings.Contains(err.Error(), "field email is in the oneof reach") { + t.Fatalf("err = %v", err) + } + + // The synthetic oneof alone is not refused. + optional, err := protodesc.NewFile(&descriptorpb.FileDescriptorProto{ + Name: proto.String("optional_test.proto"), + Package: proto.String("optionaltest"), + Syntax: proto.String("proto3"), + MessageType: []*descriptorpb.DescriptorProto{{ + Name: proto.String("Contact"), + Field: []*descriptorpb.FieldDescriptorProto{{Name: proto.String("nickname"), Number: proto.Int32(1), Type: str, + JsonName: proto.String("nickname"), OneofIndex: proto.Int32(0), Proto3Optional: proto.Bool(true)}}, + OneofDecl: []*descriptorpb.OneofDescriptorProto{{Name: proto.String("_nickname")}}, + }}, + }, nil) + if err != nil { + t.Fatal(err) + } + facts, err := New().Facts(dynamicpb.NewMessage(optional.Messages().Get(0))) + if err != nil || len(facts) != 1 || facts[0].GoName != "Nickname" { + t.Fatalf("Facts = %v, %v", facts, err) + } +} diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go index 1970c581d..1a71a5206 100644 --- a/languages/golang/encrypt/records.go +++ b/languages/golang/encrypt/records.go @@ -59,7 +59,8 @@ func (cph *Cipher) Seal(ctx context.Context, plan *record.Plan, rows []record.So return inst.call(ctx, inst.encryptRecord, buf(encodedSource), buf(encodedPlan), buf(opts)) }) if errors.Is(err, ErrTerm) { - return nil, cph.locateTermFailure(ctx, p, rows, err) + // The plan as given: Derive applies the extension itself. + return nil, cph.locateTermFailure(ctx, plan, rows, err) } if err != nil { return nil, err @@ -86,6 +87,11 @@ func (cph *Cipher) Seal(ctx context.Context, plan *record.Plan, rows []record.So // in order. A record from another keyset is [ErrForeignKeyset]. For // generated code. func (cph *Cipher) Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) { + if cph == nil { + // A Decrypter holding a nil *Cipher is not a nil interface, so the + // generated Decrypt cannot see this; say so instead of panicking. + return nil, fmt.Errorf("%w: Open on a nil *Cipher", ErrEncoding) + } p, err := cph.plan(plan) if err != nil { return nil, err @@ -140,6 +146,9 @@ func (cph *Cipher) Derive(ctx context.Context, plan *record.Plan, field string, // each record is opened under the keyset that sealed it, with one request // per keyset. For generated code. func (c *Client) Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) { + if c == nil { + return nil, fmt.Errorf("%w: Open on a nil *Client", ErrEncoding) + } if err := plan.Validate(); err != nil { return nil, fmt.Errorf("%w: %v", ErrEncoding, err) } @@ -324,7 +333,8 @@ func (c *Client) open(ctx context.Context, sel KeysetSelector, p *record.Plan, r // derives locally. The engine defines no match term for text that yields no // token (empty, separator-only, or shorter than the n-gram), because an // empty term would match every row; a program hands such a field a value or -// drops the match index. +// drops the match index. p is the plan without this cipher's extension, +// which Derive adds. func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows []record.Source, err error) error { for i, row := range rows { for _, f := range p.Fields { diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index 32e41dee6..dbebc375b 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -312,6 +312,12 @@ func (w *writer) codec(f *genFile) { func (w *writer) source(f *genFile) { w.p("\tSource: func(v %s) gensupport.Values {", f.typeExpr) + if f.isPointer { + // A nil element is reported by Encrypt, not a panic here. + w.p("\t\tif v == nil {") + w.p("\t\t\treturn nil") + w.p("\t\t}") + } if f.decl.Opaque { entries := make([]string, len(f.opaque)) for i, g := range f.opaque { @@ -423,6 +429,11 @@ func (w *writer) value(f *genFile) { } w.p("\t\tvar err error") for _, g := range f.fields { + if g.Verb == VerbIndex { + // An index-only field stores no ciphertext, so nothing opens + // for it: the struct gets the field's zero value. + continue + } if g.Sealed() && g.definedScalar() { // The engine returns the underlying type; the conversion // to the defined type is the struct's own. diff --git a/languages/golang/stashgen/export_test.go b/languages/golang/stashgen/export_test.go index 8b9d4bee4..3cb8bb184 100644 --- a/languages/golang/stashgen/export_test.go +++ b/languages/golang/stashgen/export_test.go @@ -11,7 +11,7 @@ import ( // type instead of holding a value of it: the type lives in a module the test // process cannot import. func GenerateFor(ctx context.Context, engine Engine, notices io.Writer, output string, source policy.Source, message policy.Message, pkgPath, typeName string) error { - return generateFor(generateConfig{output: output, engine: engine, notices: notices, ctx: ctx}, source, message, pkgPath, typeName) + return generateFor(ctx, generateConfig{output: output, engine: engine, notices: notices}, source, message, pkgPath, typeName) } // MessageType is messageType for the external tests. diff --git a/languages/golang/stashgen/generate.go b/languages/golang/stashgen/generate.go index ca3ca7470..34fb40cec 100644 --- a/languages/golang/stashgen/generate.go +++ b/languages/golang/stashgen/generate.go @@ -4,6 +4,10 @@ import ( "context" "errors" "fmt" + "go/ast" + "go/parser" + "go/token" + "go/types" "os" "path/filepath" "strings" @@ -98,9 +102,53 @@ func FromTags(ctx context.Context, engine Engine, req Request) (*File, error) { if err != nil { return nil, err } + if err := checkNamesFree(pkg.Types.Scope(), gf.typeName, src); err != nil { + return nil, err + } return &File{Path: outPath, Content: src, Notices: gf.stderrNotices()}, nil } +// checkNamesFree refuses a file that declares a package-level name the +// package already declares: written, it would not compile, and the compiler +// would point at the generated file rather than at the clash. The previous +// output is loaded as its package clause only, so its names are not in scope +// and a second run is not a clash with the first. +func checkNamesFree(scope *types.Scope, typeName string, src []byte) error { + file, err := parser.ParseFile(token.NewFileSet(), "", src, parser.SkipObjectResolution) + if err != nil { + return fmt.Errorf("stashgen: the generated file does not parse: %w", err) + } + var clash []string + taken := func(name string) { + if name != "_" && name != "init" && scope.Lookup(name) != nil { + clash = append(clash, name) + } + } + for _, decl := range file.Decls { + switch d := decl.(type) { + case *ast.FuncDecl: + if d.Recv == nil { + taken(d.Name.Name) + } + case *ast.GenDecl: + for _, spec := range d.Specs { + switch sp := spec.(type) { + case *ast.TypeSpec: + taken(sp.Name.Name) + case *ast.ValueSpec: + for _, n := range sp.Names { + taken(n.Name) + } + } + } + } + } + if len(clash) > 0 { + return fieldErr(typeName, "", "the package already declares %s, which the generated file declares too; pass -name to give the file's names a prefix (-name Rows writes EncryptRows and rowsCodec)", strings.Join(clash, ", ")) + } + return nil +} + // stderrNotices are the notices in the form the generator prints. func (f *genFile) stderrNotices() []string { var out []string diff --git a/languages/golang/stashgen/policy.go b/languages/golang/stashgen/policy.go index d8d542cab..d76f66316 100644 --- a/languages/golang/stashgen/policy.go +++ b/languages/golang/stashgen/policy.go @@ -22,7 +22,6 @@ type generateConfig struct { output string engine Engine notices io.Writer - ctx context.Context } // Output names the file to write. It is required. The file goes in a package @@ -37,9 +36,6 @@ func WithEngine(e Engine) GenerateOption { return func(c *generateConfig) { c.en // WithNotices sends the generator's notices here instead of stderr. func WithNotices(w io.Writer) GenerateOption { return func(c *generateConfig) { c.notices = w } } -// WithContext runs the generator under this context. -func WithContext(ctx context.Context) GenerateOption { return func(c *generateConfig) { c.ctx = ctx } } - // Generate writes the generated file for a message from a policy: the source // gives the facts about each field, the message's rules decide each one, and // the file is the same one stashgen writes from tags. The generated functions @@ -48,8 +44,8 @@ func WithContext(ctx context.Context) GenerateOption { return func(c *generateCo // Every field of the message needs a decision. A field that no rule decides // stops the generator with the field's name and its annotations, and so does // a field that a rule refuses with Fail. -func Generate(source policy.Source, message policy.Message, opts ...GenerateOption) error { - cfg := generateConfig{notices: os.Stderr, ctx: context.Background()} +func Generate(ctx context.Context, source policy.Source, message policy.Message, opts ...GenerateOption) error { + cfg := generateConfig{notices: os.Stderr} for _, o := range opts { o(&cfg) } @@ -57,17 +53,22 @@ func Generate(source policy.Source, message policy.Message, opts ...GenerateOpti return errors.New("stashgen: Generate needs Output(path)") } if cfg.engine == nil { - e, err := GuestEngine(cfg.ctx) + e, err := GuestEngine(ctx) if err != nil { return err } + // The engine Generate starts is its own to close; one given with + // WithEngine is the caller's. + if c, ok := e.(io.Closer); ok { + defer func() { _ = c.Close() }() + } cfg.engine = e } pkgPath, typeName, err := messageType(message.Message()) if err != nil { return err } - return generateFor(cfg, source, message, pkgPath, typeName) + return generateFor(ctx, cfg, source, message, pkgPath, typeName) } // messageType finds the package path and name of the message's struct type. @@ -87,11 +88,11 @@ func messageType(message any) (pkgPath, typeName string, err error) { // generateFor is Generate once the message's type is known by name. The // tests drive it with a type in a module the test process cannot import. -func generateFor(cfg generateConfig, source policy.Source, message policy.Message, pkgPath, typeName string) error { +func generateFor(ctx context.Context, cfg generateConfig, source policy.Source, message policy.Message, pkgPath, typeName string) error { if message.Context() == "" { return fmt.Errorf("stashgen: %s: ForMessage needs a Context", typeName) } - eqlTypes, err := cfg.engine.EQLTypes(cfg.ctx) + eqlTypes, err := cfg.engine.EQLTypes(ctx) if err != nil { return fmt.Errorf("stashgen: the engine's EQL types: %w", err) } @@ -101,11 +102,11 @@ func generateFor(cfg generateConfig, source policy.Source, message policy.Messag } outDir := filepath.Dir(cfg.output) - outName, outPath, err := outputPackage(cfg.ctx, outDir) + outName, outPath, err := outputPackage(ctx, outDir) if err != nil { return err } - msgPkg, err := loadImport(cfg.ctx, outDir, pkgPath) + msgPkg, err := loadImport(ctx, outDir, pkgPath) if err != nil { return err } @@ -165,7 +166,7 @@ func generateFor(cfg generateConfig, source policy.Source, message policy.Messag if err != nil { return err } - if err := cfg.engine.Check(cfg.ctx, gf.decl); err != nil { + if err := cfg.engine.Check(ctx, gf.decl); err != nil { return err } src, err := emit(gf) diff --git a/languages/golang/stashgen/policy_test.go b/languages/golang/stashgen/policy_test.go index 8bcf53515..a8be0cb52 100644 --- a/languages/golang/stashgen/policy_test.go +++ b/languages/golang/stashgen/policy_test.go @@ -148,6 +148,13 @@ func TestGenerateRefusals(t *testing.T) { {"a struct field with no fact", policy.SourceFunc(func(any) ([]policy.Fact, error) { return []policy.Fact{{Message: "m", Name: "id", GoName: "Id", Kind: "int64"}}, nil }), individualRules(policy.Otherwise(policy.Passthrough())), "pb.Individual.Name: the source gave no fact for this field"}, + {"two facts that name one field", policy.SourceFunc(func(any) ([]policy.Fact, error) { + facts, _ := individualFacts(nil) + return append(facts, policy.Fact{Message: "m", Name: "email", GoName: "Email", Kind: "string"}), nil + }), individualRules(), "pb.Individual.Email: two facts name this field"}, + {"a decision that does not parse", policy.SourceFunc(individualFacts), + individualRules(policy.When(policy.Field("name"), policy.Encrypt(), policy.Name("name,shred"))), + "pb.Individual.Name: the policy's decision does not parse"}, {"no context", policy.SourceFunc(individualFacts), policy.ForMessage(struct{}{}, "", base), "ForMessage needs a Context"}, {"an index the engine refuses", policy.SourceFunc(individualFacts), individualRules(policy.When(policy.Field("name"), policy.EncryptIndex(policy.Match(policy.IndexOption{Key: "k", Value: "3"})))), @@ -167,7 +174,7 @@ func TestGenerateRefusals(t *testing.T) { } func TestGenerateNeedsOutputAndAMessage(t *testing.T) { - if err := stashgen.Generate(policy.SourceFunc(individualFacts), individualRules()); err == nil || !strings.Contains(err.Error(), "needs Output") { + if err := stashgen.Generate(context.Background(), policy.SourceFunc(individualFacts), individualRules()); err == nil || !strings.Contains(err.Error(), "needs Output") { t.Fatalf("err = %v", err) } type local struct{ A int } @@ -183,7 +190,7 @@ func TestGenerateNeedsOutputAndAMessage(t *testing.T) { } // With no WithEngine the embedded guest answers; the output directory // is no module, so the run stops there or, with no guest built, before. - err = stashgen.Generate(policy.SourceFunc(individualFacts), policy.ForMessage(&local{}, "c", base), stashgen.Output(filepath.Join(t.TempDir(), "x_stash.go"))) + err = stashgen.Generate(context.Background(), policy.SourceFunc(individualFacts), policy.ForMessage(&local{}, "c", base), stashgen.Output(filepath.Join(t.TempDir(), "x_stash.go"))) if err == nil { t.Fatal("a message in no module generated") } diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index 77db9649f..0bd64dd65 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -102,6 +102,10 @@ func TestRefusals(t *testing.T) { {"a model with no field for an output", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt,index=equality\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext `stash:\"email\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "", `no field for the equality term of "email"`}, {"a model field with the wrong type", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail string `stash:\"email\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "Email", "has type string, and the ciphertext of \"email\" is encrypt.Ciphertext"}, {"a model field naming an index the field lacks", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext `stash:\"email\"`\n\tEq encrypt.EqualityTerm `stash:\"email,equality\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "Eq", `field "email" has no equality index`}, + {"a model that binds one output twice", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext `stash:\"email\"`\n\tCopy encrypt.Ciphertext `stash:\"email\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "Copy", `"email" is already bound`}, + {"a model that names a field the struct lacks", user(ctx+"\tID int64 `stash:\"id,passthrough\"`\n\tEmail string `stash:\"email,encrypt\"`") + "\ntype Row struct {\n\tID int64 `stash:\"id\"`\n\tEmail encrypt.Ciphertext `stash:\"email\"`\n\tPhone encrypt.Ciphertext `stash:\"phone\"`\n}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "Phone", `declares no field named "phone"`}, + {"an opaque struct with two fields that write one name", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tUserID string\n\tUserId string"), stashgen.Request{Type: "User"}, "UserId", `two fields write the name "user_id"`}, + {"a package that already declares a name the file writes", user(ctx+"\tEmail string `stash:\"email,encrypt\"`") + "\nvar codec = \"json\"\n", stashgen.Request{Type: "User"}, "", "the package already declares codec"}, {"a model for an opaque struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tEmail string") + "\ntype Row struct{}\n", stashgen.Request{Type: "User", Models: []stashgen.ModelRequest{{Name: "Rows", Type: "Row"}}}, "", "needs no model"}, {"a type that is not a struct", "package users\n\ntype User int\n", stashgen.Request{Type: "User"}, "", "the generator reads a struct"}, {"a type the package lacks", "package users\n", stashgen.Request{Type: "User"}, "", "has no type User"}, diff --git a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden index aaca5c3fd..97f66da60 100644 --- a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden @@ -60,6 +60,9 @@ var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndivid Declaration: declaration, PrintsPlaintext: true, Source: func(v *pb.Individual) gensupport.Values { + if v == nil { + return nil + } return gensupport.Values{ "id": v.Id, "name": v.Name, diff --git a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden index aaca5c3fd..97f66da60 100644 --- a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden +++ b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden @@ -60,6 +60,9 @@ var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndivid Declaration: declaration, PrintsPlaintext: true, Source: func(v *pb.Individual) gensupport.Values { + if v == nil { + return nil + } return gensupport.Values{ "id": v.Id, "name": v.Name, From 62f223d0cd7148781ca440f267d04c245d103b2b Mon Sep 17 00:00:00 2001 From: Lindsay Holmwood Date: Wed, 7 Oct 2026 11:50:51 +1100 Subject: [PATCH 16/30] chore(golang): mark stashgen output as linguist-generated About 4,000 of this branch's added lines are machine output: the *_stash.go files stashgen writes, and the *.golden files its tests compare against. GitHub shows them expanded next to the hand-written source, which buries the code a reviewer actually needs to read. linguist-generated makes GitHub collapse them in diffs by default and leaves them out of language statistics. It changes nothing for git, CI or the build, and the files can still be expanded on demand. Every *_stash.go file in the tree carries stashgen's "DO NOT EDIT" header, so the pattern catches no hand-written file. Claude-Session: https://claude.ai/code/session_01WQQPj7Y5yASArvZaHgFpCt --- languages/golang/.gitattributes | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 languages/golang/.gitattributes diff --git a/languages/golang/.gitattributes b/languages/golang/.gitattributes new file mode 100644 index 000000000..a6c9fc2f6 --- /dev/null +++ b/languages/golang/.gitattributes @@ -0,0 +1,11 @@ +# Generated artifacts. Marked `linguist-generated` so GitHub collapses them in +# pull-request / commit diffs by default and excludes them from repository +# language statistics. +# +# - *_stash.go : stashgen output ("Code generated by stashgen. DO NOT EDIT.") +# - *.golden : stashgen's expected output, compared by its golden tests +# +# This is a GitHub display hint only: the files remain tracked, diffable, and +# reviewable on demand, and nothing about Git, CI, or the build changes. +*_stash.go linguist-generated +*.golden linguist-generated From b7a4a7297f7939d06a0bde220d51f680cd62ba0d Mon Sep 17 00:00:00 2001 From: Lindsay Holmwood Date: Wed, 7 Oct 2026 12:21:34 +1100 Subject: [PATCH 17/30] chore(golang): keep generated *_stash.go files expanded in review Go principle 10 in docs/sdk-design-principles.md says a change to which fields are encrypted shows as a change to a committed file that a reviewer reads. The *_stash.go files are that committed record, and for a type declared through a policy they are the only place a rule change shows. Marking them linguist-generated made GitHub collapse them by default, so a reviewer could pass over exactly the change the principle wants read. The *.golden files stay marked: they are the generator's test expectations, not the declarations a program ships. Claude-Session: https://claude.ai/code/session_01WQQPj7Y5yASArvZaHgFpCt --- languages/golang/.gitattributes | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/languages/golang/.gitattributes b/languages/golang/.gitattributes index a6c9fc2f6..21d155a76 100644 --- a/languages/golang/.gitattributes +++ b/languages/golang/.gitattributes @@ -1,11 +1,10 @@ -# Generated artifacts. Marked `linguist-generated` so GitHub collapses them in -# pull-request / commit diffs by default and excludes them from repository -# language statistics. +# stashgen's golden test expectations. Marked `linguist-generated` so GitHub +# collapses them in pull-request and commit diffs by default and excludes them +# from repository language statistics. This is a GitHub display hint only: +# nothing about Git, CI, or the build changes. # -# - *_stash.go : stashgen output ("Code generated by stashgen. DO NOT EDIT.") -# - *.golden : stashgen's expected output, compared by its golden tests -# -# This is a GitHub display hint only: the files remain tracked, diffable, and -# reviewable on demand, and nothing about Git, CI, or the build changes. -*_stash.go linguist-generated +# The generated *_stash.go files are deliberately NOT marked. They are the +# committed record of which fields are encrypted, with which indexes and under +# which context, so a change to them must stay expanded for the reviewer to +# read (docs/sdk-design-principles.md, Go principle 10). *.golden linguist-generated From 49e7480fcf82c879656bd5bf4954f3ded4feb14d Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 19:09:53 -0700 Subject: [PATCH 18/30] fix(golang): Decrypt errors hold no plaintext and wrap ErrEncoding The third cipherstash-bot review on #1094 (Go principle 12): when a stored value did not fit its field's Go type, the error carried the value, so a sealed uint8 that opened as 300 returned "300 does not fit a uint8". The value is decrypted plaintext, and an error is what a program logs. The same held for an opaque JSON number, a map key inside a sealed map, and encoding/json's own errors, which quote the input. Every conversion error now names types only. Get and Opaque wrap encrypt.ErrEncoding, which doc.go already promised for a stored value that does not fit its declaration, and encoding/json's errors are no longer wrapped. TestDecryptErrorsHoldNoPlaintext fails on each case against the previous code. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/encrypt/gensupport/codec.go | 10 +++-- .../golang/encrypt/gensupport/convert.go | 30 ++++++++----- .../gensupport/gensupport_internal_test.go | 43 +++++++++++++++++++ 3 files changed, 69 insertions(+), 14 deletions(-) diff --git a/languages/golang/encrypt/gensupport/codec.go b/languages/golang/encrypt/gensupport/codec.go index df9255ae6..7600a01e3 100644 --- a/languages/golang/encrypt/gensupport/codec.go +++ b/languages/golang/encrypt/gensupport/codec.go @@ -278,7 +278,7 @@ func Get[T any](vals Values, name string) (T, error) { return out, nil } if err := convert(v, &out); err != nil { - return out, fmt.Errorf("gensupport: field %q: %w", name, err) + return out, fmt.Errorf("gensupport: field %q: %w: %w", name, encrypt.ErrEncoding, err) } return out, nil } @@ -333,7 +333,8 @@ func opaqueBytes(fields any) ([]byte, error) { if err != nil { // encoding/json refuses NaN and the infinities, which a sealed float // field outside an opaque struct accepts. - return nil, fmt.Errorf("%w: the opaque value does not encode: %v", encrypt.ErrEncoding, err) + // Its error is not wrapped: encoding/json's text quotes the value. + return nil, fmt.Errorf("%w: the opaque value does not encode as JSON (NaN, an infinity, or a type encoding/json refuses)", encrypt.ErrEncoding) } return encoded, nil } @@ -348,10 +349,11 @@ func Opaque[T any](vals Values, out *T) error { } encoded, ok := v.([]byte) if !ok { - return fmt.Errorf("gensupport: the opaque value opened as %T, not bytes", v) + return fmt.Errorf("gensupport: %w: the opaque value opened as %T, not bytes", encrypt.ErrEncoding, v) } if err := json.Unmarshal(encoded, out); err != nil { - return fmt.Errorf("gensupport: the opaque value does not decode: %w", err) + // Its error is not wrapped: encoding/json's text quotes the value. + return fmt.Errorf("gensupport: %w: the opaque value does not decode into a %T", encrypt.ErrEncoding, *out) } return nil } diff --git a/languages/golang/encrypt/gensupport/convert.go b/languages/golang/encrypt/gensupport/convert.go index dfc3819d0..22162f8ea 100644 --- a/languages/golang/encrypt/gensupport/convert.go +++ b/languages/golang/encrypt/gensupport/convert.go @@ -2,6 +2,7 @@ package gensupport import ( "encoding/json" + "errors" "fmt" "math" "reflect" @@ -17,6 +18,9 @@ import ( // the same family, narrower at most. A value outside the target's range, or // of another family, is an error. // +// An error names types and positions, never the value: the value is +// decrypted plaintext, and an error is what a program logs. +// // The switch names the built-in types; a type defined over one (type Status // string, time.Duration), or a slice or map of any readable type, is read // through its underlying type by [convertVia], the one place this package @@ -84,7 +88,7 @@ func convert(v any, out any) error { *out = f case float64: if f != 0 && !math.IsInf(f, 0) && !math.IsNaN(f) && (math.Abs(f) > math.MaxFloat32 || math.Abs(f) < math.SmallestNonzeroFloat32) { - return fmt.Errorf("%v does not fit a float32", f) + return outOfRange(*out) } *out = float32(f) default: @@ -210,7 +214,7 @@ func convertVia(v any, out any) error { for key, item := range vals { slot := reflect.New(t.Elem()) if err := convert(item, slot.Interface()); err != nil { - return fmt.Errorf("entry %q: %w", key, err) + return fmt.Errorf("an entry of %s: %w", t, err) } result.SetMapIndex(reflect.ValueOf(key).Convert(t.Key()), slot.Elem()) } @@ -262,19 +266,20 @@ func widenNumber(n json.Number, out any) (any, error) { case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: i, err := strconv.ParseInt(string(n), 10, 64) if err != nil { - return nil, fmt.Errorf("%s is not an integer that fits an int64", n) + return nil, errors.New("the stored number is not an integer that fits an int64") } return i, nil case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: u, err := strconv.ParseUint(string(n), 10, 64) if err != nil { - return nil, fmt.Errorf("%s is not an integer that fits a uint64", n) + return nil, errors.New("the stored number is not an integer that fits a uint64") } return u, nil case reflect.Float32, reflect.Float64, reflect.Interface: f, err := n.Float64() if err != nil { - return nil, err + // strconv's error quotes the number. + return nil, errors.New("the stored number does not fit a float64") } return f, nil } @@ -285,6 +290,11 @@ func mismatch(v, want any) error { return fmt.Errorf("opened as %T, not %T", v, want) } +// outOfRange names the target type and not the value, which is plaintext. +func outOfRange(want any) error { + return fmt.Errorf("the stored value does not fit a %T", want) +} + // setInt writes an integer of any decoded width into a signed target, within // its range. func setInt[T ~int | ~int8 | ~int16 | ~int32 | ~int64](v any, out *T, lo, hi int64) error { @@ -298,14 +308,14 @@ func setInt[T ~int | ~int8 | ~int16 | ~int32 | ~int64](v any, out *T, lo, hi int n = int64(i) case uint64: if i > math.MaxInt64 { - return fmt.Errorf("%d does not fit a %T", i, *out) + return outOfRange(*out) } n = int64(i) default: return mismatch(v, *out) } if n < lo || n > hi { - return fmt.Errorf("%d does not fit a %T", n, *out) + return outOfRange(*out) } *out = T(n) return nil @@ -322,19 +332,19 @@ func setUint[T ~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64](v any, out *T, hi u n = i case int32: if i < 0 { - return fmt.Errorf("%d does not fit a %T", i, *out) + return outOfRange(*out) } n = uint64(i) case int64: if i < 0 { - return fmt.Errorf("%d does not fit a %T", i, *out) + return outOfRange(*out) } n = uint64(i) default: return mismatch(v, *out) } if n > hi { - return fmt.Errorf("%d does not fit a %T", n, *out) + return outOfRange(*out) } *out = T(n) return nil diff --git a/languages/golang/encrypt/gensupport/gensupport_internal_test.go b/languages/golang/encrypt/gensupport/gensupport_internal_test.go index 40b028f3d..c3dd82f39 100644 --- a/languages/golang/encrypt/gensupport/gensupport_internal_test.go +++ b/languages/golang/encrypt/gensupport/gensupport_internal_test.go @@ -2,7 +2,9 @@ package gensupport import ( "encoding/json" + "errors" "fmt" + "math" "reflect" "strings" "testing" @@ -231,6 +233,47 @@ func userCodec() *Codec[user, encryptedUser] { }) } +// A value that does not fit its Go type is decrypted plaintext: the error +// names the types and wraps ErrEncoding, and never holds the value. +func TestDecryptErrorsHoldNoPlaintext(t *testing.T) { + cases := []struct { + name string + value any + read func(Values) error + leak string + }{ + {"uint8 range", uint32(300), func(v Values) error { _, err := Get[uint8](v, "f"); return err }, "300"}, + {"int range", uint64(1<<63 + 4242), func(v Values) error { _, err := Get[int](v, "f"); return err }, "4242"}, + {"negative uint", int64(-4242), func(v Values) error { _, err := Get[uint](v, "f"); return err }, "4242"}, + {"float32 range", float64(4.242e300), func(v Values) error { _, err := Get[float32](v, "f"); return err }, "4.242"}, + {"json int", json.Number("4242.5"), func(v Values) error { _, err := Get[int64](v, "f"); return err }, "4242"}, + {"json uint", json.Number("-4242"), func(v Values) error { _, err := Get[uint64](v, "f"); return err }, "4242"}, + {"json float", json.Number("4242e999"), func(v Values) error { _, err := Get[float64](v, "f"); return err }, "4242"}, + {"map key", Values{"secret-key-4242": "x"}, func(v Values) error { _, err := Get[map[string]int](v, "f"); return err }, "4242"}, + {"opaque bytes", []byte(`{"n":4242}`), func(v Values) error { + var out struct{ N uint8 } + return Opaque(Values{OpaqueField: v["f"]}, &out) + }, "4242"}, + } + for _, tc := range cases { + err := tc.read(Values{"f": tc.value}) + if err == nil { + t.Fatalf("%s: no error", tc.name) + } + if !errors.Is(err, encrypt.ErrEncoding) { + t.Errorf("%s: %v does not wrap ErrEncoding", tc.name, err) + } + if strings.Contains(err.Error(), tc.leak) { + t.Errorf("%s: the error holds the value: %v", tc.name, err) + } + } + // encoding/json's own error ("json: unsupported value: NaN") quotes the + // value, so it is not wrapped. + if _, err := opaqueBytes(struct{ F float64 }{math.NaN()}); !errors.Is(err, encrypt.ErrEncoding) || strings.Contains(err.Error(), "json:") { + t.Errorf("opaqueBytes NaN: %v", err) + } +} + func TestSplitFailsClosedInBothDirections(t *testing.T) { c := userCodec() if c.err != nil { From 115c78334be7c639602849f4a71c32cf4b35ac71 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 19:15:16 -0700 Subject: [PATCH 19/30] fix(golang): docs say what this build does; -redact covers %#v The third cipherstash-bot review on #1094, the claims and print items: - The batch claim. Encrypt and Decrypt send one ZeroKMS request for each 500 sealed values, plus one the first time a keyset is used, not one request per call. The READMEs, doc.go and the comment stashgen writes on every generated Encrypt and Decrypt now say so. - Database libraries. No test runs database/sql, pgx, sqlx or GORM, so the docs now name the interfaces instead: Ciphertext and each term type implement driver.Valuer and sql.Scanner, one column each. - The Rust bytes. A field lowered from data seals as vitaminc's tagged leaf whatever its kind, which a Rust record opens only through a Value field. Three comments said a uint32 or string field wrote the bytes a bare Rust u32 or String does. - What this build refuses. index=json and an index with options are marked refused and listed under "When stashgen stops"; the options refusal now names the real limit (a query term uses only the default options) rather than the record plan, which carries them. The two Go-only tag words, opaque and json, are named. Only a policy sets Identity, and the README says why. A type with nothing sealed on its own gets no Fields, and the README says why. - Go 1.26, not 1.24, matching go.mod. The step 5 snippet compiles (opened, not people, and err is checked before defer). The stashgen quick start uses index= until EQL types land. The CI recipe also fails on a generated file that was never committed. - "a auth" and "a encrypt client" read "an". - Print methods. The print notice checks the signature, not only the name: a LogValue() any is not a slog.LogValuer, so slog prints every field. -redact also writes GoString, which %#v calls, and refuses a struct that already has one. testusers.Secret is generated with -redact, and TestARedactedStructPrintsNoSealedField formats it with %v, %+v, %#v, %s and slog. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/auth/doc.go | 2 +- languages/golang/auth/strategy.go | 2 +- languages/golang/cmd/stashgen/README.md | 34 +++-- languages/golang/cmd/stashgen/main.go | 2 +- languages/golang/encrypt/README.md | 25 +++- languages/golang/encrypt/ciphertext.go | 4 +- languages/golang/encrypt/client.go | 2 +- languages/golang/encrypt/credentials.go | 4 +- languages/golang/encrypt/doc.go | 25 +++- .../golang/encrypt/example/user_stash.go | 7 +- languages/golang/encrypt/gensupport/codec.go | 8 +- .../golang/encrypt/gensupport/declaration.go | 5 +- languages/golang/encrypt/guest_test.go | 2 +- .../internal/testusers/account_stash.go | 7 +- .../internal/testusers/document_stash.go | 7 +- .../internal/testusers/everything_stash.go | 8 +- .../encrypt/internal/testusers/kinds_stash.go | 7 +- .../internal/testusers/lookup_stash.go | 7 +- .../encrypt/internal/testusers/probe_stash.go | 7 +- .../internal/testusers/secret_stash.go | 122 ++++++++++++++++++ .../encrypt/internal/testusers/user_stash.go | 7 +- .../encrypt/internal/testusers/users.go | 9 ++ languages/golang/encrypt/kinds_test.go | 18 +++ languages/golang/encrypt/term.go | 6 +- languages/golang/internal/record/record.go | 8 +- languages/golang/stashgen/emit.go | 9 +- languages/golang/stashgen/engine.go | 7 +- .../golang/stashgen/enginetest/enginetest.go | 2 +- languages/golang/stashgen/policy_test.go | 2 +- languages/golang/stashgen/read.go | 35 ++++- languages/golang/stashgen/refusal_test.go | 27 +++- .../cases/accounts/account_stash.go.golden | 12 +- .../contacts/contactstash_stash.go.golden | 7 +- .../cases/documents/document_stash.go.golden | 7 +- .../cases/embedded/patient_stash.go.golden | 7 +- .../foreign/individualstash_stash.go.golden | 7 +- .../cases/orders/order_stash.go.golden | 7 +- .../cases/orders/refund_stash.go.golden | 15 ++- .../testdata/cases/users/user_stash.go.golden | 7 +- .../policy_individual_stash.go.golden | 7 +- 40 files changed, 384 insertions(+), 109 deletions(-) create mode 100644 languages/golang/encrypt/internal/testusers/secret_stash.go diff --git a/languages/golang/auth/doc.go b/languages/golang/auth/doc.go index e98bc4db2..b0bf4f16d 100644 --- a/languages/golang/auth/doc.go +++ b/languages/golang/auth/doc.go @@ -29,7 +29,7 @@ // For authentication and refresh, use [ProfileStore.AccessKey], // [ProfileStore.OIDC], [ProfileStore.DeviceSession], or [ProfileStore.Auto]. // Each returns a [Strategy], which is what encrypt.NewCredentials takes -// for the bearer token: the only way a token reaches a encrypt client. +// for the bearer token: the only way a token reaches an encrypt client. // A raw token — [ProfileStore.Token]'s included — cannot be refreshed when // it expires, and would bypass the cross-process lock a device-session // refresh holds with the CLI, so encrypt does not accept one. A diff --git a/languages/golang/auth/strategy.go b/languages/golang/auth/strategy.go index b44b494bc..6bd6a272a 100644 --- a/languages/golang/auth/strategy.go +++ b/languages/golang/auth/strategy.go @@ -50,7 +50,7 @@ func strategyConfig(opts []StrategyOption) strategyOptions { // Strategy is a Rust stack-auth strategy retained inside the credential // guest: the source of the bearer token encrypt.NewCredentials takes. // Close drops its cached credential; closing the parent profile closes all -// its strategies. It is the caller's to close: a encrypt client that +// its strategies. It is the caller's to close: an encrypt client that // was given it asks it for tokens but never closes it, so it must stay open // until the client is closed. type Strategy struct { diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index fe26a9dee..657159d57 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -8,7 +8,7 @@ Your program calls those functions, and it never builds or names a plan. ## Use the SDK 1. Add the generator to your module. - This needs Go 1.24 or later. + This needs Go 1.26 or later. ```sh go get -tool github.com/cipherstash/stack/languages/golang/cmd/stashgen @@ -21,8 +21,8 @@ Your program calls those functions, and it never builds or names a plan. type User struct { _ struct{} `stash:"context=users"` ID int64 `stash:"id,passthrough"` - Email string `stash:"email,encrypt_into=TextEq"` - Name string `stash:"name,encrypt_into=TextEq"` + Email string `stash:"email,encrypt,index=equality;match"` + Name string `stash:"name,encrypt"` } ``` @@ -39,12 +39,13 @@ Your program calls those functions, and it never builds or names a plan. ```go encrypted, err := users.Encrypt(ctx, cipher, people) - people, err := users.Decrypt(ctx, cipher, encrypted) - query, err := users.Fields.Email.Query(ctx, cipher, "bob@example.com") + opened, err := users.Decrypt(ctx, cipher, encrypted) + term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com") ``` 6. Store the encrypted type. - Each field is one column, so `database/sql`, pgx, sqlx and GORM take it as it is. + Each sealed field is one or more byte columns: `Email.Ciphertext`, `Email.Equality`, `Email.Match`. + `encrypt.Ciphertext` and each term type implement `driver.Valuer` and `sql.Scanner`, so a database library binds and scans each one as bytes; map each one to its own column. 7. Run the generator again after each change to the struct or to a tag. A change to the fields of the struct stops the build until you do. @@ -52,9 +53,11 @@ Your program calls those functions, and it never builds or names a plan. 8. In CI, run the generator and fail when a generated file changes. ```sh - go generate ./... && git diff --exit-code + go generate ./... && git diff --exit-code && test -z "$(git status --porcelain)" ``` + `git diff` sees only files git already tracks; the `git status` check also fails on a generated file that was never committed. + The rest of this file is the reference. ## Struct tags @@ -67,7 +70,8 @@ The first part of a tag is the field's name, which is the column name in a datab | `stash:"email,encrypt_into=TextEq"` | seal the field into one EQL value, with the terms that EQL type has | | `stash:"notes,encrypt"` | seal the field, with no index | | `stash:"email,encrypt,index=equality;match"` | seal the field, and derive each index beside it | -| `stash:"attrs,index=json"` | derive the index alone; no ciphertext is stored, so `Decrypt` leaves the field at its zero value | +| `stash:"score,index=ore"` | derive the index alone; no ciphertext is stored, so `Decrypt` leaves the field at its zero value | +| `stash:"attrs,index=json"` | refused in this build: the engine does not derive the `json` index yet | | `stash:"id,passthrough"` | store the field as it is | | `stash:"-"` | leave the field out | | `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value | @@ -77,7 +81,10 @@ A `match` index needs text with at least one token: the engine derives no match `Encrypt` then fails for the whole batch, with an error naming the row, the field and the index. So an optional or short value does not belong under `match`: give the field `equality` alone, or make the value required. An index takes its options in parentheses after its name, separated by commas: `index=equality;match(k=3)`. -These words are the same as the Rust API's words for the same behaviour. +This build refuses an index with options: a query term is derived with the default options only, so a stored term with other options would never match one. +These words are the same as the Rust API's words for the same behaviour, with two that only Go has. +`opaque` seals the whole struct as one `bytes` field holding a JSON document (see "What crosses the binding"). +`json` names the JSON index, which the Rust API does not declare yet; this build refuses it. An embedded struct of your own adds its tagged fields to the outer struct. An embedded struct from another package takes one tag for all of its fields: `stash:",passthrough"` stores them as they are, and `stash:"-"` leaves them out. @@ -94,8 +101,9 @@ For `-type User`, the file `user_stash.go` holds: A passthrough field keeps its Go type. A field with `encrypt_into` holds one EQL value. A field with `encrypt` or `index=` holds a struct with one field for each output, such as `Email.Ciphertext` and `Email.Equality`. -- `Encrypt` and `Decrypt`, which take a slice and return a slice, and send one ZeroKMS request for all of it. +- `Encrypt` and `Decrypt`, which take a slice and return a slice, and send one ZeroKMS request for each 500 sealed values in it, plus one the first time a keyset is used. - `Fields`, with one entry for each sealed field. + An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried. An entry encrypts one value, and it has a query method only for what the field declares: `Query` for `encrypt_into`, and `Equality`, `Match`, `Ore` or `Ope` for `index=`. - `String` and `LogValue` on `EncryptedUser`, which print the passthrough fields and hide the sealed ones. - A copy of the fields of `User`, so a change to them stops the build. @@ -111,7 +119,7 @@ Generated code uses no reflection, and no function in it panics. | `-for P.F` | `T` declares the tags for `F`, a type in another package. Each field of `T` names a field of `F` with the same name and type, and every exported field of `F` is named. | | `-model Name=R` | A model `R` for separate columns: one field for each column, each tagged with the output it holds, `stash:"email"` or `stash:"email,equality"`. Writes `EncryptName` and `DecryptName`. Any number. | | `-model Name=R:D` | The same, for an `R` that cannot carry tags. The struct `D` in your package declares them. | -| `-redact` | Write `String` and `LogValue` methods on `T`. | +| `-redact` | Write `String`, `GoString` and `LogValue` methods on `T`. | | `-output file` | The file to write. The default is the type's name in lower case, with `_stash.go`. | `stashgen` loads the package with `golang.org/x/tools/go/packages` and reads types, not text. @@ -135,6 +143,7 @@ The error names the type and the field, and never a value. - an index or an EQL type that does not apply to the field's Go type, such as `match` on an `int32`; - a field type that the engine cannot seal; - an EQL type that the engine cannot produce yet; +- an index the engine does not derive yet (`json`), or an index with options; - a `passthrough` field that has an index; - a model with a field that has no tag, or with no field for an output; - two structs in one package that would both write `Encrypt`; @@ -172,13 +181,14 @@ func main() { The first rule that matches a field decides it. Every field needs a decision: a field no rule decides stops the generator with the field's name and its annotations. `Name` sets the column name, and `Identity` keeps the field's context when its column is renamed. +Only a policy can set `Identity`; no tag spells it yet, because how a declaration changes over time is not decided. The generated file goes in a package of your own, and the functions take and return pointers to the message. ## Printing A generated type hides its sealed fields when a program prints or logs it. The struct you wrote is not protected: `stashgen` warns when it has sealed fields and no `String` and `LogValue` methods, and the program prints the same warning to stderr once for each type. -`-redact` makes `stashgen` write those two methods on the struct. +`-redact` makes `stashgen` write those two methods on the struct, and `GoString` for `%#v`. No warning, error or log line holds a plaintext value. ## What crosses the binding diff --git a/languages/golang/cmd/stashgen/main.go b/languages/golang/cmd/stashgen/main.go index 5064338d0..1371a56de 100644 --- a/languages/golang/cmd/stashgen/main.go +++ b/languages/golang/cmd/stashgen/main.go @@ -57,7 +57,7 @@ func run(args []string, dir string, stdout, stderr io.Writer, newEngine func(con fs.StringVar(&req.Name, "name", "", "write EncryptN, DecryptN and NFields instead of Encrypt, Decrypt and Fields") fs.StringVar(&req.For, "for", "", "P.F: -type declares the tags for F, a type in package P") fs.Var(&models, "model", "Name=R or Name=R:D: a model R for separate columns; writes EncryptName and DecryptName (repeatable)") - fs.BoolVar(&req.Redact, "redact", false, "write String and LogValue methods on the -type struct") + fs.BoolVar(&req.Redact, "redact", false, "write String, GoString and LogValue methods on the -type struct") fs.StringVar(&req.Output, "output", "", "the file to write (default: the type's name in lower case, with _stash.go)") fs.Usage = func() { fmt.Fprintln(stderr, "usage: stashgen -type T [-name N] [-for P.F] [-model Name=R[:D]]... [-redact] [-output file]") diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index 00caee33d..357e36371 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -14,7 +14,7 @@ The package reference is on [pkg.go.dev]; the generator's reference is ## Use the SDK -1. Add the generator to your module. This needs Go 1.24 or later. +1. Add the generator to your module. This needs Go 1.26 or later. ```sh go get -tool github.com/cipherstash/stack/languages/golang/cmd/stashgen @@ -45,17 +45,25 @@ The package reference is on [pkg.go.dev]; the generator's reference is ```go client, err := encrypt.NewClient(ctx) + if err != nil { + return err + } defer client.Close() cipher := client.Keyset(encrypt.KeysetName("tenant-42")) - encrypted, err := users.Encrypt(ctx, cipher, people) // one ZeroKMS request - people, err := users.Decrypt(ctx, cipher, encrypted) + encrypted, err := users.Encrypt(ctx, cipher, people) + opened, err := users.Decrypt(ctx, cipher, encrypted) term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com") ``` -6. Store the encrypted type. Each field is one or more byte columns, so - `database/sql`, pgx, sqlx and GORM take it as it is: - `e.Email.Ciphertext`, `e.Email.Equality`, `e.Email.Match`. + `Encrypt` and `Decrypt` send one ZeroKMS request for each 500 sealed + values in the batch, plus one request the first time a keyset is used. + +6. Store the encrypted type. Each sealed field is one or more byte columns: + `e.Email.Ciphertext`, `e.Email.Equality`, `e.Email.Match`. `Ciphertext` + and each term type implement `driver.Valuer` and `sql.Scanner`, so a + database library binds and scans each column as bytes; map each one to + its own column. 7. Run the generator again after each change to the struct or to a tag. A change to the fields of the struct stops the build until you do. @@ -63,9 +71,12 @@ The package reference is on [pkg.go.dev]; the generator's reference is 8. In CI, run the generator and fail when a generated file changes. ```sh - go generate ./... && git diff --exit-code + go generate ./... && git diff --exit-code && test -z "$(git status --porcelain)" ``` + `git diff` sees only files git already tracks; the `git status` check + also fails on a generated file that was never committed. + The rest of this file is the reference. [`example/`](example/) is the eight steps as a program. diff --git a/languages/golang/encrypt/ciphertext.go b/languages/golang/encrypt/ciphertext.go index 41b39a05e..242d85d67 100644 --- a/languages/golang/encrypt/ciphertext.go +++ b/languages/golang/encrypt/ciphertext.go @@ -15,8 +15,8 @@ import ( // vitaminc-encrypt and must never scan or marshal where one belongs. // // A sealed field is one scalar — a string, a number, a bool or a []byte, or -// a type defined over one — and seals as the typed leaf a Rust record -// derives. A struct, slice or map seals only as part of an opaque struct, +// a type defined over one — and seals as vitaminc's tagged leaf, which a +// Rust record opens when its field is a Value (not a bare u32 or String). A struct, slice or map seals only as part of an opaque struct, // which crosses as one JSON document and is one leaf; stashgen refuses it // anywhere else, because the engine would seal it as a tree of leaves and a // column holds one. diff --git a/languages/golang/encrypt/client.go b/languages/golang/encrypt/client.go index acaa16868..361106ada 100644 --- a/languages/golang/encrypt/client.go +++ b/languages/golang/encrypt/client.go @@ -100,7 +100,7 @@ func NewClient(ctx context.Context, opts ...ClientOption) (_ *Client, err error) if explicit, ok := creds.(*explicitCredentials); ok && err == nil && explicit.token == nil { // Knowable from the credentials as they were built: the one place // a missing token source is decided. - err = fmt.Errorf("%w: NewCredentials needs a auth strategy for the token", ErrEncoding) + err = fmt.Errorf("%w: NewCredentials needs an auth strategy for the token", ErrEncoding) } wasm := cfg.guest if err == nil && wasm == nil { diff --git a/languages/golang/encrypt/credentials.go b/languages/golang/encrypt/credentials.go index 5e62e6966..192d6dbcf 100644 --- a/languages/golang/encrypt/credentials.go +++ b/languages/golang/encrypt/credentials.go @@ -22,7 +22,7 @@ import ( // profile, in the Rust client's order. [NewCredentials] takes a client id, // a client key and a strategy explicitly; [OIDCFederation] mints the token // from an identity provider's. Those three are the only implementations: -// the interface is sealed, so a bearer token always comes from a auth +// the interface is sealed, so a bearer token always comes from an auth // strategy. A raw token cannot be refreshed when it expires, and a source // outside the strategies would bypass the cross-process refresh lock the // device session shares with the CLI (a refresh token used twice gets the @@ -57,7 +57,7 @@ type resolvedCredentials struct { // ClientKey is the client key. NewClient consumes it whatever the // outcome, as [NewClientKey] describes. ClientKey *ClientKey - // Token supplies the bearer token for every request: a auth + // Token supplies the bearer token for every request: an auth // strategy, outside the package's own tests. Token tokenSource // Close, when not nil, releases what the credentials hold open — the diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index 9679ce799..cd9ff82f1 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -4,7 +4,7 @@ // // # Use the SDK // -// 1. Add the generator to your module (Go 1.24 or later): +// 1. Add the generator to your module (Go 1.26 or later): // // go get -tool github.com/cipherstash/stack/languages/golang/cmd/stashgen // @@ -28,21 +28,32 @@ // 5. Call the generated functions where you write and read: // // client, err := encrypt.NewClient(ctx) +// if err != nil { +// return err +// } +// defer client.Close() // cipher := client.Keyset(encrypt.KeysetName("tenant-42")) -// encrypted, err := users.Encrypt(ctx, cipher, people) // []users.EncryptedUser, one request -// people, err := users.Decrypt(ctx, cipher, encrypted) // []users.User +// encrypted, err := users.Encrypt(ctx, cipher, people) // []users.EncryptedUser +// opened, err := users.Decrypt(ctx, cipher, encrypted) // []users.User // term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com") // -// 6. Store the encrypted type. Each field is one or more columns of bytes -// ([Ciphertext] and the term types), so database/sql, pgx, sqlx and GORM take -// it as it is. +// Encrypt and Decrypt send one ZeroKMS request for each 500 sealed values in +// the batch, plus one the first time a keyset is used. +// +// 6. Store the encrypted type. Each sealed field is one or more columns of +// bytes ([Ciphertext] and the term types), each implementing driver.Valuer +// and sql.Scanner, so a database library binds and scans each one as bytes; +// map each one to its own column. // // 7. Run the generator again after each change to the struct or to a tag. A // change to the fields of the struct stops the build until you do. // // 8. In CI, run the generator and fail when a generated file changes: // -// go generate ./... && git diff --exit-code +// go generate ./... && git diff --exit-code && test -z "$(git status --porcelain)" +// +// git diff sees only tracked files; the git status check also fails on a +// generated file that was never committed. // // The rest is the reference. The generator's own reference — the tag // grammar, the flags, what it writes and what it refuses — is in diff --git a/languages/golang/encrypt/example/user_stash.go b/languages/golang/encrypt/example/user_stash.go index f588db10d..6f7f399b8 100644 --- a/languages/golang/encrypt/example/user_stash.go +++ b/languages/golang/encrypt/example/user_stash.go @@ -109,13 +109,14 @@ var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ }, }) -// Encrypt seals each User in one ZeroKMS request. The result has one element -// for each input, in the same order. +// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values. +// The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, main []User) ([]EncryptedUser, error) { return codec.Encrypt(ctx, cipher, main) } -// Decrypt opens each EncryptedUser in one ZeroKMS request. +// Decrypt opens each EncryptedUser, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/gensupport/codec.go b/languages/golang/encrypt/gensupport/codec.go index 7600a01e3..a4cc56a74 100644 --- a/languages/golang/encrypt/gensupport/codec.go +++ b/languages/golang/encrypt/gensupport/codec.go @@ -86,7 +86,7 @@ func (c *Codec[P, E]) notices() { }) } -// Encrypt seals every value in one request. The result has one element for +// Encrypt seals every value, with one request for each 500 sealed values. The result has one element for // each value, in the same order. func (c *Codec[P, E]) Encrypt(ctx context.Context, cipher *encrypt.Cipher, values []P) ([]E, error) { c.notices() @@ -134,7 +134,7 @@ func (c *Codec[P, E]) Encrypt(ctx context.Context, cipher *encrypt.Cipher, value return out, nil } -// Decrypt opens every value in one request. +// Decrypt opens every value, with one request for each 500 sealed values. func (c *Codec[P, E]) Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []E) ([]P, error) { c.notices() if c.err != nil { @@ -314,12 +314,12 @@ type RecordsCodec[P, R any] struct { decrypt func(context.Context, encrypt.Decrypter, []R) ([]P, error) } -// Encrypt seals every value into a model row, in one request. +// Encrypt seals every value into a model row, with one request for each 500 sealed values. func (c *RecordsCodec[P, R]) Encrypt(ctx context.Context, cipher *encrypt.Cipher, values []P) ([]R, error) { return c.encrypt(ctx, cipher, values) } -// Decrypt opens every model row, in one request. +// Decrypt opens every model row, with one request for each 500 sealed values. func (c *RecordsCodec[P, R]) Decrypt(ctx context.Context, d encrypt.Decrypter, rows []R) ([]P, error) { return c.decrypt(ctx, d, rows) } diff --git a/languages/golang/encrypt/gensupport/declaration.go b/languages/golang/encrypt/gensupport/declaration.go index 8228890e8..5ea336c3c 100644 --- a/languages/golang/encrypt/gensupport/declaration.go +++ b/languages/golang/encrypt/gensupport/declaration.go @@ -10,8 +10,9 @@ import ( ) // Kind is a field's wire type: the data form of the Rust chain's `::`, -// chosen by stashgen from the field's Go type. A String or UInt32 field -// seals as the typed leaf a Rust record derives. Every sealed field has a +// chosen by stashgen from the field's Go type. It decides the terms a field +// derives; every field seals as vitaminc's tagged leaf whatever its kind, +// which a Rust record opens when its field is a Value. Every sealed field has a // scalar kind: stashgen refuses a struct, slice or map outside an opaque // struct, and an opaque struct seals as Bytes (one JSON document). Untyped // names a field with no declared type and is not what generated code writes. diff --git a/languages/golang/encrypt/guest_test.go b/languages/golang/encrypt/guest_test.go index ae3190e4c..fe35aa631 100644 --- a/languages/golang/encrypt/guest_test.go +++ b/languages/golang/encrypt/guest_test.go @@ -851,7 +851,7 @@ func TestTransportSendCounterAndResponseHeaders(t *testing.T) { func ExampleNewClient() { // With no options, NewClient uses AutoCredentials. To supply the - // credentials yourself, the token comes from a auth strategy — + // credentials yourself, the token comes from an auth strategy — // here an access key; live_test.go has a real round trip. The caller // opened the store and the strategy, and closes them after the client. ctx := context.Background() diff --git a/languages/golang/encrypt/internal/testusers/account_stash.go b/languages/golang/encrypt/internal/testusers/account_stash.go index 92f4c3f47..2ff74373b 100644 --- a/languages/golang/encrypt/internal/testusers/account_stash.go +++ b/languages/golang/encrypt/internal/testusers/account_stash.go @@ -112,13 +112,14 @@ var accountCodec = gensupport.New(gensupport.Generated[Account, EncryptedAccount }, }) -// EncryptAccount seals each Account in one ZeroKMS request. The result has one -// element for each input, in the same order. +// EncryptAccount seals each Account, with one ZeroKMS request for each 500 +// sealed values. The result has one element for each input, in the same order. func EncryptAccount(ctx context.Context, cipher *encrypt.Cipher, values []Account) ([]EncryptedAccount, error) { return accountCodec.Encrypt(ctx, cipher, values) } -// DecryptAccount opens each EncryptedAccount in one ZeroKMS request. +// DecryptAccount opens each EncryptedAccount, with one ZeroKMS request for each +// 500 sealed values. func DecryptAccount(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedAccount) ([]Account, error) { return accountCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/document_stash.go b/languages/golang/encrypt/internal/testusers/document_stash.go index 351c19547..9aaba5d79 100644 --- a/languages/golang/encrypt/internal/testusers/document_stash.go +++ b/languages/golang/encrypt/internal/testusers/document_stash.go @@ -78,13 +78,14 @@ var documentCodec = gensupport.New(gensupport.Generated[Document, EncryptedDocum }, }) -// EncryptDocument seals each Document in one ZeroKMS request. The result has -// one element for each input, in the same order. +// EncryptDocument seals each Document, with one ZeroKMS request for each 500 +// sealed values. The result has one element for each input, in the same order. func EncryptDocument(ctx context.Context, cipher *encrypt.Cipher, values []Document) ([]EncryptedDocument, error) { return documentCodec.Encrypt(ctx, cipher, values) } -// DecryptDocument opens each EncryptedDocument in one ZeroKMS request. +// DecryptDocument opens each EncryptedDocument, with one ZeroKMS request for +// each 500 sealed values. func DecryptDocument(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedDocument) ([]Document, error) { return documentCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/everything_stash.go b/languages/golang/encrypt/internal/testusers/everything_stash.go index 2caf2f81a..a69e6ed37 100644 --- a/languages/golang/encrypt/internal/testusers/everything_stash.go +++ b/languages/golang/encrypt/internal/testusers/everything_stash.go @@ -180,13 +180,15 @@ var everythingCodec = gensupport.New(gensupport.Generated[Everything, EncryptedE }, }) -// EncryptEverything seals each Everything in one ZeroKMS request. The result -// has one element for each input, in the same order. +// EncryptEverything seals each Everything, with one ZeroKMS request for each +// 500 sealed values. The result has one element for each input, in the same +// order. func EncryptEverything(ctx context.Context, cipher *encrypt.Cipher, values []Everything) ([]EncryptedEverything, error) { return everythingCodec.Encrypt(ctx, cipher, values) } -// DecryptEverything opens each EncryptedEverything in one ZeroKMS request. +// DecryptEverything opens each EncryptedEverything, with one ZeroKMS request +// for each 500 sealed values. func DecryptEverything(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedEverything) ([]Everything, error) { return everythingCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/kinds_stash.go b/languages/golang/encrypt/internal/testusers/kinds_stash.go index d052e08d8..bc4d290d5 100644 --- a/languages/golang/encrypt/internal/testusers/kinds_stash.go +++ b/languages/golang/encrypt/internal/testusers/kinds_stash.go @@ -486,13 +486,14 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ }, }) -// EncryptKinds seals each Kinds in one ZeroKMS request. The result has one -// element for each input, in the same order. +// EncryptKinds seals each Kinds, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func EncryptKinds(ctx context.Context, cipher *encrypt.Cipher, values []Kinds) ([]EncryptedKinds, error) { return kindsCodec.Encrypt(ctx, cipher, values) } -// DecryptKinds opens each EncryptedKinds in one ZeroKMS request. +// DecryptKinds opens each EncryptedKinds, with one ZeroKMS request for each 500 +// sealed values. func DecryptKinds(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedKinds) ([]Kinds, error) { return kindsCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/lookup_stash.go b/languages/golang/encrypt/internal/testusers/lookup_stash.go index ee461c65e..1e99becd3 100644 --- a/languages/golang/encrypt/internal/testusers/lookup_stash.go +++ b/languages/golang/encrypt/internal/testusers/lookup_stash.go @@ -104,13 +104,14 @@ var lookupCodec = gensupport.New(gensupport.Generated[Lookup, EncryptedLookup]{ }, }) -// EncryptLookup seals each Lookup in one ZeroKMS request. The result has one -// element for each input, in the same order. +// EncryptLookup seals each Lookup, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func EncryptLookup(ctx context.Context, cipher *encrypt.Cipher, values []Lookup) ([]EncryptedLookup, error) { return lookupCodec.Encrypt(ctx, cipher, values) } -// DecryptLookup opens each EncryptedLookup in one ZeroKMS request. +// DecryptLookup opens each EncryptedLookup, with one ZeroKMS request for each +// 500 sealed values. func DecryptLookup(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedLookup) ([]Lookup, error) { return lookupCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/probe_stash.go b/languages/golang/encrypt/internal/testusers/probe_stash.go index b17727cb9..c5ff15e9b 100644 --- a/languages/golang/encrypt/internal/testusers/probe_stash.go +++ b/languages/golang/encrypt/internal/testusers/probe_stash.go @@ -154,13 +154,14 @@ var probeCodec = gensupport.New(gensupport.Generated[Probe, EncryptedProbe]{ }, }) -// EncryptProbe seals each Probe in one ZeroKMS request. The result has one -// element for each input, in the same order. +// EncryptProbe seals each Probe, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func EncryptProbe(ctx context.Context, cipher *encrypt.Cipher, values []Probe) ([]EncryptedProbe, error) { return probeCodec.Encrypt(ctx, cipher, values) } -// DecryptProbe opens each EncryptedProbe in one ZeroKMS request. +// DecryptProbe opens each EncryptedProbe, with one ZeroKMS request for each 500 +// sealed values. func DecryptProbe(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedProbe) ([]Probe, error) { return probeCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/secret_stash.go b/languages/golang/encrypt/internal/testusers/secret_stash.go new file mode 100644 index 000000000..ba91fb8d2 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/secret_stash.go @@ -0,0 +1,122 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +type EncryptedSecret struct { + ID int64 + Value EncryptedSecretValue +} + +type EncryptedSecretValue struct { + Ciphertext encrypt.Ciphertext +} + +func (e EncryptedSecret) String() string { + return gensupport.Redacted("EncryptedSecret", map[string]any{"ID": e.ID}, "Value") +} + +func (e EncryptedSecret) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Value") +} + +// Written because of -redact: Secret no longer prints its sealed fields. +func (s Secret) String() string { + return gensupport.Redacted("Secret", map[string]any{"ID": s.ID}, "Value") +} + +func (s Secret) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": s.ID}, "Value") +} + +// GoString is what %#v prints; without it %#v prints every field. +func (s Secret) GoString() string { + return gensupport.Redacted("Secret", map[string]any{"ID": s.ID}, "Value") +} + +// Stops compiling when Secret gains, loses, reorders or retypes a field. +var _ = secretShape(Secret{}) + +type secretShape struct { + _ struct{} + ID int64 + Value string +} + +var secretDeclaration = gensupport.Declare("secrets"). + Passthrough("id"). + Encrypt("value", gensupport.String) + +var secretCodec = gensupport.New(gensupport.Generated[Secret, EncryptedSecret]{ + TypeName: "Secret", + Declaration: secretDeclaration, + Source: func(v Secret) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "value": v.Value, + } + }, + Seal: func(rec gensupport.Record) (EncryptedSecret, error) { + var e EncryptedSecret + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedSecret{}, err + } + e.Value = EncryptedSecretValue{Ciphertext: rec["value"].Ciphertext} + return e, nil + }, + Open: func(e EncryptedSecret) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "value": {Ciphertext: e.Value.Ciphertext}, + } + }, + Value: func(e EncryptedSecret, vals gensupport.Values) (Secret, error) { + var v Secret + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return Secret{}, err + } + if v.Value, err = gensupport.Get[string](vals, "value"); err != nil { + return Secret{}, err + } + return v, nil + }, +}) + +// EncryptSecret seals each Secret, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. +func EncryptSecret(ctx context.Context, cipher *encrypt.Cipher, values []Secret) ([]EncryptedSecret, error) { + return secretCodec.Encrypt(ctx, cipher, values) +} + +// DecryptSecret opens each EncryptedSecret, with one ZeroKMS request for each +// 500 sealed values. +func DecryptSecret(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedSecret) ([]Secret, error) { + return secretCodec.Decrypt(ctx, d, encrypted) +} + +var SecretFields = struct { + Value SecretValueField +}{ + Value: SecretValueField{gensupport.NewField[string](secretDeclaration, "value")}, +} + +type SecretValueField struct { + field gensupport.Field[string] +} + +func (f SecretValueField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedSecretValue, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedSecretValue{Ciphertext: out.Ciphertext}, err +} diff --git a/languages/golang/encrypt/internal/testusers/user_stash.go b/languages/golang/encrypt/internal/testusers/user_stash.go index ea5f8d14a..124d1a5fe 100644 --- a/languages/golang/encrypt/internal/testusers/user_stash.go +++ b/languages/golang/encrypt/internal/testusers/user_stash.go @@ -124,13 +124,14 @@ var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ }, }) -// Encrypt seals each User in one ZeroKMS request. The result has one element -// for each input, in the same order. +// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values. +// The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, testusers []User) ([]EncryptedUser, error) { return codec.Encrypt(ctx, cipher, testusers) } -// Decrypt opens each EncryptedUser in one ZeroKMS request. +// Decrypt opens each EncryptedUser, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/users.go b/languages/golang/encrypt/internal/testusers/users.go index 2ab1d2780..3e3766d82 100644 --- a/languages/golang/encrypt/internal/testusers/users.go +++ b/languages/golang/encrypt/internal/testusers/users.go @@ -47,6 +47,15 @@ type Document struct { Tags []string } +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Secret -name Secret -redact + +// Secret is generated with -redact, so it prints no sealed field. +type Secret struct { + _ struct{} `stash:"context=secrets"` + ID int64 `stash:"id,passthrough"` + Value string `stash:"value,encrypt"` +} + //go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Probe -name Probe // Probe has one field of each scalar type the ORE and OPE indexes order, for diff --git a/languages/golang/encrypt/kinds_test.go b/languages/golang/encrypt/kinds_test.go index fcade46d3..56e5e77e7 100644 --- a/languages/golang/encrypt/kinds_test.go +++ b/languages/golang/encrypt/kinds_test.go @@ -5,6 +5,8 @@ import ( "context" "database/sql" "errors" + "fmt" + "log/slog" "math" "reflect" "strings" @@ -293,3 +295,19 @@ func TestARenamedFieldOpensUnderItsIdentity(t *testing.T) { t.Fatal("renamed without its identity opened the old data") } } + +// A struct generated with -redact prints no sealed field under any verb: +// %v and %+v call String, %#v calls GoString, and slog calls LogValue. +func TestARedactedStructPrintsNoSealedField(t *testing.T) { + s := testusers.Secret{ID: 7, Value: "hunter2-plaintext"} + for _, verb := range []string{"%v", "%+v", "%#v", "%s"} { + if got := fmt.Sprintf(verb, s); strings.Contains(got, "hunter2") { + t.Errorf("%s printed the sealed field: %s", verb, got) + } + } + var buf bytes.Buffer + slog.New(slog.NewTextHandler(&buf, nil)).Info("secret", "s", s) + if strings.Contains(buf.String(), "hunter2") { + t.Errorf("slog printed the sealed field: %s", buf.String()) + } +} diff --git a/languages/golang/encrypt/term.go b/languages/golang/encrypt/term.go index 4d54115e6..b019f68fe 100644 --- a/languages/golang/encrypt/term.go +++ b/languages/golang/encrypt/term.go @@ -51,9 +51,9 @@ var ( ) // MatchOption is an option of the match index. None is defined yet: the -// engine's data plan carries the match index under its default options, and -// a non-default option has no wire form (stack-encrypt plan builder, -// Additions item 7). +// record plan can carry match options, but the guest derives a query term +// under the default options only, so a stored term with other options would +// match no query. type MatchOption interface { matchOption() } diff --git a/languages/golang/internal/record/record.go b/languages/golang/internal/record/record.go index 36c7659c4..6e5c55fb7 100644 --- a/languages/golang/internal/record/record.go +++ b/languages/golang/internal/record/record.go @@ -28,9 +28,11 @@ import ( ) // Kind is a plan field's declared type: the data form of the Rust chain's -// `::`. A field typed uint32 or string lowers to a real u32 or String -// leaf, the same bytes a Rust record derives; every other kind, and Untyped, -// seals in vitaminc's self-describing tagged encoding as one leaf. +// `::`. It decides which terms the field derives, not how it seals: +// a field lowered from data seals in vitaminc's self-describing tagged leaf +// encoding whatever its kind. A Rust record derives the same terms, and +// opens the leaf only when its field is a Value; a Rust record over a bare +// u32 or String writes a different leaf, which neither side can open. type Kind string // The kinds. The names are vitaminc's ValueKind names, frozen. diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index dbebc375b..9894fc9dd 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -154,6 +154,11 @@ func (w *writer) printMethods(f *genFile) { w.p("func (%s %s) LogValue() slog.Value {", r, f.typeName) w.p("\treturn gensupport.RedactedLog(%s%s)", shown, hidden) w.p("}") + w.nl() + w.p("// GoString is what %%#v prints; without it %%#v prints every field.") + w.p("func (%s %s) GoString() string {", r, f.typeName) + w.p("\treturn gensupport.Redacted(%q, %s%s)", f.typeName, shown, hidden) + w.p("}") } // opaqueShape writes the struct an opaque value crosses the binding as: the @@ -455,12 +460,12 @@ func (w *writer) value(f *genFile) { func (w *writer) functions(f *genFile) { w.nl() - w.b.WriteString(wrapComment(fmt.Sprintf("%s seals each %s in one ZeroKMS request. The result has one element for each input, in the same order.", f.encryptFn, f.typeName), 80)) + w.b.WriteString(wrapComment(fmt.Sprintf("%s seals each %s, with one ZeroKMS request for each 500 sealed values. The result has one element for each input, in the same order.", f.encryptFn, f.typeName), 80)) w.p("func %s(ctx context.Context, cipher *encrypt.Cipher, %s []%s) ([]%s, error) {", f.encryptFn, f.paramName, f.typeExpr, f.encName) w.p("\treturn %s.Encrypt(ctx, cipher, %s)", f.codecVar, f.paramName) w.p("}") w.nl() - w.b.WriteString(wrapComment(fmt.Sprintf("%s opens each %s in one ZeroKMS request.", f.decryptFn, f.encName), 80)) + w.b.WriteString(wrapComment(fmt.Sprintf("%s opens each %s, with one ZeroKMS request for each 500 sealed values.", f.decryptFn, f.encName), 80)) w.p("func %s(ctx context.Context, d encrypt.Decrypter, encrypted []%s) ([]%s, error) {", f.decryptFn, f.encName, f.typeExpr) w.p("\treturn %s.Decrypt(ctx, d, encrypted)", f.codecVar) w.p("}") diff --git a/languages/golang/stashgen/engine.go b/languages/golang/stashgen/engine.go index 279132247..edcf3f071 100644 --- a/languages/golang/stashgen/engine.go +++ b/languages/golang/stashgen/engine.go @@ -144,7 +144,12 @@ func lowerDeclaration(d Declaration, eqlTypes []EQLType) (*record.Plan, error) { } for _, idx := range f.Indexes { if len(idx.Options) > 0 { - return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("index %s: the engine cannot carry index options yet", idx)} + // The record plan carries match options, but the guest's + // query export derives a term under the default options + // only, so a stored term with others would match no query. + // The rule belongs in the guest (se_plan_check) once se_term + // takes options. + return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("index %s: a query term uses only the default index options, so this build refuses options", idx)} } out, ok := outputOfIndex[idx.Name] if !ok { diff --git a/languages/golang/stashgen/enginetest/enginetest.go b/languages/golang/stashgen/enginetest/enginetest.go index 10de7a3c0..5a817c87a 100644 --- a/languages/golang/stashgen/enginetest/enginetest.go +++ b/languages/golang/stashgen/enginetest/enginetest.go @@ -54,7 +54,7 @@ func (e Static) Check(ctx context.Context, d stashgen.Declaration) error { } for _, idx := range f.Indexes { if len(idx.Options) > 0 { - return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("index %s: the engine cannot carry index options yet", idx)} + return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("index %s: a query term uses only the default index options, so this build refuses options", idx)} } if err := indexApplies(idx.Name, f.GoType); err != nil { return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: err.Error()} diff --git a/languages/golang/stashgen/policy_test.go b/languages/golang/stashgen/policy_test.go index a8be0cb52..e33c0afc6 100644 --- a/languages/golang/stashgen/policy_test.go +++ b/languages/golang/stashgen/policy_test.go @@ -158,7 +158,7 @@ func TestGenerateRefusals(t *testing.T) { {"no context", policy.SourceFunc(individualFacts), policy.ForMessage(struct{}{}, "", base), "ForMessage needs a Context"}, {"an index the engine refuses", policy.SourceFunc(individualFacts), individualRules(policy.When(policy.Field("name"), policy.EncryptIndex(policy.Match(policy.IndexOption{Key: "k", Value: "3"})))), - "pb.Individual.Name: index match(k=3): the engine cannot carry index options"}, + "pb.Individual.Name: index match(k=3): a query term uses only the default index options"}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index bc6032741..d9b710709 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -228,7 +228,7 @@ func (r *reader) build(collected *collected, typeName string, valueNamed *types. } f.redact = true f.redactRecv = strings.ToLower(req.Type[:1]) - for _, m := range []string{"String", "LogValue"} { + for _, m := range []string{"String", "GoString", "LogValue"} { if hasMethod(types.NewPointer(tagged), m) { return nil, fmt.Errorf("stashgen: -redact: %s already has a %s method", req.Type, m) } @@ -272,7 +272,7 @@ func (r *reader) build(collected *collected, typeName string, valueNamed *types. if f.decl.Opaque { sealedCount = 1 } - if sealedCount > 0 && !f.redact && (!hasMethod(valueNamed, "String") || !hasMethod(valueNamed, "LogValue")) { + if sealedCount > 0 && !f.redact && (!isStringer(valueNamed) || !isLogValuer(valueNamed)) { f.printsPlaintext = true if foreign { f.notices = append(f.notices, fmt.Sprintf("%s prints its sealed fields in the clear, and stashgen cannot add print methods to a type from another package.", typeName)) @@ -475,6 +475,37 @@ func hasMethod(t types.Type, name string) bool { return false } +// isStringer is true when t has fmt.Stringer's method, signature included: +// a String with another signature is not what fmt calls. +func isStringer(t types.Type) bool { + sig := methodSig(t, "String") + return sig != nil && sig.Params().Len() == 0 && sig.Results().Len() == 1 && + types.Identical(sig.Results().At(0).Type(), types.Typ[types.String]) +} + +// isLogValuer is true when t has slog.LogValuer's method, signature +// included: a LogValue returning anything but slog.Value is not what slog +// calls, and slog then prints every field. +func isLogValuer(t types.Type) bool { + sig := methodSig(t, "LogValue") + if sig == nil || sig.Params().Len() != 0 || sig.Results().Len() != 1 { + return false + } + named, ok := sig.Results().At(0).Type().(*types.Named) + return ok && named.Obj().Pkg() != nil && named.Obj().Pkg().Path() == "log/slog" && named.Obj().Name() == "Value" +} + +func methodSig(t types.Type, name string) *types.Signature { + ms := types.NewMethodSet(t) + for i := range ms.Len() { + if ms.At(i).Obj().Name() == name { + sig, _ := ms.At(i).Type().(*types.Signature) + return sig + } + } + return nil +} + func shapeOf(st *types.Struct, typeExpr func(types.Type) string) []shapeField { out := make([]shapeField, 0, st.NumFields()) for i := range st.NumFields() { diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index 0bd64dd65..51ca412bc 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -84,7 +84,7 @@ func TestRefusals(t *testing.T) { {"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "has no EQL type TextMatch"}, {"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"}, {"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"}, - {"an index option the engine cannot carry", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "cannot carry index options"}, + {"an index option a query term cannot use", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "a query term uses only the default index options"}, {"a passthrough field that has an index", user(ctx + "\tID int64 `stash:\"id,passthrough,index=equality\"`"), stashgen.Request{Type: "User"}, "ID", "passthrough field has no index"}, {"an embedded struct from another package with no tag", user(ctx + "\tgorm.Model\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Model", "cannot carry tags; tag the field"}, {"an embedded struct with a verb other than passthrough", user(ctx + "\tgorm.Model `stash:\",encrypt\"`\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "Model", "takes only passthrough"}, @@ -201,8 +201,13 @@ func TestOtherLibrariesTagsAreCopied(t *testing.T) { } } +// withSlog adds log/slog to user's imports. +func withSlog(src string) string { + return strings.Replace(src, "\t\"time\"\n", "\t\"log/slog\"\n\t\"time\"\n", 1) +} + func TestAStructWithPrintMethodsGetsNoNotice(t *testing.T) { - src := user(ctx+"\tEmail string `stash:\"email,encrypt\"`") + "\nfunc (User) String() string { return \"\" }\nfunc (User) LogValue() any { return nil }\n" + src := withSlog(user(ctx+"\tEmail string `stash:\"email,encrypt\"`")) + "\nfunc (User) String() string { return \"\" }\nfunc (User) LogValue() slog.Value { return slog.Value{} }\n" file, err := generate(t, src, stashgen.Request{Type: "User"}) if err != nil { t.Fatal(err) @@ -215,6 +220,24 @@ func TestAStructWithPrintMethodsGetsNoNotice(t *testing.T) { } } +// A method named String or LogValue with another signature is not what fmt +// or slog calls, so the struct still prints its sealed fields. +func TestAPrintMethodWithTheWrongSignatureStillGetsANotice(t *testing.T) { + for name, methods := range map[string]string{ + "LogValue any": "\nfunc (User) String() string { return \"\" }\nfunc (User) LogValue() any { return nil }\n", + "String []byte": "\nfunc (User) String() []byte { return nil }\nfunc (User) LogValue() slog.Value { return slog.Value{} }\n", + } { + src := withSlog(user(ctx+"\tEmail string `stash:\"email,encrypt\"`")) + methods + file, err := generate(t, src, stashgen.Request{Type: "User"}) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(file.Content), "PrintsPlaintext: true") { + t.Errorf("%s: PrintsPlaintext not set", name) + } + } +} + // The refusals hold against the engine stashgen ships with, not only the // fake: the embedded guest, through GuestEngine. Skips only when the guest // is not built. The cases are the ones a reviewer found the two engines diff --git a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden index 0d08a56fd..adf627185 100644 --- a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden @@ -41,6 +41,11 @@ func (a Account) LogValue() slog.Value { return gensupport.RedactedLog(map[string]any{"Model": a.Model}, "Email") } +// GoString is what %#v prints; without it %#v prints every field. +func (a Account) GoString() string { + return gensupport.Redacted("Account", map[string]any{"Model": a.Model}, "Email") +} + // Stops compiling when Account gains, loses, reorders or retypes a field. var _ = accountShape(Account{}) @@ -122,13 +127,14 @@ var codec = gensupport.New(gensupport.Generated[Account, EncryptedAccount]{ }, }) -// Encrypt seals each Account in one ZeroKMS request. The result has one element -// for each input, in the same order. +// Encrypt seals each Account, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, accounts []Account) ([]EncryptedAccount, error) { return codec.Encrypt(ctx, cipher, accounts) } -// Decrypt opens each EncryptedAccount in one ZeroKMS request. +// Decrypt opens each EncryptedAccount, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedAccount) ([]Account, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden index facaff393..e7a36d034 100644 --- a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden @@ -109,13 +109,14 @@ var codec = gensupport.New(gensupport.Generated[crm.Contact, EncryptedContact]{ }, }) -// Encrypt seals each crm.Contact in one ZeroKMS request. The result has one -// element for each input, in the same order. +// Encrypt seals each crm.Contact, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, contacts []crm.Contact) ([]EncryptedContact, error) { return codec.Encrypt(ctx, cipher, contacts) } -// Decrypt opens each EncryptedContact in one ZeroKMS request. +// Decrypt opens each EncryptedContact, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedContact) ([]crm.Contact, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden index e34e0e77c..80d8b7b6d 100644 --- a/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden @@ -78,13 +78,14 @@ var codec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ }, }) -// Encrypt seals each Document in one ZeroKMS request. The result has one -// element for each input, in the same order. +// Encrypt seals each Document, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, documents []Document) ([]EncryptedDocument, error) { return codec.Encrypt(ctx, cipher, documents) } -// Decrypt opens each EncryptedDocument in one ZeroKMS request. +// Decrypt opens each EncryptedDocument, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedDocument) ([]Document, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden index ce3945fce..07daea939 100644 --- a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden @@ -114,13 +114,14 @@ var codec = gensupport.New(gensupport.Generated[Patient, EncryptedPatient]{ }, }) -// Encrypt seals each Patient in one ZeroKMS request. The result has one element -// for each input, in the same order. +// Encrypt seals each Patient, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, embedded []Patient) ([]EncryptedPatient, error) { return codec.Encrypt(ctx, cipher, embedded) } -// Decrypt opens each EncryptedPatient in one ZeroKMS request. +// Decrypt opens each EncryptedPatient, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedPatient) ([]Patient, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden index 97f66da60..fa453c9f1 100644 --- a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden @@ -120,13 +120,14 @@ var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndivid }, }) -// Encrypt seals each pb.Individual in one ZeroKMS request. The result has one -// element for each input, in the same order. +// Encrypt seals each pb.Individual, with one ZeroKMS request for each 500 +// sealed values. The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, individuals []*pb.Individual) ([]EncryptedIndividual, error) { return codec.Encrypt(ctx, cipher, individuals) } -// Decrypt opens each EncryptedIndividual in one ZeroKMS request. +// Decrypt opens each EncryptedIndividual, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedIndividual) ([]*pb.Individual, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden index 7effe1280..960ad4e05 100644 --- a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden @@ -122,13 +122,14 @@ var orderCodec = gensupport.New(gensupport.Generated[Order, EncryptedOrder]{ }, }) -// EncryptOrder seals each Order in one ZeroKMS request. The result has one -// element for each input, in the same order. +// EncryptOrder seals each Order, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func EncryptOrder(ctx context.Context, cipher *encrypt.Cipher, values []Order) ([]EncryptedOrder, error) { return orderCodec.Encrypt(ctx, cipher, values) } -// DecryptOrder opens each EncryptedOrder in one ZeroKMS request. +// DecryptOrder opens each EncryptedOrder, with one ZeroKMS request for each 500 +// sealed values. func DecryptOrder(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedOrder) ([]Order, error) { return orderCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden index cdd36738c..7f0397577 100644 --- a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden @@ -14,6 +14,9 @@ import ( // Stops compiling when the library does not accept this version of generated file. const _ = gensupport.GeneratedVersion1 +// Refund prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + type EncryptedRefund struct { ID int64 Reason eql.TextEq @@ -41,8 +44,9 @@ var declaration = gensupport.Declare("refunds"). EncryptInto("reason", gensupport.String, "TextEq") var codec = gensupport.New(gensupport.Generated[Refund, EncryptedRefund]{ - TypeName: "Refund", - Declaration: declaration, + TypeName: "Refund", + Declaration: declaration, + PrintsPlaintext: true, Source: func(v Refund) gensupport.Values { return gensupport.Values{ "id": v.ID, @@ -77,13 +81,14 @@ var codec = gensupport.New(gensupport.Generated[Refund, EncryptedRefund]{ }, }) -// Encrypt seals each Refund in one ZeroKMS request. The result has one element -// for each input, in the same order. +// Encrypt seals each Refund, with one ZeroKMS request for each 500 sealed +// values. The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, orders []Refund) ([]EncryptedRefund, error) { return codec.Encrypt(ctx, cipher, orders) } -// Decrypt opens each EncryptedRefund in one ZeroKMS request. +// Decrypt opens each EncryptedRefund, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedRefund) ([]Refund, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden index 5224ae7a4..ffff9bb1b 100644 --- a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden @@ -92,13 +92,14 @@ var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ }, }) -// Encrypt seals each User in one ZeroKMS request. The result has one element -// for each input, in the same order. +// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values. +// The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, users []User) ([]EncryptedUser, error) { return codec.Encrypt(ctx, cipher, users) } -// Decrypt opens each EncryptedUser in one ZeroKMS request. +// Decrypt opens each EncryptedUser, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden index 97f66da60..fa453c9f1 100644 --- a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden +++ b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden @@ -120,13 +120,14 @@ var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndivid }, }) -// Encrypt seals each pb.Individual in one ZeroKMS request. The result has one -// element for each input, in the same order. +// Encrypt seals each pb.Individual, with one ZeroKMS request for each 500 +// sealed values. The result has one element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, individuals []*pb.Individual) ([]EncryptedIndividual, error) { return codec.Encrypt(ctx, cipher, individuals) } -// Decrypt opens each EncryptedIndividual in one ZeroKMS request. +// Decrypt opens each EncryptedIndividual, with one ZeroKMS request for each 500 +// sealed values. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedIndividual) ([]*pb.Individual, error) { return codec.Decrypt(ctx, d, encrypted) } From 9f7e191cb90a7e4dd76207c846026a7e26fc11ce Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 19:17:13 -0700 Subject: [PATCH 20/30] fix(golang): a field added to an embedded struct stops the build; CI catches an uncommitted generated file The third cipherstash-bot review on #1094 (Go principles 1, 9 and 10): - The shape check embedded the same named type the user's struct did, so a field added to an embedded Person, or to gorm.Model, still converted. The generated code reads fields through the embedded struct one by one, so Encrypt dropped the new field with no error. stashgen now also writes a shape for each embedded struct it reads (recursively, skipping one tagged `stash:"-"`). One from another package with an unexported field cannot convert, and the generator's notice says so: the compiler finds a removed or retyped field and CI finds an added one. TestAFieldAddedToAnEmbeddedStructStopsTheBuild adds a field to the embedded case's Person and checks the stale file fails to build at the embedded shape. - The "Generated code is committed" step ran git diff, which ignores untracked files, so a //go:generate line whose output was never committed passed. It now also fails on untracked files. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- .github/workflows/tests-golang.yml | 9 +++- languages/golang/stashgen/emit.go | 15 +++++- languages/golang/stashgen/golden_test.go | 48 +++++++++++++++++++ languages/golang/stashgen/read.go | 48 +++++++++++++++++++ .../cases/accounts/account_stash.go.golden | 10 ++++ .../cases/embedded/patient_stash.go.golden | 9 ++++ 6 files changed, 136 insertions(+), 3 deletions(-) diff --git a/.github/workflows/tests-golang.yml b/.github/workflows/tests-golang.yml index db3025508..67531f441 100644 --- a/.github/workflows/tests-golang.yml +++ b/.github/workflows/tests-golang.yml @@ -201,12 +201,19 @@ jobs: # file is committed, and this fails when `go generate` would change # one. It runs the real stashgen against the guest just built, so it # also proves the generator and the engine agree on the module's own - # examples. + # examples. `git diff` sees only tracked files, so the status check + # catches a generated file that was never committed. - name: Generated code is committed working-directory: languages/golang run: | CGO_ENABLED=0 go generate ./... git diff --exit-code -- . + untracked="$(git status --porcelain -- .)" + if [ -n "$untracked" ]; then + echo "::error::go generate wrote files that are not committed:" + echo "$untracked" + exit 1 + fi # Linux only: macOS and Windows would report the same findings. Needs no # guests: the packages embed a directory and compile without them. diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index 9894fc9dd..02153f67f 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -185,8 +185,19 @@ func (w *writer) shape(f *genFile) { w.p("// Stops compiling when %s gains, loses, reorders or retypes a field.", f.typeName) w.p("var _ = %s(%s)", f.shapeName, f.shapeSource) w.nl() - w.p("type %s struct {", f.shapeName) - for _, s := range f.shapeFields { + w.shapeStruct(f.shapeName, f.shapeFields) + for _, e := range f.embeddedShapes { + w.nl() + w.p("// Stops compiling when the embedded %s gains, loses, reorders or retypes a field.", strings.TrimSuffix(e.source, "{}")) + w.p("var _ = %s(%s)", e.name, e.source) + w.nl() + w.shapeStruct(e.name, e.fields) + } +} + +func (w *writer) shapeStruct(name string, fields []shapeField) { + w.p("type %s struct {", name) + for _, s := range fields { if s.embedded { w.p("\t%s", s.typeExpr) } else { diff --git a/languages/golang/stashgen/golden_test.go b/languages/golang/stashgen/golden_test.go index 1c00fbcab..4f368ce5a 100644 --- a/languages/golang/stashgen/golden_test.go +++ b/languages/golang/stashgen/golden_test.go @@ -164,3 +164,51 @@ func diff(want, got string) string { } return b.String() } + +// A field added to an embedded struct stops the build, as one added to the +// outer struct does: the outer shape embeds the same named type, so only +// the embedded struct's own shape notices. +func TestAFieldAddedToAnEmbeddedStructStopsTheBuild(t *testing.T) { + caseDir := filepath.Join("testdata", "cases", "embedded") + golden, err := os.ReadFile(filepath.Join(caseDir, "patient_stash.go.golden")) + if err != nil { + t.Fatal(err) + } + tmp := t.TempDir() + if err := copyTree(caseDir, tmp); err != nil { + t.Fatal(err) + } + testdata, err := filepath.Abs("testdata") + if err != nil { + t.Fatal(err) + } + gomod, err := os.ReadFile(filepath.Join(tmp, "go.mod")) + if err != nil { + t.Fatal(err) + } + gomod = bytes.ReplaceAll(gomod, []byte("../../stubsdk"), []byte(filepath.Join(testdata, "stubsdk"))) + gomod = bytes.ReplaceAll(gomod, []byte("../../stubgorm"), []byte(filepath.Join(testdata, "stubgorm"))) + src, err := os.ReadFile(filepath.Join(tmp, "embedded.go")) + if err != nil { + t.Fatal(err) + } + grown := bytes.Replace(src, []byte("\tnotes string\n"), []byte("\tnotes string\n\tPhone string `stash:\"phone,encrypt\"`\n"), 1) + if bytes.Equal(grown, src) { + t.Fatal("the case's Person changed; update this test") + } + for name, data := range map[string][]byte{"go.mod": gomod, "embedded.go": grown, "patient_stash.go": golden} { + if err := os.WriteFile(filepath.Join(tmp, name), data, 0o600); err != nil { + t.Fatal(err) + } + } + cmd := exec.Command("go", "build", "./...") + cmd.Dir = tmp + cmd.Env = append(os.Environ(), "GOPROXY=off", "GOWORK=off", "GOFLAGS=-mod=mod", "CGO_ENABLED=0") + out, err := cmd.CombinedOutput() + if err == nil { + t.Fatal("the stale file still compiles after Person gained a field") + } + if !strings.Contains(string(out), "patientShapePerson") { + t.Fatalf("the build failed, but not at the embedded shape:\n%s", out) + } +} diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index d9b710709..5a5827fd4 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -32,6 +32,10 @@ type genFile struct { shapeName string shapeSource string // "User{}", "crm.Contact{}" shapeFields []shapeField + // One shape for each embedded struct the file reads fields through: + // the outer shape embeds the same named type, so a field added to it + // would still convert. + embeddedShapes []embeddedShape // Generated names. encName string // "EncryptedUser" @@ -65,6 +69,12 @@ type shapeField struct { typeExpr string } +type embeddedShape struct { + name string // "patientShapePerson" + source string // "Person{}" + fields []shapeField +} + // encMember is one member of the encrypted struct: a field, or an embedded // struct carried through from the plaintext type. type encMember struct { @@ -254,6 +264,11 @@ func (r *reader) build(collected *collected, typeName string, valueNamed *types. default: f.shapeFields = shapeOf(valueStruct, r.typeExpr) } + if f.shapeName != "" { + if notice := r.embedShapes(f, valueStruct, f.shapeName); notice != "" { + conversionNotice = notice + } + } if err := r.buildFields(collected); err != nil { return nil, err @@ -506,6 +521,39 @@ func methodSig(t types.Type, name string) *types.Signature { return nil } +// embedShapes adds a shape for each embedded struct of st the file reads, +// recursively, so a field added to one stops the build as a field added to +// the outer struct does. An embedded struct left out with `stash:"-"` is not +// read and gets none. One from another package with an unexported field +// cannot convert; the notice it returns says CI finds a field added to it. +func (r *reader) embedShapes(f *genFile, st *types.Struct, prefix string) string { + var notice string + for i := range st.NumFields() { + fld := st.Field(i) + if !fld.Embedded() { + continue + } + if t, err := parseTag(st.Tag(i)); err == nil && t.Omit { + continue + } + named, inner, ok := embeddedStruct(fld.Type()) + if !ok || named == nil { + continue + } + name := prefix + named.Obj().Name() + samePkg := named.Obj().Pkg() != nil && named.Obj().Pkg().Path() == r.pkg.PkgPath + if !samePkg && hasUnexported(inner) { + notice = fmt.Sprintf("%s has unexported fields, so Go cannot convert it to a copy of its fields: the compiler finds a removed or retyped field of it, and CI finds an added one.", r.typeExpr(named)) + continue + } + f.embeddedShapes = append(f.embeddedShapes, embeddedShape{name: name, source: r.typeExpr(named) + "{}", fields: shapeOf(inner, r.typeExpr)}) + if n := r.embedShapes(f, inner, name); n != "" { + notice = n + } + } + return notice +} + func shapeOf(st *types.Struct, typeExpr func(types.Type) string) []shapeField { out := make([]shapeField, 0, st.NumFields()) for i := range st.NumFields() { diff --git a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden index adf627185..c333d3462 100644 --- a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden @@ -57,6 +57,16 @@ type accountShape struct { token string } +// Stops compiling when the embedded gorm.Model gains, loses, reorders or retypes a field. +var _ = accountShapeModel(gorm.Model{}) + +type accountShapeModel struct { + ID uint + CreatedAt time.Time + UpdatedAt time.Time + DeletedAt gorm.DeletedAt +} + var declaration = gensupport.Declare("accounts"). Passthrough("id"). Passthrough("created_at"). diff --git a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden index 07daea939..2cbee5d7b 100644 --- a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden @@ -54,6 +54,15 @@ type patientShape struct { Chart []byte } +// Stops compiling when the embedded Person gains, loses, reorders or retypes a field. +var _ = patientShapePerson(Person{}) + +type patientShapePerson struct { + Name string + Email string + notes string +} + var declaration = gensupport.Declare("patients"). EncryptInto("name", gensupport.String, "TextEq"). EncryptIndex("email", gensupport.String, encrypt.Equality). From 8337266513b718bd03736df184f854249b19b4ba Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 19:18:45 -0700 Subject: [PATCH 21/30] fix(golang)!: the policy path checks names and takes its output as a parameter The third cipherstash-bot review on #1094 (Go principles 1 and G6): - Generate wrote its file without the name check FromTags runs, and had no way to set a name, so two messages generated into one package both declared Encrypt and the compiler reported the clash in the generated file. Generate now loads the output package (without the file it replaces) and refuses a clash, and WithName gives the names a prefix as -name does on the tag path. - The output path was a required option, found missing only at run time. It is now a parameter: stashgen.Generate(ctx, source, message, "../individuals/individual_stash.go") and Output is gone. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/cmd/stashgen/README.md | 3 +- languages/golang/stashgen/export_test.go | 7 ++- languages/golang/stashgen/generate.go | 6 +-- languages/golang/stashgen/policy.go | 63 +++++++++++++++++++----- languages/golang/stashgen/policy_test.go | 37 +++++++++++++- 5 files changed, 97 insertions(+), 19 deletions(-) diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 657159d57..6d926fccc 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -173,7 +173,7 @@ var Individuals = policy.ForMessage(&pb.Individual{}, policy.Context("individual //go:generate go run ../cmd/genencrypt func main() { err := stashgen.Generate(context.Background(), protosource.New(), rules.Individuals, - stashgen.Output("../individuals/individual_stash.go")) + "../individuals/individual_stash.go") ... } ``` @@ -183,6 +183,7 @@ Every field needs a decision: a field no rule decides stops the generator with t `Name` sets the column name, and `Identity` keeps the field's context when its column is renamed. Only a policy can set `Identity`; no tag spells it yet, because how a declaration changes over time is not decided. The generated file goes in a package of your own, and the functions take and return pointers to the message. +`stashgen.WithName("Individual")` gives the file's names a prefix, as `-name` does: the second message generated into one package needs one, and `Generate` refuses a file whose names the package already declares. ## Printing diff --git a/languages/golang/stashgen/export_test.go b/languages/golang/stashgen/export_test.go index 3cb8bb184..50f6bbc3f 100644 --- a/languages/golang/stashgen/export_test.go +++ b/languages/golang/stashgen/export_test.go @@ -11,7 +11,12 @@ import ( // type instead of holding a value of it: the type lives in a module the test // process cannot import. func GenerateFor(ctx context.Context, engine Engine, notices io.Writer, output string, source policy.Source, message policy.Message, pkgPath, typeName string) error { - return generateFor(ctx, generateConfig{output: output, engine: engine, notices: notices}, source, message, pkgPath, typeName) + return GenerateForNamed(ctx, engine, notices, output, "", source, message, pkgPath, typeName) +} + +// GenerateForNamed is GenerateFor with WithName. +func GenerateForNamed(ctx context.Context, engine Engine, notices io.Writer, output, name string, source policy.Source, message policy.Message, pkgPath, typeName string) error { + return generateFor(ctx, generateConfig{output: output, name: name, engine: engine, notices: notices}, source, message, pkgPath, typeName) } // MessageType is messageType for the external tests. diff --git a/languages/golang/stashgen/generate.go b/languages/golang/stashgen/generate.go index 34fb40cec..ae584b43d 100644 --- a/languages/golang/stashgen/generate.go +++ b/languages/golang/stashgen/generate.go @@ -102,7 +102,7 @@ func FromTags(ctx context.Context, engine Engine, req Request) (*File, error) { if err != nil { return nil, err } - if err := checkNamesFree(pkg.Types.Scope(), gf.typeName, src); err != nil { + if err := checkNamesFree(pkg.Types.Scope(), gf.typeName, src, "pass -name to give the file's names a prefix (-name Rows writes EncryptRows and rowsCodec)"); err != nil { return nil, err } return &File{Path: outPath, Content: src, Notices: gf.stderrNotices()}, nil @@ -113,7 +113,7 @@ func FromTags(ctx context.Context, engine Engine, req Request) (*File, error) { // would point at the generated file rather than at the clash. The previous // output is loaded as its package clause only, so its names are not in scope // and a second run is not a clash with the first. -func checkNamesFree(scope *types.Scope, typeName string, src []byte) error { +func checkNamesFree(scope *types.Scope, typeName string, src []byte, remedy string) error { file, err := parser.ParseFile(token.NewFileSet(), "", src, parser.SkipObjectResolution) if err != nil { return fmt.Errorf("stashgen: the generated file does not parse: %w", err) @@ -144,7 +144,7 @@ func checkNamesFree(scope *types.Scope, typeName string, src []byte) error { } } if len(clash) > 0 { - return fieldErr(typeName, "", "the package already declares %s, which the generated file declares too; pass -name to give the file's names a prefix (-name Rows writes EncryptRows and rowsCodec)", strings.Join(clash, ", ")) + return fieldErr(typeName, "", "the package already declares %s, which the generated file declares too; %s", strings.Join(clash, ", "), remedy) } return nil } diff --git a/languages/golang/stashgen/policy.go b/languages/golang/stashgen/policy.go index d76f66316..e75452766 100644 --- a/languages/golang/stashgen/policy.go +++ b/languages/golang/stashgen/policy.go @@ -20,14 +20,16 @@ type GenerateOption func(*generateConfig) type generateConfig struct { output string + name string engine Engine notices io.Writer } -// Output names the file to write. It is required. The file goes in a package -// of your own, not in the package of the generated type, so the generate -// program never imports a file that it wrote. -func Output(path string) GenerateOption { return func(c *generateConfig) { c.output = path } } +// WithName gives the generated names a prefix, as -name does on the tag +// path: WithName("Individual") writes EncryptIndividual, DecryptIndividual +// and IndividualFields. A package holds one Encrypt, so the second message +// generated into one package needs a name. +func WithName(name string) GenerateOption { return func(c *generateConfig) { c.name = name } } // WithEngine checks the declaration with this engine instead of the one the // SDK embeds. @@ -36,21 +38,26 @@ func WithEngine(e Engine) GenerateOption { return func(c *generateConfig) { c.en // WithNotices sends the generator's notices here instead of stderr. func WithNotices(w io.Writer) GenerateOption { return func(c *generateConfig) { c.notices = w } } -// Generate writes the generated file for a message from a policy: the source -// gives the facts about each field, the message's rules decide each one, and -// the file is the same one stashgen writes from tags. The generated functions -// take and return pointers to the message. +// Generate writes the generated file for a message from a policy to output: +// the source gives the facts about each field, the message's rules decide +// each one, and the file is the same one stashgen writes from tags. The +// generated functions take and return pointers to the message. output goes +// in a package of your own, not in the package of the generated type, so the +// generate program never imports a file that it wrote. // // Every field of the message needs a decision. A field that no rule decides // stops the generator with the field's name and its annotations, and so does // a field that a rule refuses with Fail. -func Generate(ctx context.Context, source policy.Source, message policy.Message, opts ...GenerateOption) error { - cfg := generateConfig{notices: os.Stderr} +func Generate(ctx context.Context, source policy.Source, message policy.Message, output string, opts ...GenerateOption) error { + cfg := generateConfig{output: output, notices: os.Stderr} for _, o := range opts { o(&cfg) } if cfg.output == "" { - return errors.New("stashgen: Generate needs Output(path)") + return errors.New("stashgen: Generate needs the path of the file to write") + } + if cfg.name != "" && (!isIdent(cfg.name) || strings.ToUpper(cfg.name[:1]) != cfg.name[:1]) { + return fmt.Errorf("stashgen: WithName(%q) must be an exported Go name", cfg.name) } if cfg.engine == nil { e, err := GuestEngine(ctx) @@ -161,7 +168,7 @@ func generateFor(ctx context.Context, cfg generateConfig, source policy.Source, } } - r := &reader{pkg: msgPkg, req: Request{Type: typeName}, eql: eqlTypes, imports: newImportSet(), outPkgName: outName, outPkgPath: outPath} + r := &reader{pkg: msgPkg, req: Request{Type: typeName, Name: cfg.name}, eql: eqlTypes, imports: newImportSet(), outPkgName: outName, outPkgPath: outPath} gf, err := r.build(collected, display, named, st, true, nil) if err != nil { return err @@ -173,12 +180,44 @@ func generateFor(ctx context.Context, cfg generateConfig, source policy.Source, if err != nil { return err } + if err := checkOutputNamesFree(ctx, outDir, cfg.output, display, src); err != nil { + return err + } for _, n := range gf.stderrNotices() { fmt.Fprintln(cfg.notices, n) } return (&File{Path: cfg.output, Content: src}).Write() } +// checkOutputNamesFree is the tag path's name check for the policy path: +// the output package, without the file being replaced, must not declare a +// name the generated file declares. A directory with no other Go file +// declares nothing. +func checkOutputNamesFree(ctx context.Context, dir, output, display string, src []byte) error { + others, err := filepath.Glob(filepath.Join(dir, "*.go")) + if err != nil { + return err + } + outAbs, err := filepath.Abs(output) + if err != nil { + return err + } + n := 0 + for _, o := range others { + if !sameFile(o, outAbs) { + n++ + } + } + if n == 0 { + return nil + } + pkg, err := loadPackage(ctx, dir, output) + if err != nil { + return err + } + return checkNamesFree(pkg.Types.Scope(), display, src, "pass WithName to give the file's names a prefix (WithName(\"Individual\") writes EncryptIndividual)") +} + // structFields indexes a struct's fields by the proto name in their protobuf // tag and by Go name. func structFields(st *types.Struct) (byProtoName, byGoName map[string]*types.Var) { diff --git a/languages/golang/stashgen/policy_test.go b/languages/golang/stashgen/policy_test.go index e33c0afc6..68d09af73 100644 --- a/languages/golang/stashgen/policy_test.go +++ b/languages/golang/stashgen/policy_test.go @@ -3,6 +3,7 @@ package stashgen_test import ( "bytes" "context" + "io" "os" "path/filepath" "strings" @@ -174,7 +175,7 @@ func TestGenerateRefusals(t *testing.T) { } func TestGenerateNeedsOutputAndAMessage(t *testing.T) { - if err := stashgen.Generate(context.Background(), policy.SourceFunc(individualFacts), individualRules()); err == nil || !strings.Contains(err.Error(), "needs Output") { + if err := stashgen.Generate(context.Background(), policy.SourceFunc(individualFacts), individualRules(), ""); err == nil || !strings.Contains(err.Error(), "needs the path") { t.Fatalf("err = %v", err) } type local struct{ A int } @@ -190,8 +191,40 @@ func TestGenerateNeedsOutputAndAMessage(t *testing.T) { } // With no WithEngine the embedded guest answers; the output directory // is no module, so the run stops there or, with no guest built, before. - err = stashgen.Generate(context.Background(), policy.SourceFunc(individualFacts), policy.ForMessage(&local{}, "c", base), stashgen.Output(filepath.Join(t.TempDir(), "x_stash.go"))) + err = stashgen.Generate(context.Background(), policy.SourceFunc(individualFacts), policy.ForMessage(&local{}, "c", base), filepath.Join(t.TempDir(), "x_stash.go")) if err == nil { t.Fatal("a message in no module generated") } } + +// The policy path keeps the tag path's name check: a second message written +// into a package that already declares Encrypt is refused, naming WithName, +// and WithName gives the file its own names. +func TestGenerateRefusesANameThePackageDeclares(t *testing.T) { + dir := policyModule(t) + if err := os.WriteFile(filepath.Join(dir, "individuals", "other.go"), []byte("package individuals\n\nfunc Encrypt() {}\n"), 0o600); err != nil { + t.Fatal(err) + } + out := filepath.Join(dir, "individuals", "individual_stash.go") + err := stashgen.GenerateFor(context.Background(), enginetest.Static{}, io.Discard, out, policy.SourceFunc(individualFacts), individualRules(), "example.com/app/pb", "Individual") + if err == nil || !strings.Contains(err.Error(), "already declares Encrypt") || !strings.Contains(err.Error(), "WithName") { + t.Fatalf("err = %v", err) + } + if _, statErr := os.Stat(out); statErr == nil { + t.Fatal("a file was written after a refusal") + } + if err := stashgen.GenerateForNamed(context.Background(), enginetest.Static{}, io.Discard, out, "Individual", policy.SourceFunc(individualFacts), individualRules(), "example.com/app/pb", "Individual"); err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(out) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(got), "func EncryptIndividual(") { + t.Fatalf("WithName did not prefix the names:\n%s", got) + } + // A rerun over its own output is not a clash with itself. + if err := stashgen.GenerateForNamed(context.Background(), enginetest.Static{}, io.Discard, out, "Individual", policy.SourceFunc(individualFacts), individualRules(), "example.com/app/pb", "Individual"); err != nil { + t.Fatalf("rerun: %v", err) + } +} From c1741bac5e1435d7c92ed6562986157186f138bf Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 19:20:52 -0700 Subject: [PATCH 22/30] refactor(golang)!: names that do not repeat their package, in Go's spelling The third cipherstash-bot review on #1094 (Go principles 8 and 13), before the first release makes these names permanent: - auth.ErrAuthTransport, auth.ErrAuthConfig, auth.ErrAuthOther and auth.WithAuthBaseURL repeat the package name; they are now auth.ErrTransport, auth.ErrConfig, auth.ErrOther and auth.WithBaseURL. The internal/guest names stay. - gensupport.UInt32/UInt64 and record.UInt32/UInt64 are Uint32/Uint64, as the standard library and stashgen's own KindUint spell them. Generated files and goldens are regenerated. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc --- languages/golang/auth/README.md | 2 +- languages/golang/auth/errors.go | 12 ++-- languages/golang/auth/strategy.go | 12 ++-- languages/golang/auth/strategy_test.go | 68 +++++++++---------- languages/golang/auth/transport.go | 6 +- languages/golang/encrypt/credentials.go | 4 +- languages/golang/encrypt/credentials_test.go | 8 +-- .../golang/encrypt/example/user_stash.go | 2 +- .../golang/encrypt/gensupport/declaration.go | 6 +- .../gensupport/gensupport_internal_test.go | 4 +- languages/golang/encrypt/guest_test.go | 6 +- .../encrypt/internal/testusers/kinds_stash.go | 10 +-- .../encrypt/internal/testusers/probe_stash.go | 4 +- .../encrypt/internal/testusers/user_stash.go | 2 +- .../golang/encrypt/live_internal_test.go | 2 +- languages/golang/encrypt/memory_test.go | 2 +- languages/golang/encrypt/options_test.go | 12 ++-- languages/golang/internal/record/record.go | 6 +- .../golang/internal/record/record_test.go | 2 +- languages/golang/stashgen/emit.go | 4 +- languages/golang/stashgen/engine.go | 10 +-- .../stubsdk/encrypt/gensupport/gensupport.go | 4 +- 22 files changed, 94 insertions(+), 94 deletions(-) diff --git a/languages/golang/auth/README.md b/languages/golang/auth/README.md index 2b22e327d..79c6d60eb 100644 --- a/languages/golang/auth/README.md +++ b/languages/golang/auth/README.md @@ -88,7 +88,7 @@ The OIDC provider is a one-method `Token(context.Context) (string, error)` interface, called on every token fetch for the JWT of the user the call is for; each distinct JWT is exchanged once while its CTS token lasts. Use `auth.OAuth2TokenSource(source)` to adapt a -`golang.org/x/oauth2.TokenSource`. `WithAuthBaseURL(url)` overrides service +`golang.org/x/oauth2.TokenSource`. `WithBaseURL(url)` overrides service discovery for local tests or a custom CTS host; `WithCacheCapacity(n)` sets how many users' CTS tokens an OIDC strategy keeps (1024 unless set), sized to the users it serves within a CTS token's lifetime. diff --git a/languages/golang/auth/errors.go b/languages/golang/auth/errors.go index 9dfbb7ab9..72eeba7a9 100644 --- a/languages/golang/auth/errors.go +++ b/languages/golang/auth/errors.go @@ -46,12 +46,12 @@ var ( ErrUsageLimit = guest.ErrAuthUsageLimit // ErrNotAuthenticated means no usable auth credential is available. ErrNotAuthenticated = guest.ErrAuthNotAuthenticated - // ErrAuthTransport is a failed auth HTTP exchange or response read. - ErrAuthTransport = guest.ErrAuthTransport - // ErrAuthConfig is invalid auth configuration or token data. - ErrAuthConfig = guest.ErrAuthConfig - // ErrAuthOther is an auth failure outside the actionable categories above. - ErrAuthOther = guest.ErrAuthOther + // ErrTransport is a failed auth HTTP exchange or response read. + ErrTransport = guest.ErrAuthTransport + // ErrConfig is invalid auth configuration or token data. + ErrConfig = guest.ErrAuthConfig + // ErrOther is an auth failure outside the actionable categories above. + ErrOther = guest.ErrAuthOther // ErrMemoryLock is guest memory that could not be locked in RAM (or, on // Linux, excluded from core dumps). Open returns it under // [RequireLockedMemory]; otherwise [ProfileStore.MemoryLockError] diff --git a/languages/golang/auth/strategy.go b/languages/golang/auth/strategy.go index 6bd6a272a..5fadc7bce 100644 --- a/languages/golang/auth/strategy.go +++ b/languages/golang/auth/strategy.go @@ -21,8 +21,8 @@ type strategyOptions struct { cacheCapacity *uint32 } -// WithAuthBaseURL overrides CTS service discovery for one strategy. -func WithAuthBaseURL(url string) StrategyOption { +// WithBaseURL overrides CTS service discovery for one strategy. +func WithBaseURL(url string) StrategyOption { return func(o *strategyOptions) { o.baseURL = url } } @@ -82,7 +82,7 @@ func (s *ProfileStore) newStrategy(ctx context.Context, config any, device bool) // stays in the guest after construction; it is not sent on each Token call. func (s *ProfileStore) AccessKey(ctx context.Context, crn, key string, opts ...StrategyOption) (*Strategy, error) { if crn == "" || key == "" { - return nil, ErrAuthConfig + return nil, ErrConfig } o := strategyConfig(opts) return s.newStrategy(ctx, struct { @@ -100,7 +100,7 @@ func (s *ProfileStore) AccessKey(ctx context.Context, crn, key string, opts ...S // JWTs that cache holds. func (s *ProfileStore) OIDC(ctx context.Context, crn string, provider OIDCProvider, opts ...StrategyOption) (*Strategy, error) { if crn == "" || provider == nil { - return nil, ErrAuthConfig + return nil, ErrConfig } o := strategyConfig(opts) id := s.root.inst.transport.register(provider) @@ -125,7 +125,7 @@ func (s *ProfileStore) OIDC(ctx context.Context, crn string, provider OIDCProvid // CLI, then the guest re-reads and saves before the lock is released. func (s *ProfileStore) DeviceSession(ctx context.Context, opts ...StrategyOption) (*Strategy, error) { if s.dir == guestRoot { - return nil, ErrAuthConfig + return nil, ErrConfig } o := strategyConfig(opts) return s.newStrategy(ctx, struct { @@ -147,7 +147,7 @@ func (s *ProfileStore) Auto(ctx context.Context, opts ...StrategyOption) (*Strat } if key, keySet := os.LookupEnv("CS_CLIENT_ACCESS_KEY"); keySet { if !crnSet { - return nil, ErrAuthConfig + return nil, ErrConfig } return s.AccessKey(ctx, crn, key, opts...) } diff --git a/languages/golang/auth/strategy_test.go b/languages/golang/auth/strategy_test.go index 7fa1c3d95..3715ea115 100644 --- a/languages/golang/auth/strategy_test.go +++ b/languages/golang/auth/strategy_test.go @@ -90,7 +90,7 @@ func TestAccessKeyStrategyCachesAndPreservesRequest(t *testing.T) { t.Fatal(err) } defer profile.Close() - strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithAuthBaseURL(server.URL)) + strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -154,7 +154,7 @@ func TestOIDCStrategyFederatesEachProviderTokenOnce(t *testing.T) { return "", errors.New("provider did not receive the Token caller's context") } return idp, nil - }), WithAuthBaseURL(server.URL)) + }), WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -210,7 +210,7 @@ func TestOIDCStrategyCacheCapacityZeroExchangesEveryCall(t *testing.T) { defer profile.Close() strategy, err := profile.OIDC(context.Background(), testCRN, OIDCProviderFunc(func(context.Context) (string, error) { return "idp-a", nil - }), WithAuthBaseURL(server.URL), WithCacheCapacity(0)) + }), WithBaseURL(server.URL), WithCacheCapacity(0)) if err != nil { t.Fatal(err) } @@ -241,7 +241,7 @@ func TestUsageLimitIsPreservedAcrossGuest(t *testing.T) { t.Fatal(err) } defer profile.Close() - strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithAuthBaseURL(server.URL)) + strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -269,7 +269,7 @@ func TestDeviceRefreshReportsInvalidClient(t *testing.T) { if err != nil { t.Fatal(err) } - strategy, err := ws.DeviceSession(context.Background(), WithAuthBaseURL(server.URL)) + strategy, err := ws.DeviceSession(context.Background(), WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -312,7 +312,7 @@ func TestAutoPrefersAccessKeyThenDeviceSession(t *testing.T) { } t.Setenv("CS_CLIENT_ACCESS_KEY", "CSAKtestKeyId.testKeySecret") t.Setenv("CS_WORKSPACE_CRN", testCRN) - strategy, err := profile.Auto(context.Background(), WithAuthBaseURL(server.URL)) + strategy, err := profile.Auto(context.Background(), WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -327,7 +327,7 @@ func TestAutoPrefersAccessKeyThenDeviceSession(t *testing.T) { if err := os.Unsetenv("CS_CLIENT_ACCESS_KEY"); err != nil { t.Fatal(err) } - strategy, err = profile.Auto(context.Background(), WithAuthBaseURL(server.URL)) + strategy, err = profile.Auto(context.Background(), WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -342,7 +342,7 @@ func TestAutoPrefersAccessKeyThenDeviceSession(t *testing.T) { if err := profile.ClearCurrentWorkspace(context.Background()); err != nil { t.Fatal(err) } - if _, err := profile.Auto(context.Background(), WithAuthBaseURL(server.URL)); !errors.Is(err, ErrNotAuthenticated) { + if _, err := profile.Auto(context.Background(), WithBaseURL(server.URL)); !errors.Is(err, ErrNotAuthenticated) { t.Fatalf("no credentials: error = %v, want %v", err, ErrNotAuthenticated) } } @@ -372,7 +372,7 @@ func TestDeviceSessionFreshTokenDoesNotTakeRefreshLock(t *testing.T) { if err != nil { t.Fatal(err) } - strategy, err := workspace.DeviceSession(context.Background(), WithAuthBaseURL("https://cts.example.com")) + strategy, err := workspace.DeviceSession(context.Background(), WithBaseURL("https://cts.example.com")) if err != nil { t.Fatal(err) } @@ -423,7 +423,7 @@ func TestDeviceSessionMissingAndInvalidProfilesKeepTheirErrors(t *testing.T) { if err != nil { t.Fatal(err) } - strategy, err := workspace.DeviceSession(context.Background(), WithAuthBaseURL("https://cts.example.com")) + strategy, err := workspace.DeviceSession(context.Background(), WithBaseURL("https://cts.example.com")) if err != nil { t.Fatal(err) } @@ -448,28 +448,28 @@ func TestAutoUsesEnvironmentPresenceAndProfileExistence(t *testing.T) { } t.Setenv("CS_WORKSPACE_CRN", testCRN) t.Setenv("CS_CLIENT_ACCESS_KEY", "") - if _, err := profile.Auto(context.Background()); !errors.Is(err, ErrAuthConfig) { - t.Fatalf("set but empty access key: error = %v, want %v", err, ErrAuthConfig) + if _, err := profile.Auto(context.Background()); !errors.Is(err, ErrConfig) { + t.Fatalf("set but empty access key: error = %v, want %v", err, ErrConfig) } // A key that does not parse is a configuration error like the empty one, // the class Rust's AutoStrategy reports, not a malformed-input error. t.Setenv("CS_CLIENT_ACCESS_KEY", "not-a-key") - if _, err := profile.Auto(context.Background()); !errors.Is(err, ErrAuthConfig) { - t.Fatalf("malformed access key: error = %v, want %v", err, ErrAuthConfig) + if _, err := profile.Auto(context.Background()); !errors.Is(err, ErrConfig) { + t.Fatalf("malformed access key: error = %v, want %v", err, ErrConfig) } - if _, err := profile.AccessKey(context.Background(), "invalid", "CSAKtestKeyId.testKeySecret"); !errors.Is(err, ErrAuthConfig) { - t.Fatalf("malformed CRN for access key: error = %v, want %v", err, ErrAuthConfig) + if _, err := profile.AccessKey(context.Background(), "invalid", "CSAKtestKeyId.testKeySecret"); !errors.Is(err, ErrConfig) { + t.Fatalf("malformed CRN for access key: error = %v, want %v", err, ErrConfig) } provider := OIDCProviderFunc(func(context.Context) (string, error) { return "", nil }) - if _, err := profile.OIDC(context.Background(), "invalid", provider); !errors.Is(err, ErrAuthConfig) { - t.Fatalf("malformed CRN for OIDC: error = %v, want %v", err, ErrAuthConfig) + if _, err := profile.OIDC(context.Background(), "invalid", provider); !errors.Is(err, ErrConfig) { + t.Fatalf("malformed CRN for OIDC: error = %v, want %v", err, ErrConfig) } if err := os.Unsetenv("CS_CLIENT_ACCESS_KEY"); err != nil { t.Fatal(err) } t.Setenv("CS_WORKSPACE_CRN", "invalid") - if _, err := profile.Auto(context.Background()); !errors.Is(err, ErrAuthConfig) { - t.Fatalf("invalid CRN without key: error = %v, want %v", err, ErrAuthConfig) + if _, err := profile.Auto(context.Background()); !errors.Is(err, ErrConfig) { + t.Fatalf("invalid CRN without key: error = %v, want %v", err, ErrConfig) } if err := os.Unsetenv("CS_WORKSPACE_CRN"); err != nil { t.Fatal(err) @@ -525,7 +525,7 @@ func TestDeviceRefreshLockPreventsReplay(t *testing.T) { errCh <- err return } - strategy, err := ws.DeviceSession(context.Background(), WithAuthBaseURL(server.URL)) + strategy, err := ws.DeviceSession(context.Background(), WithBaseURL(server.URL)) if err != nil { errCh <- err return @@ -578,7 +578,7 @@ func TestDeviceRefreshReportsInvalidGrant(t *testing.T) { if err != nil { t.Fatal(err) } - strategy, err := ws.DeviceSession(context.Background(), WithAuthBaseURL(server.URL)) + strategy, err := ws.DeviceSession(context.Background(), WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -613,7 +613,7 @@ func TestAuthRequestsIdentifyTheLibraryNotGo(t *testing.T) { t.Fatal(err) } defer profile.Close() - strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithAuthBaseURL(server.URL)) + strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -654,14 +654,14 @@ func TestAuthTransportErrorNamesTheHTTPStatusNotTheBody(t *testing.T) { t.Fatal(err) } defer profile.Close() - strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithAuthBaseURL(server.URL)) + strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } defer strategy.Close() _, err = strategy.Token(context.Background()) - if !errors.Is(err, ErrAuthTransport) { - t.Fatalf("Token error = %v, want ErrAuthTransport", err) + if !errors.Is(err, ErrTransport) { + t.Fatalf("Token error = %v, want ErrTransport", err) } if want := "cipherstash: auth transport failed: HTTP 403"; err.Error() != want { t.Fatalf("Token error = %q, want %q", err, want) @@ -680,14 +680,14 @@ func TestAuthTransportErrorWithoutAResponseNamesNoStatus(t *testing.T) { t.Fatal(err) } defer profile.Close() - strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithAuthBaseURL(addr)) + strategy, err := profile.AccessKey(context.Background(), testCRN, "CSAKtestKeyId.testKeySecret", WithBaseURL(addr)) if err != nil { t.Fatal(err) } defer strategy.Close() _, err = strategy.Token(context.Background()) - if !errors.Is(err, ErrAuthTransport) || strings.Contains(err.Error(), "HTTP") { - t.Fatalf("Token error = %v, want a bare ErrAuthTransport", err) + if !errors.Is(err, ErrTransport) || strings.Contains(err.Error(), "HTTP") { + t.Fatalf("Token error = %v, want a bare ErrTransport", err) } } @@ -719,13 +719,13 @@ func TestOutOfRangeAuthStatusIsTransport(t *testing.T) { t.Fatal(err) } defer profile.Close() - strategy, err := profile.AccessKey(ctx, testCRN, "CSAKtestKeyId.testKeySecret", WithAuthBaseURL("https://cts.invalid")) + strategy, err := profile.AccessKey(ctx, testCRN, "CSAKtestKeyId.testKeySecret", WithBaseURL("https://cts.invalid")) if err != nil { t.Fatal(err) } defer strategy.Close() - if _, err := strategy.Token(ctx); !errors.Is(err, ErrAuthTransport) { - t.Fatalf("Token: %v, want ErrAuthTransport", err) + if _, err := strategy.Token(ctx); !errors.Is(err, ErrTransport) { + t.Fatalf("Token: %v, want ErrTransport", err) } }) } @@ -760,7 +760,7 @@ func TestOpenWithoutProfileRunsAccessKeyAndRefusesProfileReads(t *testing.T) { } t.Setenv("CS_CLIENT_ACCESS_KEY", "CSAKtestKeyId.testKeySecret") t.Setenv("CS_WORKSPACE_CRN", testCRN) - strategy, err := store.Auto(ctx, WithAuthBaseURL(server.URL)) + strategy, err := store.Auto(ctx, WithBaseURL(server.URL)) if err != nil { t.Fatal(err) } @@ -782,7 +782,7 @@ func TestOpenWithoutProfileRunsAccessKeyAndRefusesProfileReads(t *testing.T) { func TestStrategyReportsItsStoresMemoryLockLive(t *testing.T) { ctx := context.Background() _, s := profile(t) - strategy, err := s.AccessKey(ctx, testCRN, "CSAKtestKeyId.testKeySecret", WithAuthBaseURL("https://cts.invalid")) + strategy, err := s.AccessKey(ctx, testCRN, "CSAKtestKeyId.testKeySecret", WithBaseURL("https://cts.invalid")) if err != nil { t.Fatal(err) } diff --git a/languages/golang/auth/transport.go b/languages/golang/auth/transport.go index 78a894548..d2cea0f65 100644 --- a/languages/golang/auth/transport.go +++ b/languages/golang/auth/transport.go @@ -44,7 +44,7 @@ const maxAuthResponseBytes = 16 << 20 // authHTTPStatus records the status of the last HTTP response the transport // received during one guest call. Only a status code crosses the guest ABI, // so without it a refused exchange (the edge in front of CTS answering 403) -// reaches the caller as a bare ErrAuthTransport. It lives on the call's +// reaches the caller as a bare ErrTransport. It lives on the call's // context, which wazero hands to the host import, so concurrent calls on // different profiles never see each other's status. type authHTTPStatus struct{ code int } @@ -56,11 +56,11 @@ func withAuthHTTPStatus(ctx context.Context) (context.Context, *authHTTPStatus) return context.WithValue(ctx, authHTTPStatusKey{}, status), status } -// wrap names the HTTP status of a refused exchange on an ErrAuthTransport +// wrap names the HTTP status of a refused exchange on an ErrTransport // ("cipherstash: auth transport failed: HTTP 403"). The body is never // included: it may be an HTML error page, or echo a credential. func (s *authHTTPStatus) wrap(err error) error { - if err == nil || !errors.Is(err, ErrAuthTransport) || s.code == 0 || (s.code >= 200 && s.code < 300) { + if err == nil || !errors.Is(err, ErrTransport) || s.code == 0 || (s.code >= 200 && s.code < 300) { return err } return fmt.Errorf("%w: HTTP %d", err, s.code) diff --git a/languages/golang/encrypt/credentials.go b/languages/golang/encrypt/credentials.go index 192d6dbcf..50caf3002 100644 --- a/languages/golang/encrypt/credentials.go +++ b/languages/golang/encrypt/credentials.go @@ -210,7 +210,7 @@ func (autoCredentials) resolve(ctx context.Context, opts resolveOptions) (*resol } return nil, fmt.Errorf("%w: no token: set %s and %s, or run `stash auth login`: %w", ErrNoCredentials, envAccessKey, envWorkspaceCRN, err) - case errors.Is(err, auth.ErrAuthConfig) && accessKeyConfigured(): + case errors.Is(err, auth.ErrConfig) && accessKeyConfigured(): // The status covers every configuration fault the guest reports; // name the variables only when they are what was configured. return nil, fmt.Errorf("encrypt: credentials: check %s and %s: %w", envAccessKey, envWorkspaceCRN, err) @@ -234,7 +234,7 @@ func (autoCredentials) resolve(ctx context.Context, opts resolveOptions) (*resol // developer profile. // // opts configure the federation strategy as they would -// auth.ProfileStore.OIDC: auth.WithAuthBaseURL pins the CTS +// auth.ProfileStore.OIDC: auth.WithBaseURL pins the CTS // endpoint for these credentials alone (without it, CS_CTS_HOST overrides // the endpoint, else it is discovered), and auth.WithCacheCapacity // sets how many users' tokens are kept (1024 unless set). diff --git a/languages/golang/encrypt/credentials_test.go b/languages/golang/encrypt/credentials_test.go index bd12205f2..fc14a75df 100644 --- a/languages/golang/encrypt/credentials_test.go +++ b/languages/golang/encrypt/credentials_test.go @@ -317,7 +317,7 @@ func TestAutoCredentialsMissing(t *testing.T) { { name: "an access key with no workspace CRN", env: map[string]string{envAccessKey: testAccessKey, envClientID: testClientID, envClientKey: testClientKey}, - want: []error{auth.ErrAuthConfig}, + want: []error{auth.ErrConfig}, }, } { t.Run(tc.name, func(t *testing.T) { @@ -729,7 +729,7 @@ func TestNewCredentialsReportsTheStrategysMemoryLock(t *testing.T) { t.Fatal(err) } defer store.Close() - strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, auth.WithAuthBaseURL("https://cts.example.com")) + strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, auth.WithBaseURL("https://cts.example.com")) if err != nil { t.Fatal(err) } @@ -769,7 +769,7 @@ func TestRequireLockedMemoryRefusesUnlockedCredentials(t *testing.T) { t.Fatal(err) } t.Cleanup(func() { _ = store.Close() }) - strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, auth.WithAuthBaseURL(cts.URL)) + strategy, err := store.AccessKey(ctx, testCRN, testAccessKey, auth.WithBaseURL(cts.URL)) if err != nil { t.Fatal(err) } @@ -874,7 +874,7 @@ func TestNewCredentialsLeavesTheStrategyToTheCaller(t *testing.T) { } defer store.Close() cts := newStub(t, http.StatusUnauthorized, "", "nope") - strategy, err := store.AccessKey(ctx, testCRN, "CSAKtest.key", auth.WithAuthBaseURL(cts.URL)) + strategy, err := store.AccessKey(ctx, testCRN, "CSAKtest.key", auth.WithBaseURL(cts.URL)) if err != nil { t.Fatal(err) } diff --git a/languages/golang/encrypt/example/user_stash.go b/languages/golang/encrypt/example/user_stash.go index 6f7f399b8..6f8944703 100644 --- a/languages/golang/encrypt/example/user_stash.go +++ b/languages/golang/encrypt/example/user_stash.go @@ -55,7 +55,7 @@ type userShape struct { var declaration = gensupport.Declare("users"). Passthrough("id"). EncryptIndex("email", gensupport.String, encrypt.Equality, encrypt.Match()). - EncryptIndex("age", gensupport.UInt32, encrypt.Equality, encrypt.Ore) + EncryptIndex("age", gensupport.Uint32, encrypt.Equality, encrypt.Ore) var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ TypeName: "User", diff --git a/languages/golang/encrypt/gensupport/declaration.go b/languages/golang/encrypt/gensupport/declaration.go index 5ea336c3c..34377ac10 100644 --- a/languages/golang/encrypt/gensupport/declaration.go +++ b/languages/golang/encrypt/gensupport/declaration.go @@ -19,15 +19,15 @@ import ( type Kind string // The kinds. int8, int16 and int32 are Int32; int and int64 are Int64; -// uint8, uint16 and uint32 are UInt32; uint and uint64 are UInt64; []byte is +// uint8, uint16 and uint32 are Uint32; uint and uint64 are Uint64; []byte is // Bytes. A type defined over one of these has its underlying kind. const ( Untyped Kind = "" Bool Kind = "bool" Int32 Kind = "int32" Int64 Kind = "int64" - UInt32 Kind = "uint32" - UInt64 Kind = "uint64" + Uint32 Kind = "uint32" + Uint64 Kind = "uint64" Float32 Kind = "float32" Float64 Kind = "float64" String Kind = "string" diff --git a/languages/golang/encrypt/gensupport/gensupport_internal_test.go b/languages/golang/encrypt/gensupport/gensupport_internal_test.go index c3dd82f39..059348b92 100644 --- a/languages/golang/encrypt/gensupport/gensupport_internal_test.go +++ b/languages/golang/encrypt/gensupport/gensupport_internal_test.go @@ -18,7 +18,7 @@ import ( func TestDeclarationLowersToTheEnginesPlan(t *testing.T) { d := Declare("users"). Passthrough("id"). - EncryptIndex("age", UInt32, encrypt.Equality, encrypt.Ore). + EncryptIndex("age", Uint32, encrypt.Equality, encrypt.Ore). EncryptIndex("email", String, encrypt.Equality, encrypt.Match()). Encrypt("notes", String). Index("score", Int64, encrypt.Ope). @@ -28,7 +28,7 @@ func TestDeclarationLowersToTheEnginesPlan(t *testing.T) { t.Fatal(err) } want := &record.Plan{Context: []string{"users"}, Fields: []record.Field{ - {Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Ore}}, + {Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Ore}}, {Name: "email", Kind: record.String, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Match}}, {Name: "notes", Kind: record.String, Outputs: []record.Output{record.Ciphertext}}, {Name: "score", Kind: record.Int64, Outputs: []record.Output{record.Ope}}, diff --git a/languages/golang/encrypt/guest_test.go b/languages/golang/encrypt/guest_test.go index fe35aa631..5da718efa 100644 --- a/languages/golang/encrypt/guest_test.go +++ b/languages/golang/encrypt/guest_test.go @@ -578,7 +578,7 @@ func mustHex(s string) []byte { // users package lowers its tags to. func usersPlan() *record.Plan { return &record.Plan{Context: []string{"users"}, Fields: []record.Field{ - {Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Ore}}, + {Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Ore}}, {Name: "email", Kind: record.String, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Match}}, }} } @@ -670,7 +670,7 @@ func TestGuestRefusesMalformedInputsBeforeState(t *testing.T) { def := c.DefaultKeyset() plan := usersPlan() floatAge := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Float64, Outputs: []record.Output{record.Ciphertext, record.Equality}}}} - matchInt := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Match}}}} + matchInt := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Match}}}} calls := map[string]func() error{ "float under equality": func() error { _, err := def.Derive(ctx, floatAge, "age", record.Equality, 1.5); return err }, "integer under match": func() error { _, err := def.Derive(ctx, matchInt, "age", record.Match, uint32(1)); return err }, @@ -707,7 +707,7 @@ func TestPlanCheckAnswersWithoutACipher(t *testing.T) { t.Fatalf("a good plan: %v", err) } refused := []*record.Plan{ - {Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.UInt32, Outputs: []record.Output{record.Ciphertext, record.Match}}}}, + {Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Match}}}}, {Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Float64, Outputs: []record.Output{record.Equality}}}}, } for i, p := range refused { diff --git a/languages/golang/encrypt/internal/testusers/kinds_stash.go b/languages/golang/encrypt/internal/testusers/kinds_stash.go index bc4d290d5..335b2496c 100644 --- a/languages/golang/encrypt/internal/testusers/kinds_stash.go +++ b/languages/golang/encrypt/internal/testusers/kinds_stash.go @@ -200,11 +200,11 @@ var kindsDeclaration = gensupport.Declare("kinds"). EncryptIndex("i32", gensupport.Int32, encrypt.Equality). EncryptIndex("i", gensupport.Int64, encrypt.Ore). Encrypt("i64", gensupport.Int64). - EncryptIndex("u8", gensupport.UInt32, encrypt.Equality). - Encrypt("u16", gensupport.UInt32). - EncryptIndex("u32", gensupport.UInt32, encrypt.Equality, encrypt.Ore). - Encrypt("u", gensupport.UInt64). - EncryptIndex("u64", gensupport.UInt64, encrypt.Ope). + EncryptIndex("u8", gensupport.Uint32, encrypt.Equality). + Encrypt("u16", gensupport.Uint32). + EncryptIndex("u32", gensupport.Uint32, encrypt.Equality, encrypt.Ore). + Encrypt("u", gensupport.Uint64). + EncryptIndex("u64", gensupport.Uint64, encrypt.Ope). Encrypt("f32", gensupport.Float32). EncryptIndex("f64", gensupport.Float64, encrypt.Ore). EncryptIndex("by", gensupport.Bytes, encrypt.Equality). diff --git a/languages/golang/encrypt/internal/testusers/probe_stash.go b/languages/golang/encrypt/internal/testusers/probe_stash.go index c5ff15e9b..aac72d4e0 100644 --- a/languages/golang/encrypt/internal/testusers/probe_stash.go +++ b/languages/golang/encrypt/internal/testusers/probe_stash.go @@ -75,8 +75,8 @@ type probeShape struct { } var probeDeclaration = gensupport.Declare("prop"). - EncryptIndex("u32", gensupport.UInt32, encrypt.Ore, encrypt.Ope). - EncryptIndex("u64", gensupport.UInt64, encrypt.Ore, encrypt.Ope). + EncryptIndex("u32", gensupport.Uint32, encrypt.Ore, encrypt.Ope). + EncryptIndex("u64", gensupport.Uint64, encrypt.Ore, encrypt.Ope). EncryptIndex("i64", gensupport.Int64, encrypt.Ore, encrypt.Ope). EncryptIndex("s", gensupport.String, encrypt.Ore, encrypt.Ope). EncryptIndex("b", gensupport.Bytes, encrypt.Ore, encrypt.Ope) diff --git a/languages/golang/encrypt/internal/testusers/user_stash.go b/languages/golang/encrypt/internal/testusers/user_stash.go index 124d1a5fe..844826abd 100644 --- a/languages/golang/encrypt/internal/testusers/user_stash.go +++ b/languages/golang/encrypt/internal/testusers/user_stash.go @@ -61,7 +61,7 @@ type userShape struct { var declaration = gensupport.Declare("users"). Passthrough("id"). - EncryptIndex("age", gensupport.UInt32, encrypt.Equality, encrypt.Ore). + EncryptIndex("age", gensupport.Uint32, encrypt.Equality, encrypt.Ore). EncryptIndex("email", gensupport.String, encrypt.Equality, encrypt.Match()). Encrypt("notes", gensupport.String). Omit("internal") diff --git a/languages/golang/encrypt/live_internal_test.go b/languages/golang/encrypt/live_internal_test.go index 504a0423e..63db30a94 100644 --- a/languages/golang/encrypt/live_internal_test.go +++ b/languages/golang/encrypt/live_internal_test.go @@ -52,7 +52,7 @@ func liveClient(t *testing.T) *Client { t.Cleanup(func() { _ = store.Close() }) var strategyOpts []auth.StrategyOption if cts := os.Getenv("STACK_ENCRYPT_TEST_CTS_HOST"); cts != "" { - strategyOpts = append(strategyOpts, auth.WithAuthBaseURL(cts)) + strategyOpts = append(strategyOpts, auth.WithBaseURL(cts)) } strategy, err := store.AccessKey(t.Context(), crn, accessKey, strategyOpts...) if err != nil { diff --git a/languages/golang/encrypt/memory_test.go b/languages/golang/encrypt/memory_test.go index 9a0d7e598..6be1561b5 100644 --- a/languages/golang/encrypt/memory_test.go +++ b/languages/golang/encrypt/memory_test.go @@ -148,7 +148,7 @@ func TestRequireLockedMemoryRefusesACallerStoreUnlocked(t *testing.T) { fmt.Println("case skipped: mlock succeeds under RLIMIT_MEMLOCK=0") return } - strategy, err := store.AccessKey(ctx, "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY", "CSAKtestKeyId.testKeySecret", auth.WithAuthBaseURL("https://cts.invalid")) + strategy, err := store.AccessKey(ctx, "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY", "CSAKtestKeyId.testKeySecret", auth.WithBaseURL("https://cts.invalid")) if err != nil { t.Fatal(err) } diff --git a/languages/golang/encrypt/options_test.go b/languages/golang/encrypt/options_test.go index c4b066258..85d1481b0 100644 --- a/languages/golang/encrypt/options_test.go +++ b/languages/golang/encrypt/options_test.go @@ -94,7 +94,7 @@ func TestTheSameCredentialsPassedTwiceAreNotConsumed(t *testing.T) { // does not panic, whichever constructor made them. func TestCredentialsAreComparable(t *testing.T) { a := OIDCFederation("crn:a", nil) - b := OIDCFederation("crn:b", nil, auth.WithAuthBaseURL("https://cts.example.com")) + b := OIDCFederation("crn:b", nil, auth.WithBaseURL("https://cts.example.com")) if a == b || AutoCredentials() != AutoCredentials() { t.Fatal("unexpected comparison result") } @@ -195,12 +195,12 @@ func TestOIDCFederationResolvesTheKeyLikeAuto(t *testing.T) { } // No CRN is a configuration error from the strategy, before any // provider or key is asked. - if _, err := OIDCFederation("", provider).resolve(context.Background(), resolveOptions{Transport: http.DefaultTransport}); !errors.Is(err, auth.ErrAuthConfig) { - t.Fatalf("OIDCFederation with no CRN: %v, want ErrAuthConfig", err) + if _, err := OIDCFederation("", provider).resolve(context.Background(), resolveOptions{Transport: http.DefaultTransport}); !errors.Is(err, auth.ErrConfig) { + t.Fatalf("OIDCFederation with no CRN: %v, want ErrConfig", err) } } -// OIDCFederation's strategy options reach the strategy: WithAuthBaseURL +// OIDCFederation's strategy options reach the strategy: WithBaseURL // pins CTS for these credentials, over CS_CTS_HOST, which here names a // decoy that fails the test if it is asked. func TestOIDCFederationTakesStrategyOptions(t *testing.T) { @@ -208,7 +208,7 @@ func TestOIDCFederationTakesStrategyOptions(t *testing.T) { cleanEnv(t, filepath.Join(t.TempDir(), "absent")) cts := newAuthServer(t) decoy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - t.Errorf("CS_CTS_HOST was asked (%s) though WithAuthBaseURL pinned CTS", r.URL.Path) + t.Errorf("CS_CTS_HOST was asked (%s) though WithBaseURL pinned CTS", r.URL.Path) http.NotFound(w, r) })) t.Cleanup(decoy.Close) @@ -216,7 +216,7 @@ func TestOIDCFederationTakesStrategyOptions(t *testing.T) { t.Setenv(envClientID, testClientID) t.Setenv(envClientKey, testClientKey) provider := auth.OIDCProviderFunc(func(context.Context) (string, error) { return "idp-token", nil }) - creds := OIDCFederation(testCRN, provider, auth.WithAuthBaseURL(cts.URL)) + creds := OIDCFederation(testCRN, provider, auth.WithBaseURL(cts.URL)) resolved, err := creds.resolve(context.Background(), resolveOptions{Transport: http.DefaultTransport}) if err != nil { t.Fatal(err) diff --git a/languages/golang/internal/record/record.go b/languages/golang/internal/record/record.go index 6e5c55fb7..0cb1b2fa5 100644 --- a/languages/golang/internal/record/record.go +++ b/languages/golang/internal/record/record.go @@ -41,8 +41,8 @@ const ( Bool Kind = "bool" Int32 Kind = "int32" Int64 Kind = "int64" - UInt32 Kind = "uint32" - UInt64 Kind = "uint64" + Uint32 Kind = "uint32" + Uint64 Kind = "uint64" Float32 Kind = "float32" Float64 Kind = "float64" String Kind = "string" @@ -217,7 +217,7 @@ func (p *Plan) Validate() error { func (k Kind) known() bool { switch k { - case Untyped, Bool, Int32, Int64, UInt32, UInt64, Float32, Float64, String, Bytes: + case Untyped, Bool, Int32, Int64, Uint32, Uint64, Float32, Float64, String, Bytes: return true } return false diff --git a/languages/golang/internal/record/record_test.go b/languages/golang/internal/record/record_test.go index 8b00cd51d..c2e71b798 100644 --- a/languages/golang/internal/record/record_test.go +++ b/languages/golang/internal/record/record_test.go @@ -11,7 +11,7 @@ import ( func users() *Plan { return &Plan{Context: []string{"users"}, Fields: []Field{ - {Name: "age", Kind: UInt32, Outputs: []Output{Ciphertext, Equality, Ore}}, + {Name: "age", Kind: Uint32, Outputs: []Output{Ciphertext, Equality, Ore}}, {Name: "email", Kind: String, Outputs: []Output{Ciphertext, Equality, Match}}, {Name: "notes", Kind: String, Outputs: []Output{Ciphertext}}, }} diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index 02153f67f..4f0e9043a 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -228,9 +228,9 @@ func kindExpr(t GoType) string { case KindUint: switch t.Basic { case "uint8", "uint16", "uint32", "byte": - return "gensupport.UInt32" + return "gensupport.Uint32" } - return "gensupport.UInt64" + return "gensupport.Uint64" case KindFloat: if t.Basic == "float32" { return "gensupport.Float32" diff --git a/languages/golang/stashgen/engine.go b/languages/golang/stashgen/engine.go index edcf3f071..ab104ef7d 100644 --- a/languages/golang/stashgen/engine.go +++ b/languages/golang/stashgen/engine.go @@ -184,10 +184,10 @@ func wireKind(t GoType) record.Kind { return record.Int32 case "gensupport.Int64": return record.Int64 - case "gensupport.UInt32": - return record.UInt32 - case "gensupport.UInt64": - return record.UInt64 + case "gensupport.Uint32": + return record.Uint32 + case "gensupport.Uint64": + return record.Uint64 case "gensupport.Float32": return record.Float32 case "gensupport.Float64": @@ -208,7 +208,7 @@ func kindOfWire(k record.Kind) Kind { return KindBytes case record.Int32, record.Int64: return KindInt - case record.UInt32, record.UInt64: + case record.Uint32, record.Uint64: return KindUint case record.Float32, record.Float64: return KindFloat diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go index 1df868db6..a64e9e2d0 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go @@ -29,8 +29,8 @@ const ( Bool Kind = "bool" Int32 Kind = "int32" Int64 Kind = "int64" - UInt32 Kind = "uint32" - UInt64 Kind = "uint64" + Uint32 Kind = "uint32" + Uint64 Kind = "uint64" Float32 Kind = "float32" Float64 Kind = "float64" String Kind = "string" From 2a802ad1fad8ff4f57d33a124414c7e7fef1407a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 21:15:36 -0700 Subject: [PATCH 23/30] docs(stack-encrypt): drop the Go plan.Custom note, the package is removed here The note on plan.Custom and plantest snapshots described a Go package this PR deletes. Go has no release yet, so no user ever saw plan.Custom bind a text part; the crate CHANGELOG ships to crates.io and should not name it. --- packages/stack-encrypt/CHANGELOG.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index e27aeac6f..8f0ecf0a6 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -48,14 +48,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 indexed fields under one label are refused for the same reason the builder refuses them (`PlanError::SharedIdentity`). A record a Go program wrote with a `label=` tag is unaffected; one written with a `context=` - tag is not readable through a plan. For the same reason Go - `plan.Custom("notes/v1")` now binds the label `["notes", "v1"]`, not the - one text part `"notes/v1"`: a row written under the old form does not - decrypt and its terms match no query, with no error, and a one-segment - `Custom("ctx")` is refused. `plantest` snapshots now spell every context - as its segments (`context ["notes", "v1"]`), so a golden file changes on - regeneration and refuses the old spelling until it does; check each - `target Custom` column before recording it. + tag is not readable through a plan. - **Every data plan field seals the tagged `FfiValue` leaf, whatever its `"type"`**, as every field did before; the type admits indexes and checks kinds and changes no bytes, so a row written without a type opens under a From 34f95f7a573c8c4c582665cd2f3a120c2c9c0e46 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 21:30:26 -0700 Subject: [PATCH 24/30] fix(golang): parseTarget refuses an unknown kind or index The keys of a target entry were strict, but two values were not: plaintext became a record.Kind and each index a record.Output with no check. A typo in the Rust serialiser ("strng", "equality") then reached stashgen as an EQL type with no kind or no indexes, and no error. Both are now refused; "json", the SteVec document index no plan output carries, is accepted. record.Kind's check is exported as Known for it. --- languages/golang/encrypt/checker.go | 12 +++++++++++- languages/golang/encrypt/targets_test.go | 2 ++ languages/golang/internal/record/record.go | 5 +++-- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/languages/golang/encrypt/checker.go b/languages/golang/encrypt/checker.go index 3677a5c9c..7c0a0ee4d 100644 --- a/languages/golang/encrypt/checker.go +++ b/languages/golang/encrypt/checker.go @@ -122,6 +122,9 @@ func parseTarget(entry vcvalue.Object) (record.Target, error) { var kind string kind, err = targetOptionalString(f) t.Plaintext = record.Kind(kind) + if err == nil && !t.Plaintext.Known() { + err = fmt.Errorf("plaintext %q is not a kind", kind) + } case "sql_domain": t.SQLDomain, err = targetString(f) case "indexes": @@ -134,7 +137,14 @@ func parseTarget(entry vcvalue.Object) (record.Target, error) { if !ok { return t, fmt.Errorf("an index is %T, not a string", item) } - t.Indexes = append(t.Indexes, record.Output(s)) + switch o := record.Output(s); o { + // "json" is the SteVec document index, which no plan output + // carries. + case record.Equality, record.Match, record.Ore, record.Ope, "json": + t.Indexes = append(t.Indexes, o) + default: + return t, fmt.Errorf("unknown index %q", s) + } } case "query": t.Query, err = targetOptionalString(f) diff --git a/languages/golang/encrypt/targets_test.go b/languages/golang/encrypt/targets_test.go index ec4a91b5f..69a007acf 100644 --- a/languages/golang/encrypt/targets_test.go +++ b/languages/golang/encrypt/targets_test.go @@ -47,6 +47,8 @@ func TestParseTargetsReadsEachEntry(t *testing.T) { "list not a list": vcvalue.Object{{Key: "targets", Value: "x"}}, "entry not object": vcvalue.Object{{Key: "targets", Value: []any{"TextEq"}}}, "numeric plaintext": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "plaintext", Value: int64(1)}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "misspelled kind": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "plaintext", Value: "strng"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "misspelled index": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{"equality"}}, vcvalue.Field{Key: "producible", Value: true})}}}, "unknown key": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "kind", Value: "string"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, "no name": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, "no producible": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}})}}}, diff --git a/languages/golang/internal/record/record.go b/languages/golang/internal/record/record.go index 0cb1b2fa5..63805b3e6 100644 --- a/languages/golang/internal/record/record.go +++ b/languages/golang/internal/record/record.go @@ -193,7 +193,7 @@ func (p *Plan) Validate() error { return fmt.Errorf("record: field %q: identity %q is used twice", f.Name, identity) } identities[identity] = true - if !f.Kind.known() { + if !f.Kind.Known() { return fmt.Errorf("record: field %q: unknown kind %q", f.Name, f.Kind) } if len(f.Outputs) == 0 { @@ -215,7 +215,8 @@ func (p *Plan) Validate() error { return nil } -func (k Kind) known() bool { +// Known reports whether k is a kind that the engine names; Untyped is one. +func (k Kind) Known() bool { switch k { case Untyped, Bool, Int32, Int64, Uint32, Uint64, Float32, Float64, String, Bytes: return true From ac6897af39f0c47136f802e4695c7cb0107cdd94 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 6 Oct 2026 21:30:27 -0700 Subject: [PATCH 25/30] docs(golang): say what the SDK does not protect, and fix the example's Decrypt - The example sealed through Extend("tenant-42") and decrypted through the Client, which refuses every such row; it now decrypts through the cipher, and the Cipher, Decrypter and README docs say a Client opens only rows sealed with no extension. - doc.go says the guest holds nothing but the client key and keyset cache between calls; after Decrypt one unwiped copy of each opened string remains, as the skipped residency test records. The test comment no longer names a vitaminc line nobody has confirmed. - A generated Decrypt over a struct with index-only fields says those fields come back zero and must not be encrypted again to update the row, and the stashgen README says the same: re-encrypting stores the term for zero. - The generated Encrypt and Decrypt comments, records.go and gensupport count the keyset request beside the one per 500 sealed values. - The README lists which terms a database can compare: ORE terms sort almost correctly as plain bytes, so ORDER BY on them is wrong without an error. - The tag table warns that the name and context= are part of every stored value's context, and the cipher section that a value is not bound to its row. --- languages/golang/cmd/stashgen/README.md | 12 ++++++++ languages/golang/encrypt/README.md | 23 ++++++++++++++- languages/golang/encrypt/cipher.go | 4 ++- languages/golang/encrypt/doc.go | 11 +++++--- languages/golang/encrypt/example/main.go | 7 +++-- .../golang/encrypt/example/user_stash.go | 7 +++-- .../golang/encrypt/gensupport/gensupport.go | 2 +- .../internal/testusers/account_stash.go | 5 ++-- .../internal/testusers/document_stash.go | 5 ++-- .../internal/testusers/everything_stash.go | 6 ++-- .../encrypt/internal/testusers/kinds_stash.go | 5 ++-- .../internal/testusers/lookup_stash.go | 11 ++++++-- .../encrypt/internal/testusers/probe_stash.go | 5 ++-- .../internal/testusers/secret_stash.go | 5 ++-- .../encrypt/internal/testusers/user_stash.go | 7 +++-- languages/golang/encrypt/records.go | 21 +++++++------- languages/golang/encrypt/roundtrip_test.go | 11 ++++---- languages/golang/encrypt/term.go | 5 +++- languages/golang/stashgen/emit.go | 28 +++++++++++++++++-- .../cases/accounts/account_stash.go.golden | 5 ++-- .../contacts/contactstash_stash.go.golden | 5 ++-- .../cases/documents/document_stash.go.golden | 5 ++-- .../cases/embedded/patient_stash.go.golden | 5 ++-- .../foreign/individualstash_stash.go.golden | 5 ++-- .../cases/orders/order_stash.go.golden | 5 ++-- .../cases/orders/refund_stash.go.golden | 5 ++-- .../testdata/cases/users/user_stash.go.golden | 7 +++-- .../policy_individual_stash.go.golden | 5 ++-- 28 files changed, 158 insertions(+), 69 deletions(-) diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 6d926fccc..edd32c117 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -76,6 +76,18 @@ The first part of a tag is the field's name, which is the column name in a datab | `stash:"-"` | leave the field out | | `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value | +> **Take care** +> +> The name and the `context=` value are part of the encryption context of each stored value. +> If you change either one, the rows that you stored before the change do not decrypt, and queries do not find them. +> Do not change them after you store rows. No tag can rename a column and keep its context yet. + +> **Take care** +> +> Do not encrypt a decrypted value again to update its row. +> After `Decrypt`, each index-only field is zero, so `Encrypt` stores the term for zero, and a query on that field then matches the wrong rows with no error. +> To update one column, use its `Fields` entry, such as `Fields.Email.Encrypt`. + The index names are `equality`, `match`, `ore`, `ope` and `json`. A `match` index needs text with at least one token: the engine derives no match term for an empty string, separator-only text, or text shorter than the n-gram length (3 characters), because an empty term would match every row. `Encrypt` then fails for the whole batch, with an error naming the row, the field and the index. diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index 357e36371..62b504757 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -135,6 +135,14 @@ written through `Extend("tenant-42")` opens and matches only through a cipher with the same extension. No call takes a keyset or a context, so the three cannot use different ones. +> **Take care** +> +> The context binds each value to its table, its column and the cipher's +> extension. It does not bind the value to its row, so a value copied to +> another row of the same table decrypts there with no error. The Rust engine +> can bind a value to its row with `context_field`. The Go SDK cannot do this +> yet. + `Client.DefaultKeyset()` is the keyset a ZeroKMS administrator set for the client; `Client.Keyset(encrypt.KeysetName(..))` or `Client.Keyset(id)` any other, loaded on first use. `Cipher.KeysetID(ctx)` resolves it. @@ -143,7 +151,9 @@ other, loaded on first use. `Cipher.KeysetID(ctx)` resolves it. `users.Decrypt` takes a `Decrypter`: the `*Cipher`, which refuses a row another keyset sealed with `ErrForeignKeyset` before any key is retrieved, or -the `*Client`, which opens each row under the keyset that sealed it. +the `*Client`, which opens each row under the keyset that sealed it. The +`*Client` opens only rows that a cipher with no extension sealed; a row +written through `Extend` opens only through a cipher with the same extension. ## What is stored @@ -167,6 +177,17 @@ language. `EqualityTerm.Equal` compares in constant time; `OreTerm.Compare` and `OpeTerm.Compare` order as the plaintexts; `MatchTerm.Positions` decodes the token positions. +A database can compare some terms by itself: + +- `Equality`: compare with `=`. +- `Ope`: compare with `<`, `>` and `ORDER BY`. The byte order is the + plaintext order. +- `Ore`: do not compare in SQL. Plain byte order is almost always right, and + sometimes wrong, with no error. Compare in Go with `OreTerm.Compare`, or + use an EQL type (`encrypt_into=`), which the database compares correctly. +- `Match`: do not compare in SQL. Decode the positions in Go with + `MatchTerm.Positions`, or use an EQL type. + ## Errors The generator and the compiler find a mistake in a declaration, so no call diff --git a/languages/golang/encrypt/cipher.go b/languages/golang/encrypt/cipher.go index 3d5f58281..0d5322653 100644 --- a/languages/golang/encrypt/cipher.go +++ b/languages/golang/encrypt/cipher.go @@ -13,7 +13,9 @@ import ( // the engine directly. A Cipher opens only its own keyset's ciphertexts — a // leaf sealed under another keyset is refused as [ErrForeignKeyset] before // any key is retrieved. To open ciphertexts from any keyset, pass the -// [Client] where a [Decrypter] is taken. +// [Client] where a [Decrypter] is taken. A Client opens only rows that a +// cipher with no extension encrypted; rows sealed through [Cipher.Extend] +// open only through a cipher with the same extension. // // A Cipher holds no guest state: the keyset is selected on every call, and // loaded by the guest on first use, so one is cheap to make per request or diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index cd9ff82f1..71d4de4b0 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -178,8 +178,11 @@ // the guest's memory that cannot be locked. A Client prints its memory state // ([Client.String]) and logs it ([Client.LogValue]). // -// Between calls the guest holds the client key and its keyset cache and -// nothing else: data keys are per-call values wiped when the export returns, -// and every buffer staged for a call is wiped before the call's result is -// returned. +// Between calls the guest holds the client key and its keyset cache. Data +// keys are per-call values, wiped when the export returns, and every buffer +// staged for a call is wiped before the call's result is returned. One +// exception remains: after Decrypt, one unwiped copy of each opened string +// stays in the guest's freed heap until that memory is reused. The test +// TestPlaintextDoesNotRemainInGuestMemoryAfterDecrypt records the gap and is +// skipped until vitaminc wipes that copy. package encrypt diff --git a/languages/golang/encrypt/example/main.go b/languages/golang/encrypt/example/main.go index 2bd2f43bd..2157699d7 100644 --- a/languages/golang/encrypt/example/main.go +++ b/languages/golang/encrypt/example/main.go @@ -55,9 +55,10 @@ func run(ctx context.Context) error { fmt.Printf("id %d matches bob@example.com: %v\n", e.ID, term.Equal(e.Email.Equality)) } - // Decrypt through the cipher, which refuses another keyset's rows, or - // through the client, which opens each row under the keyset that sealed it. - back, err := Decrypt(ctx, client, encrypted) + // Decrypt through the cipher. It carries the extension that Encrypt + // used, and it refuses rows from another keyset. The client would refuse + // these rows: it opens only rows sealed with no extension. + back, err := Decrypt(ctx, cipher, encrypted) if err != nil { return fmt.Errorf("decrypt: %w", err) } diff --git a/languages/golang/encrypt/example/user_stash.go b/languages/golang/encrypt/example/user_stash.go index 6f8944703..13cf2b6ef 100644 --- a/languages/golang/encrypt/example/user_stash.go +++ b/languages/golang/encrypt/example/user_stash.go @@ -109,14 +109,15 @@ var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ }, }) -// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values. -// The result has one element for each input, in the same order. +// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values, +// plus one the first time a keyset is used. The result has one element for each +// input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, main []User) ([]EncryptedUser, error) { return codec.Encrypt(ctx, cipher, main) } // Decrypt opens each EncryptedUser, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/gensupport/gensupport.go b/languages/golang/encrypt/gensupport/gensupport.go index 3fa87b363..27bf21fb8 100644 --- a/languages/golang/encrypt/gensupport/gensupport.go +++ b/languages/golang/encrypt/gensupport/gensupport.go @@ -6,7 +6,7 @@ // [Declaration] from the struct's tags, hands the library its conversions in // a [Generated] value, and prints through [Redacted] and [RedactedLog]. The // library lowers the declaration to the data plan the engine reads, sends a -// slice of values as one request, and reports the two notices that the +// slice of values in one guest call, and reports the two notices that the // generator also prints. No function in this package panics. package gensupport diff --git a/languages/golang/encrypt/internal/testusers/account_stash.go b/languages/golang/encrypt/internal/testusers/account_stash.go index 2ff74373b..c07695012 100644 --- a/languages/golang/encrypt/internal/testusers/account_stash.go +++ b/languages/golang/encrypt/internal/testusers/account_stash.go @@ -113,13 +113,14 @@ var accountCodec = gensupport.New(gensupport.Generated[Account, EncryptedAccount }) // EncryptAccount seals each Account, with one ZeroKMS request for each 500 -// sealed values. The result has one element for each input, in the same order. +// sealed values, plus one the first time a keyset is used. The result has one +// element for each input, in the same order. func EncryptAccount(ctx context.Context, cipher *encrypt.Cipher, values []Account) ([]EncryptedAccount, error) { return accountCodec.Encrypt(ctx, cipher, values) } // DecryptAccount opens each EncryptedAccount, with one ZeroKMS request for each -// 500 sealed values. +// 500 sealed values, plus one the first time a keyset is used. func DecryptAccount(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedAccount) ([]Account, error) { return accountCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/document_stash.go b/languages/golang/encrypt/internal/testusers/document_stash.go index 9aaba5d79..086fa0941 100644 --- a/languages/golang/encrypt/internal/testusers/document_stash.go +++ b/languages/golang/encrypt/internal/testusers/document_stash.go @@ -79,13 +79,14 @@ var documentCodec = gensupport.New(gensupport.Generated[Document, EncryptedDocum }) // EncryptDocument seals each Document, with one ZeroKMS request for each 500 -// sealed values. The result has one element for each input, in the same order. +// sealed values, plus one the first time a keyset is used. The result has one +// element for each input, in the same order. func EncryptDocument(ctx context.Context, cipher *encrypt.Cipher, values []Document) ([]EncryptedDocument, error) { return documentCodec.Encrypt(ctx, cipher, values) } // DecryptDocument opens each EncryptedDocument, with one ZeroKMS request for -// each 500 sealed values. +// each 500 sealed values, plus one the first time a keyset is used. func DecryptDocument(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedDocument) ([]Document, error) { return documentCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/everything_stash.go b/languages/golang/encrypt/internal/testusers/everything_stash.go index a69e6ed37..8be911965 100644 --- a/languages/golang/encrypt/internal/testusers/everything_stash.go +++ b/languages/golang/encrypt/internal/testusers/everything_stash.go @@ -181,14 +181,14 @@ var everythingCodec = gensupport.New(gensupport.Generated[Everything, EncryptedE }) // EncryptEverything seals each Everything, with one ZeroKMS request for each -// 500 sealed values. The result has one element for each input, in the same -// order. +// 500 sealed values, plus one the first time a keyset is used. The result has +// one element for each input, in the same order. func EncryptEverything(ctx context.Context, cipher *encrypt.Cipher, values []Everything) ([]EncryptedEverything, error) { return everythingCodec.Encrypt(ctx, cipher, values) } // DecryptEverything opens each EncryptedEverything, with one ZeroKMS request -// for each 500 sealed values. +// for each 500 sealed values, plus one the first time a keyset is used. func DecryptEverything(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedEverything) ([]Everything, error) { return everythingCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/kinds_stash.go b/languages/golang/encrypt/internal/testusers/kinds_stash.go index 335b2496c..3eb5866b2 100644 --- a/languages/golang/encrypt/internal/testusers/kinds_stash.go +++ b/languages/golang/encrypt/internal/testusers/kinds_stash.go @@ -487,13 +487,14 @@ var kindsCodec = gensupport.New(gensupport.Generated[Kinds, EncryptedKinds]{ }) // EncryptKinds seals each Kinds, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func EncryptKinds(ctx context.Context, cipher *encrypt.Cipher, values []Kinds) ([]EncryptedKinds, error) { return kindsCodec.Encrypt(ctx, cipher, values) } // DecryptKinds opens each EncryptedKinds, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func DecryptKinds(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedKinds) ([]Kinds, error) { return kindsCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/lookup_stash.go b/languages/golang/encrypt/internal/testusers/lookup_stash.go index 1e99becd3..0ae987a04 100644 --- a/languages/golang/encrypt/internal/testusers/lookup_stash.go +++ b/languages/golang/encrypt/internal/testusers/lookup_stash.go @@ -105,13 +105,20 @@ var lookupCodec = gensupport.New(gensupport.Generated[Lookup, EncryptedLookup]{ }) // EncryptLookup seals each Lookup, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func EncryptLookup(ctx context.Context, cipher *encrypt.Cipher, values []Lookup) ([]EncryptedLookup, error) { return lookupCodec.Encrypt(ctx, cipher, values) } // DecryptLookup opens each EncryptedLookup, with one ZeroKMS request for each -// 500 sealed values. +// 500 sealed values, plus one the first time a keyset is used. +// +// DecryptLookup leaves Score and Code at the zero value: an index-only field +// stores no ciphertext, so nothing opens for it. Do not pass a decrypted value +// back to EncryptLookup to update its row, because EncryptLookup would then +// store the term for zero. Derive the index-only terms again from the real +// values through LookupFields. func DecryptLookup(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedLookup) ([]Lookup, error) { return lookupCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/probe_stash.go b/languages/golang/encrypt/internal/testusers/probe_stash.go index aac72d4e0..807745930 100644 --- a/languages/golang/encrypt/internal/testusers/probe_stash.go +++ b/languages/golang/encrypt/internal/testusers/probe_stash.go @@ -155,13 +155,14 @@ var probeCodec = gensupport.New(gensupport.Generated[Probe, EncryptedProbe]{ }) // EncryptProbe seals each Probe, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func EncryptProbe(ctx context.Context, cipher *encrypt.Cipher, values []Probe) ([]EncryptedProbe, error) { return probeCodec.Encrypt(ctx, cipher, values) } // DecryptProbe opens each EncryptedProbe, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func DecryptProbe(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedProbe) ([]Probe, error) { return probeCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/secret_stash.go b/languages/golang/encrypt/internal/testusers/secret_stash.go index ba91fb8d2..9f85d763c 100644 --- a/languages/golang/encrypt/internal/testusers/secret_stash.go +++ b/languages/golang/encrypt/internal/testusers/secret_stash.go @@ -95,13 +95,14 @@ var secretCodec = gensupport.New(gensupport.Generated[Secret, EncryptedSecret]{ }) // EncryptSecret seals each Secret, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func EncryptSecret(ctx context.Context, cipher *encrypt.Cipher, values []Secret) ([]EncryptedSecret, error) { return secretCodec.Encrypt(ctx, cipher, values) } // DecryptSecret opens each EncryptedSecret, with one ZeroKMS request for each -// 500 sealed values. +// 500 sealed values, plus one the first time a keyset is used. func DecryptSecret(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedSecret) ([]Secret, error) { return secretCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/internal/testusers/user_stash.go b/languages/golang/encrypt/internal/testusers/user_stash.go index 844826abd..97f1cae6e 100644 --- a/languages/golang/encrypt/internal/testusers/user_stash.go +++ b/languages/golang/encrypt/internal/testusers/user_stash.go @@ -124,14 +124,15 @@ var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ }, }) -// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values. -// The result has one element for each input, in the same order. +// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values, +// plus one the first time a keyset is used. The result has one element for each +// input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, testusers []User) ([]EncryptedUser, error) { return codec.Encrypt(ctx, cipher, testusers) } // Decrypt opens each EncryptedUser, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go index 1a71a5206..4b4606dd3 100644 --- a/languages/golang/encrypt/records.go +++ b/languages/golang/encrypt/records.go @@ -14,20 +14,21 @@ import ( // The three methods take and return the internal record types, so a program // cannot call them with anything but what a generated declaration lowered // to; the generated functions (users.Encrypt, users.Decrypt, users.Fields) -// are the API. Every call is one guest call, and one batched ZeroKMS request -// however many rows it carries (one request per 500 sealed leaves). +// are the API. Every call is one guest call. It sends one ZeroKMS request +// for each 500 sealed values, plus one the first time a keyset is used. // Decrypter opens records: a *Cipher, which refuses a record sealed under // another keyset before any key is retrieved, or a *Client, which opens each -// record under the keyset that sealed it. Generated Decrypt functions take +// record under the keyset that sealed it. A *Client opens only records that +// a cipher with no extension sealed. Generated Decrypt functions take // one. Its method is for generated code; a program does not call it. type Decrypter interface { Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) } // Seal encrypts rows under the plan, through this cipher's keyset and with -// its extension: the sealed fields of every row in one request, in order. -// For generated code. +// its extension: the sealed fields of every row, in order, with one ZeroKMS +// request for each 500 sealed values. For generated code. func (cph *Cipher) Seal(ctx context.Context, plan *record.Plan, rows []record.Source) ([]record.Sealed, error) { p, err := cph.plan(plan) if err != nil { @@ -83,9 +84,9 @@ func (cph *Cipher) Seal(ctx context.Context, plan *record.Plan, rows []record.So return sealed, nil } -// Open decrypts records sealed under the plan: every row in one request, -// in order. A record from another keyset is [ErrForeignKeyset]. For -// generated code. +// Open decrypts records sealed under the plan: every row, in order, with one +// ZeroKMS request for each 500 sealed values. A record from another keyset is +// [ErrForeignKeyset]. For generated code. func (cph *Cipher) Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) { if cph == nil { // A Decrypter holding a nil *Cipher is not a nil interface, so the @@ -143,8 +144,8 @@ func (cph *Cipher) Derive(ctx context.Context, plan *record.Plan, field string, } // Open decrypts records sealed under the plan by any keyset of this client: -// each record is opened under the keyset that sealed it, with one request -// per keyset. For generated code. +// each record is opened under the keyset that sealed it, with one ZeroKMS +// request for each 500 sealed values from each keyset. For generated code. func (c *Client) Open(ctx context.Context, plan *record.Plan, records []record.Sealed) ([]record.Source, error) { if c == nil { return nil, fmt.Errorf("%w: Open on a nil *Client", ErrEncoding) diff --git a/languages/golang/encrypt/roundtrip_test.go b/languages/golang/encrypt/roundtrip_test.go index ae88fd562..2af6348aa 100644 --- a/languages/golang/encrypt/roundtrip_test.go +++ b/languages/golang/encrypt/roundtrip_test.go @@ -323,13 +323,12 @@ func TestPlaintextDoesNotRemainInGuestMemory(t *testing.T) { // into the guest. It fails today: exactly one copy of each opened string // stays in the guest's freed heap. The host's copy is wiped (the output // buffer goes through se_dealloc) and the FfiValue's Protected payloads are -// wiped on drop; the copy that stays is made inside vitaminc-aead-value's -// FfiValue decoder, which copies the decrypted leaf's bytes into a new -// Protected (value.rs, the `tags::STRING` arm) and leaves the AEAD output -// it copied from to drop unwiped. That is vitaminc's to fix; the skip -// records it, and the test runs again once it is. +// wiped on drop, so the copy is made on vitaminc's decrypt path before the +// plaintext is wrapped in Protected; which line makes it is not yet known. +// That is vitaminc's to fix; the skip records it, and the test runs again +// once it is. func TestPlaintextDoesNotRemainInGuestMemoryAfterDecrypt(t *testing.T) { - t.Skip("known: one unwiped copy of each opened string remains after Decrypt; see the comment above (vitaminc-aead-value FfiValue decode)") + t.Skip("known: one unwiped copy of each opened string remains after Decrypt, made on vitaminc's decrypt path; run this test again when vitaminc wipes it") c := deterministicClient(t) cipher := c.DefaultKeyset() const plaintext = "residency-probe-4111-b1c2d3e4f5" diff --git a/languages/golang/encrypt/term.go b/languages/golang/encrypt/term.go index b019f68fe..8e3bf55a0 100644 --- a/languages/golang/encrypt/term.go +++ b/languages/golang/encrypt/term.go @@ -121,7 +121,10 @@ func (t MatchTerm) Positions() ([]uint16, error) { // OreTerm is an order-revealing term (CLLW ORE): the raw term bytes. Two // terms derived under the same keyset and context order as their plaintexts -// through Compare and Less; plain byte order says nothing. +// through Compare and Less. Plain byte order is almost always the same +// order, and sometimes not, with no error: do not compare ORE terms in SQL +// (ORDER BY, <, >) on a byte column. Compare them in Go, or store the field +// as an EQL type, which the database compares correctly. type OreTerm []byte // Compare orders two ORE terms as their plaintexts: -1, 0 or +1. Ports the diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index 4f0e9043a..c0cd9368a 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -471,12 +471,25 @@ func (w *writer) value(f *genFile) { func (w *writer) functions(f *genFile) { w.nl() - w.b.WriteString(wrapComment(fmt.Sprintf("%s seals each %s, with one ZeroKMS request for each 500 sealed values. The result has one element for each input, in the same order.", f.encryptFn, f.typeName), 80)) + w.b.WriteString(wrapComment(fmt.Sprintf("%s seals each %s, with one ZeroKMS request for each 500 sealed values, plus one the first time a keyset is used. The result has one element for each input, in the same order.", f.encryptFn, f.typeName), 80)) w.p("func %s(ctx context.Context, cipher *encrypt.Cipher, %s []%s) ([]%s, error) {", f.encryptFn, f.paramName, f.typeExpr, f.encName) w.p("\treturn %s.Encrypt(ctx, cipher, %s)", f.codecVar, f.paramName) w.p("}") w.nl() - w.b.WriteString(wrapComment(fmt.Sprintf("%s opens each %s, with one ZeroKMS request for each 500 sealed values.", f.decryptFn, f.encName), 80)) + w.b.WriteString(wrapComment(fmt.Sprintf("%s opens each %s, with one ZeroKMS request for each 500 sealed values, plus one the first time a keyset is used.", f.decryptFn, f.encName), 80)) + var indexOnly []string + for _, g := range f.fields { + if g.Verb == VerbIndex { + indexOnly = append(indexOnly, g.GoName) + } + } + if len(indexOnly) > 0 { + // Read, change and save is the usual update, and it would store the + // term for zero: say so on the function a program calls. + w.p("//") + w.b.WriteString(wrapComment(fmt.Sprintf("%s leaves %s at the zero value: an index-only field stores no ciphertext, so nothing opens for it. Do not pass a decrypted value back to %s to update its row, because %s would then store the term for zero. Derive the index-only terms again from the real values through %s.", + f.decryptFn, joinNames(indexOnly), f.encryptFn, f.encryptFn, f.fieldsVar), 80)) + } w.p("func %s(ctx context.Context, d encrypt.Decrypter, encrypted []%s) ([]%s, error) {", f.decryptFn, f.encName, f.typeExpr) w.p("\treturn %s.Decrypt(ctx, d, encrypted)", f.codecVar) w.p("}") @@ -604,3 +617,14 @@ func (m *genModel) fieldFor(g *genField, out *output) string { } return "/* unbound " + g.Name + " */" } + +// joinNames lists Go names in prose: "A", "A and B", "A, B and C". +func joinNames(names []string) string { + switch len(names) { + case 0: + return "" + case 1: + return names[0] + } + return strings.Join(names[:len(names)-1], ", ") + " and " + names[len(names)-1] +} diff --git a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden index c333d3462..b63fb9cad 100644 --- a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden @@ -138,13 +138,14 @@ var codec = gensupport.New(gensupport.Generated[Account, EncryptedAccount]{ }) // Encrypt seals each Account, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, accounts []Account) ([]EncryptedAccount, error) { return codec.Encrypt(ctx, cipher, accounts) } // Decrypt opens each EncryptedAccount, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedAccount) ([]Account, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden index e7a36d034..00d723533 100644 --- a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden @@ -110,13 +110,14 @@ var codec = gensupport.New(gensupport.Generated[crm.Contact, EncryptedContact]{ }) // Encrypt seals each crm.Contact, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, contacts []crm.Contact) ([]EncryptedContact, error) { return codec.Encrypt(ctx, cipher, contacts) } // Decrypt opens each EncryptedContact, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedContact) ([]crm.Contact, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden index 80d8b7b6d..d2ff0276a 100644 --- a/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/documents/document_stash.go.golden @@ -79,13 +79,14 @@ var codec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ }) // Encrypt seals each Document, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, documents []Document) ([]EncryptedDocument, error) { return codec.Encrypt(ctx, cipher, documents) } // Decrypt opens each EncryptedDocument, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedDocument) ([]Document, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden index 2cbee5d7b..1adbd28a5 100644 --- a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden @@ -124,13 +124,14 @@ var codec = gensupport.New(gensupport.Generated[Patient, EncryptedPatient]{ }) // Encrypt seals each Patient, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, embedded []Patient) ([]EncryptedPatient, error) { return codec.Encrypt(ctx, cipher, embedded) } // Decrypt opens each EncryptedPatient, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedPatient) ([]Patient, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden index fa453c9f1..b6dd34a2f 100644 --- a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden @@ -121,13 +121,14 @@ var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndivid }) // Encrypt seals each pb.Individual, with one ZeroKMS request for each 500 -// sealed values. The result has one element for each input, in the same order. +// sealed values, plus one the first time a keyset is used. The result has one +// element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, individuals []*pb.Individual) ([]EncryptedIndividual, error) { return codec.Encrypt(ctx, cipher, individuals) } // Decrypt opens each EncryptedIndividual, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedIndividual) ([]*pb.Individual, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden index 960ad4e05..519b731c2 100644 --- a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden @@ -123,13 +123,14 @@ var orderCodec = gensupport.New(gensupport.Generated[Order, EncryptedOrder]{ }) // EncryptOrder seals each Order, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func EncryptOrder(ctx context.Context, cipher *encrypt.Cipher, values []Order) ([]EncryptedOrder, error) { return orderCodec.Encrypt(ctx, cipher, values) } // DecryptOrder opens each EncryptedOrder, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func DecryptOrder(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedOrder) ([]Order, error) { return orderCodec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden index 7f0397577..a9d7a066b 100644 --- a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden @@ -82,13 +82,14 @@ var codec = gensupport.New(gensupport.Generated[Refund, EncryptedRefund]{ }) // Encrypt seals each Refund, with one ZeroKMS request for each 500 sealed -// values. The result has one element for each input, in the same order. +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, orders []Refund) ([]EncryptedRefund, error) { return codec.Encrypt(ctx, cipher, orders) } // Decrypt opens each EncryptedRefund, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedRefund) ([]Refund, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden index ffff9bb1b..9a82606d3 100644 --- a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden @@ -92,14 +92,15 @@ var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ }, }) -// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values. -// The result has one element for each input, in the same order. +// Encrypt seals each User, with one ZeroKMS request for each 500 sealed values, +// plus one the first time a keyset is used. The result has one element for each +// input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, users []User) ([]EncryptedUser, error) { return codec.Encrypt(ctx, cipher, users) } // Decrypt opens each EncryptedUser, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { return codec.Decrypt(ctx, d, encrypted) } diff --git a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden index fa453c9f1..b6dd34a2f 100644 --- a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden +++ b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden @@ -121,13 +121,14 @@ var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndivid }) // Encrypt seals each pb.Individual, with one ZeroKMS request for each 500 -// sealed values. The result has one element for each input, in the same order. +// sealed values, plus one the first time a keyset is used. The result has one +// element for each input, in the same order. func Encrypt(ctx context.Context, cipher *encrypt.Cipher, individuals []*pb.Individual) ([]EncryptedIndividual, error) { return codec.Encrypt(ctx, cipher, individuals) } // Decrypt opens each EncryptedIndividual, with one ZeroKMS request for each 500 -// sealed values. +// sealed values, plus one the first time a keyset is used. func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedIndividual) ([]*pb.Individual, error) { return codec.Decrypt(ctx, d, encrypted) } From 57e9ad1971a17ce0b76323d713977564e2981121 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 7 Oct 2026 02:27:22 -0700 Subject: [PATCH 26/30] feat(stack-encrypt): a data plan can take its context from a field of the record The plan builder has three context sources: the plan, the call, and a field of the value (`FieldsBuilder::context_field`, the derive's `#[stash(context_field)]`). The data grammar `dynamic::record` lowers had only the first: a plan held one `Label` and every field's `"context"` had to sit under it. ADR-0007 listed `context_field` among the typed, Rust-only parts, but it is not typed, and the ADR's own rule says a capability the grammar cannot express is added to the grammar once so every author gets it. A binding could not bind a row to its own context at all. The grammar gains a plan-level `"context_field": ""` key beside the field specs. Under it the named field is a passthrough of type string whose value in each record (`"tenants/acme"`) is the context every other field is sealed under, and each field's `"context"` is its identity alone, a one-segment label, extended by the caller's parts as before. `Plan::with_context_field` holds the rules (the field exists, is passthrough only, is a string or untyped, and every label has one segment), `lower()` goes through the builder's own `context_field` over a `Value` (which now implements `IntoLabel`; a value that is not text is the new `LabelError::NotText`), and the extension still becomes `.extend(..)`. Without the key the two-segment and shared-prefix rules hold exactly as they did, so every existing plan parses and seals the same bytes; a one-segment label is now the plan's refusal rather than the field's, which is the same `Error::Plan` from `plan()`. `record::decrypt` and `check_record` take the context the caller expects, the chain's `open(record).context(expected)`, and refuse a record whose stored context differs with `Error::ContextMismatch` before any key is requested; `None` opens each record under what it stores, and `Some` on a plan with a context of its own is the chain's `TwoContextSources`. `Plan::label` is an `Option` now, as the builder's is, and `Plan::context_field` names the field. The cross-author fixture gains a `context_field` case: a Rust chain over `Value` fields with `context_field` and the data plan each seal a note under `tenants/acme`, each opens the other's under the stored context and refuses it under another, and both derive the term under `tenants/acme/text`. Only that key was appended; the existing records are as committed, and each test's update path now rewrites only the entries it owns. --- docs/plans/2026-10-04-plan-builder.md | 7 + packages/stack-encrypt/CHANGELOG.md | 19 + ...-through-a-plan-never-a-second-executor.md | 11 + .../fuzz/fuzz_targets/check_record.rs | 2 +- packages/stack-encrypt/src/descriptor.rs | 5 + packages/stack-encrypt/src/dynamic/record.rs | 880 ++++++++++++++++-- packages/stack-encrypt/src/dynamic/value.rs | 20 + packages/stack-encrypt/src/plan/mod.rs | 12 +- .../stack-encrypt/tests/fixtures/README.md | 24 +- .../tests/fixtures/record_lowering.json | 65 ++ .../stack-encrypt/tests/record_lowering.rs | 377 +++++++- 11 files changed, 1328 insertions(+), 94 deletions(-) diff --git a/docs/plans/2026-10-04-plan-builder.md b/docs/plans/2026-10-04-plan-builder.md index b596cf753..bd45d5433 100644 --- a/docs/plans/2026-10-04-plan-builder.md +++ b/docs/plans/2026-10-04-plan-builder.md @@ -16,6 +16,13 @@ the builder itself, #1071), with a second round of decisions: decrypt is spelled `open`, the two starts and the three context sources, the typed verb and the picker, the derive narrowed before it emits the plan, and EQL types assembled per language with no registry. +Amended 2026-10-07 (#1094): where this document says the typed parts "have +no data form" and lists `context_field` among them ("Consolidation", and +item 3 under "Why the first draft was dropped"), it is wrong about +`context_field`. That verb is not typed, and the data grammar now spells it +as a plan-level `"context_field"` key; the Go SDK spells it as the +`context_field` tag word. The typed parts that have no data form are +`encrypt_into`, the picker and the one-value start. **Date:** 2026-10-04 **Issue:** #1046 **Builds on:** #1050 (one context per column; `Label`, `Describe`), #971 (EQL v3 diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 8f0ecf0a6..fa493e9ad 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -122,6 +122,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **A data plan can take its context from a field of the record**, as a + Rust chain's `FieldsBuilder::context_field` and the derive's + `#[stash(context_field)]` do: a plan-level `"context_field": ""` + key beside the field specs (`dynamic::record::plan`, + `Plan::with_context_field`). The named field's value in each record (a + label such as `"tenants/acme"`) is the context every other field is + sealed under, and the field is carried as a passthrough of type + `"string"`, so the record stores its own context. Under a context field + each field's `"context"` is its identity alone, a one-segment label, + extended as before; without the key the two-segment and shared-prefix + rules hold and every existing plan seals the same bytes. + `record::decrypt` and `record::check_record` take the context the caller + expects (an `Option