refactor(golang): one DeterministicSource, protoc-gen-go's GoName, and the review's small findings

DeterministicSource existed twice — stack-encrypt's tests/common and the
guest's deterministic.rs — and the Go fixture test depends on the two
deriving the same bytes. It now lives once in stack-kms behind its
test-support feature, beside FakeDataKeySource; the stack-encrypt tests
and the guest import it, and the guest's deterministic-kms feature no
longer carries its own sha2. The rebuilt test guest still opens the
fixture, which is the proof the move changed no bytes.

protosource.GoName is protoc-gen-go's GoCamelCase word for word: a digit
ends a word (foo_1bar is Foo_1Bar, sha256sum is Sha256Sum), a leading
underscore is X, a dot is an underscore unless a lower-case letter
follows; TestGoName held the wrong value for foo_1bar. scalar's enum
branch has a test through a dynamic descriptor, since testpb declares no
enum. stack-guest-abi's crate doc names the guest's real path. The three
documents say what a match index needs — text with at least one token,
three characters for the n-gram — that the whole batch fails otherwise,
and that an optional or short value does not belong under match.

Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc

Author: coderdan

languages/golang/cmd/stashgen/README.md

@@ -73,8 +73,9 @@ The first part of a tag is the field's name, which is the column name in a datab
 | `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value |
 
 The index names are `equality`, `match`, `ore`, `ope` and `json`.
-A `match` index needs text with at least one token: the engine derives no match term for an empty or separator-only string, or one shorter than the n-gram, because an empty term would match every row.
-`Encrypt` then returns an error naming the row, the field and the index; give the field a value, or drop the index.
+A `match` index needs text with at least one token: the engine derives no match term for an empty string, separator-only text, or text shorter than the n-gram length (3 characters), because an empty term would match every row.
+`Encrypt` then fails for the whole batch, with an error naming the row, the field and the index.
+So an optional or short value does not belong under `match`: give the field `equality` alone, or make the value required.
 An index takes its options in parentheses after its name, separated by commas: `index=equality;match(k=3)`.
 These words are the same as the Rust API's words for the same behaviour.
 

languages/golang/encrypt/README.md

@@ -143,6 +143,13 @@ and `Equality`, `Match`, `Ore` or `Ope` (the term types). A field with
 An `opaque` struct is one `Sealed` field. Every stored type implements
 `driver.Valuer` and `sql.Scanner`.
 
+A `match` index needs text with at least one token. The engine derives no
+match term for an empty string, separator-only text, or text shorter than the
+n-gram length (3 characters), because an empty term would match every row.
+`Encrypt` then fails for the whole batch with `ErrTerm`, naming the row, the
+field and the index. So an optional or short value does not belong under
+`match`: give such a field `equality` alone, or make the value required.
+
 Terms are byte-equal to the ones the Rust crate derives, so a term from
 `users.Fields.Email.Equality` compares against a stored term written from any
 language. `EqualityTerm.Equal` compares in constant time; `OreTerm.Compare`

languages/golang/encrypt/doc.go

@@ -93,6 +93,13 @@
 // [Match], [Ore] and [Ope]; [JSON] is declared and refused until the engine
 // derives it. Every stored type implements driver.Valuer and sql.Scanner.
 //
+// A match index needs text with at least one token: the engine derives no
+// match term for an empty string, separator-only text, or text shorter than
+// the n-gram length (3 characters), because an empty term would match every
+// row. Encrypt then fails for the whole batch with [ErrTerm], naming the row,
+// the field and the index, so an optional or short value does not belong
+// under match.
+//
 // # Transport and auth
 //
 // The guest imports exactly two host functions: an HTTP send, served by any

languages/golang/encrypt/guest/Cargo.lock

@@ -30,7 +30,7 @@ crate-type = ["cdylib", "rlib"]
 # network. The Go tests load it to open the fixture's records and to run
 # hermetic round trips; `mise run wasm:guest:build:deterministic` builds it
 # beside the real guest. Never the embedded guest.
-deterministic-kms = ["stack-kms/test-support", "dep:sha2"]
+deterministic-kms = ["stack-kms/test-support"]
 
 [dependencies]
 # The stack crates with default features off: no reqwest, no native TLS —
@@ -56,8 +56,6 @@ vitaminc-aead-value = "0.5.1"
 vitaminc-protected = "0.5.1"
 
 futures = { version = "0.3", default-features = false, features = ["executor"] }
-# Only the deterministic-kms test build derives keys itself.
-sha2 = { version = "0.10", optional = true }
 serde = "1"
 serde_json = "1"
 uuid = "1"

languages/golang/encrypt/guest/Cargo.toml

@@ -1,116 +1,13 @@
-//! The deterministic key source of the `deterministic-kms` test build: a
-//! copy of `DeterministicSource` in stack-encrypt's `tests/common/mod.rs`,
-//! which sealed the record fixture (`tests/fixtures/record_lowering.json`).
+//! The deterministic key source of the `deterministic-kms` test build:
+//! `stack-kms`'s [`DeterministicSource`] (behind its `test-support` feature),
+//! the one definition that also sealed stack-encrypt's record fixture
+//! (`tests/fixtures/record_lowering.json`). A Go test that loads this build
+//! with the fixture's seed opens the records Rust sealed and derives the same
+//! term bytes, because both run the same code.
 //!
-//! Every data key is `SHA-256(seed ‖ "key" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, its
-//! tag `SHA-256(seed ‖ "tag" ‖ 0 ‖ descriptor ‖ 0 ‖ iv)`, and the IV
-//! `SHA-256(seed ‖ "iv" ‖ 0 ‖ descriptor ‖ 0 ‖ counter)[..16]`, where
-//! `descriptor` is the context the leaf is sealed under as ZeroKMS renders
-//! it. A leaf opens under its own field's descriptor and no other, as under
-//! ZeroKMS. The index key is `FakeDataKeySource`'s, the same every test in
-//! the repository derives terms under. So a Go test that loads this build
-//! with the fixture's seed opens the records Rust sealed and derives the
-//! same term bytes.
-//!
-//! It is a test double. The feature that compiles it is off by default, the
-//! Go package never embeds this build, and the fixture README beside the
-//! JSON file is the one definition both copies follow.
-
-use std::borrow::Cow;
-use std::sync::atomic::{AtomicU64, Ordering};
-
-use sha2::{Digest, Sha256};
-use stack_kms::{
-    DataKey, DataKeySource, DataKeyWithTag, FakeDataKeySource, GenerateKeyPayload, IdentifiedBy,
-    IndexKey, IndexKeySource, RetrieveKeyPayload, UnverifiedContext,
-};
-use uuid::Uuid;
-
-/// The seeded source. See the [module docs](self).
-pub struct DeterministicSource {
-    seed: [u8; 32],
-    counter: AtomicU64,
-    index: FakeDataKeySource,
-}
-
-impl DeterministicSource {
-    /// A source over `seed`.
-    pub fn new(seed: [u8; 32]) -> Self {
-        Self {
-            seed,
-            counter: AtomicU64::new(0),
-            index: FakeDataKeySource::new(),
-        }
-    }
-
-    fn derive(&self, what: &str, descriptor: &str, salt: &[u8]) -> [u8; 32] {
-        let mut hasher = Sha256::new();
-        hasher.update(self.seed);
-        hasher.update(what.as_bytes());
-        hasher.update([0u8]);
-        hasher.update(descriptor.as_bytes());
-        hasher.update([0u8]);
-        hasher.update(salt);
-        hasher.finalize().into()
-    }
-}
-
-impl DataKeySource for DeterministicSource {
-    async fn generate_keys(
-        &self,
-        payloads: Vec<GenerateKeyPayload<'_>>,
-        _keyset_id: Option<Uuid>,
-        _unverified_context: Option<Cow<'_, UnverifiedContext>>,
-    ) -> Result<Vec<DataKeyWithTag>, stack_kms::Error> {
-        Ok(payloads
-            .into_iter()
-            .map(|payload| {
-                let n = self.counter.fetch_add(1, Ordering::SeqCst);
-                let iv_bytes = self.derive("iv", payload.descriptor, &n.to_le_bytes());
-                let mut iv = stack_kms::Iv::default();
-                let width = iv.len();
-                iv.copy_from_slice(&iv_bytes[..width]);
-                let key = self.derive("key", payload.descriptor, &iv);
-                let tag = self.derive("tag", payload.descriptor, &iv);
-                DataKeyWithTag {
-                    key: DataKey { iv, key },
-                    tag: tag.to_vec(),
-                    decryption_policy: payload.decryption_policy,
-                }
-            })
-            .collect())
-    }
-
-    async fn retrieve_keys(
-        &self,
-        payloads: Vec<RetrieveKeyPayload<'_>>,
-        _keyset_id: Option<Uuid>,
-        _unverified_context: Option<&UnverifiedContext>,
-    ) -> Result<Vec<DataKey>, stack_kms::Error> {
-        payloads
-            .iter()
-            .map(|payload| {
-                let iv: stack_kms::Iv = *payload.iv.as_ref();
-                let tag = self.derive("tag", payload.descriptor, &iv);
-                if tag[..] != *payload.tag {
-                    return Err(stack_kms::Error::RetrieveKey(
-                        stack_kms::RetrieveKeyError::FailedRetrieval(
-                            "the tag is not this descriptor's".to_string(),
-                        ),
-                    ));
-                }
-                let key = self.derive("key", payload.descriptor, &iv);
-                Ok(DataKey { iv, key })
-            })
-            .collect()
-    }
-}
+//! It is a test double. The feature that compiles it is off by default, and
+//! the build it produces goes under languages/golang/encrypt/testdata, which
+//! `go build` and the package's `//go:embed wasm` both ignore, so no Go
+//! binary carries it; the tests read it from disk.
 
-impl IndexKeySource for DeterministicSource {
-    async fn load_index_key(
-        &self,
-        keyset_id: Option<IdentifiedBy>,
-    ) -> Result<(Uuid, IndexKey), stack_kms::Error> {
-        self.index.load_index_key(keyset_id).await
-    }
-}
+pub use stack_kms::DeterministicSource;

languages/golang/encrypt/guest/src/deterministic.rs

@@ -8,7 +8,6 @@ package protosource
 
 import (
 	"fmt"
-	"strings"
 
 	"google.golang.org/protobuf/proto"
 	"google.golang.org/protobuf/reflect/protoreflect"
@@ -82,27 +81,42 @@ func scalar(fd protoreflect.FieldDescriptor, v protoreflect.Value) string {
 	return v.String()
 }
 
-// GoName is the Go field name protoc-gen-go gives a proto field: the first
-// letter capitalised, and an underscore before a lower-case letter dropped
-// with that letter capitalised. medicare_no becomes MedicareNo; foo_1bar
-// keeps its underscore, as protoc-gen-go does.
+// GoName is the Go field name protoc-gen-go gives a proto field: its
+// GoCamelCase, word for word. A word starts at an underscore or a capital;
+// a digit is a word of its own, so the letter after it starts a new word
+// (foo_1bar is Foo_1Bar, sha256sum is Sha256Sum); an underscore before a
+// lower-case letter is dropped and the letter capitalised; a leading
+// underscore becomes X (_x is XX); a dot becomes an underscore.
 func GoName(protoName string) string {
-	var b strings.Builder
-	upper := true
+	var b []byte
 	for i := 0; i < len(protoName); i++ {
 		c := protoName[i]
 		switch {
-		case c == '_' && i+1 < len(protoName) && protoName[i+1] >= 'a' && protoName[i+1] <= 'z':
-			upper = true
+		case c == '.' && i+1 < len(protoName) && isASCIILower(protoName[i+1]):
+			// Skip over '.' in ".{{lowercase}}".
 		case c == '.':
-			b.WriteByte('_')
-		case upper && c >= 'a' && c <= 'z':
-			b.WriteByte(c - 'a' + 'A')
-			upper = false
+			b = append(b, '_')
+		case c == '_' && (i == 0 || protoName[i-1] == '.'):
+			// An initial '_' (or one after '.') becomes 'X', so the name
+			// starts with a capital.
+			b = append(b, 'X')
+		case c == '_' && i+1 < len(protoName) && isASCIILower(protoName[i+1]):
+			// Skip over '_' in "_{{lowercase}}".
+		case c >= '0' && c <= '9':
+			b = append(b, c)
 		default:
-			b.WriteByte(c)
-			upper = false
+			// A letter starts a word, capitalised; the lower-case run after
+			// it is the rest of the word.
+			if isASCIILower(c) {
+				c -= 'a' - 'A'
+			}
+			b = append(b, c)
+			for ; i+1 < len(protoName) && isASCIILower(protoName[i+1]); i++ {
+				b = append(b, protoName[i+1])
+			}
 		}
 	}
-	return b.String()
+	return string(b)
 }
+
+func isASCIILower(c byte) bool { return 'a' <= c && c <= 'z' }

languages/golang/encrypt/policy/protosource/protosource.go

@@ -4,6 +4,11 @@ import (
 	"strings"
 	"testing"
 
+	"google.golang.org/protobuf/proto"
+	"google.golang.org/protobuf/reflect/protodesc"
+	"google.golang.org/protobuf/reflect/protoreflect"
+	"google.golang.org/protobuf/types/descriptorpb"
+
 	"github.com/cipherstash/stack/languages/golang/encrypt/policy"
 	"github.com/cipherstash/stack/languages/golang/encrypt/policy/protosource/internal/testpb"
 )
@@ -62,10 +67,52 @@ func TestNotAMessage(t *testing.T) {
 }
 
 func TestGoName(t *testing.T) {
-	cases := map[string]string{"id": "Id", "medicare_no": "MedicareNo", "foo_1bar": "Foo_1bar", "Already": "Already", "a_b_c": "ABC", "x__y": "X_Y"}
+	// protoc-gen-go's GoCamelCase, including the cases a reviewer found the
+	// first version wrong on: a digit ends a word, a leading underscore is X.
+	cases := map[string]string{
+		"id": "Id", "medicare_no": "MedicareNo", "foo_1bar": "Foo_1Bar", "sha256sum": "Sha256Sum", "_x": "XX",
+		"Already": "Already", "a_b_c": "ABC", "x__y": "X_Y", "a.b": "AB", "a.B": "A_B", "x_Y": "X_Y", "ab1": "Ab1",
+	}
 	for in, want := range cases {
 		if got := GoName(in); got != want {
 			t.Errorf("GoName(%q) = %q, want %q", in, got, want)
 		}
 	}
 }
+
+// scalar spells an enum option by its value name: a dynamic enum field
+// stands in for a generated one, since testpb declares no enum.
+func TestScalarSpellsAnEnumByName(t *testing.T) {
+	file, err := protodesc.NewFile(&descriptorpb.FileDescriptorProto{
+		Name:    proto.String("enum_test.proto"),
+		Package: proto.String("enumtest"),
+		Syntax:  proto.String("proto3"),
+		EnumType: []*descriptorpb.EnumDescriptorProto{{
+			Name: proto.String("Level"),
+			Value: []*descriptorpb.EnumValueDescriptorProto{
+				{Name: proto.String("LEVEL_UNSPECIFIED"), Number: proto.Int32(0)},
+				{Name: proto.String("LEVEL_HIGH"), Number: proto.Int32(2)},
+			},
+		}},
+		MessageType: []*descriptorpb.DescriptorProto{{
+			Name: proto.String("Holder"),
+			Field: []*descriptorpb.FieldDescriptorProto{{
+				Name: proto.String("level"), Number: proto.Int32(1),
+				Type:     descriptorpb.FieldDescriptorProto_TYPE_ENUM.Enum(),
+				TypeName: proto.String(".enumtest.Level"),
+				JsonName: proto.String("level"),
+			}},
+		}},
+	}, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	fd := file.Messages().Get(0).Fields().Get(0)
+	if got := scalar(fd, protoreflect.ValueOfEnum(2)); got != "LEVEL_HIGH" {
+		t.Fatalf("scalar(enum 2) = %q, want LEVEL_HIGH", got)
+	}
+	// A number with no name falls back to the number.
+	if got := scalar(fd, protoreflect.ValueOfEnum(7)); got != "7" {
+		t.Fatalf("scalar(enum 7) = %q", got)
+	}
+}