diff --git a/docs/dev/capabilities.yml b/docs/dev/capabilities.yml index 8f543457..7e4b960c 100644 --- a/docs/dev/capabilities.yml +++ b/docs/dev/capabilities.yml @@ -29,7 +29,7 @@ # Bumped whenever the capabilities list changes, so a consumer can tell how current # the manifest it validated against was (the published catalog stamps this number). -version: 1 +version: 2 capabilities: - id: secret-injection @@ -89,3 +89,14 @@ capabilities: Per-app operator-supplied configuration. A manifest `config:` block renders a form at install and a Settings editor on the app page, injecting each answer under its own env-var name into the target service's compose override. + + - id: mail-enable-enum + since: "0.x" + ref: "#302; APP_MANIFEST.md # D3; SERVICE_PROVISIONING.md # BYO outgoing mail" + summary: > + Manifest-declared mail.env value maps. An app whose mail config is a + boot-validated enum (twenty's EMAIL_DRIVER, vaultwarden's SMTP_SECURITY) + declares, per app-owned var, the token for each mail state (from: bound | + encryption); the brain resolves and stamps the final value under the app's + own name in both bound and unbound states, where compose interpolation can't + remap one enum's tokens onto another's. diff --git a/docs/dev/catalog-import-gaps.md b/docs/dev/catalog-import-gaps.md index 685fa465..0adf3aed 100644 --- a/docs/dev/catalog-import-gaps.md +++ b/docs/dev/catalog-import-gaps.md @@ -208,7 +208,7 @@ Unlike `docs/progress/` entries (frozen ADR snapshots), this file is **mutable b - **Trigger:** `EMAIL_DRIVER` — Twenty's outbound-mail switch is a **boot-validated enum** (`logger` | `smtp`, default `logger`), not a boolean. The established malmo BYO-mail compose gate (`${MALMO_MAIL_HOST:+smtp}`, as `gitea`/`kimai`/`paperless-ngx` use to turn mail on only when a provider is bound) expands to the empty string on an unbound box, and `EMAIL_DRIVER=""` fails Twenty's config validation at startup — verified live: the container exits 1 with `EMAIL_DRIVER must be one of the following values` / `Config variables validation failed`. - **What breaks:** malmo's registered SMTP provider can't be auto-injected into Twenty by env the way it is for boolean-gated apps. Mail itself is not broken — the admin enables it in-app under **Settings → Emails** (Twenty's DB-backed config layer, which overrides the env default), so invitations and notifications send after one manual step — but the "it just works from the box's mail provider" auto-wire that `mail: {optional: true}` + `MALMO_MAIL_*` gives other apps does not apply, so the manifest declares no `mail:` binding. - **Why malmo can't satisfy it (v1):** the BYO-mail injection (#122) is shaped for a boolean/`:+`-gated enable flag; an enum driver that boot-rejects empty-string can't be gated by a single compose expansion (there is no `smtp`-when-bound / `logger`-when-unbound substitution). A malmo-side fix would be injecting a "mail-bound" boolean or a driver sentinel (e.g. `MALMO_MAIL_DRIVER`) the compose could map onto the enum's two valid values. -- **Status:** open — shipped degraded (`catalog/twenty`); mail configured in-app, the same path `listmonk` takes for its DB-stored SMTP settings. New gap-class: recurs for any app whose mail-enable is a boot-validated enum rather than a boolean. +- **Status:** implemented (#302 — manifest `mail.env` value maps, `APP_MANIFEST.md` # D3, `SERVICE_PROVISIONING.md` # BYO outgoing mail). The manifest declares, per app-owned mail env var, the token for each mail state (`from: bound|encryption`) and the brain resolves + stamps the final value in Go — closing both this enable-enum facet (twenty's `EMAIL_DRIVER`) and the security-enum facet (vaultwarden's `SMTP_SECURITY`), neither of which compose interpolation can remap. Emitted in both bound and unbound states so the enum boots valid unbound (killing the empty-string boot-reject the `:+` gate caused). Recorded in [`capabilities.yml`](capabilities.yml) under id `mail-enable-enum`. The brain mechanism ships here; wiring `catalog/twenty`'s manifest (`EMAIL_DRIVER`) and the store repo's `apps/vaultwarden` (`SMTP_SECURITY`) plus the live send check are the coordinated catalog/store follow-up. ### nonroot-data-ownership — formbricks (2026-06-20) diff --git a/docs/progress/README.md b/docs/progress/README.md index 4b848f3e..ddfb3161 100644 --- a/docs/progress/README.md +++ b/docs/progress/README.md @@ -202,3 +202,4 @@ Oldest first; append new entries to the bottom. | [reconcile-pending-recreate.md](reconcile-pending-recreate.md) — Make the reconcile pass converge env-restamping drift on a **running** container (**closes #268**), the gap `manifest-config-block.md` (#264) documented and `RebindMail` shared. `SetConfig`/`RebindMail` are brain-commits-first: store + override/.env, then `compose up -d` a running instance. A failed `compose up` left a container that **kept running** on its old env — reconcile re-created an already-running container only on resource-limit drift, never env drift — so it stayed stale until the user retried (a fallen-over container or brain restart already converged via the "no containers" branch). **Fix:** a `pending_recreate` boolean on `instances` (column + idempotent migration + `Instance.PendingRecreate` + `SetInstancePendingRecreate`). New `recreateRunning(ctx, inst)` helper wraps the running-instance edit's `compose up`: on failure it **marks** the instance pending (the committed override/.env is the reconstructible intent), on success it **clears** the marker; `SetConfig`/`RebindMail` route through it. `Reconcile`'s already-up branch now recreates when resource-limit policy drifted **or** the marker is set — one `compose up -d` converges both (env read at container-create) — and clears the marker on success, with the `restore()` rewind guarded to the resource-stanza patch while the marker (not a file rewind) makes the env recreate retryable; the "no containers" branch also clears the marker after a bring-up so a pending-and-fallen-over instance is satisfied without a redundant later recreate. `Start` also clears the marker on a successful recreate, so a Stop→Start cycle doesn't leave a stale marker for reconcile to redundantly retry (a self-review catch). Covers config + mail + any future env-restamping op in **one place** (not special-cased to config). `APP_LIFECYCLE.md`'s reconcile drift list gained the pending-recreate case as a fourth bullet. Startup-pass cadence unchanged (no timer; converges on next brain start) — matches the issue's "self-healing-on-restart edge" framing and the spec's no-reconciler-loop stance. `make check` green | done | | [hosted-grow-root-disk.md](hosted-grow-root-disk.md) — Grow the hosted image's root filesystem to fill the whole provider disk on boot. The image bakes a fixed **8 GiB** root (sparse raw stays small) but nothing grew it onto the far larger provider disk, so a box ran on ~8 GiB — docker image storage + the brain's SQLite store share that one volume, so a single app install can fill it and the brain's first store write 500s login (the box looks offline though it is up). **Fix:** a runtime `repart.d` definition (`Type=root`, `GrowFileSystem=yes`, no size cap) + a `malmo-grow-root.service` oneshot running `systemd-repart --dry-run=no`, ordered `Before=docker.service host-agent.service` and **fail-closed** (`Requires=` from both, via a docker drop-in and host-agent's unit) so the write-heavy services refuse to start on an un-grown root. Build-time root stays pinned at 8 GiB; the two repart definitions are deliberately separate. Lean set gains `systemd-repart` + `libfdisk1` only (no `libcryptsetup`, cryptsetup cut preserved). Boot-proof asserts the tool is present and the unit reached `active`; real full-disk growth is a provider-box acceptance step. **Known:** the stock `systemd-repart.service` also runs the same config (harmless double execution — idempotent; consolidation deferred pending a live re-test), and a pre-existing unrelated `/setup` 503 boot-proof failure reproduces on clean `main`, tracked separately | done | | [hosted-setup-boot-proof-race.md](hosted-setup-boot-proof-race.md) — Close the `/setup` 503 boot-proof failure [hosted-grow-root-disk.md](hosted-grow-root-disk.md) tracked separately, and confirm the wildcard-TLS/`:443` path is green. The `unseeded` boot polled `POST /api/v1/setup` for **403** but broke on any of `403\|503\|409\|200`, so a transient **503** — Caddy answering "no ready `/api` upstream" in the first second after the control-plane stack comes up, before the brain's listener + dashboard route land — ended the loop and failed the proof, though the box is correct (the brain returns 403 unconditionally on hosted, no 503 path; the diag showed all four containers "Up <1–2 seconds"). **Fix:** break only on a definitive `403\|409\|200` and ride through `502\|503`, exactly as the `/api/v1/me` poll above already does; a genuinely stuck `/setup` still fails after the 30s window. The louder "`:443` never binds" symptom was a **broken-build artifact, not a live regression**: the build-and-boot CI job (`CI / Cloud image`, `publish=false`) passed both the `unseeded` and `seeded` boots (`seeded` hard-asserts `:443` bound + `caddy: wildcard TLS configured`), and every known real-box root cause (`certificates.automate` #301, seed-fetch keep-alive, static resolver) is already fixed in-tree — so no product change. Adds `docs/dev/hosted-boot-proof.md` (runbook: happy-path flow, brain-log milestones, symptom→where-to-look, how to run) and de-stales `TESTING.md`'s `/setup` gate description (the as-built SSO/403, superseding the secret 401/200). Test-lane + docs only | done | +| [mail-enum-value-maps.md](mail-enum-value-maps.md) — Generalize BYO-mail injection so an app whose mail config is a **boot-validated enum** auto-wires, where compose interpolation can't remap tokens (**closes #302**, extends [byo-outgoing-mail.md](byo-outgoing-mail.md) #122). Compose `environment:` can substitute / default / test-presence but not remap one enum onto another, so twenty's `EMAIL_DRIVER` (boot-rejects the empty string `${MALMO_MAIL_HOST:+smtp}` expands to unbound) and vaultwarden's `SMTP_SECURITY` (`off\|starttls\|force_tls` vs malmo's `none\|starttls\|tls`) stayed degraded. **Fix:** manifest `mail.env` declares, per app-owned var, the token for each mail state (`from: bound\|encryption` + a domain-covering `map`); the brain resolves the final token in Go (`mailAppEnvLines`) and stamps it under the app's own name — no `MALMO_` indirection, the same direct-stamp convention `config:` uses. `validateMail` reuses the config `app_env` guards (uppercase, never `MALMO_`/loader var) + requires the map cover its domain exactly. **Emitted in both bound and unbound states** (the load-bearing change) — unbound resolves the `none`/`unbound` tokens so the enum is present-and-valid and never boot-rejects; `rewriteEnvMail` strips the declared vars by name (not just the `MALMO_MAIL_` prefix) and re-resolves on rebind, no duplicate lines. Ledger `mail-enable-enum — twenty` → implemented; `capabilities.yml` gains id `mail-enable-enum` (version 1→2). Brain mechanism only — `catalog/twenty` + store-repo `apps/vaultwarden` manifest wiring + live send checks are the coordinated follow-up (catalog cut over to the store repo). `make check` green | done | diff --git a/docs/progress/mail-enum-value-maps.md b/docs/progress/mail-enum-value-maps.md new file mode 100644 index 00000000..5914a1ec --- /dev/null +++ b/docs/progress/mail-enum-value-maps.md @@ -0,0 +1,39 @@ +# Mail enum value maps — manifest-declared `mail.env`, brain-resolved tokens + +- **Status:** done +- **Date:** 2026-07-08 +- **Specs touched:** `APP_MANIFEST.md` (# D3 — new `mail.env` subsection), `SERVICE_PROVISIONING.md` (# BYO outgoing mail — enum-config paragraph), `docs/dev/catalog-import-gaps.md` (`mail-enable-enum — twenty` flipped to implemented), `docs/dev/capabilities.yml` (new id `mail-enable-enum`, version 1→2) + +Closes #302, extending [byo-outgoing-mail.md](byo-outgoing-mail.md) (#122). BYO mail injects malmo-canonical `MALMO_MAIL_*` and relies on the app's compose to adapt them, but compose `environment:` interpolation can only substitute / default (`:-`) / test-presence (`:+`) — it cannot **remap one enum's tokens onto another's**. Two shipped apps are degraded purely by this: twenty's `EMAIL_DRIVER` (`logger|smtp`, boot-rejects the empty string the `${MALMO_MAIL_HOST:+smtp}` gate expands to on an unbound box) and vaultwarden's `SMTP_SECURITY` (`off|starttls|force_tls`, a value remap off malmo's `none|starttls|tls`). This generalizes the projection the brain already hand-computes for `MALMO_MAIL_USE_TLS`/`USE_SSL`/`DSN` (which `mail.go` does in Go *because* "compose can't derive these from the string") into a manifest-declared value map. + +## What was done + +### Manifest (`internal/manifest/manifest.go`) + +- `Mail` gains `Env map[string]MailEnvMap`; new `MailEnvMap{From, Map}`. Each key is the app's own env-var name; `from` names a mail-state domain and `map` gives the app's token for each domain value. +- Two domains: `encryption` (`none|starttls|tls`, the bound provider's mode — treated as `none` when unbound) and the synthetic `bound` (`bound|unbound`). Domain names/values are exported constants (`MailFromEncryption`, `MailFromBound`, `MailBound`, `MailUnbound`) so lifecycle resolution references them, not string literals. The `encryption` domain values are kept as literals here (not imported from `store`) so the manifest layer stays free of a `store` dependency; a lifecycle test guards the two vocabularies against drift. +- `validateMail` extended: each env name is validated exactly like a config `app_env` (reuses `configEnvName` + the `MALMO_` / `reservedConfigEnv` guards — the token lands in `.env` under the app's own name, so it can't be allowed to clobber the `MALMO_` family or a loader var), `from` must be a known domain, and `map` must cover that domain **exactly** (right count + every key present + non-empty token) so resolution never hits an undeclared token. + +### Lifecycle (`internal/lifecycle/mail.go`, `lifecycle.go`) + +- New `mailAppEnvLines(mail, bound)` resolves each declared var to its token for the current state and renders `APP_VAR=token`, keys sorted for a byte-stable `.env`. `bound == nil` (unbound) resolves the `none`/`unbound` tokens. Validation guarantees the map covers the domain, so the lookup can't miss. +- **Emitted in both states** — the load-bearing change. Unlike `MALMO_MAIL_*` (bound-only), the declared vars are stamped whether or not a provider is bound, so an enum-driver app is present-and-valid unbound instead of boot-rejecting on an empty value. `writeEnv` (install) and `rewriteEnvMail` (rebind) both take `man.Mail` (already in scope at each call site — no extra manifest read) and append `mailAppEnvLines` after the `MALMO_MAIL_*` block. +- `rewriteEnvMail` now strips the declared vars **by name** (not just the `MALMO_MAIL_` prefix) before re-resolving, so a rebind — including an unbind, which drops `MALMO_MAIL_*` but keeps the enum vars with their unbound tokens — re-stamps cleanly with no duplicate lines. + +## Verification + +- `internal/manifest`: `mail.env` maps parse for both domains; `validateMail` rejects unknown `from`, an incomplete domain, an extra key, a misspelled key at the right count, an empty token, a `MALMO_`-prefixed name, a lowercase name, and a reserved runtime name (`PATH`). +- `internal/lifecycle`: install unbound stamps `EMAIL_DRIVER=logger` + `SMTP_SECURITY=off` and no `MALMO_MAIL_*`; install bound stamps `smtp`/`force_tls` alongside the `MALMO_MAIL_*` family; rebind bound→unbound→bound re-stamps the right tokens each way with exactly one line per var (no duplicates); a unit test drives `mailAppEnvLines` across all three encryption modes (also guarding the manifest-literal ↔ `store.MailEncryption*` alignment) and asserts an empty `mail.env` yields no lines (existing mail-only apps unaffected). +- Coverage on new code: `mailAppEnvLines` 100%, `validateMail` 100%; the mail branches added to `writeEnv`/`rewriteEnvMail` are all exercised (their residual uncovered lines are pre-existing `os` error paths). +- `make check` green. + +## Known gaps & deviations + +- **Catalog/store manifest wiring is a coordinated follow-up, not in this repo.** The `catalog/` tree was cut over to the store repo (`07aac3a`), so `catalog/twenty` (`EMAIL_DRIVER`) and the store repo's `apps/vaultwarden` (`SMTP_SECURITY`, its `outbound-mail-enum-mismatch` limitation) — plus the issue's live send checks for both — ship separately against this brain mechanism. The ledger entry is flipped to `implemented` per the "mechanism shipped" convention and names both apps for the revisit. +- **Reserved-name validation on `mail.env` keys is slightly beyond the issue's literal list** (it named only the domain-coverage + unknown-`from` rules). Kept deliberately: these vars land in the same `.env` as `MALMO_MAIL_*`, so a `MALMO_`-prefixed or loader-var key could clobber a brain-owned or process-critical value; the guard reuses the existing config `app_env` validator, matching the issue's "same direct-stamp convention as `config:`" framing. +- **Compose still needs a pass-through line** (`EMAIL_DRIVER: ${EMAIL_DRIVER}`) in the app's compose — the remap happens in Go, the compose just forwards the resolved value. That line is part of the store-repo manifest work above. + +## What's next + +- Wire `catalog/twenty` (`EMAIL_DRIVER` via `from: bound`) and the store repo's `apps/vaultwarden` (`SMTP_SECURITY` via `from: encryption`), flip vaultwarden's `outbound-mail-enum-mismatch` and twenty's row toward `full` after a live send check. +- The `capabilities.yml` id `mail-enable-enum` now fires the mechanical re-screen of any app that recorded a wait on that gap-class (`CAPABILITIES.md`). diff --git a/docs/specs/APP_MANIFEST.md b/docs/specs/APP_MANIFEST.md index 1620ea5c..54f2abd1 100644 --- a/docs/specs/APP_MANIFEST.md +++ b/docs/specs/APP_MANIFEST.md @@ -256,6 +256,22 @@ environment: MAILER_FROM: "${MALMO_MAIL_FROM:-kimai@example.com}" ``` +**Enum-style config a compose gate can't reach (`mail.env`).** Some apps take a mail setting as a *boot-validated enum* rather than a boolean or free string — twenty's `EMAIL_DRIVER` (`logger` | `smtp`, rejects empty) or vaultwarden's `SMTP_SECURITY` (`off` | `starttls` | `force_tls`, whose tokens differ from malmo's `none`/`starttls`/`tls`). Compose `environment:` interpolation can only substitute, default (`:-`), or test-presence (`:+`); it cannot *remap* one enum's tokens onto another's, and the established `${MALMO_MAIL_HOST:+…}` gate expands to the empty string on an unbound box, which boot-rejects the enum. So the manifest declares the mapping and the brain resolves the final token in Go, stamping it into the `.env` under the app's own name — no `MALMO_` indirection, the same direct-stamp convention `config:` uses (# D4): + +```yaml +mail: + optional: true + env: + EMAIL_DRIVER: # twenty — is a provider bound? + from: bound # domain: bound | unbound + map: { bound: smtp, unbound: logger } + SMTP_SECURITY: # vaultwarden — the provider's mode + from: encryption # domain: none | starttls | tls (unbound ⇒ none) + map: { none: "off", starttls: starttls, tls: force_tls } +``` + +Each entry's `from` names a mail-state domain — `bound` (whether any provider is bound) or `encryption` (the bound provider's mode, treated as `none` when unbound) — and `map` must cover that domain exactly (every value, none extra), so resolution never hits an undeclared token. The var name is validated like a config `app_env` (uppercase identifier, never the `MALMO_` prefix or a loader/runtime var). **Unlike the rest of `MALMO_MAIL_*`, these vars are stamped in *both* states:** unbound resolves the `unbound`/`none` tokens, so the enum is always present and valid and the app never boot-rejects on an empty value. The app's compose passes the resolved value straight through (`EMAIL_DRIVER: ${EMAIL_DRIVER}`) — no remap, because the brain already did it. + v1 admits only `optional: true` — an app that *can't* run unbound (`optional: false` or a bare `mail: {}`) is rejected at parse, because a box with no registered providers couldn't install it. Required-mail semantics (blocking install until a provider is picked) is a possible later loosening; declare-and-degrade is the v1 contract. ### D4. User-supplied configuration diff --git a/docs/specs/SERVICE_PROVISIONING.md b/docs/specs/SERVICE_PROVISIONING.md index 5e9475ce..1a93a7c6 100644 --- a/docs/specs/SERVICE_PROVISIONING.md +++ b/docs/specs/SERVICE_PROVISIONING.md @@ -202,6 +202,8 @@ MALMO_MAIL_DSN=smtps://box%40example.com:...@smtp.fastmail.com:465 The DSN scheme is `smtps://` for implicit TLS and `smtp://` otherwise (SMTP-URL consumers negotiate STARTTLS opportunistically; an app needing the exact mode reads `MALMO_MAIL_ENCRYPTION`). `MALMO_MAIL_USE_TLS` / `MALMO_MAIL_USE_SSL` are boolean projections of that mode (STARTTLS vs implicit TLS, at most one true) for apps that take two separate flags — e.g. Django's `EMAIL_USE_TLS` / `EMAIL_USE_SSL`, which Paperless surfaces — since a compose file can't derive a boolean from the encryption string. Credentials are URL-escaped. The app's compose maps the vars to whatever it expects, per the family contract above — with a compose default for the unbound case so absence degrades cleanly (Kimai: `MAILER_URL: "${MALMO_MAIL_DSN:-null://null}"`). +**Enum-style mail config (`mail.env`, `APP_MANIFEST.md` # D3):** where a compose `:+` gate can't remap the box's mail state onto an app's own enum tokens — an `EMAIL_DRIVER` that boot-rejects the empty string, an `SMTP_SECURITY` whose values differ from malmo's `none`/`starttls`/`tls` — the manifest declares the mapping (per app-owned var, `from: bound|encryption` + a token map) and the brain resolves the final token in Go, stamping it under the app's own var name. These are emitted in **both** bound and unbound states (unbound → the `unbound`/`none` token), so the enum is always present-and-valid — the one part of the family the unbound case still writes, precisely so the app doesn't boot-reject with mail off. + **Propagation:** env is read at container create, so a rebind re-stamps the `.env` and recreates the instance's containers immediately (stopped instances pick it up at next start). Editing or deleting a *provider* does not re-stamp bound apps: they keep the previously injected values until their next rebind or reinstall — v1 accepts this lag, and the Settings UI says so. Deleting a provider unbinds its apps in the brain's state (the next rebind of each app drops the vars). **Explicitly not in v1** (deferral, not rejection — `NEXT.md` # Outgoing mail): no malmo-run relay/smarthost, no per-app rate limiting or queue, no inbound mail anything. If email grows more surface (a default box-wide provider, brain-sent notification email riding the same providers), this section promotes to its own `OUTGOING_MAIL.md`. diff --git a/internal/lifecycle/lifecycle.go b/internal/lifecycle/lifecycle.go index 1861cb70..ad190154 100644 --- a/internal/lifecycle/lifecycle.go +++ b/internal/lifecycle/lifecycle.go @@ -668,7 +668,7 @@ func (m *Manager) install(ctx context.Context, man *manifest.Manifest, composeBy if err := m.writeOverride(id, man, composeBytes, pins, iso, lim); err != nil { return rollback(fmt.Errorf("override: %w", err)) } - if err := m.writeEnv(id, slug, iso); err != nil { + if err := m.writeEnv(id, slug, iso, man.Mail); err != nil { return rollback(fmt.Errorf("env: %w", err)) } @@ -1656,7 +1656,7 @@ func (m *Manager) writeOverride(id string, man *manifest.Manifest, composeBytes return os.WriteFile(filepath.Join(m.instanceDir(id), "compose.override.yml"), out, 0o644) } -func (m *Manager) writeEnv(id, slug string, iso isolation) error { +func (m *Manager) writeEnv(id, slug string, iso isolation, mail *manifest.Mail) error { dataDir, _ := filepath.Abs(filepath.Join(m.instanceDir(id), "data")) // MALMO_APP_URL is the app's own public URL (apps that build absolute links // read it). Hosted is HTTPS at "..malmo.network"; appliance is @@ -1722,9 +1722,15 @@ func (m *Manager) writeEnv(id, slug string, iso isolation) error { if err != nil && !errors.Is(err, store.ErrNotFound) { return fmt.Errorf("load mail binding: %w", err) } + var bound *store.MailProvider if err == nil { + bound = &mp lines = append(lines, mailEnvLines(mp)...) } + // Manifest-declared mail.env vars (#302) are emitted in both states — unbound + // resolves the none/unbound tokens — so an enum-driver app boots valid without + // a provider bound, where the MALMO_MAIL_* family injects nothing. + lines = append(lines, mailAppEnvLines(mail, bound)...) env := strings.Join(append(lines, ""), "\n") return os.WriteFile(filepath.Join(m.instanceDir(id), ".env"), []byte(env), 0o644) } diff --git a/internal/lifecycle/lifecycle_mail_test.go b/internal/lifecycle/lifecycle_mail_test.go index 292062b6..8fb2e74b 100644 --- a/internal/lifecycle/lifecycle_mail_test.go +++ b/internal/lifecycle/lifecycle_mail_test.go @@ -8,10 +8,12 @@ import ( "context" "os" "path/filepath" + "slices" "strings" "testing" "time" + "github.com/malmoos/malmo/internal/manifest" "github.com/malmoos/malmo/internal/store" ) @@ -37,6 +39,32 @@ services: image: traefik/whoami:v1.10.3 ` +// mailEnumManifest declares mail.env maps for both domains (#302): a `bound` +// enum (twenty's EMAIL_DRIVER) and an `encryption` enum (vaultwarden's +// SMTP_SECURITY). `off` is quoted so YAML keeps it a string, not a boolean. +const mailEnumManifest = ` +id: mailenumapp +manifest_version: 1 +name: Mail Enum App +version: "1.0" +compose_file: compose.yml +main_service: app +main_port: 8080 +preferred_slugs: [mailenumapp] +permissions: + internet: true + lan: false +mail: + optional: true + env: + EMAIL_DRIVER: + from: bound + map: { bound: smtp, unbound: logger } + SMTP_SECURITY: + from: encryption + map: { none: "off", starttls: starttls, tls: force_tls } +` + func testProvider() store.MailProvider { return store.MailProvider{ ID: "mp_test", Label: "Fastmail", Host: "smtp.fastmail.com", Port: 465, @@ -194,6 +222,127 @@ func countCalls(calls []call, method string) int { return n } +func installEnumMailApp(t *testing.T, e *testEnv, providerID string) (store.Instance, string) { + t.Helper() + e.writeCatalogApp(t, "mailenumapp", mailCompose, mailEnumManifest) + e.docker.digests[testImage] = testDigest + inst, err := e.m.Install(context.Background(), "mailenumapp", + Owner{UserID: "u_admin", Username: "admin"}, store.ScopeHousehold, nil, providerID, nil, nil) + if err != nil { + t.Fatalf("install: %v", err) + } + env, err := os.ReadFile(filepath.Join(e.stateDir, "instances", inst.ID, ".env")) + if err != nil { + t.Fatalf("read env: %v", err) + } + return inst, string(env) +} + +// A mail.env-declaring app installed unbound gets the declared enum vars stamped +// with their none/unbound tokens (so it boots valid), and no MALMO_MAIL_* (#302). +func TestInstallUnboundStampsEnumMailVars(t *testing.T) { + e := newTestEnv(t) + _, env := installEnumMailApp(t, e, "") + if strings.Contains(env, "MALMO_MAIL_") { + t.Fatalf("unbound install must inject no MALMO_MAIL_ vars, got:\n%s", env) + } + for k, v := range map[string]string{"EMAIL_DRIVER": "logger", "SMTP_SECURITY": "off"} { + if got := envValue(env, k); got != v { + t.Errorf("unbound %s = %q, want %q", k, got, v) + } + } +} + +// Bound, the declared enum vars resolve their mapped tokens and coexist with the +// MALMO_MAIL_* family. +func TestInstallBoundStampsEnumMailVars(t *testing.T) { + e := newTestEnv(t) + if err := e.store.CreateMailProvider(testProvider()); err != nil { // tls + t.Fatalf("create provider: %v", err) + } + _, env := installEnumMailApp(t, e, "mp_test") + for k, v := range map[string]string{"EMAIL_DRIVER": "smtp", "SMTP_SECURITY": "force_tls"} { + if got := envValue(env, k); got != v { + t.Errorf("bound %s = %q, want %q", k, got, v) + } + } + if got := envValue(env, "MALMO_MAIL_HOST"); got != "smtp.fastmail.com" { + t.Errorf("MALMO_MAIL_HOST = %q, want smtp.fastmail.com", got) + } +} + +// A rebind re-stamps the declared enum vars for the new state — including the +// unbind, where MALMO_MAIL_* is stripped but the enum vars flip to their unbound +// tokens and stay present (a bare strip would boot-reject twenty's driver) — with +// no duplicate lines left behind. +func TestRebindReStampsEnumMailVars(t *testing.T) { + e := newTestEnv(t) + if err := e.store.CreateMailProvider(testProvider()); err != nil { + t.Fatalf("create provider: %v", err) + } + inst, _ := installEnumMailApp(t, e, "mp_test") + envPath := filepath.Join(e.stateDir, "instances", inst.ID, ".env") + + if err := e.m.RebindMail(context.Background(), inst.ID, ""); err != nil { + t.Fatalf("unbind: %v", err) + } + raw, _ := os.ReadFile(envPath) + if strings.Contains(string(raw), "MALMO_MAIL_") { + t.Fatalf("unbind must strip MALMO_MAIL_ vars, got:\n%s", raw) + } + for k, v := range map[string]string{"EMAIL_DRIVER": "logger", "SMTP_SECURITY": "off"} { + if got := envValue(string(raw), k); got != v { + t.Errorf("unbound %s = %q, want %q", k, got, v) + } + if n := strings.Count(string(raw), k+"="); n != 1 { + t.Fatalf("%s stamped %d times after unbind, want 1:\n%s", k, n, raw) + } + } + + if err := e.m.RebindMail(context.Background(), inst.ID, "mp_test"); err != nil { + t.Fatalf("rebind: %v", err) + } + raw, _ = os.ReadFile(envPath) + for k, v := range map[string]string{"EMAIL_DRIVER": "smtp", "SMTP_SECURITY": "force_tls"} { + if got := envValue(string(raw), k); got != v { + t.Errorf("rebound %s = %q, want %q", k, got, v) + } + if n := strings.Count(string(raw), k+"="); n != 1 { + t.Fatalf("%s stamped %d times after rebind, want 1:\n%s", k, n, raw) + } + } +} + +// mailAppEnvLines resolves each domain and each encryption mode. The encryption +// cases also guard manifest's literal domain values against store.MailEncryption* +// drift, since the two vocabularies are only bridged at resolution time. +func TestMailAppEnvLinesResolvesDomains(t *testing.T) { + mail := &manifest.Mail{ + Optional: true, + Env: map[string]manifest.MailEnvMap{ + "EMAIL_DRIVER": {From: manifest.MailFromBound, Map: map[string]string{"bound": "smtp", "unbound": "logger"}}, + "SMTP_SECURITY": {From: manifest.MailFromEncryption, Map: map[string]string{"none": "off", "starttls": "starttls", "tls": "force_tls"}}, + }, + } + if got, want := mailAppEnvLines(mail, nil), []string{"EMAIL_DRIVER=logger", "SMTP_SECURITY=off"}; !slices.Equal(got, want) { + t.Fatalf("unbound = %v, want %v (sorted by name)", got, want) + } + for enc, sec := range map[string]string{ + store.MailEncryptionNone: "off", + store.MailEncryptionSTARTTLS: "starttls", + store.MailEncryptionTLS: "force_tls", + } { + p := testProvider() + p.Encryption = enc + if got, want := mailAppEnvLines(mail, &p), []string{"EMAIL_DRIVER=smtp", "SMTP_SECURITY=" + sec}; !slices.Equal(got, want) { + t.Errorf("encryption %s: got %v, want %v", enc, got, want) + } + } + if lines := mailAppEnvLines(&manifest.Mail{Optional: true}, nil); lines != nil { + t.Errorf("empty mail.env must yield no lines, got %v", lines) + } +} + func TestMailDSN(t *testing.T) { p := testProvider() diff --git a/internal/lifecycle/mail.go b/internal/lifecycle/mail.go index 66225a63..f1df9e18 100644 --- a/internal/lifecycle/mail.go +++ b/internal/lifecycle/mail.go @@ -15,9 +15,11 @@ import ( "net/url" "os" "path/filepath" + "sort" "strconv" "strings" + "github.com/malmoos/malmo/internal/manifest" "github.com/malmoos/malmo/internal/store" ) @@ -41,6 +43,43 @@ func mailEnvLines(p store.MailProvider) []string { } } +// mailAppEnvLines resolves each manifest-declared mail.env var to its token for +// the box's current mail state and renders it as `APP_VAR=token` (#302). Unlike +// the MALMO_MAIL_* family — injected only when bound — these are emitted in both +// states, so an app whose mail switch is a boot-validated enum is present and +// valid even unbound (bound == nil): `encryption` reads none, `bound` reads +// unbound. Keys are sorted so a rewrite produces a byte-stable .env. Validation +// guarantees every declared map covers its domain, so the lookup can't miss. +func mailAppEnvLines(mail *manifest.Mail, bound *store.MailProvider) []string { + if mail == nil || len(mail.Env) == 0 { + return nil + } + names := make([]string, 0, len(mail.Env)) + for name := range mail.Env { + names = append(names, name) + } + sort.Strings(names) + lines := make([]string, 0, len(names)) + for _, name := range names { + em := mail.Env[name] + var domainValue string + switch em.From { + case manifest.MailFromEncryption: + domainValue = store.MailEncryptionNone // unbound ⇒ no encryption + if bound != nil { + domainValue = bound.Encryption + } + case manifest.MailFromBound: + domainValue = manifest.MailUnbound + if bound != nil { + domainValue = manifest.MailBound + } + } + lines = append(lines, name+"="+em.Map[domainValue]) + } + return lines +} + // mailDSN renders a provider as a Symfony-style SMTP URL // (smtp[s]://user:pass@host:port). Implicit TLS is the smtps scheme; starttls // and none both stay smtp:// — SMTP URL consumers negotiate STARTTLS @@ -90,7 +129,7 @@ func (m *Manager) RebindMail(ctx context.Context, id, providerID string) error { } else if err := m.store.SetInstanceMailBinding(id, providerID); err != nil { return fmt.Errorf("bind mail provider: %w", err) } - if err := m.rewriteEnvMail(id); err != nil { + if err := m.rewriteEnvMail(id, man.Mail); err != nil { return fmt.Errorf("rewrite env: %w", err) } @@ -106,30 +145,47 @@ func (m *Manager) RebindMail(ctx context.Context, id, providerID string) error { return nil } -// rewriteEnvMail re-stamps only the MALMO_MAIL_* lines of an instance's .env -// from the current binding, leaving every other line byte-identical — unlike a -// full writeEnv it needs no install-time isolation state, and a stable secret -// can't be re-rolled by accident. -func (m *Manager) rewriteEnvMail(id string) error { +// rewriteEnvMail re-stamps only the mail-owned lines of an instance's .env from +// the current binding — the MALMO_MAIL_* family plus any manifest-declared +// mail.env vars (#302) — leaving every other line byte-identical, so unlike a +// full writeEnv it needs no install-time isolation state and a stable secret +// can't be re-rolled by accident. The declared vars are stamped under the app's +// own names, so they're stripped by name (not the MALMO_MAIL_ prefix) and +// re-resolved for the new state — including unbound, which the MALMO_MAIL_* +// family drops but the declared enum vars keep (with their unbound tokens). +func (m *Manager) rewriteEnvMail(id string, mail *manifest.Mail) error { path := filepath.Join(m.instanceDir(id), ".env") raw, err := os.ReadFile(path) if err != nil { return err } + declared := map[string]bool{} + if mail != nil { + for name := range mail.Env { + declared[name] = true + } + } lines := strings.Split(strings.TrimRight(string(raw), "\n"), "\n") kept := lines[:0] for _, l := range lines { - if !strings.HasPrefix(l, "MALMO_MAIL_") { - kept = append(kept, l) + if strings.HasPrefix(l, "MALMO_MAIL_") { + continue + } + if name, _, ok := strings.Cut(l, "="); ok && declared[name] { + continue } + kept = append(kept, l) } mp, err := m.store.GetInstanceMailProvider(id) if err != nil && !errors.Is(err, store.ErrNotFound) { return err } + var bound *store.MailProvider if err == nil { + bound = &mp kept = append(kept, mailEnvLines(mp)...) } + kept = append(kept, mailAppEnvLines(mail, bound)...) env := strings.Join(append(kept, ""), "\n") return os.WriteFile(path, []byte(env), 0o644) } diff --git a/internal/manifest/manifest.go b/internal/manifest/manifest.go index 830961ab..0ad5abb6 100644 --- a/internal/manifest/manifest.go +++ b/internal/manifest/manifest.go @@ -423,6 +423,31 @@ type ServiceDep struct { // every app must still install and run unbound (validateMail rejects false). type Mail struct { Optional bool `yaml:"optional"` + + // Env auto-wires app-owned mail env vars whose valid values are an enum the + // app boot-validates, where a compose `${MALMO_MAIL_*:+…}` gate can't remap + // one token set onto another (#302). Each key is the app's own env-var name; + // the brain resolves the mapped token for the box's current mail state and + // stamps it into the .env directly — no MALMO_ indirection, the same as a + // config field. Emitted in both bound and unbound states (unbound uses the + // none/unbound tokens), so an enum-driver app is present-and-valid on a box + // with no provider registered. Absent ⇒ only the MALMO_MAIL_* family is + // injected. See APP_MANIFEST.md # D3, SERVICE_PROVISIONING.md # BYO outgoing mail. + Env map[string]MailEnvMap `yaml:"env,omitempty"` +} + +// MailEnvMap projects one malmo mail-state domain onto an app's own token set +// (APP_MANIFEST.md # D3). From names the source domain and Map gives the app's +// token for each value in it: +// +// encryption — the bound provider's mode: none | starttls | tls (unbound ⇒ none) +// bound — whether any provider is bound: bound | unbound +// +// Map must cover its domain exactly (every value, none extra) so resolution +// never hits an undeclared token in any mail state (validateMail enforces it). +type MailEnvMap struct { + From string `yaml:"from"` + Map map[string]string `yaml:"map"` } // ConfigField is one user-supplied configuration field (APP_MANIFEST.md # D4). @@ -684,10 +709,36 @@ func (m *Manifest) validateServices() error { return nil } +// Mail-state domain names a MailEnvMap.From may take (APP_MANIFEST.md # D3). +const ( + MailFromEncryption = "encryption" + MailFromBound = "bound" +) + +// Values of the synthetic `bound` domain (whether a provider is bound). +const ( + MailBound = "bound" + MailUnbound = "unbound" +) + +// mailEnvDomains maps each MailEnvMap.From to the exact token set its Map must +// cover (APP_MANIFEST.md # D3). The encryption domain values mirror +// store.MailEncryption* (none|starttls|tls); kept as literals here so the +// manifest layer stays free of a store dependency, and lifecycle resolution +// bridges the two (a lifecycle test guards the alignment). +var mailEnvDomains = map[string][]string{ + MailFromEncryption: {"none", "starttls", "tls"}, + MailFromBound: {MailBound, MailUnbound}, +} + // validateMail checks the optional outgoing-mail block (APP_MANIFEST.md # D). // Absent ⇒ no-op. v1 supports only `optional: true`: an app that cannot run // unbound would have to fail install on every box with no provider registered, -// so required mail is rejected until a real consumer needs it. +// so required mail is rejected until a real consumer needs it. Each mail.env +// entry (#302) is validated like a config app_env — the token lands in the .env +// under the app's own name, so it must be a bare uppercase identifier that can't +// clobber the MALMO_ family or a loader var — plus its map must cover the From +// domain exactly, so resolution never hits an undeclared token. func (m *Manifest) validateMail() error { if m.Mail == nil { return nil @@ -695,6 +746,33 @@ func (m *Manifest) validateMail() error { if !m.Mail.Optional { return fmt.Errorf("mail: v1 supports only optional mail (set `optional: true`; apps must run unbound)") } + for name, em := range m.Mail.Env { + if !configEnvName.MatchString(name) { + return fmt.Errorf("mail.env: name %q must be an uppercase env-var name (e.g. EMAIL_DRIVER)", name) + } + if strings.HasPrefix(name, "MALMO_") { + return fmt.Errorf("mail.env: name %q may not use the reserved MALMO_ prefix", name) + } + if reservedConfigEnv[name] { + return fmt.Errorf("mail.env: name %q is a reserved runtime variable and may not be set", name) + } + domain, ok := mailEnvDomains[em.From] + if !ok { + return fmt.Errorf("mail.env[%s]: unknown from %q (allowed: encryption, bound)", name, em.From) + } + if len(em.Map) != len(domain) { + return fmt.Errorf("mail.env[%s]: map must cover exactly the %s domain %v", name, em.From, domain) + } + for _, v := range domain { + token, ok := em.Map[v] + if !ok { + return fmt.Errorf("mail.env[%s]: map is missing key %q for domain %s", name, v, em.From) + } + if token == "" { + return fmt.Errorf("mail.env[%s]: map[%s] token must not be empty", name, v) + } + } + } return nil } diff --git a/internal/manifest/manifest_test.go b/internal/manifest/manifest_test.go index 0acf4679..712091cc 100644 --- a/internal/manifest/manifest_test.go +++ b/internal/manifest/manifest_test.go @@ -159,6 +159,83 @@ main_port: 8001 } } +func TestParseMailEnvMap(t *testing.T) { + src := []byte(` +id: twenty +manifest_version: 1 +name: Twenty +version: "1.0" +compose_file: compose.yml +main_service: app +main_port: 3000 +mail: + optional: true + env: + EMAIL_DRIVER: + from: bound + map: { bound: smtp, unbound: logger } + SMTP_SECURITY: + from: encryption + map: { none: "off", starttls: starttls, tls: force_tls } +`) + m, err := Parse(src) + if err != nil { + t.Fatalf("parse: %v", err) + } + if got := m.Mail.Env["EMAIL_DRIVER"]; got.From != MailFromBound || got.Map[MailBound] != "smtp" || got.Map[MailUnbound] != "logger" { + t.Fatalf("EMAIL_DRIVER map not parsed: %+v", got) + } + if got := m.Mail.Env["SMTP_SECURITY"]; got.From != MailFromEncryption || got.Map["tls"] != "force_tls" || got.Map["none"] != "off" { + t.Fatalf("SMTP_SECURITY map not parsed: %+v", got) + } +} + +func TestValidateMailEnvRejects(t *testing.T) { + cases := map[string]string{ + "unknown from": ` EMAIL_DRIVER: + from: mystery + map: { bound: smtp, unbound: logger }`, + "incomplete encryption domain": ` SMTP_SECURITY: + from: encryption + map: { none: "off", tls: force_tls }`, + "extra key": ` EMAIL_DRIVER: + from: bound + map: { bound: smtp, unbound: logger, other: x }`, + "misspelled key (right count)": ` EMAIL_DRIVER: + from: bound + map: { bound: smtp, notunbound: logger }`, + "empty token": ` EMAIL_DRIVER: + from: bound + map: { bound: smtp, unbound: "" }`, + "malmo-prefixed name": ` MALMO_MAIL_HOST: + from: bound + map: { bound: smtp, unbound: logger }`, + "lowercase name": ` email_driver: + from: bound + map: { bound: smtp, unbound: logger }`, + "reserved runtime name": ` PATH: + from: bound + map: { bound: smtp, unbound: logger }`, + } + for name, env := range cases { + src := []byte(` +id: app +manifest_version: 1 +name: App +version: "1.0" +compose_file: compose.yml +main_service: app +main_port: 8080 +mail: + optional: true + env: +` + env + "\n") + if _, err := Parse(src); err == nil { + t.Errorf("%s: parse must reject the mail.env map", name) + } + } +} + func TestParseSecretsNormalizesBytes(t *testing.T) { src := []byte(` id: kan