Skip to content

fix(helm): start cleanly on TLS/Redis defaults, move webhook and registry keys to Secrets - #48

Open
Andrew Grathwohl (agrathwohl) wants to merge 2 commits into
mainfrom
converge/fire-322
Open

Andrew Grathwohl (agrathwohl) wants to merge 2 commits into
mainfrom
converge/fire-322

Conversation

@agrathwohl

@agrathwohl Andrew Grathwohl (agrathwohl) commented Oct 5, 2026 •

Copy link
Copy Markdown

Fixes FIRE-322

What was broken and what changed

The chart failed to start in several configurations, including its defaults.

  • The generate-certs init container ran as UID 1000 and wrote a certificate key that the firewall image (UID 1001) could not read, so nginx crash-looped on Permission denied. initContainers.certGenerator.securityContext.runAsUser now defaults to 1001.

  • Default TLS installs failed because the certificate SAN list was empty. It now always includes the Service name and its DNS forms, plus any configured domains. The cert script runs with set -e, so a failed step stops it.

  • tls.existingSecret without tls.certManager could not load certificates because the keys did not match the filenames nginx reads. tls.crt and tls.key always mount as fullchain.pem and privkey.pem. The new tls.remapKeys defaults to true; setting it to false mounts secrets keyed fullchain.pem and privkey.pem without remapping. tls.certManager now only controls the ca.crt projection.

  • With Redis enabled, Kubernetes service-link environment variables from a Service named redis set REDIS_PORT and caused a ValueError at startup. enableServiceLinks is now false on the pod.

  • The default redis.host: redis failed to connect because nginx's resolver ignores DNS search domains. Bare hosts now render as <host>.<release namespace>.svc.<clusterDomain>. A new wait-for-redis init container holds the pod in Init until Redis accepts TCP connections.

  • webhook.authHeader and pathRouting.privateRegistry.apiKey rendered in plain text in the ConfigMap. They move to Secrets, passed as environment variables WEBHOOK_AUTH_HEADER and PRIVATE_REGISTRY_KEY. New existingSecret and existingSecretKey options support existing Secrets for both.

  • Default single-replica installs had a PodDisruptionBudget with minAvailable: 1 that blocked node drains. The PDB now only renders when replicaCount > 1, or autoscaling.minReplicas > 1 with autoscaling on.

Upgrade notes

  • If you use a TLS secret keyed fullchain.pem/privkey.pem that worked without certManager, set tls.remapKeys: false.
  • Single-replica installs no longer include a PDB.
  • If you override initContainers.certGenerator.securityContext, its runAsUser must match the firewall UID 1001.

How it was validated

Tested on kind with Kubernetes 1.36.1 and firewall image 2.9.3. The chart on main reproduced the privkey.pem Permission denied, fullchain.pem No such file, and REDIS_PORT ValueError crashes.

Verified all pods reach Ready state for:

  • Default install
  • tls.existingSecret (served the secret's certificate)
  • Secret keyed fullchain.pem/privkey.pem with tls.remapKeys: false
  • Redis enabled alongside a Service named redis (3 client connections in redis-cli client list)
  • Webhook and private registry configured (values resolved in /app/config.env from Secrets, not the ConfigMap)

Set redis.host to an invalid address and confirmed the pod remained in Init state, logging Waiting for nope.<namespace>.svc.cluster.local/6379.

Passed helm lint, kubeconform against Kubernetes 1.30.0 for the default install, examples, and 4 code paths, and helm unittest with PR #10's 2 test files, 5 of 5 passing.

What is left

  • cloudformation/firewall-eks.yaml ChartVersion default is still 0.11.7.
  • Basic auth password and private registry username/password remain in the ConfigMap, as moving them requires a firewall image change.

Note

Medium Risk
Changes default TLS mounting, Redis connectivity, PDB presence, and secret wiring—upgrades may need tls.remapKeys: false or lose single-replica PDBs, but fixes prior crash-loop defaults.

Overview
Helm chart 0.12.0 fixes several default-install crash paths and tightens how TLS, Redis, and secrets are handled.

Startup and TLS: The cert generator init container runs as UID 1001 so nginx can read privkey.pem, always adds Service DNS SANs (plus clusterDomain), and uses set -e. Existing TLS secrets mount via tls.remapKeys (tls.crt/tls.key → fullchain.pem/privkey.pem by default); tls.certManager now only controls optional ca.crt projection.

Redis: Bare redis.host names are FQDN-qualified for nginx’s resolver, a wait-for-redis init container blocks until TCP is up, and enableServiceLinks: false avoids REDIS_PORT service-link env collisions.

Secrets: webhook.authHeader and pathRouting.privateRegistry.apiKey are removed from the ConfigMap; chart-created or existingSecret values are injected as WEBHOOK_AUTH_HEADER and PRIVATE_REGISTRY_KEY.

Ops: PDB renders only when more than one replica is always running; README documents new values (clusterDomain, Redis host behavior, TLS/PDB notes).

Reviewed by Cursor Bugbot for commit 07df1e8. Configure here.

@agrathwohl
Andrew Grathwohl (agrathwohl) requested a review from a team as a code owner October 5, 2026 18:28

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issue.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 07df1e8. Configure here.

name: {{ .privateRegistry.existingSecret | default (printf "%s-private-registry" (include "socket-firewall.fullname" $)) }}
key: {{ .privateRegistry.existingSecretKey }}
{{- end }}
{{- end }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secret updates skip pod rollout

Medium Severity

Moving webhook.authHeader and pathRouting.privateRegistry.apiKey out of the ConfigMap means a Helm upgrade that only changes those values no longer touches checksum/config. Pods keep the old WEBHOOK_AUTH_HEADER and PRIVATE_REGISTRY_KEY until they are recreated, so rotated credentials do not take effect.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 07df1e8. Configure here.

The ConfigMap still renders two secrets in plain text. This commit moves
them to Secrets.

Remove `socket.basicAuthPassword` from the ConfigMap. Store it in the
`<fullname>-basic-auth` Secret and pass it as the
`SOCKET_BASIC_AUTH_PASSWORD` environment variable. Add
`socket.basicAuthExistingSecret` and `socket.basicAuthExistingSecretKey`
(defaults to `SOCKET_BASIC_AUTH_PASSWORD`) to support using an existing
Secret instead.

Remove `pathRouting.privateRegistry.password` from the ConfigMap. Store
it in the `<fullname>-private-registry` Secret alongside the API key and
pass it as `PATH_ROUTING_PRIVATE_REGISTRY_PASSWORD`. Add
`pathRouting.privateRegistry.existingSecretPasswordKey` (defaults to
`PATH_ROUTING_PRIVATE_REGISTRY_PASSWORD`). If
`pathRouting.privateRegistry.existingSecret` is set and
`pathRouting.privateRegistry.username` is present, the chart reads the
password key. Otherwise, it reads the API key. Usernames stay in the
ConfigMap.

No firewall image change is needed. The 2.9.3 config tool checks an
environment variable override before `socket.yml` for every key (key
path uppercased, dots to underscores). Verified that
`socket-proxy-config-tool generate` with the rendered ConfigMap and
these two environment variables writes the basic auth password and
`PRIVATE_REGISTRY_AUTH_CREDENTIAL` as `username:password` to
`/app/config.env`.

Add a `checksum/secret` pod annotation using the rendered
`templates/secret.yaml`. Without it, changing the webhook auth header or
registry API key no longer restarted pods once those values left the
ConfigMap. With it, any change to a chart-created Secret value triggers
a rollout.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to add CI for this, in order to catch future regressions?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants