feat(automation): provision the KV store secret for Replicated installs

[hieptl]

Description

The automation service only offers its key-value store when AUTOMATION_KV_SECRET is set, and nothing in the chart or the Replicated manifests set it. On a self-hosted install:

• GET /api/automation/v1/capabilities does not list kvStore;
• a run gets no AUTOMATION_KV_TOKEN.

Every run starts in a fresh sandbox there, so an automation that keeps state between runs has nowhere to put it. The automation templates that track work across runs (issue polling, GitHub code review and issue-to-PR dispatch, the Jira poller) either fail or repeat their work.

This provisions the secret the same way the git sync secret is provisioned (#1213):

• replicated/config.yaml: hidden, generated automation_kv_secret password item.
• replicated/secrets.yaml: passes it to the openhands-secrets chart when automations are enabled.
• charts/openhands-secrets: config.automation_kv_secret value and the automation-kv-secret Secret (key kv-secret).
• charts/openhands/charts/automation: kvSecretFromSecret value and an AUTOMATION_KV_SECRET env entry on the automation pods. The secretKeyRef is optional, so an install without the Secret keeps starting with the store off.
• charts/openhands/values.yaml: mirrors kvSecretFromSecret under automation:.
• replicated/openhands.yaml: the item joins secretsChecksum.
• charts/openhands/templates/troubleshoot/support-bundle.yaml: a new app/automation-config-snapshot collector reports whether AUTOMATION_KV_SECRET is set (set / empty_or_unset, never the value), so a support bundle shows whether the KV store is configured. It runs in the automation pod, the only pod that receives the variable.

Checks run locally:

• helm unittest charts/openhands (182 passed), helm unittest charts/openhands/charts/automation (26 passed) and helm unittest charts/openhands-secrets (12 passed), including the three new suites below.
• python3 scripts/check_secret_checksum.py: all 46 password fields covered.
• pytest scripts/test_keycloak_realm_template.py scripts/test_openhands_secrets_template.py scripts/test_replicated_resolver_label.py: 29 passed.
• helm template renders the Secret only when automation.enabled is true, and the env entry on both the migrate and the automation containers.

Deployed to the Replicated fleet VM shared-4 (clean install) with the OHE test procedure, together with the related changes:

• OpenHands-Cloud branch hieple/deploy-2026-10-01 (ac80547) is main plus:
  • the chart change from feat(automation): provision the KV store secret for Replicated installs #1326 (AUTOMATION_KV_SECRET);
  • automation pinned to sha-03687e7 (feat: support agent profiles in cloud mode automation#541; image revision label 03687e70dfc0);
  • agent-canvas pinned to sha-9f60167 (feat(automations): show templates on cloud backends and offer native git integrations OpenHands#17830; image revision label 9f601672d942), which ships the extensions npm package at feat(automations): run the automation templates on OpenHands Cloud and Enterprise extensions#712's head (bf7e3212);
  • EXTENSIONS_REF: hieptl/ohe-3160 under global.agentServerEnv, so sandboxes load the skills from the extensions branch.
• Published as Replicated release 1731 on channel hieple/deploy-2026-10-01. The fleet run succeeded: https://github.com/OpenHands/infra/actions/runs/36819742198
• Checked on https://app.shared-4.replicated.all-hands-testing.dev without signing in:
  • /server_info: app_version 1.67.0, sdk_version 1.49.6.
  • /api/automation/health and /api/automation/ready: 200. /api/automation/openapi.json reports version 1.16.0; the chart's default image is 1.15.1, so the pinned build is the one running.
  • The deployed /canvas/ bundle (278 assets) contains the script bundles of the extensions branch (CloudConversations, _deliver_cloud, the $AUTH_HEADER skill text) and the Canvas changes (mcp-native-panel, the cloud-providers-configured query).
• Not checked, because it needs a signed-in session on the instance and I had no credentials for it: GET /api/automation/v1/capabilities (expected to list agentProfiles and kvStore), a preflight with a profile selected, and an end-to-end run of a template.

Helm Chart Checklist

• I have tested the chart upgrade path from the previous version
• I have verified backwards compatibility with existing values.yaml configurations
• I have updated the chart's README.md if there are any breaking changes or new required values

Additional Notes

• On upgrade, KOTS generates the value for existing installs the first time the config is rendered; the automation pods pick it up when they next restart. They carry no secret checksum annotation, the same as for the git sync secret.
• The value signs KV tokens and encrypts stored KV values, so changing it later makes existing automation state unreadable. The config item's help text says so.
• Chart tests: charts/openhands-secrets/tests/automation_kv_secret_test.yaml (no Secret when automations are disabled; the value under the key the automation chart reads) and charts/openhands/charts/automation/tests/kv_secret_env_test.yaml (the env entry with its optional secret reference; no entry when the reference is unset), and charts/openhands/tests/automation_support_bundle_test.yaml (the support bundle collector targets the automation pod and is left out when automations are disabled).
• The value is used as given: I round-tripped a KV value through the automation service's own encryption with a 32-character alphanumeric secret, the form RandomString 32 produces.
• Related: feat: support agent profiles in cloud mode automation#541, feat(automations): run the automation templates on OpenHands Cloud and Enterprise extensions#712, feat(automations): show templates on cloud backends and offer native git integrations OpenHands#17830 (OHE-3160).

Demo Video

demo.mov

[hieptl]

Merge order (OHE-3160)

This PR is one of four that together bring the automation templates to OpenHands Cloud and Enterprise. Please merge them in this order:

1. feat: support agent profiles in cloud mode automation#541: agent profiles in cloud mode, and a setup preflight that accepts a selected profile
2. feat(automation): provision the KV store secret for Replicated installs #1326 (this PR): the automation key-value store secret for self-hosted installs
3. feat(automations): run the automation templates on OpenHands Cloud and Enterprise extensions#712: template scripts and skills that run on Cloud and Enterprise
4. feat(automations): show templates on cloud backends and offer native git integrations OpenHands#17830: Canvas: templates on cloud backends, native git integrations

Required

• feat(automations): run the automation templates on OpenHands Cloud and Enterprise extensions#712 has to be merged and released to npm before feat(automations): show templates on cloud backends and offer native git integrations OpenHands#17830. Canvas currently pins the extensions npm package to Explain invitation behavior in the org membership KOTS help text #712's head commit by git URL for testing; that pin must be replaced with the released version before #17830 merges.

Recommended

• feat: support agent profiles in cloud mode automation#541 and feat(automation): provision the KV store secret for Replicated installs #1326 go first, so the backend already supports what Canvas then exposes. They do not depend on each other or on Explain invitation behavior in the org membership KOTS help text #712, and each can be merged on its own.
• feat(automations): show templates on cloud backends and offer native git integrations OpenHands#17830 goes last. Merged earlier, Canvas would show templates on cloud backends that the backend cannot run yet.

After merging

• automation needs a release, and the automation image tag in the OpenHands-Cloud chart (currently 1.15.1) bumped to it, for refactor(tests): improve test quality for create_github_app #541 to reach Enterprise installs.
• Canvas needs a release, and the agent-canvas image tag in the chart (currently 1.24.0) bumped to it.

[jpshackelford]

Thanks @hieptl — this matches what I was independently writing up in #1327 (now closed, ~8h apart). Core wiring looks right and the fleet-deploy evidence is more than I had. A few carryovers from #1327:

Requested changes

1. Bump chart versions. Subchart values changed; without the bump, Replicated release packaging, helm dep update, and consumers pinning by version won't register this as a new release. Neither Chart.yaml is in the diff, so leaving these here rather than inline:

• charts/openhands-secrets/Chart.yaml: 0.1.24 → 0.1.25
• charts/openhands/Chart.yaml: 0.74.0 → 0.74.1

2. Add AUTOMATION_KV_SECRET to SECRET_PRESENCE in charts/openhands/templates/troubleshoot/support-bundle.yaml (also not in the diff, so dropping here):

-                "AUTOMATION_WEBHOOK_SECRET", "AUTOMATIONS_SERVICE_KEY",
+                "AUTOMATION_WEBHOOK_SECRET", "AUTOMATION_KV_SECRET", "AUTOMATIONS_SERVICE_KEY",

Supportability: a Replicated support bundle should answer "is KV configured on this install?" in one line. Without this, the exact bug this PR fixes (KV silently off because the secret was never wired) is invisible in any future bundle, and we'd be asking customers to re-run bundles after we notice KV endpoints 503ing. One-char diff; big diagnostic payoff.

Nit

One inline comment on replicated/config.yaml suggesting a rotation warning in the help text, modelled on litellm_salt_key's precedent.

Sidebar: pre-existing secretsChecksum arity bug

Unrelated to this PR but worth linking so it isn't lost: the printf format in replicated/openhands.yaml only has 45 %s placeholders, so inserting at position 32 shifts trailing args off the hash. Doesn't affect AUTOMATION_KV_SECRET itself (position 32 is inside the hashed range), but tracked in #1328 — either PR can land first.

---

This review was posted by an AI agent (OpenHands) on behalf of @jpshackelford.

[hieptl]

Thanks for the careful review @jpshackelford, and for carrying the points over from #1327. Going through each one:

1. Chart version bumps: left as they are

In this repo the chart version: is owned by release-please, so I have not bumped either Chart.yaml by hand:

• The header of .github/workflows/release.yml says: "release-please owns the chart version: in Chart.yaml; contributors no longer bump it by hand".
• Both charts are release-please lines: .release-please-manifest.openhands.json is at 0.74.0 and .release-please-manifest.openhands-secrets.json at 0.1.24. A manual bump would put Chart.yaml ahead of its manifest and leave the release PR to reconcile it.
• It matches the change this PR is modelled on: the git sync secret (feat: wire the git sync wrapping secret and a /workspace volume #1213) touched the same set of files without bumping either chart.

The concern behind the request is still covered, as far as I can tell: the feat commits here touch both charts/openhands and charts/openhands-secrets, so the next release PRs should pick up both lines, and in the meantime the publish-charts checks on this PR publish preview charts for anyone who wants to install it before it merges.

If you would still prefer a manual bump I am happy to add it. I just did not want to work against the release flow.

2. AUTOMATION_KV_SECRET in the support bundle: done in 7fa9178, in a slightly different place

Fully agree on the supportability gap, and thanks for raising it. One adjustment to where the check lives:

• The openhands-config-snapshot collector execs into the app pod (app=openhands, container openhands).
• AUTOMATION_KV_SECRET is only injected into the automation pods (charts/openhands/charts/automation/templates/_env.yaml). The app pod never receives it, unlike AUTOMATION_WEBHOOK_SECRET and AUTOMATIONS_SERVICE_KEY, which it does.

So added to that SECRET_PRESENCE list it would report empty_or_unset on every install, including healthy ones, which would point support in the wrong direction.

Instead there is now an app/automation-config-snapshot collector next to the other automation collectors. It runs in the automation pod (app=automation) and reports the same set / empty_or_unset shape, never the value:

{
  "secret_presence": {
    "AUTOMATION_KV_SECRET": "set"
  }
}

It is inside the existing automation.enabled block, so it is skipped when automations are off. charts/openhands/tests/automation_support_bundle_test.yaml (b2da1f4) covers both cases.

3. Help text nit: applied

Replied inline; applied verbatim in 21a228d.

Sidebar: secretsChecksum arity (#1328)

Thanks for linking it. Two notes:

• Both PRs edit the same secretsChecksum line in replicated/openhands.yaml, so whichever lands second needs a small rebase. If fix: secretsChecksum printf arity so trailing secrets actually rotate #1328 goes first I will rebase this one and keep placeholders and arguments equal with the extra item (48 each).
• One observation that may be useful for fix: secretsChecksum printf arity so trailing secrets actually rotate #1328's description. I rendered a small printf with more arguments than placeholders through helm template, and Go does not drop the surplus ones: it appends them as %!(EXTRA string=…), and the sha256sum changed when I changed only a surplus argument. So today a trailing secret should still change the checksum, and this PR does not push anything out of the hash. I only checked this with Helm's template engine, not with KOTS itself, and matching the placeholder count is clearly the right fix either way since the current output leans on an error format.

The description is updated to mention the new collector and help text. Let me know if anything else should change.

[jpshackelford]

Thanks again for the great work on this!

Great catch also about properly detecting the secret and adding it to the support bundle. You were right to put the check in the automation pod rather than extending the openhands SECRET_PRESENCE list.

Looking forward to being able to use this myself!

[openhands-release-bot]

🚀 Released in openhands/0.75.0.