Skip to content

feat(sandbox-env): optional KEDA autoscaling for tenant warm pools - #7007

Closed
pedrofrxncx wants to merge 1 commit into
mainfrom
feat/tenant-pool-keda
Closed

feat(sandbox-env): optional KEDA autoscaling for tenant warm pools#7007
pedrofrxncx wants to merge 1 commit into
mainfrom
feat/tenant-pool-keda

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Tenant warm pools (tenantPools[]) were fixed-size only, while the generic warmPool could already be autoscaled. Each entry now takes an optional autoscaling block that renders a KEDA ScaledObject against the pool's native scale subresource.

Off by default — the chart still installs on clusters without KEDA, and a tenant pool stays a fixed size until someone opts in.

Why a pool can't be a constant

A warm pool holds a ready pod only while size > claim rate × warm time. Measured in prod on 2026-08-07: a claim every ~40s (~1.5/min) against ~2.5min from pod creation to a serving dev server, so a fixed size 2 supplied 0.8 warm pods/min against 1.5/min demanded — the pool was empty essentially 100% of the time. Neither term is constant, so a static number is wrong in both directions as traffic moves.

No default trigger, deliberately

The obvious move — copy the generic pool's agent_sandbox_claim_creation_total query — is wrong here. That metric is labelled by sandbox_template, and every tenant pool binds the one template this chart renders, so it would scale each tenant pool on the whole environment's claim rate.

Studio's own per-pool gauge studio_sandbox_pool_pods{pool,state} is the signal that distinguishes them. The docs point at state="bound" (pods actually handed to a user) rather than state="ready", which counts "config posted", not "dev server up" — it reads full while the pool is useless, which is exactly what hid the 2026-08-07 sizing bug.

So triggers is BYO and validated at template time: a ScaledObject with none would sit at minReplicaCount forever, which is a dormant deploy rather than a failure. maxReplicaCount >= minReplicaCount is validated too.

Operational caveat

size remains the seeded / Argo-reapplied replica count. Enabling this also requires the pool to be covered by the Argo app's ignoreDifferences on SandboxWarmPool /spec/replicas, or selfHeal reverts the scaler on every reconcile. Called out in values.yaml, the README, and the example.

Testing

  • helm lint + render clean.
  • New examples/values-tenant-pool-keda.yaml, rendered in CI.
  • New CI step asserts the triggers-required validation fails the render (verified locally, along with the min/max check).
  • Rendered the real 0.16.2 → 0.17.0 diff against production studio-sandbox-prod values: additive ScaledObject, the housekeeper legacy-idle-path drop, and label bumps. Both SandboxTemplate specs are byte-identical, so this does not roll the warm pool.

Chart 0.16.370.17.0.

Follow-up

Consumer side: decocms/deco-apps-cd#TBD enables it for tenant-electrolux-prod (needs this chart released first).


Summary by cubic

Tenant warm pools (tenantPools[]) were fixed-size only, which can't track a claim rate that varies over time. Each entry now takes an optional autoscaling block that renders a KEDA ScaledObject against the pool's native scale subresource; it's off by default, so the chart still installs on clusters without KEDA.

Autoscaling

  • No default trigger ships: the operator's claim metric is labelled by sandbox_template, which every tenant pool shares, so it can't tell pools apart.
  • Scale on Studio's per-pool studio_sandbox_pool_pods{pool,state="bound"} gauge — state="ready" counts "config posted", not a serving dev server.
  • triggers is required and validated at template time, since a ScaledObject with none sits at minReplicaCount forever. maxReplicaCount >= minReplicaCount is validated too.

Migration

  • size remains the seeded / Argo-reapplied replica count; enabling autoscaling requires adding the pool to the Argo app's ignoreDifferences on SandboxWarmPool /spec/replicas, or selfHeal reverts the scaler.
  • Chart version bumps to 0.17.0.

Written for commit 0cb92ec. Summary will update on new commits.

Review in cubic

Tenant warm pools were fixed-size only, while the generic warmPool could be
autoscaled. A tenant pool holds a ready pod only while `size > claim rate x
warm time`, so the right size moves with the org's traffic and a static number
is either idle spend or a guaranteed cold start.

Each `tenantPools` entry now takes an optional `autoscaling` block rendering a
KEDA ScaledObject against the pool's native scale subresource. Off by default,
so the chart still installs on clusters without KEDA.

No default trigger ships, deliberately: the operator's
`agent_sandbox_claim_creation_total` is labelled by `sandbox_template`, and
every tenant pool binds the ONE template this chart renders — copying the
generic pool's query would scale each tenant pool on the whole environment's
claim rate. Studio's per-pool `studio_sandbox_pool_pods{pool,state}` gauge is
the signal that distinguishes them, and the docs point at `state="bound"`
rather than `state="ready"` (which counts "config posted", not "dev server
up"). `triggers` is required and validated at template time: a ScaledObject
with none sits at minReplicaCount forever.

`size` stays the seeded / Argo-reapplied replica count, so enabling this also
needs the pool covered by the Argo app's ignoreDifferences on SandboxWarmPool
/spec/replicas — noted in values.yaml, the README and the example.

Chart 0.16.37 -> 0.17.0. CI renders the example and asserts the
triggers-required validation fails the render.
@pedrofrxncx

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR sat past the bot's 48h merge window, main has moved on, and its CI results no longer reflect the current base. This is a housekeeping close, not a rejection of the change — if the underlying problem still exists, the bot will find it again and open a fresh, rebased PR.

[studio-bot:stale-close]

@pedrofrxncx pedrofrxncx closed this Sep 8, 2026
@pedrofrxncx
pedrofrxncx deleted the feat/tenant-pool-keda branch September 8, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant