feat(sandbox-env): optional KEDA autoscaling for tenant warm pools - #7007
Closed
pedrofrxncx wants to merge 1 commit into
Closed
feat(sandbox-env): optional KEDA autoscaling for tenant warm pools#7007pedrofrxncx wants to merge 1 commit into
pedrofrxncx wants to merge 1 commit into
Conversation
Tenant warm pools were fixed-size only, while the generic warmPool could be
autoscaled. A tenant pool holds a ready pod only while `size > claim rate x
warm time`, so the right size moves with the org's traffic and a static number
is either idle spend or a guaranteed cold start.
Each `tenantPools` entry now takes an optional `autoscaling` block rendering a
KEDA ScaledObject against the pool's native scale subresource. Off by default,
so the chart still installs on clusters without KEDA.
No default trigger ships, deliberately: the operator's
`agent_sandbox_claim_creation_total` is labelled by `sandbox_template`, and
every tenant pool binds the ONE template this chart renders — copying the
generic pool's query would scale each tenant pool on the whole environment's
claim rate. Studio's per-pool `studio_sandbox_pool_pods{pool,state}` gauge is
the signal that distinguishes them, and the docs point at `state="bound"`
rather than `state="ready"` (which counts "config posted", not "dev server
up"). `triggers` is required and validated at template time: a ScaledObject
with none sits at minReplicaCount forever.
`size` stays the seeded / Argo-reapplied replica count, so enabling this also
needs the pool covered by the Argo app's ignoreDifferences on SandboxWarmPool
/spec/replicas — noted in values.yaml, the README and the example.
Chart 0.16.37 -> 0.17.0. CI renders the example and asserts the
triggers-required validation fails the render.
Collaborator
Author
|
Closing as stale: this PR sat past the bot's 48h merge window, main has moved on, and its CI results no longer reflect the current base. This is a housekeeping close, not a rejection of the change — if the underlying problem still exists, the bot will find it again and open a fresh, rebased PR. [studio-bot:stale-close] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tenant warm pools (
tenantPools[]) were fixed-size only, while the genericwarmPoolcould already be autoscaled. Each entry now takes an optionalautoscalingblock that renders a KEDAScaledObjectagainst the pool's native scale subresource.Off by default — the chart still installs on clusters without KEDA, and a tenant pool stays a fixed
sizeuntil someone opts in.Why a pool can't be a constant
A warm pool holds a ready pod only while
size > claim rate × warm time. Measured in prod on 2026-08-07: a claim every ~40s (~1.5/min) against ~2.5min from pod creation to a serving dev server, so a fixed size 2 supplied 0.8 warm pods/min against 1.5/min demanded — the pool was empty essentially 100% of the time. Neither term is constant, so a static number is wrong in both directions as traffic moves.No default trigger, deliberately
The obvious move — copy the generic pool's
agent_sandbox_claim_creation_totalquery — is wrong here. That metric is labelled bysandbox_template, and every tenant pool binds the one template this chart renders, so it would scale each tenant pool on the whole environment's claim rate.Studio's own per-pool gauge
studio_sandbox_pool_pods{pool,state}is the signal that distinguishes them. The docs point atstate="bound"(pods actually handed to a user) rather thanstate="ready", which counts "config posted", not "dev server up" — it reads full while the pool is useless, which is exactly what hid the 2026-08-07 sizing bug.So
triggersis BYO and validated at template time: aScaledObjectwith none would sit atminReplicaCountforever, which is a dormant deploy rather than a failure.maxReplicaCount >= minReplicaCountis validated too.Operational caveat
sizeremains the seeded / Argo-reapplied replica count. Enabling this also requires the pool to be covered by the Argo app'signoreDifferencesonSandboxWarmPool /spec/replicas, or selfHeal reverts the scaler on every reconcile. Called out invalues.yaml, the README, and the example.Testing
helm lint+ render clean.examples/values-tenant-pool-keda.yaml, rendered in CI.studio-sandbox-prodvalues: additiveScaledObject, the housekeeper legacy-idle-path drop, and label bumps. BothSandboxTemplatespecs are byte-identical, so this does not roll the warm pool.Chart
0.16.37→0.17.0.Follow-up
Consumer side: decocms/deco-apps-cd#TBD enables it for
tenant-electrolux-prod(needs this chart released first).Summary by cubic
Tenant warm pools (
tenantPools[]) were fixed-size only, which can't track a claim rate that varies over time. Each entry now takes an optionalautoscalingblock that renders a KEDAScaledObjectagainst the pool's native scale subresource; it's off by default, so the chart still installs on clusters without KEDA.Autoscaling
sandbox_template, which every tenant pool shares, so it can't tell pools apart.studio_sandbox_pool_pods{pool,state="bound"}gauge —state="ready"counts "config posted", not a serving dev server.triggersis required and validated at template time, since aScaledObjectwith none sits atminReplicaCountforever.maxReplicaCount >= minReplicaCountis validated too.Migration
sizeremains the seeded / Argo-reapplied replica count; enabling autoscaling requires adding the pool to the Argo app'signoreDifferencesonSandboxWarmPool /spec/replicas, or selfHeal reverts the scaler.Written for commit 0cb92ec. Summary will update on new commits.