chore: throttle deployment bursts that overloaded the Lagoon core

[dan2k3k4]

Incident response — recommend priority merge

Bulk deployment triggers overloaded the Lagoon core with concurrent builds. Two independent flood sources found and staged:

1. Scheduled redeploys (polydock:dispatch-scheduled-redeploys)

• Was: up to 50 per run × every 10 minutes (≤300 builds/hour), with next_redeploy_at IS NULL DESC ordering that front-loads whole cohorts (adopted projects, newly-enabled store apps) the moment they become eligible.
• Now: POLYDOCK_DEPLOY_MAX_PER_RUN default 10, schedule hourly, ordering most-outdated first (never-redeployed = oldest, then oldest last_deployed_at). A backlog drains at ≤10/hour instead of spiking.

2. Pre-warm refresh (EnsureUnallocatedAppInstancesJob)

• Was: ALL stale instances queued for removal in one pass, then the ENTIRE deficit recreated in one loop — each recreation is a full Lagoon create+deploy. A store app with a same-day-created pool of 20 going stale = 20 concurrent rebuilds.
• Now: one refresh batch per hour globally (atomic cache gate), batch capped at POLYDOCK_PREWARM_BATCH (default 10), oldest instances first (the removal query already ordered by created_at). The creation loop is capped per app per pass — since in-progress instances count toward the pool, large deficits fill across passes instead of as one wall of builds.

Both env knobs are tunable without deploys if 10/hour needs adjusting.

Test-pinned behavior (5 new tests)

• Oldest-first trigger order (never → ancient → recent untouched)
• Per-run cap respected (existing test, still green)
• Refresh: one capped oldest-first batch, gate blocks within the hour, next batch after 61 minutes
• Creation deficit fills 2-at-a-time under a batch cap of 2

Testing

• php artisan test — 454 passed
• ./vendor/bin/phpstan analyse — no errors; Pint clean

Greptile Summary

This incident-response PR throttles two sources of Lagoon build bursts: scheduled redeploys are reduced from 50 every 10 minutes to a maximum of 10 per hour with oldest-first ordering, and pre-warm instance refreshes are now bounded by a shared global budget (one batch per hour, capped at POLYDOCK_PREWARM_BATCH) dispatched in oldest-stale-first order across all store apps.

• Scheduled redeploys (DispatchScheduledRedeploysCommand): default cap dropped from 50 → 10, schedule changed from every-10-min → hourly, and ordering changed to last_deployed_at IS NULL DESC, last_deployed_at ASC so adopted/newly-enabled instances drain gradually instead of spiking.
• Pre-warm refresh (EnsureUnallocatedAppInstancesJob): refreshStaleInstances() extracts the refresh logic into a single global-budget method with a Cache::add hourly gate and oldest-app-first ordering; instance creation is also capped per-app-per-pass by the same prewarm_batch config value.
• Five new tests cover oldest-first trigger order, the hourly gate, cross-app budget sharing, and capped creation deficits.

Confidence Score: 5/5

The change is safe to merge — it strictly reduces Lagoon build concurrency through a smaller default cap, an hourly schedule, and a cache-gated global refresh budget, all backed by five targeted tests.

Both throttle mechanisms are well-scoped: the scheduled-redeploy cap and ordering change are straightforward and directly tested; the refreshStaleInstances refactor correctly gates behind Cache::add, shares the budget across apps oldest-first, and the tests exercise the gate, the ordering, and the creation cap. No data is at risk; the worst-case outcome of a misconfiguration is slower pool replenishment, not data loss or additional build bursts.

No files require special attention. The config/polydock.php comment about prewarm_batch scope and the N+1 query pattern in refreshStaleInstances are worth a follow-up but do not block merging.

Important Files Changed

Filename	Overview
app/Jobs/EnsureUnallocatedAppInstancesJob.php	Extracts refresh logic into refreshStaleInstances() with a global hourly cache gate and budget-shared oldest-first ordering; caps creation per-app via prewarm_batch. Minor issues: config comment misleads on per-app vs global scope; N+1 DB queries per refreshable app (exists + min + count).
app/Console/Commands/DispatchScheduledRedeploysCommand.php	Default cap reduced from 50 → 10, ordering changed to oldest-last-deployed-at-first; logic is clean and well-tested.
config/polydock.php	Adds prewarm_batch config key and updates max_per_run default to 10; the inline comment describes the value as 'per store app' which is accurate for creation but misleading for the refresh code path where it is a global cross-app budget.
routes/console.php	Redeploy schedule changed from everyTenMinutes() to hourly() with explanatory comment; onOneServer() added for correctness on multi-server deployments.
tests/Feature/Deployment/ScheduledRedeployCommandTest.php	New test verifies oldest-first ordering (never → ancient → recent untouched); clear and correct.
tests/Feature/Jobs/EnsureUnallocatedThrottlingTest.php	New test file covers the hourly gate, oldest-first batch, cross-app budget sharing, and capped creation deficit; uses $this->travel() to verify the gate resets correctly.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[EnsureUnallocatedAppInstancesJob::handle] --> B[Load AVAILABLE apps needing maintenance]
    B --> C[refreshStaleInstances]
    C --> D{Any refreshable apps?}
    D -- No --> E[return 0]
    D -- Yes --> F{Cache::add prewarm-refresh-hourly-gate}
    F -- Already set within 1 hour --> E
    F -- Added new hour --> G[Sort apps by oldest stale instance created_at]
    G --> H[Iterate apps, share global budget=prewarm_batch]
    H --> I[queueUnallocatedInstancesForRemoval up to budget]
    I --> J[budget -= queued]
    J --> K{budget < 1?}
    K -- Yes --> L[break]
    K -- No --> H
    L --> M[Main loop: foreach app]
    E --> M
    M --> N{Excess unallocated?}
    N -- Yes --> O[queueUnallocatedInstancesForRemoval excess]
    N -- No --> P[app->refresh]
    O --> P
    P --> Q{needed > 0?}
    Q -- No --> R[continue]
    Q -- Yes --> S[needed = min needed prewarm_batch]
    S --> T{app supports_pre_warming?}
    T -- No --> R
    T -- Yes --> U[Create up to prewarm_batch PENDING_PRE_CREATE instances]
    U --> R

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[EnsureUnallocatedAppInstancesJob::handle] --> B[Load AVAILABLE apps needing maintenance]
    B --> C[refreshStaleInstances]
    C --> D{Any refreshable apps?}
    D -- No --> E[return 0]
    D -- Yes --> F{Cache::add prewarm-refresh-hourly-gate}
    F -- Already set within 1 hour --> E
    F -- Added new hour --> G[Sort apps by oldest stale instance created_at]
    G --> H[Iterate apps, share global budget=prewarm_batch]
    H --> I[queueUnallocatedInstancesForRemoval up to budget]
    I --> J[budget -= queued]
    J --> K{budget < 1?}
    K -- Yes --> L[break]
    K -- No --> H
    L --> M[Main loop: foreach app]
    E --> M
    M --> N{Excess unallocated?}
    N -- Yes --> O[queueUnallocatedInstancesForRemoval excess]
    N -- No --> P[app->refresh]
    O --> P
    P --> Q{needed > 0?}
    Q -- No --> R[continue]
    Q -- Yes --> S[needed = min needed prewarm_batch]
    S --> T{app supports_pre_warming?}
    T -- No --> R
    T -- Yes --> U[Create up to prewarm_batch PENDING_PRE_CREATE instances]
    U --> R

Loading

Comments Outside Diff (1)

1. app/Jobs/EnsureUnallocatedAppInstancesJob.php, line 89-100 (link)

   Removal and creation can both fire in the same pass

   When stale instances are queued for PENDING_PRE_REMOVE, they are assigned a user_group_id and excluded from unallocated_instances_count (which only counts user_group_id IS NULL rows). After $app->refresh() on line 89, the deficit that drives $needed is therefore inflated by exactly the number of instances just queued for removal. In a pass where, say, 2 are refreshed and the target pool is 5, $needed can become 3 (capped to 2), triggering 2 simultaneous PENDING_PRE_CREATE instances while the 2 removal pipelines also eventually trigger full Lagoon deploys. The effective per-pass Lagoon pressure is up to 2 × prewarm_batch rather than prewarm_batch. This is bounded, but worth noting as an untested interaction — the existing refresh test does not assert on how many new instances are created in that same pass.

_{Reviews (2): Last reviewed commit: "chore: throttle deployment bursts that o..." | Re-trigger Greptile}

[greptile-apps]

Single global gate starves all apps except the first

Cache::add('prewarm-refresh-hourly-gate', true, 3600) uses one key for every store app. Because PolydockStoreApp::query()...->get() returns rows in primary-key order and no orderBy is applied, the lowest-ID app that has stale instances wins the cache gate on every hour boundary. Every other app with stale instances is blocked for that entire hour. If App A continuously regenerates stale instances (e.g. a pool of 50, batch of 10 → 5 hours of back-to-back wins), App B’s stale instances accumulate unattended for that whole window — the scenario this PR is trying to prevent.

Changing the cache key to 'prewarm-refresh-hourly-gate-'.$app->id gives each store app its own independent hourly gate. The global Lagoon pressure is then num_apps × prewarm_batch per hour; if that ceiling is still too high, the batch size can be tuned, or a separate global counter added on top.