Skip to content

Stale rescheduler re-publishes in-flight matches, multiplying load #174

Description

@razam-sherwani

Dependency: READY — start now. No blockers, independent of #172/#173. Pair with the per-match worker watchdog in cs3600_2026#35 — the watchdog stops a match becoming stale, this stops the stale one being cloned.

Stale rescheduler re-publishes in-flight work

GameMatchRepository.findStaleWaitingMatches selects status IN ('waiting','in_progress') AND scheduledAt <= :cutoff, and StaleGameMatchRescheduler (@Scheduled(fixedRate = 60000), threshold 120 min) re-publishes without cancelling the original. A slow or wedged match is therefore re-enqueued every cycle past the threshold, and times_scheduled climbs — the rescheduler multiplies load exactly under the stress it fires in.

Fix

  • Exclude in_progress from the stale sweep, or verify the worker is actually dead before requeuing.
  • Cancel/dedupe the prior copy before re-publishing so a match cannot exist in the queue twice.
  • Add a bound on times_scheduled after which a match is failed rather than requeued.

Pairs with the per-match worker watchdog in cs3600_2026#35 (kill a wedged play_game so it never becomes stale in the first place). Part of the wait-time track (#172, #173).

Note on measuring the fix: the rescheduler overwrites scheduled_at, so historical wait-time queries (started_at − scheduled_at) must filter times_scheduled = 1 for clean numbers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    High PrioritybugSomething isn't workingreadyNo open blockers - can be started now

    Type

    No type

    Fields

    Stage

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions