Skip to content

Surface why a run is stuck in PENDING (scheduling observability in run view) #2565

Description

@Mbeaulne

Summary

Runs sometimes stuck in PENDING forever. UI gives no signal why. Root cause only found by digging into the k8s clusters (pod scheduling). Want that signal surfaced in the run view / logs.

Example

https://oasis.shopify.io/runs-v2/019f87586deb1c5ee708 — stuck in PENDING.

Problem

  • PENDING node shows no reason, no events, no pod status.
  • Debugging needs manual kubectl into GKE/Nebius to find the scheduling failure (Unschedulable, quota, ImagePullBackOff, SA/IAM, etc.).
  • Slow, and needs cluster access most users don't have.

Desired

  • Run view surfaces scheduling/pod state for a PENDING node: launch status, k8s events (FailedScheduling, quota exceeded, image pull errors...), the reason it hasn't launched.
  • A "why pending" panel or event feed. Even a raw dump of launcher/pod events in the UI logs would help.

Acceptance

  • A PENDING node shows a reason / event feed for why it hasn't launched.
  • Common stuck causes (unschedulable, quota, image pull, SA/IAM) visible without kubectl.

Related

Reported by Matt Beaulne (@matt.beaulne) via River.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions