Skip to content

Fix stale PR checker timeouts#390

Merged
redreceipt merged 2 commits into
mainfrom
agent/retry-stale-pr-checker
Jul 11, 2026
Merged

Fix stale PR checker timeouts#390
redreceipt merged 2 commits into
mainfrom
agent/retry-stale-pr-checker

Conversation

@redreceipt

@redreceipt redreceipt commented Jul 11, 2026

Copy link
Copy Markdown
Member

Summary

  • Extend the GitHub GraphQL execution deadline from 10 to 30 seconds for the repository-heavy PR query.
  • Retry the stale PR aggregation once before degrading.
  • Send an explicit warning through the manager Slack webhook when GitHub remains unavailable.
  • Keep normal PR and stale-issue reminders on the public Slack webhook.
  • Cover the timeout, retry, manager-warning, and empty-Linear-fallback paths with regression tests.

Root cause

At the July 11 10:00 EDT production run, the differential/crossroads-anywhere GraphQL request exceeded the client's default deadline. post_stale() caught the aggregate GitHubDataError and replaced the PR result with an empty mapping. Once there were no matching stale Linear issues, the function returned before calling Slack, making the scheduled run look like it never happened.

Impact

Transient GitHub latency should now recover on the retry. If both attempts fail, the manager Slack channel receives a concise degraded-status message while the public channel receives only normal PR and stale-issue reminders.

Validation

  • python -m unittest discover -s tests -p 'test_*.py' — 107 tests passed
  • ruff check .
  • ruff format --check .
  • mypy .
  • git diff --check

Proof

Before, the July 11 production worker discarded the PR reminder section at 10:00 EDT:

WARNING Failed to fetch GitHub PRs for differential/crossroads-anywhere
WARNING Skipping GitHub PR review reminders: Failed to fetch complete GitHub PR data

After, a non-posting run of the patched job against the real GitHub and Linear APIs produced the expected PR payload:

elapsed=10.7s posts=1
has_prs=True
has_stale_issues=False
has_degraded_notice=False
payload_chars=3823

@redreceipt redreceipt temporarily deployed to bug-board-agent-retry-s-9oj5bx July 11, 2026 15:24 Inactive
@redreceipt redreceipt temporarily deployed to bug-board-agent-retry-s-9oj5bx July 11, 2026 15:28 Inactive
@redreceipt redreceipt marked this pull request as ready for review July 11, 2026 15:30
@redreceipt redreceipt requested a review from Copilot July 11, 2026 15:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves reliability of the scheduled stale PR checker by increasing the GitHub GraphQL client execution timeout, retrying transient GitHub PR aggregation failures once, and emitting an explicit degraded-status notification to the manager Slack webhook when GitHub PR data is still unavailable.

Changes:

  • Increase GitHub GraphQL client execute_timeout to 30s to accommodate repo-heavy queries.
  • Add a single retry for PR-review reminders fetch and post a manager-only degraded-status notice when GitHub PR data remains incomplete.
  • Add regression tests for timeout/slow-query configuration, retry behavior, manager-warning behavior, and empty-Linear-fallback behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
jobs.py Adds a one-time retry wrapper for PR fetches and posts a manager-only degraded-status message when GitHub PR data remains unavailable.
github.py Introduces a 30s GraphQL execution timeout constant and passes it to the GraphQL client.
tests/test_jobs.py Adds/updates tests covering retry behavior and manager degraded-status notifications.
tests/test_gql_client_requests.py Adds a test asserting the GitHub GraphQL client is constructed with the configured execution timeout.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread jobs.py
Comment on lines +624 to +630
if github_prs_unavailable:
post_to_manager_slack(
"*Stale PR Check Unavailable*\n\n"
"GitHub did not return complete PR data after retrying, "
"so review reminders were skipped.\n\n"
f"<{os.getenv('APP_URL')}|View Bug Board>"
)
@redreceipt redreceipt merged commit ff2cdd6 into main Jul 11, 2026
6 checks passed
@redreceipt redreceipt deleted the agent/retry-stale-pr-checker branch July 11, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants