Fix stale PR checker timeouts#390
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Improves reliability of the scheduled stale PR checker by increasing the GitHub GraphQL client execution timeout, retrying transient GitHub PR aggregation failures once, and emitting an explicit degraded-status notification to the manager Slack webhook when GitHub PR data is still unavailable.
Changes:
- Increase GitHub GraphQL client
execute_timeoutto 30s to accommodate repo-heavy queries. - Add a single retry for PR-review reminders fetch and post a manager-only degraded-status notice when GitHub PR data remains incomplete.
- Add regression tests for timeout/slow-query configuration, retry behavior, manager-warning behavior, and empty-Linear-fallback behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
jobs.py |
Adds a one-time retry wrapper for PR fetches and posts a manager-only degraded-status message when GitHub PR data remains unavailable. |
github.py |
Introduces a 30s GraphQL execution timeout constant and passes it to the GraphQL client. |
tests/test_jobs.py |
Adds/updates tests covering retry behavior and manager degraded-status notifications. |
tests/test_gql_client_requests.py |
Adds a test asserting the GitHub GraphQL client is constructed with the configured execution timeout. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+624
to
+630
| if github_prs_unavailable: | ||
| post_to_manager_slack( | ||
| "*Stale PR Check Unavailable*\n\n" | ||
| "GitHub did not return complete PR data after retrying, " | ||
| "so review reminders were skipped.\n\n" | ||
| f"<{os.getenv('APP_URL')}|View Bug Board>" | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause
At the July 11 10:00 EDT production run, the
differential/crossroads-anywhereGraphQL request exceeded the client's default deadline.post_stale()caught the aggregateGitHubDataErrorand replaced the PR result with an empty mapping. Once there were no matching stale Linear issues, the function returned before calling Slack, making the scheduled run look like it never happened.Impact
Transient GitHub latency should now recover on the retry. If both attempts fail, the manager Slack channel receives a concise degraded-status message while the public channel receives only normal PR and stale-issue reminders.
Validation
python -m unittest discover -s tests -p 'test_*.py'— 107 tests passedruff check .ruff format --check .mypy .git diff --checkProof
Before, the July 11 production worker discarded the PR reminder section at 10:00 EDT:
After, a non-posting run of the patched job against the real GitHub and Linear APIs produced the expected PR payload: