Extend RHACS Central route wait to ride out DNS negative caching - #163
Merged
Conversation
The 'Wait for StackRox Central to return 200 status' task retried for about 5 minutes. A transient NXDOMAIN for the *.apps route hostname gets negative-cached along the resolver chain (dyn.redhatworkshops.io publishes SOA minimum 86400), so a momentary blip at first query could outlast the whole retry budget and fail the provision. Seen on prod1 jobs 40181 and 37175 (agd-v2.rhacs-demo-cnv) while the wildcard record verifiably existed. Extend the wait to 30 minutes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rhjcd
requested review from
agonzalezrh,
fridim,
rut31337,
stencell and
wkulhanek
as code owners
July 15, 2026 20:09
ahsen-shah
approved these changes
Jul 15, 2026
Contributor
Author
|
Follow-ups filed: GPTEINFRA-17341 (SOA minimum on dyn.redhatworkshops.io), GPTEINFRA-17342 (CoreDNS per-zone forwarding on prod0/prod1), campaign tracker RHDPOPS-23814. Companion defensive fix for the same item's other flake: rhpds/agnosticd-v2#185. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ocp4_workload_rhacs task 'Wait for StackRox Central to return 200 status' retried for about 5 minutes (60 x 5s). A transient NXDOMAIN for the *.apps route hostname gets negative-cached along the resolver chain — dyn.redhatworkshops.io publishes SOA minimum 86400 (24h) — so a momentary resolution blip at first query can outlast the entire retry budget and fail the provision, even though the wildcard record exists on both authoritative servers.
Observed on prod1 (apps.ocpv-infra02.wdc07) jobs 40181 (2026-07-15) and 37175 (2026-07-14), both agd-v2.rhacs-demo-cnv: 60 consecutive attempts over 5m23s all failed with 'Name or service not known' for central-stackrox.apps.cluster-.dyn.redhatworkshops.io while api.cluster- resolved fine in the same job, and 14 live pool clusters resolved correctly on both ddns01 and ddns02 during investigation.
This change extends the wait to 30 minutes (90 x 20s), which outlasts typical resolver negative-cache windows (OpenShift CoreDNS denial cache caps at 900s). No happy-path cost: the task exits on the first 200.
A companion infra request is being filed to lower the SOA minimum on dyn.redhatworkshops.io, which addresses the root cause for all items using that zone.
Deployment note: dev stage of agd-v2.rhacs-demo-cnv tracks main for collections; prod pins tag rhacs-demo-cnv-1.0.1 and needs a 1.0.2 tag plus an agnosticv prod.yaml bump to pick this up.