From be6401ef6daf313a725c1c170b8a149a2ccaaedc Mon Sep 17 00:00:00 2001 From: Sohan Kunkerkar Date: Tue, 25 Aug 2026 12:19:52 -0400 Subject: [PATCH 1/3] enable-qe-catalogsource-disconnected: honor OO_INDEX on non-C2S clusters OO_INDEX is only read inside mirror_optional_images(), which the main flow calls under `if [ $mirror -eq 1 ]` -- C2S/SC2S clusters only. On every other disconnected cluster the function never runs, so the override was read nowhere and create_catalog_sources used a mirror_index_image hardcoded to quay.io/openshift-qe-optional-operators/aosqe-index:v.. That leaves a job with no way to pin an index when the kube-version-derived tag is unavailable upstream. It is currently blocking the kueue-operator disconnected periodics on all three branches: every run that installs successfully then fails pulling aosqe-index:v1.35 with `manifest unknown`, and setting OO_INDEX has no effect. Set mirror_index_image in both branches of the existing conditional, and in the override branch route the image through the pull-through proxy fronting its registry -- MIRROR_PROXY_REGISTRY_QUAY for quay.io, MIRROR_PROXY_REGISTRY otherwise. Those match the ICSP mappings create_settled_icsp already emits. The default path is unchanged, so jobs that do not set OO_INDEX cannot regress. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Hn12gqKYyvoriFifL12kX --- ...enable-qe-catalogsource-disconnected-commands.sh | 13 +++++++++++-- .../enable-qe-catalogsource-disconnected-ref.yaml | 6 ++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh index 0152b7a396e92..b5eb486deebeb 100644 --- a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh +++ b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh @@ -635,11 +635,20 @@ kube_minor=$(oc version -o json |jq -r '.serverVersion.minor' | sed 's/+$//') if [[ $OO_INDEX == "" ]];then origin_index_image="quay.io/openshift-qe-optional-operators/aosqe-index:v${kube_major}.${kube_minor}" + mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/openshift-qe-optional-operators/aosqe-index:v${kube_major}.${kube_minor}" else origin_index_image="$OO_INDEX" + # Route the override through the pull-through proxy that fronts its registry, so + # OO_INDEX also takes effect on non-C2S clusters. There mirror_optional_images is + # never called, and the catalogsource pulls mirror_index_image directly; without + # this the override silently had no effect and the job stayed pinned to the + # kube-version-derived aosqe-index tag. + case "$OO_INDEX" in + quay.io/*) mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/${OO_INDEX#quay.io/}" ;; + */*/*) mirror_index_image="${MIRROR_PROXY_REGISTRY}/${OO_INDEX#*/}" ;; + *) mirror_index_image="$OO_INDEX" ;; + esac fi - -mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/openshift-qe-optional-operators/aosqe-index:v${kube_major}.${kube_minor}" echo "origin_index_image: ${origin_index_image}" echo "mirror_index_image: ${mirror_index_image}" diff --git a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml index 822220c2a8c1e..416e1c35abb4b 100644 --- a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml +++ b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml @@ -16,6 +16,12 @@ ref: default: "" documentation: |- The index image URL. For example: quay.io/openshift-qe-optional-operators/ocp4-index:v4.17 + When unset, the index defaults to + quay.io/openshift-qe-optional-operators/aosqe-index:v., + derived from the cluster's Kubernetes version. Set this to pin a specific index, + for example when the version-derived tag is unavailable upstream. The catalogsource + pulls the override through the pull-through proxy that fronts its registry + (MIRROR_PROXY_REGISTRY_QUAY for quay.io images, MIRROR_PROXY_REGISTRY otherwise). - name: CATALOGSOURCE_NAME default: qe-app-registry documentation: |- From 2c54ce5174fa6a4f722b4b3b08b1bc6de103137b Mon Sep 17 00:00:00 2001 From: Sohan Kunkerkar Date: Tue, 25 Aug 2026 12:27:21 -0400 Subject: [PATCH 2/3] Address review: match fronted registries explicitly, leave C2S untouched Two fixes from review feedback: The `*/*/*` case pattern was not a reliable "has a registry host" test. It missed single-slash references like registry.redhat.io/index:v4.18, which fell through unproxied and would be pulled from the original registry on a disconnected cluster, and it would have stripped the first component of an unqualified foo/bar/baz. Match the registries create_settled_icsp actually fronts instead, and fail with an explicit error on anything else rather than constructing a reference the proxy cannot resolve. Separately, the previous commit overrode mirror_index_image on C2S/SC2S too. There mirror_optional_images re-publishes the index under the canonical targetCatalog/targetTag, so the catalogsource must keep pointing at the canonical value. Scope the override to `mirror -eq 0`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Hn12gqKYyvoriFifL12kX --- ...-qe-catalogsource-disconnected-commands.sh | 29 +++++++++++++------ ...ble-qe-catalogsource-disconnected-ref.yaml | 11 +++++-- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh index b5eb486deebeb..77d473d861fb5 100644 --- a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh +++ b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh @@ -635,18 +635,29 @@ kube_minor=$(oc version -o json |jq -r '.serverVersion.minor' | sed 's/+$//') if [[ $OO_INDEX == "" ]];then origin_index_image="quay.io/openshift-qe-optional-operators/aosqe-index:v${kube_major}.${kube_minor}" - mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/openshift-qe-optional-operators/aosqe-index:v${kube_major}.${kube_minor}" else origin_index_image="$OO_INDEX" - # Route the override through the pull-through proxy that fronts its registry, so - # OO_INDEX also takes effect on non-C2S clusters. There mirror_optional_images is - # never called, and the catalogsource pulls mirror_index_image directly; without - # this the override silently had no effect and the job stayed pinned to the - # kube-version-derived aosqe-index tag. +fi + +# On C2S/SC2S, mirror_optional_images re-publishes origin_index_image under the +# canonical aosqe targetCatalog/targetTag, so the catalogsource keeps pointing there. +mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/openshift-qe-optional-operators/aosqe-index:v${kube_major}.${kube_minor}" + +if [[ $OO_INDEX != "" && $mirror -eq 0 ]]; then + # Nothing re-publishes the override on a non-C2S cluster, so the catalogsource has + # to pull it directly -- which in a disconnected install means going through one of + # the pull-through proxies create_settled_icsp configures. Only those registries are + # fronted, so match them explicitly and fail loudly on anything else rather than + # building a reference the proxy cannot resolve. case "$OO_INDEX" in - quay.io/*) mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/${OO_INDEX#quay.io/}" ;; - */*/*) mirror_index_image="${MIRROR_PROXY_REGISTRY}/${OO_INDEX#*/}" ;; - *) mirror_index_image="$OO_INDEX" ;; + quay.io/openshifttest/*|quay.io/openshift-qe-optional-operators/*|quay.io/olmqe/*) + mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/${OO_INDEX#quay.io/}" ;; + registry.redhat.io/*|brew.registry.redhat.io/*|registry.stage.redhat.io/*|registry-proxy.engineering.redhat.com/*) + mirror_index_image="${MIRROR_PROXY_REGISTRY}/${OO_INDEX#*/}" ;; + *) + echo "ERROR: OO_INDEX=${OO_INDEX} is not in a registry fronted by the mirror proxy." + echo "Supported: quay.io/{openshifttest,openshift-qe-optional-operators,olmqe}, registry.redhat.io, brew.registry.redhat.io, registry.stage.redhat.io, registry-proxy.engineering.redhat.com" + exit 1 ;; esac fi echo "origin_index_image: ${origin_index_image}" diff --git a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml index 416e1c35abb4b..a6f75660b7d54 100644 --- a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml +++ b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml @@ -19,9 +19,14 @@ ref: When unset, the index defaults to quay.io/openshift-qe-optional-operators/aosqe-index:v., derived from the cluster's Kubernetes version. Set this to pin a specific index, - for example when the version-derived tag is unavailable upstream. The catalogsource - pulls the override through the pull-through proxy that fronts its registry - (MIRROR_PROXY_REGISTRY_QUAY for quay.io images, MIRROR_PROXY_REGISTRY otherwise). + for example when the version-derived tag is unavailable upstream. + On C2S/SC2S clusters the index is re-published under the canonical aosqe + catalog, so any registry works. On other disconnected clusters the catalogsource + pulls the override directly and it must live in a registry fronted by the mirror + proxy: quay.io/openshifttest, quay.io/openshift-qe-optional-operators, + quay.io/olmqe, registry.redhat.io, brew.registry.redhat.io, + registry.stage.redhat.io or registry-proxy.engineering.redhat.com. + Anything else fails the step with an explicit error. - name: CATALOGSOURCE_NAME default: qe-app-registry documentation: |- From a818a8d4af04e664d7173c6a87a47fd55c690eda Mon Sep 17 00:00:00 2001 From: Sohan Kunkerkar Date: Tue, 25 Aug 2026 19:02:06 -0400 Subject: [PATCH 3/3] Fall back to the canonical index instead of failing on unproxied OO_INDEX Most OO_INDEX values in the repo are ci-operator pipeline references that resolve to registry.buildXX.ci.openshift.org pullspecs, which the mirror proxy does not front. Failing the step on those would turn a value that is ignored today into a hard error across every disconnected job that sets one. Warn and keep using the version-derived index instead. Behaviour for any value the proxy cannot serve is then identical to before this change. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Hn12gqKYyvoriFifL12kX --- ...enable-qe-catalogsource-disconnected-commands.sh | 13 ++++++++----- .../enable-qe-catalogsource-disconnected-ref.yaml | 4 +++- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh index 77d473d861fb5..f6cd38ba28595 100644 --- a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh +++ b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh @@ -647,17 +647,20 @@ if [[ $OO_INDEX != "" && $mirror -eq 0 ]]; then # Nothing re-publishes the override on a non-C2S cluster, so the catalogsource has # to pull it directly -- which in a disconnected install means going through one of # the pull-through proxies create_settled_icsp configures. Only those registries are - # fronted, so match them explicitly and fail loudly on anything else rather than - # building a reference the proxy cannot resolve. + # fronted, so match them explicitly and fall back to the canonical index on + # anything else rather than building a reference the proxy cannot resolve. case "$OO_INDEX" in quay.io/openshifttest/*|quay.io/openshift-qe-optional-operators/*|quay.io/olmqe/*) mirror_index_image="${MIRROR_PROXY_REGISTRY_QUAY}/${OO_INDEX#quay.io/}" ;; registry.redhat.io/*|brew.registry.redhat.io/*|registry.stage.redhat.io/*|registry-proxy.engineering.redhat.com/*) mirror_index_image="${MIRROR_PROXY_REGISTRY}/${OO_INDEX#*/}" ;; *) - echo "ERROR: OO_INDEX=${OO_INDEX} is not in a registry fronted by the mirror proxy." - echo "Supported: quay.io/{openshifttest,openshift-qe-optional-operators,olmqe}, registry.redhat.io, brew.registry.redhat.io, registry.stage.redhat.io, registry-proxy.engineering.redhat.com" - exit 1 ;; + # Not a registry the proxy fronts (for example a ci-operator pipeline + # pullspec on registry.buildXX.ci.openshift.org). Constructing a proxied + # reference for it would not resolve, so keep the previous behaviour of + # using the canonical index and say so instead of failing the job. + echo "WARNING: OO_INDEX=${OO_INDEX} is not in a registry fronted by the mirror proxy; using ${mirror_index_image}" + echo "WARNING: proxied registries are quay.io/{openshifttest,openshift-qe-optional-operators,olmqe}, registry.redhat.io, brew.registry.redhat.io, registry.stage.redhat.io, registry-proxy.engineering.redhat.com" ;; esac fi echo "origin_index_image: ${origin_index_image}" diff --git a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml index a6f75660b7d54..f282e3f9d3da4 100644 --- a/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml +++ b/ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml @@ -26,7 +26,9 @@ ref: proxy: quay.io/openshifttest, quay.io/openshift-qe-optional-operators, quay.io/olmqe, registry.redhat.io, brew.registry.redhat.io, registry.stage.redhat.io or registry-proxy.engineering.redhat.com. - Anything else fails the step with an explicit error. + Any other value (for example a ci-operator pipeline pullspec) is not proxied; + the step logs a warning and falls back to the version-derived index, which is + the behaviour it had before this override was honored. - name: CATALOGSOURCE_NAME default: qe-app-registry documentation: |-