Skip to content

WIP: OCPBUGS-105876: Wire availableInertia into the olm StatusSyncer - #233

Draft
fgiudici wants to merge 2 commits into
openshift:mainfrom
fgiudici:ocpbugs-105876-available-inertia
Draft

WIP: OCPBUGS-105876: Wire availableInertia into the olm StatusSyncer#233
fgiudici wants to merge 2 commits into
openshift:mainfrom
fgiudici:ocpbugs-105876-available-inertia

Conversation

@fgiudici

@fgiudici fgiudici commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

clusteroperator/olm may intermittently flips Available=False for a sub-second during OCP upgrades tripping the CVO invariant monitor test.
See OCPBUGS-105876 for the full scenario analysis and reasoning.

The idea is to add a new availableInertia grace period on library-go's StatusSyncer (see openshift/library-go#2426).

Not mergeable as-is, currently in draft:
WithAvailableInertia doesn't exist in the currently vendored library-go. The second commit manually overlays the vendored status_controller.go with the unmerged branch's content (content only; go.mod/go.sum/vendor/modules.txt intentionally untouched) so this branch builds and tests locally.
It must be dropped and replaced with a real go get github.com/openshift/library-go@<sha> && go mod vendor once the library-go PR merges upstream.
make verify-deps will fail on this in its current state.

Importing the library from the PR branch requires more work since the newer version triggers cascade dependencies updates. This will be worked on later, the idea here was to start to share the PR to discuss the approach first.

fgiudici and others added 2 commits August 14, 2026 17:34
clusteroperator/olm briefly flips Available=False (sub-second) during
OCP upgrades because catalogd's soft anti-affinity can co-schedule
both replicas on one master; when MCO drains that node both pods get
evicted and AvailableReplicas hits 0 momentarily. StatusSyncer
currently propagates that with no grace period.

Chain WithAvailableInertia (5s) onto the existing
NewClusterOperatorStatusController call, mirroring the WithDegradedInertia
pattern already available on StatusSyncer. Requires the availableInertia
addition to openshift/library-go's StatusSyncer (see next commit).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
WithAvailableInertia (used by the previous commit) does not exist in
the currently vendored library-go (v0.0.0-20251107090138-0de9712313a5)
— it only exists on the local, unpushed "availableInertia" branch of
openshift/library-go. Overlay that branch's content here (content
only; go.mod, go.sum, and vendor/modules.txt are intentionally
untouched) purely so this branch builds and tests locally today.

`go mod tidy`/`go mod vendor` were not used for a real bump: this
repo's current dependency graph has an unrelated, pre-existing break
(k8s.io/kubectl needs k8s.io/api/scheduling/v1alpha1, missing from the
k8s.io/api version tidy wants to pick) that surfaces as soon as a full
re-resolution is attempted.

Not mergeable as-is: `make verify-deps` will fail on this (it reruns
go mod tidy/vendor from scratch and there's no deps.diff registering
this override). This commit exists to keep the branch buildable while
local; it must be dropped/replaced once the library-go PR is pushed,
reviewed, and merged, via a real
`go get github.com/openshift/library-go@<merged-sha> && go mod vendor`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 14, 2026
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 14, 2026
@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Aug 14, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@fgiudici: This pull request references Jira Issue OCPBUGS-105876, which is invalid:

  • expected the bug to target either version "5.1.0." or "openshift-5.1.0.", but it targets "5.0.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

clusteroperator/olm intermittently flips Available=False for a
sub-second during OCP upgrades (reason
CatalogdDeploymentCatalogdControllerManager_Deploying), tripping the
CVO invariant monitor test. Root cause (confirmed from CI artifacts,
OCPBUGS-105876): catalogd's soft anti-affinity can co-schedule both
replicas on one master node; when MCO drains it, both pods get evicted
(the PDB's unhealthyPodEvictionPolicy: AlwaysAllow permits this) and
AvailableReplicas briefly hits 0.

The fix is a new availableInertia grace period on library-go's
StatusSyncer, mirroring the existing 2-minute degradedInertia
(openshift/library-go, branch availableInertia:
https://github.com/fgiudici/openshift.library-go/tree/availableInertia,
not yet opened as a PR upstream). This PR wires the consuming side:

  • cmd/cluster-olm-operator/main.go: chain
    .WithAvailableInertia(status.MustNewInertia(5 * time.Second).Inertia)
    onto the existing NewClusterOperatorStatusController call.

Not mergeable as-is — do not merge:

  • WithAvailableInertia doesn't exist in the currently vendored
    library-go. The second commit manually overlays the vendored
    status_controller.go with the unmerged branch's content (content
    only; go.mod/go.sum/vendor/modules.txt intentionally untouched) so
    this branch builds and tests locally. It must be dropped and
    replaced with a real go get github.com/openshift/library-go@<sha> && go mod vendor once the library-go PR merges upstream.
  • make verify-deps will fail on this in its current state.

A temporary test-only tolerance is already in place in
openshift/origin (openshift/origin#31518) to keep CI green in the
meantime.

Opening as WIP for visibility/review of the wiring approach; will
un-WIP and drop the overlay commit once library-go merges.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7abd3a50-23b1-4599-b67b-e2d784fdf579

📥 Commits

Reviewing files that changed from the base of the PR and between 5123647 and 8bec84c.

⛔ Files ignored due to path filters (1)
  • vendor/github.com/openshift/library-go/pkg/operator/status/status_controller.go is excluded by !**/vendor/**, !vendor/**
📒 Files selected for processing (1)
  • cmd/cluster-olm-operator/main.go

Walkthrough

The cluster operator status controller now applies a 5-second availability inertia instead of its default configuration.

Changes

Cluster operator status

Layer / File(s) Summary
Configure availability inertia
cmd/cluster-olm-operator/main.go
The cluster operator status controller now uses a 5-second availability inertia.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: 🟡 Moderate · up to 8bec8

The change currently relies on an unreleased library-go API and a manual vendor overlay without synchronized dependency metadata, causing dependency verification to fail. It is not merge-ready until the upstream dependency lands and vendoring is regenerated.

Suggested reviewers: tmshort

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes only controller wiring and vendored status code; the diff adds no test files or Ginkgo test-title declarations.
Test Structure And Quality ✅ Passed The PR changes only production code and a vendored status controller; no Ginkgo test files or test blocks changed, so these test-structure requirements are not applicable.
Microshift Test Compatibility ✅ Passed The PR changes only main.go and a vendored status controller; the diff adds no Ginkgo tests or test declarations, so MicroShift test compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The diff changes only operator wiring and vendored status code; it adds no Ginkgo e2e tests or multi-node test assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed The cumulative PR diff only wires status available inertia and adds its library implementation; it introduces no affinity, spread, replica, selector, toleration, or PDB scheduling constraint.
Ote Binary Stdout Contract ✅ Passed The PR diff only adds available-inertia status handling; it introduces no stdout writes or suite setup, and existing fmt.Println calls are unchanged.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR changes only main.go and vendored status_controller.go; the diff adds no Ginkgo tests, IPv4 assumptions, or external connectivity.
No-Weak-Crypto ✅ Passed The PR changes only availability inertia wiring and status handling; added lines contain no MD5, SHA1, DES, RC4, Blowfish, ECB, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed The PR changes Go status logic and a vendored library file only; no added privilege settings or container/Kubernetes manifest changes were found.
No-Sensitive-Data-In-Logs ✅ Passed The diff adds only availability inertia wiring and status handling; it adds no logging, secret formatting, or sensitive-data output.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: wiring availableInertia into the OLM StatusSyncer.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fgiudici
Once this PR has been reviewed and has the lgtm label, please assign joelanford for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tmshort

tmshort commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

/lgtm
to get the CI to run

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 19, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test openshift-e2e-aws
/test openshift-e2e-aws-customnoupgrade
/test openshift-e2e-aws-devpreview
/test openshift-e2e-aws-techpreview

@tmshort

tmshort commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

/test deps
/test images
/test lint

@openshift-ci

openshift-ci Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

@fgiudici: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/deps 8bec84c link true /test deps

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants