fix(performance): scope Deployment/StatefulSet cache to operator-managed objects [RHDHBUGS-3569] by rm3l · Pull Request #3333 · redhat-developer/rhdh-operator

rm3l · 2026-08-06T07:05:27Z

Description

The metadata informers for Deployments and StatefulSets cache PartialObjectMetadata for every object of those kinds cluster-wide, regardless of whether they are managed by the operator. Predicates only control whether a reconcile is enqueued — they do not reduce cache size.

On large multi-tenant clusters (e.g. Dev Sandbox with 2000 users), this causes the operator to OOM-crash: heap grows from ~17 MB to ~800 MB, with 63% consumed by ObjectMeta.Unmarshal in the metadata informer LIST path. This happens even with ENABLE_CACHE_LABEL_FILTER=true and TransformStripManagedFields active, since those only cover Secrets/ConfigMaps.

This PR:

Adds unconditional cache.ByObject label selectors for Deployments and StatefulSets filtered on app.kubernetes.io/name=backstage, so only operator-managed objects are cached. This is server-side filtering and is not a breaking change: unlike Secrets/ConfigMaps, users do not create Deployments/StatefulSets for the operator; it creates them itself and always sets this label.
Replaces WatchesMetadata with Owns() for idiomatic ownerReference-based event mapping. The operator already sets controllerutil.SetControllerReference on all Deployments/StatefulSets it creates.

Which issue(s) does this PR fix or relate to

Fixes RHDHBUGS-3569

PR acceptance criteria

Tests
Documentation

How to test changes / Special notes to the reviewer

Perf team should verify memory usage with the 2000-user Dev Sandbox test. Expected result: operator memory drops from ~800 MB to tens of MB.

…naged objects The metadata informers for Deployments and StatefulSets cached PartialObjectMetadata for every object of those kinds cluster-wide, causing OOM on large multi-tenant clusters (~800 MB heap with 2000 users). Add unconditional cache.ByObject label selectors for Deployments and StatefulSets filtered on app.kubernetes.io/name=backstage, so only operator-managed objects are cached. Replace WatchesMetadata with Owns() for idiomatic ownerReference-based event mapping. Assisted-by: Claude <noreply@anthropic.com>

codecov · 2026-08-06T07:09:48Z

Codecov Report

❌ Patch coverage is 0% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.57%. Comparing base (7ab9ff2) to head (c08f089).
⚠️ Report is 25 commits behind head on release-1.10.

Files with missing lines	Patch %	Lines
internal/controller/watchers.go	0.00%	20 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff                @@
##           release-1.10    #3333      +/-   ##
================================================
+ Coverage         61.05%   61.57%   +0.52%     
================================================
  Files                37       37              
  Lines              2108     2090      -18     
================================================
  Hits               1287     1287              
+ Misses              686      668      -18     
  Partials            135      135

Flag	Coverage Δ
nightly	`?`
unittests	`61.57% <0.00%> (+0.52%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/controller/watchers.go	`0.00% <0.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… watch events Add predicate-based V(1) logging to the Owns() watchers for Deployments and StatefulSets, preserving the "enqueuing reconcile on change of" debug messages that were previously emitted by requestByAppLabels. Each log entry now also includes the event type (create, update, delete). Assisted-by: Claude <noreply@anthropic.com>

sonarqubecloud · 2026-08-06T07:19:09Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

rm3l · 2026-08-06T07:20:01Z

/agentic_review

rm3l · 2026-08-06T07:21:51Z

/cherrypick main

openshift-cherrypick-robot · 2026-08-06T07:21:56Z

@rm3l: once the present PR merges, I will cherry-pick it on top of main in a new PR and assign it to you.

Details

In response to this:

/cherrypick main

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

rhdh-qodo-merge · 2026-08-06T07:25:16Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Cache selector too broad 🐞 Bug ➹ Performance

Description

cmd/main.go scopes the Deployment/StatefulSet cache using only app.kubernetes.io/name=backstage,
which is not an operator-specific label; unrelated Backstage workloads with the same conventional
label can still be cached, reducing the effectiveness of this memory-usage fix on multi-tenant
clusters. This does not enqueue reconciles by itself (Owns() maps by ownerReferences), but it can
still inflate the informer cache.

Code

cmd/main.go[R157-160]

+	mgrOpts.Cache.ByObject = map[client.Object]cache.ByObject{
+		&appsv1.Deployment{}: {
+			Label: backstageLabelSelector,
+		},

Relevance

●● Moderate
Would require new operator-specific/instance labeling strategy; no clear precedent they require
narrower cache selectors.
PR-#2070
PR-#3312

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The cache filter is keyed solely on utils.BackstageAppLabel: utils.BackstageAppName (i.e.,
app.kubernetes.io/name=backstage). The operator’s common labeling helper sets only
app.kubernetes.io/name and app.kubernetes.io/instance, and setMetaInfo() applies those labels
broadly, but there is no operator-specific management label to prevent unrelated Backstage workloads
from matching the cache selector. Also, the new watch mechanism uses Owns(), which maps events by
ownerReferences (so cache inclusion is the main remaining scope concern).

cmd/main.go[142-164]
pkg/utils/utils.go[37-50]
pkg/model/runtime.go[204-214]
internal/controller/watchers.go[78-104]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The cache filter for Deployments/StatefulSets matches any object labeled `app.kubernetes.io/name=backstage`, which is a generic application label and does not uniquely identify operator-managed resources.

### Issue Context
- The cache is configured to include Deployments/StatefulSets with `app.kubernetes.io/name=backstage`.
- The operator sets `app.kubernetes.io/name=backstage` on its resources, but does not set a distinct "managed-by"/"operator" label.
- Watches now use `Owns()`, so unrelated objects won't enqueue reconciles, but they may still be cached and consume memory.

### Fix Focus Areas
- cmd/main.go[142-164]
- pkg/utils/utils.go[37-50]
- pkg/model/runtime.go[204-214]

### Suggested fix
1. Introduce and consistently set a dedicated operator-management label on all operator-created resources (e.g. `app.kubernetes.io/managed-by: rhdh-operator` or `rhdh.redhat.com/managed-by: rhdh-operator`).
2. Change the Deployment/StatefulSet `cache.ByObject` selector to match that operator-specific label (optionally alongside `app.kubernetes.io/name=backstage`).
3. Ensure the label is applied centrally (e.g., in `setMetaInfo()` or `utils.SetKubeLabels()`) so it cannot be lost via overlays/patches.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Context used

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 18 rules

✅ Cross-repo context

Not relevant to this PR: redhat-developer/rhdh

Not relevant to this PR: redhat-developer/rhdh-plugins

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}

Previous review results

Review updated until commit c08f089 ⚖️ Balanced

Results up to commit c08f089 ⚖️ Balanced

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Cache selector too broad 🐞 Bug ➹ Performance

Description

cmd/main.go scopes the Deployment/StatefulSet cache using only app.kubernetes.io/name=backstage,
which is not an operator-specific label; unrelated Backstage workloads with the same conventional
label can still be cached, reducing the effectiveness of this memory-usage fix on multi-tenant
clusters. This does not enqueue reconciles by itself (Owns() maps by ownerReferences), but it can
still inflate the informer cache.

Code

cmd/main.go[R157-160]

+	mgrOpts.Cache.ByObject = map[client.Object]cache.ByObject{
+		&appsv1.Deployment{}: {
+			Label: backstageLabelSelector,
+		},

Relevance

●● Moderate
Would require new operator-specific/instance labeling strategy; no clear precedent they require
narrower cache selectors.
PR-#2070
PR-#3312

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The cache filter is keyed solely on utils.BackstageAppLabel: utils.BackstageAppName (i.e.,
app.kubernetes.io/name=backstage). The operator’s common labeling helper sets only
app.kubernetes.io/name and app.kubernetes.io/instance, and setMetaInfo() applies those labels
broadly, but there is no operator-specific management label to prevent unrelated Backstage workloads
from matching the cache selector. Also, the new watch mechanism uses Owns(), which maps events by
ownerReferences (so cache inclusion is the main remaining scope concern).

cmd/main.go[142-164]
pkg/utils/utils.go[37-50]
pkg/model/runtime.go[204-214]
internal/controller/watchers.go[78-104]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The cache filter for Deployments/StatefulSets matches any object labeled `app.kubernetes.io/name=backstage`, which is a generic application label and does not uniquely identify operator-managed resources.

### Issue Context
- The cache is configured to include Deployments/StatefulSets with `app.kubernetes.io/name=backstage`.
- The operator sets `app.kubernetes.io/name=backstage` on its resources, but does not set a distinct "managed-by"/"operator" label.
- Watches now use `Owns()`, so unrelated objects won't enqueue reconciles, but they may still be cached and consume memory.

### Fix Focus Areas
- cmd/main.go[142-164]
- pkg/utils/utils.go[37-50]
- pkg/model/runtime.go[204-214]

### Suggested fix
1. Introduce and consistently set a dedicated operator-management label on all operator-created resources (e.g. `app.kubernetes.io/managed-by: rhdh-operator` or `rhdh.redhat.com/managed-by: rhdh-operator`).
2. Change the Deployment/StatefulSet `cache.ByObject` selector to match that operator-specific label (optionally alongside `app.kubernetes.io/name=backstage`).
3. Ensure the label is applied centrally (e.g., in `setMetaInfo()` or `utils.SetKubeLabels()`) so it cannot be lost via overlays/patches.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

rm3l · 2026-08-06T07:32:16Z

/build-images

github-actions · 2026-08-06T07:35:46Z

✅ PR images built successfully!

Images are available for testing:

Operator: quay.io/rhdh-community/operator:0.10.4-pr-3333-c08f089
Bundle: quay.io/rhdh-community/operator-bundle:0.10.4-pr-3333-c08f089
Catalog: quay.io/rhdh-community/operator-catalog:0.10.4-pr-3333-c08f089

Also available with PR number tag:

quay.io/rhdh-community/operator:0.10.4-pr-3333
quay.io/rhdh-community/operator-bundle:0.10.4-pr-3333
quay.io/rhdh-community/operator-catalog:0.10.4-pr-3333

_{Triggered by @rm3l}

rhdh-qodo-merge · 2026-08-06T07:53:06Z

PR Summary by Qodo

Scope Deployment/StatefulSet cache to operator-managed objects to prevent OOM

🐞 Bug fix ✨ Enhancement 🕐 20-40 Minutes

AI Description

• Add server-side label selectors so Deployment/StatefulSet informers only cache operator-managed
 objects.
• Replace metadata watches with ownerReference-based Owns() watches for idiomatic event-to-CR
 mapping.
• Preserve optional Secret/ConfigMap cache filtering while merging all ByObject cache rules.

Diagram

graph TD
  A["cmd/main.go"] --> B["Manager cache opts"] --> C["ByObject label filters"] --> D["Informer cache"] --> E{{"Kubernetes API"}}
  F["BackstageReconciler"] --> G["Owns() watchers"] --> D
  G --> H["Deployment / StatefulSet"]

High-Level Assessment

The following are alternative approaches to this PR:

1. Keep WatchesMetadata + custom metadata-only cache

➕ Keeps cached objects minimal (PartialObjectMetadata) even for owned workloads
➕ Avoids any future risk of full-object caching for workloads
➖ More complex wiring (custom GVK metadata types + map funcs)
➖ Still requires careful cache scoping to avoid cluster-wide metadata LIST/WATCH pressure

2. Namespace-scoped cache (multi-namespace manager)

➕ Reduces cache footprint without relying on labels
➖ Not viable if the operator must watch many/all namespaces
➖ Operationally complex (must keep namespace list in sync)

3. Disable caching for apps resources and use direct client reads

➕ Eliminates cache growth from apps resources entirely
➖ Harder to build efficient event-driven reconciliation
➖ More apiserver calls; can hurt performance and rate limits

Recommendation: The PR’s approach (server-side cache scoping via cache.ByObject selectors + Owns() mapping) is the best tradeoff: it directly fixes the root cause (cluster-wide apps metadata caching), keeps the controller idiomatic (ownerReferences), and remains low-maintenance. The main assumption to validate is that all operator-created Deployments/StatefulSets consistently carry the app.kubernetes.io/name=backstage label and have proper controller ownerReferences.

Files changed (2) +57 / -66

Bug fix (2) +57 / -66

main.goAdd ByObject label-scoped cache for Deployments/StatefulSets +33/-9

Add ByObject label-scoped cache for Deployments/StatefulSets

• Configures controller-runtime cache to only store Deployments/StatefulSets labeled 'app.kubernetes.io/name=backstage', preventing cluster-wide metadata caching. Refactors the existing Secret/ConfigMap label filter to extend the same 'ByObject' map rather than overwriting it.

cmd/main.go

watchers.goSwitch workload watches to ownerReference-based Owns() with event logging +24/-57

Switch workload watches to ownerReference-based Owns() with event logging

• Replaces 'WatchesMetadata' + label-based request mapping with 'Owns(&Deployment{})' and 'Owns(&StatefulSet{})' so events map to Backstage CRs via ownerReferences. Adds V(1) predicate logging for create/update/delete events and removes the obsolete 'requestByAppLabels' helper.

internal/controller/watchers.go

rhdh-qodo-merge · 2026-08-06T07:56:32Z

Code review by qodo was updated up to the latest commit c08f089

openshift-ci Bot added the do-not-merge/work-in-progress label Aug 6, 2026

rm3l changed the title ~~fix(RHDHBUGS-3569): scope Deployment/StatefulSet cache to operator-managed objects~~ fix(performance): scope Deployment/StatefulSet cache to operator-managed objects [RHDHBUGS-3569] Aug 6, 2026

rm3l force-pushed the fix/release-1.10/RHDHBUGS-3569--operator-oom-deployment-statefulset-metadata-informers-cache-all-cluster-objects branch from 02399cd to c08f089 Compare August 6, 2026 07:18

rm3l marked this pull request as ready for review August 6, 2026 07:51

rm3l requested a review from a team as a code owner August 6, 2026 07:51

openshift-ci Bot removed the do-not-merge/work-in-progress label Aug 6, 2026

openshift-ci Bot requested review from OpinionatedHeron and subhashkhileri August 6, 2026 07:51

rhdh-qodo-merge Bot added enhancement New feature or request Bug fix labels Aug 6, 2026

Navigation Menu

Uh oh!

fix(performance): scope Deployment/StatefulSet cache to operator-managed objects [RHDHBUGS-3569] - #3333

fix(performance): scope Deployment/StatefulSet cache to operator-managed objects [RHDHBUGS-3569]#3333
rm3l wants to merge 2 commits into
redhat-developer:release-1.10from
rm3l:fix/release-1.10/RHDHBUGS-3569--operator-oom-deployment-statefulset-metadata-informers-cache-all-cluster-objects

rm3l commented Aug 6, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 6, 2026 •

edited

Loading

Uh oh!

sonarqubecloud Bot commented Aug 6, 2026

Uh oh!

rm3l commented Aug 6, 2026

Uh oh!

rm3l commented Aug 6, 2026

Uh oh!

openshift-cherrypick-robot commented Aug 6, 2026

Uh oh!

rhdh-qodo-merge Bot commented Aug 6, 2026 •

edited

Loading

Uh oh!

rm3l commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

rhdh-qodo-merge Bot commented Aug 6, 2026

Uh oh!

rhdh-qodo-merge Bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rm3l commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Which issue(s) does this PR fix or relate to

PR acceptance criteria

How to test changes / Special notes to the reviewer

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonarqubecloud Bot commented Aug 6, 2026

Quality Gate passed

Uh oh!

rm3l commented Aug 6, 2026

Uh oh!

rm3l commented Aug 6, 2026

Uh oh!

openshift-cherrypick-robot commented Aug 6, 2026

Uh oh!

rhdh-qodo-merge Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Previous review results

Uh oh!

rm3l commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

rhdh-qodo-merge Bot commented Aug 6, 2026

PR Summary by Qodo

Uh oh!

rhdh-qodo-merge Bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rm3l commented Aug 6, 2026 •

edited

Loading

codecov Bot commented Aug 6, 2026 •

edited

Loading

rhdh-qodo-merge Bot commented Aug 6, 2026 •

edited

Loading