test(ci): operator Fedora PostgreSQL 15 to 18 dump/restore upgrade path

[zdrapela]

Summary

Companion to #5141 (Helm Fedora dump/restore) and #5139 (RHEL Helm POSTGRESQL_UPGRADE=copy) for RHIDP-14594.

This PR exercises the Operator-local Fedora PostgreSQL path on e2e-ocp-operator-nightly:

PG15 → (CI verification) → pg_dumpall / wipe PVC / restore → PG18 → (CI verification)

• Default operator image: RELATED_IMAGE_postgresql=quay.io/fedora/postgresql-15:latest
• Upgrade target: quay.io/fedora/postgresql-18:latest (via oc set env on the operator controller)
• Orchestration: .ci/pipelines/jobs/ocp-operator.sh (handle_ocp_operator) + .ci/pipelines/lib/postgres.sh
• Artifacts: showcase-operator-pg15, showcase-operator-pg18
• Showcase-only (RBAC + runtime skipped) to keep the job focused on DB upgrade evidence

Why dump/restore (and why skip PG16)

Same Fedora constraints as #5141:

• Upstream does not support in-container upgrades on Fedora (sclorg/postgresql-container@b86ea1a)
• quay.io/fedora/postgresql-18 advertises POSTGRESQL_PREV_VERSION=16 but ships postgresql-17 bins
• No quay.io/fedora/postgresql-17 image
• Logical dump/restore can jump 15 → 18 in one maintenance window (see PostgreSQL — Upgrading a PostgreSQL Cluster, Upgrading Data via pg_dumpall)

The Operator already defaults to Fedora PG15 for local DB (RELATED_IMAGE_postgresql), so this is the natural community/operator path.

---

Documentation A — General guidance (operators / docs)

Use this for product/community docs. Do not restate the full PostgreSQL upgrade procedure — follow upstream and only call out RHDH Operator / OpenShift specifics.

Upstream procedure (source of truth)

Major-version upgrades that move data with a logical dump are documented here:

• PostgreSQL 18 docs — Upgrading a PostgreSQL Cluster
  • § Upgrading Data via pg_dumpall (pg_dumpall → new cluster → psql restore)
  • Alternatives (pg_upgrade, logical replication) are not what this Fedora operator-local path uses

Also useful:

• pg_dumpall / SQL dump

When this RHDH guidance applies

• RHDH deployed with the Operator and local PostgreSQL (spec.database.enableLocalDb: true, default)
• Postgres image family is Fedora (quay.io/fedora/postgresql-* via RELATED_IMAGE_postgresql on the operator Deployment / CSV)
• In-place POSTGRESQL_UPGRADE=copy is not reliable (see Summary)

RHDH Operator / OpenShift delta (on top of upstream dump/restore)

Map the upstream dump/restore steps onto the operator-managed local DB as follows:

1. Downtime — Schedule a maintenance window; Backstage must not write while the volume is replaced.
2. Baseline — Record SHOW server_version and the current Postgres container image on backstage-psql-<cr-name>-0.
3. Dump — From the running PG15 pod, run pg_dumpall (logical backup of all DBs/roles). Keep the dump until verification succeeds.
   Note: Upstream prefers dumping with the newer client when both majors can run in parallel. With a single PVC this path dumps from the old pod before wipe.
4. Point the operator at PG18 — Update RELATED_IMAGE_postgresql on the RHDH operator controller Deployment to quay.io/fedora/postgresql-18:latest and wait for the operator rollout to complete. The operator injects this image into the local DB StatefulSet on reconcile.
5. Replace the data volume — Delete the backstage-psql-<cr-name> StatefulSet and its PVC so the next reconcile can initdb a fresh PG18 datadir. Keep the backstage-psql-secret-<cr-name> Secret so RHDH credentials stay valid.
6. Wait for PG18 — Allow the operator to recreate the StatefulSet; confirm the pod image contains postgresql-18 and is Ready. Do not set POSTGRESQL_UPGRADE (this is not a copy upgrade).
7. Restore — Load the dump into the Ready PG18 pod with psql (see upstream). Image init may already create roles/globals; tolerate benign “already exists” errors, then confirm application databases.
8. Collation — After the major jump, run ALTER DATABASE … REFRESH COLLATION VERSION on postgres, template1, and user databases (common post-major cleanup; not a substitute for upstream migration notes).
9. Reconnect RHDH — Restart the backstage-<cr-name> Deployment and wait until the previous pod UID is gone and a new Ready pod exists before trusting the instance.
10. Verify — Confirm Postgres reports 18.x / fedora/postgresql-18, then run your normal RHDH smoke checks.

What not to do on Fedora (operator-local)

• Do not rely on POSTGRESQL_UPGRADE=copy / hardlink for this Fedora jump.
• Do not require a PG16 intermediate hop; dump/restore can go 15→18 directly.
• Do not delete the Postgres password Secret (backstage-psql-secret-<cr-name>) when wiping the PVC unless you also reconfigure RHDH.
• Do not change only the StatefulSet image without updating RELATED_IMAGE_postgresql — the operator will reconcile the image back from the env var.

---

CI flow (this PR)

install RHDH operator
  → set RELATED_IMAGE_postgresql=fedora/postgresql-15
  → deploy Backstage CR (local DB)
  → wait Ready + Backstage healthy
  → [CI only] seed catalog proof + Playwright (showcase-operator-pg15)
  → capture RHDH pod UID
  → oc exec … pg_dumpall -U postgres  →  DUMPFILE
  → set RELATED_IMAGE_postgresql=fedora/postgresql-18 (+ operator rollout)
  → delete backstage-psql StatefulSet + PVC (keep secret)
  → wait Ready (image contains postgresql-18)
  → oc exec -i … psql -U postgres  < DUMPFILE
  → ALTER DATABASE … REFRESH COLLATION VERSION
  → oc rollout restart deployment/backstage-rhdh
  → wait previous RHDH pod UID terminated + new Ready pod
  → [CI only] API + Playwright proof (showcase-operator-pg18)

CI entrypoints

Step	Code
Overall sequence	handle_ocp_operator in .ci/pipelines/jobs/ocp-operator.sh
RELATED_IMAGE switch	set_operator_postgresql_related_image
Wait Ready + image substr	wait_for_postgres_ready
pg_dumpall	postgres_dumpall_to_file
Wipe STS/PVC	postgres_wipe_persistent_volume
Restore	postgres_restore_dumpall_file
Collation	refresh_postgres_collation_versions
Pod certainty	get_rhdh_pod_uid / ensure_rhdh_pod_replaced
CI data proof	seed_pg_upgrade_data_proof / assert_pg_upgrade_data_proof_api
CI UI proof	e2e-tests/playwright/e2e/pg-upgrade-data-proof.spec.ts (PG_UPGRADE_DATA_PROOF=1)

---

Test plan

• /test e2e-ocp-operator-nightly — PG15 and PG18 Playwright phases green
• Artifacts under showcase-operator-pg15 and showcase-operator-pg18
• Version evidence: Fedora 15.x → 18.x (no PG16 hop)
• Previous RHDH pod UID terminated before post-upgrade Playwright
• Catalog persistence proof (API + UI) after dump/restore

Related

• Helm Fedora dump/restore: test(ci): fedora helm PostgreSQL 15 to 18 dump/restore upgrade path #5141
• Helm RHEL copy path: test(ci): exercise Helm PostgreSQL 15-16-18 upgrade path #5139
• Jira: RHIDP-14594
• PostgreSQL: Upgrading a PostgreSQL Cluster (pg_dumpall method)
• sclorg: Upgrading Database, Do not support upgrades on Fedora

Made with Cursor

[zdrapela]

/test e2e-ocp-operator-nightly

[rhdh-qodo-merge]

PR Summary by Qodo

CI: verify operator-local Fedora PostgreSQL 15→18 dump/restore upgrade path

🧪 Tests ✨ Enhancement 🕐 40+ Minutes

AI Description

• Add an OCP operator-nightly CI flow to validate Fedora Postgres 15→18 dump/restore upgrades.
• Persist upgrade evidence via diagnostics/artifacts and a seeded catalog entity checked
 post-upgrade.
• Introduce reusable PostgreSQL/Backstage upgrade helpers for operator and Helm-managed deployments.

Diagram

graph TD
  A(["e2e-ocp-operator-nightly"]) --> B[".ci/pipelines/jobs/ocp-operator.sh"] --> C["Operator controller (RELATED_IMAGE_postgresql)"] --> D[("Operator-local Postgres (PG15 PVC)")] --> E["pg_dumpall.sql"] --> F[("Operator-local Postgres (PG18 PVC)")] --> G["Backstage restart + Playwright verify"]
  subgraph Legend
    direction LR
    _job(["CI job"]) ~~~ _cfg["Config/script"] ~~~ _db[("Database/PVC")
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Use in-place pg_upgrade (where supported)

• ➕ Faster than logical dump/restore for large datasets
• ➕ Preserves roles/permissions with fewer restore-time warnings
• ➖ Not supported/reliable for Fedora container images per upstream constraints referenced in the PR
• ➖ Adds significant operational complexity to CI and diverges from intended Fedora path

2. Validate persistence via DB-level assertions only (skip UI/Playwright)

• ➕ Less flaky than full UI tests and faster runtime
• ➕ Directly verifies critical tables/rows post-restore
• ➖ Weaker end-to-end signal: may miss app-level schema/migration issues
• ➖ Harder to keep assertions stable across Backstage/catalog changes

Recommendation: Keep the PR’s dump/restore approach: it aligns with Fedora image constraints and provides credible end-to-end evidence by seeding a catalog entity, upgrading, and verifying via API/UI. The added helper library is a good direction as it centralizes operator vs Helm detection and reduces future duplication.

Files changed (5) +926 / -54

Tests (2) +189 / -54

ocp-operator.shAdd operator-local PG15→PG18 dump/restore verification phases +142/-54

Add operator-local PG15→PG18 dump/restore verification phases

• Reworks the operator CI job to run showcase-only, explicitly pin RELATED_IMAGE_postgresql to PG15 then switch to PG18, and orchestrate pg_dumpall → PVC wipe → restore. Adds phase-scoped artifact/diagnostics saving, Backstage pod replacement waiting, and optional seed/verify steps to prove catalog persistence across the upgrade.

.ci/pipelines/jobs/ocp-operator.sh

pg-upgrade-data-proof.spec.tsAdd Playwright check for seeded entity post-upgrade +47/-0

Add Playwright check for seeded entity post-upgrade

• Adds a Playwright spec gated by PG_UPGRADE_DATA_PROOF=1 that navigates the Catalog UI and asserts the seeded “PG Upgrade Data Proof” component remains visible after the database upgrade sequence.

e2e-tests/playwright/e2e/pg-upgrade-data-proof.spec.ts

Other (3) +737 / -0

postgres.shIntroduce reusable Postgres upgrade/diagnostics helper library +620/-0

Introduce reusable Postgres upgrade/diagnostics helper library

• Adds a shared bash library for locating the managed Postgres pod/STS across install methods, waiting for readiness with image assertions, dumping/restoring via pg_dumpall, wiping PVCs, and refreshing collation versions. Includes operator-specific support for updating RELATED_IMAGE_postgresql, plus utilities to seed and verify a persistence-proof catalog entity and ensure Backstage pods are replaced after upgrade hops.

.ci/pipelines/lib/postgres.sh

catalog-info.yamlAdd catalog entity used as upgrade persistence proof +13/-0

Add catalog entity used as upgrade persistence proof

• Defines a small Backstage Component entity (pg-upgrade-data-proof) tagged for RHIDP-14594, used as the durable record that should survive the Postgres major upgrade.

.ci/pipelines/resources/pg-upgrade/catalog-info.yaml

data-proof-server.yamlAdd in-cluster static server manifest for catalog-info fixture +104/-0

Add in-cluster static server manifest for catalog-info fixture

• Adds a ConfigMap+Deployment+Service that can serve the catalog-info.yaml from within the cluster using a Python or NodeJS runtime chosen at deploy time. This avoids reliance on external GitHub/raw URLs during CI while still allowing Backstage to register a catalog location.

.ci/pipelines/resources/pg-upgrade/data-proof-server.yaml

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.96%. Comparing base (d090bd1) to head (c3b1f29).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5145      +/-   ##
==========================================
- Coverage   63.69%   59.96%   -3.74%     
==========================================
  Files         123      111      -12     
  Lines        2424     2198     -226     
  Branches      572      527      -45     
==========================================
- Hits         1544     1318     -226     
- Misses        878      879       +1     
+ Partials        2        1       -1     

Flag	Coverage Δ
rhdh	59.96% <ø> (-3.74%)	⬇️

Components	Coverage Δ
Backend plugins	∅ <ø> (∅)
Backend app	66.66% <ø> (ø)
Frontend app	58.89% <ø> (ø)
Plugin utils	∅ <ø> (∅)
Dynamic plugins utils	∅ <ø> (∅)

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d090bd1...c3b1f29. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (1) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Context used

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 46 rules

✅ Skills: e2e-verify-fix, e2e-diagnose-and-fix

✅ Cross-repo context

  Explored: repo: redhat-developer/rhdh-chart (sha: e476f298)

  Explored: repo: redhat-developer/rhdh-operator (sha: cde4968e)

  Not relevant to this PR: redhat-developer/rhdh-plugins

  Not relevant to this PR: redhat-developer/rhdh-local

1. Wrong default proof repo 🐞 Bug ☼ Reliability

Description

pg_upgrade_data_proof_target_url defaults PG_UPGRADE_PROOF_REPO to "zdrapela/rhdh", so seeding the
catalog location can point at the wrong GitHub repo/SHA and fail to load the new catalog-info.yaml.
This can break the PG upgrade verification flow when the override isn’t set, despite the repo
already exposing REPO_OWNER/REPO_NAME defaults.

Code

.ci/pipelines/lib/postgres.sh[R531-547]

+pg_upgrade_data_proof_target_url() {
+  if [[ -n "${PG_UPGRADE_PROOF_URL:-}" ]]; then
+    echo "${PG_UPGRADE_PROOF_URL}"
+    return 0
+  fi
+
+  local sha="${PULL_PULL_SHA:-${PULL_BASE_SHA:-}}"
+  if [[ -z "${sha}" ]]; then
+    sha=$(git -C "${DIR}/../.." rev-parse HEAD 2> /dev/null || true)
+  fi
+  local repo="${PG_UPGRADE_PROOF_REPO:-zdrapela/rhdh}"
+  if [[ -z "${sha}" ]]; then
+    log::error "Cannot resolve pg-upgrade proof catalog URL (set PG_UPGRADE_PROOF_URL or PULL_PULL_SHA)"
+    return 1
+  fi
+  echo "https://github.com/${repo}/blob/${sha}/.ci/pipelines/resources/pg-upgrade/catalog-info.yaml"
+}

Relevance

⭐⭐⭐ High

Repo already centralizes REPO_OWNER/REPO_NAME defaults; hardcoded fork repo default likely fixed.

PR-#3340

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The helper constructs the target catalog URL using a hard-coded default repo, while the CI env
already defines the current repo owner/name and should be the source of truth for constructing
GitHub URLs.

.ci/pipelines/lib/postgres.sh[528-547]
.ci/pipelines/env_variables.sh[16-17]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`pg_upgrade_data_proof_target_url()` hard-codes a fallback repo (`zdrapela/rhdh`) when `PG_UPGRADE_PROOF_REPO` is unset. That makes the seeded catalog location URL potentially point to an unrelated repository and prevents the catalog entity from being registered.

### Issue Context
The CI environment already defines `REPO_OWNER` and `REPO_NAME` for the current repository; use those as the default (or compute from git remote) instead of a personal fork.

### Fix Focus Areas
- .ci/pipelines/lib/postgres.sh[531-547]
- .ci/pipelines/env_variables.sh[16-17]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Restore hides SQL errors 🐞 Bug ≡ Correctness

Description

postgres_restore_dumpall_file runs psql with ON_ERROR_STOP=0, so SQL errors during restore won’t
necessarily fail the step and the pipeline can proceed with a partially restored database. This
undermines the upgrade verification and the “persistence proof” signal because data can be missing
while the job continues.

Code

.ci/pipelines/lib/postgres.sh[R345-351]

+  log::info "Restoring pg_dumpall into ${pg_pod} from ${dumpfile}"
+  # Ignore benign "already exists" errors from globals/roles created by image init.
+  if ! oc exec -i -n "${namespace}" "${pg_pod}" -- psql -U postgres -v ON_ERROR_STOP=0 < "${dumpfile}"; then
+    log::error "psql restore failed"
+    return 1
+  fi
+  log::info "pg_dumpall restore completed"

Relevance

⭐⭐⭐ High

ON_ERROR_STOP=0 can mask restore failures; correctness signal likely enforced for upgrade
verification.

PR-#3340
PR-#4288

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The restore step explicitly disables stop-on-error and only checks the command exit status, allowing
statement-level restore failures to go unnoticed.

.ci/pipelines/lib/postgres.sh[329-352]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`postgres_restore_dumpall_file()` restores using `psql -v ON_ERROR_STOP=0`, which allows SQL errors to be ignored and still exit successfully. This can produce false-positive upgrade results.

### Issue Context
The comment mentions ignoring benign "already exists" errors, but the current implementation suppresses *all* statement errors. We need to still ignore known-safe conflicts while failing the restore on unexpected errors.

### Fix Focus Areas
- .ci/pipelines/lib/postgres.sh[329-352]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. expect imported from @playwright/test 📘 Rule violation ≡ Correctness

Description

The new Playwright spec imports expect from @playwright/test instead of
@support/coverage/test, which violates the required coverage-enabled test harness import pattern.
This can break coverage collection and consistency across E2E specs.

Code

e2e-tests/playwright/e2e/pg-upgrade-data-proof.spec.ts[R1-2]

+import { expect } from "@playwright/test";
+import { test } from "@support/coverage/test";

Relevance

⭐⭐⭐ High

Coverage harness pattern is enforced across Playwright specs; expect should be imported from
@support/coverage/test.

PR-#4798

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1798 requires that Playwright specs under e2e-tests/playwright/ import test and
expect from @support/coverage/test and not from @playwright/test. The added spec imports
expect from @playwright/test on line 1.

Rule 1798: E2E specs must import test/expect from @support/coverage/test, not @playwright/test
e2e-tests/playwright/e2e/pg-upgrade-data-proof.spec.ts[1-2]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Playwright spec imports `expect` from `@playwright/test`, but compliance requires importing both `test` and `expect` from `@support/coverage/test`.

## Issue Context
This ensures coverage hooks and consistent test setup across the E2E suite.

## Fix Focus Areas
- e2e-tests/playwright/e2e/pg-upgrade-data-proof.spec.ts[1-2]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Hardcoded helm release in diagnostics 🐞 Bug ⚙ Maintainability

Description

dump_postgres_diagnostics hardcodes helm get values rhdh, so when the Helm release name differs it
will fetch the wrong release (or fail) and reduce the usefulness of the captured diagnostics. This
is low-impact because it only affects the Helm diagnostics branch and is already guarded with `||
true`.

Code

.ci/pipelines/lib/postgres.sh[R147-149]

+      log::info "Helm values snapshot (postgresql image):"
+      helm get values rhdh -n "${namespace}" 2>&1 | grep -A20 -E 'postgresql:|^  image:' || true
+    fi

Relevance

⭐⭐ Medium

Hardcoding release may be tolerated in CI; no close accept/reject precedent found.

PR-#4288

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The diagnostics helper uses a fixed Helm release name even though the CI environment defines the
release name variable and the library otherwise aims to be reusable across install methods.

.ci/pipelines/lib/postgres.sh[141-149]
.ci/pipelines/env_variables.sh[71-77]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The Helm diagnostics path uses a hard-coded release name (`rhdh`) rather than the configured release name.

### Issue Context
The repo defines `RELEASE_NAME`, and this helper library is intended to be install-method aware.

### Fix Focus Areas
- .ci/pipelines/lib/postgres.sh[141-149]
- .ci/pipelines/env_variables.sh[71-77]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5145
• quay.io/rhdh-community/rhdh:pr-5145-c3b1f29f

[zdrapela]

/test e2e-ocp-operator-nightly

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[zdrapela]

/test e2e-ocp-operator-nightly

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[zdrapela]

/test e2e-ocp-operator-nightly

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[zdrapela]

/test e2e-ocp-operator-nightly

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[openshift-ci]

@zdrapela: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ocp-helm	c3b1f29	link	true	/test e2e-ocp-helm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[zdrapela]

/test e2e-ocp-operator-nightly

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud