feat: delete Backup objects not present in the pgBackRest catalog

[ermakov-oleg]

Problem

CloudNativePG does not delete Backup objects; the component taking the backups is expected to (the in-tree Barman code and plugin-barman-cloud both do). This plugin runs pgbackrest expire in the repository but leaves the Kubernetes objects behind, so they accumulate without bound and the operator keeps all of them in its informer cache. On a fleet of ~340 clusters we reached 112k Backup objects and operator OOMs before an external cleanup job caught up.

What this PR does

Adds a CatalogMaintenanceRunnable to the instance sidecar, ported from plugin-barman-cloud (internal/cnpgi/instance/retention.go) and reduced to the one operation pgBackRest does not do itself (no retention enforcement, no status update). On the current primary it periodically reads the catalog with pgbackrest info and deletes the completed Backup objects of the cluster whose backup ID is no longer in the catalog.

New Archive field: instanceSidecarConfiguration.catalogMaintenanceIntervalSeconds (default 1800, 0 disables the maintenance, otherwise at least 60). manifest.yaml and the CRD are regenerated. No RBAC change: the instance service account already holds list/get/delete on backups from CloudNativePG.

When the catalog is trusted

pgbackrest info reports a missing stanza or an unreadable repository inside the JSON with exit 0 and still returns what the other repositories hold. The cleanup therefore acts only when the catalog:

• is for the expected stanza,
• reports every configured repository as readable (ok or no backup; a stanza status of mixed is the normal answer when one repository holds backups and another is still empty),
• lists at least one backup — an empty catalog is never acted upon, because a freshly recreated stanza looks identical.

How a Backup is matched

The Backup RPC now records the cluster UID, the stanza and the repository locations (endpointURL/bucket + destinationPath, in configuration order) in status.pluginMetadata. Only objects whose recorded values all equal the current ones are considered. Consequences worth knowing:

• Objects created by plugin versions before this change carry no location and are never deleted; remove them once by hand (or with an existing cleanup job) after upgrading.
• Repointing an Archive, adding/removing/reordering its repositories or renaming the stanza makes the previously recorded objects unmatchable; they are kept, and each cycle logs how many objects it kept for that reason.
• Only Backups labelled cnpg.io/cluster=<cluster> are listed. ScheduledBackup and kubectl cnpg backup set the label; a hand-applied Backup manifest without it is never cleaned.
• Deletes carry the object UID as a precondition, so a Backup recreated under the same name between the list and the delete is left alone. Backups that completed after the catalog was read are skipped. Failed backups are not in the catalog and are not touched; a follow-up PR will add an age-based rule for them.

Load

The list is narrowed to the cluster label, paginated (500 per page, an expired continue token restarts the pass once), replicas do not read the Archive, and the first cycle is jittered over the default interval so a fleet-wide rollout does not fire every primary at the same second.

Credits

The catalog gate per repository, the repository locations in the metadata and the UID precondition come from #153 by @chobostar, who opened an independent implementation of the same feature on the same day and closed it in favour of this one. Co-authored-by is on the commit.

Unrelated e2e fixes carried in this PR

Two separate commits make the e2e suite runnable again; happy to split them into their own PR if you prefer.

• test/e2e/internal/objectstore/minio.go now pulls quay.io/minio/minio:latest: the minio/minio repository is gone from Docker Hub (the registry answers 404), so every e2e run currently fails with ImagePullBackOff before a single spec starts.
• The suite installs CloudNativePG v1.30.0 instead of main. Since cloudnative-pg#11319 (2026-08-27) new instances bootstrap through an init container instead of a Job, and recovering a cluster needs the restore hooks served from the instance sidecar (plugin-barman-cloud did that in feat: serve restore hooks from the instance sidecar cloudnative-pg/plugin-barman-cloud#1025). This plugin still serves them from the job sidecar only, so on CNPG main every recovery-based test hangs on the bootstrap-instance init container. That gap is real for the next CloudNativePG release and deserves its own PR; it is not addressed here.

Unit tests cover the filter chain, the catalog gate (including real pgbackrest info output with per-repository statuses), the location matching, the UID precondition, pagination and the 410 restart, the interval semantics and cancellation; make lint/make test/make build pass.

https://claude.ai/code/session_01NRY9m8yhU4NVCCBHnURaL9

[chobostar]

I opened #153 independently for the same problem. Yours came first and places the work as a periodic runnable like plugin-barman-cloud does, so I am closing mine. Three things from my branch that may be worth taking.

1. UID precondition on the delete

Between the List and the Delete a Backup with the same name can be recreated, and the
delete takes the new one.

preconditions := client.Preconditions{UID: &backup.UID}
err := cli.Delete(ctx, backup, preconditions)
if err != nil && !apierrors.IsNotFound(err) && !apierrors.IsConflict(err) {
    // conflict = the precondition did not hold, the listed object is already gone
    errs = append(errs, ...)
}

2. Record the repository, not only the stanza

Closes the known gap about an Archive repointed to another bucket with the same stanza. Add a
third entry to backupResultMetadata, rendered from the configuration in pgBackRest's order:

<endpointURL>/<bucket><destinationPath>[,<endpointURL>/<bucket><destinationPath>...]

and require it to match in useSameBackupLocation. No behaviour change for older objects: they
carry no clusterUID and your filter already skips them.

3. The status gate assumes a single repository

catalogIsAuthoritative accepts stanza status 0 only. With two repositories that state is never
reached: a base backup goes to the default repository
(backup.c#L2555-L2561)
while archive-push writes to all of them, so the second one sits at code 2 and the stanza
reports 4 "mixed"
(info.c#L828).
Multiple repositories are not wired through the plugin today — --repo appears nowhere in the
tree, no test configures more than one — so this only matters if that changes.

One caveat for that day: code 4 is also what a repository that could not be read produces, and
pgbackrest info then still returns the other repositories' backups, so the per-repository
statuses in repo[] have to be checked as well. I have that implemented with tests if it ever
becomes relevant.

[ermakov-oleg]

Ported from #153, with you as co-author on the commit:

• UID precondition on the delete, Conflict tolerated like NotFound.
• repositories in pluginMetadata (<endpointURL>/<bucket><destinationPath>, comma-joined, configuration order), required to match.
• Per-repository status gate: every configured repository must report ok or no backup, stanza mixed is accepted. Plus the stanza name check and the refusal of an empty catalog.
• Backups without a backupID are skipped.

Unchanged: objects without location metadata are never deleted, and the cleanup stays a periodic runnable.

Thanks for closing #153 and for the write-up.