feat: delete the Backup resources of expired backups

[chobostar]

feat: delete the Backup resources of expired backups

pgBackRest enforces the retention policy at the end of every base backup and drops the expired
backups from the repository. Nothing removes the Backup resources that describe them, so they
accumulate for as long as the cluster lives and keep advertising restore points whose data is
long gone.

Why nothing removes them today

CloudNativePG does have this cleanup, but only for the backups it takes itself:
deleteBackupsNotInCatalog
is called from
backupMaintenance
on the in-tree barmanObjectStore path, and its
useSameBackupLocation
filter returns false for any cluster without spec.backup.barmanObjectStore, which is every
plugin based backup. So even if the code were reached, it would skip our resources.

The barman-cloud plugin ran into the same gap and reimplemented the cleanup in its own sidecar:
deleteBackupsNotInCatalog,
attributing resources through a clusterUID recorded in the
plugin metadata.
This plugin had no equivalent.

What it looks like in practice

Numbers from one of our staging environments, running this plugin on 120 CloudNativePG clusters
with a two day retention:

backups.postgresql.cnpg.io resources	3417, across 42 namespaces
growth	~60 a day, one per cluster per scheduled backup (59, 59, 59, 59, 60, 60, 60, 60, 60, 61 over the last ten days)
oldest resource	five months old, completed, describing data the repository dropped after two days
per cluster	~28 on average, over 150 for the oldest ones
size	~1.6 KB each, ~5.3 MB in etcd

Those 5 MB are not about to fill anything up on their own: at 60 objects a day, backups alone
would need decades to matter. What makes it worth fixing is the kind of budget they spend.

etcd's backend is hard capped, at
2 GiB by default and 8 GiB as the largest size upstream suggests.
That cap is per member and shared by every object in the Kubernetes cluster, and reaching it does
not degrade backups, it stops the cluster: etcd raises a cluster-wide NOSPACE alarm and enters
"a maintenance mode which only accepts key reads and deletes"
(maintenance guide) until an
operator frees space, defragments and disarms the alarm. Every controller in the cluster is
read-only until then.

Against that budget this is a leak, in the precise sense: it grows linearly with the size of the
fleet and with uptime, it is never reclaimed, and nothing about it is load bearing. It also costs
more than the size of the objects. A Backup is created and then patched as it runs and
completes, and every revision stays in the backend until the apiserver's next compaction, every
five minutes by
default.
Compaction only frees those revisions inside the file; the file itself does not shrink until
someone defragments, since
"deleting application data does not reclaim the space on disk".
Beyond the storage, every one of those objects is also carried by the operator's informer cache
and returned by every LIST over backups.

And the other half of the motivation has nothing to do with space: each of those 3400 resources
claims a backup exists.

What the change does

The instance sidecar reads the catalog right after pgbackrest backup returns, which is exactly
when the retention policy has just run, and deletes the Backup resources whose
status.backupID is no longer in it. That costs one additional pgbackrest info per backup.

What keeps it from deleting a valid resource

This code deletes Kubernetes resources, so most of the diff is about not deleting the wrong
ones:

• The catalog has to be complete. pgbackrest info does not fail when a repository of a
  stanza cannot be read: it flags that repository
  (infoStanzaErrorAdd
  sets code 99) and still returns what the others hold. Acting on that answer would delete the
  resources of every backup living only in the unreachable repository. The catalog is now used
  only when it reports exactly the configured number of repositories and each of them could be
  read in full.
• A repository holding no backup yet is a complete answer, not a partial one. pgBackRest
  writes a backup to a single repository
  (backup.c:
  "repo option not specified, defaulting to repo1"), so the second repository of a stanza sits at
  code 2 and the stanza reports 4 "mixed"
  (info.c),
  which is the normal steady state rather than a failure. Codes 0 and 2 are accepted; 1, 3, 5 and
  99 are not.
• The location is recorded and compared. The cluster UID, the stanza and the repositories
  (endpoint, bucket and path, in pgBackRest's own order) go into status.pluginMetadata, and a
  resource is only considered when all of the entries it carries match the current ones. This
  covers an Archive repointed at another bucket, a stanza rename, and a cluster recreated under
  a name whose previous Backup resources outlived it.
• Only completed backups with an ID are touched. Failed resources are kept as the record of
  the failure, running ones are none of our business.
• An empty catalog is never acted upon. A backup has just been taken, so an empty catalog
  means the repository is not the one we expect, not that everything expired.
• Deletes carry the resource UID as a precondition, so a resource recreated between the
  listing and the deletion is not removed by mistake; the resulting conflict is tolerated, as is
  a NotFound.
• A cleanup failure is logged and swallowed. It must not turn a backup that succeeded into a
  failed one, which is also what CloudNativePG's own backupMaintenance does.

Backup resources are excluded from the sidecar client cache: they are listed once per backup,
and caching them would need a cluster wide watch, while the Role CloudNativePG builds for the
instance grants list, get and delete
namespaced.
No RBAC change is needed, and no CRD change either.

What to expect after upgrading

• Resources written before this change carry no recorded location. They are attributed to the
  current one, since that is the only thing they can be checked against, and they never acquire
  one: if the stanza or the repositories of a cluster ever change, those older resources are
  deleted by the next backup. Their data is not touched. This is the deliberate trade that makes
  the change useful for existing installations; the README says so.
• A location is compared as a whole, so renaming the endpoint of the same bucket, or reordering
  the repositories of an Archive, leaves the resources written before the change behind for
  good. They have to be removed by hand. This direction leaks rather than over-deletes, which is
  the way round we chose.
• The first backup after the upgrade clears the accumulated backlog and takes a little longer.
• Each deletion makes the operator's Backup controller log a Reconciler error … not found
  line (cloudnative-pg#11483),
  so that first backup emits a burst of them. Nothing is wrong, but log based alerting will
  notice.

Out of scope

Deleting a Backup resource still removes nothing from the repository, which is
#30 and the opposite
direction. It cannot be done the same way: the
Backup service
in cnpg-i has no deletion RPC, so the plugin would have to own a finalizer, and pgBackRest only
exists in the sidecar image, so that finalizer could only be cleared while a primary is running.
I have a separate write-up of the design and its failure modes and will post it on that issue.

Worth noting that this change makes resource deletion routine, so whatever semantics #30 lands
on will apply to many more deletions than today.

Testing

• 33 specs in internal/cnpgi/instance and 18 in internal/pgbackrest/catalog, covering each
  attribution rule and each catalog criterion separately.
• The catalog criteria are pinned both from parsed pgbackrest info output, so the JSON tags
  cannot silently stop matching, and from literal catalogs, so each criterion fails on its own.
• Mutation checked: removing the UID precondition, the stanza status check, the repository loop,
  the recorded clusterUID, the stanza comparison, or the error aggregation each makes the suite
  fail.
• golangci-lint with the repository configuration reports no issues; controller-gen and
  kustomize produce no changes to config/, api/ or manifest.yaml.
• No e2e test: a case with retention.full: 1, two backups and an assertion that the first
  Backup resource disappears would pin the whole path including RBAC and the real status codes.
  Happy to add it if you would like it in this PR.

Open questions

1. Legacy resources. They are attributed to the current location, which is what lets an
   existing installation clean up its backlog, but it means a later repoint deletes them.
   barman-cloud chose the opposite: its
   comparison
   requires the clusterUID to match, so resources without one are never deleted. A middle
   ground would be a one time adoption that writes the current location into matching legacy
   resources on the first run. Tell me which you prefer.
2. Placement. barman-cloud runs its cleanup as a periodic runnable on the primary. Here it
   runs inside the backup RPC, which is exactly when expiry happened and needs no extra runnable
   or interval setting, at the cost of not reacting to an expire run out of band.
3. A latent bug noticed while working on this, not touched here:
   PgbackrestBackupDatabase.RepoKey
   is tagged json:"repo_key" while pgBackRest emits repo-key, so the field never unmarshals.
   It is unused today, so nothing is broken, but it is exactly what Deleting the Backup resource doesn't delete the backup files from the MinIO bucket #30 would need to tell which
   repository a backup lives in. Say the word and I will send a separate PR.

[chobostar]

Closing in favour of #152, opened first and implementing the same cleanup as a periodic runnable
on the primary, matching plugin-barman-cloud. The parts of this branch worth keeping — a UID
precondition on the delete, recording the repository alongside the stanza, and the stanza status
gate with multiple repositories — are in a comment on that PR. The branch stays available if any
of it is wanted as a patch.