Reload-safe path for named_collections — typed CRD field or a reload-safe settings category

[ovidiubuligan]

Problem

Any change to spec.settings.extraConfig rolls the ClickHouse pods, even when the change is a
named_collections entry — a section ClickHouse itself applies via SYSTEM RELOAD CONFIG
without a restart (verified on 26.7.1). On single-replica clusters that turns a URL tweak into
a full ClickHouse outage window.

The cause is that extraConfig is one opaque raw blob, hardcoded restart-required in
internal/controller/clickhouse/config.go:

Name:            ExtraConfigFileName,
requiresRestart: true,
Getter:          func(r *clickhouseReconciler) []byte { return r.Cluster.Spec.Settings.ExtraConfig.Raw },

We understand why (#15): arbitrary user YAML can't be classified, so restart-always is the only
safe default given the current API shape. This issue asks for an API shape that lets the
operator know a setting is reload-safe — with named_collections as the concrete, common case.

Why named_collections specifically

• They're a natural fit for CRs: per-environment endpoints/credentials for url()/S3/dictionary
  sources (our case: a refreshable materialized view pulling a dimension table from an internal
  REST API — the collection holds the per-env URL).
• They're reload-safe by ClickHouse's own semantics, and the operator already trusts that: the
  config-reload revision tracking introduced in Don't restart pods on some config changes #115 reads its marker back through a reserved
  named collection. The reload plumbing (restartable vs reloadable hash split,
  checksum/reloadable-configuration on the ConfigMap, operator-driven SYSTEM RELOAD CONFIG)
  already exists and already carries extraUsersConfig — named collections just can't reach it.

Proposal

Option A (preferred): a first-class typed field, e.g.

spec:
  settings:
    namedCollections:
      ctx_customers:
        url: "http://my-service.my-ns.svc:5400/Customers"
        format: "JSONEachRow"

rendered into its own config.d file whose generator sets requiresRestart: false, riding the
existing #115 reload path. Being typed, it also becomes validatable (see bonus below) and could
later support valueFrom secret refs for credentials.

Option B (more general): an explicitly reload-safe raw section, e.g.
spec.settings.extraReloadableConfig — same pass-through semantics as extraConfig, but the
user asserts reload-safety by choosing the field, and the operator classifies it into the
reloadable bucket. This covers the whole category of reload-safe server settings without the
operator having to classify anything itself.

Option C (least preferred): an opt-out annotation excluding extraConfig from the restart
hash — mentioned only for completeness; it's a footgun compared to A/B.

Bonus: the misplacement trap A would also close

Trying to dodge the restart by putting named_collections under extraUsersConfig fails
silently: the users-config parser ignores server-tree sections — server starts clean, no log
warning, system.named_collections empty (verified on clickhouse-server:26.7.1.1315-alpine;
the same YAML via config.d registers fine). Until a typed field exists, a validating-webhook
check rejecting server-tree keys inside extraUsersConfig would convert that silent no-op into
a loud error.

Environment

• operator v0.0.7 (ghcr.io), behavior re-verified unchanged on main @ fc046cc1
• ClickHouse 26.7.1.1315-alpine, Replicated database, 1–2 replicas per cluster, AKS

[GrigoryPervakov]

Good point, we need to think about how to do it without bloating the spec.
Option A is hard to support because we would need to always keep track of the config schema in ClickHouse and keep it the same.

As a workaround, you may mount a custom ConfigMap into the config dir. The operator knows nothing about such ConfigMap content updates, so the Pod controller would update it without restart.

[melancholictheory]

On the schema tracking concern, that objection lands on option A but not on option B. extraReloadableConfig would be a raw pass-through exactly like extraConfig, with no typed fields and nothing for the operator to keep in sync with ClickHouse. The only difference is which bucket the generator puts it in, so it rides the existing reloadable hash split instead of the restart one. The assertion of reload-safety comes from the user picking that field, not from the operator classifying anything.

The cost is that a user can put a restart-only setting there and get a config that silently does not take effect until the next restart. That is a real footgun, though a narrower one than the current behaviour of restarting on every extraConfig edit.

The custom ConfigMap workaround should work for the immediate problem. Worth noting it does not compose with extraConfig: both would target /etc/clickhouse-server/config.d, and the operator projects colliding mount paths into a single projected volume, so anyone using both is relying on that merge rather than on a documented contract.

Separately I opened #317 for the misplacement trap in the bonus section, since it needs no API decision either way. It warns when extraUsersConfig carries a server-tree section rather than rejecting, so a cluster that already has one stays updatable. Easy to turn into a hard error if you would rather.