redisdistributed: Redis alert rules are not namespace-scoped — fire cluster-wide, RedisTooManyMasters/RedisMissingMaster broken in multi-tenant clusters

[PiyushSingh-ZS]

charts/redisdistributed/templates/alerts.yaml renders a per-release PrometheusRule, but 6 of the 7 alert expressions have no namespace selector. They evaluate redis_* metrics across every namespace in the cluster, while stamping the release's namespace/service as static labels — so an alert about another namespace's Redis gets mislabeled as this release.

Only RedisDown is correctly scoped (it already filters {namespace="{{ .Release.Namespace }}", instance=~"{{ .Release.Name }}-redis-.*"}).

Affected exprs (templates/alerts.yaml)

Line	Alert	Expr
15	RedisMissingMaster	(count(redis_instance_info{role="master"}) or vector(0)) < 1
27	RedisTooManyMasters	count(redis_instance_info{role="master"}) > 1
39	RedisDisconnectedSlaves	count without (instance, job) (redis_connected_slaves) - sum without (instance, job) (redis_connected_slaves) - 1 > 0
51	RedisReplicationBroken	delta(redis_connected_slaves[1m]) < 0
63	RedisClusterFlapping	changes(redis_connected_slaves[1m]) > 1
75	RedisRejectedConnections	increase(redis_rejected_connections_total[1m]) > 0

Two are outright broken in multi-tenant clusters

When more than one release of this chart (or any scraped Redis) exists in the cluster:

• RedisTooManyMasters (count(redis_instance_info{role="master"}) > 1) — counts every master in every namespace. Each Redis instance is its own master, so with ≥2 scraped Redis instances this is permanently true → constant false-positive alert.
• RedisMissingMaster (count(...) or vector(0) < 1) — only drops to 0 if every Redis master in the whole cluster disappears at once, so it will never fire when a single release's master is actually gone (another namespace's master keeps the count ≥ 1). Silent failure of a critical alert.

The redis_connected_slaves-based rules keep the namespace label via without(instance, job), so they group per-namespace, but still emit alerts for other namespaces' Redis under this release's labels.

Real-world impact

Observed in a cluster running this chart in zopnight-beta and zopnight-stage (plus other Redis in auth, auth-stage, kops-dev, internal-docs) — the master-count alerts are meaningless there.

Suggested fix

Add namespace="{{ .Release.Namespace }}" to every metric selector, matching what RedisDown already does. E.g.:

count(redis_instance_info{role="master", namespace="{{ .Release.Namespace }}"}) > 1
delta(redis_connected_slaves{namespace="{{ .Release.Namespace }}"}[1m]) < 0
increase(redis_rejected_connections_total{namespace="{{ .Release.Namespace }}"}[1m]) > 0