Summary
When the OCM service log API returns HTTP 429 (rate limit exceeded), ocm-agent fleet mode enters an infinite retry loop. The restoreNotificationStatus() function rolls back lastTransitionTime after every failed send, causing canSendNotification() to return true on every reconciliation cycle (~15 min). This generates ~24 wasted API requests per hour, per affected cluster, indefinitely.
Root Cause
The issue is in pkg/handlers/webhookrhobsreceiver.go in the processAlert function (line 415-423). The code flow:
updateNotificationStatus() sets lastTransitionTime = now and increments FiringNotificationSentCount (written to K8s MFNR)
sendNotification() POSTs to OCM API → receives 429
restoreNotificationStatus() rolls back both lastTransitionTime and FiringNotificationSentCount to their pre-update values
On the next reconciliation (~15 min later), canSendNotification() reads the rolled-back lastTransitionTime, finds it is older than ResendWait (24h), and returns true
Steps 1-4 repeat indefinitely
The rollback was designed as a workaround for the optimistic update pattern (status is updated before the send, so it must be rolled back on failure). However, it does not distinguish between transient errors (where retry is appropriate) and rate-limit errors (where retry makes the problem worse).
Impact
API waste: Each 15-min cycle fires 1 initial POST + 2 SDK retries per cluster = 3 requests. With 2 clusters affected, that is 6 requests/cycle or ~24 requests/hour, all returning 429.
Envoy/Limitador pressure: The fleet service account's request-rate quota (enforced by the Envoy sidecar via Limitador at limitador.app-sre-rate-limiting.svc:8081) is consumed by these wasted retries, potentially starving legitimate service log sends for other clusters on the same management cluster.
Silent service log loss: The service log is never created — the customer never gets notified — but the system keeps trying without success.
Observed Behavior
Error in ocm-agent-fleet logs:
error="can't post service log: status is 429, identifier is '8', code is 'OCM-CA-8' and operation identifier is '...': Exceeds rate limit request"
Proposed Fix
Two changes:
- pkg/ocm/ocm.go — Return typed RateLimitError on 429
Add a RateLimitError type and return it from SendServiceLog when the response status is 429. This allows callers to distinguish rate limiting from other failures.
- pkg/handlers/webhookrhobsreceiver.go — Rate-limit-aware backoff
Summary
When the OCM service log API returns HTTP 429 (rate limit exceeded), ocm-agent fleet mode enters an infinite retry loop. The restoreNotificationStatus() function rolls back lastTransitionTime after every failed send, causing canSendNotification() to return true on every reconciliation cycle (~15 min). This generates ~24 wasted API requests per hour, per affected cluster, indefinitely.
Root Cause
The issue is in pkg/handlers/webhookrhobsreceiver.go in the processAlert function (line 415-423). The code flow:
updateNotificationStatus() sets lastTransitionTime = now and increments FiringNotificationSentCount (written to K8s MFNR)
sendNotification() POSTs to OCM API → receives 429
restoreNotificationStatus() rolls back both lastTransitionTime and FiringNotificationSentCount to their pre-update values
On the next reconciliation (~15 min later), canSendNotification() reads the rolled-back lastTransitionTime, finds it is older than ResendWait (24h), and returns true
Steps 1-4 repeat indefinitely
The rollback was designed as a workaround for the optimistic update pattern (status is updated before the send, so it must be rolled back on failure). However, it does not distinguish between transient errors (where retry is appropriate) and rate-limit errors (where retry makes the problem worse).
Impact
API waste: Each 15-min cycle fires 1 initial POST + 2 SDK retries per cluster = 3 requests. With 2 clusters affected, that is 6 requests/cycle or ~24 requests/hour, all returning 429.
Envoy/Limitador pressure: The fleet service account's request-rate quota (enforced by the Envoy sidecar via Limitador at limitador.app-sre-rate-limiting.svc:8081) is consumed by these wasted retries, potentially starving legitimate service log sends for other clusters on the same management cluster.
Silent service log loss: The service log is never created — the customer never gets notified — but the system keeps trying without success.
Observed Behavior
Error in ocm-agent-fleet logs:
Proposed Fix
Two changes:
Add a RateLimitError type and return it from SendServiceLog when the response status is 429. This allows callers to distinguish rate limiting from other failures.