Skip to content

fix(jobs): circuit breaker consecutiveFailures only increments on exhausted retries — real disable threshold is much higher than configured #220

Description

@walidboulanouar

Problem

In packages/jobs/src/webhook-sweeper.ts, consecutiveFailures only increments when the full delivery retry chain is permanently exhausted (not on intermediate retries). A successful mid-run delivery resets the counter to 0. This means:

  • Configured threshold: 5
  • Real effective threshold: 5 permanently exhausted chains (not 5 simple failures)
  • A flaky endpoint that fails 4× then succeeds 1× then fails 4× again never trips the breaker

Also: consecutiveFailures is reset to 0 on ANY success in a sweep window, even if 4 other deliveries in the same window permanently failed.

Fix

Increment on every delivery failure (not just exhausted chain) and only reset on a fully-clean sweep window:

// On each permanent failure:
consecutiveFailures += 1;

// On success — only reset if NO failures in this sweep:
if (allSucceeded) consecutiveFailures = 0;
// Otherwise: leave counter as-is

Severity

MEDIUM — circuit breaker provides false safety; persistently flaky endpoints never auto-disable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions