Node.js EurekaClient: retry loop has no circuit breaker or inter-cycle backoff

[balhar-jakub]

Description

The Node.js onboarding enabler EurekaClient has no circuit breaker for its retry logic. When Eureka is unreachable, the client enters an unbounded retry loop with no backoff between interval cycles.

Retry mechanism

Per-request retries use a linear delay (EurekaClient.js:703-704):

const nextRetryDelay = this.config.eureka.requestRetryDelay * (retryAttempt + 1);
// = 500ms, 1000ms, 1500ms → burst exhausts in ~3s

With defaults (defaultConfig.js:56-59):

heartbeatInterval: 30000,
registryFetchInterval: 30000,
maxRetries: 3,
requestRetryDelay: 500,

Interval timers never stop

Heartbeats and registry fetches use setInterval that fires every 30s forever (EurekaClient.js:313-316, 347-352):

startHeartbeats() {
    this.heartbeat = setInterval(() => this.renew(), 30000);
}
startRegistryFetches() {
    this.registryFetch = setInterval(() => this.fetchRegistry(...), 30000);
}

clearInterval is only called in stop(). There is no pause or backoff on failure.

Resulting failure pattern

Each 30s tick spawns a fresh request with its own independent 3-retry burse, producing this pattern indefinitely:

t=0:   heartbeat burst (4 attempts, ~3s) — all fail
t=0:   registry burst (4 attempts, ~3s) — all fail
t=30:  heartbeat burst again
t=30:  registry burst again
...repeats forever

What is missing

• No consecutive failure counter that spans interval cycles
• No exponential backoff between cycles (30s interval is fixed)
• No circuit breaker state (closed → open → half-open)
• No pause/suspend mechanism when Eureka is persistently unreachable

Expected behavior

After N consecutive interval-cycle failures, the client should either:

• Increase the interval between cycles (exponential backoff), or
• Enter a circuit-open state where it stops attempting until a cool-down period elapses

Files

• onboarding-enabler-nodejs/src/EurekaClient.js:313-316 (startHeartbeats)
• onboarding-enabler-nodejs/src/EurekaClient.js:347-352 (startRegistryFetches)
• onboarding-enabler-nodejs/src/EurekaClient.js:698-710 (retry logic)
• onboarding-enabler-nodejs/src/defaultConfig.js:56-59 (defaults)

[balhar-jakub]

Problem Analysis — Node.js EurekaClient Circuit Breaker (#4775)

1. Refined Problem Statement

Persona: A system programmer deploying a Node.js-based service on z/OS using the Zowe onboarding enabler. They configure their service to register with the Zowe Discovery Service (Eureka) for high availability.

Real-world scenario: During a scheduled z/OS maintenance window, the Discovery Service is restarted. The Node.js EurekaClient enters an unbounded retry storm: every 30 seconds it fires 4 HTTP requests (heartbeat + registry fetch, each with 3 retries). There is no circuit breaker, no exponential backoff between cycles, and no pause mechanism. The client bombards the network indefinitely until the Discovery Service comes back — which could be hours.

Root cause: startHeartbeats() (EurekaClient.js:314) and startRegistryFetches() (EurekaClient.js:348) use setInterval with a fixed 30s timer. Each tick spawns an independent request with its own 3-retry burst (requestRetryDelay: 500ms, linear). There is no state spanning cycles — no consecutive failure counter, no backoff, no circuit breaker.

Impact: Request storms, log spam (warn/error on every failure), network congestion. In HA deployments with many Node.js services, this can cause cascading load on the Discovery Service cluster.

2. Current Architecture Touchpoints

Touchpoint	File	What happens
Heartbeat timer	EurekaClient.js:313-316	setInterval(() => this.renew(), heartbeatInterval) — fires every 30s forever
Registry fetch timer	EurekaClient.js:347-352	setInterval(() => this.fetchRegistry(...), registryFetchInterval) — fires every 30s forever
Per-request retry	EurekaClient.js:698-710	Linear delay: requestRetryDelay * (retryAttempt + 1) → 500ms, 1000ms, 1500ms. Exhausts in ~3s.
Configuration	defaultConfig.js:56-59	heartbeatInterval: 30000, registryFetchInterval: 30000, maxRetries: 3, requestRetryDelay: 500
Stop/cleanup	EurekaClient.js:stop()	clearInterval(this.heartbeat), clearInterval(this.registryFetch) — only called on application shutdown

The module lives entirely within onboarding-enabler-nodejs/. No other Zowe components depend on Node.js EurekaClient internals — it's consumed as an npm package by downstream Node.js services.

3. Scope

In scope:

• Add a circuit breaker state machine (CLOSED → OPEN → HALF_OPEN) to the Node.js EurekaClient
• Replace setInterval with a setTimeout-based scheduling loop that respects circuit breaker state
• Add exponential backoff between heartbeat/registry fetch cycles when the circuit is OPEN
• Make the circuit breaker configurable via eureka-client.yml (opt-out: circuitBreaker.enabled: false falls back to legacy setInterval)
• Unit tests for the circuit breaker state machine (mocha + sinon + chai)
• Integration tests for EurekaClient with circuit breaker wired in

Out of scope:

• Changes to the Java-side Discovery Service or Eureka server
• Changes to the Eureka REST protocol
• Circuit breaker for the initial registration call — only heartbeats and registry fetches
• Any new npm dependencies — the circuit breaker must be implemented in pure JavaScript
• Changes to other Node.js services (the enabler is the library; services consume it)
• Documentation updates (handled separately)

4. Acceptance Criteria

1. AC1 — Circuit opens: After N consecutive interval-cycle failures (configurable, default 5), the circuit transitions from CLOSED → OPEN and heartbeat/registry fetches are suspended.
2. AC2 — Requests blocked while OPEN: While the circuit is OPEN, no HTTP requests are made to the Discovery Service. The client logs the blocked attempt at WARN level.
3. AC3 — HALF_OPEN probe: After the cooldown period elapses (configurable, default 60s), the circuit transitions to HALF_OPEN and allows exactly one probe request per cycle.
4. AC4 — Success closes circuit: If the probe request succeeds, the circuit transitions HALF_OPEN → CLOSED, resets the failure counter, and resumes normal operation.
5. AC5 — Failure re-opens circuit: If the probe request fails, the circuit transitions HALF_OPEN → OPEN and a new cooldown period begins with exponential backoff.
6. AC6 — Exponential backoff: The cooldown period doubles after each consecutive failure cycle (baseCooldown × 2^(failureCount-1)), capped at a configurable maximum (default 5 minutes).
7. AC7 — Backward compatibility: When config.eureka.circuitBreaker.enabled is set to false, the existing setInterval behavior is preserved with no change in functionality.
8. AC8 — Observable state transitions: Circuit state changes (OPEN, HALF_OPEN, CLOSED) are logged at appropriate levels (WARN for OPEN, INFO for HALF_OPEN/CLOSED).
9. AC9 — Clean shutdown: EurekaClient.stop() correctly clears all setTimeout handles regardless of circuit state. No timers leak on stop/restart.

5. Design Questions for Architect

1. State machine design: Should the circuit breaker be a standalone class (CircuitBreaker.js) separate from EurekaClient.js, or integrated directly into the client? Trade-offs: testability and reuse vs. simplicity.
2. Timer implementation: Should heartbeat and registry fetch share one circuit breaker instance, or should each have its own independent circuit breaker? The current setInterval timers fire independently.
3. Configuration surface: What config keys should live under eureka.circuitBreaker? Candidates: enabled, maxFailures, cooldownTime, backoffMax. Should cooldown time be called cooldownTime or baseCooldown? Should maxFailures be failureThreshold?
4. API design for EurekaClient integration: What methods should the circuit breaker expose? allowRequest(), recordSuccess(), recordFailure(), reset(), getNextCooldown(). Should state transitions be exposed as events (EventEmitter) or callbacks?
5. HALF_OPEN semantics: When in HALF_OPEN state, should allowRequest() return true exactly once (and subsequent calls return false until the probe resolves), or return a special value like 'HALF_OPEN' to signal the transition?
6. Backward compatibility check: The renew() method currently takes no callback parameter. Circuit breaker integration requires knowing whether a heartbeat succeeded or failed. Should renew() be modified to accept a callback, or should the scheduling loop wrap the call differently?
7. Existing retry logic interaction: The per-request retry burst (maxRetries: 3, requestRetryDelay: 500) in eurekaRequest() currently retries independent of circuit state. When the circuit breaker is active, should per-request retries still apply, or should the circuit breaker supersede them?

6. Constraints

• Zero new npm dependencies — the circuit breaker must be pure JavaScript (Node.js standard library only)
• Backward compatible — existing consumers of the eureka-js-client npm package must not break
• Configurable defaults — all circuit breaker parameters must be overridable in eureka-client.yml
• Node.js 16+ compatibility — the Zowe onboarding enabler supports Node.js 16 and above
• ES module format — the module uses CommonJS (require/module.exports), not ES modules
• EPL-2.0 license — all new files must carry the EPL-2.0 license header (no dual-licensing, no MIT/Apache fragments)

---

Prior art: PR #4781 (closed without merge, 2026-07-07) implemented a working solution with 170 passing tests. The architect should review that approach but design the solution independently — the prior PR demonstrates feasibility, not the final design.