[FavorMax]

Overview

All four issues target backend/src/jobs/verificationWorker.ts. On main that file is currently a placeholder: runCycle() returns 2, processEvent() only logs, and isRetryableError() always returns false. So nothing drives verification requests through the pipeline. A full worker existed until 93d39ce ("fix: resolve remaining TypeScript compilation errors…") replaced it.

The issues depend on each other, so they're delivered together, in dependency order:

1. Restore the worker pipeline from 2998b4c, its last complete version. Every service it uses still exists on main with the same API.
2. [Backend] [Feature] Implement node/queue abstraction so multiple workers can run concurrently #694: the queue it claims work from, with parallel lanes.
3. [Backend] [Feature] Link worker processing back to VerificationJob state transitions #696: job state transitions through the state-machine service.
4. [Backend] [Feature] Implement per-request retry and dead-letter handling in the worker #693: retry and dead-lettering when work fails.
5. [Backend] [Feature] Implement worker heartbeat, health, and telemetry reporting #695: heartbeats, a health endpoint and per-request timing.

Related Issues

• Closes [Backend] [Feature] Implement per-request retry and dead-letter handling in the worker #693: Implement per-request retry and dead-letter handling in the worker
• Closes [Backend] [Feature] Implement node/queue abstraction so multiple workers can run concurrently #694: Implement node/queue abstraction so multiple workers can run concurrently
• Closes [Backend] [Feature] Implement worker heartbeat, health, and telemetry reporting #695: Implement worker heartbeat, health, and telemetry reporting
• Closes [Backend] [Feature] Link worker processing back to VerificationJob state transitions #696: Link worker processing back to VerificationJob state transitions

Changes

#696 Worker → VerificationJob state transitions

• Each claimed request drives its VerificationJob through verificationService, the existing state-machine service. The transitions are pending → processing → tee_verifying → minting → completed, plus failed.
• Each field is saved at the stage that produces it:
  • teeAttestationHash, teeSignature and codeMeasurementHash on entering tee_verifying
  • stellarTransactionHash on entering minting
  • the minted certificateId on completed (new)
• Jobs are created with manifestHash and requestId, so contract-event ingestion (advanceFromAttestationEvent / completeFromMintEvent) can match events back to them.
• The registry-authorization precheck runs before the attestation is submitted.
• Crash recovery is unchanged: processing resumes from the job's current state, and an already-submitted mint is confirmed rather than rebuilt.
• Kept from main: verifyManifestForJob and startManifestRehashWorker, the manifest re-hash check.

#694 Queue abstraction and concurrent workers

• New services/workerQueue.service.ts defines a WorkerQueue contract with three operations: claim, release and deadLetter.
• MongoWorkerQueue implements it on top of the existing atomic findOneAndUpdate claim and its time-limited lease. Two workers can never hold the same request, a crashed worker's request is reclaimed when its lease expires, and every later write is checked against the lease.
• The worker gets its work only from the queue (deps.queue). deps.events now handles only progress writes.
• New VERIFICATION_WORKER_CONCURRENCY setting (default 1). Each cycle runs that many lanes in parallel, sharing the batch budget. Any number of worker processes can share one database.

#693 Retry policy and dead letters

• retryDelayMs(attempt, baseMs) doubles the delay on each attempt and caps it at 15 minutes.
• Transient errors are released back to the queue for the next attempt. Transient means SPV gateway errors, RPC and network errors, and an unknown transaction outcome.
• A permanent failure, or a transient one on the last allowed attempt:
  1. marks the linked job failed with the reason, through the state machine;
  2. records the request in the new deadLetteredRequests collection (models/DeadLetteredRequest.model.ts): event, job, CIDs, requester, attempts, reason, error code, whether the final error was retryable, the worker and a timestamp;
  3. marks the request failed.
• Dead-lettering is idempotent per request. If the worker has lost its lease, the dead-letter entry it wrote is removed again, so a request another worker still owns is never quarantined.
• An SPV "not verified" verdict is a normal rejection, not a dead letter. It's an answer, not a processing failure.

#695 Heartbeat, health and telemetry

• New workerHeartbeats collection (models/WorkerHeartbeat.model.ts) with fields workerId, status, startedAt, lastHeartbeatAt, lastProcessedLedger, processedCount, failedCount and lastProcessingMs.
• The worker writes a heartbeat at start, every VERIFICATION_WORKER_HEARTBEAT_INTERVAL_MS (default 15 s), after each cycle, and on stop. A failed heartbeat write is logged, never fatal.
• New GET /api/v1/worker/health (controllers/worker.controller.ts → services/workerHeartbeat.service.ts → WorkerHeartbeat model). A worker counts as alive while it reports running and its last heartbeat is newer than WORKER_HEARTBEAT_STALE_MS (default 60 s). The endpoint returns 200 while at least one worker is alive and 503 otherwise.
• Every request logs its processing duration (durationMs), including attempts that end in a retry or a dead letter.

Configuration

The new optional settings, all with defaults, are documented in .env.example:

• VERIFICATION_WORKER_CONCURRENCY
• VERIFICATION_WORKER_HEARTBEAT_INTERVAL_MS
• WORKER_HEARTBEAT_STALE_MS

No new dependencies.

Tests

• __tests__/verificationWorker.test.ts (the existing suite) now uses a mocked WorkerQueue and expects deadLetter where requests now fail permanently. Three fixtures were updated to match the restored behaviour: createJob now receives manifestHash and requestId, and tee_verifying jobs carry the codeMeasurementHash the state machine requires. New cases cover concurrency lanes, the back-off curve, persisting the certificate ID, dead-letter ordering, SPV rejections not being dead-lettered, duration logging and heartbeats.
• services/__tests__/attestationMint.integration.test.ts (from [Backend] [Testing] Integration test attestation and mint flow with mocked RPC #725) can't run on main, because its MongoMemoryServer setup is commented out. This PR restores that setup and adds two things: a dead-letter assertion, and a case where two workers with 3 lanes each drain 12 requests against real MongoDB and mint each one exactly once.
• New services/__tests__/workerQueue.service.test.ts (real MongoDB):
  • 8 workers claim 40 requests concurrently and no request is handed out twice;
  • a held lease blocks other claims, and an expired lease can be reclaimed;
  • release makes a request claimable again only from its retry time;
  • dead-lettering stores the right fields, is idempotent, and backs out when the lease is lost.
• New __tests__/worker.health.test.ts (real MongoDB): heartbeat upserts, the alive/stale rules, and GET /api/v1/worker/health returning 200 or 503.

Verification Results

Commands were run from backend/. Jest was run with jest.config.ts minus setupFilesAfterEnv, because the setup file it names is missing on main. The full-suite runs used dummy values for the required environment variables, because config/env.ts calls process.exit when they are missing.

• Affected suites: all pass.
  • __tests__/verificationWorker.test.ts: 42 of 42 pass. On main it fails 24 of 30.
  • services/__tests__/attestationMint.integration.test.ts: 11 of 11 pass. On main the suite can't run.
  • services/__tests__/workerQueue.service.test.ts: 8 of 8 pass.
  • __tests__/worker.health.test.ts: 6 of 6 pass.
• Full suite, compared with main at 658d68d:
  • main: 158 failed, 331 passed
  • this branch: 124 failed, 392 passed
  • No new failures, test for test. 34 tests that fail on main now pass (24 in the worker suite, 10 in the integration suite). The remaining failures are the same pre-existing ones on both, including one in verificationRequestEvent.service.test.ts, which this PR doesn't touch.
• Type check: a scratch tsconfig rooted at every changed and new source file reports no errors.
• Full tsc / pnpm --filter @stellarproof/backend run build: fails on main and on this branch, because of a syntax error in services/kms.service.ts around line 430. That's a broken merge of feat(backend): certificate analytics, revocation, SPV linkage, KMS re-encryption (#741, #742, #743, #746) #926 and feat(kms): key expiry, scheduled rotation, and master-key rewrap #927, which this PR doesn't touch. CI's "Build Backend" step will fail for the same reason; main itself has been red since a53f674.
• Lint: not run. ESLint isn't installed or configured in the backend package.
• CI: pending

Acceptance Criteria

The acceptance criteria are the same for all four issues. I've ticked each one only where this PR provides evidence for it.

#693 Retry and dead-letter handling

• Retry policy with exponential back-off (retryDelayMs; release with back-off is tested)
• Failures routed to the dead-letter collection (deadLetteredRequests; tested on real MongoDB and end to end)
• The linked job is marked failed with the reason (through the state machine, before dead-lettering; tested)
• Layered architecture: worker → workerQueue / verificationService → DeadLetteredRequest / VerificationJob / VerificationRequestEvent models
• Data comes from the database: dead letters and job state are persisted and asserted on reload
• Real .env credentials and real-life data: not done (mongodb-memory-server)
• API versioning: not applicable. This is worker-side behaviour with no endpoint.
• Production ready: idempotent dead-lettering, lease-safe writes, typed outcomes
• Proof of work screenshot: not attached (test output above)
• PR includes Closes #693 and a summary

#694 Queue abstraction and concurrent workers

• Claim/release queue abstraction (WorkerQueue, MongoWorkerQueue)
• Multiple concurrent workers supported (any number of processes, plus VERIFICATION_WORKER_CONCURRENCY lanes each; two-worker end-to-end test)
• No double-claiming (8 workers × 40 requests, and the two-worker mint test: each request is claimed and minted exactly once)
• Layered architecture: worker → workerQueue service → VerificationRequestEvent model
• Data comes from the database: claims, leases and releases are MongoDB writes
• Real .env credentials and real-life data: not done (mongodb-memory-server)
• API versioning: not applicable. No endpoint.
• Production ready: atomic claims, lease expiry recovery, lease-guarded writes
• Proof of work screenshot: not attached (test output above)
• PR includes Closes #694 and a summary

#695 Heartbeat, health and telemetry

• Worker heartbeats are persisted (workerHeartbeats: workerId, status, lastProcessedLedger and counters)
• Worker health endpoint (GET /api/v1/worker/health, 200 or 503)
• Processing durations are logged (durationMs on every finished attempt)
• Layered architecture: worker.controller → workerHeartbeat.service → WorkerHeartbeat model
• Data comes from the database: health is computed from stored heartbeats
• Real .env credentials and real-life data: not done (mongodb-memory-server)
• API versioning: /api/v1/worker/health
• Production ready: heartbeat failures never stop processing; the health list is bounded
• Proof of work screenshot: not attached (route tests show the 200 and 503 responses)
• PR includes Closes #695 and a summary

#696 Worker → VerificationJob state transitions

• Jobs move through the full stage sequence (asserted in order against real MongoDB by the integration suite)
• Attestation hashes and signatures are persisted (teeAttestationHash, teeSignature, codeMeasurementHash, stellarTransactionHash, plus certificateId)
• Reuses the state-machine service (verificationService.updateJobStatus; illegal transitions are rejected by the existing VALID_TRANSITIONS)
• Layered architecture: worker → verificationService → VerificationJob model
• Data comes from the database: job state is read and written through MongoDB
• Real .env credentials and real-life data: not done (mongodb-memory-server)
• API versioning: not applicable. Worker-side.
• Proof of work screenshot: not attached (test output above)
• PR includes Closes #696 and a summary

Notes

These are pre-existing problems on main that this PR doesn't fix:

• services/kms.service.ts has a syntax error, so tsc and the backend build fail.
• jest.config.ts points at a setup file that doesn't exist.
• One assertion in verificationRequestEvent.service.test.ts fails on main and fails here too.

[drips-wave]

@FavorMax Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Tybravo]

@FavorMax

Thank you for the tasks done