|
| 1 | +# Stovepipe breakage repair coordination |
| 2 | + |
| 3 | +Stovepipe accepts a broken check, decides whether to start a repair or attach it to existing work, and tracks progress until validation confirms recovery. A repair can cover several checks. The [workflow map](workflow.md#workflow) shows this work running alongside validation. |
| 4 | + |
| 5 | +`buildanalysis.Controller.Process` persists observations and manages failure episodes and recovery evidence. Each episode also stores its authoritative `FixID` assignment, updated by `fix`. The `fix` queue stage owns repair decisions, associations, and submission; `fixsignal` observes execution and resolves repairs. Both stages implement [`consumer.Controller`](../../../platform/consumer/controller.go) through `Process(context.Context, consumer.Delivery) error`. [`fixer.RepairEvaluator`](../../../stovepipe/extension/fixer/fixer.go) recommends Start, Attach, or Defer; `fixer.RepairExecutor` submits and observes work through a host-supplied implementation. |
| 6 | + |
| 7 | +**Implementation status:** the [queue messages and topic keys](../../../stovepipe/core/messagequeue/messagequeue.go), [extension interfaces](../../../stovepipe/extension/fixer/fixer.go), and initial [entity](../../../stovepipe/entity/fix.go) and [storage](../../../stovepipe/extension/storage/fix_store.go) contracts are defined. The persistence layer will implement the layout below by keeping active IDs on the existing Queue and assignments in failure-episode records. `buildanalysis` and both fixer runtime controllers remain to be implemented. |
| 8 | + |
| 9 | +## 1. Persist the failure episode in buildanalysis |
| 10 | + |
| 11 | +After persisting observations, `buildanalysis.Controller.Process` creates or updates an episode identified by `(queue, check key, generation)` in the same delivery, then publishes [`FixRequest`](../../../stovepipe/core/messagequeue/proto/fix.proto) with that identity to `TopicKeyFix`. The same handoff applies when recovery or new evidence changes the episode. Persist before publishing; retry failed publication with the same message identity. |
| 12 | + |
| 13 | +### Required failure-episode contract |
| 14 | + |
| 15 | +The episode entity and store are part of the build-analysis implementation. They must support the following data and behavior; the field names below describe the required contract rather than an implemented Go type. |
| 16 | + |
| 17 | +| Field | Required content | |
| 18 | +| --- | --- | |
| 19 | +| `Queue` | Validation queue containing the check and its repairs. | |
| 20 | +| `Key` | Stable, opaque check identity, including its predicate and configuration. Preserve case and trailing spaces. | |
| 21 | +| `Generation` | Positive episode number. Recovery followed by another failure creates a new generation. Retain previous generations. | |
| 22 | +| `Failure.BuildID` | Build containing the first positive failure evidence for this episode. Preserve this initial reference as later observations arrive. | |
| 23 | +| `Failure.ObservationID` | Optional reference to persisted diagnostics within that build, when supplied by build analysis. | |
| 24 | +| `Failure.URI` | Optional retained revision for provenance or ordering after build archival. | |
| 25 | +| `Recovery` | Optional verified recovery evidence with a required validating `BuildID` and optional observation ID and revision URI. Absence means recovery is unverified. It can be recorded before a fix is assigned. | |
| 26 | +| `FixID` | Current assigned logical repair ID, scoped to the same queue. Empty means unassigned. This is the authoritative repair assignment. | |
| 27 | +| `Version` | Optimistic concurrency version covering evidence, recovery, and assignment. | |
| 28 | + |
| 29 | +The episode store must provide a primary-key read for any retained `(queue, key, generation)` and conditional updates that preserve unrelated fields. Build-analysis updates preserve `FixID`; fixer updates preserve failure and recovery evidence. Competing assignments use an expected version and expected current owner. Stale observations preserve verified recovery; recurrence creates a new generation. Diagnosis and root cause can emerge later and are resolved through extension-owned context. |
| 30 | + |
| 31 | +The episode reference matches [`entity.BreakageRef`](../../../stovepipe/entity/fix.go). The fixer stores that reference in `Fix.Breakages`; the build-analysis episode stores the assignment and evidence. This uses one episode record for both health and repair association. |
| 32 | + |
| 33 | +Retain enough observation provenance to verify report completeness, coverage, ordering, and configuration compatibility. Positive failures can trigger repair before the whole build completes. For failure-only reports, absence establishes recovery only after complete, comparable validation of that check. A missing or partial report leaves recovery unverified. Successful-check records are unnecessary in the CI artifact when coverage comes from the job configuration. |
| 34 | + |
| 35 | +For provenance, follow `Fix.Breakages` to the retained episode, then `Failure.BuildID` to [`Build.RequestID`](../../../stovepipe/entity/build.go) and [`Request.URI`](../../../stovepipe/entity/request.go). If builds or requests can be archived before episodes and repairs, the episode must retain the original request ID and revision needed for that lookup. |
| 36 | + |
| 37 | +Publish an updated `FixRequest` after recovery, assignment changes requiring follow-up, or evidence changes that warrant reevaluation. Reconcile missed or dead-lettered handoffs and revisit broken episodes that still need repair, so interrupted processing eventually resumes. |
| 38 | + |
| 39 | +## 2. Evaluate the episode against active repairs |
| 40 | + |
| 41 | +`fix.Controller.Process` reloads the failure episode. An unassigned, unrecovered episode proceeds to evaluation. An assigned episode resumes or signals its existing repair. Recovery before assignment resolves the pending request. |
| 42 | + |
| 43 | +Load the queue through [`QueueStore.Get`](../../../stovepipe/extension/storage/queue_store.go). Read each repair in `Queue.ActiveFixIDs` through [`FixStore.Get`](../../../stovepipe/extension/storage/fix_store.go), and identify compatible candidates. Compatibility covers the queue, tracked ref, configuration, and whether the repair can still cover another episode. A candidate-loading error leaves the request pending for normal retry. |
| 44 | + |
| 45 | +Call `fixer.RepairEvaluator.Evaluate` with the queue, episode reference, and candidate IDs. The implementation resolves diagnostics, investigation progress, and changes through injected dependencies, following the [`conflict.Analyzer`](../../../submitqueue/extension/conflict/conflict.go) pattern. |
| 46 | + |
| 47 | +| Decision | Meaning | Controller action | |
| 48 | +| --- | --- | --- | |
| 49 | +| `Start` | The evidence supports a new repair. | Persist a repair and its episode association, then submit it. | |
| 50 | +| `Attach` | An existing repair is expected to cover the episode. | Revalidate the returned fix ID and associate the episode with it. | |
| 51 | +| `Defer` | More evidence or investigation is needed. | Keep the request pending with `delivery.Hold(...)`. | |
| 52 | + |
| 53 | +For example, several checks can fail because an entity is missing one generated field. The evaluator can attach them to a repair adding that field, or defer while a plausible overlapping repair investigates. A bounded waiting policy escalates episodes that remain undecidable or permits independent work. |
| 54 | + |
| 55 | +## 3. Persist the repair and assignment |
| 56 | + |
| 57 | +Recheck the episode before applying a decision. An existing assignment takes precedence over a replayed evaluation; verified recovery resolves the pending request. |
| 58 | + |
| 59 | +For `Start`: |
| 60 | + |
| 61 | +1. Persist a queued `Fix` containing its initial `BreakageRef`. |
| 62 | +2. Add its ID to `Queue.ActiveFixIDs` through `QueueStore.Update`. |
| 63 | +3. Conditionally assign the episode's `FixID`. |
| 64 | +4. Confirm that the episode remains assigned to this repair and still needs repair before submission. |
| 65 | + |
| 66 | +Queue updates use the existing `Queue.Version`. On a version conflict, reload the queue and merge the membership change while preserving pipeline and enablement fields. Copy slices before changing membership. The version guards the queue update, not the entire evaluation. Concurrent evaluations can start overlapping repairs; occasional duplicate repair attempts are an accepted tradeoff. |
| 67 | + |
| 68 | +For `Attach`, validate the selected repair's state and eligibility, persist the episode reference on the repair, and conditionally assign the episode to that fix. If the selected repair closes or the episode changes, reevaluate. If a worker needs new evidence, use the host's membership-refresh protocol before relying on coverage. |
| 69 | + |
| 70 | +The episode's `FixID` is authoritative. `Fix.Breakages` supplies navigation from a repair to its episodes and may temporarily contain a candidate association that did not win. Reload episode assignments when observing a repair and finish interrupted writes through replay. If another assignment wins, resolve unused queued work and remove its active ID. A crash before active membership is persisted can leave an unused, unsubmitted repair record; build-analysis reconciliation retries the still-unassigned episode. Once the repair is listed, reconciliation can discover it by queue key and finish or resolve its initial association. |
| 71 | + |
| 72 | +## 4. Trigger the repair |
| 73 | + |
| 74 | +`fix.Controller.Process` calls `fixer.RepairExecutor.Submit` after persisting the repair, active membership, and episode assignment. This follows the trigger-and-monitor split used by `build` and `buildsignal`. Service wiring selects the evaluator and executor for each queue. |
| 75 | + |
| 76 | +Before enabling dispatch, refine durable execution state to retain attempt identity, submission intent, and submission outcome. Submission must distinguish accepted, definitely rejected before acceptance, and unknown outcomes; the initial error-only `Submit` contract needs to be refined accordingly. Persist attempt identity before calling the executor and observe accepted or ambiguous submissions through [`FixSignal`](../../../stovepipe/core/messagequeue/proto/fixsignal.proto). Replaying a submission depends on the executor's idempotency contract. Optional metadata describes provider context; it does not establish acceptance. |
| 77 | + |
| 78 | +## 5. Observe execution and check recovery |
| 79 | + |
| 80 | +`fixsignal.Controller.Process` reloads the repair and the failure episodes referenced by `Fix.Breakages`. It calls `fixer.RepairExecutor.Observe`, persists execution progress, and uses each currently assigned episode's recovery evidence to determine repair resolution. Use `delivery.Hold(...)` while execution or verification is pending. |
| 81 | + |
| 82 | +`ExecutionRef` is an opaque logical execution handle. Published change URIs, native run IDs, landed revisions, diagnostics, and code-review status belong in `Metadata`. Provider success advances execution progress; `buildanalysis` establishes recovery. A failed or ineffective attempt can be retried within the same episode. Record new attempts explicitly and prevent older observations from overwriting them. |
| 83 | + |
| 84 | +`buildanalysis` records recovery for the exact episode and publishes its `FixRequest`. The `fix` stage resolves an unassigned request or signals its assigned repair. Health recovery and execution completion remain separate: after all associated episodes recover, continue reconciliation until execution is settled or stopped. Retain the active ID while either verification or execution remains unresolved. Persist terminal repair state before removing the ID from `Queue.ActiveFixIDs`; a stale terminal entry is filtered during candidate discovery and removed by replay. |
| 85 | + |
| 86 | +## 6. Handle overlap discovered during execution |
| 87 | + |
| 88 | +A worker can discover overlap after repairs start. Persist a reassociation intent, then publish the affected episode's `FixRequest`. The `fix` stage validates expected assignment and repair versions, updates membership and the episode's `FixID`, and signals affected repairs. Preserve prior repair records and transfer only covered episodes when overlap is partial. |
| 89 | + |
| 90 | +Define how the worker acknowledges supersession and how publication ownership is enforced. Updating an episode assignment alone does not stop an external worker. Keep this protocol in the host integration. |
| 91 | + |
| 92 | +## Queue delivery and wiring |
| 93 | + |
| 94 | +| Topic | Message | Partition | Processing | |
| 95 | +| --- | --- | --- | --- | |
| 96 | +| `TopicKeyFix` (`fix`) | `FixRequest`: queue, check key, generation | Episode identity | Reload the failure episode; evaluate, persist assignment, and submit or signal a repair. | |
| 97 | +| `TopicKeyFixSignal` (`fixsignal`) | `FixSignal`: queue, fix ID | Fix ID | Reload the repair and failure episodes; observe execution and check recovery. | |
| 98 | + |
| 99 | +Register both controllers in [service wiring](../../../service/stovepipe/server/main.go), following [`build`](../../../stovepipe/controller/build/build.go) and [`buildsignal`](../../../stovepipe/controller/buildsignal/buildsignal.go). Each exposes `Process`, `Name`, `TopicKey`, and `ConsumerGroup` through `consumer.Controller`. |
| 100 | + |
| 101 | +Return `nil` on completion, propagate failures through normal error classification, and call `Hold` then return `nil` while waiting. Reload persisted state on every delivery. Use stable message identities for each durable transition; recovery and new attempts need distinct identities. Persist state before publishing a dependent message, and reconcile failed or dead-lettered handoffs. |
| 102 | + |
| 103 | +## Data and storage |
| 104 | + |
| 105 | +| Record | Primary key | Contents | |
| 106 | +| --- | --- | --- | |
| 107 | +| `Fix`, through `FixStore` | `(queue, fix ID)` | Episode references, execution handle, lifecycle, metadata, version, and timestamps. | |
| 108 | +| Existing `Queue`, through `QueueStore` | `(name)` | Existing pipeline coordination plus `ActiveFixIDs`, sharing `Queue.Version`. | |
| 109 | +| Failure episode, managed by buildanalysis | `(queue, check key, generation)` | Initial failure provenance, verified recovery, assigned `FixID`, and version. | |
| 110 | + |
| 111 | +All access remains primary-key-based. Candidate discovery reads `Queue.ActiveFixIDs` and then individual fixes. Keep the active set bounded; detailed history remains in repair and failure-episode records. A separate binding table and active-work store are unnecessary. |
| 112 | + |
| 113 | +`fix.metadata` follows the existing [request-log metadata pattern](../../../stovepipe/entity/request_log.go). Nil and empty metadata maps are equivalent and persist as `{}`. Nil and empty `Queue.ActiveFixIDs` both persist as `[]` and read as nil. Conditional updates replace complete snapshots. Every queue writer preserves the active IDs alongside existing coordination fields. Queue updates can conflict with ingestion or validation updates and retry using the latest full snapshot. |
| 114 | + |
| 115 | +For an existing database, add and backfill `queue.active_fix_ids` with an empty JSON array before running the new queue-store code. The persistence layer must also supply schemas for fresh databases. |
| 116 | + |
| 117 | +## Implementation |
| 118 | + |
| 119 | +1. **Implement persistence and episode tracking.** Implement `FixStore`, consolidate active repair IDs on `Queue`, and keep assignment in the build-analysis episode. Supply the episode data and primary-key operations listed above, preserve historical generations and provenance, and publish/reconcile `FixRequest` handoffs. |
| 120 | +2. **Complete durable execution state.** Define attempt identity, submission outcomes, and replayable reassociation before enabling dispatch. Refine `Fix` and executor results as needed. |
| 121 | +3. **Implement the queue stages.** Add `fix.Controller.Process` and `fixsignal.Controller.Process`, queue membership updates, episode assignment, bounded deferral, and reconciliation; register both consumers. |
| 122 | +4. **Implement extensions.** Supply `RepairEvaluator` and `RepairExecutor`, including evidence refresh, observation correlation, and overlap handling. Keep transport adapters in host integration. |
| 123 | +5. **Verify the lifecycle.** Cover concurrent membership updates, duplicate deliveries, interrupted associations, recovery before submission, candidate closure, ambiguous submission, provider retries, late overlap, partial recovery, and repeated failure episodes. |
0 commit comments