feat(stovepipe): define repair coordination contracts and RFC

Author: mnoah1

Makefile

@@ -629,6 +629,7 @@ GO_GENERATE_PACKAGES := \
 	./stovepipe/core/requestlog/... \
 	./stovepipe/extension/buildissue/... \
 	./stovepipe/extension/buildrunner/... \
+	./stovepipe/extension/fixer/... \
 	./stovepipe/extension/projectstatus/... \
 	./stovepipe/extension/queueconfig/... \
 	./stovepipe/extension/sourcecontrol/... \

doc/rfc/stovepipe/fixer.md

@@ -0,0 +1,50 @@
+# First-failure repair submission
+
+Build analysis records confirmed failures. The `fix` queue step selects the first failed target for each build and triggers one repair through a Go extension. Later failed targets remain recorded and their fix requests no-op. Submission can happen while the build is running. See the [workflow](workflow.md#workflow).
+
+## 1. Record the failure
+
+[`buildanalysis.Controller.Process`](../../../stovepipe/controller/buildanalysis/buildanalysis.go) obtains confirmed `BuildIssue{TaskID, UnitID}` observations from [`buildissue.Detector`](../../../stovepipe/extension/buildissue/buildissue.go). The detector owns provider confirmation rules.
+
+Persist the observation through [`BuildIssueStore`](../../../stovepipe/extension/storage/build_issue_store.go) and reconcile its [`OpenBuildIssue`](../../../stovepipe/entity/open_build_issue.go). Preserve the opening's `FirstObservedBuildID` and any `FixID` when updating observation fields. A reopened issue has a new opening build. Build IDs are opaque identities.
+
+Publish [`FixRequest`](../../../stovepipe/core/messagequeue/proto/fix.proto) with the queue, detecting BuildID, TaskID, UnitID, and FirstObservedBuildID. Partition by detecting BuildID. Replay publication after duplicate observation writes so interrupted handoffs can finish.
+
+## 2. Select one target for the build
+
+[`fix.Controller.Process`](../../../stovepipe/controller/fix/fix.go) reloads the Build and Request, checks queue identity, and verifies the persisted failure and issue opening.
+
+1. If `Build.FirstFixID` is empty, persist a queued [`Fix`](../../../stovepipe/entity/fix.go) and conditionally select it through the existing `Build.Version` guard.
+2. Keep the selected ID for the lifetime of the build. Requests for other targets acknowledge without submitting anything.
+3. A duplicate request for the selected target can finish preparation while the repair is still queued. Recheck the issue opening and persist its `FixID` while preserving observation fields.
+4. Mark an unselected competing candidate or a queued repair whose opening recovered as skipped. Preserve the build's selection.
+
+The first eligible request processed wins. Build analysis continues collecting all observations. Each build has its own selection; cross-build or shared-cause deduplication requires later decision logic.
+
+## 3. Trigger once
+
+[`RepairExecutor.Submit(ctx, Fix) (string, error)`](../../../stovepipe/extension/fixer/fixer.go) is the provider interface. Resolve the executor and finish local preparation before claiming dispatch.
+
+Conditionally persist `FixSubmitting` before making the outbound call. Only the delivery whose conditional write succeeds calls Submit. Redelivery after that marker acknowledges without another call, including after a timeout, process restart, or result-persistence failure.
+
+On success, store `FixSubmitted` and the nonempty opaque `ExecutionRef`. On an error or empty reference, store `FixSubmissionUnknown` and the diagnostic in metadata. A trigger error can follow acceptance, so it does not establish execution failure. A failed result write leaves the durable submitting marker and does not repeat the outbound call.
+
+This is at-most-once dispatch. A crash after the submitting marker and before the call may leave the repair unlaunched. A lost response leaves acceptance unknown. Both cases retain the build selection and require explicit follow-up rather than another automatic trigger. The provider's own execution behavior remains its responsibility.
+
+## 4. Retain provenance
+
+| Record | Key | Repair data |
+| --- | --- | --- |
+| Fix | queue/fix ID | Initiating BuildID, issue references, submission state, optional execution reference and metadata, version, timestamps. |
+| Existing Build | queue/build ID | Write-once `FirstFixID` under the existing snapshot version. |
+| OpenBuildIssue | queue/task/unit | Most recently selected `FixID` alongside the opening and observation state. |
+
+Navigate `Fix.BuildID -> Build.RequestID -> Request.URI` for the initiating change. `Fix.Issues` retains the opening identity even if another build later assigns its own repair to that opening. Build analysis owns failure and recovery evidence.
+
+The fixer adds one table and uses primary-key operations. Apply the schema additions and backfill the existing build/open-issue columns before deployment. Metadata contains optional provider details and submission diagnostics.
+
+## Integration and follow-up work
+
+[`core/fixrequest.Publish`](../../../stovepipe/core/fixrequest/publish.go) is the build-analysis handoff after observation and positive open-issue persistence. The build-analysis owner supplies that reconciliation and calls the helper, including on replay. The reference host retains its no-op detector until the integration lands.
+
+The initial implementation ends after recording the trigger result. A future `fixsignal` step can observe execution and correlate authoritative recovery. A future decision policy can attach related failures to an existing repair or start additional fixes for unrelated causes. Those features are separate from this one-shot submission flow.

doc/rfc/stovepipe/workflow.md

@@ -169,6 +169,8 @@ Per-stage design detail lives under `steps/` so this doc stays a pipeline overvi
 - [buildsignal.md](steps/buildsignal.md) — the poll loop: hold-based re-poll cadence, per-build partitioning, and the fail-closed handoff to record
 - [record.md](steps/record.md) — turning a terminal build outcome into an immutable validation fact, monotonic last-green advancement and ref promotion, and the hook event announcing the outcome. Project analysis already runs inline through `projectstatus.Resolver` and may later use results calculated by build analysis
 
+For repair decisions, execution, and recovery, see the [repair RFC](fixer.md).
+
 ## Dedup, idempotency, and history rewrites
 
 Ingestion is idempotent on `(Queue, head URI)`, so duplicate poller reports — and any future webhook producer racing the poller — converge on one Request. The pipeline persists the Queue's last-green URI and per-URI greenness, so it must tolerate a **history rewrite**: when `SourceControl` reports that the last-green URI is no longer an ancestor of the current head, `process` falls back to a full-monorepo build rather than trusting a baseline that no longer exists on the branch. The system converges to a correct greenness rather than wedging on a stale pointer.

stovepipe/core/messagequeue/README.md

@@ -11,3 +11,9 @@ Payloads are defined in proto3 (`proto/`, generated into `protopb/`) and seriali
 - **buildsignal** (`TopicKeyBuildSignal`, `BuildSignal`) — build publishes the build id here after triggering; buildsignal holds the delivery between polls until the build reaches a terminal status. Partitioned by build id, so each build's poll loop is an independent partition.
 - **buildanalysis** (`TopicKeyBuildAnalysis`, `BuildAnalysis`) — identifies a persisted build for issue detection, partitioned by build id. Build publishes a start notice and buildsignal publishes a completion notice with a distinct message id. See [build analysis](../../controller/buildanalysis/README.md) for polling and persistence behavior.
 - **record** (`TopicKeyRecord`, `Record`) — buildsignal publishes the request id after persisting the terminal outcome; record reloads the request and records its validation fact. Partitioned by request id.
+
+## Repair stage contract
+
+**fix** (`TopicKeyFix`, [`FixRequest`](proto/fix.proto)) identifies a persisted observation and issue opening using queue, detecting build, task, unit, and first-observed build. Partition by detecting build ID. The `consumer.Controller` selects the first failed target through `Build.FirstFixID`, persists a submitting marker, and calls `RepairExecutor.Submit` once. Later failed targets no-op.
+
+Persist observations and their current opening before publishing a request. Replay publication after duplicate observation writes so a crash between persistence and handoff can recover. Redelivery can complete preparation, but a durable submitting marker suppresses any further outbound call. Execution monitoring and verified recovery are follow-up work.

stovepipe/core/messagequeue/messagequeue.go

@@ -56,6 +56,9 @@ type (
 	// Record is the payload buildsignal publishes to the record stage once a
 	// build reaches a terminal status: the build id to record.
 	Record = protopb.Record
+
+	// FixRequest identifies a confirmed issue opening and the detecting build.
+	FixRequest = protopb.FixRequest
 )
 
 // marshalOpts keeps the JSON field names identical to the proto field names

stovepipe/core/messagequeue/messagequeue_test.go

@@ -66,6 +66,40 @@ func TestRecordRoundTrip(t *testing.T) {
 	assert.True(t, proto.Equal(rec, got), "round-tripped Record should equal the original")
 }
 
+func TestFixMessagesRoundTrip(t *testing.T) {
+	for _, tt := range []struct {
+		name    string
+		message proto.Message
+	}{
+		{
+			name: "issue request",
+			message: &FixRequest{
+				QueueName: "repo/main",
+				BuildId:   "build-42", TaskId: "compile", UnitId: "package", FirstObservedBuildId: "build-41",
+			},
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			data, err := Marshal(tt.message)
+			require.NoError(t, err)
+			got := tt.message.ProtoReflect().New().Interface()
+			require.NoError(t, Unmarshal(data, got))
+			assert.True(t, proto.Equal(tt.message, got))
+		})
+	}
+}
+
+func TestFixRequestWireFormat(t *testing.T) {
+	message := &FixRequest{QueueName: "repo/main", BuildId: "build-42", TaskId: "check-a", FirstObservedBuildId: "build-41"}
+	data, err := Marshal(message)
+	require.NoError(t, err)
+	assert.JSONEq(t, `{"queue_name":"repo/main","build_id":"build-42","task_id":"check-a","first_observed_build_id":"build-41"}`, string(data))
+
+	var got FixRequest
+	require.NoError(t, Unmarshal([]byte(`{"queue_name":"repo/main","build_id":"build-42","task_id":"check-a","first_observed_build_id":"build-41","future_field":true}`), &got))
+	assert.True(t, proto.Equal(message, &got))
+}
+
 // TestWireFormat locks the protojson encoding decision the contract relies on:
 // snake_case field names (UseProtoNames).
 func TestWireFormat(t *testing.T) {
@@ -80,7 +114,7 @@ func TestWireFormat(t *testing.T) {
 // no topic_keys option names an unknown key.
 func TestTopicKeysBindEveryTopicKey(t *testing.T) {
 	bound := map[string]int{}
-	for _, m := range []proto.Message{&ProcessRequest{}, &BuildRequest{}, &BuildSignal{}, &BuildAnalysis{}, &Record{}} {
+	for _, m := range []proto.Message{&ProcessRequest{}, &BuildRequest{}, &BuildSignal{}, &BuildAnalysis{}, &Record{}, &FixRequest{}} {
 		keys := TopicKeys(m)
 		require.NotEmpty(t, keys, "message must declare a non-empty topic_keys option")
 		for _, key := range keys {
@@ -94,6 +128,7 @@ func TestTopicKeysBindEveryTopicKey(t *testing.T) {
 		TopicKeyBuildSignal,
 		TopicKeyBuildAnalysis,
 		TopicKeyRecord,
+		TopicKeyFix,
 	}
 
 	valid := map[string]bool{}

stovepipe/core/messagequeue/proto/BUILD.bazel

@@ -3,6 +3,7 @@ exports_files(
         "build.proto",
         "buildanalysis.proto",
         "buildsignal.proto",
+        "fix.proto",
         "process.proto",
         "record.proto",
     ],

stovepipe/core/messagequeue/proto/fix.proto

@@ -0,0 +1,36 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";
+
+package uber.stovepipe.messagequeue;
+
+import "api/base/messagequeue/proto/messagequeue.proto";
+
+option go_package = "github.com/uber/submitqueue/stovepipe/core/messagequeue/protopb";
+option java_multiple_files = true;
+option java_outer_classname = "FixRequestProto";
+option java_package = "com.uber.submitqueue.stovepipe.messagequeue";
+
+// FixRequest identifies a confirmed issue opening and the build requesting repair.
+// The observation and open issue must be persisted before publication.
+message FixRequest {
+    option (uber.base.messagequeue.topic_keys) = "fix";
+
+    string queue_name = 1;
+    string build_id = 2;
+    string task_id = 3;
+    string unit_id = 4;
+    string first_observed_build_id = 5;
+}

stovepipe/core/messagequeue/protopb/BUILD.bazel

@@ -6,6 +6,7 @@ go_library(
         "build.pb.go",
         "buildanalysis.pb.go",
         "buildsignal.pb.go",
+        "fix.pb.go",
         "process.pb.go",
         "record.pb.go",
     ],