Skip to content

feat(messagequeue): shard MySQL queue tables by tenant - #681

Open
behinddwalls wants to merge 4 commits into
mainfrom
preetam/mq-tenant-shard
Open

feat(messagequeue): shard MySQL queue tables by tenant#681
behinddwalls wants to merge 4 commits into
mainfrom
preetam/mq-tenant-shard

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Why?

The MySQL message queue keyed every table on topic and partition key. That cannot serve as a Vitess vindex because SubmitQueue uses partition keys for ordering units such as request and build IDs; one business queue would therefore scatter across shards and share shard capacity with unrelated queues.

What?

  • Add a tenant shard identity to every MySQL MQ table, primary key, store query, message, and admin operation while preserving partition_key as the ordering unit.
  • Use readable ASCII VARCHAR fields for tenant, topic, and consumer identifiers and explicit utf8mb4 VARCHAR fields for partition keys and message IDs, keeping composite indexes under InnoDB's 3072-byte limit.
  • Publish through one Message function with named MessageParams fields; tenant is required and mirrored into queue_name delivery metadata.
  • Discover partitions from an authoritative MQ_TENANTS list parsed by shared service wiring; isolate discovery, lease, heartbeat, and shutdown failures per tenant.
  • Map SubmitQueue queue names to tenants at publish/wiring boundaries and reject Stovepipe ingest for unconfigured tenants before side effects.
  • Add schema checks, shard-linter coverage, and cross-tenant ACK/GC and DLQ isolation tests.

Test Plan

✅ Platform publish and hook tests
✅ 28 focused controller and wiring test suites
✅ ./tool/bazel test //platform/extension/messagequeue/mysql/... //tool/linter/queueshard/... --test_output=errors
✅ ./tool/bazel test //test/integration/extension/messagequeue/mysql:go_default_test --test_output=errors
✅ ./tool/bazel build //...
✅ make fmt && make gazelle

@behinddwalls behinddwalls mentioned this pull request Sep 7, 2026
Open
@behinddwalls
feat(messagequeue): shard MySQL queue tables by tenant
c3c5b16 
Vitess needs a stable vindex that is not the Kafka-style partition key. Prefix every queue table and store query with tenant, discover partitions from the configured tenant list, and map SubmitQueue queueName onto tenant at wiring so one busy queue cannot scatter across shards.

## Summary

### Why?

The MySQL message queue keyed every table on (topic, partition_key). That cannot be a Vitess vindex: SubmitQueue already uses partition_key for ordering (for example a build ID on build-signal), so hashing it would scatter one queue across shards and still mix tenants on a given shard. Isolation has to be an explicit tenant column that SubmitQueue maps from queueName at the wiring boundary.

### What?

- Add Message.Tenant and require it on publish; platform/publish stamps it from queue_name metadata or context.
- Rewrite the five queue schemas so PKs lead with tenant; drop cross-tenant indexes; keep KEY idx_offset for InnoDB AUTO_INCREMENT.
- Drive subscriber discovery, leases, heartbeats, and fair-share from Params.Tenants instead of SELECT DISTINCT partition_key.
- Wire tenants from SubmitQueue/Runway queue config and Stovepipe MQ_TENANTS.
- Require --tenant on topic-scoped admin commands; teach queueshard the MQ schema root.

## Test Plan

✅ `./tool/bazel test //platform/extension/messagequeue/... //tool/linter/queueshard/...`

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	platform/extension/messagequeue/mysql/schema/queue_messages.sql
#	service/runway/server/config.go
#	test/integration/extension/messagequeue/mysql/queue_test.go

# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto c9ee7a8
# Last command done (1 command done):
#    pick 8940d26 # feat(messagequeue): shard MySQL queue tables by tenant
# Next command to do (1 remaining command):
#    pick c79649f # fix(messagequeue): harden tenant shard isolation
# You are currently rebasing branch 'preetam/mq-tenant-shard' on 'c9ee7a84'.
#
# Changes to be committed:
#	modified:   doc/rfc/index.md
#	new file:   doc/rfc/messagequeue-tenant-sharding.md
#	modified:   platform/base/messagequeue/message.go
#	modified:   platform/extension/messagequeue/mysql/BUILD.bazel
#	modified:   platform/extension/messagequeue/mysql/constants.go
#	modified:   platform/extension/messagequeue/mysql/ctl/commands.go
#	modified:   platform/extension/messagequeue/mysql/ctl/lib/admin.go
#	modified:   platform/extension/messagequeue/mysql/ctl/lib/admin_test.go
#	modified:   platform/extension/messagequeue/mysql/delivery_state_store.go
#	modified:   platform/extension/messagequeue/mysql/delivery_state_store_test.go
#	modified:   platform/extension/messagequeue/mysql/message_store.go
#	modified:   platform/extension/messagequeue/mysql/message_store_test.go
#	modified:   platform/extension/messagequeue/mysql/mock_stores.go
#	modified:   platform/extension/messagequeue/mysql/offset_store.go
#	modified:   platform/extension/messagequeue/mysql/offset_store_test.go
#	modified:   platform/extension/messagequeue/mysql/partition_lease_store.go
#	modified:   platform/extension/messagequeue/mysql/partition_lease_store_test.go
#	modified:   platform/extension/messagequeue/mysql/publisher.go
#	modified:   platform/extension/messagequeue/mysql/publisher_test.go
#	modified:   platform/extension/messagequeue/mysql/schema/queue_delivery_state.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_messages.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_offsets.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_partition_leases.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_subscriber_heartbeats.sql
#	modified:   platform/extension/messagequeue/mysql/sql.go
#	modified:   platform/extension/messagequeue/mysql/stores.go
#	modified:   platform/extension/messagequeue/mysql/subscriber.go
#	modified:   platform/extension/messagequeue/mysql/subscriber_heartbeat_store.go
#	modified:   platform/extension/messagequeue/mysql/subscriber_heartbeat_store_test.go
#	modified:   platform/extension/messagequeue/mysql/subscriber_test.go
#	new file:   platform/extension/messagequeue/mysql/tenants.go
#	modified:   platform/publish/publish.go
#	modified:   platform/publish/publish_test.go
#	modified:   service/runway/server/config.go
#	modified:   service/runway/server/main.go
#	modified:   service/stovepipe/docker-compose.yml
#	modified:   service/stovepipe/server/main.go
#	modified:   service/submitqueue/gateway/server/main.go
#	modified:   service/submitqueue/orchestrator/server/main.go
#	modified:   test/e2e/runway/harness_test.go
#	modified:   test/e2e/runway/suite_test.go
#	modified:   test/e2e/submitqueue/BUILD.bazel
#	modified:   test/e2e/submitqueue/harness_test.go
#	modified:   test/integration/extension/messagequeue/mysql/queue_test.go
#	modified:   test/integration/submitqueue/core/consumer/consumer_test.go
#	modified:   test/integration/submitqueue/gateway/BUILD.bazel
#	modified:   test/integration/submitqueue/gateway/suite_test.go
#	modified:   tool/linter/queueshard/main.go
#	modified:   tool/linter/queueshard/main_test.go
#
@behinddwalls
fix(messagequeue): harden tenant shard isolation
718d045 
## Summary

### Why?

The initial tenant-sharding implementation could exceed InnoDB's composite-key limit, infer incomplete tenant lists from extension overrides, and let one tenant's storage failure block unrelated tenants. Stovepipe could also accept work for a tenant its subscribers would never discover.

### What?

- Use readable ASCII VARCHAR identifiers for tenants, topics, and consumer names while preserving utf8mb4 partition keys and message IDs.
- Validate backend identifier contracts before database access and require an authoritative MQ_TENANTS list for consumer services.
- Reject unconfigured Stovepipe queues before side effects.
- Isolate discovery and shutdown failures per tenant and aggregate errors after all tenants are attempted.
- Narrow the queueshard AUTO_INCREMENT exception and add schema, cross-tenant ACK/GC, and DLQ isolation coverage.

## Test Plan

✅ `./tool/bazel test //platform/extension/messagequeue/mysql/... //tool/linter/queueshard/... --test_output=errors`
✅ `./tool/bazel test //test/integration/extension/messagequeue/mysql:go_default_test --test_output=errors`
✅ `./tool/bazel build //service/runway/server/... //service/stovepipe/server/... //service/submitqueue/orchestrator/server/...`
✅ `make fmt && make gazelle`

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	service/runway/server/config.go
#	service/runway/server/main.go

# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto c9ee7a8
# Last commands done (2 commands done):
#    pick 8940d26 # feat(messagequeue): shard MySQL queue tables by tenant
#    pick c79649f # fix(messagequeue): harden tenant shard isolation
# No commands remaining.
# You are currently rebasing branch 'preetam/mq-tenant-shard' on 'c9ee7a84'.
#
# Changes to be committed:
#	modified:   doc/rfc/messagequeue-tenant-sharding.md
#	modified:   platform/extension/messagequeue/mysql/BUILD.bazel
#	new file:   platform/extension/messagequeue/mysql/identifier.go
#	modified:   platform/extension/messagequeue/mysql/publisher.go
#	modified:   platform/extension/messagequeue/mysql/publisher_test.go
#	modified:   platform/extension/messagequeue/mysql/schema/queue_delivery_state.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_messages.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_offsets.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_partition_leases.sql
#	modified:   platform/extension/messagequeue/mysql/schema/queue_subscriber_heartbeats.sql
#	modified:   platform/extension/messagequeue/mysql/subscriber.go
#	modified:   platform/extension/messagequeue/mysql/subscriber_test.go
#	modified:   platform/extension/messagequeue/mysql/tenants.go
#	new file:   platform/extension/messagequeue/mysql/tenants_test.go
#	modified:   service/runway/server/config.go
#	modified:   service/runway/server/docker-compose.yml
#	modified:   service/runway/server/main.go
#	modified:   service/stovepipe/server/main.go
#	modified:   service/submitqueue/docker-compose.yml
#	modified:   service/submitqueue/orchestrator/server/docker-compose.yml
#	modified:   service/submitqueue/orchestrator/server/main.go
#	modified:   stovepipe/controller/ingest.go
#	modified:   stovepipe/controller/ingest_test.go
#	modified:   test/integration/extension/messagequeue/mysql/BUILD.bazel
#	modified:   test/integration/extension/messagequeue/mysql/queue_test.go
#	new file:   test/integration/extension/messagequeue/mysql/tenant_isolation_test.go
#	modified:   tool/linter/queueshard/main.go
#	modified:   tool/linter/queueshard/main_test.go
#
@behinddwalls
behinddwalls force-pushed the preetam/mq-tenant-shard branch from c79649f to 718d045 Compare September 7, 2026 19:48
@behinddwalls
behinddwalls changed the base branch from land-terminology to main September 7, 2026 19:48
behinddwalls
behinddwalls commented Sep 7, 2026
View reviewed changes
Comment thread platform/extension/messagequeue/mysql/tenants.go Outdated
behinddwalls
behinddwalls commented Sep 7, 2026
View reviewed changes
Comment thread platform/publish/publish.go Outdated
@behinddwalls @cursoragent
refactor(messagequeue): make publish tenant explicit
6a6bf07 
## Summary

### Why?

Tenant selects the physical MQ shard and is therefore a first-class publish input, not metadata to infer implicitly. Parsing MQ_TENANTS is service wiring behavior rather than a responsibility of the MySQL backend.

### What?

- Require tenant explicitly in platform publish and hook APIs, propagate it to queue_name metadata, and reject conflicting metadata.
- Pass each producer's authoritative queue through SubmitQueue, Runway, Stovepipe, hooks, and tests independently from partition keys.
- Move required-tenant parsing into a shared service/messagequeue wiring package.

## Test Plan

✅ Focused unit tests across publish, hook, service wiring, Runway, Stovepipe, and SubmitQueue controllers
✅ `./tool/bazel build //...`
✅ `make fmt && make gazelle`

Co-authored-by: Cursor <cursoragent@cursor.com>
behinddwalls
behinddwalls commented Sep 7, 2026
View reviewed changes
Comment thread platform/publish/publish.go Outdated
@behinddwalls @cursoragent
refactor(messagequeue): consolidate publish arguments
cfce51e 
## Summary

### Why?

Adding tenant to the positional publish API made future message attributes likely to trigger another repository-wide signature migration and left metadata as a separate method variant.

### What?

Replace Message and MessageWithMetadata with one Message function taking documented MessageParams. Named fields keep tenant, message identity, payload, partitioning, and optional metadata explicit at every call site while allowing additive evolution.

## Test Plan

✅ Platform publish and hook tests
✅ 28 focused controller and wiring test suites
✅ `./tool/bazel build //...`
✅ `make fmt && make gazelle`

Co-authored-by: Cursor <cursoragent@cursor.com>
@behinddwalls
behinddwalls marked this pull request as ready for review September 7, 2026 20:34
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners September 7, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sbalabanov sbalabanov Awaiting requested review from sbalabanov sbalabanov is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@behinddwalls