add persistent experiment administration (#4776) by mariusae · Pull Request #4776 · meta-pytorch/monarch

mariusae · 2026-09-02T15:39:30Z

Summary:

Let any process submit, inspect, and target persistent scale experiments through replicated SQLite state. Nodes claim work addressed to their PID, run echo or streaming-transfer measurements after warmup, and publish status, timing, and failure details for remote inspection.

Walkthrough: admin.rs implements experiment creation, targeted source/destination selection, node listing, and result inspection over a remote Chrysalis SQLite endpoint. main.rs exposes those commands alongside the runner, and the README walks through MAST launch, topology, experiment submission, bandwidth tests, and result collection.

Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba

Walkthrough document: https://mdoc.internalmeta.com/doc/1dbe7d3b-7aab-48ad-be8b-01bd98b9e9a7

Reviewed By: shayne-fletcher

Differential Revision: D117242783

meta-codesync · 2026-09-02T15:40:15Z

@mariusae has exported this pull request. If you are a Meta employee, you can view the originating Diff in D117242783.

Summary: Pull Request resolved: meta-pytorch#4776 Let any process submit, inspect, and target persistent scale experiments through replicated SQLite state. Nodes claim work addressed to their PID, run echo or streaming-transfer measurements after warmup, and publish status, timing, and failure details for remote inspection. Walkthrough: `admin.rs` implements experiment creation, targeted source/destination selection, node listing, and result inspection over a remote Chrysalis SQLite endpoint. `main.rs` exposes those commands alongside the runner, and the README walks through MAST launch, topology, experiment submission, bandwidth tests, and result collection. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1dbe7d3b-7aab-48ad-be8b-01bd98b9e9a7 Reviewed By: shayne-fletcher Differential Revision: D117242783

Summary: Pull Request resolved: meta-pytorch#4772 Introduce the dependency-free wire types at the base of Chrysalis and document the architecture and review sequence. PIDs name process incarnations, while fixed-width QUIC connection IDs embed the destination PID and a process-local demultiplexing suffix. Walkthrough: Start in `chrysalis-core`, where `Pid` defines process identity and `ConnectionId` defines the routable QUIC wire layout. Then read `chrysalis/DESIGN.md` for the complete model and `chrysalis/WALKTHROUGH.md` for the order in which the stack assembles it. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/58f98269-78c5-42b3-a1a4-6e9a7a1adaf2 Reviewed By: shayne-fletcher Differential Revision: D117242759

Summary: Pull Request resolved: meta-pytorch#4773 Add the carrier layer that moves atomic datagrams without interpreting QUIC. A common socket contract supports UDP, Unix datagrams, and in-process channels, while `DatagramSocketSet` selects a carrier from an opaque destination address. Walkthrough: Read `datagram.rs` for the transport-independent address, receive, and transmit contracts. Then follow the concrete implementations in `udp.rs`, `unix.rs`, and `inproc.rs`; `socket_set.rs` composes those carriers behind one socket and `shutdown.rs` provides structured teardown. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/aebf6303-aaad-470c-ae40-e0a0f60550bf Reviewed By: shayne-fletcher Differential Revision: D117242760

Summary: Pull Request resolved: meta-pytorch#4774 Route datagrams by the destination PID embedded in each QUIC connection ID. Routes reference lifecycle gates, so expiring one gate fences every derived destination without synchronously deleting a subtree of forwarding entries. Walkthrough: Begin with `route.rs`: `RouteGate` supplies the shared fence, `Route` binds a destination to a next hop and gate, and `Router` performs forwarding. `switch.rs` then demultiplexes local PIDs and delegates every non-local datagram to that router without inspecting QUIC streams. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/15fc8742-10cf-4dbd-af8d-70a218e2d8c7 Reviewed By: shayne-fletcher Differential Revision: D117242761

Summary: Pull Request resolved: meta-pytorch#4775 Add the ownership and completion model shared by transport drivers and runtime adapters. Bounded ingress, completion credits, ordered submissions, operation IDs, and explicit wakeups make resource retention and backpressure part of the API without depending on Tokio or a QUIC implementation. Walkthrough: `command.rs` and `submission.rs` define accepted work and ownership transfer. `completion.rs` defines every resource-return and lifecycle event. `credit.rs`, `queue.rs`, and `ingress.rs` enforce bounded admission, while `id.rs` and `notify.rs` provide driver-scoped identity and runtime-neutral wakeups. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/a39624e5-45e3-4696-8d96-77c9e2f95eb6 Reviewed By: shayne-fletcher Differential Revision: D117242762

Summary: Add a runtime-neutral UDP engine that owns stable packet slots on a dedicated driver thread. The engine submits batched receive and transmit operations through io_uring and exposes the packet-I/O boundary needed by the QUIC state machine. Walkthrough: `config.rs` defines bounded ring and buffer sizing. `socket.rs` transfers a bound UDP socket into the engine. `driver.rs` owns the ring, receive and transmit slot lifecycles, GSO submissions, GRO splitting, completion processing, and structured shutdown. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/96ad08dc-3b3a-4fb4-9c4f-59d5a9415577 Reviewed By: shayne-fletcher Differential Revision: D117242763

Summary: Pull Request resolved: meta-pytorch#4776 Let any process submit, inspect, and target persistent scale experiments through replicated SQLite state. Nodes claim work addressed to their PID, run echo or streaming-transfer measurements after warmup, and publish status, timing, and failure details for remote inspection. Walkthrough: `admin.rs` implements experiment creation, targeted source/destination selection, node listing, and result inspection over a remote Chrysalis SQLite endpoint. `main.rs` exposes those commands alongside the runner, and the README walks through MAST launch, topology, experiment submission, bandwidth tests, and result collection. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1dbe7d3b-7aab-48ad-be8b-01bd98b9e9a7 Reviewed By: shayne-fletcher Differential Revision: D117242783

Summary: Define the final identity, admission, buffer-ownership, command, and packet-I/O boundaries used by the runtime-neutral quiche endpoint. The crate binds quiche to Chrysalis CIDs and completion transport without introducing an async runtime. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/6ae92406-f8be-46ae-8b73-6d4b869276c2 Reviewed By: shayne-fletcher Differential Revision: D117242765

Summary: Implement the final runtime-neutral quiche endpoint over bounded completion transport and packet slots. The driver authenticates certificate-derived PIDs, routes fixed CIDs directly, bounds server admission and retained completions, schedules only runnable streams, preserves directional operation order, returns ownership on cancellation, and reclaims terminal state. Document the end-to-end control and data lifecycle from connection creation through stream send, stream closure, and connection reclamation. Walkthrough: `driver.rs` owns endpoint admission, completion credits, shutdown, packet-loop orchestration, and the lifecycle overview. `driver/network.rs` owns CID routing, Retry, connection admission, packet generation, and lifecycle statistics. `driver/connection.rs` owns authenticated connection state plus ordered send and receive queues, runnable-stream scheduling, cancellation, half-state transitions, and reclamation. `driver/tests.rs` exercises handshake, multiplexing, backpressure, shutdown, authentication, cancellation, and stream control. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/c932f625-1719-4ce0-9502-6074ffcb7f68 Reviewed By: shayne-fletcher Differential Revision: D117242766

Summary: Pull Request resolved: meta-pytorch#4766 Translate the completion-driven Chrysalis transport into ergonomic Tokio connections and streams without moving protocol ownership into the runtime. The adapter preserves bounded admission, caller buffer ownership, operation cancellation, and structured driver shutdown. Name every transport constructor `spawn_client`, `spawn_server`, `spawn_duplex`, or `spawn_duplex_routed` so the API makes creation of the driver thread and completion pump explicit. Walkthrough: `Transport::spawn_*` constructs the low-level endpoint, starts its dedicated driver thread, and starts the Tokio completion pump. Connection and stream handles translate command receipts and operation completions into Tokio futures while cancellation remains an explicit message to the runtime-neutral driver. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1c2c7a60-06fb-493d-9ce2-96f18e099b75 Reviewed By: shayne-fletcher Differential Revision: D117242767

Summary: Compose routed datagram sockets, direct io_uring UDP, pooled QUIC connections, and link-local stream protocols behind one carrier-neutral transport. Application and control streams share physical carriers while retaining distinct process routing identities. Document the synthetic `SocketAddr` projection that lets unmodified quiche identify paths over opaque carrier addresses, including its stable bidirectional mapping and the fork that removing it would otherwise require. Walkthrough: `packet_io.rs` adapts switched datagrams to the quiche packet-slot contract. `quic_transport.rs` owns endpoint tasks, connection pooling, dialing, and accepted streams across routed and direct UDP carriers. `link_local.rs` multiplexes registered control protocols over reserved streams, and `lib.rs` exposes the composed transport surface. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/c70ac010-2206-4e75-b662-cee3411ff5a8 Reviewed By: shayne-fletcher Differential Revision: D117242768

Summary: Define the versioned delegated-namespace messages and a bounded deterministic frame codec. The protocol covers publication snapshots and deltas, cumulative acknowledgements, resolution, enumeration, rejection, and link-scoped identity. Walkthrough: `protocol.rs` defines typed identifiers, locators, process entries, consistency modes, and every message variant. `codec.rs` assigns the wire tags, enforces frame and collection bounds, and encodes or decodes each field without depending on session or network tasks. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1d9e8e3d-57d2-443e-b9ae-0bc87a88ff30 Reviewed By: shayne-fletcher Differential Revision: D117242769

Summary: Implement the authoritative delegated namespace and parent-scoped resolver cache as deterministic state machines. Publication snapshots, contiguous deltas, ownership, revisions, replay, and negative-cache expiry can now be tested without transport timing. Walkthrough: `state.rs` validates commands, stages snapshots, applies atomic directory changes, assigns revisions, and emits cumulative link responses. `cache.rs` accepts ordered resolution updates from one authority and retains positive or expiring negative results without extending replayed deadlines. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/f890ab4b-ac9f-423a-bb41-5e818abc6dd2 Reviewed By: shayne-fletcher Differential Revision: D117242770

Summary: Connect the deterministic nameserver protocol to ordered framed streams. Parent sessions negotiate versions, preserve request ordering, translate committed state effects into acknowledgements, correlate resolution and enumeration responses, and bind learned state to one link lifetime. Walkthrough: `stream.rs` performs bounded asynchronous frame reads and writes over Chrysalis streams. `session.rs` owns handshake state, pending request correlation, committed-command callbacks, publication responses, enumeration, and terminal protocol errors without owning socket tasks. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/9a1ade49-3a74-4c22-825f-62e8cb812987 Reviewed By: shayne-fletcher Differential Revision: D117242771

Summary: Export one complete child process subtree through a link-scoped publication session. The child sends chunked snapshots followed by contiguous deltas, advances only through cumulative acknowledgements, and resnapshots when the parent detects a gap. Walkthrough: `ChildSession` owns publication sequence, snapshot incarnation, chunking, pending changes, and request correlation. Its event interface separates deterministic protocol progress from the task that carries messages and supplies the current subtree. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/60c62871-5eb2-48da-aa2c-3734a5d78ae2 Reviewed By: shayne-fletcher Differential Revision: D117242772

Summary: Compose deterministic namespace state with framed parent and child tasks, gated route projection, cached resolution, paginated enumeration, and supervised parent reconnection. Expiring one link gate immediately fences every route learned from that subtree while asynchronous cleanup proceeds. Walkthrough: `service.rs` serializes authoritative commands and queries. `routes.rs` projects directory changes into shared-gate router entries. `link.rs` drives one parent publication and resolution session, `server.rs` accepts child links, `upstream.rs` presents the active parent, and `manager.rs` supervises configuration, reconnect, status, resolution, and enumeration. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/650999f8-d5fd-4927-93bb-0f22b18c437a Reviewed By: shayne-fletcher Differential Revision: D117242773

Summary: Pull Request resolved: meta-pytorch#4765 Present identity, carriers, transport, nameserver state, parent linkage, and registered link-local protocols through one embeddable `Node`. The facade owns structured startup and shutdown while exposing a small serve, join, dial, accept, enumerate, and protocol API. Walkthrough: `node.rs` composes transport and namespace services and supervises their lifetimes. `link_protocol.rs` registers singleton link-local handlers and starts them for every live link. `socket.rs` adapts accepted Chrysalis streams to Tokio I/O, while `lib.rs` exposes the public embedding surface. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/94be4b47-26d5-4988-a16b-3c91a0d8a068 Reviewed By: shayne-fletcher Differential Revision: D117242774

Summary: Obtain Meta mTLS material and issue a distinct leaf certificate for each Chrysalis process. Certificate-derived PIDs remain self-certifying while the core transport and Node crates stay independent of Meta identity infrastructure. Walkthrough: The provider loads the host identity chain, creates process-specific key material, signs a constrained leaf certificate, validates the resulting chain, and returns the certificate, key, and derived endpoint identity consumed by Chrysalis. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/d06242da-b4c6-40e8-9ef1-71a3232d5884 Reviewed By: shayne-fletcher Differential Revision: D117242775

Summary: Replace the cr-sqlite extension plan with a Chrysalis-local replication boundary built entirely on ordinary SQLite. Applications register trusted table descriptors and explicitly capture full-row upserts or deletes in the same transaction as their domain writes. The design keeps site IDs, logical versions, mutation records, row winners, and peer frontiers in normal `__chrysalis_*` tables. Peers exchange schema digests only; they never supply executable SQL. Concurrent writes converge with deterministic row-level last-writer-wins ordering on `(version, site ID)`. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/fa07097b-024d-4e75-8cd8-15f215de8b72 Reviewed By: shayne-fletcher Differential Revision: D117242777

Summary: Implement Chrysalis-local SQLite replication without a native extension. Trusted local table descriptors initialize application tables, atomic explicit captures append full-row upserts or deletes, and durable site/version metadata supports deterministic row-level last-writer-wins convergence. The version-five stream protocol exchanges schema digests rather than executable SQL, preserves origin identities while relaying mutations, filters by topology scope, bounds batches, and advances durable per-peer frontiers only after acknowledgement. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/238b8581-9ba5-437c-8aee-503392f793e0 Reviewed By: shayne-fletcher Differential Revision: D117242778

Summary: Run bidirectional SQLite synchronization on every adjacent Chrysalis link. Durable per-origin frontiers and incremental origin discovery prevent loops and preserve progress when subtree membership changes without assuming that relayed sites share one scalar database version. Walkthrough: `topology.rs` registers the reserved SQLite link protocol with `Node`, advertises each child subtree explicitly, advertises the complement toward children, and republishes aggregate membership upstream. The README follows scope changes, durable frontiers, origin discovery, schema propagation, and operational usage. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/8625eb6f-b458-4863-b259-db69d2a218e0 Reviewed By: shayne-fletcher Differential Revision: D117242779

Summary: Resolve direct addresses and `mast://` names into join settings, then expose serving, process enumeration, stream piping, and local SQLite inspection through the public Node API. The SQLite shell uses ordinary libSQL directly and carries no native extension resource. Walkthrough: `chrysalis-resolver` parses deployment references and resolves MAST tasks. In `chrysalis-cli`, `address.rs` normalizes carrier and join syntax, `identity.rs` selects identity providers, `socket.rs` bridges streams, `sqlite_shell.rs` owns local database commands, and `main.rs` assembles the command surface. Replicated applications use the typed `chrysalis-sqlite` API instead of attempting to infer mutations from arbitrary shell SQL. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/7ace3758-d098-4a0c-9837-e49b902c9cee Reviewed By: shayne-fletcher Differential Revision: D117242780

Summary: Launch many real Chrysalis processes in flat or task-head topologies, record lineage, wait for namespace convergence, exercise fresh streams, and run direct network baselines. Only task heads require UDP; local leaves use Unix datagrams under one supervisor. Walkthrough: `benchmark.rs` owns process topology, worker supervision, convergence, echo, throughput, and experiment execution. `persist.rs` defines trusted replicated table schemas and captures node, experiment, target, claim, and result rows atomically through the local SQLite mutation log. `network_baseline.rs` isolates carrier and QUIC measurements, while `mast.rs` packages only the Rust binary—there is no SQLite extension artifact. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/4e601137-b930-44b0-914f-6ca3b85705b0 Reviewed By: shayne-fletcher Differential Revision: D117242781

Summary: Pull Request resolved: meta-pytorch#4776 Let any process submit, inspect, and target persistent scale experiments through replicated SQLite state. Nodes claim work addressed to their PID, run echo or streaming-transfer measurements after warmup, and publish status, timing, and failure details for remote inspection. Walkthrough: `admin.rs` implements experiment creation, targeted source/destination selection, node listing, and result inspection over a remote Chrysalis SQLite endpoint. `main.rs` exposes those commands alongside the runner, and the README walks through MAST launch, topology, experiment submission, bandwidth tests, and result collection. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1dbe7d3b-7aab-48ad-be8b-01bd98b9e9a7 Reviewed By: shayne-fletcher Differential Revision: D117242783

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 2, 2026

meta-codesync Bot added the meta-exported label Sep 2, 2026

mariusae force-pushed the export-D117242783 branch from 879b6c8 to 337e461 Compare September 2, 2026 16:23

meta-codesync Bot changed the title ~~add persistent experiment administration~~ add persistent experiment administration (#4776) Sep 2, 2026

mariusae force-pushed the export-D117242783 branch from 337e461 to c756544 Compare September 2, 2026 16:29

mariusae force-pushed the export-D117242783 branch from c756544 to 3d4f19b Compare September 2, 2026 16:38

mariusae force-pushed the export-D117242783 branch from 3d4f19b to aa310d8 Compare September 2, 2026 16:45

mariusae force-pushed the export-D117242783 branch from aa310d8 to 4a69cd6 Compare September 2, 2026 16:54

mariusae force-pushed the export-D117242783 branch from 4a69cd6 to 73efa81 Compare September 2, 2026 17:00

mariusae added 5 commits September 4, 2026 08:07

mariusae force-pushed the export-D117242783 branch from 73efa81 to f07fac4 Compare September 4, 2026 15:17

mariusae force-pushed the export-D117242783 branch from f07fac4 to c5c6b06 Compare September 4, 2026 16:05

mariusae force-pushed the export-D117242783 branch from c5c6b06 to bae7f38 Compare September 4, 2026 16:31

mariusae force-pushed the export-D117242783 branch from bae7f38 to 9bd2ae6 Compare September 4, 2026 16:45

mariusae force-pushed the export-D117242783 branch from f403a96 to f236e15 Compare September 4, 2026 17:52

mariusae force-pushed the export-D117242783 branch from f236e15 to 2596a78 Compare September 4, 2026 17:58

mariusae added 16 commits September 4, 2026 12:50

mariusae force-pushed the export-D117242783 branch from 2596a78 to ec7f2b7 Compare September 4, 2026 19:54

mariusae force-pushed the export-D117242783 branch from ec7f2b7 to f316643 Compare September 4, 2026 19:59

Navigation Menu

Uh oh!

add persistent experiment administration (#4776) - #4776

add persistent experiment administration (#4776)#4776
mariusae wants to merge 22 commits into
meta-pytorch:mainfrom
mariusae:export-D117242783

mariusae commented Sep 2, 2026 •

edited by meta-codesync Bot

Loading

Uh oh!

meta-codesync Bot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mariusae commented Sep 2, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meta-codesync Bot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mariusae commented Sep 2, 2026 •

edited by meta-codesync Bot

Loading