Make actor message handling abortable (#4665) - #4665
Open
thedavekwon wants to merge 6 commits into
Open
Conversation
meta-cla
Bot
added
the
CLA Signed
Contributor
|
@thedavekwon has exported this pull request. If you are a Meta employee, you can view the originating Diff in D115847469. |
thedavekwon
force-pushed
the
export-D115847469
branch
from
September 1, 2026 21:24
1a1aed2 to
986af40
Compare
meta-codesync
Bot
changed the title
Summary: Pull Request resolved: meta-pytorch#4646 Use terminal `ActorSupervisionEvent`s for successful, failed, and zombie child completion, and remove the parallel `Signal::ChildStopped` path. Each child enqueues exactly one terminal event while retaining its direct-parent ownership edge. The event carries private, non-serialized `child_to_unlink` metadata for that hop; the parent removes the edge when it consumes the event, before invoking the actor supervision handler. `actor_id` continues to identify the original event subject when failures propagate through multiple levels. The parent child map is therefore a barrier for unconsumed completion events. This prevents teardown from consuming the final event before observing the corresponding unlink while preserving event delivery before terminal status becomes externally visible. Custom actor loops consume the same terminal events. Differential Revision: D114913598
Summary: Pull Request resolved: meta-pytorch#4664 Replace the separate Kill and Abort vocabulary with one forced actor operation: Abort. Make `abort()` the primary API and retain `kill()` only as a compatibility alias. Design invariant: every forced actor termination has one control signal, one terminal cause, and one result: `Abort` and `Aborted`. Callers and supervision no longer distinguish semantically identical forced-stop paths. Update internal producers, consumers, status reporting, documentation, and tests to use the unified outcome without changing cancellation behavior. Differential Revision: D115847493
Summary: Move cooperative shutdown out of the runtime teardown path and into `Actor::handle_stop`, while keeping the actor loop available until the actor requests exit. Design invariant: the runtime delivers `Stop` and `DrainAndStop`, but the actor owns how cooperative shutdown completes. `Stop` invokes the hook immediately; `DrainAndStop` closes handler ingress and queues the hook behind already accepted work. The default hook closes ingress, preserves the requested mode and reason, forwards that request to current direct children, and exits only after every direct child completes. Overrides may return and complete asynchronously through ordinary messages or supervision events. A direct child remains in the parent ownership map until the parent consumes its terminal supervision event. On receipt, the runtime removes the exact direct-child edge and invokes `handle_child_stopped` before `handle_supervision_event`. The child map is therefore the completion barrier during cooperative shutdown, and child failures remain observable before the parent completes. `ActorSupervisionEvent.actor_id` remains the propagated event subject; private, non-serialized `child_to_unlink` records only the hop-local ownership edge. Remote actors remain represented by local `Supervisor` proxies rather than being inserted into the local ownership topology. A proxy forwards stop requests before or after `Linked` and remains alive until remote terminal, unlink, or liveness handling resolves its local lifecycle. Once an actor loop returns, any child still linked is residual. The following diff transfers those children to proc ownership and aborts them; later diffs make active user hooks Abort-aware. Differential Revision: D110433174
Summary: After an actor loop exits, detach every remaining direct child, promote it to a proc root, mark it `Zombie`, and request Abort without traversing descendants or waiting for completion. Design invariant: every live actor has exactly one current owner: one live parent or the proc root index. The child parent-link lock serializes completion routing with forced detachment. While the parent loop is active, a terminal event retains the parent link until that event is consumed. Once the parent loop exits, any still-linked child is residual and teardown transfers it to proc ownership. If detachment wins before completion routes, the later terminal event remains available to introspection but does not re-enter supervision. Each actor handles only its own direct residual children. Proc teardown remains root-oriented, remote descendants use liveness orphan handling, and no stuck descendant can pin an ancestor teardown. Differential Revision: D114913597
Summary: Remove the per-actor cleanup deadline and its Rust, Python, documentation, and test configuration surface. Transport flush timeouts remain unchanged. Design invariant: cleanup is ordinary user code, not a separate teardown phase with its own runtime timeout policy. Graceful and failure exits await cleanup normally; the following Abort-aware lifecycle diff is solely responsible for skipping or preempting cleanup during forced termination. This leaves deadline and escalation policy with explicit teardown callers and avoids competing cleanup, actor, and proc timers. Reviewed By: mariusae Differential Revision: D110433730
thedavekwon
added a commit
to thedavekwon/monarch
that referenced
this pull request
Sep 2, 2026
Summary: Pull Request resolved: meta-pytorch#4665 Make `Abort` cancel active asynchronous message handling at its next yield. Abort is published through a sticky side channel and has priority whenever cancellation and handler progress are both ready. The standard actor loop observes this channel while idle and while polling `work.handle(...)`; `Signal::Abort` is removed. Only user message handling is preemptible. `Actor::init`, stop and supervision hooks, cleanup, synchronous handler code, and custom `ActorInstance` loops are not cancelled. After handler cancellation, the existing error path performs child teardown, cleanup, supervision reporting, introspection shutdown, and terminal status publication. Proc timeout escalation marks an already proc-owned root actor `Zombie` before publishing Abort. Forced parent teardown detaches each residual direct child, promotes it to proc ownership, marks it `Zombie`, and publishes Abort, allowing ancestor teardown to continue. Design invariant: once the standard actor loop observes Abort, it drops any in-flight asynchronous handler and starts no subsequent handler. Abort never bypasses lifecycle teardown; `Zombie` remains nonterminal until the actor publishes its true terminal status. Differential Revision: D115847469
meta-codesync
Bot
changed the title
thedavekwon
force-pushed
the
export-D115847469
branch
from
September 2, 2026 07:40
986af40 to
361a40d
Compare
thedavekwon
added a commit
to thedavekwon/monarch
that referenced
this pull request
Sep 2, 2026
Summary: Pull Request resolved: meta-pytorch#4665 Make `Abort` cancel active asynchronous message handling at its next yield. Abort is published through a sticky side channel and has priority whenever cancellation and handler progress are both ready. The standard actor loop observes this channel while idle and while polling `work.handle(...)`; `Signal::Abort` is removed. Only user message handling is preemptible. `Actor::init`, stop and supervision hooks, cleanup, synchronous handler code, and custom `ActorInstance` loops are not cancelled. After handler cancellation, the existing error path performs child teardown, cleanup, supervision reporting, introspection shutdown, and terminal status publication. Proc timeout escalation marks an already proc-owned root actor `Zombie` before publishing Abort. Forced parent teardown detaches each residual direct child, promotes it to proc ownership, marks it `Zombie`, and publishes Abort, allowing ancestor teardown to continue. Design invariant: once the standard actor loop observes Abort, it drops any in-flight asynchronous handler and starts no subsequent handler. Abort never bypasses lifecycle teardown; `Zombie` remains nonterminal until the actor publishes its true terminal status. Differential Revision: D115847469
thedavekwon
force-pushed
the
export-D115847469
branch
from
September 2, 2026 07:44
361a40d to
a8e2f65
Compare
Summary: Pull Request resolved: meta-pytorch#4665 Make `Abort` cancel active asynchronous message handling at its next yield. Abort is published through a sticky side channel and has priority whenever cancellation and handler progress are both ready. The standard actor loop observes this channel while idle and while polling `work.handle(...)`; `Signal::Abort` is removed. Only user message handling is preemptible. `Actor::init`, stop and supervision hooks, cleanup, synchronous handler code, and custom `ActorInstance` loops are not cancelled. After handler cancellation, the existing error path performs child teardown, cleanup, supervision reporting, introspection shutdown, and terminal status publication. Proc timeout escalation marks an already proc-owned root actor `Zombie` before publishing Abort. Forced parent teardown detaches each residual direct child, promotes it to proc ownership, marks it `Zombie`, and publishes Abort, allowing ancestor teardown to continue. Design invariant: once the standard actor loop observes Abort, it drops any in-flight asynchronous handler and starts no subsequent handler. Abort never bypasses lifecycle teardown; `Zombie` remains nonterminal until the actor publishes its true terminal status. Differential Revision: D115847469
thedavekwon
force-pushed
the
export-D115847469
branch
from
September 2, 2026 07:48
a8e2f65 to
5120bae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Make
Abortcancel active asynchronous message handling at its next yield. Abort is published through a sticky side channel and has priority whenever cancellation and handler progress are both ready. The standard actor loop observes this channel while idle and while pollingwork.handle(...);Signal::Abortis removed.Only user message handling is preemptible.
Actor::init, stop and supervision hooks, cleanup, synchronous handler code, and customActorInstanceloops are not cancelled. After handler cancellation, the existing error path performs child teardown, cleanup, supervision reporting, introspection shutdown, and terminal status publication.Proc timeout escalation marks an already proc-owned root actor
Zombiebefore publishing Abort. Forced parent teardown detaches each residual direct child, promotes it to proc ownership, marks itZombie, and publishes Abort, allowing ancestor teardown to continue.Design invariant: once the standard actor loop observes Abort, it drops any in-flight asynchronous handler and starts no subsequent handler. Abort never bypasses lifecycle teardown;
Zombieremains nonterminal until the actor publishes its true terminal status.Differential Revision: D115847469