stop actor event loops before Python exits (#4757)

[shayne-fletcher]

Summary:

GitHub issue: #4749
GitHub pull request: #4757

each PythonActor instance has its own asyncio event loop running on a dedicated daemon OS thread, created through Python's threading.Thread. today Python can begin exiting while one of those threads is still alive, which can crash the whole process. specifically, CPython can force an actor thread to exit while it still has Rust/PyO3 calls on its stack. that forced unwind aborts the process with FATAL: exception not rethrown.

cpuhrsch's core dumps establish this exact failure path, and D117279357 identified and field-tested the key mitigation: stop actor event loops before Python finalizes. this diff builds on that idea and refines how the loops are found and stopped:

• each actor-loop thread is marked with its loop before it starts, allowing shutdown to use Python's existing list of live threads instead of adding another global registry or lock;
• the existing shutdown_tokio_runtime handler performs the work, so no second production atexit handler is added;
• shutdown asks every marked loop to stop, releases the GIL so the threads can run, and polls Python's live-thread list until the shared deadline. this keeps shutdown bounded even on Python 3.14, where Thread.is_alive() can itself block;
• every loop shares one 250 ms deadline instead of receiving its own timeout. the actual bound is 250 ms plus one final scan.

the guard is deliberately best effort. it covers loops visible while it is scanning, but it can miss a loop created after it has scanned and found none.

together with this fix, a deterministic subprocess test is added that follows the real shutdown path and reproduces precisely the observed failure: without the markers, Python 3.12 aborts during shutdown; with them, every actor loop stops before exit on Python 3.12 and GIL-enabled Python 3.14. focused Rust tests cover multiple loops and the deadline path.

this is a temporary mitigation. it stops leftover actor loops just before Python exits, but it does not fix why they are still running. today actor cleanup can return before its loop thread ends, and the root client can report Stopped without running that cleanup. the pytokio-removal work will provide the lifecycle fix. once every actor owns and waits for its loop thread on every shutdown path, this mitigation and its marker can be removed.

the Jobs API creates a simple process tree: it provisions worker host processes with run_worker_loop_forever(), and those worker hosts launch actor processes through bootstrap_main. a worker host hands control to Rust and ends with process::exit(0), so it never runs Python finalization. the client and actor processes return through Python and do run Py_FinalizeEx; those are the processes this mitigation protects.

Differential Revision: D117568016

[meta-codesync]

@shayne-fletcher has exported this pull request. If you are a Meta employee, you can view the originating Diff in D117568016.

[shayne-fletcher]

Fixes #4749