Repository navigation
Make parallelism thread- and core-safe (bounded, no recursive spawn) - #17
Merged
Merged
Conversation
Map of every process pool, in-worker subprocess, and backend that spawns work, how each is sized, and the exact nesting call chains (batch x singleresidue, evolution, direct singleresidue, trajectory). Findings that drive the bounding work: - Process counts on live paths are bounded to cores only by convention: the outer pool threads a reduced n_cpus/inner_cpus into the inner pool; there is no single shared budget enforcing it. - No BLAS/OpenMP thread pinning anywhere, so cores^2 native threads are reachable on every path. - Pools use the default fork start method, unsafe after threads start.
…fork-safe pools Bound every parallel path so nested pools can never multiply into a fork bomb that takes down the machine. Three changes, all drawing from one new shared-budget module (utils/concurrency.py): - Single core budget. resolve_concurrency(n_procs, n_items) splits the physical-core budget into (outer, inner) with outer * inner <= cores, and resolve_pool_size() caps a flat pool the same way. The batch executor (dir_frustration), the mutation scan pool (mutate_res_scan_parallel), and the evolution precompute pool all call these instead of each invoking cpu_count() independently, so no path can size a pool against full cores while already inside another pool. - Per-worker native-thread pinning. apply_thread_limits() pins OMP/OPENBLAS/MKL/NUMEXPR/VECLIB/BLIS thread counts to 1 (via setdefault, so an explicit operator value wins). It runs at the very top of frustrapy/__init__.py from a dependency-free leaf module (_threadlimits.py), before any numpy/BLAS import, so the parent and every spawn/fork worker start single-threaded. Without this, cores worker processes could each spin cores BLAS threads (cores**2). - Fork-safe start method. Pools are built from get_pool_context() (forkserver, else spawn, else default) with a worker initializer that re-pins threads, avoiding the fork-after-threads deadlock and orphaned LAMMPS children. close()/join() already run in finally on every path. Default behaviour is unchanged: dir_frustration with no n_procs stays serial. Verified output is byte-identical: 92 tests pass, including the 1CRN all-three-modes parity gate and the FrustraEvo byte-identical gate.
Add running tests that exercise the nested fork-bomb path rather than only the sizing math. A dir_frustration(n_procs=2) singleresidue batch runs the two-level pool nest while a sampler thread watches the live lmp_serial descendant count, asserting the peak stays within the shared core budget; a second test drives a pool from the shared fork-safe context with a raising worker and confirms close()/join() leave no orphan processes. Observed peak on a 14-core box is 7 concurrent lmp_serial (well under budget); an unbounded version stacking two cpu_count() pools would reach ~28. Full suite green (94 passed), 1CRN all 3 modes and FrustraEvo outputs byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make the parallelism thread- and core-safe so it cannot recursively spawn work.
The problem
batch whose structures each open their own mutation pool multiplied process
counts, and each worker process could spin one BLAS/OpenMP thread per core, so
total threads approached cores squared.
The fix
their worker count and inner pools get the remaining per-worker share, so the total
number of live workers never exceeds the physical core count.
process does not itself fan out across all cores.
no orphan LAMMPS processes leak.
Verification
asserts it stays within the budget; a test checks clean shutdown with no leftover
children. Existing outputs stay byte-identical.
Docs
pinning, and how the knobs compose.
Base branch is
stagingso this shows only the parallelism-safety diff.