Skip to content

fix(demos): cap build parallelism to avoid gateway OOM#62

Merged
mfaferek93 merged 1 commit into
mainfrom
fix/demo-build-oom
Jun 5, 2026
Merged

fix(demos): cap build parallelism to avoid gateway OOM#62
mfaferek93 merged 1 commit into
mainfrom
fix/demo-build-oom

Conversation

@mfaferek93
Copy link
Copy Markdown
Contributor

Problem

Docker builds for the demos fail intermittently with:

c++: fatal error: Killed signal terminated program cc1plus
... ResourceExhausted: cannot allocate memory

The build always dies in ros2_medkit_gateway. It is not a compile error and not architecture-specific (diagnostic_bridge and the other packages build cleanly). It is an out-of-memory kill of the compiler.

Root cause

The gateway has a few heavily-templated translation units (HTTP handlers) that peak at ~1.6 GB RSS each when compiling. colcon/make compiled them in parallel, one job per CPU core, so total peak scaled with core count. On many-core or RAM-limited Docker hosts (e.g. Docker Desktop), several heavy units overlap and exceed the memory limit, so cc1plus is OOM-killed. Whether a given machine succeeds depended on core count and scheduling timing, so it was flaky (passed on 8-core/8 GB, failed on a higher-core host even at 12 GB).

Fix

Limit build parallelism in the four demos that build the gateway from source: --executor sequential (one package at a time) + MAKEFLAGS='-j 2' (at most two compilers at once). Peak drops to ~3.2 GB, making the build deterministic across hosts.

manymove_industrial already has its own parallelism knob (MANYMOVE_COLCON_WORKERS) and is left unchanged.

Verification

Built the gateway from main inside a container hard-capped at --memory=6g --memory-swap=6g:

Finished <<< ros2_medkit_gateway [2min 19s]
Summary: 4 packages finished

Passes at 6 GB with margin; the previous unbounded build was OOM-killed under the same cap.

Gateway has a few heavily-templated TUs (~1.6 GB RSS each); compiling
them all in parallel (one per core) OOM-killed cc1plus on many-core or
RAM-limited Docker hosts. Use a sequential executor + MAKEFLAGS='-j 2'
(peak ~3.2 GB, verified building under a 6 GB cap).
@mfaferek93 mfaferek93 merged commit 3c69954 into main Jun 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants