back: deterministic readiness gate + MLDv2 flood fix for flaky backend tests (fixes #450) - #452
Merged
Conversation
Deduplicate the capture-path logic and the VLAN/VXLAN interface predicates that were inlined (slightly differently) across network.py, emulator.py and network_topology.py. Also unit-test the capture path helpers.
…net#450) Networks are IPv4-only (use_v6=False), yet the switches emit a steady IPv6 MLDv2 multicast-report flood that crowds every capture buffer and drops the real ARP/ICMP exchange. "not igmp" only excludes IPv4 IGMP, so also exclude all IPv6 traffic.
Replace the fixed network_configuration_time sleeps (3s default, 7s RSTP, 33s STP) with a readiness gate that polls until OVS STP/RSTP state settles, all VXLAN VTEP peers are reachable and every capture file is non-empty, then retries a cold capture once before failing. Fixes flaky backend tests that raced the emulation startup. Also: - drop network_configuration_time from MiminetTopology (no longer used) - add stp_enabled() helper, reuse the capture path helpers for the missing-file / stale-capture checks
…d harness back/ovs-init.sh starts OVS exactly once (ovs-ctl, then a ovs-vswitchd fallback only if it did not start). It replaces three copies of the same snippet, including the prod image ENTRYPOINT.sh, which had the same rogue second ovs-vswitchd bug that caused random "No such RSTP object" failures. - image: ovs-init.sh is baked to /app/ovs-init.sh, run by ENTRYPOINT.sh - harness: scripts/back-test.sh calls it from the repo mount - CI: back_test.yml runs it via sudo - docs: DEVELOPMENT.md examples use it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The backend emulation suite (
back/tests/test_miminet_back.py) was flaky in CI: it relied on fixed sleeps (network_configuration_time: 3 s default, 7 s RSTP, 33 s STP) to wait for the network to be "ready" before capturing traffic, and the packet captures let IPv6 MLDv2 multicast-report traffic crowd out the real ARP/ICMP exchange. Random failures such asNo such RSTP objectwere the result (see #450).This PR replaces the sleeps with a deterministic readiness gate, fixes the capture filter, retries cold captures, and de-duplicates OVS startup across the image, CI and the local harness — fixing the same rogue-second-
ovs-vswitchdbug in the prod image too.Changes
1. Deterministic readiness gate —
back/src/network.py,back/src/network_topology.py,back/src/tasks.pytasks.pyretries the emulation (bounded, up to 4 attempts) when a network that defines jobs produces an animation without any meaningful host traffic.network_configuration_timeand its per-STP-mode sleep selection fromMiminetTopology.2. Capture filter fix —
back/src/network_topology.pynot igmp and not ip6: the switches emit a steady IPv6 MLDv2 multicast-report flood that fills every capture buffer and drops the real ARP/ICMP exchange. Networks are IPv4-only (use_v6=False), so IPv6 is pure noise.3. OVS single source of truth —
back/ovs-init.sh(new),back/Dockerfile,back/ENTRYPOINT.sh,scripts/back-test.sh,.github/workflows/back_test.yml,DEVELOPMENT.mdback/ovs-init.shstarts OVS exactly once (ovs-ctl start, with a fallback spawn ofovs-vswitchdonly if it did not start).ovs-vswitchdbug in the prod imageENTRYPOINT.sh— a second instance fights over the bridges and produces randomNo such RSTP objectfailures./app/ovs-init.sh; the local harness (scripts/back-test.sh) and CI (back_test.yml) call the same file, so all three paths stay in sync.4. Small refactors —
back/src/net_utils/,back/src/emulator.pyback/tests/test_captures.pywith 3 unit tests for the capture path helpers.Validation
Risks / notes for maintainers
ENTRYPOINT.shchange routes prod containers through/app/ovs-init.sh. Behavior is identical when OVS starts cleanly (a singleovs-vswitchd); it only changes the failure mode from "two competing instances" to "one correct instance".DEVELOPMENT.mdstill documents "Expected: 24 passed" for the backend suite; it now collects 27 tests (24 emulation + 3 new unit tests). That doc line was left unchanged to keep the diff minimal.Fixes #450.