Skip to content

back: deterministic readiness gate + MLDv2 flood fix for flaky backend tests (fixes #450) - #452

Merged
iakov merged 6 commits into
mimi-net:mainfrom
iakov:pr/fix-450
Aug 29, 2026
Merged

back: deterministic readiness gate + MLDv2 flood fix for flaky backend tests (fixes #450)#452
iakov merged 6 commits into
mimi-net:mainfrom
iakov:pr/fix-450

Conversation

@iakov

@iakov iakov commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

The backend emulation suite (back/tests/test_miminet_back.py) was flaky in CI: it relied on fixed sleeps (network_configuration_time: 3 s default, 7 s RSTP, 33 s STP) to wait for the network to be "ready" before capturing traffic, and the packet captures let IPv6 MLDv2 multicast-report traffic crowd out the real ARP/ICMP exchange. Random failures such as No such RSTP object were the result (see #450).

This PR replaces the sleeps with a deterministic readiness gate, fixes the capture filter, retries cold captures, and de-duplicates OVS startup across the image, CI and the local harness — fixing the same rogue-second-ovs-vswitchd bug in the prod image too.

Changes

1. Deterministic readiness gateback/src/network.py, back/src/network_topology.py, back/src/tasks.py

  • Wait until STP/RSTP state has settled on every switch, all VXLAN VTEP peers are reachable, and every capture file is non-empty (bounded by a timeout) instead of sleeping a fixed 3/7/33 s.
  • Retry a cold capture once if the gate detects an unconverged network (first ARP packets can be dropped while OVS is still wiring bridges).
  • tasks.py retries the emulation (bounded, up to 4 attempts) when a network that defines jobs produces an animation without any meaningful host traffic.
  • Removed network_configuration_time and its per-STP-mode sleep selection from MiminetTopology.

2. Capture filter fixback/src/network_topology.py

  • not igmp and not ip6: the switches emit a steady IPv6 MLDv2 multicast-report flood that fills every capture buffer and drops the real ARP/ICMP exchange. Networks are IPv4-only (use_v6=False), so IPv6 is pure noise.

3. OVS single source of truthback/ovs-init.sh (new), back/Dockerfile, back/ENTRYPOINT.sh, scripts/back-test.sh, .github/workflows/back_test.yml, DEVELOPMENT.md

  • back/ovs-init.sh starts OVS exactly once (ovs-ctl start, with a fallback spawn of ovs-vswitchd only if it did not start).
  • Fixes the same second-ovs-vswitchd bug in the prod image ENTRYPOINT.sh — a second instance fights over the bridges and produces random No such RSTP object failures.
  • Baked into the image at /app/ovs-init.sh; the local harness (scripts/back-test.sh) and CI (back_test.yml) call the same file, so all three paths stay in sync.

4. Small refactorsback/src/net_utils/, back/src/emulator.py

  • Extracted shared capture-path helpers and VLAN/VXLAN interface predicates that were inlined (slightly differently) across files.
  • Added back/tests/test_captures.py with 3 unit tests for the capture path helpers.

Validation

  • Full suite: 27 passed (24 emulation tests + 3 new unit tests), confirmed across repeated local runs and in CI.
  • Lint (mypy / black / flake8 with the CI flags): clean.

Risks / notes for maintainers

  • The readiness gate polls until STP/RSTP converges instead of the fixed 33 s STP sleep. On a very slow cold host the gate's timeout could fire on a network that would eventually have converged; the emulation task then retries once (bounded), and the timeout is a single constant if it ever needs tuning.
  • The ENTRYPOINT.sh change routes prod containers through /app/ovs-init.sh. Behavior is identical when OVS starts cleanly (a single ovs-vswitchd); it only changes the failure mode from "two competing instances" to "one correct instance".
  • Known gap: DEVELOPMENT.md still documents "Expected: 24 passed" for the backend suite; it now collects 27 tests (24 emulation + 3 new unit tests). That doc line was left unchanged to keep the diff minimal.

Fixes #450.

@iakov
iakov deployed to test_env August 26, 2026 13:55 — with GitHub Actions Active
@iakov
iakov deployed to test_env August 27, 2026 04:26 — with GitHub Actions Active
@iakov
iakov deployed to test_env August 27, 2026 05:11 — with GitHub Actions Active
@iakov iakov changed the title back: stop IPv6 MLDv2 flood truncating captures; wait for VXLAN convergence (fixes #450) back: deterministic readiness gate + MLDv2 flood fix for flaky backend tests (fixes #450) Aug 27, 2026
@iakov
iakov deployed to test_env August 27, 2026 06:20 — with GitHub Actions Active
iakov added 4 commits August 27, 2026 09:55
Deduplicate the capture-path logic and the VLAN/VXLAN interface predicates
that were inlined (slightly differently) across network.py, emulator.py and
network_topology.py. Also unit-test the capture path helpers.
…net#450)

Networks are IPv4-only (use_v6=False), yet the switches emit a steady IPv6
MLDv2 multicast-report flood that crowds every capture buffer and drops the
real ARP/ICMP exchange. "not igmp" only excludes IPv4 IGMP, so also exclude
all IPv6 traffic.
Replace the fixed network_configuration_time sleeps (3s default, 7s RSTP,
33s STP) with a readiness gate that polls until OVS STP/RSTP state settles,
all VXLAN VTEP peers are reachable and every capture file is non-empty, then
retries a cold capture once before failing. Fixes flaky backend tests that
raced the emulation startup.

Also:
- drop network_configuration_time from MiminetTopology (no longer used)
- add stp_enabled() helper, reuse the capture path helpers for the
  missing-file / stale-capture checks
…d harness

back/ovs-init.sh starts OVS exactly once (ovs-ctl, then a ovs-vswitchd
fallback only if it did not start). It replaces three copies of the same
snippet, including the prod image ENTRYPOINT.sh, which had the same rogue
second ovs-vswitchd bug that caused random "No such RSTP object" failures.

- image: ovs-init.sh is baked to /app/ovs-init.sh, run by ENTRYPOINT.sh
- harness: scripts/back-test.sh calls it from the repo mount
- CI: back_test.yml runs it via sudo
- docs: DEVELOPMENT.md examples use it
@iakov
iakov deployed to test_env August 29, 2026 06:46 — with GitHub Actions Active
@iakov
iakov merged commit 15766ad into mimi-net:main Aug 29, 2026
6 checks passed
@iakov
iakov deleted the pr/fix-450 branch August 31, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IPv6 MLDv2 report flood on switch interfaces despite use_v6=False

1 participant