Skip to content

test(scenario): assert offline peer catches up after reconnect - #135

Draft
D4ryl00 wants to merge 2 commits into
berty:mainfrom
D4ryl00:test/replication-convergence
Draft

test(scenario): assert offline peer catches up after reconnect#135
D4ryl00 wants to merge 2 commits into
berty:mainfrom
D4ryl00:test/replication-convergence

Conversation

@D4ryl00

@D4ryl00 D4ryl00 commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

What

Adds an end-to-end scenario test asserting that a peer which goes offline, misses messages, and then reconnects receives every missed message via the orbit-db head exchange — with no application-level retry loop.

It runs on the full protocol service (NewTestingProtocolWithMockedPeers): create a 2-peer multi-member group, take one peer offline (sever the mocknet link + disconnect), have the other send 20 messages, reconnect, then assert the offline peer catches up on all 20 through GroupMessageList.

Why

Catch-up after an offline window had no test coverage — the existing scenario tests use a static ConnectAll topology and never drop a peer. This fills that gap on the real service stack (real AppMessageSend / GroupMessageList, real gossipsub + head exchange).

The one mocknet compromise (documented in the test)

The ideal test would restore the link and let the transport reconverge on its own (discovery poll → BackoffConnector re-dial → topic re-join → head exchange). A probe confirmed mocknet does not do this: after the link is restored the peer never re-dials on its own (NotConnected for 60s+), because mocknet wires peers directly via ConnectAll rather than through the discovery/backoff machinery that drives reconnection on a real network.

So the test re-establishes the connection with a single ConnectPeers — modelling the one reconnect that discovery would produce — and asserts that this single reconnect is enough: the head exchange re-fires and backfills the whole log, no retry loop.

Note — this single reconnect is reliable, but not because the full service is immune. The underlying flake is a go-libp2p-pubsub connect/disconnect race: on a fast reconnect the new-peer event is deduped against the not-yet-removed stale peer entry, so the peer's subscriptions are never re-propagated and it ends up in 0 topics despite a healthy connection. This isolated, clean reconnect rarely trips it; the busier bertyreplication topology hits it ~1/100 and needs a reconnect retry loop. So it's low-probability here, not immunity — 30× clean is not proof of zero.

Notes

  • Named TestScenario_* so it runs in the flappy CI lane (TEST_STABILITY=flappy, testman -retry=10), matching the other networking scenario tests.
  • Ran 30x under -race locally without a failure (~1.8s each).

Signed-off-by: D4ryl00 d4ryl00@gmail.com

D4ryl00 added 2 commits June 25, 2026 00:19
Adds an end-to-end test on the full protocol service for the catch-up path
that bare orbit-db-over-mocknet tests cannot cover: a peer goes offline,
misses messages, reconnects, and is expected to receive every missed
message via the orbit-db head exchange — with no application-level retry.

mocknet does not run the discovery poll + BackoffConnector that re-dials a
dropped peer on a real network (a probe confirms a peer never reconnects on
its own after the link is restored), so the test re-establishes the
connection with a single ConnectPeers, modelling the one reconnect discovery
would produce. The reconnect then re-fires the head exchange, which backfills
the whole log; the full-service gossipsub machinery converges the join
without any retry loop.

Ran 30x under -race without a failure.

Signed-off-by: D4ryl00 <d4ryl00@gmail.com>
The catch-up relies on a single reconnect not tripping a go-libp2p-pubsub
connect/disconnect race that can drop a peer's subscriptions; this isolated
clean reconnect rarely hits it, but it is not immunity (the busier
bertyreplication topology hits it ~1/100).

Signed-off-by: D4ryl00 <d4ryl00@gmail.com>
@D4ryl00
D4ryl00 marked this pull request as draft June 25, 2026 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant