test(scenario): assert offline peer catches up after reconnect - #135
Draft
D4ryl00 wants to merge 2 commits into
Draft
test(scenario): assert offline peer catches up after reconnect#135D4ryl00 wants to merge 2 commits into
D4ryl00 wants to merge 2 commits into
Conversation
Adds an end-to-end test on the full protocol service for the catch-up path that bare orbit-db-over-mocknet tests cannot cover: a peer goes offline, misses messages, reconnects, and is expected to receive every missed message via the orbit-db head exchange — with no application-level retry. mocknet does not run the discovery poll + BackoffConnector that re-dials a dropped peer on a real network (a probe confirms a peer never reconnects on its own after the link is restored), so the test re-establishes the connection with a single ConnectPeers, modelling the one reconnect discovery would produce. The reconnect then re-fires the head exchange, which backfills the whole log; the full-service gossipsub machinery converges the join without any retry loop. Ran 30x under -race without a failure. Signed-off-by: D4ryl00 <d4ryl00@gmail.com>
The catch-up relies on a single reconnect not tripping a go-libp2p-pubsub connect/disconnect race that can drop a peer's subscriptions; this isolated clean reconnect rarely hits it, but it is not immunity (the busier bertyreplication topology hits it ~1/100). Signed-off-by: D4ryl00 <d4ryl00@gmail.com>
D4ryl00
marked this pull request as draft
June 25, 2026 07:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an end-to-end scenario test asserting that a peer which goes offline, misses messages, and then reconnects receives every missed message via the orbit-db head exchange — with no application-level retry loop.
It runs on the full protocol service (
NewTestingProtocolWithMockedPeers): create a 2-peer multi-member group, take one peer offline (sever the mocknet link + disconnect), have the other send 20 messages, reconnect, then assert the offline peer catches up on all 20 throughGroupMessageList.Why
Catch-up after an offline window had no test coverage — the existing scenario tests use a static
ConnectAlltopology and never drop a peer. This fills that gap on the real service stack (realAppMessageSend/GroupMessageList, real gossipsub + head exchange).The one mocknet compromise (documented in the test)
The ideal test would restore the link and let the transport reconverge on its own (discovery poll →
BackoffConnectorre-dial → topic re-join → head exchange). A probe confirmed mocknet does not do this: after the link is restored the peer never re-dials on its own (NotConnectedfor 60s+), because mocknet wires peers directly viaConnectAllrather than through the discovery/backoff machinery that drives reconnection on a real network.So the test re-establishes the connection with a single
ConnectPeers— modelling the one reconnect that discovery would produce — and asserts that this single reconnect is enough: the head exchange re-fires and backfills the whole log, no retry loop.Note — this single reconnect is reliable, but not because the full service is immune. The underlying flake is a go-libp2p-pubsub connect/disconnect race: on a fast reconnect the new-peer event is deduped against the not-yet-removed stale peer entry, so the peer's subscriptions are never re-propagated and it ends up in 0 topics despite a healthy connection. This isolated, clean reconnect rarely trips it; the busier
bertyreplicationtopology hits it ~1/100 and needs a reconnect retry loop. So it's low-probability here, not immunity — 30× clean is not proof of zero.Notes
TestScenario_*so it runs in the flappy CI lane (TEST_STABILITY=flappy,testman -retry=10), matching the other networking scenario tests.-racelocally without a failure (~1.8s each).Signed-off-by: D4ryl00 d4ryl00@gmail.com