OneSync: clients dropped with "Timed out after 60 seconds (2)" since frame retention change (38386+) #4394

What happened?

Since we updated FXServer from 37850 to 38386 (and later to 38435), players get dropped in waves with OneSync missed-frame reasons:

• Timed out after 60 seconds (2)
• Timed out after 60 seconds (1, <frames>, <ms>), for example (1, 0, 761) and (1, 0, 30)

Daily count of these drop reasons on one server, about 60-80 players at peak, same resources and same host throughout:

Server->client connection timed out drops stayed at the same level before and after the update. The server itself is healthy when this happens: no hitch warnings, around 90% CPU idle, no interface drops, and other outbound services on the machine keep working. Several players are dropped within a few seconds of each other. For example, 17 out of 65 players dropped between 19:57:34 and 19:58, and most of them reconnected right away.

We think this comes from bd84727 ("fix(sync): retain frames based on client silence"). The old check was client->GetLastSeen() > 5s. GetLastSeen() is an epoch timestamp, so that check was always true, and every client kept the worst-case backlog of maxSavedClientFramesWorstCase (~60 s). ShouldUseWorstCaseFrameBacklog now uses the time elapsed since the last update (since d34c31a, since the last game state update). A client that falls more than ~650 frames (~8-9 s) behind while still sending some packets therefore has its frames erased. When it NAcks one of those frames it is dropped in HandleGameStateNAck ("(1, ...)") or because the frame no longer exists ("(2)").

The bug fix itself makes sense. In practice, though, the retention window dropped from ~60 s to ~8-9 s for every client, so a short network hiccup that clients used to recover from now disconnects them.

Expected result

A client that falls behind for 10-20 seconds should recover, as it did on 37850, rather than being dropped with a 60-second timeout message.

Reproduction steps

1. Run a populated OneSync server on 38386 or newer.
2. Add a ~10-15 s stall or heavy packet loss on a client's downstream while the client keeps sending (for example with clumsy on the client, or a short network interruption).
3. Expected per the code above: the client is dropped with Timed out after 60 seconds (2) or (1, x, y), while on 37850 it would still have 60 s of frames. We have not run this controlled test yet; our data comes from the production numbers above.

Importancy

There's a workaround

Area(s)

FXServer

Specific version(s)

FXServer linux 38386 and 38435 (current recommended). Not seen on 37850. As far as we can see, the relevant code is unchanged in 38621.

Additional information

Our workaround is to roll back to 37850. A server convar for the minimum frame retention, or using the worst-case backlog whenever the client is behind (rather than only after 5 s of silence), would let us move back to recommended builds.