Skip to content

An unsatisfiable blocking reload starves a client whose cache already holds its runway #446

Description

@superuser404notfound

Split out of #442, where it is the step before the one that was reported. Measured on the harness, not from a device log.

What happens

A live session with a DVR window, viewer parked inside it, upstream stops delivering (connection stays open, no further bytes). Every segment ahead of the playhead is already in the segment cache and the local server would hand it over instantly. Playback stops anyway.

  state=playing t=60.90s edge=253.40s behind=192.50s range=1.4...253.4
[+251.96s] [NativeAVPlayerHost] #1 playbackStalled
  state=playing t=61.31s edge=253.40s behind=192.09s range=1.4...253.4
[+257.44s] [HLSLocalServer] -> 503 /media.m3u8 reason=blocking reload msn=54 unsatisfiable
[+257.45s] [NativeAVPlayerHost] #1 errorLog code=-16849 ... 'HTTP 503: Service Unavailable'
[+258.18s] [AetherEngine] #65 re-engaging stalled AVPlayer (stall + 6s without fetches): nudge seek to 59.91s

range=1.4...253.4 is the resident range since AE#441, so this is not the window overstating what the cache holds. The client is 192 s behind live with 192 s of resident content in front of it, and it stalls.

Mechanism

AVPlayer refreshes the playlist with a blocking reload (?_HLS_msn=N). Once the source stops, that request can never be satisfied, so the server holds it for liveBlockingReloadHoldSeconds and then answers 503, which is what RFC 8216bis asks for and what #167 put there. While that request is outstanding AVPlayer issues no segment requests at all, and its forward buffer is 4 s deep by our own forwardBufferDuration. From a second run, timed off the same log:

  • +89.18 poll for msn=24 goes out
  • +93.21 last segment fetch
  • +102.12 playbackStalled, buffer drained
  • +107.19 the hold expires, 503 goes back: 18.0 s, exactly 3 x TARGETDURATION (TD=6)

liveBlockingReloadHoldSeconds already carries the rule this violates: "A hold that outlives AVPlayer's ~4 s forward buffer guarantees the stall it exists to prevent." That was written about the old hardcoded 18 s under fastZap. On a stalled source 3 x TD reproduces it exactly, because the bound is only tight while the source still cuts segments.

What ends it

Two things, both accidental:

  • our own #65 re-engage nudge, one stall-recovery window later, which buys about two segments before the next hold parks it again;
  • AVPlayer demoting itself. After one -15410 'Invalid server blocking reload behavior for low latency' at +124.34 it stopped sending _HLS_msn entirely, and the same frozen source then played the whole cached runway continuously (40+ s, no stall, only harmless -12888 playlist-unchanged entries).

So the error the 503 exists to avoid is, on this path, the thing that restores playback.

Why it matters

This is the entry point of #442. A viewer parked minutes behind live does not need the newest playlist at all, but the hold starves them anyway, the stall escalates through the recovery ladder, and by the time #93 item death fires there is a position to lose. Fixing where the recovery rejoins (done, #442) does not stop the source freeze from stopping playback for someone who had everything they needed on disk.

Not fixed yet, and why

The obvious move, answering the unsatisfiable hold with the current playlist instead of holding, is exactly the -15410 case #167 put the 503 there to avoid. The measurement above says that particular -15410 is benign and in fact recovers, but that is one Mac run against a synthetic source, and #167 came off a device. Withdrawing CAN-BLOCK-RELOAD once production has halted has the same delivery problem: the client cannot learn about it without a response it will accept.

Wanted before building: a device run that says whether a fast 503 (bail immediately when liveProductionHalted, no hold) is enough to keep the fetch loop alive, or whether the advertisement itself has to go.

Reproduction

swift run aetherctl live --realtime --preroll 30 --dvr-window 1800 \
  --freeze-after 150 --rewind-before-freeze 120 --seconds 260

--freeze-after leaves the connection open and stops delivering, which is how a dead upstream usually looks and is a different failure from --drop-after's RST.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions