A producer restart that recovers a dead source leaves playbackPhase on .stalled(reconnecting:) indefinitely — the aborted producer owned the stall and the fresh one never clears it

[rrgomes]

Measured on 6.45.0, Apple TV 4K 3rd gen / tvOS 26.6, loopback-HLS VOD on the
native path. Asset: 1080p H.264 MKV, ~35 Mbit/s, ~89 min, DTS audio, PGS subs.

After the origin comes back and the reader is demonstrably serving again,
playbackPhase stays .stalled(reconnecting: true) — through normal playback,
indefinitely, until some later unrelated recovery happens to clear it. The
longest we have measured is 454.8 s, of which 298 s were with the picture
playing normally.

The rig

The origin is reached through a local reverse proxy we can kill and restart, so
only the media plane dies; nothing else about the session changes. Killing it
produces Could not connect to the server on the reader within milliseconds on
this asset, because at ~35 Mbit/s a 16 MB read-ahead is only about 4 s and the
reader is essentially always connected.

What we measured

Three arms, one session, minutes apart. The discriminator is whether the
recovery restarted the producer, which is visible in the reader's generation
counters.

arm	outage	producer restart on recovery	reader generations	phase cleared after
A	~140 s	yes	reset to pump gen=10 / prefetch gen=4	304.6 s, and only on a later restart-free recovery
B	~133 s	yes	reset to gen=1	still stalled 298 s later; see C
C (control)	12 s	no	continued pump gen=87, 88, 89	8 s

Arm B in full:

+0.000   cut
+0.168   phase -> .stalled(reconnecting: true)
+48      picture stops (read-ahead exhausted)
+133     origin restored
+133.44  pump gen=1 first data after 27ms      <- reader healthy, generations RESET
+431     picture playing normally, reader at gen=91, ~90 clean fetches,
         phase STILL .stalled(reconnecting: true)

Arm C is the same session five minutes later. A 12 s outage is too brief to
wedge the producer, so no restart: generations continued rather than resetting,
and the phase cleared 8 s after the origin returned.

The mechanism, from your own diagnostics

On the recovery in arm B:

[HLSVideoEngine] restart at idx=286: old producer wedged in a read past 5s;
                 aborted it and reopened a fresh demuxer
[HLSVideoEngine] producer restarted at idx=286 (restart took 5150ms;
                 stopWait=5005ms reopen=126ms seek=18ms build=0ms)

The aborted producer is the one that owned the reconnecting state. The fresh
demuxer was never stalled, so it has nothing to clear, and the phase keeps
reporting the reader that no longer exists.

Why a host cannot work around it

playbackPhase is the only signal a host has for "the source is delivering".
The obvious alternative — treat an advancing clock as proof — is one we
deliberately rejected after two device traces of our own: read-ahead plays on
for tens of seconds after a source dies, so a clock can advance over a dead
source; and a seek that cannot land jumps the clock while parking the phase on
.seeking. So we clear our own reconnect state only when the phase is neither
stalled nor seeking, and a phase that never leaves stalled is unrecoverable
from outside the engine.

The user-visible cost on our side is a "Reconnecting" indicator that stays up
over normally playing video, and — because we drive a group-playback feature
off the same signal — a peer that keeps telling the room it is buffering long
after it is fine.

The ask

When a producer restart replaces the reader, the stall state should go with it:
either clear it as part of the swap, or let the fresh producer publish a
non-stalled phase once it is serving. A "reader is delivering" signal that does
not depend on which producer instance owns it would also solve it from our side.

Caveat on version

We are pinned to 6.45.0 and have not yet tested 6.46.0 or 6.47.0, because we are
holding the pin while a retest on another issue is outstanding. 6.46.0 changes
producer cutting, so it may well move this. Happy to bump and re-run both arms
if that would be more useful than the measurements above.

The rig is reproducible on demand here and we can run whatever would help,
including a longer soak or a variant that wedges the producer without killing
the origin.

[superuser404notfound]

Confirmed, reproduced locally, and fixed on main in 834fc17c. The three-arm table with the generation counters was what made this findable: the reset generations name the swap, and that is exactly where the axis breaks.

What it actually is

The reading in the report is right about the outcome and one step short of the cause. The fresh demuxer does not merely have "nothing to clear": it does form an opinion, and it forms it while nobody is listening.

AVIOReader deduplicates its phase emissions through a small gate, so a flapping origin cannot spam the callback. Since #410 that gate starts with no opinion, precisely so a reader installed by a reopen can report its own recovery. What defeats it is the wiring order at the restart: the replacement demuxer is opened first and handed the sink one step later.

try fresh.open(...)                 // find_stream_info reads megabytes: gate latches .flowing into a nil sink
...
demuxer = freshDemuxer
freshDemuxer.onNetworkPhaseChanged = onNetworkPhaseChanged   // the listener arrives after the fact

By the time the sink exists, the gate believes it has already said .flowing, so the reader that is now serving the session stays silent for the rest of it, and the last word anyone heard is the aborted reader's .reconnecting. That is also why arm A cleared at +304.6 s only on a later restart-free recovery: that recovery ran through the same reader's own reconnect cycle, which flaps the gate through .reconnecting and lets the next .flowing out.

Two adjacent holes came out of the same reading:

• the replaced demuxer kept its sink, so an aborted pump that outlives the swap could still move the axis for a session it no longer feeds;
• the live reopen never wired the sink onto its fresh demuxer at all, so on that path the axis could not recover for the rest of the session either.

The fix

The gate now deduplicates for a LISTENER, not just for a reader instance: attaching a sink clears the history, because a listener that just arrived has heard nothing, whatever the reader said into the void during its own open. The sink and its gate moved under one leaf lock, since the handover thread installs while the demux thread emits. At the swap the outgoing demuxer is unwired and the incoming one takes the sink, on both the restart and the live-reopen paths.

That lands on the second option in the ask: the fresh reader publishes a non-stalled phase off its own first measured delivery, rather than anyone asserting health at the swap.

setReaderNetworkPhase also logs its transitions now:

[AetherEngine] source network axis flowing -> reconnecting
[AetherEngine] source network axis reconnecting -> flowing

The axis moves a handful of times per session, and when it latched there was nothing in the log that named it, which is what forced the reconstruction from generation counters.

Measurement

Reproduced without a device, on a rig shaped like the reported one: an origin that stops delivering on established sockets without closing them (so the reader parks in a blocking read rather than getting a refusal), brought back, with a seek past the produced window landing on top of the parked pump. That is the restart under test:

[HLSVideoEngine] restart at idx=31: old producer wedged in a read past 5s; aborted it and reopened a fresh demuxer
[HLSVideoEngine] producer restarted at idx=31 (restart took 5033ms; stopWait=5010ms reopen=21ms seek=1ms build=0ms)

Three runs per arm, identical every time:

arm	ticks reading stalled(reconnecting: true)	playing ticks after the swap	phase after the swap
before	24, to the end of the asset	0	never leaves stalled
after	3, the outage itself	21 to 22	playing at the first tick after the restart

In the before arm the clock advances at 1.0x with zero drops through all 21 of those seconds, which is the "Reconnecting over normally playing video" state described in the report.

On the version question

No need to re-run the two arms on a newer pin. The defect is present on today's main, which is 6.47.0 plus the tip: neither 6.46.0 nor 6.47.0 touches this wiring, so the 6.45.0 measurement stands as written. 834fc17c is on main and not in a release yet; a retest is worth doing once it ships in one.

The host-side rule described here (clearing reconnect state only when the phase is neither stalled nor seeking) stays correct and needs no change: a clock that advances over a dead source and a seek that cannot land are both real, and both were the reason the phase exists in the first place.

Thanks for the arm C control in particular. Without a run that recovers WITHOUT a producer restart, "the recovery cleared it" and "the reader never spoke again" look the same from outside.

[superuser404notfound]

Released in 6.48.0 (0c3bb669), so the retest has a pin to sit on. The line to grep for on a recovery that restarts the producer is source network axis reconnecting -> flowing: if the axis latches again, that line is missing rather than late, which separates a silent reader from a slow one.