SegmentCache.init's stale-session sweep deletes a concurrent >1h session's directory — the age check has no liveness half

[cmcpherson274]

A code-read finding, disclosed before we have reproduced the failure end-to-end — filing it because concurrent engines make it reachable and the sweep's own comment says it was designed for a world without them.

The read

SegmentCache.init sweeps sibling session directories under <tmp>/aether-segments/ (Video/SegmentCache.swift:107–124 at 6.56.3): every entry that is not the current session and whose directory creation date is nil or older than one hour is removed. The comment above the baseDir says the prefix "lets sweepStaleSessionDirs() find sibling dirs from crashed sessions" — with one engine per process at a time, the sibling is always dead, and the sweep is exactly right.

With two engines alive in one process the same predicate reads differently: the directory's creation date is the session's start time, so any session older than one hour is indistinguishable from a crashed one. A viewer an hour into a film (or a long live timeshift hold) whose host then constructs a second engine — a preview tile, a second surface, even a quick failed tune that gets as far as init — has their segment directory deleted underneath the running session.

What we have not done

We have not run the two-engine >1 h scenario to observe the failure mode (whether store() fails soft, whether an .alwaysMapped read faults, whether playback just re-fetches). Our surfaces tear engines down well inside an hour today, so this is ahead of us rather than behind us — but the N-surface direction (the preview tile shipped today, per the other issue) makes it a matter of time.

The ask

A liveness check next to the age check. In-process is enough: a process-global registry of live session IDs (the sweep already knows its own), so the predicate becomes "not mine, not live, and older than an hour". A crashed sibling still sweeps exactly as today.

[superuser404notfound]

The read is right, and the scenario reproduces as soon as it is set up rather than waited for: backdate a live session's directory past the cutoff (its creation date is the session's start time, so an hour of playback does the same thing), construct a second cache against the same base, and the running session's directory and every segment in it are gone. That is the first of the seven tests in 2254067c on fix/ae451-sweep-liveness.

The failure mode is worse than a re-fetch, which is the part worth having measured before this ships anywhere near an N-surface host.

The bookkeeping outlives the file. entries[index] still names the deleted segment, so mediaSegmentURL hands the server a URL, the server takes the file-backed path, stats nothing, and answers 404 for an in-range VOD index. AVPlayer treats that as terminal loadFailed. The #50 in-range-is-never-404 rule was only ever enforced in the data path (classifySegmentResponse); the file path had its own fileSize == 0 branch that went straight to 404. So the sweep does not cost a re-fetch, it ends the item. The same lie propagates to every other consumer of "is this stored": the AE#421 split that tells a starved consumer from a silent one, and the byte ledger.

Two halves, both in that commit:

Liveness. Each cache holds an flock(2) on <sessionDir>/session.lock for its whole life, and the sweep skips a candidate whose marker is still held. The age check stays as the outer bound, which also closes the window between createDirectory and taking the lock.

The reason it is a file lock and not the in-process registry suggested here: flock locks belong to the open file description, so a second cache in the same process is refused exactly like one in another process, and the kernel drops the lock when the process dies, so a genuinely crashed sibling sweeps precisely as it does today. A registry gets the in-process case only, and NSTemporaryDirectory() is per user for a non-sandboxed process, so two aetherctl runs (or a run alongside a non-sandboxed host) sit in the same base directory and can do this to each other. A directory with no marker at all is still decided by age alone, so nothing about the crashed-sibling path changes.

The bookkeeping. An entry is now checked where it is redeemed: peekURL and fetch drop an entry whose file has gone (and the ledger with it), the file path is bound to the same classification as the data path so an in-range index gets a retriable 503 and never a 404, and a store into a directory that vanished restores the directory and re-takes the marker instead of leaving the session permanently unable to write. Without the drop, the 503 would just repeat forever: the producer asks whether the segment is stored, and the stale entry keeps answering yes.

Two of the tests are the ones that would fail if liveness had been read too broadly: a crashed sibling past the cutoff is still swept, and a marker file that no one holds does not protect anything.

Not in a release yet.

[superuser404notfound]

Released in 6.56.4 (31f3a5a0), so this no longer needs a main checkout to retest. The fix is 2254067c, merged at eeda5d3f.