fix(engine): exports place sound and picture right when a source's audio and video start apart

[miguel-heygen]

What

Exports of a source whose audio and video streams start at different times (MP4, MOV, MPEG-TS) now place picture and sound where the browser preview shows them. That covers the whole clip, every render path (in-process, distributed, HDR layers) and loops.

Scope: MP4, MOV and MPEG-TS sources. WebM and MKV are a known gap that a follow-up fixes: the Matroska demuxer reports start_time 0 for every stream even when the first video packet is 7 s in, so this change cannot see the offset there and those files export exactly as on main.

A user reported an export that lost its first ~7 s of picture and whose sound was wrong. One source property reproduces both symptoms: per-stream start_time values that differ, e.g. a video stream that starts 7 s after its audio. Keyframe-cut screen recordings have this shape, and so do recorders that start audio before the first video frame.

Why

On main, for a 12 s clip with the source played from 0:

source	output second: source second heard (main)	browser
video starts 7 s after audio	0:src7 ... 4:src11, 5:src-2, 6..11: silent	0:src0 ... 11:src11
audio starts 7 s after video	0:src0 ... 4:src4, 5..11: silent	0..6: silent, 7:src0 ... 11:src4

The picture of the video-late source has two problems on main:

• It shows the first frame frozen for 7 s, where the browser shows nothing.
• It freezes on the last frame for the final 7 s of the clip, where the browser keeps playing.

Root cause, in packages/engine:

• Audio (audioMixer.ts, both the <video> and the <audio> path):
  • -ss <mediaStart> -t <dur> went before -i. Even -ss 0 makes the mov demuxer seek the default stream (video) and move every other stream to the keyframe it found, so audio before the video's first keyframe was cut.
  • The extracted WAV has no timestamps and is placed sample 0 at the clip start, so a late audio stream played early by its offset.
• Picture:
  • Frames are sampled with fps=...:start_time=0, which fills the slots before the first frame with that frame.
  • resolvePlayableVideoDuration returned the video stream's own length. Its callers treat it as an end time on the media timeline (the extraction window, the held tail, the loop cycle), so a late stream was cut, or wrapped, lead seconds early.

Related work

None open. The sibling fix for a split scene's in-point lives in the producer's media collector and does not touch these files.

How

• ffprobe.ts: one helper measures how far after the file's media time 0 a stream starts. Media time 0 is the earliest audio/video stream start, which is the browser's origin. It feeds AudioMetadata.latestStreamLeadSeconds and VideoMetadata.videoStreamLeadSeconds, and both come from the probe each path already makes.
• audioMixer.ts: one extractAudioSegment serves both audio paths.
  • When mediaStart is at or past every stream's start, it keeps today's fast input seek, with the same argv and output.
  • Otherwise it reads without an input seek: atrim=start=S:end=S+D,asetpts=PTS-S/TB,aresample=async=1:first_pts=0. This trims by timestamp first and only then pads the leading gap, so memory stays bounded by the clip, not by the stream offset. The trim also goes in front of the rate-lane graph.
• videoFrameExtractor.ts:
  • resolvePlayableVideoDuration now returns the video's end on the media timeline (lead + stream length). The extraction window, the held tail, the loop cycle and frame coverage all read it from there.
  • The final-frame probe converts its stream-relative timestamp by the lead.
  • isVideoHiddenBeforeStreamStart is the one rule for "nothing on screen yet". Both getFrameIndexAtTime (the SDR and distributed lookup) and the HDR layer blit use it.
• Distributed plans (distributed/shared.ts) carry videoStreamLeadSeconds the same way they carry videoStreamStartSeconds. Older plans read it as 0.

Decisions (reversible):

• Before a late video stream starts, export shows nothing, not a frozen first frame.
  • Measured by seeking hyperframes snapshot on sources with a 0.1, 0.5, 1, 2, 3 and 7 s video lead. Chrome paints the upcoming first frame when it is under 1 s away and nothing when it is 1 s or more away (3 s lead: black at 2.00 s, first frame at 2.03 s).
  • The rule copies that, so the common small offsets (a video starting a frame or two late) render exactly as before.
• Audio track selection is unchanged. FFmpeg still picks the track it picks today.

Test plan

New packages/engine/src/services/streamStartOffset.test.ts (real FFmpeg, skipped without it). It builds the sources from lavfi; a tone of 200 + 100·s Hz names each source second.

Sound, checked as a per-second source map:

• <video> and <audio> elements, from media start 0 and 3;
• an MPEG-TS remux (clock starts at 1.4 s, so no stream starts at 0);
• a clip with a rate lane;
• a clip that ends before the audio starts.

Picture:

• no frame while a late video's first frame is ≥ 1 s away, frames after that;
• the extraction window covers the whole media length;
• a loop wraps on the full media length.

Other tests:

• hdrCompositor.test.ts: the HDR blit paints nothing for a late stream before it starts.
• videoMetadata.test.ts: plans round-trip the lead.

Red without each fix (each mutant applied alone, then restored):

fix removed	failing test
lead dropped from the playable duration	"plays a late video stream to its end on the media timeline and loops on its full length"
plan drops videoStreamLeadSeconds	"round-trips a non-zero video stream start and lead"
HDR blit ignores the rule	"paints at 1 s of a video whose stream starts 7 s in: false"
rate-lane graph without the timestamp trim	"video-late.mp4 as <video> from 3 s (rate lane)"
lead not measured from the earliest stream	"hides a late video stream's lead in a transport whose clock starts at 1.4 s"
final-frame timestamp without the lead	"holds a late video stream's last frame past its end"
hidden rule ignores mediaStart	"shows a late video stream from a trim at 6.5 s / 10 s on the media timeline"
HDR blit ignores the extraction start	"paints at 1 s of a clip from 6.5 s whose stream starts 7 s in: true"
whole audio fix reverted (previous head)	7 of the original 8

Runs:

• New tests pass 3 runs in a row.
• audioMixer*, videoFrameExtractor*, audioFxRender, audioVolumeEnvelope, videoFrameInjector: green. ffprobe.test.ts: the only failures are 4 tests that need LFS fixtures my checkout skipped, the same as on main.
• Producer: hdrCompositor, captureHdr* and videoFrameCoverage green (93); distributed/ green (232, bun).

Memory, reading a 12 s window from a file whose audio starts 1 h late:

filter	peak RSS
previous head (pad, then trim)	1.59 GB
this head (trim, then pad)	20 MB

End to end, hyperframes render --format mp4 --fps 30 --quality standard --workers auto on Linux:

source	main	this branch
video 7 s late, 12 s clip	audio 0:src7 ... 4:src11, 5:src-2, 6..11 silent; t=1 frozen first frame	audio 0:src0 ... 11:src11; t=1 black, as the browser
video 7 s late, 27 s clip	v0.8.146: t=20.5 and t=25.5 both frozen on source second 12	t=20.5 shows 13 and t=25.5 shows 18; the browser snapshot shows 18 at 25.5; audio 0..19, then silent
audio 7 s late	audio 0:src0 ... 4:src4, 5..11 silent	0..6 silent, 7:src0 ... 11:src4; picture identical to main
both streams at 0		output file byte-identical to main (same md5)

• Unit tests added/updated
• Manual testing performed
• Documentation updated (if applicable)
• Comments follow CONTRIBUTING.md "Comments"

Not verified:

• Chrome's audio timing for these files is inferred from the container timestamps, not measured.
• The browser's picture rule was measured on paused seeks (snapshot), not during continuous playback.
• HDR: there is a unit test for the blit, and I checked that FFmpeg's HDR raw extraction fills the leading slots the same way as SDR (360 frames for 12 s of a 7 s-late source). I did not render a skewed HDR source end to end.
• Distributed: the plan round trip is tested, but no distributed render was run end to end.
• Not checked on Windows or on the macOS GPU path.
• WebM/MKV: not fixed here (see Scope). FLV-like files that report a stream start but no stream duration are handled in that follow-up too.
• Out of scope, and wrong on main too: a ramped (rate-lane) clip on a source whose audio starts late plays the ramp offset. That is a follow-up.

[github-actions]

Edit accuracy: accurate 2061 (base branch 2061), smooth 1572 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)