Repository navigation
fix(studio): timeline composition clips show each tile's own frame instead of one repeated poster - #5046
Draft
miguel-heygen wants to merge 21 commits into
Draft
fix(studio): timeline composition clips show each tile's own frame instead of one repeated poster#5046miguel-heygen wants to merge 21 commits into
miguel-heygen wants to merge 21 commits into
Conversation
…stead of one repeated poster
Edit accuracy: accurate 1437 (base branch 2055), smooth 1233 of thoseThe gate fails: the passing count fell from 2055 to 1437. In the base grid but not run (618)
Quarantined, measured but not gated (0) |
…band; strips wait out zooms
…eline scrolls or zooms
…r, holding frames until the next
… for a poster first
…rn the frame shape
…unk by the thumbnail URL builder
…edits keep a clip's frame shape
…nce per frame change
…next-open poster render
…ead of rebuilding them each frame
…che scans that cannot change anything
…l lease per chunk per clip
…recheck at commit
…etween render and commit
Fallow audit reportFound 42 findings. Duplication (41)
Health (1)
Generated by fallow. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A composition clip's timeline strip now shows, in each tile, the frame the composition renders at that tile's time. Before, every tile repeated one poster frame, so a ten-second day-to-night scene looked like ten copies of mid-morning.
The picture also fills the clip's full height: the name now sits on the frame with a shadow that reads on light and dark frames, instead of on a black band above it. While the frames load, the clip shows its own fill colour and its name, with no grey pulse.
Why
CompositionThumbnailbuilt one poster URL (atseekTime + duration / 2) and every tile rendered that same image. Video clips already map tiles to distinct frames; composition clips never did.How
planCompositionStrip): each tile's centre maps onto a time grid whose step is the largest power of two no longer than one tile, measured from the tile's real width. Neighbouring tiles therefore always get different frames (down to 1/32 s) and each frame falls inside its own tile's time span. The last tile, which can run past the clip's end, shows the clip's last frame. Chunk boundaries come from the clip's range and the grid step alone, so zooming within one power of two or scrolling asks for exactly the same chunks again.ThumbnailTiles), so only their chunks are requested. A chunk is oneGET .../thumbnail/<comp>?times=a,b,..., and the server renders its times in ascending order in one generation turn, so the adapter's forward-only warm page serves every frame without reloading. It returns them side by side in one image (sharp, already a studio-server dependency). A malformedtimes, ortimeswith PNG, is refused with 400 rather than quietly rendering one frame.background=1). It queues behind all visible work, and if it is already rendering when a visible thumbnail arrives, it is stopped and queued again behind that work, so it still finishes afterwards. The render slot stays taken until the stopped render has actually let go of the browser, so two renders never overlap. Visible work never interrupts other visible work. Before, a clip's first frames could wait for that whole render.generateThumbnailFramesthat loads the composition once and seeks and captures every time in that page. The dev server implements it (it opened a page per frame before); the CLI preview server already reuses one warm page across a strip's ascending times, and an adapter without the method keeps rendering frames one by one.perf-ceilings.jsongo up by exactly that, with the reason recorded.revisionparameter, which every chunk URL carries.is-composition; their content starts at the top of the clip, and their name drops the band background and uses--timeline-thumbnail-label-shadow. That token was a single 1 px drop shadow, which barely showed on near-white frames, so it is now a tight dark halo plus the drop shadow. Video, image and text clips keep their band.times, ortimeswith a source-size output (format=pngoroutput=source), is refused with 400.Measured
CI, the "Studio: timeline viewport gate" job, each arm on one runner: main at 184d254 (run) against this PR at 7fa8fb0 (run).
That fixture's clips are plain elements (its default profile), so it measures the timeline's render path at scale rather than composition strips.
Opening the 12-scene work-count fixture (same CI jobs, counted until thumbnail work goes quiet):
Server time per thumbnail request on the Studio dev server, one open of the same fixture on a shared Linux box (request start to finish, so it includes waiting behind the other in-flight request): main's poster median 10.9 s (12 requests); this PR's 3-frame strip median 11.8 s (12 strips, all finished, none aborted). The strip's two extra frames cost about 0.9 s because they come from the same page load.
A visible clip's frames requested while the next-open poster renders, on the Studio dev server with the same fixture. The poster request starts, and one second later the clip's frames are requested the way each arm's timeline asks (main: one frame; this PR: a 3-frame strip). Three rounds per arm, all within 0.1 s of each other:
Against the 8 ms per-frame bar: the CI gate reports frame intervals, not per-frame main-thread work, so per-frame work comes from a bench of scroll, zoom and playback on the CLI preview server, with the strip fixture, on a shared 8-core Linux box (load 3.5 to 8). Five alternating main / this PR rounds at this head's tile code; the zoom pass steps the zoom slider from max to min in 21 steps:
Scroll and playback are level with main. Neither side meets the 8 ms bar on this box. The remaining zoom cost is the feature itself: tiles change frames as the zoom moves (about 22 ms more style and paint per pass), strip loads from the scroll before it land during the zoom (about 28 ms of IPC and 17 ms of garbage collection), and about 23 ms of script. At this PR's first strip version the zoom median was 6 frames on main against 11 here. Giving every tile its own compositor layer was tried and measured worse (54 frames over 8 ms against main's 37 in five rounds), so it is not in this PR.
Before / After
Same fixture built for this PR (a 10 s "Day" scene from a near-white dawn to night, a 120 s "Long" one, a 10 s "Night" scene from night to dawn), same zoom, same viewport, current headless Chrome. "Loading" is taken with every thumbnail request held back.
Before
Loading: the black name band over a grey fill.
Loaded: one repeated frame per clip, under the band.
After
Loading: each clip's own fill with its name, no band, no grey.
Loaded: each tile shows its own moment, filling the clip's height; the names read on the near-white Day frame and the dark Night frame.
Test plan
CompositionThumbnail.test.ts: an 8 s clip in a 500 px strip of 71 px tiles asks for one strip at 0.5 s ... 7.5 s; each tile shows the frame inside its own span ([0, 1, 2, 3, 5, 6, 7, 7], the last tile running past the clip), at its own cell and one frame's aspect. It fails on main, where every tile is the same poster, and it also fails if the span is taken as range / tile count. Planner properties over five clip shapes: each frame falls inside its tile and is later than the previous tile's, chunks hold at most 8 ascending times, and every zoom inside one power of two asks for identical chunks.thumbnail.test.ts:times=3,1,2renders 1, 2, 3 in that order and returns one 3-cell image, with each cell holding its frame's colour. Malformedtimes, more than 8 times, ortimeswith PNG gets a 400 and renders nothing. A strip or a single frame of a composition three folders deep is cached under a short name. All of these fail on main's route; the single-frame long-path case is the existing bug.thumbnailGenerationCoordinator.test.ts: queued single frames start before queued strips, and strips before the background poster, each kept in order. It fails on main's first-in-first-out queue. A strip arriving while the background poster renders stops it, starts only once the stopped render returns, then the poster runs again to completion (fails with preemption removed, and fails if the slot is freed before the stopped render returns); a single frame arriving while a strip renders waits for it.CompositionThumbnail.test.ts: a tile keeps its shown strip leased while the next one loads, so it is never released and re-fetched; a clip one tile wide shows its middle frame.TimelineClip.test.tsxandtimelineMotionStyles.test.ts: only composition clips carryis-composition, their content starts at the top, and their name has no band and uses the shadow token.CompositionThumbnail.test.ts: nothing is drawn over the clip's fill while the poster loads (main draws the grey pulse). Tiles exactly one frame wide are filled edge to edge, so neighbours meet without a seam, and it fails if every tile letterboxes. A portrait frame in a tile held at the 48 px minimum width letterboxes at its own aspect.CompositionThumbnail.test.ts: a clip asks for nothing before it has a width and then for its strip alone (no poster), and a tile keeps its last frame until its new chunk arrives; both fail with the poster-first tile.CompositionThumbnail.test.ts: a tile whose strip fails shows the poster, and an edit that turns the frames portrait re-tiles the clip at the new shape; each fails with its fix removed.vite.browser.test.ts: a 3-frame strip opens one page, loads it once and seeks to 1, 2 and 3 in order.thumbnail.test.ts: an adapter withgenerateThumbnailFramesgets one call with the ascending times and no single-frame calls.thumbnailScheduler.test.ts: a lease taken before a failure sees the retried result after the quiet period (fails on main's replace-the-entry retry).CompositionThumbnail.test.ts: a portrait clip keeps its 48 px tiles while an edit's frames load.thumbnailScheduler.test.ts: a rich load waits whilenoteMotion()keeps arriving and starts once the timeline has been still for 150 ms.CompositionThumbnail.test.ts: a clip whose 8 tiles share a chunk takes one lease on it (16 with per-tile leases), and a priority change moves that lease instead of taking another.src/player,src/hooksandsrc/components: 6118 tests pass, three runs in a row; studio-server routes: 527 tests pass, three runs in a row.Size
About 1,450 changed lines, over the usual 1,000. The strip, the render order, the retry fix and the zoom work are kept together because each one is needed for the others to behave, and they ship as one change.
Known limits
The next-open poster is best effort. If visible thumbnails keep interrupting it for longer than the client's 30 s thumbnail limit, that render is dropped, and the poster is rendered on the next open instead.
When zooming crosses into a new power of two, a tile shows its previous frame (now at the wrong time) until its new chunk arrives. So does a trim that moves a clip edge into another grid cell.
A composition whose frames are not 16:9 asks for its first chunks twice on the first open: once planned at 16:9, once at its real shape.
With a tile between one and two grid steps wide, up to half of a chunk's frames are not shown at that zoom; they are what the next zoom step shows.
A drilled-in element clip whose box changes size over time has every frame fitted to the first frame's cell.
A composition clip longer than its composition asks for times past the composition's end; the frame shown is whatever the composition renders there on its own.