Skip to content

perf(rust): reuse the encoder input and apply the residual in place - #27

Open
seo-rii wants to merge 2 commits into
fix/rust-wide-image-residualfrom
perf/rust-encode-preprocessing
Open

seo-rii wants to merge 2 commits into
fix/rust-wide-image-residualfrom
perf/rust-encode-preprocessing

Conversation

@seo-rii

@seo-rii seo-rii commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Stacked on #14 (fix/rust-wide-image-residual). Review only b892e14. It merges cleanly with #13, and the merged tree passes all tests.

Problem

Trustmark::encode did the same full-resolution work twice, and apply_residual made more full-size float copies than it needed:

  • encode cloned the full-resolution image twice and ran crop → resize → normalize twice to build the same input tensor: once for the encoder and once just to subtract it from the encoder's output. With the antialiased resize in fix(rust): antialias downscaling to the model input size #13, that duplicate pass costs more.
  • apply_residual converted the input to RGBA32F, cloned it, converted the upscaled residual to RGBA32F, converted the result again, and then copied it in with imageops::replace. That kept several full-size float images alive at once.

Change

The public API is unchanged.

encode:

  • Preprocesses the input once, from a borrowed image, through a new TryFrom<&ModelImage>. The old owned conversion now delegates to it, so decode is unchanged.
  • Passes the input tensor to ORT by reference and reuses it for the residual subtraction.

apply_residual:

  • Converts the input once (RGB32F, or RGBA32F when it has alpha), consuming it.
  • Upscales the residual once with the same Triangle filter.
  • Adds the residual in place to the three colour channels and leaves alpha alone.
  • The float operations and their order are exactly the same as before.
  • Full-size float buffers drop from about 84 bytes per pixel to 24 for RGB inputs, and from about 76 to 28 for RGBA.

Bit-for-bit equivalence

A scratch harness hashed the output pixel type and all of its raw f32 bytes, then decoded the result. All 53/53 cases were identical to the base branch and decoded to the right watermark:

  • Q, P, B and C on ghost.png (RGBA), ufo_240.jpg, ripley.jpg, a 3672×2448 upscale, 4032×1500 (wide), 1500×4032 (tall) and a 3072×3072 RGBA upscale;
  • Q on a 5472×3648 (20 MP) image;
  • Q and B with L8, La8, Rgb16, Rgba16, Rgb32F and Rgba32F inputs.

Measurements

Release build, variant Q, base and new runs interleaved with alternating order. The machine was shared and heavily loaded; the cleanest pass ran at a load average of about 8–9.

Image Encode, base: median (range of round medians) Encode, new: median (range)
9 MP 508 ms (449–955) 333 ms (301–738)
20 MP 1022 ms (942–1328) 588 ms (549–656)
  • The new build was faster in every paired round for the large images: 8 of 8 in this pass, and 19 of 20 in earlier, heavier-load passes.
  • Small images (ufo_240, ghost) show no measurable change. The expected saving there is under 1 ms.

Peak memory during encode, measured from a warmed-up model with VmHWM reset just before the call (stable across runs):

Image Base New
20 MP RGB ~1210 MB ~466–482 MB
9 MP RGB ~606 MB ~262–274 MB
9.4 MP RGBA ~607–614 MB ~283–325 MB

Tests

Not run: the criterion encode bench, and timings on the tree merged with #13.

🤖 Generated with Claude Code

`Trustmark::encode` cloned the full-resolution input twice and ran the
crop/resize/normalize preprocessing a second time just to subtract the
same tensor from the encoder output. `apply_residual` then converted
and cloned the input to RGBA32F several times, converted the upscaled
residual to RGBA32F, and copied the result back into yet another float
image.

Build the model input once from a borrowed `ModelImage` (new
`TryFrom<&ModelImage>`; the owned conversion delegates to it), pass the
tensor to the encoder by reference and reuse it for the residual.
Convert the input to RGB32F or RGBA32F once, consuming it, and add the
upscaled residual to its color channels in place, so at most two
full-size float images are alive.

The float operations and their order are unchanged, so outputs are
bit-identical; a unit test pins the residual arithmetic, alpha handling
and output formats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 21, 2026 04:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants