Repository navigation
Conversation
Every constructor created both ONNX Runtime sessions, so a process that only embeds or only reads watermarks still read both model files, paid for both sessions at start-up and kept both in memory. Add `Trustmark::encoder_only(models, variant, version, options)` and `Trustmark::decoder_only(models, variant, options)`. They read only the one model file and create only its session; `decoder_only` takes no `Version` because decoding reads it from the watermark. `decode` on an encoder-only model returns the new `Error::DecoderNotLoaded`, and `encode` on a decoder-only model returns `Error::EncoderNotLoaded` before the payload is checked against a version it was never given. `encoder_provider()` and `decoder_provider()` now return `Option`, with `None` for a session that was not loaded. `new` and `with_options` are unchanged and still load both. Measured on CPU (variant Q, 16-core x86-64, default options, 10 fresh processes each, medians): loading both takes 164 ms and peaks at 141 MiB RSS (293 MiB after one encode and decode); encoder only 69 ms / 52 MiB (144 MiB after one encode); decoder only 123 ms / 84 MiB (173 MiB after one decode). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Trustmark::newalways creates both the encoder and the decoder session. An embed-only or detect-only service pays for the model it never uses.Change
Trustmark::encoder_only(models, variant, version, options)andTrustmark::decoder_only(models, variant, options).decoder_onlytakes noVersion.newandwith_optionsare unchanged.Error::EncoderNotLoaded/Error::DecoderNotLoaded.encodechecks for the encoder before validating the payload against a version that a decoder-only instance never received.encoder_provider()/decoder_provider()now returnOption<&ProviderInfo>. This changes a signature introduced, and not yet released, in the base PR.loadEncoder/loadDecoder, added in perf(python): load only the networks in use (loadEncoder/loadDecoder) #19.Measurements
CPU, 16 cores, Q, default options. Medians of 10 fresh processes. Peak RSS is VmHWM; a process uses about 20 MiB before loading.
Tests
--features cudaand no GPU, 30 pass and 3 are ignored.--include-ignored.-D warnings(default features and all features),cargo doc,cargo publish --dry-run --lockedand MSRV 1.88 all pass.🤖 Generated with Claude Code