feat(sdk): pure scale-limit sizing helpers and a segment-integrity ADR (DSPX-4650) - #1020
Open
dmihalcik-virtru wants to merge 1 commit into
Open
feat(sdk): pure scale-limit sizing helpers and a segment-integrity ADR (DSPX-4650)#1020dmihalcik-virtru wants to merge 1 commit into
dmihalcik-virtru wants to merge 1 commit into
Conversation
Item 0 of DSPX-4648: fix the per-segment integrity algorithm before any of the 50 TiB limits are derived from it, and extract the size arithmetic those limits need into pure functions. Decision (spec/DSPX-4650-segment-integrity-algorithm.md): per-segment integrity stays GMAC -- free, since getSignature just returns the AES-GCM auth tag the cipher already computed -- while the root signature stays HS256. No default changes. Every limit is nonetheless derived for HS256, the more expensive case at 56 vs 36 bytes per manifest segment entry, so moving the default later loosens nothing and needs no re-derivation. Adds lib/tdf3/src/utils/scale-limits.ts: base64Length, perSegmentEntryBytes, finalSegmentEntryBytes, segmentCountFor, estimateSegmentsArrayBytes, estimateManifestBytes and minSegmentSizeFor. All pure, all functions of sizes only, so 50 TiB behaviour is reachable in a unit test rather than only by encrypting 50 TiB. Nothing calls them yet; this is behaviour-neutral. The constants are pinned against the real serializer rather than a hand-built entry: tests/mocha/helpers/write-tdf.ts drives writeStream with no KAS and no network, and the specs measure what a 5th segment actually adds to an emitted manifest for both GMAC and HS256. Perturbing any entry-cost constant by one byte fails 13 of the 19 tests. Also moves the plan document into spec/ as DSPX-4648-web-sdk-large-files.md. DSPX-4650
|
Warning Review limit reachedNext included review available in 59 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
dmihalcik-virtru
added this pull request to stack #1033
September 10, 2026 20:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Item 0 of the 50 TiB workstream. Two things, both prerequisites for everything after them, neither of which changes any behaviour:
spec/DSPX-4650-segment-integrity-algorithm.mdrecords that per-segment integrity stays GMAC and the root signature stays HS256 — the current defaults — and why.lib/tdf3/src/utils/scale-limits.tsaddsbase64Length,perSegmentEntryBytes,finalSegmentEntryBytes,segmentCountFor,estimateSegmentsArrayBytes,estimateManifestBytesandminSegmentSizeFor. All pure, all functions of sizes only. Nothing calls them yet.Also moves the plan document into
spec/asDSPX-4648-web-sdk-large-files.md, matching the existingspec/DSPX-*.mdconvention.Why the ADR blocks the rest
The manifest carries one
segments[]entry per segment, and the digest in that entry is the only part of the manifest that scales with payload size:{"hash":"…"},)At 50 TiB with 16 MiB segments that is 118 MB of manifest versus 184 MB — a 1.56× factor on the single quantity the whole plan is trying to bound. Picking a segment size, a manifest cap, or a maximum encryptable payload before fixing this constant means deriving all three twice.
The decision: GMAC is free —
getSignature()(lib/tdf3/src/tdf.ts:369-372) returnscontent.slice(-16), the AES-GCM auth tag the cipher already computed. HS256 is a separate HMAC-SHA256 pass over every segment's ciphertext: at 50 TiB, 50 TiB of extra hashing on encrypt and another 50 TiB on decrypt-verify, competing with the encrypt for the same WebCrypto queue. It buys an integrity value nominally independent of the cipher, butgetSignaturekeys the HMAC from the same unwrapped key, so that independence is weaker in practice than in principle. Not worth it per-segment; kept for the root signature, where it is computed once over the concatenated segment hashes rather than over the payload.But every limit is still derived for HS256.
perSegmentEntryBytes(alg)takes the algorithm as a parameter, so the limits are a function of it rather than of the shipped default. The landing point in item 1 (16 MiB segments, 256 MiB cap) is chosen to fit 50 TiB at 56 B/entry; running at 36 B/entry just leaves headroom. If the default ever moves, nothing needs re-deriving and no limit silently loosens.Why extract the arithmetic
Whether a manifest will fit, how big a payload may be, what segment size to pick and how much to prefetch are all pure functions of sizes — none of them need a byte of payload. Separating them means the behaviour at 50 TiB is reachable in a unit test instead of only by encrypting 50 TiB. That is the whole testing strategy for DSPX-4648, and it only works if the estimate provably matches what the serializer really emits.
How the constants are pinned
Not by asserting on a hand-built entry — that would just restate the bug.
lib/tests/mocha/helpers/write-tdf.tsdriveswriteStreamdirectly with mock keys, no KAS and no network, so a real container can be written with a 64-byte segment size. The specs then measure deltas against real emitted manifests, for both algorithms:segmentsarray minus a 4-segment one equalsperSegmentEntryBytes(alg)exactly;segmentSize/encryptedSegmentSize) costsfinalSegmentEntryBytes(...)exactly;estimateSegmentsArrayBytesnever under-estimates, swept over[0, 1, 63, 64, 65, 127, 128, 200, 640]bytes.If the serializer ever changes shape — a renamed field, a new per-segment key — those fail rather than the estimate silently drifting. Verified red-first: perturbing
SEGMENT_ENTRY_OVERHEAD_BYTESby one byte fails 13 of the 19 tests.The estimate is deliberately an over-estimate: every entry is counted with its separating comma even though the last has none, which pays for the closing bracket with a byte to spare. Under-estimating would let a doomed encrypt start.
The numbers
A table-driven block asserts the plan's sizing directly, at 50 TiB, in milliseconds:
minSegmentSizeFor(50 TiB) === 12,800— the floor imposed byMAX_GCM_INVOCATIONS_PER_KEYfrom #1018. Every segment size under consideration is orders of magnitude above it, so the IV ceiling is not the binding constraint, but it is the floor any future default has to clear.How to test
19 passing, ~30 ms. Full suite: 410 passing, 0 failing.
Risk
None to runtime behaviour — no existing code path calls any of this. The risk is that the estimator is wrong, which is what the serializer-agreement tests are for.