Repository navigation
ci(tls): investigate native retention qualification failures - #219
vincentkoc wants to merge 78 commits into
Conversation
Checkpoint the reviewed TLS ownership repair before integrating current main. Native transport takeover, completion custody, shutdown joins, and regression coverage are retained together. The generic TLS retention case still needs qualification; no landing claim is made.
* commit '191755ec115ece3dd642eba3cb0143cecb5e5d7a': fix(tls): enforce current trust before session resumption (#213)
|
Current diagnostic handoff at The latest Darwin capture, artifact13065901291, has verified archive integrity: all 30 manifest members match, and all four captured controls match this head. GC32 completed, but both dropped objects were collected, the watched word remained zero, and there were no changing writes. Sealing stops at line547 because the target is absent from the snapshot. The later original-failure-preservation and writer-join gates at lines599–605 never execute. Both causal proof flags remain false; source restoration and cleanup succeeded. This probe does not establish the startup retainer or a scanner defect. Linux ARM64 has separate useful evidence: writer capture41233926703 observes a physical deinit stack store followed by an accepted conservative word, while explicitly leaving earlier source-variable/register origin unproven. Repair qualification41235963876, artifact13055982502, has verified outer/nested digests and all 25 manifest members. Its six XML reports contain 283 passes, eight skips and zero failures, including both original retention regressions. This supports the narrow absent-config guard, rather than proving the earlier heap-payload producer. Runtime PR215 at Next owner step: preserve the diagnostic route and original assertions, obtain original-failure-preserving producer/lifetime attribution on those two targets, then repair and repeat the affected native qualification. The latest first-change-context path was not exercised. Earlier successful diagnostic capture for The earlier destroyed-owner admission gap is resolved by the successful capture run. Runtime and fixture bytes remain unchanged. Artifact The capture observes the actual WeakRef target matching the destroyed late callback with null native data, callback return, and 31 subsequent synchronous GCs in the matching VM. The owner's real mark byte changes The direct root category is |
* ci(tls): reuse matched symbols from the retained ARM64 build * ci(tls): inspect retained symbols with LLVM 23
Reuse the immutable shipping-debug artifact, capture the actual unchanged fixture child and trace its matching weak target through the late callback and final explicit collection. Dump verifier marker data only after completed marking; keep raw path interpretation separate from runtime fixes.
Retain entry identities, raw hit counts and final owner/GC correlation on diagnostic failure without changing the capture gates.
Stack reconciliation
The transport/completion repair in #215 landed as d9be087. GitHub retargeted this diagnostic branch to main and exposed a squash-ancestry conflict. Signed merge 75c46a1 joins that main commit without rewriting the diagnostic history. The resulting diff contains only the 27 diagnostic controls, byte-identical to their previous head d8cb272. Production source and tests now match main. Existing exact-source native proof remains valid; it is not relabelled as merged-head evidence. Redundant diagnostic workflows triggered by the merge are canceled; no completed probe is rerun.
Completed canonical native replay
The unchanged D4 fixture now completed all six canonical files on both affected production hosts. macOS x64 run 41268294394: 282 passes, eight existing skips, one unchanged D4 failure, zero errors. Windows ARM64 run 41269000950: 280 passes, ten existing skips, one unchanged D4 failure, zero errors. Both completion regressions pass. All 291 cases on each host are accounted for; all five remaining suites pass. Source 510fef5, engine b82bc917e99110b61c3a21202d6def7457aa194c and original fixture SHA-256 d4c04a72b89bc7d33c292a16a57c38f5da540815dc2e141adfd58c1bd9973af2 remain pinned.
macOS proof artifact 13068122421 / ZIP 133233c20b3d6dfe376bd9dc92d47cc61445ad28c8d266d203ab39e98d869e38; Windows proof artifact 13068940537 / ZIP a61a0711efe13653f0ff9340617846d67c660276cd7fea1840e24a194a134816. Native identities, safe ZIP entries, CRC, all manifest records, archived controls, source/index restoration and raw XML were verified.
The first Windows replay stopped before native execution at stale Git index stat metadata. Signed controller d8cb272 refreshes stat metadata before the unchanged clean-index gate and retries only Windows. It does not stage content. Its scoped review found no P0-P2 issue. Neither replay masks the D4 assertion: both jobs remain red with true/false/true. This closes the missing-suite coverage gap for the partial transport/completion repair in #215. Physical retaining-owner attribution remains unresolved here; older pending statements below are historical.
Problem and scope
Investigate the two remaining native TLS collection failures associated with #215 using unchanged assertions and immutable shipping artifacts. This PR contains diagnostic controls; PR215 owns the runtime repair. #220 is merged here and supplies the saved-artifact reader. No release, publication or consumer pins change.
Complete canonical native replay
Run 41268294394 at signed controls
8ebb643978cb5e1f4fc71c8ab77c7a66174272f5reuses the original510product binaries and unchanged D4 fixture on macOS x64 and Windows ARM64. It collects all six canonical suites before reporting failure, closing the five-suite coverage gap left by first-suite fail-fast. It removes the experimental8918fixture substitution. Native results are pending; the existing D4 assertion stays strict and red when it fails. Other four targets retain their existing exact-tree proofs. Independent P0–P2 review: scoped-clean, 0.83; Python/YAML syntax and diff checks passed. No runtime code, test assertion or published pin changes.The unchanged D4 assertion already fails on pre-PR shipping
191755/ engineb82on macOS x64 and both Windows hosts in baseline run 41254624952: true/false/false. Exact pre-guard source454a78a4d3b0202ecdbedf7dd051fc84ffc59e9ealready fails on macOS x64 and Windows ARM64 with the current true/false/true signature in run 40897692970. Both completion regressions pass. Both small original XML ZIPs were independently verified, including their embedded full commit, safe entries, CRC and SHA-256:12909164077/ed996fc58953bb9e8db17f9df758f08415c26d9181342ee9b02dd42c62663f35;12909298742/8499ed51deab631648cdca0de898bdd6482f52c0f7bb2fe9cec892efba74ca99. This establishes an inherited failing predicate, not an identical retaining mechanism. PR215 is a partial transport/completion lifetime repair; this diagnostic PR retains the unknown physical-retainer investigation.Merge status
Held. Six-target matrix 41238111110 built all six artifacts from signed source
510fef558b464c58de1209298f638fa476d03eef; both Windows compatibility jobs passed. The six canonical suites passed on Linux x64/ARM64 and macOS ARM64 (283 passes/eight existing skips each), and Windows x64 (281 passes/ten existing skips).macOS x64 and Windows ARM64 still fail
TLS Duplex cycles collect after startup while a retained origin keeps its wrapper alive: expected retainedAlive/droppedCollected/releasedCollected = true/true/true; observed true/false/true. Each first suite has nine passes, one skip and this failure; the other five suites did not run. Both completion regressions passed. All six XML ZIPs and reports were verified. Failed XML artifacts: macOS x6413057222077, SHA-256f861e28e80493a0735ae0b7c76ce8a1b5e99906671f8029354700009573fa60f; Windows ARM6413056908293, SHA-2561e5ebf80850ae6ab76d987a0810db47600cf7ae0d55ddb5302d471dc2ae69086.PR215's
670b23frestores the complete510 source tree and original checkpoint after reverting experimental fixture relocation8918. The paired checkpoint experiment showed frame/job-phase sensitivity, which does not identify a retaining writer. The experimental replay passed four hosts; both Windows hosts failed initial Git cleanliness admission before execution. Neither is a six-target pass or a basis for weakening the original fixture. Bun's GC evidence rule applies.Established Linux ARM64 repair
Shipping-setting reproduction and writer-to-root capture established a stale inactive-config payload copy and real conservative root on source454/engineb82, preserving callbackRoot:true and one ECANCELED settlement. Capture artifact
13054827777and all16 manifest entries were verified.PR215 guards both absent-config clears while preserving live destruction and StartTLS move-before-callback ordering. Native shipping-profile qualification passed283 cases/eight skips, including both collection regressions. Its runtime-file SHA-256
2a40a7df44cfbc9344eb92ced128da282ade137fe2a08f5319d85b23c297dfb0matches510. Artifact13055982502, ZIP SHA-25689e13e7c8367fdade283db9ecfb558cb1294517cfa117fe8e7e8c0c87a9d1287:25 manifest entries and six XML reports verified.Independent optimized ARM64 disassembly verifies the None branch bypasses all192-byte payload loads/spills; Some preserves its payload and drop. Artifact
13056802915, ZIP SHA-256506778c06ee69bea7332832f72d760195b4eb55ce5a4dd7063f5f75d1c5c1698: every manifest entry and both controls verified. This does not prove other platforms' retaining paths.Darwin x64 physical evidence
All Darwin probes admit the actual matrix-consumed artifact
13057470428:232,201,333 bytes; ZIP SHA-256a4d8b786d7bb87d166f757e089ac1705f4ac0701504038ebbecd438c9d6e06d6. Profile SHA-25652f2048920d8b31ef4471e121cc06ad7b4828761a2c3278349f66ed9b4acaf2b; dSYM SHA-25645aa427a18e5f97dda7a9fb23641ee066ecc5084de26d121b54530f62c41b0c6; shared UUID4c4c4432-5555-3144-a125-d2b0a4297913. The unsigned dry-run x64 contract is admitted without resigning bytes. A profile is not the production executable.Production heap capture41247950843 preserved the original failure and identified dropped Duplex6696/native TLS6776 in a68-node SCC. Its69-node backward closure has only environment492 outside the cycle, via saved-register names7 and4/8; environment492 has no recorded incoming edge or roots entry. No ancestor reaches any431 recorded roots or synthetic0. Snapshot SHA-256
20d517e1fa8970fff15b268f25fbabaf1e5472f788c22055e635c68295bff01d. The snapshot performs an additional full GC and does not alone establish the original checkpoint's physical root.Synchronous physical capture41251044278, controls
d8abd7aa122ad36845872807fab26b92721c5764, passed. Artifact13061207699:365,694 bytes; ZIP SHA-256771d55dbc9ad07756ce33bd360fa1f470638cca41f502e593a4ebd0caa195dd6; all29 manifest entries and four controls verified. Both selected XML controls and original/tagged/debugged direct children preserve true/false/true; source restoration, exact profile identity, normal ASLR and advancing synchronous stop IDs were verified.Independent raw reconstruction finds one accepted stack word holding environment492 (
0x42a13c4c900) at raw-span offset1024. It equalsexecuteModuleProgramSP / FP minus224 atInterpreter.cpp:1839, called byJSModuleRecord::evaluate:276. Its bitmap word changes0→65536 at bit16 with CF0: a new mark. After original GC32 completes, the same process emits snapshot SHA-2569e132d96600b004d89b60af5fc721a2c1dba2cefe04b487933f9c42410efbd69, identifying dropped Duplex6702/native TLS6782 with the same68-node SCC/69-node closure shape and sole external environment492. No ancestor reaches recorded roots.This establishes a concrete accepted word and new mark in the matched profile. The graph is from a later snapshot GC; imports, tags, allocations, profile code and debugger stops are observer effects. Source declaration order, zero graph edges and SP equality do not establish a scanner defect, local ownership or writer.
Static writer inventory41252056087 passed. Artifact
13061219282:98,020 bytes; ZIP SHA-25673259c3a04df23c4b8373cabcdc9ec46c4489310508f9297b4bd8d4979ee4cf3;11 manifest entries/two controls verified. The244 physicalexecuteModulePrograminstructions allocate224 bytes below FP, placeProtoCallFrameat FP minus216, and call VM entry at0x103cf8527(return0x103cf852c). The observed word is eight bytes before that object. No writer follows from proximity. Assembly labels fragment the VM-entry symbol listing; its six instructions are not a full VM-entry disassembly.Supporting metadata and static proofs: Darwin cleanup disassembly, AST layout witness, scanner/mark inventory. Reconstructed layouts do not prove optimized locals. Numeric Variable names identify saved generator/async virtual-register slots, not physical offsets: producer, marking/graph owner.
Writer probe: original failure not preserved
Signed controls
97b715d73c1349b302394206a426bc3e63412a3ereached GC32 and owned cleanup in native run 41261815813, but the required original-failure assertion stayed red. All four controls retained true/false/true; the debugger arm again produced true/true/true, and both dropped tags were absent. This time the admitted GC30 word was already zero: no value changes occurred, so the added first-change context never executed. Artifact13065901291:497,282 bytes; ZIP SHA-256904c86b1a5aaca7896b552f3dd7e1325212b7c23d098f15f73a3341e7978d320. All30 manifest entries, four exact controls, four original-red arms, immutable identities, source restoration and controller/supervisor cleanup were verified. No retaining-writer or lifetime claim follows. Two late-window runs now demonstrate this observation boundary; static packet/symbol analysis below identifies the observed clearing path.This controller revision adds11 lines to record module/UUID, file PC, symbol, at most64 physical instructions and at most256 caller frames for only the first observed value change. Fresh local P0–P2 autoreview completed scoped-clean (confidence 0.90); syntax, diff, outgoing-data, lint and source checks passed. The context path remains natively unexercised. The GC30–32 window, stop-ID temporal join and original-failure assertion remain required; earlier/equal-value writers and intended lifetime remain unknown, and both causal-proof flags remain false.
Static attribution of the observed timer clear
The runtime FP records from run 41261158810 now bind to physical calls in the exact Darwin x64 profile. Artifact
13057470428from source build 41238111110, source510fef558b464c58de1209298f638fa476d03eef, supplies profile SHA-25652f2048920d8b31ef4471e121cc06ad7b4828761a2c3278349f66ed9b4acaf2band dSYM SHA-25645aa427a18e5f97dda7a9fb23641ee066ecc5084de26d121b54530f62c41b0c6; both report UUID4C4C4432-5555-3144-A125-D2B0A4297913. Inspection was static, with LLDB initialization and symbol-file script loading disabled; no inferior was launched.Normalized file calls and saved returns agree at all three boundaries:
0x1010094edcallsJSC__VM__collectAsyncand returns at0x1010094f2;0x100551c0dcallsclearStackBelowCallerand returns at0x100551c12;0x100551cd9calls the__bzerostub and returns at0x100551cde. Symbol provenance reaches the repeating GC timer'sperform_gcatGarbageCollectionController.rs:216andsecureZeroBytes<char[2048]>atbindings.cpp:4983.The helper's prologue reserves its 2 KiB buffer at
[FP−0x810, FP−0x10). The watched word lies inside it at offset0x350; captured RAX/RBX equal its start, and the recovered FP/SP fit the intervening stack pushes. An independent read-only check confirmed the hashes, physical calls and geometry. At this write the address belongs to the helper's local buffer: prior module-frame ownership cannot be carried forward. This establishes the clearing path in the debugger arm, which collected the target. It does not identify the cleared pointer as the dropped retainer, its original store or intended lifetime, explain why the timer intervened under observation, or supply the missing original-failure retained-root/writer join. Both causal-proof flags remain false.Synchronous LLDB event ownership, atomic progress/capture checkpoints, the independent330-second supervisor, cooperative240-second deadline from before launch, strict stop IDs/reasons/watchpoint identity, admitted-inferior cleanup and cleanup-gated sealing remain. All stop/observation/file/byte limits are unchanged. The original fixture and runtime are unchanged.
Late-window run 41261158810, signed controls
16faa52ec209105e8cdd8172c32666bc206f2c20, completed GC32 and owned cleanup but failed the original-failure-preserved assertion. All four controls retained true/false/true; the debugger arm produced true/true/true and both dropped tags were absent from its snapshot. Its only observed value change, at GC30, cleared the watched word to zero; GC32 used the same SP and returned zero. The initial pointer therefore cannot be identified as the dropped retaining environment in this run. Artifact13065426695:496,136 bytes; ZIP SHA-256f4993e771568a79da81006f4ea480f0248567f6a03a37ece4fa366d43ef28463. All30 manifest entries, four exact controls, four original-red arms, immutable identities, source restoration and controller/supervisor cleanup were verified. Retained dyld data maps the clear PC to libsystem_platform.dylib, UUID78C9DCC0-2FBA-3BE4-A1DD-DA618362AE68, image offset0x34cd. This identifies the image, not the function or caller. Static CollectCallEdges log entries are not runtime caller evidence. This result does not qualify an original-retainer or runtime-fix claim.Bounded synchronous run 41260381945, signed controls
056f35af435a32f842446d4284a921843ee03a27, reached its cooperative240-second deadline at GC5 after718 value changes (505 during GC1); its last stop ID was11447. LLDB batch returned zero despite the recorded Python assertion; the required seal stayed red. Artifact13065057117:220,434 bytes; ZIP SHA-2564cb592e0141ceb71a3ed3d11fa206ed2e80c02235e5b7a5ef77cf87fd65b0a68. All28 manifest entries, four exact controls, four original-red arms, immutable identities, source restoration and controller/supervisor cleanup were verified. No GC32, snapshot or writer join was completed. This supports removing startup watch traffic, without promising the late window will finish.The preceding async run 41259547032 failed at GC1/stop15 before retaining any handled value changes. Artifact
13064741937: 88,786 bytes; ZIP SHA-2561be03f2958372785a6a52b1ad4b2b72afd8ef6b7d2764054feefe113aba3b09a. All 28 manifest entries, four exact controls, four original-failure arms, immutable source/profile/dSYM/fixture identities, source restoration and owned-inferior cleanup were verified. The retained log evaluates stop15 twice: sequence1202 reports should-stop true, sequence1216 false, then sequence1217 resumes privately while the controller is inspecting live threads. LLDB event removal can execute stop actions and resume; synchronous resume owns event delivery until a settled stop. This supports the controller cutover, but does not establish the precise Apple1700 double-evaluation cause or a TLS runtime defect. No snapshot/join or writer proof came from this failed run.The current probe watches one eight-byte physical address from GC30 through GC32. It must match the actual GC32 module-frame SP, accepted scan word, new mark and later-snapshot owner, including the last preceding value-changing write at the raw span capture. Address reuse is recorded rather than assumed to preserve frame ownership. Each observed OnModify change retains old/new values, stop ID, native PC and post-store general registers; the Bun image mapping is recorded once. Decoding and focused writer-stack/operand/lifetime capture remain separate. Admission/GC32 physical frames and failure diagnostics remain. Bounds are240 seconds/30,000 stops/12,000 observations/32 files/32MiB. Unexpected stops fail. No runtime or fixture repair follows from an incomplete capture.
13062091211:385,054 bytes, ZIP SHA-256629e362bfdc3791402c3b93520825bbf4c8d48e9ad5f888c5ee6668bfc52c898.13061887594:40,609 bytes, ZIP SHA-2566d783fcdff6dc09af151b8ba2935a39bee07ca48bd3855f92e8d5e3f1a59cf55. No capture-cost verdict.7e718d305357fd9dc4fb1efcbba147701668815c: explicit write-always API admitted by exact CI LLDB1700; still GC1/stop23/Trace2, zero writes, watched value0. Added PC/SP/FP/description identifies file PC0x103c4b38ainappendJSCellOrAuxiliary. Artifact13062577275:40,838 bytes, ZIP SHA-2561e4a7589a15eedd3ceff0dd2971c673d3ddb4a2d250697df1169816d96dbc26d.All three failed writer archives have26 verified manifest entries, four exact controls, four unchanged-failure controls, restored source and successful owned-inferior cleanup. Write-always did not resolve the trace. Its cause remains unknown; it must not be treated as a valid watched store. Fresh independent P0–P2 review found no actionable issue in the write-mode/metadata change; AST, schema and diff checks passed. Physical writer attribution and affected native qualification remain required before PR215 can land.
Internal debugger-state capture41254624942, signed controls
988374576502970a9f87e49994c2fe85e4c2c73f, failed atGC1 on the same Trace2/zero-write condition. Artifact13062023699:94,765 bytes; ZIP SHA-256d3e22bc458d942e78f28bdfa026f3b6fd835972db6324cfdf76cb1ca77f78eac. Root and independent audit verified all26 manifest entries/four exact controls/four original-failure controls/source restoration/owned-inferior cleanup. Exact LLDB1700 admitted both4096-entry circular log handlers and all debugger-state commands. Raw dumps are retained indebugger.log; their command return output is empty.The hardware watchpoint's remoteZ2 admission succeeds, yet it records no hit and its word remains0. Public mark breakpoints5/6 report disabled/hit0. The remoteZ0/z0/resume/stop sequence and completed internal plans nevertheless show physical sites13/14 stepped at file PCs
0x103c4b37d→0x103c4b384→0x103c4b38abeforeGC32, ending in the fatal trace. This establishes premature interaction with those disabled sites, not a runtime writer or a precise LLDB implementation defect. A recognized stop can no longer hide an unexpected simultaneous stop on another thread; unexpected Trace remains fatal.Breakpoint-lifecycle capture41255238480, signed controls
1c8fc4bbed1aeb7dac655afc83d0746d047b2fdb, passed the earlier trace boundary and reachedGC9, then failed the12,000-write observation cap, not the deadline. Artifact13063490369:374,601 bytes; ZIP SHA-256a45789ba3c165a6d2003830edef14592068cf0c13f6eb635dade1706f75552f1. Root and independent audit verified all26 manifest entries/four controls/four original-failure arms/exact identities/source restoration/owned-inferior cleanup. History contains963 value changes and11,037 equal-value writes;8,794 events are at the clock routine's frame-pointer push. GC32/snapshot/owner-writer proof is incomplete. Lint, source-lints and autofix passed this head.Scan/gather/mark breakpoints are created only while stopped atGC32 entry. After all pending observations finish, checked deletion removes the exact controller-owned capture breakpoints and watchpoint before later snapshot/release GCs. Unused disabled/thread options were removed from the point helper. Bounded logging is an explicit observer effect. Native evidence establishes progress past the original trace failure; full lifecycle qualification remains pending completedGC32.
Value-history capture41255962432, signed controls
36c0f839049ff82f4af34f7ef19fff54456beb9c, failed the240-second deadline atGC4 after657 value changes. Artifact13063780694:284,316 bytes; ZIP SHA-2562d0da5a8a624b72cf3190ce99357d68035e8f6698db62dd97097ffcb1ce10c32. All26 manifest entries/four exact controls/four original-failure arms/identities/source restoration/owned-inferior cleanup were verified. The208 detailed contexts produced10,036 frame rows. Retained packets repeatedly query JIT memory-region dirty-page metadata; no precise CPU attribution is claimed. GC32/owner-writer proof remains incomplete.Next capture acceptance rule
One cold native qualification of the same committed workflow and exact admitted profile/fixture must complete originalGC32 within the existing240 seconds/30,000 stops/12,000 observations/32-files32MiB bounds, preserve all four true/false/true controls and source identity/restoration, and join the same physical watched address to the accepted scan word, a new mark and later-snapshot owner. An observed transition from another value to that owner cell is required. A partial trace, a matching baseline failure, changed checkpoint, or weakened assertion cannot pass.
The scoped repair removes per-write full stack walks, symbol lookup, function decoding and its cache. Every OnModify event instead keeps old/new values, native PCs and the once-recorded Bun image mapping and post-store general registers; failure diagnostics and initial/GC32 physical frames remain. Required per-write stack-walk/decoder work becomes zero. Candidate-writer decoding and focused stack/operand/lifetime proof are a separate next stage before any fixture or runtime repair. Equal-value stores and final-writer attribution remain gaps. This is a bounded evidence qualification, not a general performance benchmark or a claim of a runtime fix.
Raw-history qualification 41256773872, signed controls
3e0bc52bc860da6f9cdac3dc744d0b6f8a43de58, was cancelled at the 15-minute job limit. Artifact13063374820is 36,943 bytes, ZIP SHA-256a8d9efcd028992bfd44235ea6ae9d9cedf41a54c2708000c90aa584ad6504ed5. All 24 manifest entries, four exact controls, four original failing arms, profile/dSYM/fixture identities, and source restoration are verified. The archive contains 750 textual watchpoint old/new pairs (249 distinct post-values), but nocapture.json, snapshot or join. The runner terminated LLDB/debugserver after cancellation; controller-owned inferior cleanup, GC progress and a candidate writer are unproven. The 240-second internal deadline did not yield a saved capture; this observer route failed qualification. No product or fixture change follows from this incomplete trace.