Skip to content

perf(analyze): keep NTFS file identities as one bit per MFT record - #2015

Merged
tw93 merged 1 commit into
windowsfrom
analyze-ntfs-record-bitmap
Nov 16, 2026
Merged

tw93 merged 1 commit into
windowsfrom
analyze-ntfs-record-bitmap

Conversation

@tw93

@tw93 tw93 commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Follow-up to #2012. Directory records carry no link count, so #2012 keeps every file's ID, which took sampled peak memory from about 22 MiB to 85 MiB at 1,000,000 files. On NTFS the file ID's low 48 bits are the dense MFT record number, so this keeps one bit per record in 32 KiB pages created on first use: 1,000,000 records hold 128 KiB. ReFS and other file systems keep the hashed set.

Verified on macOS with -race and with Windows vet, test build and golangci-lint; this PR runs the native Windows tests.

#2012 kept every file's ID in a map because directory records carry no
link count, which took sampled peaks from about 22 MiB to 85 MiB at
1,000,000 files. On NTFS the file ID is a file reference whose low 48
bits are the dense MFT record number, so a bitmap in lazily created
32 KiB pages holds 1,000,000 records in 128 KiB, the layout Mole for
Windows uses in src/Mole.Core/DiskScan/FileIdSet.cs. The volume's file
system name selects it; ReFS and other IDs keep the hashed set. The
sequence number is dropped, so a record reused by a new file during the
walk reads as a link of the old one, a race no live walk can rule out.

Also renames the folded-directory test that no longer asserts reuse and
registers the new files in install.ps1's inventory.

Verified: bitmap tests pass under -race on macOS (each of 600,000
records reported new exactly once across eight workers, 1,000,000
records in four pages, NTFS links deduplicated without touching the
hashed set); GOOS=windows vet, test build and golangci-lint pass.
@tw93
tw93 merged commit d4b1b7a into windows Nov 16, 2026
5 checks passed
@tw93
tw93 deleted the analyze-ntfs-record-bitmap branch November 16, 2026 03:07
tw93 added a commit that referenced this pull request Nov 16, 2026
…status

The Home hardlink entry still called #2012 a draft; it merged as be700a9 and the MFT bitmap landed as #2015. Mole Mac gets a Pending row from source reading (per-child walks own their dedup set) with the runtime check that decides it. Verified with scripts/audit_incidents.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant