Repository navigation
perf(analyze): keep NTFS file identities as one bit per MFT record - #2015
Merged
Merged
Conversation
#2012 kept every file's ID in a map because directory records carry no link count, which took sampled peaks from about 22 MiB to 85 MiB at 1,000,000 files. On NTFS the file ID is a file reference whose low 48 bits are the dense MFT record number, so a bitmap in lazily created 32 KiB pages holds 1,000,000 records in 128 KiB, the layout Mole for Windows uses in src/Mole.Core/DiskScan/FileIdSet.cs. The volume's file system name selects it; ReFS and other IDs keep the hashed set. The sequence number is dropped, so a record reused by a new file during the walk reads as a link of the old one, a race no live walk can rule out. Also renames the folded-directory test that no longer asserts reuse and registers the new files in install.ps1's inventory. Verified: bitmap tests pass under -race on macOS (each of 600,000 records reported new exactly once across eight workers, 1,000,000 records in four pages, NTFS links deduplicated without touching the hashed set); GOOS=windows vet, test build and golangci-lint pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2012. Directory records carry no link count, so #2012 keeps every file's ID, which took sampled peak memory from about 22 MiB to 85 MiB at 1,000,000 files. On NTFS the file ID's low 48 bits are the dense MFT record number, so this keeps one bit per record in 32 KiB pages created on first use: 1,000,000 records hold 128 KiB. ReFS and other file systems keep the hashed set.
Verified on macOS with
-raceand with Windows vet, test build and golangci-lint; this PR runs the native Windows tests.