Skip to content

Fix lore-revision vfs feature: forward-port projfs/serve.rs to the public crate APIs - #83

Open
kelvincai522 wants to merge 1 commit into
EpicGames:mainfrom
kelvincai522:vfs-fix-main
Open

Fix lore-revision vfs feature: forward-port projfs/serve.rs to the public crate APIs#83
kelvincai522 wants to merge 1 commit into
EpicGames:mainfrom
kelvincai522:vfs-fix-main

Conversation

@kelvincai522

Copy link
Copy Markdown

Problem

lore-revision/src/projfs/serve.rs (the Windows ProjFS provider behind lore clone --virtual) was imported as an unmodified "Initial code copy from private repo" and has never compiled in the open-source tree. Building with the vfs feature fails on Windows:

cargo build --features lore-revision/vfs

with 8 errors, because the surrounding public crates moved on while serve.rs stayed frozen.

Fix

Adapt serve.rs to the current public APIs (no behavior change):

  • From<GUID> for Context violated the orphan rule (both types are foreign now that Context lives in lore_storage) → inline the existing transmute_copy into context_from_guid().
  • VirtualLayer.module.path is Option<PathBuf>.as_deref().expect() (×4).
  • serve() returns () → handle repository.require_path() inline instead of ?.
  • require_path()? in the i32/HRESULT ProjFS callback → map_err to a named E_INVALIDARG.
  • private immutable::ReadOptionslore_storage::options::ReadOptions.
  • drop the now-unused DOT_URC import.

Verification

Built --release --features lore-revision/vfs on Windows (MSVC). lore clone --virtual then serves a ProjFS mount that hydrates lazily: a ~15.7 GB logical tree materialized at ~0.4 MB on disk, and opening a single file pulled down only that file (placeholder → hydrated, ~110 ms).

Scope: Windows/ProjFS only (#[cfg(all(target_family = "windows", feature = "vfs"))]).

The `vfs` feature (lore-revision/vfs) does not compile in the open-source
tree: lore-revision/src/projfs/serve.rs was imported as an unmodified copy
from the private repo ("Initial code copy from private repo") and never built
against the public crate APIs, so `--features lore-revision/vfs` fails on
Windows with 8 errors. Adapt serve.rs to the current public APIs:

- From<GUID> for Context violated the orphan rule (both types are foreign now
  that Context lives in lore_storage) -> inline the transmute_copy into the
  existing context_from_guid() helper.
- VirtualLayer module.path is now Option<PathBuf> -> .as_deref().expect() (x4).
- serve() returns () -> handle repository.require_path() inline instead of `?`.
- require_path()? inside the i32/HRESULT ProjFS callback -> map_err to a named
  E_INVALIDARG (no implicit InvalidArguments -> i32 conversion exists).
- immutable::ReadOptions re-export is private -> lore_storage::options::ReadOptions.
- Drop the now-unused crate::repository::DOT_URC import.

With this, `cargo build --release --features lore-revision/vfs` succeeds and
`lore clone --virtual` serves a ProjFS mount that hydrates files lazily
(verified: a 15.7 GB logical tree materialized at ~0.4 MB on disk; opening a
file pulls down only that file).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Kelvin Cai <kelvin.cai@impossible.place>
@kelvincai522

Copy link
Copy Markdown
Author

Sharing some follow-on VFS work that builds on this PR — for context only, not to expand this PR (it should stay the focused compile-fix).

On top of this forward-port I've prototyped a FUSE backend for lore clone --virtual on Linux (the counterpart to the ProjFS provider). It's on the fuse-vfs branch of my fork; design notes + findings are in docs/fuse-vfs.md there.

FUSE provider (lore-revision/src/fuse/serve.rs): overlay-style — a backing fd opened before mount2, backing-first passthrough with virtual-tree fallback + hydrate-on-read, non-blocking async-spawn callbacks, and write-through materialization. Lazy reads + write-through verified on both Linux/FUSE and Windows/ProjFS (≈15 GB logical tree served at a few MB on disk; opening a file pulls only that file, and a read-back md5 is identical across platforms). One structural note: because FUSE replaces the directory (vs ProjFS overlaying it), the in-mount .lore store gets shadowed and lore's synchronous store reads re-enter the mount; serving the immutable store from outside the mount (--use-shared-store) avoids it — which also matches the roadmap's "serve from the shared store" intent. Details in the doc.

clone --virtual base fix (a behavior change — flagging for your view): the virtual path returns before store_current_anchor, so the local branch stays at revision 0; editing in the mount + stage/commit then diverges from the remote and push is refused. Setting the anchor to the served revision before serving makes stop → stage <file> → commit → push work as a fast-forward, with no sync. If revision-0 was intentional pending the VFS/locking work, happy to drop or rework this.

Two things that look like core-VFS / roadmap territory rather than provider bugs (same on ProjFS and FUSE):

  1. Commit/push while serving blocks on the exclusive repository flock — the serving process is both the file provider and the lock-holder, so a concurrent stage/commit can't acquire it. Seems to map to the "scalable file locking" roadmap item.
  2. No in-place re-serve — re-running clone --virtual on an existing clone errors ("Repository already exist"), and there's no mount/serve command, so serving is effectively one-shot per clone (a mount-lifecycle gap).

Once this PR lands I can rebase the FUSE work onto main (so the projfs forward-port drops out of its diff) and open it as a separate PR. Mainly wanted to surface the direction and get your read on the avoid-sync behavior question.

@mjansson

mjansson commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

The VFS feature is currently in heavy development internally at Epic - not projfs/fuse specifically, so we will revisit this when the foundational work is done and merged.

@ajcarberry ajcarberry added the area:core Core library and its interfaces (lib, C API); revision, storage, transport, protocol internals label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Core library and its interfaces (lib, C API); revision, storage, transport, protocol internals

Development

Successfully merging this pull request may close these issues.

3 participants