Ghost LRU entries persist indefinitely after file deletion — file-not-found path doesn't evict from index

[im-nick-adams]

Summary

When a CAS blob is deleted from disk (e.g., via the overwrite path in Add()), but a concurrent GET has already looked up the key in the LRU and released the lock, the subsequent os.Open fails with "not exists." The warning is logged, but the stale LRU entry is never removed, causing the same warning to fire on every subsequent access until the server is restarted.

Impact

On a moderately active server (~10 concurrent Bazel users), we observed 20,145 phantom warnings over 2 days (~49/minute), producing persistent Lost inputs no longer available remotely failures for Bazel clients using --remote_download_toplevel. A restart temporarily resolves the issue by rebuilding the LRU from disk, but phantoms accumulate again within hours.

Root Cause

In cache/disk/disk.go, availableOrTryProxy():

// Slow path retry:
c.mu.Lock()
item, listElem = c.lru.Get(key)
if listElem != nil {
    blobPath = path.Join(c.dir, c.FileLocation(...))
    f, err = os.Open(blobPath)
}
c.mu.Unlock()

if err != nil {
    // WARNING: logs but does NOT evict from LRU
    log.Printf("Warning: expected %q to exist on disk, undersized cache?", blobPath)
}

Compare to the decompression-error path ~20 lines later, which correctly self-heals:

if err != nil {
    log.Printf("Warning: expected item to be on disk, but something happened...")
    c.mu.Lock()
    c.lru.RemoveElement(listElem)  // ← correctly removes dead entry
    c.mu.Unlock()
}

Suggested Fix

Add LRU eviction in the file-not-found path, matching the existing decompression-error pattern:

if err != nil {
    log.Printf("Warning: expected %q to exist on disk, undersized cache?", blobPath)
    c.mu.Lock()
    if listElem != nil {
        c.lru.RemoveElement(listElem)
    }
    c.mu.Unlock()
}

This makes ghost entries self-healing on first access rather than persisting indefinitely.

Environment

• bazel-remote commit: b857daf1f63c641dc3fe6105a674a6e9ed81cf35
• Go 1.25.6
• Local disk storage (no NFS), zstd compression mode
• ~847K cached files, 126 GB on disk, 1.2 TB max

Reproduction

1. Start bazel-remote with multiple concurrent clients
2. Trigger overwrites (same CAS hash uploaded by different clients, or after bazel clean --expunge)
3. Concurrent GET requests for overwritten blobs hit the race window
4. Once triggered, the phantom entry persists forever — visible via repeated "expected to exist on disk" warnings for the same path

[mostynb]

Added a slightly different mitigation in #894 - remove the lru entry while still holding the lock instead of re-acquiring it. But I want to think about the root issue a bit more before landing this.

[im-nick-adams]

No problem. Really appreciate the quick response.

[mostynb]

BTW are you using a specific version of bazel? It is kind of unexpected that clients would re-upload the same blobs so frequently.

[im-nick-adams]

We're using 9.1.0. Below is some additional follow that I've been doing this afternoon with Claude.

---

Follow-up investigation — root cause of the overwrites

After deploying the ghost-eviction fix, we dug into what's producing the overwrites (1.47 GB overwritten_bytes_total) that create the race window in the first place.

In a multi-user environment where 50+ developers share a remote cache, the pattern is:

1. Client A builds target //foo:bar — produces libfoo.a (hash abc123), calls FindMissingBlobs → server says "missing" → uploads it (file: abc123-4990-111111111)
2. Sometime later, Client B does a bazel clean --expunge and rebuilds the same target. Since their local output base is empty, they rebuild libfoo.a locally, producing the same hash abc123.
3. Client B calls FindMissingBlobs — server says "not missing" (it exists) — so Client B does not re-upload the CAS blob. ✓
4. However, the action cache (AC) PUT is unconditional. Client B uploads a new AC entry for the same action key. The AC entry overwrites the old one (different random suffix), and the old AC file is queued for async deletion. This is expected and correct.

But there's a scenario where CAS does get re-uploaded:

5. If the ghost LRU bug (the original issue) causes FindMissingBlobs → Contains() → returns true (ghost says "exists") → client skips upload → later GET fails → Bazel retries the action with --experimental_remote_cache_eviction_retries → re-executes locally → uploads the result → now it's a genuine CAS PUT for a hash that already exists in the LRU (but the old file was already deleted) → new file is created → but nothing to delete (old file was already gone) → no overwrite in this path.

So the 1.47 GB overwritten is likely driven by concurrent FindMissingBlobs races where two clients both see "missing" before either has finished uploading, rather than post-expunge re-uploads.

Suggested additional hardening (separate from the ghost fix):

In the CAS Put() path, before writing to disk, check if the key already exists in the LRU with the same size. If so, verify the file exists on disk (os.Stat). If the file is present and healthy, skip the write entirely — the content is guaranteed identical (content-addressed). This eliminates the overwrite storm and the associated race window.

// Proposed no-op optimization for CAS Put:
if kind == cache.CAS {
    c.mu.Lock()
    item, elem := c.lru.Get(key)
    c.mu.Unlock()
    if elem != nil && !isSizeMismatch(size, item.size) {
        existingPath := path.Join(c.dir, c.FileLocation(kind, item.legacy, hash, item.size, item.random))
        if _, statErr := os.Stat(existingPath); statErr == nil {
            // File exists with correct hash+size — skip redundant write
            _, _ = io.Copy(io.Discard, r) // drain the reader
            return nil
        }
    }
}

Theoretically this would reduce overwritten_bytes_total to near-zero and eliminate the race window that causes the ghost entries in the first place.

---

Some commentary is coming from Claude :)

[im-nick-adams]

We've also added a complementary hardening on top of #894 that skips redundant CAS Put operations entirely when the blob already exists on disk:

im-nick-adams@e6f9bc6

Since CAS is content-addressed, a Put for an existing hash is guaranteed to produce identical content. Skipping the write avoids the overwrite path in lru.Add() which deletes the old file and creates the race window in the first place. This should reduce overwritten_bytes_total to near-zero in multi-user environments where concurrent clients frequently build the same targets.

[ulrfa]

When a CAS blob is deleted from disk (e.g., via the overwrite path in Add()), but a concurrent GET has already looked up the key in the LRU and released the lock, the subsequent os.Open fails with "not exists."

The overwrite path in lru.Add is not discarding keys, it instead atomically (while the lock is held) swaps the path (the random value) for the key, and queues the old file for async deletion. The slow path in disk.availableOrTryProxy handles this by re-acquiring the lock, looking up the key again to get the current path, and opening it. Since the lookup and open both happen under the lock, the file cannot be evicted or replaced between them. And once os.Open returns successfully, Linux guarantees the open file descriptor remains usable even if the path is subsequently unlinked.

The only scenario I’m aware of, where the slow path itself can fail with “not exists” is if the item was evicted to make room for other files, but if recently accessed files (e.g. touched by FindMissingBlobs) are being evicted, the cache is too small. Could that be what triggered this scenario?

[mostynb]

Since CAS is content-addressed, a Put for an existing hash is guaranteed to produce identical content. Skipping the write avoids the overwrite path in lru.Add() which deletes the old file and creates the race window in the first place. This should reduce overwritten_bytes_total to near-zero in multi-user environments where concurrent clients frequently build the same targets.

There is a downside to doing that- if there is a corrupt blob on disk then you would need admin intervention to replace it.

The only scenario I’m aware of, where the slow path itself can fail with “not exists” is if the item was evicted to make room for other files, but if recently accessed files (e.g. touched by FindMissingBlobs) are being evicted, the cache is too small. Could that be what triggered this scenario?

Assuming this is correct, there should not be any cache eviction happening: ~847K cached files, 126 GB on disk, 1.2 TB max

Another possiblity, that @ulrfa mentioned in the PR, is that another process is removing files in the cache directory (for example another bazel-remote instance). @im-nick-adams: could this be the case?

[im-nick-adams]

This extension is enabled in the Web Worker Extension Host because it prefers to run there. Learn More

Definitely a good point.

Another possiblity, that @ulrfa mentioned in the PR, is that another process is removing files in the cache directory (for example another bazel-remote instance). @im-nick-adams: could this be the case?

We are only running a single bazel-remote instance. That said, when we initially ran into this issue, the cache was configured too small and we were definitely running into evictions. So that could have been the instigator of the scenario (we had initially mis-configured the cache to be only 100GB after switching to a NAS).

[mostynb]

Pushed a change which removes the index element if the slow path fails, and logs the fast/slow path and the last reported error. I wonder if you could try this and report back any failure logs that you see?

[im-nick-adams]

Great. One of my co-workers is going to build and deploy this. We'll get back to you on whether it addresses the issue.

A note, we did see the build failures start to crop up again today even with the larger cache size deployed... We were able to get back to a functional state after restarting bazel-remote.