Skip to content

[BUG] Lucene: GPU search fails when a segment's accepted-ordinal set is empty (topK collapses to 0) #2599

Description

@imotov

Describe the bug
When CuVS2510GPUVectorsReader.search() runs against a segment where the accept-docs bitset leaves no live vector ordinals, the effective top-k is clamped to zero and the search returns no result rows at all, which then trips an assertion (and, with assertions off, throws).

The mechanism is entirely in CuVS2510GPUVectorsReader.java:

  1. The early-out at line 460 only covers fieldEntry.count() == 0 || knnCollector.k() == 0. It does not cover "the segment has vectors, but every one of them is filtered out".
  2. Line 490 computes the cuVS top-k from the prefilter cardinality:
    topK = Math.min(knnCollector.k() + 10, mask[0].cardinality());
    With a zero-cardinality mask this yields topK == 0.
  3. SearchResultsImpl.create() (SearchResultsImpl.java:45) loops for (long i = 0; i < topK * numberOfQueries; i++) and only appends a per-query map once count == topK. With topK == 0 the loop body never runs, so it returns an empty list rather than a list holding one empty map.
  4. Back in the reader, assert searchResult.size() == 1; (line 566) fails.

This is not test-only. With assertions disabled the assert is skipped and line 571 calls searchResult.getFirst() on an empty LinkedList, which throws NoSuchElementException. That propagates out of the catch (Throwable t) { Utils.handleThrowable(t); } block unchanged, since Utils.handleThrowable rethrows RuntimeException as-is. So a production query over a segment whose vectors are all deleted/filtered out fails instead of returning zero hits.

This affects both branches of the search: cagra=false in the reproduction below, but the topK clamp happens before the CAGRA/brute-force split, so the CAGRA branch is exposed the same way.

Steps/Code to reproduce bug

cd java/cuvs-lucene
mvn test -Dtest=TestCuVSDeletedDocuments#testVectorSearchWithMixedDeletedAndMissingVectors -Dtests.seed=2FEE712B9F46AA6B

Fails deterministically for that seed on a machine with a supported GPU:

[ERROR] com.nvidia.cuvs.lucene.TestCuVSDeletedDocuments.testVectorSearchWithMixedDeletedAndMissingVectors
java.lang.AssertionError
    at __randomizedtesting.SeedInfo.seed([2FEE712B9F46AA6B:95289329B99FD2D2]:0)
    at com.nvidia.cuvs.lucene.CuVS2510GPUVectorsReader.search(CuVS2510GPUVectorsReader.java:566)
    at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.search(PerFieldKnnVectorsFormat.java:311)
    ...
    at com.nvidia.cuvs.lucene.TestCuVSDeletedDocuments.testVectorSearchWithMixedDeletedAndMissingVectors(TestCuVSDeletedDocuments.java:175)

Instrumenting the reader just after line 490 confirms the cause. Over the leaves visited for this seed:

DBG k=23 topK=27 cardinality=27 acceptedOrdsLen=43 fieldEntryCount=43 cagra=true
DBG k=23 topK=16 cardinality=16 acceptedOrdsLen=20 fieldEntryCount=20 cagra=true
DBG k=23 topK=23 cardinality=-1 acceptedOrdsLen=-1 fieldEntryCount=1  cagra=false
DBG k=23 topK=1  cardinality=1  acceptedOrdsLen=1  fieldEntryCount=1  cagra=false
DBG k=23 topK=1  cardinality=1  acceptedOrdsLen=1  fieldEntryCount=1  cagra=false
DBG k=23 topK=23 cardinality=-1 acceptedOrdsLen=-1 fieldEntryCount=2  cagra=true
DBG k=23 topK=0  cardinality=0  acceptedOrdsLen=1  fieldEntryCount=1  cagra=false   <-- assert fires here

The offending leaf is a one-document segment holding a single vector whose document was deleted, so getAcceptOrds() accepts nothing.

Note the failing call at TestCuVSDeletedDocuments.java:175 is the unfiltered query (filter = null). No user filter is needed — the live-docs bitset alone is enough, because acceptDocs is non-null whenever the segment has deletions. Any tiny segment whose vector-bearing documents are all deleted reproduces it.

Expected behavior
Test doesn't fail. A segment in which the filter (or the live-docs bitset) accepts zero vector ordinals should contribute zero hits, not fail. Lucene's contract is that search() collects nothing for such a leaf.

Environment details (please complete the following information):

  • Environment location: Bare-metal
  • Method of cuVS install: from source (branch-26.10, at 92796ea8)

Additional context

Possible approaches:

  1. Extend the early-out at line 460 to also return when the accepted-ordinal set is empty:
    if (acceptDocs != null && mask[0].isEmpty()) {
      return;
    }
    This is the narrowest fix and keeps the GPU out of a query that cannot produce results.
  2. Independently, make the reader tolerate an empty result list rather than assuming exactly one row — the current assert searchResult.size() == 1 / searchResult.getFirst() pair turns any zero-row response into a hard failure. Guarding with if (searchResult.isEmpty()) return; would stop this class of bug from being a production-visible exception.
  3. Optionally, decide whether SearchResultsImpl.create() should return numberOfQueries empty maps instead of an empty list when topK == 0, so the "one map per query" invariant its callers assume actually holds. That is a cuvs-java-level change and would fix every caller at once.

Fixes 1 and 2 are complementary; 1 alone avoids a pointless GPU round-trip, 2 alone makes the failure mode benign.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

LucenebugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions