Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench

[nvzm123]

Summary

This PR adds a built-in pylucene backend to cuVS Bench for building and searching local Lucene vector indexes through PyLucene and cuVS-Lucene.

It supports:

• Lucene101AcceleratedHNSWCodec for cuVS-assisted HNSW construction with Lucene HNSW search;
• CuVS2510GPUSearchCodec for GPU CAGRA construction and search;
• packaged configurations, deterministic parameter sweeps, automatic tuning, dry runs, index reuse, forced rebuilds, query batching, and standard cuVS Bench CSV results; and
• h5py as an explicit dependency for the existing dataset-preparation path.

Selecting --backend pylucene is sufficient to activate the backend. It resolves a matching local cuvs-java/cuvs-lucene JAR pair and native-library path from conventional build or Maven locations, while retaining paired explicit overrides for nonstandard installations. Missing or mismatched prerequisites fail before JVM startup with actionable messages.

This PR also owns the PyLucene end-to-end suite. Pytest owns its scenarios, assertions, parameterization, and reporting under cuvs_bench/tests/pylucene; reusable Python helpers remain non-test modules, and test-only Java adapters live under python/cuvs_bench/tests/java. Those adapters are compiled into a temporary directory before the process-wide JVM starts and are excluded from published packages.

HNSW parameters and topology

The HNSW backend exposes these benchmark parameters:

• m maps to AcceleratedHNSWParams.maxConn;
• ef_construction maps to AcceleratedHNSWParams.beamWidth;
• both inputs use cuVS-Lucene's SAME_GRAPH_FOOTPRINT heuristic to derive the underlying CAGRA build parameters;
• num_candidates controls Lucene's KnnFloatVectorQuery candidate budget and may be swept independently for each built index; and
• direct_single_segment requests one direct Lucene segment without a force merge and verifies the committed topology.

Stock PyLucene instantiates codecs through no-argument constructors. The backend therefore compiles a small Java adapter before JVM startup that delegates to the production accelerated codec with the requested heuristic parameters. It does not reimplement the production codec.

Automatic tuning covers m, ef_construction, and num_candidates, with the candidate lower bound resolved from top_k. Build parameters and segment count are recorded in schema-v4 provenance so incompatible indexes cannot be silently reused.

Runtime and integrity behavior

The backend validates the PyLucene 10.2 ABI, configured JAR roles and matching Maven versions, native-library paths, Lucene SPI codecs, dataset shape, and index location before execution. It requires the base cuvs-java JAR and standard thin cuvs-lucene JAR, rejects native-classifier/fat artifacts, and initializes PyLucene and the JVM lazily.

HNSW preserves cuVS-Lucene's production behavior: it uses the accelerated writer when available and intentionally falls back to Lucene's CPU writer otherwise. CAGRA requires GPU support and is validated fail-closed. A generic cuVS-unavailable CAGRA failure is augmented with the relevant PyLucene/JAR/native-library setup checks without changing unrelated Java errors. CAGRA segment metadata, vector dimensions and counts, file coverage, headers, footers, and checksums are verified before results are accepted.

Atomic, commit-bound provenance protects index reuse. Failed new builds remove only their partial output, and preflight failures preserve an existing index. --force removes the prior index before replacement, so a failed forced rebuild leaves no previous index to reuse. In-process backend results use the shared direct CSV exporter and preserve build/search identity, latency percentiles, and failure handling; result filenames and CSV rows retain algorithm, group, and optional scope identity.

Requirements and limits

• FLOAT32 Euclidean/L2 datasets, at most 4096 dimensions, and at least two indexed vectors
• latency mode with one search thread; query batching through --batch-size
• num_candidates >= top_k for HNSW; this is Lucene's candidate budget, not a direct cuVS ef_search setting
• CAGRA searches with k <= 1024
• direct_single_segment remains subject to Lucene's per-indexing-thread hard RAM limit and fails if Lucene commits more than one segment
• process-wide PyLucene JVM settings cannot change after initialization

PyLucene 10.2 remains a source-built external dependency. This PR depends on the production codec and compatibility changes in NVIDIA/cuvs#2475 until that PR is merged. The Bench guide pins the exact tested producer revision rather than a moving PR head.

Test coverage

Non-live pytest coverage includes backend registration, zero-boilerplate runtime discovery, paired override validation, configuration expansion and tuning, parameter validation, adapter compilation and classpath failures, JVM and SPI validation, configured-artifact role validation, index build/reuse/cleanup, single- and multi-segment topology, search candidate propagation, result export, provenance mismatches, CAGRA corruption detection, built-wheel resource ownership, and CLI failure handling.

The opt-in live matrix explicitly distinguishes:

• CPU HNSW build and HNSW search;
• GPU CAGRA build followed by one-layer or three-layer HNSW search; and
• GPU CAGRA build and CAGRA search.

GPU-required cases assert the accelerated writer, reader, and query implementations and fail on unavailable cuVS or CPU fallback. CPU cases explicitly verify and report stock Lucene HNSW execution.

Coverage includes a single live document, one and ten segments, 10-to-1 and 100-to-10 force merges, CAGRA searchWidth values 1, 16, and 32, deletion, vectorless documents, selective filtering, persisted HNSW graph degree/layers, an exact Lucene filtered-search boundary, deterministic brute-force recall, duplicate-hit exclusion, inactive/filter-rejected document exclusion, and rank-one self matches. CAGRA-built HNSW also covers top_k=2000 with num_candidates=2500; direct CAGRA search retains its supported k <= 1024 boundary. The Java-side query bridge verifies the retained production searchWidth value rather than echoing Python configuration.

CAGRA configurations use graphDegree=32, intermediateGraphDegree=64, and enough documents to avoid cuVS clamping warnings, including 24,832 vectors for the three-layer case. Every live path case captures process output and fails on the known cuVS graph-clamping diagnostics before printing a clear CPU HNSW, GPU/CAGRA-built HNSW, or GPU CAGRA-search label.

Artifact gates require the production Java adapter and PyLucene YAML resources in built distributions while rejecting test Java, PyLuceneTestSupport, and .class payloads. The wheel-content test runs in the standard Python test environment with CUDA disabled, so it does not require nvcc. A Bench-owned formatting POM, the pre-commit matcher, and the Spotless wrapper cover both production and test Java roots without adding a Maven artifact to the package tree.

Validation

Final revisions:

• cuVS Bench PR Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385: e54979c3b5aba2f53a77720f7ebc3dd0e16618fa
• cuVS producer PR Add PyLucene integration and CPU/GPU end-to-end tests #2475: 57ce4920374dce1cecd09091566b18710fe01c82

Validated on an NVIDIA A10G with JDK 22 and Lucene/PyLucene 10.2. The GPU run used the official RAPIDS 26.12 development nightly libcuvs 26.12.00a8 (cuda12_260910004918_29e1101b), built from the tested main revision 29e1101b merged into both branches. The runtime reported cuVS 26.12.0 and used the RMM 26.12 ABI; the cuvs-java and cuvs-lucene 26.12.0 JARs were rebuilt from the merged producer branch.

After that full run, PR #2475 merged current main (ef29c4cfd53082d31c1fc05ec35251c835120efa) in 57ce4920374dce1cecd09091566b18710fe01c82. That merge adds only the four upstream CMake dependency-discovery changes and leaves the validated Java, cuvs-lucene, PyLucene, and test inputs unchanged; git diff --check passed, and the GPU suite was not redundantly rerun.

Producer validation

• full GPU-enabled cuvs-java and cuvs-lucene Maven suites: 497 tests, 0 failures, 0 errors, 29 skipped (cuvs-java: 112 with 1 skipped; cuvs-lucene: 385 with 28 skipped, including both passing ThinJarContentsIT cases)

cuVS Bench validation

• python -m pytest -q -s cuvs_bench/tests/pylucene --run-pylucene: 393 passed in 55.05 seconds
• relevant shell syntax checks: passed

The Java suites exercised the native GPU paths and emitted no version-mismatch or linkage errors. Their inherited randomized and small-dataset cases emitted native cuVS graph-parameter diagnostics (271 warning lines from cuvs-java and 2,651 from cuvs-lucene). The PyLucene suite emitted no cuVS configuration or CPU-fallback warnings; it emitted the expected JVM notice for the incubating vector module.

Related work

• NVIDIA/cuvs#2475
• NVIDIA/cuvs#1907
• NVIDIA/cuvs-lucene#174 (historical source of the producer-side PyLucene compatibility work)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.