Add ContainsCursor.NextGeq: successor queries from cursor state#47
Merged
Conversation
NextGeq(x) returns the smallest set value >= x, sharing the cursor's cache and leaving the cursor positioned on the returned value, so interleaving Contains and NextGeq with non-decreasing arguments keeps both on their fast paths. This is the successor primitive for leapfrog intersection: a caller scanning a stream against a sparse bitmap can jump directly to the bitmap's next admissible value instead of probing rejected candidates one by one. Resolution reuses the Contains machinery (same-key cache, maxKey short-circuit, searchFrom gallop, plain search on backward jumps), walks past empty containers, and adds a word-masked scan (bitmap.nextGeq) for the next set bit at or after a position inside a bitmap container. Verified against a sort.Search reference over ToArray on both fixture shapes (~600K probes each: leapfrog, random with backward jumps, interleaved with Contains, exact boundaries), plus empty/emptied-container and container-gap edge cases; race-clean. Leapfrog benchmark: 32 ns per successor jump, 0 allocs.
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
Author
|
After testing, performance is worse compared to regular BMW advance, so let's avoid adding more complexity to sroar EDIT: reopened as there is a case for filter driven advance |
There was a problem hiding this comment.
Pull request overview
This PR adds a successor query primitive to the existing ContainsCursor acceleration mechanism, enabling efficient “next ≥ x” advancement while preserving cursor locality. This supports leapfrog-style intersection by allowing consumers to advance directly to the next admissible value rather than probing candidate-by-candidate.
Changes:
- Adds
(*ContainsCursor).NextGeq(x) (uint64, bool)to return the smallest set value ≥xwhile updating the cursor position. - Introduces
bitmap.nextGeq(y)to find the next set bit within a bitmap container using a masked word scan. - Adds equivalence, edge-case, and benchmark coverage for
NextGeqagainst a reference successor implementation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| contains_cursor.go | Adds NextGeq successor API and cursor-state updates during successor resolution. |
| container.go | Adds bitmap.nextGeq for in-container successor scanning in bitmap containers. |
| contains_cursor_nextgeq_test.go | Adds differential tests, edge-case tests, and a benchmark for NextGeq. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Review follow-ups: - Same-key probes now resolve within the cached container without touching the keys node or the arena; the maxKey range check folds into the resolution switch so cache hits skip it (sound: lastKey is a resolved key <= maxKey or the unmatchable sentinel). Leapfrog successor jumps: 31.7 -> 18.5 ns/op. - Extract arrayGeqPos (lower bound from the cached position) out of arrayHas and share it between Contains and NextGeq; Contains keeps its single-call structure and its benchmarks are unchanged. - Fix the no-successor comment: also reached when x is past the last set value of a normal trailing container.
…nextgeq # Conflicts: # contains_cursor.go
… array fast path BenchmarkNextGeqLeapfrog runs on denseBitmap (all array containers, ~650 values each) with ~1-2000 jumps, so 98% of probes stay in one container and 0% ever reach a bitmap container: it measures only the arrayGeqPos same-key fast path, never bitmap.nextGeq or the cross-container walk. Its comment is corrected to say so. Add two benchmarks that exercise the paths the PR actually adds: - BenchmarkNextGeqBitmap: an all-bitmap-container fixture (every 5th value per key) so the timed path is bitmap.nextGeq (verified: 100% bitmap landings). - BenchmarkNextGeqCrossContainer: jumps > 65536 over mixedBitmap so every probe changes key and hits the searchFrom gallop + fresh-container walk, including empty-gap skips (verified: 100% cross-key, mixed array/bitmap). Also add targeted TestNextGeqEdgeCases cases for the same-key "cached container exhausted -> idx=lastIdx+1 -> later container" branch, for both the bitmap and array container shapes (the random differential reaches the array shape but never the bitmap one).
aliszka
approved these changes
Jul 7, 2026
The arrayGeqPos refactor dropped arrayHas's equality and in-gap early returns, adding a caller-side re-compare and an unconditional arrPos store to every probe — the two shapes that dominate reject-heavy Contains workloads (+3-5% on the new gap-shape bench, ~+1.5% end-to-end on filtered BM25 reject walks). Restore arrayHas and derive NextGeq's successor from the lower-bound invariant arrayHas already leaves in arrPos: on a miss the successor is c[arrPos]. Same-key jumps keep their fast path (18.8ns vs 30.5ns without it); at_cursor_hit probes recover 3.6%.
aliszka
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
NextGeq(x) (uint64, bool)toContainsCursor: the smallest set value ≥ x, resolved from the cursor's existing cache and leaving the cursor positioned on the result — so interleavedContains/NextGeqwith non-decreasing arguments stay on their fast paths. Follows up #44; independent of #45/#46.Why
This is the successor primitive for leapfrog intersection. Downstream (Weaviate BM25 filtered search), the merged tombstone+filter bitmap keeps only 2–8% of docs on update-heavy segments; the scan currently walks posting lists candidate by candidate and probes the filter on each (~49% of the filtered query loop, at the per-probe memory floor). With
NextGeq, the roles invert when the filter is sparse: jump straight to the filter's next admissible docID and advance the posting lists to it — skipping the reject walks and, when postings are dense relative to the filter, whole block decodes. Verified planning estimate for that wiring: 19–41% of the filtered query loop.How
Containsmachinery unchanged: same-key cache,maxKeyshort-circuit,searchFromgallop forward, plain search on backward jumps.bitmap.nextGeq(y): word-masked scan for the next set bit at or after a position (mask0xFFFF >> (y & 0xF)matches the MSB-first bit order used bybitmapMaskand the iterator).Verification
Differential against a
sort.Searchreference overToArray()on both fixture shapes (~600K probes each): leapfrog ascending, fully random (forward + backward jumps), interleaved withContains, and exact boundary probes (first/last/beyond). Edge cases: nil cursor/bitmap, empty bitmap, emptied container skipped, container-gap crossing, dense bitmap-container scans. Race-clean; full suite green.Benchmark
(dense fixture, ascending jumps with random 1–2000 gaps — the sparse-filter iteration pattern)