Skip to content

feat(algo): shared zero-copy CSR bridge for GDS functions - #54

Merged
adsharma merged 2 commits into
LadybugDB:mainfrom
zachwinter:feat/gds-zero-copy-bridge
Aug 13, 2026
Merged

feat(algo): shared zero-copy CSR bridge for GDS functions#54
adsharma merged 2 commits into
LadybugDB:mainfrom
zachwinter:feat/gds-zero-copy-bridge

Conversation

@zachwinter

Copy link
Copy Markdown
Contributor

The "incorporate once, reuse for the 100+ GDS functions" piece from the #47 review discussion, now that both halves of the substrate are in ladybug main.

One helper, buildUndirectedCSR() (gds_csr_bridge.{h,cpp}), consumed by every icebug-backed GDS function:

  • Zero-copy path: PROJECT_GRAPH pins each rel table's arrow CSR on the graph entry (ladybug#800) → getCSRArrowArrays().symmetrize() (ladybug#799, undirected per-row-sorted view) → import the C-ABI ArrowArrays → reinterpret int64 buffers as uint64 in place → NetworKit::GraphR. No edge is copied between projection and algorithm.
  • Scan fallback: filtered projections, multi-node-table graphs, manual-transaction projections, and dimension mismatches fall back to the original fwd+bwd storage scan. Behavior unchanged wherever the pinned CSR can't represent the projection.

GDS_PAGE_RANK consumes it in this PR (its InMemGraph copy is gone); #47's node2vec rebases onto the same helper next, and GDS_LOUVAIN follows.

Verification: traced both paths live in the shell — plain projection takes zero-copy, filtered projection takes fallback, ranks digit-identical (0.479730 / 0.173423×3 on the star). The filtered-vs-plain equivalence is now a test case; full algo suite 71/71.

Note: needs a ladybug build containing ladybug#800 + #799 (both merged to main) — the extensions CI ladybug pin may need to advance first.

🤖 Generated with Claude Code

One helper — buildUndirectedCSR() — now feeds every icebug-backed GDS
function, replacing the per-algorithm InMemGraph scan:

Zero-copy path: PROJECT_GRAPH pins each rel table's arrow CSR on the graph
entry (ladybug#800); the bridge fetches it, symmetrize()s it into the
undirected per-row-sorted view (ladybug#799), imports the C-ABI ArrowArrays,
and reinterprets the int64 buffers as uint64 in place. No edge is copied
between the projection and NetworKit::GraphR.

Scan fallback: filtered projections, multi-node-table graphs,
manual-transaction projections, and dimension mismatches fall back to the
original fwd+bwd storage scan — behavior unchanged for everything the
pinned CSR can't represent.

GDS_PAGE_RANK consumes the helper (graph name threaded through its bind
data for the entry lookup). Both paths verified live and digit-identical
on the star graph; the filtered-projection equivalence is a test case.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@adsharma

Copy link
Copy Markdown
Contributor

Looks great! Two simple issues to look into:

1. Parallel-edge multiplicity diverges between the two paths (real bug)

This is the significant one. The two paths build different undirected graphs for the same data whenever the rel table has parallel edges (multiple rows from the same (from,to)), yet the commit claims "digit-identical, behavior unchanged."

  • Zero-copy path goes through symmetrize(), which is explicitly a simple-graph builder. The two-pointer merge dedupes (if (v != last)) consecutive duplicates and collapses reciprocal pairs, mirroring scipy sparse-addition semantics. Two parallel rows (u,v),(u,v) plus (v,u) collapse to a
    single structural entry on each side → degrees 1/1.
  • Scan fallback uses InMemGraph::insertNbr, which (per the impl) does not dedupe — it appends every scanned fwd+bwd edge. Same data → degrees 2/2.

So GDS_PAGE_RANK('G') now silently returns different scores on multigraph data than it did before this commit (and than the fallback/filtered projection of the same data does). PageRank is degree-sensitive, so the results genuinely change. The star-graph verification the commit cites can't
catch this because the star has no parallel edges. The new test case shares that blind spot.

Suggested fix: make the two paths produce the same graph. Either the fallback should also coalesce to simple (dedupe via the rel of the old buildCSR), or — better — drop multiplicity consistently and document that GDS operates on the simple projection. Whatever you choose, you need a test
with parallel edges that pins the expected (and ideally cross-checked) scores.

2. Stale-data window: pinned CSR can silently serve the wrong edge set

The materialized CSR is pinned on the graph entry at PROJECT_GRAPH time. The go-guard in fromMaterializedCsr only checks the node dimension (indptr->length() == numNodes + 1). If the rel table is mutated after projection (rows inserted/deleted) but the node cardinality is unchanged, the
zero-copy path happily serves the stale CSR while the scan fallback (and pre-commit behavior) reads live storage. Same statement, different answer depending on which path silently engages.

CSRMetadata already carries a changeEpoch watermark (used for the sortedByDest staleness check) — you could validate the pinned CSR against the rel tables' current change epoch in addition to the dimension check, so staleness forces the fallback instead of silently serving a snapshot.

…och check

Per review:

1. Multiplicity: symmetrize() builds a simple graph; the scan fallback kept
   parallel-edge multiplicity, so degree-sensitive algorithms diverged
   between paths on multigraph data. The fallback now sorts + uniques each
   row — both paths build the same simple undirected projection, documented
   as the GDS contract. Test: multigraph (parallel + reciprocal edges)
   through both paths, cross-checked against the equivalent clean graph.

2. Staleness: the pinned CSR now carries the rel table's change epoch
   (ladybug#806); the bridge compares it to the table's current epoch and
   falls back to live storage on any mismatch — closing the window where a
   rel mutation at constant node cardinality served a stale snapshot.
   Test: mutate after projection, assert live ranks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zachwinter

Copy link
Copy Markdown
Contributor Author

Both addressed in d879219. You were right on 1 — the star graph couldn't see it, and "digit-identical" was only true for simple graphs.

1. Multiplicity: took the "drop multiplicity consistently" option — the scan fallback now sorts + uniques each row, so both paths build the same simple undirected projection (symmetrize()'s A + A.T semantics), documented as the GDS contract in the bridge header. Test: multigraph with parallel (0,1)×2 + reciprocal (1,0) + (1,2) through both paths, cross-checked against the equivalent clean path graph — all three produce 0.256757 / 0.486486 / 0.256757.

2. Staleness: CSRMetadata doesn't carry a materialization-time epoch, so LadybugDB/ladybug#806 (small follow-up, up now) records each rel table's changeEpoch on the graph entry — captured before the materializing scan, so a racing mutation reads as stale. The bridge compares it to the table's current epoch and any mismatch forces the fallback. Test: mutate the rel table after projection at constant node cardinality — exactly the window you described — and assert the ranks reflect live storage. That test fails without the epoch check.

Suite 73/73. This PR now depends on ladybug#806 (plus #800/#799 already in main) for CI.

@adsharma
adsharma merged commit 4a990bd into LadybugDB:main Aug 13, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants