feat(algo): shared zero-copy CSR bridge for GDS functions - #54
Conversation
One helper — buildUndirectedCSR() — now feeds every icebug-backed GDS function, replacing the per-algorithm InMemGraph scan: Zero-copy path: PROJECT_GRAPH pins each rel table's arrow CSR on the graph entry (ladybug#800); the bridge fetches it, symmetrize()s it into the undirected per-row-sorted view (ladybug#799), imports the C-ABI ArrowArrays, and reinterprets the int64 buffers as uint64 in place. No edge is copied between the projection and NetworKit::GraphR. Scan fallback: filtered projections, multi-node-table graphs, manual-transaction projections, and dimension mismatches fall back to the original fwd+bwd storage scan — behavior unchanged for everything the pinned CSR can't represent. GDS_PAGE_RANK consumes the helper (graph name threaded through its bind data for the entry lookup). Both paths verified live and digit-identical on the star graph; the filtered-projection equivalence is a test case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Looks great! Two simple issues to look into: 1. Parallel-edge multiplicity diverges between the two paths (real bug)This is the significant one. The two paths build different undirected graphs for the same data whenever the rel table has parallel edges (multiple rows from the same (from,to)), yet the commit claims "digit-identical, behavior unchanged."
So GDS_PAGE_RANK('G') now silently returns different scores on multigraph data than it did before this commit (and than the fallback/filtered projection of the same data does). PageRank is degree-sensitive, so the results genuinely change. The star-graph verification the commit cites can't Suggested fix: make the two paths produce the same graph. Either the fallback should also coalesce to simple (dedupe via the rel of the old buildCSR), or — better — drop multiplicity consistently and document that GDS operates on the simple projection. Whatever you choose, you need a test 2. Stale-data window: pinned CSR can silently serve the wrong edge setThe materialized CSR is pinned on the graph entry at PROJECT_GRAPH time. The go-guard in fromMaterializedCsr only checks the node dimension (indptr->length() == numNodes + 1). If the rel table is mutated after projection (rows inserted/deleted) but the node cardinality is unchanged, the CSRMetadata already carries a changeEpoch watermark (used for the sortedByDest staleness check) — you could validate the pinned CSR against the rel tables' current change epoch in addition to the dimension check, so staleness forces the fallback instead of silently serving a snapshot. |
…och check Per review: 1. Multiplicity: symmetrize() builds a simple graph; the scan fallback kept parallel-edge multiplicity, so degree-sensitive algorithms diverged between paths on multigraph data. The fallback now sorts + uniques each row — both paths build the same simple undirected projection, documented as the GDS contract. Test: multigraph (parallel + reciprocal edges) through both paths, cross-checked against the equivalent clean graph. 2. Staleness: the pinned CSR now carries the rel table's change epoch (ladybug#806); the bridge compares it to the table's current epoch and falls back to live storage on any mismatch — closing the window where a rel mutation at constant node cardinality served a stale snapshot. Test: mutate after projection, assert live ranks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Both addressed in d879219. You were right on 1 — the star graph couldn't see it, and "digit-identical" was only true for simple graphs. 1. Multiplicity: took the "drop multiplicity consistently" option — the scan fallback now sorts + uniques each row, so both paths build the same simple undirected projection (symmetrize()'s A + A.T semantics), documented as the GDS contract in the bridge header. Test: multigraph with parallel 2. Staleness: CSRMetadata doesn't carry a materialization-time epoch, so LadybugDB/ladybug#806 (small follow-up, up now) records each rel table's Suite 73/73. This PR now depends on ladybug#806 (plus #800/#799 already in main) for CI. |
The "incorporate once, reuse for the 100+ GDS functions" piece from the #47 review discussion, now that both halves of the substrate are in ladybug main.
One helper,
buildUndirectedCSR()(gds_csr_bridge.{h,cpp}), consumed by every icebug-backed GDS function:PROJECT_GRAPHpins each rel table's arrow CSR on the graph entry (ladybug#800) →getCSRArrowArrays().symmetrize()(ladybug#799, undirected per-row-sorted view) → import the C-ABI ArrowArrays → reinterpret int64 buffers as uint64 in place →NetworKit::GraphR. No edge is copied between projection and algorithm.GDS_PAGE_RANKconsumes it in this PR (itsInMemGraphcopy is gone); #47's node2vec rebases onto the same helper next, and GDS_LOUVAIN follows.Verification: traced both paths live in the shell — plain projection takes zero-copy, filtered projection takes fallback, ranks digit-identical (
0.479730 / 0.173423×3on the star). The filtered-vs-plain equivalence is now a test case; full algo suite 71/71.Note: needs a ladybug build containing ladybug#800 + #799 (both merged to main) — the extensions CI ladybug pin may need to advance first.
🤖 Generated with Claude Code