Skip to content

fix(query): widen UNION output schema to the union of all branches - #366

Open
teipsum wants to merge 1 commit into
GrafeoDB:mainfrom
teipsum:fix-union-output-arity
Open

fix(query): widen UNION output schema to the union of all branches#366
teipsum wants to merge 1 commit into
GrafeoDB:mainfrom
teipsum:fix-union-output-arity

Conversation

@teipsum

@teipsum teipsum commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Fixes #365

Problem

UNION built its result schema from the first branch alone, so a variable (SPARQL) or column (Cypher/GQL) that appears only in a later branch was dropped instead of present-with-unbound. This produced four observable failures (full repros in the linked issue):

  • Silent SELECT-* corruption (most severe): a SELECT * over heterogeneous-name branches returned a later branch's values under the first branch's variable names, with no error.
  • GRAFEO-X001 at plan time: a projection / ORDER BY / aggregate above the UNION referencing a later-branch-only variable could not resolve it.
  • GRAFEO-V001 at run time: a narrower later branch was indexed against the wider first-branch width.
  • Cypher/GQL silent truncation: a wider later branch's extra columns were dropped.

Per SPARQL 1.1, the result variable domain is the union of the branches' in-scope variables (§18.2.1) and the result is the multiset union of the partial solution mappings (§18.5): a variable unbound in a branch is unbound there, never removed.

Root cause

  • RDF/SPARQLcrates/grafeo-engine/src/query/planner/rdf/mod.rs, plan_union: the branch loop assigned output columns/types only on the first iteration and returned that schema; RdfUnionOperator::next then forwarded each branch's DataChunk verbatim. SELECT * additionally yields an empty projection list that bypasses the by-name guard, so first-branch names win silently.
  • Cypher/GQLcrates/grafeo-engine/src/query/planner/lpg/join.rs, plan_union: the same first-branch-only seed, forwarded through the core UnionOperator. Because Cypher projects inside each branch (below the union), there is no node above the union to fail, so a wider later branch is silently truncated.

Fix

The two legs reconcile differently, because their UNION semantics differ, so the fix lands at each site (one logical change, two operator/planner sites):

  • RDF/SPARQL (by variable name): plan_union now computes the union of every branch's variable set (first-seen order), with each column's type taken from the first branch that binds it (falling back to Any on disagreement). Each branch is wrapped in a projection that maps the unified columns to the branch's local column when present, or to a Value::Null constant when absent — reusing the existing RdfProjectOperator. The padding is a genuine per-chunk-width copy, not a merely-wider declared schema, because DISTINCT and ORDER BY copy at each chunk's own column count against a fixed-width builder. RdfUnionOperator carries the unified width and debug_asserts that every emitted chunk matches it; the single-branch fast path is preserved; and the stale "UNION of INSERT" docstring is refreshed.
  • Cypher/GQL (by position): Cypher/GQL UNION aligns columns by position, with the output names taken from the first branch that reaches each position, so plan_union widens to the maximum branch arity and null-pads the trailing columns of narrower branches via the core ProjectOperator. Column-name compatibility validation (rejecting mismatched names) is intentionally out of scope; this targets only the drop mechanism.

The change is planner-local: no executor, operator-trait, logical-operator, or on-disk change.

Tests

The existing UNION spec cases were all same-arity, so the heterogeneous path was uncovered. Added:

  • SPARQL (tests/spec/rdf/sparql/union_minus.gtest): the X001 and V001 repros; a SELECT * anti-corruption guard asserting the widened columns [s, x, y] with correct null padding; reversed-branch and three-branch variants; DISTINCT over unbound; ORDER BY with nulls last; a same-arity control.
  • Cypher (tests/spec/lpg/cypher/reading_clauses.gtest): positive-correctness oracles for the wider-second-branch and wider-first-branch cases (the dropped column is now preserved and null-padded).
  • GQL (tests/spec/lpg/gql/14.2_set_operations.gtest): a positional-alignment oracle asserting UNION ALL aligns branches by column position and names each output column from the first branch (distinct from the same-name dedup case, which cannot tell positional alignment apart from by-name alignment).
  • White-box planner (rdf/mod.rs): plan_union advertises the union of branch variable sets and emits genuinely null-padded, uniform-width chunks.

No existing spec test encoded the pre-fix behavior, so none required correction. cargo fmt and clippy are clean; the grafeo-engine, grafeo-core, and grafeo-spec-tests suites pass.


Summary by cubic

Fixes UNION to include all branch variables/columns and null-pad missing ones, preventing SELECT-* corruption and fixing GRAFEO-X001/GRAFEO-V001 in SPARQL and silent truncation in Cypher/GQL.

  • Bug Fixes
    • RDF/SPARQL: Planner now unions branch variable names and types (first binding wins; disagreements widen to Any). Each branch is projected to the unified schema with nulls for absent vars; RdfUnionOperator asserts uniform width. Single-branch fast path kept.
    • Cypher/GQL: UNION aligns by position, widens to the maximum arity, and null-pads trailing columns of narrower branches via ProjectOperator with Value::Null. Output names come from the first branch.
    • Planner-local change only. Added spec tests for heterogeneous UNIONs (SPARQL SELECT-*, DISTINCT, ORDER BY nulls-last; Cypher/GQL positional alignment) and a white-box planner guard.

Written for commit ad94058. Summary will update on new commits.

Review in cubic

UNION built its result schema from the first branch alone, so a variable or
column that appears only in a later branch was dropped instead of present
with an unbound value. On the SPARQL leg this surfaced three ways: a
projection / ORDER BY / aggregate above the UNION that referenced such a
variable raised GRAFEO-X001 ("Variable not found") at plan time; a narrower
later branch indexed against the wider first-branch width raised GRAFEO-V001
("Column not found") at run time; and, most dangerously, SELECT * returned a
later branch's values under the first branch's variable names with no error
(the Wildcard projection skips the by-name guard). The Cypher/GQL leg
silently truncated a UNION whose later branch is wider, dropping the extra
columns.

Per SPARQL 1.1 the result variable domain is the union of the branches'
in-scope variables (sec 18.2.1) and the result is the multiset union of the
partial solution mappings (sec 18.5): a variable unbound in a branch is
unbound there, never removed.

RDF/SPARQL planner (rdf/mod.rs plan_union): advertise the union of every
branch's variable set and wrap each branch in a projection that null-pads it
to the unified width. The padding is a genuine per-chunk-width copy rather
than a merely-wider declared schema, because DISTINCT and ORDER BY copy at
each chunk's own column count against a fixed-width builder. RdfUnionOperator
now debug-asserts the uniform width, and its stale "UNION of INSERT" docstring
is refreshed. The single-branch fast path is preserved.

LPG (Cypher/GQL) planner (lpg/join.rs plan_union): Cypher/GQL UNION aligns
columns by position with names from the first branch, so widen to the maximum
branch arity and null-pad the trailing columns of narrower branches. The two
legs reconcile differently (by variable name for SPARQL, positionally for
Cypher/GQL) and fail differently, so each keeps its own regression oracle.
Column-name compatibility validation is left out of scope.

Tests: the existing UNION spec cases were all same-arity, so heterogeneous
coverage is added -- the X001 and V001 repros, a SELECT-* anti-corruption
guard asserting the widened schema and null padding, DISTINCT-over-unbound
and ORDER-BY-nulls-last, a white-box planner test, and positive-correctness
Cypher oracles for the wider-first and wider-second branch cases.

Co-Authored-By: Claude <noreply@anthropic.com>
@teipsum
teipsum requested a review from StevenBtw as a code owner June 11, 2026 12:05

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 5 files

Re-trigger cubic

@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.03922% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/grafeo-engine/src/query/planner/rdf/mod.rs 97.52% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 70 untouched benchmarks


Comparing teipsum:fix-union-output-arity (ad94058) with main (4ebae02)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SPARQL/Cypher UNION drops variables that appear only in a later branch (silent SELECT-* corruption + GRAFEO-X001 / GRAFEO-V001)

1 participant