Skip to content

[SPARK-61384][FOLLOWUP][CONNECT] Add regression tests for the experimental Rust Spark Connect frontend - #870

Closed
dbtsai wants to merge 5 commits into
apache:masterfrom
dbtsai:codex/pr474-protocol-frontend
Closed

dbtsai wants to merge 5 commits into
apache:masterfrom
dbtsai:codex/pr474-protocol-frontend

Conversation

@dbtsai

@dbtsai dbtsai commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A follow-up of #474 (SPARK-61384). This PR was opened as part 1/6 of the split of #474. Since then #474 has put the experimental spark-connect-server frontend on master, so after merging master this PR adds only the frontend regression tests from its reviews:

  • tests/schema_consistency.rs:
    • materialized_views_can_mix_with_matching_arrow_and_ipc_batches: Arrow record batches with Utf8View columns and producer IPC with LargeUtf8 columns in one result, in both orders.
    • materialized_views_cannot_mix_with_preserved_view_ipc: the preserved Utf8View IPC that must fail with INTERNAL after one batch.
  • tests/engine_injection.rs:
    • session_release_cancels_pending_prepare_and_drops_backend_session and shutdown_cancels_pending_prepare_before_transport_drain_expires: ReleaseSession and graceful shutdown while the engine's prepare is pending.
    • requests_handled_after_shutdown_begins_do_not_create_backend_sessions: a request whose handler runs after shutdown begins gets UNAVAILABLE ("server is shutting down") without creating a backend session.
    • shutdown_releases_backend_session_while_a_result_remains_unread now keeps its unread peer connected through stop(), so serve must return through the two-second transport drain bound.
  • src/service.rs unit tests, negative_engine_ipc_row_counts_fail_before_forwarding: an engine IPC batch with a negative row count fails with INTERNAL before anything is forwarded, and the row offset and output schema stay unchanged.

It also notes in the workspace Cargo.toml that the unpublished spark-connect-server enables tonic's server and router features in its normal dependencies.

Why are the changes needed?

Each test covers a guard in the frontend that no other test reaches, so removing that guard would go unnoticed.

Does this PR introduce any user-facing change?

No. It changes only tests and a comment.

How was this patch tested?

cargo test --locked -p spark-connect-server: all 56 tests pass. Each of the three tests added for the last review fails when its guard is removed:

  • The drain test times out when serve awaits the drain without its bound.
  • The shutting-down test gets a result stream when the closing check in the session lookup is removed.
  • The negative row count test sees the batch forwarded when its guard is removed.

cargo +1.100.0 clippy -p spark-connect-server --all-targets -- -D warnings also passes.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (the schema consistency and pending prepare tests) and Claude Opus 5.5 (the drain, shutting-down and negative row count tests)

This pull request and its description were written by OpenAI Codex and Isaac.

@dbtsai
dbtsai marked this pull request as ready for review November 19, 2026 05:47
dbtsai added a commit to dbtsai/spark-connect-rust that referenced this pull request Nov 19, 2026
Stack the native JNI host, Java adapter, platform packaging and integration coverage on the standalone frontend in PR apache#870. Preserve executor-produced IPC and exercise negotiated byte chunking through real JNI.

Written by OpenAI Codex.
dbtsai added a commit to dbtsai/spark-connect-rust that referenced this pull request Nov 19, 2026
Stack the native JNI host, Java adapter, platform packaging and integration coverage on the standalone frontend in PR apache#870. Preserve executor-produced IPC and exercise negotiated byte chunking through real JNI.

Written by OpenAI Codex.
dbtsai added a commit to dbtsai/spark-connect-rust that referenced this pull request Nov 19, 2026
Stack the native JNI host, Java adapter, platform packaging and integration coverage on the standalone frontend in PR apache#870. Preserve executor-produced IPC and exercise negotiated byte chunking through real JNI.

Written by OpenAI Codex.
@dbtsai
dbtsai force-pushed the codex/pr474-protocol-frontend branch from 56640d3 to c35a96e Compare November 19, 2026 18:49
dbtsai added a commit to dbtsai/spark-connect-rust that referenced this pull request Nov 19, 2026
Stack the native JNI host, Java adapter, platform packaging and integration coverage on the standalone frontend in PR apache#870. Preserve executor-produced IPC and exercise negotiated byte chunking through real JNI.

Submit all Arrow result partitions in one Spark job, retaining execution scope through termination and draining ordered results with safe cancellation and cleanup ownership.

Written by OpenAI Codex.

@HyukjinKwon HyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

The frontend's protocol, framing and lifecycle behavior held up under review; the findings are two test gaps in the new crate, each leaving one implemented path that no test would notice losing. One is the ordering of the output-schema check relative to view materialization when record batches and producer IPC are mixed in one result. The other is the session-cancellation arm of the prepare phase, which only ReleaseSession or shutdown during a pending prepare reaches. Each comment names the case that would cover it.

Findings

2 total: 0 P0, 0 P1, 0 P2, 2 P3.

Nit (P3)

  • No test mixes record-batch and IPC output in one result — crates/spark-connect-server/src/service.rs:470 — see inline.
  • No test releases the session or shuts down while prepare is pending — crates/spark-connect-server/src/service.rs:567 — see inline.

Serve the Spark Connect protocol through an injectable execution engine, with Arrow IPC forwarding, negotiated byte chunking, session lifetime management and explicit experimental compatibility limits.
@dbtsai
dbtsai force-pushed the codex/pr474-protocol-frontend branch from c35a96e to 3df3bd1 Compare November 19, 2026 23:53

@HyukjinKwon HyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

The two test gaps raised in the previous review are closed: the new schema_consistency.rs cases cover record batches and producer IPC mixed in one result, and the new engine_injection.rs cases cover ReleaseSession and shutdown while prepare is pending. This round found three more test gaps in the new crate, each leaving one implemented guard that no test would notice losing: the two-second transport drain bound in lib.rs, the shutting-down rejection in session lookup, and the negative IPC row-count check in send_batch. Each comment names the case that would cover it. The review also covered the engine API and session and operation ID handling, Arrow schema validation and framing, the RPC dispatch and release paths, the proto crate's server feature, and the new CI workflow, with no findings.

Findings

3 total: 0 P0, 0 P1, 0 P2, 3 P3.

Nit (P3)

  • No test keeps a stalled peer connected across the shutdown drain — crates/spark-connect-server/src/lib.rs:107 — see inline.
  • No test sends a request whose handler runs after shutdown begins — crates/spark-connect-server/src/service.rs:221 — see inline.
  • No test supplies a negative engine IPC row count — crates/spark-connect-server/src/service.rs:451 — see inline.

Re-review status

Prior AI findings: 2 no longer raised, 0 still raised; additional unresolved findings in this review: 3.

Of these: 3 in code the previous review already covered.

Comment thread crates/spark-connect-server/src/lib.rs Resolved
HyukjinKwon added a commit to dbtsai/spark-connect-rust that referenced this pull request Nov 20, 2026
Resolve the conflicts with SPARK-62089's ClientCodec in
crates/spark-connect-proto/build.rs and src/lib.rs the way apache#870 does: the
generated client and the optional server bindings both use crate::ClientCodec,
and the crate docs describe both.

Regenerate crates/spark-connect-server/jvm/native-licenses/dependencies.json,
whose recorded Cargo.lock digest changes with the merged lockfile; the JNI
dependency closure is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon added a commit that referenced this pull request Nov 20, 2026
### Review split and dependency

This PR builds on the standalone Rust frontend in [#870](#870), at `c35a96ee25646efc3542748283f74dc32286ec2d`. Review the native layer in the [parent-to-head comparison](dbtsai/spark-connect-rust@c35a96e...51cfd00). Both PRs target Apache `master`, so Files changed includes the frontend until it lands. Further packaging/licensing splits and the producer/off-heap API redesign remain follow-up work.

### What changes were proposed in this pull request?

Add an experimental embedded Spark JVM execution engine for the Rust Spark Connect frontend. Rust owns the listener, protocol sessions, operations and response streams. A Java adapter invokes Spark planning and execution through JNI in the same JVM. The server and JNI crates are opt-in and unpublished.

Executors produce complete IPC streams through Spark's `ArrowConverters.toBatchWithSchemaIterator`. The adapter submits all output partitions in one Spark job. A lazy execution worker holds the task, session and SQL execution scopes through job termination. Client pulls drain partition results in order and release consumed references. Cancellation and close retain ownership until scheduler completion and safe plan cleanup; failed cancellation preserves owners and the original failure for retry.

JNI copies Java `byte[]` directly into a Rust-owned `Vec`, removing the intermediate C allocation and copy. Moving that allocation into protobuf `Bytes` and slicing negotiated transport chunks shares the payload. Normal Prost encoding copies it into the outgoing gRPC buffer. IPC is preserved without decode/re-encode, and the producer/JNI APIs are unchanged.

The inherited frontend implements negotiated IPC byte chunking. Opted-in clients receive indexed chunks for reassembly; other clients receive the complete IPC batch. There is no unconditional 3 MiB Arrow response split. The default maximum chunk payload is about 115.2 MiB, valid smaller client preferences are respected, and zero disables chunking. This adapter separately uses an approximate 2 MiB producer target and a 64 MiB post-conversion JNI safety check. Schema and eager-command envelopes retain their separate 3 MiB bound. The adapter's tests do not establish interoperability with nondefault IPC compression codecs.

The native layer also provides deferred handle cleanup and admission control, host-policy config redaction, embedded listener termination detection, Linux/macOS x86-64/ARM64 platform JARs, macOS universal and all-platform JAR assembly, version/checksum-aware loading, and generated dependency license assets.

### Why are the changes needed?

This prototype reuses the Rust protocol and transport stack while Spark remains the execution engine. Serial per-partition collection in the previous adapter prevented result tasks from running together and repeatedly entered execution/session scopes. One result job follows JVM Connect's scheduling and cumulative result-size behavior.

JIRA: https://issues.apache.org/jira/browse/SPARK-61384

### Does this PR introduce any user-facing change?

Developers can explicitly build and launch this prototype. Existing default workspace members and client APIs are unchanged. The Java adapter must also be available to executors. Supported batch queries, types, RPCs and restrictions are documented in the server README; unsupported operations return `UNIMPLEMENTED`. Authentication, authorization, TLS, Windows, full protocol parity and native library unloading are outside scope. This is intended for trusted local workloads.

All partitions may finish while a slow client pauses, so the driver can buffer the whole result. The combined serialized result is now subject to one `spark.driver.maxResultSize` budget, matching JVM Connect; that setting is not a retained-heap cap. Each task still materializes a partition result. Session/task limits, IPC checks and transport chunk sizes are not complete memory or CPU quotas. Shuffle dependency file cleanup remains rejected pending validation.

The adapter uses internal Spark APIs. Runtime validation is pinned to Spark 4.2.0-SNAPSHOT `250ccca3f2b5f043e20b415adf17bda5d202c60f`, Scala 2.13 and JDK 21. The stock assembly is used for JNI smoke and the performance comparisons below; broader Arrow-input acceptance uses a separately repaired runtime. This does not establish compatibility with other Spark builds.

### How was this patch tested?

Local macOS ARM64 validation passed:

- Fresh release JNI build and Java adapter/test compilation; pinned-stock JNI start/query/stop with `-Xcheck:jni`.
- Real-Spark scheduling checks covering actual partition overlap, one job and SQL execution, scope lifetime, exact ordered and retained IPC, cancellation before execution and during AQE planning, waiting pulls, session/engine close, caller interruption, upstream failure, empty partitions and cumulative result limits.
- Live acceptance, separate-executor strict Kryo, lifecycle and injected cleanup failures, including failed cancellation with a live worker and a buffered later partition.
- 52 JNI and 50 frontend Rust tests; C bridge checks normally and with ASAN/UBSAN; 809 client release unit tests; complete client integration compilation with and without `wasm-udf`; workspace Clippy and formatting; license freshness; 26 acceptance guards and 33 packaging tests. Unchanged Rust, C, client and packaging checks carry forward across Java-only and acceptance-label fixes.

Hosted status is separate. The previously published `3d0e23b` had four native-platform failures at generated inventory freshness on [GitHub's merge ref](https://github.com/apache/spark-connect-rust/actions/runs/44591877645), where an upstream `Cargo.lock` change alters the bound input hash. Its branch-only check passes locally. Master has advanced since this stacked base and the branch currently conflicts with it. The conflict is unresolved and CI against the latest target remains unverified.

### OSS JVM versus native benchmark (November 19, 2026)

This run compares the published all-partition implementation at `51cfd00ab49cfe00c64b6d0ccf59d6ee231a0458` against the OSS JVM Spark Connect server on a Linux ARM64 host. Both use the same 208 stock Spark 4.2.0-SNAPSHOT JARs at `250ccca3f2b5f043e20b415adf17bda5d202c60f`, JDK 21.0.13.1 and official PySpark-client 4.2.0 on Python 3.12.3, with PyArrow 26.0.0 and gRPC 1.84.0. The native library is a Rust 1.100.0 release build.

Four matched pairs ran in JVM/native, native/JVM, native/JVM, JVM/native order, with fresh server and client processes per arm. Settings: `local[2]`, 1 GiB driver heap, UTC, two input/shuffle partitions, 10,000 Arrow rows per batch and codec `none`. The Arrow workloads transfer 8 million ordered Int64 rows; range/filter uses 1 million rows and join uses 100,000 rows. Raw Arrow requests do not opt into byte chunking. Config/Analyze use 200 warmups plus 500 measured actions per fork; query/Arrow workloads use 10 warmups plus 30 measured actions. Concurrent Config uses eight channels/threads, one warmup and three measured rounds of 1,000 calls.

| Workload | OSS JVM (ms) | Native (ms) | Native latency change |
|---|---:|---:|---:|
| Config Get | 0.557 | 0.297 | -46.6% |
| Range schema analysis | 1.573 | 1.981 | +25.9% |
| Tiny SQL and collect | 34.978 | 27.563 | -21.2% |
| Range/filter aggregate and collect | 50.346 | 50.003 | -0.7% |
| Join aggregate and collect | 70.580 | 71.872 | +1.8% |
| Raw Arrow complete receipt | 249.106 | 233.610 | -6.2% |
| Raw Arrow receipt and table decode | 259.292 | 241.777 | -6.8% |
| Public `DataFrame.toArrow()` | 413.371 | 407.018 | -1.5% |
| Concurrent Config per-call latency | 0.585 | 0.633 | +8.2% |

Negative change means lower native latency. Latency columns are geometric means of four fork medians; changes are the native/JVM ratios of those means minus one. Concurrent Config measures individual RPC latency, not round time divided by calls.

Raw receipt drains ExecutePlan through completion, with IPC decoding and validation outside timing. The next row also includes decoding and Table creation. Public `toArrow()` uses the unmodified official client and includes response handling, decode, materialization and cast. Raw receipt was 2.2–10.4% faster in the four pairs; receipt plus decode was 4.4–11.4% faster. Public `toArrow()` was approximately tied, with paired changes from 0.035% to 3.3% faster. Schema analysis and concurrent Config are slower on aggregate. Metadata and query rows vary widely by fork: tiny SQL ranged from 37.2% faster to 9.4% slower, and range/filter from 19.9% faster to 45.1% slower.

All eight forks and 13,152 action records passed their checks, including exact ordered Arrow values, schemas, offsets and cleanup. Each raw Arrow action returned 800 batches and 65,248,000 IPC bytes in both arms. The table was independently recomputed from raw samples. Linux validation also passed 52 JNI and 50 frontend Rust tests, normal and ASAN/UBSAN C bridge checks, a stock-Spark JNI start/query/stop smoke and a reduced benchmark preflight, both with `-Xcheck:jni`. JNI checking was disabled for timing.

The host was nonexclusive. Server processes were pinned to CPUs 2–5 and clients to 6–7; one client CPU was 86.2% busy in a roughly 10-second pre-timing observation. This does not identify contention during measured calls; small deltas should be treated cautiously. These are local-executor, loopback, narrow uncompressed results, with no statistical-significance, distributed-performance, retained-heap or GC-improvement claim. The raw client records per-response receipt timelines in both arms. Host, partition count and JDK differ from the comparison below, so differences between the two sets of numbers do not isolate the scheduling change.

<details>
<summary>Local partition-scheduling diagnostic (macOS ARM64, November 19, 2026)</summary>

These separate instrumented macOS ARM64 measurements compare partition-scheduling designs. All arms use the same unmodified pinned Spark assembly, JDK 21.0.10, official PySpark 4.2.0, PyArrow 26.0.0 and gRPC 1.84.0. Both native arms share the exact release JNI library.

Settings: `local[2]`, 1 GiB driver heap, UTC, 8 million ordered Int64 rows, 16 partitions, a 10,000-row Arrow cap and codec `none`. Three fresh-server rounds rotate JVM, windowed and all-partition order. Each fork has 10 warmup and 10 measured actions for each workload.

| Action | JVM pooled median (ms) | Windowed alternative pooled median (ms) | All-partition pooled median (ms) | All/window median matched-fork ratio |
|---|---:|---:|---:|---:|
| Raw complete Arrow receipt | 115.554 | 160.763 | 113.576 | 0.7011 (29.9% lower) |
| Public `DataFrame.toArrow()` | 175.161 | 208.369 | 161.429 | 0.8957 (10.4% lower) |

Ratios are medians of three matched fork-median ratios, rather than ratios of pooled medians. All/window ratios were 0.668/0.701/0.767 for receipt and 0.896/0.993/0.710 for `toArrow`. JVM comparisons varied by fork and were approximately at parity: matched median ratios were 1.019 for receipt and 0.9997 for public `toArrow`. These data do not establish a native speedup over the JVM server or statistical significance.

All 360 actions passed exact ordered-value validation; 180 were measured. Each raw action returned 8 million rows, 800 batches and 65,248,000 IPC bytes. Raw timers drain protobuf responses through completion, excluding decoding and validation. Public timers include normal client response handling, decoding, materialization and cast. Event logs show 16 jobs and 15 SQL executions for the windowed alternative, versus one job and one SQL execution for this implementation and JVM. The change also removes per-batch task/session context setup, so these results do not isolate task parallelism alone.

A separate paused-consumer probe, using identical compiled probe class bytes, retained 7.700 MiB of query-slot IPC for the windowed version versus 62.148 MiB here (8.07 times as much), then validated all 8 million ordered values and complete cleanup. This counts payload arrays reachable from query slots, not retained heap or transient copies. Equivalent JVM retained IPC was not measured, so no native memory advantage over JVM is established. The window's byte setting is a learned admission target, not a hard memory bound. It also keeps per-partition result limits and can end SQL scopes while lookahead jobs remain active.

Against the previous serial adapter with two partitions, the same production class bytes reduced median matched-fork latency by 39.6% for receipt and 14.0% for public `toArrow` in a separate three-arm run. Full GCs still occurred there: five across the all-partition forks, including two measured pauses around 19–22 ms, versus zero serial and one JVM. The 16-partition comparison recorded no Full GCs; this does not establish that GC or memory pressure is solved. Allocation weights describe sampled churn, and 100 ms RSS samples can miss peaks. These narrow, uncompressed loopback experiments do not validate distributed execution, production memory safety with slow clients, wide rows or codecs.

</details>

### Was this patch authored or co-authored using generative AI tooling?

This pull request and its description were written by OpenAI Codex.

Closes #474 from dbtsai/codex/jvm-spark-connect-server-wip.

Lead-authored-by: DB Tsai <dbtsai@dbtsai.com>
Co-authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
HyukjinKwon and others added 2 commits November 20, 2026 12:40
Master now has this frontend through apache#474 (f65b054), whose frontend commit
c35a96e is this branch's frontend before the review's regression tests. Merge
with c35a96e as the base, so the frontend sources, README and manifests are
master's, and this branch keeps only the regression tests of engine_injection.rs
and schema_consistency.rs and the tonic note in the workspace Cargo.toml.

Co-authored-by: Isaac <no-reply@databricks.com>
…okup and negative IPC row counts

Address the three comments of the review
apache#870 (review):

- shutdown_releases_backend_session_while_a_result_remains_unread keeps its
  unread 1 KiB-window peer connected through server.stop(), so serve must
  return through the two-second transport drain bound of lib.rs.
- requests_handled_after_shutdown_begins_do_not_create_backend_sessions sends
  an ExecutePlan's headers before the shutdown signal and its body after, for
  a session no request used, and expects UNAVAILABLE with no factory call.
- negative_engine_ipc_row_counts_fail_before_forwarding passes an engine IPC
  batch with row_count -1 to send_batch and expects INTERNAL, an unchanged
  offset and output schema, and no response.

Each test fails when the guard it covers is removed.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon HyukjinKwon changed the title [SPARK-61384][CONNECT] Add an experimental Rust Spark Connect frontend (1/6) [SPARK-61384][FOLLOWUP][CONNECT] Add regression tests for the experimental Rust Spark Connect frontend Nov 20, 2026
…note

crates/spark-connect-server/jvm/native-licenses/dependencies.json records the
SHA-256 of the workspace Cargo.toml, which this branch changes with its note on
tonic's server features. The JNI dependency closure is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants