Skip to content

[CONNECT] Window native result fetches and recover stopped-context cleanup - #904

Draft
dbtsai wants to merge 1 commit into
apache:masterfrom
dbtsai:codex/native-connect-java-correctness
Draft

dbtsai wants to merge 1 commit into
apache:masterfrom
dbtsai:codex/native-connect-java-correctness

Conversation

@dbtsai

@dbtsai dbtsai commented Nov 20, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Replace all-partition result submission with an ordered partition fetch window inside one SQL execution and session scope. Host settings bound concurrent partitions and target undelivered IPC bytes; whole partitions still materialize, so the byte target is approximate and per query. Executor-produced IPC and the producer/JNI API are preserved.

Fix cancellation retry and cleanup after the hosting SparkContext stops, preserve the host's global SparkEnv, document and test that prepared Arrow settings stay frozen, and reject unsupported shuffle-cleanup configuration changes with rollback. Preserve serializable Arrow allocation errors as RESOURCE_EXHAUSTED and reject reuse of completed operation tasks. Add regressions for scheduling, configuration, failure ordering and native lifecycle.

Follow-up to #474.

Why are the changes needed?

A paused client currently permits the entire result to accumulate on the driver. Host shutdown can strand cleanup, and configuration changes between prepare and fetch can enable unsupported shuffle-file cleanup after its initial guard.

Does this PR introduce any user-facing change?

Yes. Fetching adapts to the observed partition sizes within a strict partition-count limit. New host SparkConf settings are spark.connect.native.fetch.maxConcurrentPartitions (scheduler default parallelism when unset/nonpositive) and spark.connect.native.fetch.maxBufferedBytes (heap/8 by default). These are soft per-query memory controls, with at least one whole partition admitted; they are not an aggregate heap quota.

Cancellation and host-stop drain can be retried without replaying partial resource teardown. Complete successful buffered results can drain after host stop; incomplete results fail. Unsupported shuffle-cleanup mutations fail with UNIMPLEMENTED and restore the previous values. These changes apply to the experimental embedded server on master.

How was this patch tested?

Java build and full native acceptance passed against the original base 32b27eb on macOS ARM64 with JDK 21, including cancellation, stopped-context cleanup, prepared configuration, skewed and paused consumers, executor Arrow errors, separate-executor conversion, result-size limits and native listener lifecycle under -Xcheck:jni. Direct stock-runtime regressions use pinned Spark 4.2.0-SNAPSHOT (250ccca3). The complete official-client acceptance also uses the documented local Arrow-input runtime repair; it does not establish unmodified Spark-input compatibility.

JNI Rust tests, workspace formatting/Clippy, native-license freshness and acceptance-runner guards passed at that base. Client/frontend/C sources are unchanged apart from a JNI comment; their previously executed checks carry forward. Hosted merge-ref CI remains pending. No latency or GC improvement is claimed for this final patch.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6)

This pull request and its description were written by OpenAI Codex.

…eanup

Bound ordered partition admission by concurrency and an approximate per-query
IPC byte target within one SQL execution scope. Preserve successful buffered
output and retry cleanup across host shutdown while retaining original errors.
Guard unsupported configuration changes and add scheduling, configuration,
executor-error and native lifecycle regressions.

Generated-by: OpenAI Codex (GPT-6)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant