Repository navigation
Conversation
…eanup Bound ordered partition admission by concurrency and an approximate per-query IPC byte target within one SQL execution scope. Preserve successful buffered output and retry cleanup across host shutdown while retaining original errors. Guard unsupported configuration changes and add scheduling, configuration, executor-error and native lifecycle regressions. Generated-by: OpenAI Codex (GPT-6)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Replace all-partition result submission with an ordered partition fetch window inside one SQL execution and session scope. Host settings bound concurrent partitions and target undelivered IPC bytes; whole partitions still materialize, so the byte target is approximate and per query. Executor-produced IPC and the producer/JNI API are preserved.
Fix cancellation retry and cleanup after the hosting SparkContext stops, preserve the host's global SparkEnv, document and test that prepared Arrow settings stay frozen, and reject unsupported shuffle-cleanup configuration changes with rollback. Preserve serializable Arrow allocation errors as RESOURCE_EXHAUSTED and reject reuse of completed operation tasks. Add regressions for scheduling, configuration, failure ordering and native lifecycle.
Follow-up to #474.
Why are the changes needed?
A paused client currently permits the entire result to accumulate on the driver. Host shutdown can strand cleanup, and configuration changes between prepare and fetch can enable unsupported shuffle-file cleanup after its initial guard.
Does this PR introduce any user-facing change?
Yes. Fetching adapts to the observed partition sizes within a strict partition-count limit. New host SparkConf settings are
spark.connect.native.fetch.maxConcurrentPartitions(scheduler default parallelism when unset/nonpositive) andspark.connect.native.fetch.maxBufferedBytes(heap/8 by default). These are soft per-query memory controls, with at least one whole partition admitted; they are not an aggregate heap quota.Cancellation and host-stop drain can be retried without replaying partial resource teardown. Complete successful buffered results can drain after host stop; incomplete results fail. Unsupported shuffle-cleanup mutations fail with UNIMPLEMENTED and restore the previous values. These changes apply to the experimental embedded server on master.
How was this patch tested?
Java build and full native acceptance passed against the original base
32b27ebon macOS ARM64 with JDK 21, including cancellation, stopped-context cleanup, prepared configuration, skewed and paused consumers, executor Arrow errors, separate-executor conversion, result-size limits and native listener lifecycle under-Xcheck:jni. Direct stock-runtime regressions use pinned Spark 4.2.0-SNAPSHOT (250ccca3). The complete official-client acceptance also uses the documented local Arrow-input runtime repair; it does not establish unmodified Spark-input compatibility.JNI Rust tests, workspace formatting/Clippy, native-license freshness and acceptance-runner guards passed at that base. Client/frontend/C sources are unchanged apart from a JNI comment; their previously executed checks carry forward. Hosted merge-ref CI remains pending. No latency or GC improvement is claimed for this final patch.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
This pull request and its description were written by OpenAI Codex.