Repository navigation
[SPARK-62111][CONNECT][PYTHON] Make the pickles and copies of DataFrames, Columns and table arguments carry their UDF functions and Python plans, which each request builds and pickles again, as PySpark does - #897
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…mes, Columns and table arguments carry their UDF functions and Python plans, which each request builds and pickles again, as PySpark does ### What changes were proposed in this pull request? The pickles and copies of DataFrames, Columns and table arguments in the Python client (`pyspark-client-rust`) carried the relation that the client built for them. The functions of their UDFs were pickled to commands once. PySpark's carry the plan objects and the functions, which each request builds and pickles again. This closes the differences that `dev/parity/known-differences.md` listed for them: - A pickled DataFrame, Column or table argument carries the functions of its UDFs and its Python plan objects. Each request of the unpickled object pickles the functions again, as PySpark's does. So: - a value that a function reads, changed after the pickle, is seen; - a Column of a UDF of a lambda fails a plain `pickle.dumps`, of the lambda's `PicklingError`; - a module-level function pickles by reference, as in PySpark. A deep copy of a DataFrame into another session holds copies of the functions. - `DataFrame(plan, session)` keeps a plan that fails to build and raises its error at each request, as PySpark's DataFrame does, instead of at construction. Examples are a UDF that does not pickle, or a read of an option that UTF-8 cannot encode. Each request gets a fresh exception, whose cause and context are copies without the frames of the build. A `KeyboardInterrupt` or `SystemExit` of the build is raised by the constructor. A pickle of such a DataFrame, Column or table argument carries the plan, and an unpickled Column builds it at each request with the request's session. Builds outside a request use the session that they concern: `df._plan.plan(client)` and `to_proto(client)` that of the client given, as PySpark's. - Pickles of plans over a DataFrame's `_plan`, and of their Columns and table arguments, need no session. They load on the server's Python worker, as PySpark Connect's plans, through `_column_reduce.sent_plan`. A plan that a cloudpickle gave back pickles plainly again, as PySpark's. - Unpickled and deep-copied Columns, sort orders, CASE WHENs, window specifications and table arguments keep the origins (call sites) of their expressions when PySpark's origin capture is on, as PySpark's unpickled expressions do. - The garbage collector sees the Python plans and unbuildable expressions that a single holder holds: Columns, `_expr`, table arguments, DataFrames, window specifications, grouped data, merge builders and `DataFrameWriterV2`. So reference cycles through them are collected, as in PySpark. ### Why are the changes needed? To close these differences from PySpark (the known-differences review). ### Does this PR introduce _any_ user-facing change? Yes: the pickles and copies above behave as PySpark's. What remains is documented in `docs/compatibility.md` and `dev/parity/known-differences.md`: - a cloudpickle of a DataFrame is larger (about 12.8 KB against PySpark's 1.2 KB for `spark.range(2)`); - a cycle across two holders that share one plan or expression is not collected; - an action of a DataFrame of an unpickled `df._plan` does not set the copies of its Observations; - the build at construction of a plan whose build succeeds there but fails later; - `to_plan(client)` of an unpickled Column of a failing plan's subquery raises the build error, where PySpark returns the expression. The core crate gains doc-hidden items for the Python client (see the 4.4 Rust migration notes). ### How was this patch tested? - New tests in `python/tests/test_dropin_offline.py`, the e2e tests of the server's Python worker in `python/tests/test_dropin_e2e.py`, and the core crate. Each round's new tests fail without its change. - Five rounds of an independent review. It compared the drop-in with PySpark 4.3.0 through probes of: - pickles, cloudpickles and copies, also of the classic Column and TableArg classes and on the server's Python worker; - plans that fail to build, of 19 kinds of errors; - Observations, subqueries, deep and large plans, threads and sessions; - origins with the capture on and off; - the garbage collector, in 46 scenarios and a stress run of six threads with concurrent collections. Every finding was fixed or documented. - The core tests (release and debug), wasm, clippy, rustdoc, the stubs, the offline suite in sorted order and under coverage, and the live e2e with and without protobuf. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62111-pickles-carry-functions-and-plans
branch
from
November 20, 2026 02:46
7377a59 to
a2428ab
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The pickles and copies of DataFrames, Columns and table arguments in the Python client (
pyspark-client-rust) carried the relation that the client built for them. The functions of their UDFs were pickled to commands once. PySpark's carry the plan objects and the functions, which each request builds and pickles again. This closes the differences thatdev/parity/known-differences.mdlisted for them:pickle.dumps, of the lambda'sPicklingError;A deep copy of a DataFrame into another session holds copies of the functions.
DataFrame(plan, session)keeps a plan that fails to build and raises its error at each request, as PySpark's DataFrame does, instead of at construction. Examples are a UDF that does not pickle, or a read of an option that UTF-8 cannot encode. Each request gets a fresh exception, whose cause and context are copies without the frames of the build. AKeyboardInterruptorSystemExitof the build is raised by the constructor. A pickle of such a DataFrame, Column or table argument carries the plan, and an unpickled Column builds it at each request with the request's session. Builds outside a request use the session that they concern:df._plan.plan(client)andto_proto(client)that of the client given, as PySpark's._plan, and of their Columns and table arguments, need no session. They load on the server's Python worker, as PySpark Connect's plans, through_column_reduce.sent_plan. A plan that a cloudpickle gave back pickles plainly again, as PySpark's._expr, table arguments, DataFrames, window specifications, grouped data, merge builders andDataFrameWriterV2. So reference cycles through them are collected, as in PySpark.Why are the changes needed?
To close these differences from PySpark (the known-differences review).
Does this PR introduce any user-facing change?
Yes: the pickles and copies above behave as PySpark's. What remains is documented in
docs/compatibility.mdanddev/parity/known-differences.md:spark.range(2));df._plandoes not set the copies of its Observations;to_plan(client)of an unpickled Column of a failing plan's subquery raises the build error, where PySpark returns the expression.The core crate gains doc-hidden items for the Python client (see the 4.4 Rust migration notes).
How was this patch tested?
python/tests/test_dropin_offline.py, the e2e tests of the server's Python worker inpython/tests/test_dropin_e2e.py, and the core crate. Each round's new tests fail without its change.Every finding was fixed or documented.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
This pull request and its description were written by Isaac.