Skip to content

[SPARK-62111][CONNECT][PYTHON] Make the pickles and copies of DataFrames, Columns and table arguments carry their UDF functions and Python plans, which each request builds and pickles again, as PySpark does - #897

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62111-pickles-carry-functions-and-plans

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

The pickles and copies of DataFrames, Columns and table arguments in the Python client (pyspark-client-rust) carried the relation that the client built for them. The functions of their UDFs were pickled to commands once. PySpark's carry the plan objects and the functions, which each request builds and pickles again. This closes the differences that dev/parity/known-differences.md listed for them:

  • A pickled DataFrame, Column or table argument carries the functions of its UDFs and its Python plan objects. Each request of the unpickled object pickles the functions again, as PySpark's does. So:
    • a value that a function reads, changed after the pickle, is seen;
    • a Column of a UDF of a lambda fails a plain pickle.dumps, of the lambda's PicklingError;
    • a module-level function pickles by reference, as in PySpark.
      A deep copy of a DataFrame into another session holds copies of the functions.
  • DataFrame(plan, session) keeps a plan that fails to build and raises its error at each request, as PySpark's DataFrame does, instead of at construction. Examples are a UDF that does not pickle, or a read of an option that UTF-8 cannot encode. Each request gets a fresh exception, whose cause and context are copies without the frames of the build. A KeyboardInterrupt or SystemExit of the build is raised by the constructor. A pickle of such a DataFrame, Column or table argument carries the plan, and an unpickled Column builds it at each request with the request's session. Builds outside a request use the session that they concern: df._plan.plan(client) and to_proto(client) that of the client given, as PySpark's.
  • Pickles of plans over a DataFrame's _plan, and of their Columns and table arguments, need no session. They load on the server's Python worker, as PySpark Connect's plans, through _column_reduce.sent_plan. A plan that a cloudpickle gave back pickles plainly again, as PySpark's.
  • Unpickled and deep-copied Columns, sort orders, CASE WHENs, window specifications and table arguments keep the origins (call sites) of their expressions when PySpark's origin capture is on, as PySpark's unpickled expressions do.
  • The garbage collector sees the Python plans and unbuildable expressions that a single holder holds: Columns, _expr, table arguments, DataFrames, window specifications, grouped data, merge builders and DataFrameWriterV2. So reference cycles through them are collected, as in PySpark.

Why are the changes needed?

To close these differences from PySpark (the known-differences review).

Does this PR introduce any user-facing change?

Yes: the pickles and copies above behave as PySpark's. What remains is documented in docs/compatibility.md and dev/parity/known-differences.md:

  • a cloudpickle of a DataFrame is larger (about 12.8 KB against PySpark's 1.2 KB for spark.range(2));
  • a cycle across two holders that share one plan or expression is not collected;
  • an action of a DataFrame of an unpickled df._plan does not set the copies of its Observations;
  • the build at construction of a plan whose build succeeds there but fails later;
  • to_plan(client) of an unpickled Column of a failing plan's subquery raises the build error, where PySpark returns the expression.
    The core crate gains doc-hidden items for the Python client (see the 4.4 Rust migration notes).

How was this patch tested?

  • New tests in python/tests/test_dropin_offline.py, the e2e tests of the server's Python worker in python/tests/test_dropin_e2e.py, and the core crate. Each round's new tests fail without its change.
  • Five rounds of an independent review. It compared the drop-in with PySpark 4.3.0 through probes of:
    • pickles, cloudpickles and copies, also of the classic Column and TableArg classes and on the server's Python worker;
    • plans that fail to build, of 19 kinds of errors;
    • Observations, subqueries, deep and large plans, threads and sessions;
    • origins with the capture on and off;
    • the garbage collector, in 46 scenarios and a stress run of six threads with concurrent collections.
      Every finding was fixed or documented.
  • The core tests (release and debug), wasm, clippy, rustdoc, the stubs, the offline suite in sorted order and under coverage, and the live e2e with and without protobuf.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

This pull request and its description were written by Isaac.

…mes, Columns and table arguments carry their UDF functions and Python plans, which each request builds and pickles again, as PySpark does

### What changes were proposed in this pull request?

The pickles and copies of DataFrames, Columns and table arguments in the Python client (`pyspark-client-rust`) carried the relation that the client built for them. The functions of their UDFs were pickled to commands once. PySpark's carry the plan objects and the functions, which each request builds and pickles again. This closes the differences that `dev/parity/known-differences.md` listed for them:

- A pickled DataFrame, Column or table argument carries the functions of its UDFs and its Python plan objects. Each request of the unpickled object pickles the functions again, as PySpark's does. So:
  - a value that a function reads, changed after the pickle, is seen;
  - a Column of a UDF of a lambda fails a plain `pickle.dumps`, of the lambda's `PicklingError`;
  - a module-level function pickles by reference, as in PySpark.
  A deep copy of a DataFrame into another session holds copies of the functions.
- `DataFrame(plan, session)` keeps a plan that fails to build and raises its error at each request, as PySpark's DataFrame does, instead of at construction. Examples are a UDF that does not pickle, or a read of an option that UTF-8 cannot encode. Each request gets a fresh exception, whose cause and context are copies without the frames of the build. A `KeyboardInterrupt` or `SystemExit` of the build is raised by the constructor. A pickle of such a DataFrame, Column or table argument carries the plan, and an unpickled Column builds it at each request with the request's session. Builds outside a request use the session that they concern: `df._plan.plan(client)` and `to_proto(client)` that of the client given, as PySpark's.
- Pickles of plans over a DataFrame's `_plan`, and of their Columns and table arguments, need no session. They load on the server's Python worker, as PySpark Connect's plans, through `_column_reduce.sent_plan`. A plan that a cloudpickle gave back pickles plainly again, as PySpark's.
- Unpickled and deep-copied Columns, sort orders, CASE WHENs, window specifications and table arguments keep the origins (call sites) of their expressions when PySpark's origin capture is on, as PySpark's unpickled expressions do.
- The garbage collector sees the Python plans and unbuildable expressions that a single holder holds: Columns, `_expr`, table arguments, DataFrames, window specifications, grouped data, merge builders and `DataFrameWriterV2`. So reference cycles through them are collected, as in PySpark.

### Why are the changes needed?

To close these differences from PySpark (the known-differences review).

### Does this PR introduce _any_ user-facing change?

Yes: the pickles and copies above behave as PySpark's. What remains is documented in `docs/compatibility.md` and `dev/parity/known-differences.md`:
- a cloudpickle of a DataFrame is larger (about 12.8 KB against PySpark's 1.2 KB for `spark.range(2)`);
- a cycle across two holders that share one plan or expression is not collected;
- an action of a DataFrame of an unpickled `df._plan` does not set the copies of its Observations;
- the build at construction of a plan whose build succeeds there but fails later;
- `to_plan(client)` of an unpickled Column of a failing plan's subquery raises the build error, where PySpark returns the expression.
The core crate gains doc-hidden items for the Python client (see the 4.4 Rust migration notes).

### How was this patch tested?

- New tests in `python/tests/test_dropin_offline.py`, the e2e tests of the server's Python worker in `python/tests/test_dropin_e2e.py`, and the core crate. Each round's new tests fail without its change.
- Five rounds of an independent review. It compared the drop-in with PySpark 4.3.0 through probes of:
  - pickles, cloudpickles and copies, also of the classic Column and TableArg classes and on the server's Python worker;
  - plans that fail to build, of 19 kinds of errors;
  - Observations, subqueries, deep and large plans, threads and sessions;
  - origins with the capture on and off;
  - the garbage collector, in 46 scenarios and a stress run of six threads with concurrent collections.
  Every finding was fixed or documented.
- The core tests (release and debug), wasm, clippy, rustdoc, the stubs, the offline suite in sorted order and under coverage, and the live e2e with and without protobuf.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62111-pickles-carry-functions-and-plans branch from 7377a59 to a2428ab Compare November 20, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant