Repository navigation
[SPARK-62114][CONNECT][PYTHON] Make the attributes and thread-local state of SparkConnectClient, and the lifetime of a session, PySpark's, of plain attributes that the requests read when they are built, in the Python client - #898
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…tate of SparkConnectClient, and the lifetime of a session, PySpark's, of plain attributes that the requests read when they are built, in the Python client ### What changes were proposed in this pull request? The `SparkConnectClient` of the Python client (`pyspark-client-rust`) kept its attributes in the Rust transport, behind Python subclasses that kept the two in step, and a kept `conf` or `client` held its session open. PySpark's client holds plain attributes, which its requests read when they are built, and does not keep the session object. This closes the differences that `dev/parity/known-differences.md` listed for them: - The client's `thread_local`, its set of tags and its lists of user context extensions are PySpark's plain `threading.local`, set and lists. So are `global_user_context_extensions`, `_session_id`, `_user_id` and `_closed`, plain attributes that a set changes for the requests. - The transport reads them on the calling thread when a call starts its requests, and when a finalizer sends its request. - A `copy.copy` of a client sends its own attributes, also at once with its client on another thread. - A call that runs during another call of the same thread, of a progress handler or a finalizer, sends its own client's attributes. The outer call then reads its client's attributes again, as they are then, as PySpark's requests do. The requests of a call's failure (the details of its error and the configs of its stack trace) and the interrupt of a Ctrl-C read the attributes of the call's own client when they are built, of a copy's execution iterator too. All per-thread state is keyed by the thread that waits for the call, so the interrupt carries the extensions of the thread, and nothing more is requested once a signal interrupted the wait. - A kept `conf` or `client` does not keep its session open. Once the program dropped the session and the objects that hold it, they raise `NO_ACTIVE_SESSION`, as PySpark's do once the session's `__del__` closed the client, and the exit sends nothing for the session. - `channelBuilder(...)` takes any object, and the session fails, when it is made, of the errors of PySpark's client. - A failed `CloneSession` asks the server for nothing more, as PySpark's `clone`: neither the details of its error nor the configs of its stack trace. The response of a clone is not checked. ### Why are the changes needed? To close these differences from PySpark (the known-differences review). ### Does this PR introduce _any_ user-facing change? Yes: the client and session behave as PySpark's above. What remains is documented in `docs/compatibility.md` and `dev/parity/known-differences.md`. The main case: a call's other requests after its first, such as the lookup of the plan compression, carry the attributes of the call's start, or of the end of a nested call of its session. The core crates gain doc-hidden items for the Python client, which the 4.4 Rust migration notes list. ### How was this patch tested? - New tests in `python/tests/test_dropin_offline.py`, `test_exit_cleanup_offline.py` and `test_dropin_e2e.py`, and in the core crates. The mock server gained knobs for them. Each round's new tests fail without its change. - Five rounds of an independent review, which compared the requests of the drop-in and of PySpark 4.3.0 through a recording proxy. It covered: - the attributes of clients and their copies, threads and thread-inherited targets; - nested calls of progress handlers and finalizers; - Ctrl-C; - clones and new sessions of other users; - the lifetimes of sessions, confs and clients; - leaks of the entries of 9000 threads, and the performance of small calls. Every finding was fixed or documented. - The core tests (release and debug), wasm, clippy, rustdoc, the stubs, the offline suite in sorted order and under coverage, and the live e2e with and without protobuf. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62114-client-attributes-and-session-lifetime
branch
from
November 20, 2026 05:00
7e585d5 to
4e227fb
Compare
HyukjinKwon
added a commit
that referenced
this pull request
Nov 20, 2026
…ssic pyspark.sql.DataFrame PySpark's docstrings as help() shows them ### What changes were proposed in this pull request? A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by their exact text, on master 0e91867. It covered the modules of `pyspark.sql` and `pyspark.errors` that both import, 124 of them: each module, each public class and function that it defines, and each public member of such a class, as Python finds it on the class. As the golden compares them (SPARK-62101), an object counts as the same if its `__doc__` is the same. Where PySpark's object has no docstring of its own and inherits one, it counts as the same if its `inspect.getdoc` is the same, as `help()` shows it. Of 5548 objects of PySpark: | | Same | Differ | Missing in the drop-in | Modules all same | |---|---|---|---|---| | Before, master 0e91867 | 4188 | 374 | 986 | 112 | | After, this PR | 4436 | 126 | 986 | 113 | The 248 that this PR makes PySpark's are those of the two DataFrame modules: | Module | Before: same / differ / missing | After | What differed | |---|---|---|---| | `pyspark.sql.connect.dataframe` | 13 / 121 / 0 | 133 / 1 / 0 | the module, the `DataFrame` class and 118 members | | `pyspark.sql.dataframe` | 5 / 128 / 0 | 133 / 0 / 0 | the module, the `DataFrame` class, 117 members, and the 9 members of `DataFrameNaFunctions` and `DataFrameStatFunctions` that copy the DataFrame's (`drop.__doc__ = DataFrame.dropna.__doc__`) | The one left of `pyspark.sql.connect.dataframe` is the `__init__` of the native class, CPython's slot (below). **Why they differed:** - **The Connect DataFrame.** PySpark Connect's `DataFrame` is a subclass of the classic one. Neither it nor its members have docstrings of their own, except its `__init__`, so `inspect.getdoc` and `help()` show those of the classic DataFrame. - Its aliases, such as `groupby = groupBy`, show the docstring of the member that they name. - The members that it wraps in a cache of functools show none: `isEmpty`, `isLocal`, `inputFiles` and `semanticHash` (`functools.cache`), and `isStreaming` (`functools.cached_property`). So does `is_cached`, of its own. The drop-in's Connect DataFrame is native, a virtual subclass of the classic one, which is not in its MRO. Its docstrings were its own doc comments. - **The classic DataFrame.** The drop-in's `pyspark.sql.DataFrame`, the public base, takes the docstrings of the native class, and the 9 NA and stat members copy the public base's. **The changes:** - **The native DataFrame carries the inherited docstrings**, as the Column and the other classes of SPARK-62101 do. - `gen_connect_docstrings.py` generates the docstrings that `inspect.getdoc` finds in PySpark of the DataFrame and of its members too (`INHERITING`). - An alias of a class that no decorator wraps has the docstring of the member that it names, as `inspect.getdoc` finds it there. Where a decorator of the class wraps it apart from that member (`with_origin_to_class` of PySpark Connect's Column, `WRAPPING_CLASS_DECORATORS`), it has none, as before. - A member that PySpark Connect wraps in a cache of functools has none, as `inspect.getdoc` finds none for it. - The native `DataFrame` of `dataframe.rs` (its DataFrame region) and 112 of its members take them. The 6 of no docstring in PySpark have none. Their doc comments are now plain comments. - The generated docstrings are exactly what `inspect` finds at run time in pyspark-client, for all 119 objects of PySpark Connect's DataFrame but its `__init__`. The `inspect.signature` and `__text_signature__` of the DataFrame of both modules and of its members are unchanged. - **The public base takes PySpark's classic docstrings.** It takes those of the native class, and, where PySpark's classic and Connect DataFrames differ, those of the classic DataFrame. These are `drop_duplicates`, `groupby`, `melt`, and the 5 members of the caches. - The generator writes them in `python/pyspark/sql/_classic_docs_generated.py` (`PUBLIC_BASES`), which its `--check` guards with the Rust file (`generated-drift.yml`), and which ruff leaves out, as the other generated files. - The 9 NA and stat copies are then PySpark's too. - **The modules have no docstring, as PySpark's.** That covers `pyspark/sql/dataframe.py` and `pyspark/sql/connect/dataframe.py`; their former text stays as a comment. - **The golden covers the DataFrame.** - `of_the_dataframe` is gone. `tests/golden/runtime_docstrings.json` now holds 5387 docstrings of 125 modules, up from 5137, of which 790 are of the `getdoc` kind, up from 677. - It leaves out only the modules of the session and its client, `pyspark.sql.connect.session` and `pyspark.sql.connect.client`, whose docstrings are not all PySpark's yet. - `test_the_docstrings_of_the_modules_are_pysparks` now compares 4241 docstrings, and asserts at least 4000, up from 3500. - **Docs:** - `dev/parity/known-differences.md` names the DataFrame among the native classes that carry the classic docstrings, and among those whose `__init__` is CPython's slot. It also says that the public base's dispatchers have PySpark's classic docstrings; - `docs/compatibility.md` describes the docstrings of both DataFrames and what the golden leaves out. **What remains** of the 126 that differ: - **Inherent, 3:** the `__init__` of `DataFrame`, `RuntimeConf` and `Observation` of Spark Connect, which have docstrings in PySpark Connect (its own of the DataFrame's, "Creates a new data frame", and copies of the others). A native class's `__init__` is CPython's slot, which keeps CPython's docstring. - **Of other work, left out of the golden:** - `pyspark.sql.connect.session` (43), after SPARK-62114 (#898); - `pyspark.sql.connect.client` and `client.core` (80, and 6 of the missing names), after the move of `client.core`. The 986 missing names are as before. They are mostly PySpark Connect's plan and expression classes (798 and 145), which the drop-in builds in Rust, and the py4j converters of `pyspark.sql.types` (28). ### Why are the changes needed? So that `help()`, IPython's `?` and the other tools that read `inspect.getdoc` show PySpark's documentation for the Connect DataFrame, as with pyspark-client; so that the classic `pyspark.sql.DataFrame`, the `DataFrameNaFunctions` and the `DataFrameStatFunctions` have PySpark's docstrings; and so that the golden guards them. ### Does this PR introduce _any_ user-facing change? Yes: - `help()` and `inspect.getdoc` of the Connect DataFrame and of 112 of its members show PySpark's classic docstrings, as with pyspark-client. They used to show the drop-in's own text. Their `__doc__` is that text, where pyspark-client's is None. - The 6 members of no docstring in pyspark-client (the 5 of the caches and `is_cached`) have none. - The `__doc__` of the classic `pyspark.sql.DataFrame` and of its 117 members, and of the 9 NA and stat members that copy them, is PySpark's. - The two module docstrings are None, as pyspark-client's. ### How was this patch tested? - **The golden test**, `test_the_docstrings_of_the_modules_are_pysparks` of `test_dropin_class_modules_offline.py`, with the new golden: - on master, 248 of 4241 differ, the 248 above; - on this branch, none. The golden reproduces when captured again with pyspark-client. `test_dropin_class_modules_offline.py` gives 260 passed; on master, 1 failed (this test) and 259 passed. - **A probe of both clients.** The generated docstrings are exactly what `inspect` finds at run time in pyspark-client: of the 119 objects of PySpark Connect's DataFrame, and of the 8 of the classic DataFrame that the public base takes in their place. Every object of the DataFrame of both modules then has PySpark's docstring, by its kind. The signatures of the 237 objects of both DataFrames are those of master. - **The sweep of both clients**, before and after, in the tables above. - **The gates:** - `cargo fmt --all --check`; - `cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings`; - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core`; - `cargo test -p pyspark-rs --lib`: 17 passed; - `gen_connect_docstrings.py --check` (both files), `gen_type_stubs.py --check`, `gen_fn_dispatch.py --check`, `check_classic_bindings.py`; - ruff 0.14.0, the no-stub audit, the guards and `test_type_stubs_offline.py` with pyright 1.1.414 (18 passed); - the offline suite of `coverage.yml`: 6884 passed, 77 skipped; - `test_dropin_e2e.py -k "dataframe or column or window or stat or na or functions or table or doc"` against a Spark 4.3.0 server: 152 passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. Closes #899 from HyukjinKwon/SPARK-62115-docstrings-dataframe. Authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The
SparkConnectClientof the Python client (pyspark-client-rust) kept its attributes in the Rust transport, behind Python subclasses that kept the two in step, and a keptconforclientheld its session open. PySpark's client holds plain attributes, which its requests read when they are built, and does not keep the session object. This closes the differences thatdev/parity/known-differences.mdlisted for them:thread_local, its set of tags and its lists of user context extensions are PySpark's plainthreading.local, set and lists. So areglobal_user_context_extensions,_session_id,_user_idand_closed, plain attributes that a set changes for the requests.copy.copyof a client sends its own attributes, also at once with its client on another thread.conforclientdoes not keep its session open. Once the program dropped the session and the objects that hold it, they raiseNO_ACTIVE_SESSION, as PySpark's do once the session's__del__closed the client, and the exit sends nothing for the session.channelBuilder(...)takes any object, and the session fails, when it is made, of the errors of PySpark's client.CloneSessionasks the server for nothing more, as PySpark'sclone: neither the details of its error nor the configs of its stack trace. The response of a clone is not checked.Why are the changes needed?
To close these differences from PySpark (the known-differences review).
Does this PR introduce any user-facing change?
Yes: the client and session behave as PySpark's above. What remains is documented in
docs/compatibility.mdanddev/parity/known-differences.md. The main case: thenext()of a copy's execution iterator reads nothing of the client, so its requests carry the context of the thread's last call. The core crates gain doc-hidden items for the Python client, which the 4.4 Rust migration notes list.How was this patch tested?
python/tests/test_dropin_offline.py,test_exit_cleanup_offline.pyandtest_dropin_e2e.py, and in the core crates. The mock server gained knobs for them. Each round's new tests fail without its change.Every finding was fixed or documented.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
This pull request and its description were written by Isaac.