Repository navigation
[SPARK-62101][CONNECT][PYTHON] Give the Connect Column, WindowSpec, TableArg, DataFrameNaFunctions, DataFrameStatFunctions and functions modules PySpark's docstrings as help() shows them, and compare the docstrings of every module - #896
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…ableArg, DataFrameNaFunctions, DataFrameStatFunctions and functions modules PySpark's docstrings as help() shows them, and compare the docstrings of every module A sweep of the docstrings of both clients, on master daacf3d, of the modules of pyspark.sql and pyspark.errors, found 65 of Spark Connect not PySpark's: - 59 of the Column, the WindowSpec, the TableArg, the DataFrameNaFunctions and the DataFrameStatFunctions and their members, of which 55 inherit their docstrings in PySpark and 4 have none (Column.astype, Column.name, Column.to_plan and TableArg.to_plan); - the module docstrings of pyspark.sql.connect.column, window, table_arg, functions and functions.builtin; - Read.print of pyspark.sql.connect.plan, which inherits its docstring in PySpark. PySpark Connect's classes there are subclasses of the classic ones, and neither they nor their members have docstrings of their own: their __doc__ is None, and inspect.getdoc and help() show those of the classic class, of their MRO. The drop-in's classes are native, virtual subclasses of the classic ones, which are not in their MRO, and their docstrings were their own doc comments, or none. - gen_connect_docstrings.py generates the docstrings that inspect.getdoc finds of the classes of INHERITING and of their members in PySpark, of the classic class (connect_doc!(Column.alias) is that of the classic Column.alias), but of an alias, of none (Column.name and Column.astype): the native Column, WindowSpec, TableArg, DataFrameNaFunctions and DataFrameStatFunctions (column.rs, window.rs, tablearg.rs, the NA functions of dataframe.rs, stat.rs) and their members take them. So inspect.getdoc and help() show PySpark's text, and their __doc__ is that text, where PySpark Connect's own is None. Their doc comments are plain comments now. - The modules connect/column.py, window.py, table_arg.py and functions/builtin.py have no docstring, as PySpark Connect's, of which their text stays as a comment; connect/functions/__init__.py has PySpark Connect's. - Read.print has the docstring of PySpark Connect's LogicalPlan.print, which inspect.getdoc finds of its own of none, as the drop-in's LogicalPlan has no print. - tests/golden/runtime_docstrings.json covers every module of pyspark.sql and pyspark.errors now: 5137 docstrings of 125 modules, of 3854 of 111. Each is of its kind, which its note gives: of the __doc__ of the object, or, of an object of no docstring of its own that inherits one (677 of them), {"getdoc": digest} of its inspect.getdoc, which help() shows. It leaves out the docstrings that are not all PySpark's yet: the DataFrame class of both DataFrame modules, and the modules of the session and its client (pyspark.sql.connect.session and pyspark.sql.connect.client). With the DataFrame class, it leaves out the docstrings that can only match PySpark's once those of the class do: the module docstrings of both DataFrame modules, and the 9 members of the classic DataFrameNaFunctions and DataFrameStatFunctions that copy the DataFrame's (drop.__doc__ = DataFrame.dropna.__doc__). The test compares at least 3500 docstrings now (3993). - known-differences.md describes the docstrings of the native classes, and docs/compatibility.md which docstrings the golden holds and leaves out. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62101-docstrings-rest2
branch
from
November 19, 2026 20:11
70fef6d to
8e82faf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by their exact text, on master daacf3d, after SPARK-62084 (#891). It covered the modules of
pyspark.sqlandpyspark.errorsthat both import, 124 of them: each module, each public class and function that it defines, and each public member of such a class, as Python finds it on the class. An object counts as the same if its__doc__is the same. Where PySpark's object has no docstring of its own and inherits one, it counts as the same if itsinspect.getdocis the same, ashelp()shows it. Of 5548 objects of PySpark:The 65 that this PR makes PySpark's are of the modules of the Column, the window, the table argument, the NA and stat functions, the functions, and the plans of Spark Connect:
pyspark.sql.connect.columnpyspark.sql.connect.windowpyspark.sql.connect.table_argpyspark.sql.connect.dataframepyspark.sql.connect.functionspyspark.sql.connect.functions.builtinpyspark.sql.connect.planThey are:
Column,WindowSpec,TableArg,DataFrameNaFunctionsandDataFrameStatFunctionsclasses and 50 of their members (Column.alias,Column.between,WindowSpec.orderBy,DataFrameNaFunctions.fill, ...).__doc__is None, andinspect.getdocandhelp()show the docstring of the classic class, from the MRO.Column.astype,Column.name,Column.to_planandTableArg.to_plan.pyspark.sql.connect.column,window,table_arg,functionsandfunctions.builtin.Read.printofpyspark.sql.connect.plan. In PySpark it inherits the docstring ofLogicalPlan.print, which the drop-in'sLogicalPlandoes not have.What still differs in these modules is the
DataFrameclass ofpyspark.sql.connect.dataframeand that module's docstring (below). Details are injira-SPARK-62101-docstrings-rest2.txt.The changes:
The native classes carry the inherited docstrings.
gen_connect_docstrings.pygenerates the docstring thatinspect.getdocfinds in PySpark for each class of its newINHERITINGand for each public member of it, which is that of the classic class:connect_doc!(Column.alias)is the classicColumn.alias's.inspect.getdocfinds none for it in PySpark:Column.nameandColumn.astype, which PySpark Connect's class decorator wraps apart from the members whose names they keep.Column(column.rs),WindowSpec(window.rs),TableArg(tablearg.rs),DataFrameNaFunctions(its region ofdataframe.rs) andDataFrameStatFunctions(stat.rs) and their members take them. Their doc comments are now plain comments.inspect.getdocandhelp()show PySpark's text, as with pyspark-client. Their__doc__is that text, where PySpark Connect's own is None.inspectfinds at run time in pyspark-client, for all 59 objects. The members'inspect.signatureand__text_signature__are unchanged.The Python modules have PySpark Connect's module docstrings.
connect/column.py,window.py,table_arg.pyandfunctions/builtin.pyhave none, and their former text stays as a comment.connect/functions/__init__.pyhas PySpark Connect's, "PySpark Functions with Spark Connect".Read.printhas the docstring of PySpark Connect'sLogicalPlan.print.The golden covers every module.
tests/golden/runtime_docstrings.jsonnow holds the digests of PySpark's docstrings of 125 modules, 5137 docstrings, up from 111 modules and 3854. These include:pyspark.sql.functionsandpyspark.sql.connect.functionswith their modules;date_part,datepartanduuidofpyspark.sql.connect.functions.builtinhave no docstring in either client, and those ofpyspark.sql.functions.builtinhave PySpark's in both.Each digest of the golden is of its kind, which a note of the golden gives:
__doc__of the object;{"getdoc": digest}, of itsinspect.getdoc, for an object of no docstring of its own that inherits one. That is whathelp()shows. 677 of them are of this kind, such as the members of PySpark Connect'sColumn, and an__init__of none, which showsobject.__init__'s.The test compares each the same way.
What the golden leaves out. It leaves out the docstrings that are not all PySpark's yet, which a follow-up makes PySpark's:
DataFrameclass of both DataFrame modules;pyspark.sql.connect.sessionandpyspark.sql.connect.client.With the
DataFrameclass, it also leaves out the docstrings that can only match PySpark's once those of the class do. These are the module docstrings of both DataFrame modules, and the 9 members of the classicDataFrameNaFunctionsandDataFrameStatFunctionsthat copy the DataFrame's (drop.__doc__ = DataFrame.dropna.__doc__).EXCLUDEDandof_the_dataframeofpython/tests/_runtime_docstrings.pysay so. The other docstrings of the two DataFrame modules are compared.The test compares more.
test_the_docstrings_of_the_modules_are_pysparksnow compares 3993 docstrings, and asserts at least 3500, up from 2500.Docs:
dev/parity/known-differences.mddescribes the docstrings of the native classes and ofRead.print;docs/compatibility.mddescribes them too, and what the golden holds and leaves out.What remains of the 374 that differ:
__init__ofRuntimeConfand ofObservation. PySpark Connect copies docstrings onto them, and a native class's__init__is CPython's slot, which keeps CPython's docstring. The other__init__slots show CPython's docstring withhelp(), as PySpark's__init__of none does.DataFrameclass of both DataFrame modules: 238. 113 of them inherit their docstrings in PySpark, which the follow-up gives the same way;pyspark.sql.connect.session(43),pyspark.sql.connect.client(1) andclient.core(79).The 986 missing names are as before. They are mostly PySpark Connect's plan and expression classes (798 and 145), which the drop-in builds in Rust, and the py4j converters of
pyspark.sql.types(28).Why are the changes needed?
So that
help(), IPython's?and the other tools that readinspect.getdocshow PySpark's documentation for Spark Connect's Column, WindowSpec, TableArg, NA and stat functions, as with pyspark-client, and so that the functions modules have PySpark Connect's module docstrings. Also, so that a test guards the docstrings of every module, except those of the follow-up, each ashelp()shows it where PySpark's inherits one.Does this PR introduce any user-facing change?
Yes:
help()andinspect.getdocof the native Column, WindowSpec, TableArg,DataFrameNaFunctionsandDataFrameStatFunctionsand of 50 of their members now show PySpark's classic docstrings, as with pyspark-client. They used to show the drop-in's own text, or none. Their__doc__is that text, where pyspark-client's is None.__doc__ofColumn.astype,Column.name,Column.to_planandTableArg.to_planis None, as pyspark-client's.help()ofRead.printofpyspark.sql.connect.planshows PySpark's.How was this patch tested?
The golden test,
test_the_docstrings_of_the_modules_are_pysparksoftest_dropin_class_modules_offline.py, with the new golden:The golden reproduces when captured again with pyspark-client.
test_dropin_class_modules_offline.pygives 260 passed; on master, 1 failed (this test) and 259 passed.A probe of both clients. The generated docstrings of the 59 objects of the five classes are exactly what
inspectfinds of them at run time in pyspark-client. Theinspect.signatureand__text_signature__of the five classes and their members are the same as on master.The sweep of both clients, before and after, in the tables above.
Official suites against the drop-in (
run_official_tests.py --dropin --only), the same on master and on this branch. These ran on the first revision of this PR, of the same code, which this revision gives docstrings:test_parity_columntest_connect_columntest_parity_stattest_connect_stattest_parity_dataframetest_parity_functionstest_connect_functiontest_parity_udtfThe one failure of
test_parity_functions, on both, istest_input_file_name_udf, because the server cannot read the local data path. Thetest_connect_*suites compare with classic PySpark (ReusedMixedTestCase), and skip where PySpark is remote only, as with pyspark-client.The gates, on master 05a0f48 (SPARK-62089), but the e2e subset, which ran before that rebase:
cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-depsof the four crates, also--features apache-spark-connect/wasm-udf;cargo test -p pyspark-rs --lib: 17 passed;gen_connect_docstrings.py --check,gen_type_stubs.py --check,gen_fn_dispatch.py --check;test_type_stubs_offline.py(14 passed, 4 skipped);coverage.yml: 6854 passed, 77 skipped;test_dropin_e2e.py -k "column or window or stat or na or functions or table or doc"against a Spark 4.3.0 server: 114 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.