Skip to content

[SPARK-62101][CONNECT][PYTHON] Give the Connect Column, WindowSpec, TableArg, DataFrameNaFunctions, DataFrameStatFunctions and functions modules PySpark's docstrings as help() shows them, and compare the docstrings of every module - #896

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62101-docstrings-rest2

Conversation

@HyukjinKwon

@HyukjinKwon HyukjinKwon commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by their exact text, on master daacf3d, after SPARK-62084 (#891). It covered the modules of pyspark.sql and pyspark.errors that both import, 124 of them: each module, each public class and function that it defines, and each public member of such a class, as Python finds it on the class. An object counts as the same if its __doc__ is the same. Where PySpark's object has no docstring of its own and inherits one, it counts as the same if its inspect.getdoc is the same, as help() shows it. Of 5548 objects of PySpark:

Same Differ Missing in the drop-in Modules all same
Before, master daacf3d 4123 439 986 107
After, this PR 4188 374 986 112

The 65 that this PR makes PySpark's are of the modules of the Column, the window, the table argument, the NA and stat functions, the functions, and the plans of Spark Connect:

Module Before: same / differ / missing After
pyspark.sql.connect.column 1 / 39 / 0 40 / 0 / 0
pyspark.sql.connect.window 12 / 6 / 0 18 / 0 / 0
pyspark.sql.connect.table_arg 1 / 6 / 0 7 / 0 / 0
pyspark.sql.connect.dataframe 2 / 132 / 0 13 / 121 / 0
pyspark.sql.connect.functions 0 / 1 / 0 1 / 0 / 0
pyspark.sql.connect.functions.builtin 562 / 1 / 0 563 / 0 / 0
pyspark.sql.connect.plan 43 / 1 / 798 44 / 0 / 798

They are:

  • 55 inherited docstrings: the Column, WindowSpec, TableArg, DataFrameNaFunctions and DataFrameStatFunctions classes and 50 of their members (Column.alias, Column.between, WindowSpec.orderBy, DataFrameNaFunctions.fill, ...).
    • PySpark Connect's classes are subclasses of the classic ones, and neither they nor these members have docstrings of their own. So their __doc__ is None, and inspect.getdoc and help() show the docstring of the classic class, from the MRO.
    • The drop-in's classes are native and are virtual subclasses of the classic ones, which are not in their MRO. Their docstrings were their own doc comments, or none.
  • 4 members of no docstring in PySpark, which had their own: Column.astype, Column.name, Column.to_plan and TableArg.to_plan.
  • 5 module docstrings: those of pyspark.sql.connect.column, window, table_arg, functions and functions.builtin.
  • Read.print of pyspark.sql.connect.plan. In PySpark it inherits the docstring of LogicalPlan.print, which the drop-in's LogicalPlan does not have.

What still differs in these modules is the DataFrame class of pyspark.sql.connect.dataframe and that module's docstring (below). Details are in jira-SPARK-62101-docstrings-rest2.txt.

The changes:

  • The native classes carry the inherited docstrings. gen_connect_docstrings.py generates the docstring that inspect.getdoc finds in PySpark for each class of its new INHERITING and for each public member of it, which is that of the classic class: connect_doc!(Column.alias) is the classic Column.alias's.

    • An alias has none, as inspect.getdoc finds none for it in PySpark: Column.name and Column.astype, which PySpark Connect's class decorator wraps apart from the members whose names they keep.
    • The native Column (column.rs), WindowSpec (window.rs), TableArg (tablearg.rs), DataFrameNaFunctions (its region of dataframe.rs) and DataFrameStatFunctions (stat.rs) and their members take them. Their doc comments are now plain comments.
    • So inspect.getdoc and help() show PySpark's text, as with pyspark-client. Their __doc__ is that text, where PySpark Connect's own is None.
    • The generated docstrings are exactly what inspect finds at run time in pyspark-client, for all 59 objects. The members' inspect.signature and __text_signature__ are unchanged.
  • The Python modules have PySpark Connect's module docstrings. connect/column.py, window.py, table_arg.py and functions/builtin.py have none, and their former text stays as a comment. connect/functions/__init__.py has PySpark Connect's, "PySpark Functions with Spark Connect".

  • Read.print has the docstring of PySpark Connect's LogicalPlan.print.

  • The golden covers every module. tests/golden/runtime_docstrings.json now holds the digests of PySpark's docstrings of 125 modules, 5137 docstrings, up from 111 modules and 3854. These include:

    • pyspark.sql.functions and pyspark.sql.connect.functions with their modules;
    • the Column, window and table argument modules of both;
    • the NA and stat functions of both DataFrame modules.

    date_part, datepart and uuid of pyspark.sql.connect.functions.builtin have no docstring in either client, and those of pyspark.sql.functions.builtin have PySpark's in both.

  • Each digest of the golden is of its kind, which a note of the golden gives:

    • by default, of the __doc__ of the object;
    • {"getdoc": digest}, of its inspect.getdoc, for an object of no docstring of its own that inherits one. That is what help() shows. 677 of them are of this kind, such as the members of PySpark Connect's Column, and an __init__ of none, which shows object.__init__'s.

    The test compares each the same way.

  • What the golden leaves out. It leaves out the docstrings that are not all PySpark's yet, which a follow-up makes PySpark's:

    • the DataFrame class of both DataFrame modules;
    • the modules of the session and its client, pyspark.sql.connect.session and pyspark.sql.connect.client.

    With the DataFrame class, it also leaves out the docstrings that can only match PySpark's once those of the class do. These are the module docstrings of both DataFrame modules, and the 9 members of the classic DataFrameNaFunctions and DataFrameStatFunctions that copy the DataFrame's (drop.__doc__ = DataFrame.dropna.__doc__). EXCLUDED and of_the_dataframe of python/tests/_runtime_docstrings.py say so. The other docstrings of the two DataFrame modules are compared.

  • The test compares more. test_the_docstrings_of_the_modules_are_pysparks now compares 3993 docstrings, and asserts at least 3500, up from 2500.

  • Docs:

    • dev/parity/known-differences.md describes the docstrings of the native classes and of Read.print;
    • docs/compatibility.md describes them too, and what the golden holds and leaves out.

What remains of the 374 that differ:

  • Inherent, 2: the __init__ of RuntimeConf and of Observation. PySpark Connect copies docstrings onto them, and a native class's __init__ is CPython's slot, which keeps CPython's docstring. The other __init__ slots show CPython's docstring with help(), as PySpark's __init__ of none does.
  • The follow-up of the DataFrame, the session and the client, 372:
    • the DataFrame class of both DataFrame modules: 238. 113 of them inherit their docstrings in PySpark, which the follow-up gives the same way;
    • the module docstrings of both: 2;
    • the 9 classic NA and stat members that copy the DataFrame's;
    • pyspark.sql.connect.session (43), pyspark.sql.connect.client (1) and client.core (79).

The 986 missing names are as before. They are mostly PySpark Connect's plan and expression classes (798 and 145), which the drop-in builds in Rust, and the py4j converters of pyspark.sql.types (28).

Why are the changes needed?

So that help(), IPython's ? and the other tools that read inspect.getdoc show PySpark's documentation for Spark Connect's Column, WindowSpec, TableArg, NA and stat functions, as with pyspark-client, and so that the functions modules have PySpark Connect's module docstrings. Also, so that a test guards the docstrings of every module, except those of the follow-up, each as help() shows it where PySpark's inherits one.

Does this PR introduce any user-facing change?

Yes:

  • help() and inspect.getdoc of the native Column, WindowSpec, TableArg, DataFrameNaFunctions and DataFrameStatFunctions and of 50 of their members now show PySpark's classic docstrings, as with pyspark-client. They used to show the drop-in's own text, or none. Their __doc__ is that text, where pyspark-client's is None.
  • The __doc__ of Column.astype, Column.name, Column.to_plan and TableArg.to_plan is None, as pyspark-client's.
  • The 5 module docstrings are PySpark Connect's, 4 of them None.
  • help() of Read.print of pyspark.sql.connect.plan shows PySpark's.

How was this patch tested?

  • The golden test, test_the_docstrings_of_the_modules_are_pysparks of test_dropin_class_modules_offline.py, with the new golden:

    • on master, 65 of 3993 differ, the 65 above;
    • on this branch, none.

    The golden reproduces when captured again with pyspark-client. test_dropin_class_modules_offline.py gives 260 passed; on master, 1 failed (this test) and 259 passed.

  • A probe of both clients. The generated docstrings of the 59 objects of the five classes are exactly what inspect finds of them at run time in pyspark-client. The inspect.signature and __text_signature__ of the five classes and their members are the same as on master.

  • The sweep of both clients, before and after, in the tables above.

  • Official suites against the drop-in (run_official_tests.py --dropin --only), the same on master and on this branch. These ran on the first revision of this PR, of the same code, which this revision gives docstrings:

    Suite Passed (skipped)
    test_parity_column 137 (2)
    test_connect_column 0 (28)
    test_parity_stat 9
    test_connect_stat 0 (18)
    test_parity_dataframe 53 (4)
    test_parity_functions 164 (4)
    test_connect_function 0 (31)
    test_parity_udtf 387 (21)

    The one failure of test_parity_functions, on both, is test_input_file_name_udf, because the server cannot read the local data path. The test_connect_* suites compare with classic PySpark (ReusedMixedTestCase), and skip where PySpark is remote only, as with pyspark-client.

  • The gates, on master 05a0f48 (SPARK-62089), but the e2e subset, which ran before that rebase:

    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps of the four crates, also --features apache-spark-connect/wasm-udf;
    • cargo test -p pyspark-rs --lib: 17 passed;
    • gen_connect_docstrings.py --check, gen_type_stubs.py --check, gen_fn_dispatch.py --check;
    • ruff 0.14.0, the no-stub audit, the guards and test_type_stubs_offline.py (14 passed, 4 skipped);
    • the offline suite of coverage.yml: 6854 passed, 77 skipped;
    • test_dropin_e2e.py -k "column or window or stat or na or functions or table or doc" against a Spark 4.3.0 server: 114 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…ableArg, DataFrameNaFunctions, DataFrameStatFunctions and functions modules PySpark's docstrings as help() shows them, and compare the docstrings of every module

A sweep of the docstrings of both clients, on master daacf3d, of the modules of pyspark.sql and pyspark.errors,
found 65 of Spark Connect not PySpark's:
- 59 of the Column, the WindowSpec, the TableArg, the DataFrameNaFunctions and the DataFrameStatFunctions and their
  members, of which 55 inherit their docstrings in PySpark and 4 have none (Column.astype, Column.name,
  Column.to_plan and TableArg.to_plan);
- the module docstrings of pyspark.sql.connect.column, window, table_arg, functions and functions.builtin;
- Read.print of pyspark.sql.connect.plan, which inherits its docstring in PySpark.

PySpark Connect's classes there are subclasses of the classic ones, and neither they nor their members have
docstrings of their own: their __doc__ is None, and inspect.getdoc and help() show those of the classic class, of
their MRO. The drop-in's classes are native, virtual subclasses of the classic ones, which are not in their MRO,
and their docstrings were their own doc comments, or none.

- gen_connect_docstrings.py generates the docstrings that inspect.getdoc finds of the classes of INHERITING and of
  their members in PySpark, of the classic class (connect_doc!(Column.alias) is that of the classic Column.alias),
  but of an alias, of none (Column.name and Column.astype): the native Column, WindowSpec, TableArg,
  DataFrameNaFunctions and DataFrameStatFunctions (column.rs, window.rs, tablearg.rs, the NA functions of
  dataframe.rs, stat.rs) and their members take them. So inspect.getdoc and help() show PySpark's text, and their
  __doc__ is that text, where PySpark Connect's own is None. Their doc comments are plain comments now.
- The modules connect/column.py, window.py, table_arg.py and functions/builtin.py have no docstring, as PySpark
  Connect's, of which their text stays as a comment; connect/functions/__init__.py has PySpark Connect's.
- Read.print has the docstring of PySpark Connect's LogicalPlan.print, which inspect.getdoc finds of its own of
  none, as the drop-in's LogicalPlan has no print.
- tests/golden/runtime_docstrings.json covers every module of pyspark.sql and pyspark.errors now: 5137 docstrings of
  125 modules, of 3854 of 111. Each is of its kind, which its note gives: of the __doc__ of the object, or, of an
  object of no docstring of its own that inherits one (677 of them), {"getdoc": digest} of its inspect.getdoc,
  which help() shows. It leaves out the docstrings that are not all PySpark's yet: the DataFrame class of both
  DataFrame modules, and the modules of the session and its client (pyspark.sql.connect.session and
  pyspark.sql.connect.client). With the DataFrame class, it leaves out the docstrings that can only match PySpark's
  once those of the class do: the module docstrings of both DataFrame modules, and the 9 members of the classic
  DataFrameNaFunctions and DataFrameStatFunctions that copy the DataFrame's (drop.__doc__ = DataFrame.dropna.__doc__).
  The test compares at least 3500 docstrings now (3993).
- known-differences.md describes the docstrings of the native classes, and docs/compatibility.md which docstrings
  the golden holds and leaves out.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62101-docstrings-rest2 branch from 70fef6d to 8e82faf Compare November 19, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant