Skip to content

[SPARK-62116][CONNECT][PYTHON] Skip the placeholder search of the plans built in Python when their bytes do not hold the type URL of a placeholder - #900

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62116-placeholder-fast-path
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62116-placeholder-fast-path

Conversation

@HyukjinKwon

@HyukjinKwon HyukjinKwon commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A follow-up of #877 (SPARK-62064), from the follow-up list of the review of the last 100 commits against pyspark-client 4.3.0.

#877 made the core resolve placeholders in the extensions of expressions. To do that, the core encodes the fields of every relation built in Python that holds expressions, and searches the encoding for the placeholder's type URL. The relations of DataFrames hold placeholders mostly at their inputs, and most plans hold no extension at all, so that encoding finds nothing in nearly every plan. On a project of 50,000 expressions it cost +27% when #877 measured it. On today's master, whose DataFrame(plan, session) costs more in Python, it is 3.3 ms of 42.6 ms, and 2.2 ms of the core's own resolution.

The Python client already has the bytes of each message that it gives the core: those of plan(session), or of the plan, the command, the analysis or the request of a session hook. The placeholder's type URL appears in those bytes wherever a placeholder is, also inside the message of an extension. So:

  • The core counts the type URLs in the client's bytes (plan::BuiltOutside::decoded, with memchr's memmem).
  • It first walks only the relations of the message and counts the placeholders there (plan::resolve_with, Search::Count).
  • A message given without its bytes is resolved everywhere, as before. This covers the core's own ML plans and the Rust API.
  • The offsets of the type URL in an encoding are found with memmem too. The result is the same, since the URL does not overlap itself.

The two counts can only agree when the fast path is safe:

  • each placeholder at a relation adds one type URL to the bytes, its own;
  • a type URL anywhere else adds one more, which the walk does not count.

Everything #877 does stays as it was:

  • placeholders in the extensions of expressions;
  • a placeholder whose common a plan changed, found by its structure and merged, both at a relation and in the message of an extension;
  • the extensions of commands and the ML commands;
  • the time of crafted deep nesting, linear in its length;
  • the grown stack of the own-fields encoding.

The five places where the Python client hands its bytes to the core now pass them through BuiltOutside::decoded:

  • DataFrame(plan, session) (dataframe_of_python_plan);
  • client.to_table and execute_command_as_iterator (client::execute_plan);
  • client._analyze (client::analyze_plan);
  • client.execute_command (_execute_command);
  • the requests of session hooks (call_execute_plan_hooks).

The doc-hidden DataFrame::from_relation, SparkSession::execute_plan_proto, execute_command_proto and analyze_plan_proto now take impl Into<BuiltOutside<_>>, so existing callers still pass the message alone. The doc-hidden plan::resolve_request takes one and returns the request.

memchr becomes a direct dependency of apache-spark-connect. It was already in its dependency tree, through regex, arrow-string and futures-util, so the build compiles no new crate. The license inventory of the JNI library, crates/spark-connect-server/jvm/native-licenses/dependencies.json, records the hash of Cargo.lock, so it is regenerated (scripts/gen_native_binary_licenses.py); its dependencies are the same.

Timings, on a loaded machine, release builds. DataFrame(plan, session) of a plan built in Python, with the plan built once outside the timing. Each value is the best of 15 alternating rounds of 15 runs. "master without the own fields" is master with #877's own-fields pass turned off, which is the cost to get back:

plan master master without the own fields this PR
a project of 1,000 expressions over a placeholder 0.764 ms 0.719 ms 0.728 ms
a project of 10,000 expressions over a placeholder 7.215 ms 6.568 ms 6.633 ms
a project of 50,000 expressions over a placeholder 42.557 ms 39.284 ms 38.821 ms
a project of 1,000 expressions over a range, of no placeholder 0.672 ms 0.629 ms 0.630 ms
a project of 10,000 expressions over a range, of no placeholder 7.321 ms 6.809 ms 7.015 ms
a project of 50,000 expressions over a range, of no placeholder 42.153 ms 39.025 ms 39.166 ms
a project of 50,000 expressions over a placeholder, one of them a plugin's expression that holds another placeholder (searched everywhere) 49.675 ms (does not resolve it) 49.735 ms

The last row is of 15 rounds of its own, of the best of 15 runs each. The medians of its rounds were 53.3 and 54.6 ms, within the spread of the rounds (49.7 to 65.9 ms).

Those builds are of master 0e91867. On master bcb378f, 9 alternating rounds gave 42.6 ms on master against 39.2 ms here over a placeholder, 42.3 ms against 39.1 ms of no placeholder, and 53.1 ms against 51.5 ms with a placeholder in an expression, of 50,000 expressions.

The core's part alone, the resolution of the relation once decoded (best of 15, release build):

plan every field searched (#877) this PR search taken the count of the type URLs
a project of 10,000 expressions over a placeholder 0.396 ms 0.004 ms at the relations 0.006 ms
a project of 50,000 expressions over a placeholder 2.231 ms 0.039 ms at the relations 0.035 ms
a project of 50,000 expressions of no placeholder 2.220 ms 0.037 ms at the relations 0.032 ms
the same over a placeholder, with one in a plugin's expression, 10,000 expressions 1.756 ms 1.739 ms everywhere 0.005 ms
the same, 50,000 expressions 10.759 ms 10.802 ms everywhere 0.037 ms

So a plan that holds a placeholder outside its relations pays only the count of the type URLs in its bytes, 0.04 ms for 400 KB. Both columns find the offsets of the URL with memmem.

Why are the changes needed?

To get back #877's cost on the plans that hold no placeholder outside their relations, which is nearly every plan, while keeping everything #877 resolves.

Does this PR introduce any user-facing change?

No. The messages are sent as before. The plans built in Python whose placeholders are all at their relations, or that hold none, are resolved faster.

How was this patch tested?

  • New Rust unit tests in plan.rs's placeholder_tests. A test-only record of the search of each resolution shows when the fast path is taken:
    • an_encoding_of_the_type_urls_of_the_placeholders_at_its_relations_alone_is_resolved_at_them: the fast path is taken for no placeholder, an input, the inputs of a join, ML transforms, a fetched argument with a summary's dataset, a WithRelations, and a changed placeholder. Each gives the relation and the levels of the full search.
    • an_encoding_of_a_type_url_elsewhere_is_resolved_everywhere: the full search is kept for a placeholder in the extension of an expression (also with one at an input), in the message of an extension (also 50 levels deep), for a string that is the type URL, and for a placeholder whose bytes hold the type URL. Each is resolved as expected, and as the full search resolves it.
    • the_commands_plans_analyses_and_requests_of_an_encoding_are_resolved_of_its_search: the same for a write and an ML fit (fast path), and for a merge condition, a command's extension and an Any of the type URL in a request's user context (full search). Plans, requests of session hooks and an analysis of two plans are covered too.
    • a_message_that_does_not_decode_is_its_decoder_s_error.
  • New offline test in python/tests/test_dropin_offline.py: test_every_message_built_in_python_sends_the_relations_of_its_dataframes_as_pyspark. It runs the five entry points (DataFrame(plan, session), to_table, execute_command, _analyze and a session hook), each with a placeholder at an input (fast path), in the extension of an expression (full search), and next to a string that is the type URL (full search). The mock server receives the relations, and the string as it is.
  • The existing tests of [SPARK-62064][CONNECT][PYTHON] Send the relations of DataFrames that the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans #877 and before pass unchanged, for placeholders in expressions, changed placeholders, deep messages, ML commands and plugins' messages.
  • The Rust gates (gates.sh build rust py):
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps, also as lint.yml runs it, with the macros and build crates and --features apache-spark-connect/wasm-udf;
    • scripts/gen_binary_licenses.py --check: LICENSE-binary and NOTICE-binary are up to date, as memchr was in the tree already;
    • scripts/gen_native_binary_licenses.py --check and the unit tests of both license generators (35 passed), as native-server.yml runs them;
    • cargo test -p pyspark-rs --lib: 17 passed.
  • cargo test -p apache-spark-connect --no-fail-fast in a debug build, with its live tests against a Spark 4.3.0 server: 1,431 passed, 0 failed, 15 ignored, of 109 test binaries and doc-test runs (the lib's 848, and tests/raw_relation.rs's 10).
  • The Python gates:
    • ruff 0.14.0;
    • test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • the stub and dispatch generators;
    • the test-list guards;
    • test_type_stubs_offline.py: 14 passed, 4 skipped;
    • the offline suite of coverage.yml: 6,913 passed, 77 skipped.
  • e2e against a Spark 4.3.0 server: test_dropin_e2e.py: 352 passed; test_ml_e2e.py: 32 passed.
  • Official suites pyspark.ml.tests.connect.test_parity_pipeline, test_parity_feature and test_parity_tuning through run_official_tests.py --dropin, on master and here: the same on both, with no unexpected failures: test_parity_pipeline 6 passed and 4 skipped, test_parity_feature 51 passed, test_parity_tuning 7 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…ns built in Python when their bytes do not hold the type URL of a placeholder

The core reads the fields of the relations and the commands of a message that the Python client
built, and the messages of its extensions, for placeholders only when the bytes that the client
gave hold a type URL of a placeholder that is not of one at the relations of the message, which
the core meets as it walks them (plan::BuiltOutside, plan::resolve_with).

- BuiltOutside::decoded counts the type URLs of placeholders in the encoding (memchr's memmem). A
  first walk of the relations counts the placeholders at them, and when the counts agree the
  message is resolved at its relations alone: a plan whose placeholders are all at its relations,
  or that holds none, costs a search of its bytes, and no encoding of its fields.
- DataFrame(plan, session), the plans of to_table, the commands, the analyses and the requests of
  the session hooks give the core their bytes; a message given without them is read everywhere,
  as before.
- The offsets of the type URL in an encoding are found with memmem too.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62116-placeholder-fast-path branch from 3e608d4 to 3a60b0d Compare November 20, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant