Repository navigation
[SPARK-62116][CONNECT][PYTHON] Skip the placeholder search of the plans built in Python when their bytes do not hold the type URL of a placeholder - #900
Closed
HyukjinKwon wants to merge 1 commit into
Conversation
…ns built in Python when their bytes do not hold the type URL of a placeholder The core reads the fields of the relations and the commands of a message that the Python client built, and the messages of its extensions, for placeholders only when the bytes that the client gave hold a type URL of a placeholder that is not of one at the relations of the message, which the core meets as it walks them (plan::BuiltOutside, plan::resolve_with). - BuiltOutside::decoded counts the type URLs of placeholders in the encoding (memchr's memmem). A first walk of the relations counts the placeholders at them, and when the counts agree the message is resolved at its relations alone: a plan whose placeholders are all at its relations, or that holds none, costs a search of its bytes, and no encoding of its fields. - DataFrame(plan, session), the plans of to_table, the commands, the analyses and the requests of the session hooks give the core their bytes; a message given without them is read everywhere, as before. - The offsets of the type URL in an encoding are found with memmem too. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62116-placeholder-fast-path
branch
from
November 20, 2026 05:46
3e608d4 to
3a60b0d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A follow-up of #877 (SPARK-62064), from the follow-up list of the review of the last 100 commits against pyspark-client 4.3.0.
#877 made the core resolve placeholders in the extensions of expressions. To do that, the core encodes the fields of every relation built in Python that holds expressions, and searches the encoding for the placeholder's type URL. The relations of DataFrames hold placeholders mostly at their inputs, and most plans hold no extension at all, so that encoding finds nothing in nearly every plan. On a project of 50,000 expressions it cost +27% when #877 measured it. On today's master, whose
DataFrame(plan, session)costs more in Python, it is 3.3 ms of 42.6 ms, and 2.2 ms of the core's own resolution.The Python client already has the bytes of each message that it gives the core: those of
plan(session), or of the plan, the command, the analysis or the request of a session hook. The placeholder's type URL appears in those bytes wherever a placeholder is, also inside the message of an extension. So:plan::BuiltOutside::decoded, with memchr'smemmem).plan::resolve_with,Search::Count).Search::Relations).Search::Everywhere), and that resolution tells these apart.memmemtoo. The result is the same, since the URL does not overlap itself.The two counts can only agree when the fast path is safe:
Everything #877 does stays as it was:
commona plan changed, found by its structure and merged, both at a relation and in the message of an extension;The five places where the Python client hands its bytes to the core now pass them through
BuiltOutside::decoded:DataFrame(plan, session)(dataframe_of_python_plan);client.to_tableandexecute_command_as_iterator(client::execute_plan);client._analyze(client::analyze_plan);client.execute_command(_execute_command);call_execute_plan_hooks).The doc-hidden
DataFrame::from_relation,SparkSession::execute_plan_proto,execute_command_protoandanalyze_plan_protonow takeimpl Into<BuiltOutside<_>>, so existing callers still pass the message alone. The doc-hiddenplan::resolve_requesttakes one and returns the request.memchrbecomes a direct dependency ofapache-spark-connect. It was already in its dependency tree, throughregex,arrow-stringandfutures-util, so the build compiles no new crate. The license inventory of the JNI library,crates/spark-connect-server/jvm/native-licenses/dependencies.json, records the hash ofCargo.lock, so it is regenerated (scripts/gen_native_binary_licenses.py); its dependencies are the same.Timings, on a loaded machine, release builds.
DataFrame(plan, session)of a plan built in Python, with the plan built once outside the timing. Each value is the best of 15 alternating rounds of 15 runs. "master without the own fields" is master with #877's own-fields pass turned off, which is the cost to get back:The last row is of 15 rounds of its own, of the best of 15 runs each. The medians of its rounds were 53.3 and 54.6 ms, within the spread of the rounds (49.7 to 65.9 ms).
Those builds are of master 0e91867. On master bcb378f, 9 alternating rounds gave 42.6 ms on master against 39.2 ms here over a placeholder, 42.3 ms against 39.1 ms of no placeholder, and 53.1 ms against 51.5 ms with a placeholder in an expression, of 50,000 expressions.
The core's part alone, the resolution of the relation once decoded (best of 15, release build):
So a plan that holds a placeholder outside its relations pays only the count of the type URLs in its bytes, 0.04 ms for 400 KB. Both columns find the offsets of the URL with
memmem.Why are the changes needed?
To get back #877's cost on the plans that hold no placeholder outside their relations, which is nearly every plan, while keeping everything #877 resolves.
Does this PR introduce any user-facing change?
No. The messages are sent as before. The plans built in Python whose placeholders are all at their relations, or that hold none, are resolved faster.
How was this patch tested?
plan.rs'splaceholder_tests. A test-only record of the search of each resolution shows when the fast path is taken:an_encoding_of_the_type_urls_of_the_placeholders_at_its_relations_alone_is_resolved_at_them: the fast path is taken for no placeholder, an input, the inputs of a join, ML transforms, a fetched argument with a summary's dataset, aWithRelations, and a changed placeholder. Each gives the relation and the levels of the full search.an_encoding_of_a_type_url_elsewhere_is_resolved_everywhere: the full search is kept for a placeholder in the extension of an expression (also with one at an input), in the message of an extension (also 50 levels deep), for a string that is the type URL, and for a placeholder whose bytes hold the type URL. Each is resolved as expected, and as the full search resolves it.the_commands_plans_analyses_and_requests_of_an_encoding_are_resolved_of_its_search: the same for a write and an ML fit (fast path), and for a merge condition, a command's extension and anAnyof the type URL in a request's user context (full search). Plans, requests of session hooks and an analysis of two plans are covered too.a_message_that_does_not_decode_is_its_decoder_s_error.python/tests/test_dropin_offline.py:test_every_message_built_in_python_sends_the_relations_of_its_dataframes_as_pyspark. It runs the five entry points (DataFrame(plan, session),to_table,execute_command,_analyzeand a session hook), each with a placeholder at an input (fast path), in the extension of an expression (full search), and next to a string that is the type URL (full search). The mock server receives the relations, and the string as it is.gates.sh build rust py):cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps, also aslint.ymlruns it, with the macros and build crates and--features apache-spark-connect/wasm-udf;scripts/gen_binary_licenses.py --check: LICENSE-binary and NOTICE-binary are up to date, asmemchrwas in the tree already;scripts/gen_native_binary_licenses.py --checkand the unit tests of both license generators (35 passed), asnative-server.ymlruns them;cargo test -p pyspark-rs --lib: 17 passed.cargo test -p apache-spark-connect --no-fail-fastin a debug build, with its live tests against a Spark 4.3.0 server: 1,431 passed, 0 failed, 15 ignored, of 109 test binaries and doc-test runs (the lib's 848, andtests/raw_relation.rs's 10).test_ruff_excludes.pyandscripts/audit_no_stubs.sh;test_type_stubs_offline.py: 14 passed, 4 skipped;coverage.yml: 6,913 passed, 77 skipped.test_dropin_e2e.py: 352 passed;test_ml_e2e.py: 32 passed.pyspark.ml.tests.connect.test_parity_pipeline,test_parity_featureandtest_parity_tuningthroughrun_official_tests.py --dropin, on master and here: the same on both, with no unexpected failures:test_parity_pipeline6 passed and 4 skipped,test_parity_feature51 passed,test_parity_tuning7 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.