A fast, native Rust client for Apache Spark Connect - and a drop-in
pyspark replacement with 100% public-API parity with PySpark 4.3.0. It
builds spark.connect protobuf plans, manages the gRPC channel, and decodes
Arrow results in Rust, speaking the same protocol and returning the same results
as the reference client.
Full documentation lives at apache.github.io/spark-connect-rust
- installation, quickstart, the DataFrame / Columns / SQL / Reading & Writing / Streaming / Catalog / Types API, Rust UDFs via WebAssembly, and the architecture.
The Rust API reference (generated from the crate) is on docs.rs/apache-spark-connect.
Python - a faster, drop-in replacement for the
pyspark-client PyPI package
(uninstall any existing pyspark / pyspark-client first):
pip install pyspark-client-rustYour Spark Connect code then runs unchanged; use it exactly like PySpark.
Rust - the native crate:
[dependencies]
apache-spark-connect = "4.3"use spark_connect::{SparkSession, functions as f, lit};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let spark = SparkSession::builder()
.remote("sc://localhost:15002")
.get_or_create()?;
let df = spark
.range(1_000_000)?
.select([(f::col("id") * lit(2)).alias("x")])
.filter((f::col("x") % lit(3)).eq(lit(0)));
println!("count = {}", df.count()?);
df.show(20)?;
Ok(())
}See the documentation for the full API, running a Spark Connect server, and more.
Write UDFs as plain Rust functions and call them directly — #[spark_wasm_udf]
compiles them to WebAssembly and ships them to the executors:
use spark_connect::{SparkSession, col};
use spark_connect_macros::spark_wasm_udf;
#[spark_wasm_udf]
mod udfs {
pub fn add_one(x: i64) -> i64 { x + 1 }
}
#[cfg(not(target_arch = "wasm32"))]
fn main() -> Result<(), Box<dyn std::error::Error>> {
let spark = SparkSession::builder()
.remote("sc://localhost:15002")
.get_or_create()?;
spark.range(5)?.select([udf::add_one(col("id"))?]).show(20)?;
Ok(())
}See the WASM UDF guide for the full setup, SQL registration, supported types, and more.
The unpublished spark-connect-server
library provides a Rust Spark Connect protocol frontend. It manages protocol sessions,
operations and Arrow IPC responses, delegating planning and execution to an
application-provided Engine; its protocol tests use injected engine fixtures.
The optional, unpublished spark-connect-jni crate embeds that frontend inside a
Spark JVM and supplies an adapter for Spark planning and execution. Both crates
are explicit opt-in builds; no standalone server executable is included. See the
server guide for the JNI prototype's
supported scope, build requirements and validation limits.
The frontend and JNI prototype are intended for local, trusted experiments. Read their security model before embedding them.
Issues are tracked in ASF JIRA under SPARK (GitHub Issues are disabled). See the contributing guide.
Apache License 2.0.