Robin Sparkless

PySpark-style DataFrames in Rust—no JVM. A DataFrame library that mirrors PySpark’s API and semantics while using Polars as the execution engine. The same engine powers the Python package sparkless v4 (pip install ./python) — a drop-in PySpark replacement with no JVM.

Why Robin Sparkless?

• Familiar API — SparkSession, DataFrame, Column, and PySpark-like functions so you can reuse patterns without the JVM.
• Polars under the hood — Fast, native Rust execution with Polars for IO, expressions, and aggregations.
• Persistence options — Global temp views (cross-session in-memory) and disk-backed saveAsTable via spark.sql.warehouse.dir.
• Sparkless backend target — Designed to power Sparkless (the Python PySpark replacement) as a Rust execution engine.

Features

Parity: 200+ fixtures validated against PySpark. Known differences from PySpark are documented in docs/PYSPARK_DIFFERENCES.md. Out-of-scope items (XML, UDTF, streaming, RDD) are in docs/DEFERRED_SCOPE.md. Full parity status: docs/PARITY_STATUS.md.

Installation

Python (sparkless v4)

A Python package sparkless provides a PySpark-like API with no JVM, backed by robin-sparkless.

Install from PyPI:

pip install "sparkless>=4,<5"

Or from this repo (for developing the Rust backend alongside Python):

pip install ./python

See python/README.md for usage and from sparkless.sql import SparkSession.

Rust

Add to your Cargo.toml:

[dependencies]
robin-sparkless = "4"

Optional features:

robin-sparkless = { version = "4", features = ["sql"] }   # spark.sql(), temp views
robin-sparkless = { version = "4", features = ["delta"] }  # Delta Lake read/write

Quick start

Rust

Engine-agnostic (ExprIr) API — recommended for new code and embeddings; uses only EngineError and robin-sparkless types:

use robin_sparkless::{col, lit_i64, gt, SparkSession};

fn main() -> Result<(), robin_sparkless::EngineError> {
    let spark = SparkSession::builder().app_name("demo").get_or_create();
    let df = spark.create_dataframe_engine(
        vec![
            (1, 25, "Alice".to_string()),
            (2, 30, "Bob".to_string()),
            (3, 35, "Charlie".to_string()),
        ],
        vec!["id", "age", "name"],
    )?;
    let adults = df.filter_expr_ir(&gt(col("age"), lit_i64(26)))?;
    adults.show(Some(10)).map_err(robin_sparkless::to_engine_error)?;
    Ok(())
}

Column (Polars) API — full PySpark-like API with Column and Expr:

use robin_sparkless::prelude::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let spark = SparkSession::builder().app_name("demo").get_or_create();
    let df = spark.create_dataframe(
        vec![
            (1, 25, "Alice".to_string()),
            (2, 30, "Bob".to_string()),
            (3, 35, "Charlie".to_string()),
        ],
        vec!["id", "age", "name"],
    )?;
    let adults = df.filter(col("age").gt(lit_i64(26).into_expr()).into_expr())?;
    adults.show(Some(10))?;
    Ok(())
}

Output (from show; run with cargo run --example demo):

shape: (2, 3)
┌─────┬─────┬─────────┐
│ id  ┆ age ┆ name    │
│ --- ┆ --- ┆ ---     │
│ i64 ┆ i64 ┆ str     │
╞═════╪═════╪═════════╡
│ 2   ┆ 30  ┆ Bob     │
│ 3   ┆ 35  ┆ Charlie │
└─────┴─────┴─────────┘

You can also wrap an existing Polars DataFrame with DataFrame::from_polars(polars_df). See docs/QUICKSTART.md for joins, window functions, and more.

Embedding robin-sparkless in your app

Use the engine-agnostic ExprIr API and *_engine() methods so your public surface does not depend on Polars types. Use prelude for the full Column API, or root imports for ExprIr; optional config from environment for session setup. Results can be returned as JSON for bindings or CLI tools.

[dependencies]
robin-sparkless = "4"

use robin_sparkless::{col, lit_i64, gt, SparkSession, SparklessConfig, to_engine_error};

fn main() -> Result<(), robin_sparkless::EngineError> {
    let config = SparklessConfig::from_env();
    let spark = SparkSession::from_config(&config);

    let df = spark.create_dataframe_engine(
        vec![
            (1i64, 10i64, "a".to_string()),
            (2i64, 20i64, "b".to_string()),
            (3i64, 30i64, "c".to_string()),
        ],
        vec!["id", "value", "label"],
    )?;
    let filtered = df.filter_expr_ir(&gt(col("id"), lit_i64(1)))?;
    let json = filtered.to_json_rows()?;
    println!("{}", json);
    Ok(())
}

Example output (from the snippet above or cargo run --example embed_readme; JSON key order may vary):

[{"id":2,"value":20,"label":"b"},{"id":3,"value":30,"label":"c"}]

Run the embed_basic example: cargo run --example embed_basic. For a minimal FFI surface, use robin_sparkless::prelude::embed and the ExprIr API: create_dataframe_engine, filter_expr_ir, select_expr_ir, collect_rows, agg_expr_ir, plus schema helpers (StructType::to_json, schema_from_json). Convert Polars errors with to_engine_error. See docs/EMBEDDING.md.

Development

Prerequisites: Rust (see rust-toolchain.toml).

This repository is a Cargo workspace. The main library is robin-sparkless (the facade); most users depend only on it. The workspace also includes robin-sparkless-core (engine-agnostic types, expression IR, config, error; no Polars) and robin-sparkless-polars (Polars backend: Column, functions, UDFs). These are publishable for advanced or minimal-use cases. make check and CI build the whole workspace.

CI runs format, clippy, audit, deny, Rust tests, and parity tests on push/PR (see .github/workflows/ci.yml).

Documentation

See CHANGELOG.md for version history.

License

MIT