3 stable releases

Uses new Rust 2021

10.0.0 Jul 15, 2022
9.0.0 Jun 13, 2022
8.0.0 May 16, 2022

#310 in Encoding

Download history 38/week @ 2022-05-13 16/week @ 2022-05-20 28/week @ 2022-05-27 25/week @ 2022-06-03 33/week @ 2022-06-10 29/week @ 2022-06-17 27/week @ 2022-06-24 14/week @ 2022-07-01 17/week @ 2022-07-08 39/week @ 2022-07-15 41/week @ 2022-07-22 34/week @ 2022-07-29 35/week @ 2022-08-05

150 downloads per month
Used in 6 crates (via ballista-core)

Apache-2.0

2.5MB
52K SLoC

DataFusion Proto

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

This crate is a submodule of DataFusion that provides a protocol buffer format for representing query plans and expressions.

Serializing Expressions

Based on examples/expr_serde.rs

use datafusion_common::Result;
use datafusion_expr::{col, lit, Expr};
use datafusion_proto::bytes::Serializeable;

fn main() -> Result<()> {
    // Create a new `Expr` a < 32
    let expr = col("a").lt(lit(5i32));

    // Convert it to an opaque form
    let bytes = expr.to_bytes()?;

    // Decode bytes from somewhere (over network, etc.)
    let decoded_expr = Expr::from_bytes(&bytes)?;
    assert_eq!(expr, decoded_expr);
    Ok(())
}

Serializing Plans

Based on examples/plan_serde.rs

use datafusion::prelude::*;
use datafusion_common::Result;
use datafusion_proto::bytes::{logical_plan_from_bytes, logical_plan_to_bytes};

#[tokio::main]
async fn main() -> Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("t1", "testdata/test.csv", CsvReadOptions::default())
        .await
        ?;
    let plan = ctx.table("t1")?.to_logical_plan()?;
    let bytes = logical_plan_to_bytes(&plan)?;
    let logical_round_trip = logical_plan_from_bytes(&bytes, &ctx)?;
    assert_eq!(format!("{:?}", plan), format!("{:?}", logical_round_trip));
    Ok(())
}

Dependencies

~39MB
~797K SLoC