12 unstable releases (4 breaking)

0.5.1 Mar 19, 2025
0.4.1 Dec 24, 2024
0.4.0 Oct 10, 2024
0.2.2 Apr 25, 2024
0.2.0 Feb 29, 2024

#358 in WebAssembly

Download history 406/week @ 2024-12-08 444/week @ 2024-12-15 512/week @ 2024-12-22 431/week @ 2024-12-29 481/week @ 2025-01-05 417/week @ 2025-01-12 503/week @ 2025-01-19 302/week @ 2025-01-26 471/week @ 2025-02-02 536/week @ 2025-02-09 492/week @ 2025-02-16 745/week @ 2025-02-23 765/week @ 2025-03-02 743/week @ 2025-03-09 903/week @ 2025-03-16 591/week @ 2025-03-23

3,037 downloads per month

Apache-2.0

40KB
717 lines

WebAssembly UDF for Apache Arrow

Crate Docs

For untrusted user-defined functions, you can compile them into WebAssembly and run them in a sandboxed environment.

Build UDF in WebAssembly

Create a project and add the following lines to your Cargo.toml:

[dependencies]
arrow-udf = "0.6"

Define your functions with the #[function] macro:

use arrow_udf::function;

#[function("gcd(int, int) -> int")]
fn gcd(mut a: i32, mut b: i32) -> i32 {
    while b != 0 {
        (a, b) = (b, a % b);
    }
    a
}

Then compile the project into WebAssembly:

cargo build --release --target wasm32-wasip1

You can find the generated WebAssembly module in target/wasm32-wasip1/release/*.wasm.

Run UDF in WebAssembly

Add the following lines to your Cargo.toml:

[dependencies]
arrow-udf-wasm = "0.5"

You can then load the WebAssembly module and call the functions:

use arrow_udf_wasm::Runtime;
use arrow_schema::DataType;
use arrow_array::RecordBatch;

// load the WebAssembly module
let binary = std::fs::read("udf.wasm").unwrap();
// create a runtime from the module
let runtime = Runtime::new(&binary).unwrap();
// list available functions in the module:
for name in runtime.functions() {
    println!("{}", name);
}
// call the function with a RecordBatch
let func = runtime.find_function("gcd", vec![DataType::Int32, DataType::Int32], DataType::Int32).unwrap();
let input: RecordBatch = ...;
let output = runtime.call(func, &input).unwrap();

The WebAssembly runtime is powered by Wasmtime. Notice that each WebAssembly instance can only run single-threaded, we maintain an instance pool internally to support parallel calls from multiple threads.

See the example for more details. To run the example:

cargo build --release -p arrow-udf-example --target wasm32-wasip1
cargo run --example wasm -- target/wasm32-wasip1/release/arrow_udf_example.wasm

Build WASM UDF at Runtime

Enable the build feature to build the wasm binary from source:

[dependencies]
arrow-udf-wasm = { version = "0.5", features = ["build"] }

You can then build the WebAssembly module at runtime:

let manifest = r#"
[dependencies]
chrono = "0.4"
"#;

let script = r#"
use arrow_udf::function;

#[function("gcd(int, int) -> int")]
fn gcd(mut a: i32, mut b: i32) -> i32 {
    while b != 0 {
        (a, b) = (b, a % b);
    }
    a
}
"#;
let binary = arrow_udf_wasm::build::build(manifest, script).unwrap();

See the build module for more details.

Dependencies

~24–35MB
~589K SLoC