4 releases

0.2.0-rc.1 Oct 15, 2024
0.1.0 Jul 15, 2024
0.1.0-rc.2 Jul 12, 2024
0.1.0-rc.1 Jul 11, 2024
0.1.0-alpha2 Jul 10, 2024

#8 in #datalake

Download history 331/week @ 2024-07-08 135/week @ 2024-07-15 42/week @ 2024-07-29 20/week @ 2024-09-16 22/week @ 2024-09-23 11/week @ 2024-09-30 2/week @ 2024-10-07 140/week @ 2024-10-14

176 downloads per month
Used in 2 crates

Apache-2.0

165KB
3.5K SLoC

Hudi logo

A native Rust library for Apache Hudi, with bindings to Python

hudi-rs ci hudi-rs codecov join hudi slack follow hudi x/twitter follow hudi linkedin

The hudi-rs project aims to broaden the use of Apache Hudi for a diverse range of users and projects.

Source Installation Command
PyPi pip install hudi
Crates.io cargo add hudi

Example usage

[!NOTE] These examples expect a Hudi table exists at /tmp/trips_table, created using the quick start guide.

Python

Read a Hudi table into a PyArrow table.

from hudi import HudiTable

hudi_table = HudiTable("/tmp/trips_table")
records = hudi_table.read_snapshot()

import pyarrow as pa
import pyarrow.compute as pc

arrow_table = pa.Table.from_batches(records)
result = arrow_table.select(
    ["rider", "ts", "fare"]).filter(
    pc.field("fare") > 20.0)
print(result)

Rust

Add crate hudi with datafusion feature to your application to query a Hudi table.
cargo new my_project --bin && cd my_project
cargo add tokio@1 datafusion@39
cargo add hudi --features datafusion

Update src/main.rs with the code snippet below then cargo run.

use std::sync::Arc;

use datafusion::error::Result;
use datafusion::prelude::{DataFrame, SessionContext};
use hudi::HudiDataSource;

#[tokio::main]
async fn main() -> Result<()> {
    let ctx = SessionContext::new();
    let hudi = HudiDataSource::new("/tmp/trips_table").await?;
    ctx.register_table("trips_table", Arc::new(hudi))?;
    let df: DataFrame = ctx.sql("SELECT * from trips_table where fare > 20.0").await?;
    df.show().await?;
    Ok(())
}

Work with cloud storage

Ensure cloud storage credentials are set properly as environment variables, e.g., AWS_*, AZURE_*, or GOOGLE_*. Relevant storage environment variables will then be picked up. The target table's base uri with schemes such as s3://, az://, or gs:// will be processed accordingly.

Contributing

Check out the contributing guide for all the details about making contributions to the project.

Dependencies

~36–54MB
~1M SLoC