3 releases
0.1.0 | Jul 15, 2024 |
---|---|
0.1.0-rc.2 | Jul 12, 2024 |
0.1.0-rc.1 | Jul 11, 2024 |
0.1.0-alpha2 |
|
#112 in Database implementations
40 downloads per month
Used in hudi
115KB
2.5K
SLoC
A native Rust library for Apache Hudi, with bindings to Python
The hudi-rs
project aims to broaden the use of Apache Hudi for a diverse range of
users and projects.
Source | Installation Command |
---|---|
PyPi | pip install hudi |
Crates.io | cargo add hudi |
Example usage
Python
Read a Hudi table into a PyArrow table.
from hudi import HudiTable
hudi_table = HudiTable("/tmp/trips_table")
records = hudi_table.read_snapshot()
import pyarrow as pa
import pyarrow.compute as pc
arrow_table = pa.Table.from_batches(records)
result = arrow_table.select(
["rider", "ts", "fare"]).filter(
pc.field("fare") > 20.0)
print(result)
Rust
Add crate `hudi` with `datafusion` feature to your application to query a Hudi table.
[dependencies]
hudi = { version = "0" , features = ["datafusion"] }
tokio = "1"
datafusion = "39.0.0"
use std::sync::Arc;
use datafusion::error::Result;
use datafusion::prelude::{DataFrame, SessionContext};
use hudi::HudiDataSource;
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
let hudi = HudiDataSource::new("/tmp/trips_table").await?;
ctx.register_table("trips_table", Arc::new(hudi))?;
let df: DataFrame = ctx.sql("SELECT * from trips_table where fare > 20.0").await?;
df.show().await?;
Ok(())
}
Work with cloud storage
Ensure cloud storage credentials are set properly as environment variables, e.g., AWS_*
, AZURE_*
, or GOOGLE_*
.
Relevant storage environment variables will then be picked up. The target table's base uri with schemes such
as s3://
, az://
, or gs://
will be processed accordingly.
Contributing
Check out the contributing guide for all the details about making contributions to the project.
Dependencies
~69MB
~1.5M SLoC