#cache #batcher #dataloader

dataloader

Rust implementation of Facebook’s DataLoader using async-await

20 releases (13 breaking)

0.14.0 Jan 12, 2021
0.13.2 Dec 10, 2020
0.13.1 Nov 11, 2020
0.12.0 Apr 28, 2020
0.5.0 Feb 3, 2017

#3 in Caching

Download history 335/week @ 2020-11-04 348/week @ 2020-11-11 365/week @ 2020-11-18 380/week @ 2020-11-25 387/week @ 2020-12-02 476/week @ 2020-12-09 511/week @ 2020-12-16 337/week @ 2020-12-23 288/week @ 2020-12-30 516/week @ 2021-01-06 990/week @ 2021-01-13 1320/week @ 2021-01-20 1213/week @ 2021-01-27 710/week @ 2021-02-03 606/week @ 2021-02-10 624/week @ 2021-02-17

1,702 downloads per month

MIT/Apache

22KB
474 lines

Dataloader

Rust Crates.io

Rust implementation of Facebook's DataLoader using async-await.

Documentation

Features

  • Batching load requests with caching
  • Batching load requests without caching

Usage

Switching runtime, by using cargo features

  • runtime-async-std (default), to use the async-std runtime
    • dataloader = "0.14"
  • runtime-tokio to use the Tokio runtime
    • dataloader = { version = "0.14", default-features = false, features = ["runtime-tokio"]}

Add to your Cargo.toml:

[dependencies]
dataloader = "0.14"
futures = "0.3"
async-trait = "0.1"

Example:

use async_trait::async_trait;
use dataloader::cached::Loader;
use dataloader::BatchFn;
use futures::executor::block_on;
use std::collections::HashMap;
use std::thread;

struct MyLoadFn;

#[async_trait]
impl BatchFn<usize, usize> for MyLoadFn {
    async fn load(&mut self, keys: &[usize]) -> HashMap<usize, usize> {
        println!("BatchFn load keys {:?}", keys);
        keys.iter()
            .map(|v| (v.clone(), v.clone()))
            .collect::<HashMap<_, _>>()
    }
}

fn main() {
    let mut i = 0;
    while i < 2 {
        let a = MyLoadFn;
        let loader = Loader::new(a).with_max_batch_size(4);

        let l1 = loader.clone();
        let h1 = thread::spawn(move || {
            let r1 = l1.load(1);
            let r2 = l1.load(2);
            let r3 = l1.load(3);

            let r4 = l1.load_many(vec![2, 3, 4, 5, 6, 7, 8]);
            let f = futures::future::join4(r1, r2, r3, r4);
            println!("{:?}", block_on(f));
        });

        let l2 = loader.clone();
        let h2 = thread::spawn(move || {
            let r1 = l2.load(1);
            let r2 = l2.load(2);
            let r3 = l2.load(3);
            let r4 = l2.load(4);
            let f = futures::future::join4(r1, r2, r3, r4);
            println!("{:?}", block_on(f));
        });

        h1.join().unwrap();
        h2.join().unwrap();
        i += 1;
    }
}

LICENSE

This project is licensed under either of

at your option.

Dependencies

~0.4–2.7MB
~55K SLoC