Diesel Streamer

A tiny diesel add-on for streaming large tables. It currently allows streaming large serial tables using a cursor-based streaming strategy.

Installation

For a regular synchronous runtime:

[dependencies]
diesel-streamer = { version = "0.1.12", features = ["sync"]}

For tokio async runtime:

[dependencies]
diesel-streamer = { version = "0.1.12", features = ["async"]}

Example Usage

Stream SomeTable that has a serial_field:

use diesel_streamer::stream_serial_table;

fn main() {
  use crate::schema::some_table::dsl::{some_table, serial_field};

  let mut conn = pool.get().await.unwrap();

  // with default chunk size of 500
  stream_serial_table!(some_table, serial_field, conn, |streamed_table_data: Vec<SomeTable>| {
    // do work here
    dbg!(streamed_table_data);
  });

  // specify chunk size, 130
  stream_serial_table!(some_table, serial_field, conn, 130, |streamed_table_data: Vec<SomeTable>| {
    // do work here
    dbg!(streamed_table_data);
  });

  // with cursor's beginning, 5.
  stream_serial_table!(some_table, serial_field, conn, 130, 5, |streamed_table_data: Vec<SomeTable>| {
    // do work here
    dbg!(streamed_table_data);
  });

  // with cursor's end, 50,
  stream_serial_table!(some_table, serial_field, conn, 130, 5, 50, |streamed_table_data: Vec<SomeTable>| {
    // do work here
    dbg!(streamed_table_data);
  });
}

Defaults:

• Chunk size: 500
• Cursor's beginning: lowest value of serial_field in the table
• Cursor's end: highest value of serial_field in the table

N/B: Generally, streaming should only be considered when there is a possibility of hitting OOM error when processing the table in question.

Contributing

Spin up a test db using docker-compose up or simply specify a DB url in .env as shown in .env.sample.

Run cargo test for tests.

License