#vec-deque #file-backed #mmap #memory-mapped #persistence

mmap-vecdeque

A file-backed memory-mapped VecDeque with deferred commits, atomic and durable persistence

3 unstable releases

0.2.1 Dec 13, 2024
0.2.0 Dec 13, 2024
0.1.0 Dec 13, 2024

#206 in Memory management

Download history 323/week @ 2024-12-11 16/week @ 2024-12-18

339 downloads per month

MIT license

22KB
497 lines

mmap-vecdeque

mmap-vecdeque is a file-backed, memory-mapped, durable, and thread-safe double-ended queue (deque) for Rust. Unlike a normal VecDeque, operations aren't immediately durable. Instead, you can batch multiple insertions, deletions, etc., and then call commit() to atomically persist all changes to disk.

Key Features

  • Deferred commits: Changes are kept in memory until commit() is called.
  • Atomic and durable commits: Once commit() returns, all changes are atomically and durably persisted to disk.
  • Configurable chunk size: Items are stored in fixed-size chunks of elements. By default, 10,000 elements per chunk.
  • Mmap-backed: Data is accessed via memory mapping for potentially high performance.
  • Iterators: iter() and iter_mut() to traverse elements.

Usage

use mmap_vecdeque::{MmapVecDeque, MmapVecDequeError};
use std::path::Path;

fn main() -> Result<(), MmapVecDequeError> {
    let dir = Path::new("my_deque_storage");
    let mut deque = MmapVecDeque::<u64>::open_or_create(dir, None)?;

    deque.push_back(42)?;
    deque.push_front(1)?;

    // Changes are not yet durable on disk
    deque.commit()?; // Now 1 and 42 are atomically committed.

    println!("Front: {:?}", deque.front()); // Some(1)
    println!("Back: {:?}", deque.back());   // Some(42)

    deque.pop_front()?; 
    // Not durable yet.
    deque.commit()?; // commit again

    println!("Front after pop: {:?}", deque.front()); // Some(42)

    // Iteration
    for val in deque.iter() {
        println!("{}", val);
    }

    Ok(())
}

Dependencies

~3–12MB
~157K SLoC