3 releases (breaking)

0.3.0 Oct 25, 2020
0.2.0 Oct 25, 2020
0.1.0 Oct 25, 2020

#269 in Database implementations

Download history 54/week @ 2024-07-27 39/week @ 2024-08-03 48/week @ 2024-08-10 37/week @ 2024-08-17 49/week @ 2024-08-24 86/week @ 2024-08-31 116/week @ 2024-09-07 90/week @ 2024-09-14 73/week @ 2024-09-21 129/week @ 2024-09-28 86/week @ 2024-10-05 107/week @ 2024-10-12 89/week @ 2024-10-19 36/week @ 2024-10-26 35/week @ 2024-11-02 66/week @ 2024-11-09

235 downloads per month
Used in 5 crates (4 directly)

MIT license

20KB
382 lines

simple_wal

A simple Rust write-ahead-logging crate.

Features

  • Optimized for sequential reads & writes
  • Easy atomic log compaction
  • Advisory locking
  • CRC32 checksums
  • Range scans
  • Persistent log entry index

lib.rs:

A simple write-ahead-logging crate.

Features

  • Optimized for sequential reads & writes
  • Easy atomic log compaction
  • Advisory locking
  • CRC32 checksums
  • Range scans
  • Persistent log entry index

The entire log is scanned through on startup in order to detect & clean interrupted writes and determine the length of the log. It's recommended to compact the log when old entries are no longer likely to be used.

Usage:

use simple_wal::LogFile;

let path = std::path::Path::new("./wal-log");

{
    let mut log = LogFile::open(path).unwrap();

    // write to log
    log.write(&mut b"log entry".to_vec()).unwrap();
    log.write(&mut b"foobar".to_vec()).unwrap();
    log.write(&mut b"123".to_vec()).unwrap();
   
    // flush to disk
    log.flush().unwrap();
}

{
    let mut log = LogFile::open(path).unwrap();

    // Iterate through the log
    let mut iter = log.iter(..).unwrap();
    assert_eq!(iter.next().unwrap().unwrap(), b"log entry".to_vec());
    assert_eq!(iter.next().unwrap().unwrap(), b"foobar".to_vec());
    assert_eq!(iter.next().unwrap().unwrap(), b"123".to_vec());
    assert!(iter.next().is_none());
}

{
    let mut log = LogFile::open(path).unwrap();

    // Compact the log
    log.compact(1).unwrap();

    // Iterate through the log
    let mut iter = log.iter(..).unwrap();
    assert_eq!(iter.next().unwrap().unwrap(), b"foobar".to_vec());
    assert_eq!(iter.next().unwrap().unwrap(), b"123".to_vec());
    assert!(iter.next().is_none());
}

Log Format:

00 01 02 03 04 05 06 07|08 09 10 11 12 13 14 15|.......|-4 -3 -2 -1|
-----------------------|-----------------------|-------|-----------|
starting index         |entry length           | entry | crc32     |
unsigned 64 bit int le |unsigned 64 bit int le | data  | 32bit, le |

Numbers are stored in little-endian format.

The first 8 bytes in the WAL is the starting index.

Each entry follows the following format:

  1. A 64 bit unsigned int for the entry size.
  2. The entry data
  3. A 32 bit crc32 checksum.

Dependencies

~1.5–2.4MB
~40K SLoC