1 unstable release

0.1.0 Aug 29, 2021

#375 in #search

46 downloads per month
Used in rosewood

MIT/Apache

11KB
186 lines

Loam

A Rust library for storing tree-like data structures to files. The intended use-case is for querying data which does not fit into RAM. The motivating project, rosewood, uses GeoSpatial data in an RTree.

Loam allows you to store anything which implements Serialize. Data is appended to the end of the file and never modified once written. This enables the use of mmap to read files without the risk of undefined behavior.

Write Example

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut writer = loam::Writer::new("../target/test.loam")?;
    let id = writer.push(&"Don't forget me!")?;
    writer.checkpoint(id)?;
    Ok(())
}

Read Example

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let reader = loam::Reader::new("../target/test.loam")?;
    let id = reader.root()?;
    let msg: String = reader.lookup(id)?;
    dbg!(msg);
    Ok(())
}

File Format

A loam file starts with a Header, followed by a series of Chunks.

Header

The header is fixed-length ASCII text (8 bytes).

Field Value Bytes
Magic loam 4
Major Version digits: 00 2
Minor Version digits: 00 2

Chunks

A chunk consists of these fields, serialized using bincode:

Field Description
Length Number of bytes in Data (variable-size integer)
Data Serialized chunk data
Checksum † CRC-32 of Length + Data (fixed-size integer)

An Id is the file offset of a chunk. It can be used to Deserialize the Data field.

† Checksums are only included if the crc feature is enabled.

Checkpoint

A checkpoint is a special chunk containing a fixed-size u64 of the root Id. A file must always end with a checkpoint, to allow readers to lookup the root without needing to scan the entire file.

Dependencies

~0.8–1.4MB
~32K SLoC