#ipfs #car #file #chunking #split #multiple #single

carbites

CAR file splitted implementation in Rust

3 releases

0.1.2 May 23, 2023
0.1.1 May 6, 2023
0.1.0 Apr 30, 2023

#11 in #chunking

MIT/Apache

345KB
458 lines

carbites

Chunking for CAR files. Split a single CAR into multiple CARs.

Original implementation in go-carbites.

Usage

Carbites supports treewalk strategy:

Treewalk - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR file has the same root CID but contains a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.

use carbites::{new_splitter, CarSplitter, Strategy};
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let file = std::fs::File::open("examples/test.car")?;
    let mut splitter = new_splitter(Strategy::Treewalk, file, 1024);

    let mut i = 0;
    while let Some(chunk) = splitter.next_chunk()? {
        let mut file = std::fs::File::create(format!("target/chunk-{}.car", i))?;
        file.write(&chunk)?;
        i += 1;
    }

    Ok(())
}

Todo

  • Add support for simple strategy (see below)

Simple - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder "empty" CID. The first CAR output has roots in the header, subsequent CARs have an empty root CID bafkqaaa as recommended.

License

Licensed under either of

at your option.

Dependencies

~3MB
~62K SLoC