3 releases
0.1.2 | May 23, 2023 |
---|---|
0.1.1 | May 6, 2023 |
0.1.0 | Apr 30, 2023 |
#11 in #chunking
41 downloads per month
345KB
458 lines
carbites
Chunking for CAR files. Split a single CAR into multiple CARs.
Original implementation in go-carbites.
Usage
Carbites supports treewalk strategy:
Treewalk - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR file has the same root CID but contains a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.
use carbites::{new_splitter, CarSplitter, Strategy};
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let file = std::fs::File::open("examples/test.car")?;
let mut splitter = new_splitter(Strategy::Treewalk, file, 1024);
let mut i = 0;
while let Some(chunk) = splitter.next_chunk()? {
let mut file = std::fs::File::create(format!("target/chunk-{}.car", i))?;
file.write(&chunk)?;
i += 1;
}
Ok(())
}
Todo
- Add support for simple strategy (see below)
Simple - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder "empty" CID. The first CAR output has roots in the header, subsequent CARs have an empty root CID bafkqaaa
as recommended.
License
Licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)
at your option.
Dependencies
~1.3–2.2MB
~42K SLoC