#chunk #file #process-file #chunking #concurrency

file-chunker

Efficiently process a file in (approximately) equally-sized parts

2 releases

0.1.1 Feb 10, 2022
0.1.0 Feb 9, 2022

#985 in Filesystem

Download history 123/week @ 2024-01-04 141/week @ 2024-01-11 61/week @ 2024-01-18 26/week @ 2024-01-25 112/week @ 2024-02-01 69/week @ 2024-02-08 50/week @ 2024-02-15 104/week @ 2024-02-22 111/week @ 2024-02-29 78/week @ 2024-03-07 96/week @ 2024-03-14 32/week @ 2024-03-21 72/week @ 2024-03-28 45/week @ 2024-04-04 43/week @ 2024-04-11 51/week @ 2024-04-18

215 downloads per month
Used in krapslog

MIT license

17KB
222 lines

file-chunker

This crate provides the FileChunker type, which is useful for efficiently reading a file in (approximately) equally-sized parts.

The original use case was to process a log file in chunks, one thread per chunk, and to guarantee that each chunk ended with a full line of text.

Example

use file_chunker::FileChunker;
let file = std::fs::File::open("/path/to/file").unwrap();
let chunker = FileChunker::new(&file).unwrap();
chunker.chunks(1024, Some('\n'))
    .unwrap()
    .iter()
    .for_each(|chunk| {
        println!("{:?}", chunk);
    });

lib.rs:

This crate provides the FileChunker type, which is useful for efficiently reading a file in (approximately) equally-sized parts.

The original use case was to process a log file in chunks, one thread per chunk, and to guarantee that each chunk ended with a full line of text.

Example

use file_chunker::FileChunker;

let file = std::fs::File::open("/path/to/file").unwrap();
let chunker = FileChunker::new(&file).unwrap();
chunker.chunks(1024, Some('\n'))
    .unwrap()
    .iter()
    .for_each(|chunk| {
        println!("{:?}", chunk);
    });

Dependencies

~290KB