#chunk #chunking #concurrency #files

file-chunker

Efficiently process a file in (approximately) equally-sized parts

2 releases

0.1.1 Feb 10, 2022
0.1.0 Feb 9, 2022

#937 in Text processing

Download history 166/week @ 2023-02-11 158/week @ 2023-02-18 156/week @ 2023-02-25 68/week @ 2023-03-04 155/week @ 2023-03-11 162/week @ 2023-03-18 135/week @ 2023-03-25 112/week @ 2023-04-01 217/week @ 2023-04-08 227/week @ 2023-04-15 124/week @ 2023-04-22 199/week @ 2023-04-29 134/week @ 2023-05-06 137/week @ 2023-05-13 45/week @ 2023-05-20 173/week @ 2023-05-27

513 downloads per month
Used in krapslog

MIT license

17KB
222 lines

file-chunker

This crate provides the FileChunker type, which is useful for efficiently reading a file in (approximately) equally-sized parts.

The original use case was to process a log file in chunks, one thread per chunk, and to guarantee that each chunk ended with a full line of text.

Example

use file_chunker::FileChunker;
let file = std::fs::File::open("/path/to/file").unwrap();
let chunker = FileChunker::new(&file).unwrap();
chunker.chunks(1024, Some('\n'))
    .unwrap()
    .iter()
    .for_each(|chunk| {
        println!("{:?}", chunk);
    });

lib.rs:

This crate provides the FileChunker type, which is useful for efficiently reading a file in (approximately) equally-sized parts.

The original use case was to process a log file in chunks, one thread per chunk, and to guarantee that each chunk ended with a full line of text.

Example

use file_chunker::FileChunker;

let file = std::fs::File::open("/path/to/file").unwrap();
let chunker = FileChunker::new(&file).unwrap();
chunker.chunks(1024, Some('\n'))
    .unwrap()
    .iter()
    .for_each(|chunk| {
        println!("{:?}", chunk);
    });

Dependencies

~225KB