2 unstable releases
0.2.0 | Jan 21, 2024 |
---|---|
0.1.0 | Jul 30, 2023 |
#647 in Filesystem
51KB
958 lines
read_chunk_iter
Iterator adapters over a reader that yields fixed-size chunks at a time.
Why not use generic iterator composition over Read objects?
A simple solution would be to use iterator adapters over the bytes of a file, e.g. &BufReader::new(file_path).bytes().chunks(CHUNK_SIZE)
, using the itertools
crate to provide the chunks
adapter. However, using generic iterator adaptors is significantly slower than a dedicated iterator, with the timing comparison in the examples
folder demonstrating a slowdown by a factor of 2.5-40.5. (This is including the use of BufReader
to reduce the number of underlying read
calls. Without such buffering, bytes()
would call read
once for each byte, resulting in a much larger slowdown.)
This crate offers two alternatives:
ChunkedReaderIter
, which synchronously reads from the underlyingRead
object and yields chunks of data when requested.ThreadedChunkedReaderIter
, which performs the reads in a separate thread and transmits chunks of data to the originating thread.
Whether to use ChunkedReaderIter
or ThreadedChunkedReaderIter
depends on whether the saved time of asynchronous reads while doing other computations outweighs the overhead of threading. Benchmark your particular use case before assuming that one is necessarily better than the other.
Features
- autodetect_vectored: Enable automatic detection of whether vectored reads offer speedups, and take advantage of them when they offer speedups. This feature requires nightly, but manual selection of vectored reads is still possible without it.
Planned features
- fadvise: Use
posix_fadvise
to signalPOSIX_FADV_SEQUENTIAL
for the whole file and to provide the option to free filesystem cache withPOSIX_FADV_DONTNEED
on yielded data. This feature will be enabled by default but will be a no-op on non-Unix systems.
Dependencies
~0–10MB
~43K SLoC