Rust-parallelfastx

A truly parallel parser for FASTA/FASTQ files.

Principle

The input file is memory-mapped then virtually split into N chunks. Each chunk is fed to a regular FASTA/FASTQ parser (here, the excellent https://github.com/markschl/seq_io library).

Rationale

Virtually all other "multithreaded" FASTA/FASTQ parsers typically use only one thread to parse the file, then they feed the parsed sequences to threads. If your disk is fast enough (> 2 GB/s) that parsing the file becomes a CPU bottleneck, then you might benefit from this library as the parsing is truly multithreaded.

How to use

see src/main.rs, should be self explanatory.

Inspiration

Inspiration for this repository is the amazing fastlwc-mt tool from https://github.com/expr-fi/fastlwc which does multi-threaded line counting.

Caveat

Input file needs to be seekable, which rules out all compression methods except blocked ones, which currently aren't supported by this library, but could be in principle.

Author

Rayan Chikhi, 2022