#lib #download #pipe #downloader #performance


Core library - fast downloader + extractor library for *.tar.gz, *tar.lz4 and *.tar.bz2 files

14 releases (4 breaking)

0.8.0 May 8, 2023
0.7.0 May 8, 2023
0.6.5 May 8, 2023
0.6.4 Feb 25, 2023
0.4.2 Nov 29, 2022
Download history 10/week @ 2023-08-06 21/week @ 2023-08-13 3/week @ 2023-08-20 16/week @ 2023-08-27 22/week @ 2023-09-03 19/week @ 2023-09-10 4/week @ 2023-09-17 5/week @ 2023-09-24 8/week @ 2023-10-01 4/week @ 2023-10-08 5/week @ 2023-10-15 7/week @ 2023-10-22 19/week @ 2023-10-29 6/week @ 2023-11-05 20/week @ 2023-11-12 6/week @ 2023-11-19

52 downloads per month
Used in pipe_downloader

MIT license

1.5K SLoC


This program is used to download, uncompress and untar files at once. So basically it does the same as command:

curl http://arvhive.tar.lz4 | lz4 -dc | tar -xv -C output_dir

result is the same as using (pipe_downloader will create directory if not empty):

pipe_downloader --url http://arvhive.tar.lz4 --output-dir output_dir

but it uses multiple cores better than piped commands like above. Additionally, it provides progress, that can be used by server to provide API. Implementation is nonblocking, based on threads (no async IO used)

It spawns 3 or more threads per download, one or more for downloading, one for decompressing and one for untaring (writing files also). It is using chunks, when chunks are filled with data they are gathered and when next chunk is full they are processed by unpacker.

No unsafe code used.

Currently, it supports:

  • archive.tar.lz4
  • archive.tar.gz
  • archive.tar.bz2
  • single_file.lz4
  • single_file.gz
  • single_file.bz2

As base mode of operation it used PARTIAL_CONTENT (206) http status code, which allows to continue operation when connection is lost (downloader waits patiently until chunk is available again). Downloader also supports normal GET (200) download when server does not support PARTIAL_CONTENT, but any network timeout/disconnection will lead to unrecoverable error then.

There is no option of restarting download after unrecoverable error or killed process.

  1. Cross compilation

cross build --release --target aarch64-unknown-linux-musl

  1. Test runs

cargo run --release --example pipe_downloader -- --output-dir linux-6.0.4 --url=https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.0.4.tar.gz


~390K SLoC