#http #request #client #retry

reqwest_resume

Wrapper that uses the Range HTTP header to resume get requests. It’s a thin wrapper around reqwest. It’s a work in progress – wrapping functionality is copied across on an as-needed basis. Feel free to open a PR/issue if you need something.

6 releases

0.3.2 Jul 8, 2020
0.3.1 May 16, 2020
0.3.0 Apr 30, 2020
0.2.1 Sep 7, 2019
0.1.0 Aug 15, 2018

#418 in Encoding

Download history 19/week @ 2021-02-20 13/week @ 2021-02-27 48/week @ 2021-03-06 35/week @ 2021-03-13 45/week @ 2021-03-20 58/week @ 2021-03-27 85/week @ 2021-04-03 18/week @ 2021-04-10 54/week @ 2021-04-17 20/week @ 2021-04-24 95/week @ 2021-05-01 22/week @ 2021-05-08 108/week @ 2021-05-15 12/week @ 2021-05-22 11/week @ 2021-05-29 17/week @ 2021-06-05

182 downloads per month
Used in 2 crates (via amadeus-commoncrawl)

MIT/Apache

13KB
186 lines

reqwest_resume

Crates.io MIT / Apache 2.0 licensed Build Status

📖 Docs | 💬 Chat

Wrapper that uses the Range HTTP header to resume get requests.

It's a thin wrapper around reqwest. It's a work in progress – wrapping functionality is copied across on an as-needed basis. Feel free to open a PR/issue if you need something.

Example

use async_compression::futures::bufread::GzipDecoder;
use futures::{io::BufReader, AsyncBufReadExt, StreamExt, TryStreamExt};
use std::io;

#[tokio::main]
async fn main() {
    let url = "http://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2018-30/warc.paths.gz";
    let body = reqwest_resume::get(url.parse().unwrap()).await.unwrap();
    // Content-Encoding isn't set, so decode manually
    let body = body
        .bytes_stream()
        .map_err(|e| io::Error::new(io::ErrorKind::Other, e));
    let body = BufReader::new(body.into_async_read());
    let mut body = GzipDecoder::new(body); // Content-Encoding isn't set, so decode manually
    body.multiple_members(true);

    let mut lines = BufReader::new(body).lines();
    while let Some(line) = lines.next().await {
        println!("{}", line.unwrap());
    }
}

License

Licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Dependencies

~6–9MB
~203K SLoC