#crawler #spider #spider-cli

archived app spider_cli

The fastest web crawler CLI written in Rust

202 stable releases

new 1.31.7 Jun 4, 2023
1.27.2 Apr 23, 2023
1.26.7 Mar 22, 2023
1.17.2 Nov 21, 2022
1.10.7 Jul 27, 2022

#328 in Web programming

Download history 280/week @ 2023-02-11 665/week @ 2023-02-18 1034/week @ 2023-02-25 173/week @ 2023-03-04 250/week @ 2023-03-11 99/week @ 2023-03-18 141/week @ 2023-03-25 43/week @ 2023-04-01 27/week @ 2023-04-08 386/week @ 2023-04-15 84/week @ 2023-04-22 181/week @ 2023-04-29 580/week @ 2023-05-06 345/week @ 2023-05-13 114/week @ 2023-05-20 246/week @ 2023-05-27

1,316 downloads per month

MIT license

3.5K SLoC

Spider CLI

crate version

A fast command line spider or crawler.


On Linux

  • OpenSSL 1.0.1, 1.0.2, 1.1.0, or 1.1.1


The CLI is a binary so do not add it to your Cargo.toml file.

cargo install spider_cli


The following can also be ran via command line to run the crawler. If you need loging pass in the -v flag.

spider -v --domain https://choosealicense.com crawl

Crawl and output all links visited to a file.

spider --domain https://choosealicense.com crawl -o > spider_choosealicense.json
spider_cli 1.31.7
madeindjs <contact@rousseau-alexandre.fr>, j-mendez <jeff@a11ywatch.com>
Multithreaded web crawler written in Rust.

    spider [OPTIONS] --domain <DOMAIN> [SUBCOMMAND]

    -b, --blacklist-url <BLACKLIST_URL>
            Comma seperated string list of pages to not crawl or regex with feature enabled

    -c, --concurrency <CONCURRENCY>
            How many request can be run simultaneously

    -d, --domain <DOMAIN>
            Domain to crawl

    -D, --delay <DELAY>
            Polite crawling delay in milli seconds

    -h, --help
            Print help information

    -r, --respect-robots-txt
            Respect robots.txt file

    -u, --user-agent <USER_AGENT>

    -v, --verbose
            Print page visited on standard output

    -V, --version
            Print version information

    crawl     crawl the website extracting links
    help      Print this message or the help of the given subcommand(s)
    scrape    scrape the website extracting html and links

All features are available except the Website struct on_link_find_callback configuration option.


~588K SLoC