571 stable releases

new 1.95.14 May 25, 2024
1.93.13 Apr 28, 2024
1.89.5 Mar 30, 2024
1.80.37 Dec 31, 2023
1.10.7 Jul 27, 2022

#326 in Web programming

Download history 5/week @ 2024-02-02 1/week @ 2024-02-09 1010/week @ 2024-02-16 1921/week @ 2024-02-23 366/week @ 2024-03-01 477/week @ 2024-03-08 1514/week @ 2024-03-15 2360/week @ 2024-03-22 1249/week @ 2024-03-29 1375/week @ 2024-04-05 737/week @ 2024-04-12 589/week @ 2024-04-19 1197/week @ 2024-04-26 2813/week @ 2024-05-03 713/week @ 2024-05-10 1752/week @ 2024-05-17

6,703 downloads per month

MIT license

11K SLoC

Spider CLI

crate version

A fast command line spider or crawler.


On Linux

  • OpenSSL 1.0.1, 1.0.2, 1.1.0, or 1.1.1


The CLI is a binary so do not add it to your Cargo.toml file.

cargo install spider_cli


The following can also be ran via command line to run the crawler. If you need loging pass in the -v flag.

spider -v --url https://choosealicense.com crawl

Crawl and output all links visited to a file.

spider --url https://choosealicense.com crawl -o > spider_choosealicense.json

Download all html to local destination. Use the option -t to pass in the target destination folder.

spider --url https://choosealicense.com download -t _temp_spider_downloads

Set a crawl budget and only crawl one domain.

spider --url https://choosealicense.com --budget "*,1" crawl -o

Set a crawl budget and only allow 10 pages matching the /blog/ path and limit all pages to 100.

spider --url https://choosealicense.com --budget "*,100,/blog/,10" crawl -o
The fastest web crawler CLI written in Rust.

Usage: spider [OPTIONS] --url <DOMAIN> [COMMAND]

  crawl     Crawl the website extracting links
  scrape    Scrape the website extracting html and links
  download  Download html markup to destination
  help      Print this message or the help of the given subcommand(s)

  -d, --url <DOMAIN>                Domain to crawl
  -r, --respect-robots-txt             Respect robots.txt file
  -s, --subdomains                     Allow sub-domain crawling
  -t, --tld                            Allow all tlds for domain
  -v, --verbose                        Print page visited on standard output
  -D, --delay <DELAY>                  Polite crawling delay in milli seconds
  -b, --blacklist-url <BLACKLIST_URL>  Comma seperated string list of pages to not crawl or regex with feature enabled
  -u, --user-agent <USER_AGENT>        User-Agent
  -B, --budget <BUDGET>                Crawl Budget
  -h, --help                           Print help
  -V, --version                        Print version

All features are available except the Website struct on_link_find_callback configuration option.


~652K SLoC