#web-crawler #web

app wrake

Collect links from the given URL

4 releases (2 breaking)

0.3.0 Nov 27, 2021
0.2.0 Nov 25, 2021
0.1.1 Nov 24, 2021
0.1.0 Nov 23, 2021

#39 in #web-crawler

MIT/Apache

13KB
214 lines

wrake 0.3.0

Collect links from the given URL

USAGE:
    wrake [OPTIONS] --url <string>

OPTIONS:
    -a, --user-agent <string>    User agent header [default: wrake]
    -d, --depth <integer>        Recursion depth [default: 2]
    -h, --help                   Print help information
    -i, --insecure-proxy         Accept invalid certificates for proxy
    -n, --no-domain-filter       Do not restrict recursion to original domain
    -p, --proxy <string>         Proxy through which to send requests
    -u, --url <string>           Target URL
    -V, --version                Print version information

Dependencies

~10–24MB
~377K SLoC