stork.rs

stork is a simple futures-based library to recursively crawl sources in a search engine-like fashion. stork was designed from the ground to have a simple API that is easy to use and can be reused across multiple protocols, yielding each result giving end users the freedom to do BFS, DFS or any type of search they may so wish.

The API is extremely unstable currently and will likely go through quite a few revisions before we get it stable, I'll keep on top of the changelogs but please keep this in mind when using the library.

i am a heron. i haev a long neck and i pick fish out of the water w/ my beak. if you dont star this repo and 10 other rust repos u enjoy i will fly into your kitchen tonight and make a mess of your pots and pans

View the docs for examples of how to use stork:

• stork
• stork_http

or look in the examples/ directory for some real-world examples!

storkcli

storkcli is built off the back of stork. It can be used to scrape websites for links using various filters, though basic right now stork gives us the ability to make this CLI as sophisticated as we like.

Usage:

Usage: ./storkcli <url> [--max-depth <max-depth>]

Link hunter with a little bit of magic.

Options:
  --max-depth       specifies how deep we should go from the origin, leave this
                    value unspecified to recurse until there's nothing left to
                    follow.
  --help            display usage information

Example:

$ ./storkcli "https://doyle.la/" --max-depth 0
↳ https://instagram.com/doyl_e
↳ https://linkedin.com/in/jordanjdoyle
↳ https://stackoverflow.com/users/2132800/jordan-doyle
↳ https://last.fm/user/doyle-
↳ https://github.com/w4
↳ mailto:jordan@doyle.la
↳ https://keybase.io/jrd

lib.rs:

stork is a simple futures-based library to recursively crawl sources in a search engine-like fashion. stork was designed from the ground to have a simple API that is easy to use and can be reused across multiple protocols, yielding each result giving end users the freedom to do BFS, DFS or any type of search they may so wish.

Your entry point into stork is the Storkable::new function. Have a look through the [Storkable] struct's documentation for your entry into the world of storking.

Note: you're probably not looking for this library on its own but a protocol implementation of it. See below for some first-party implementations:

• stork_http