18 releases (9 breaking)

0.10.2 Sep 22, 2018
0.10.1 Mar 12, 2018
0.10.0 Dec 2, 2017
0.9.0 Aug 9, 2017
0.1.0 Jun 30, 2015

#52 in Parser tooling

Download history 8/week @ 2018-12-16 137/week @ 2018-12-23 22/week @ 2018-12-30 39/week @ 2019-01-06 9/week @ 2019-01-13 12/week @ 2019-01-27 8/week @ 2019-02-03 6/week @ 2019-02-10 2/week @ 2019-02-17 5/week @ 2019-02-24 51/week @ 2019-03-03 4/week @ 2019-03-10 5/week @ 2019-03-17 197/week @ 2019-03-24

211 downloads per month
Used in 2 crates

MIT license

18KB
372 lines

robotparser-rs

Build Status Build status Coverage Status Crates.io Dependency status

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

[dependencies]
robotparser = "0.10"

Add extern crate robotparser to your crate root and your're good to go!

Examples

extern crate robotparser;

use robotparser::RobotFileParser;

fn main() {
    let parser = RobotFileParser::new("http://www.python.org/robots.txt");
    parser.read();
    assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.

Dependencies

~4MB
~86K SLoC