#firewall #crawler #bad #website #url #build

spider_firewall

Firewall to use for Spider Web Crawler

7 stable releases

new 2.30.2 Feb 19, 2025
2.28.6 Feb 18, 2025
2.28.3 Feb 17, 2025
2.28.2 Feb 16, 2025
2.28.0 Feb 15, 2025

#1731 in Web programming

Download history 348/week @ 2025-02-12

348 downloads per month
Used in spider

MIT license

15KB
181 lines

spider_firewall

A shield to prevent bad websites from messing up your system.

cargo add spider_firewall

use spider_firewall::is_bad_website_url;

fn main() {
    let domain = url::Url::parse("https://badwebsite.com").expect("parse");
    let blocked = is_bad_website_url(&domain);
}

The build can take up to 5-10 minutes to build locally.

Dependencies

~0.1–1.8MB
~24K SLoC