1 unstable release

0.2.4 Dec 22, 2023

#16 in #sitemap


Used in sws-lua

MIT/Apache

75KB
2K SLoC

Sitemap Web Scraper

Sitemap Web Scraper (sws) is a tool for simple, flexible, and yet performant web pages scraping.

It consists of a CLI written in Rust that crawls web pages and executes a Lua JIT script to scrap them, outputting results to a CSV file.

sws crawl --script examples/fandom_mmh7.lua -o result.csv

Check out the doc for more details.


lib.rs:

HTML parsing and querying with CSS selectors.

Partial port of scraper using sws-tree which uses Rc instead of references with lifetimes.

Dependencies

~2.7–8MB
~72K SLoC