16 releases (breaking)

Uses old Rust 2015

0.13.1 Dec 1, 2018
0.13.0 Oct 15, 2018
0.12.1 Jun 18, 2017
0.11.0 Mar 12, 2017
0.4.0 Jun 7, 2016

#1574 in Web programming

Download history 17/week @ 2023-05-26 2/week @ 2023-06-02 19/week @ 2023-06-16 16/week @ 2023-06-23 19/week @ 2023-06-30 3/week @ 2023-07-14 3/week @ 2023-07-21 2/week @ 2023-07-28 1/week @ 2023-08-04 17/week @ 2023-08-18 17/week @ 2023-08-25 3/week @ 2023-09-01 37/week @ 2023-09-08

74 downloads per month

MIT license

18KB
424 lines

Maman

Maman is a Rust Web Crawler saving pages on Redis.

Pages are send to list <MAMAN_ENV>:queue:maman using Sidekiq job format

{
"class": "Maman",
"jid": "b4a577edbccf1d805744efa9",
"retry": true,
"created_at": 1461789979, "enqueued_at": 1461789979,
"args": {
    "document":"<html><body><a href='#' /><a href='/new' /></html>",
    "urls": ["https://example.net/new"],
    "headers": {"content-type": "text/html"},
    "url": "https://example.net/"
    }
}

Dependencies

Installation

With cargo

cargo install maman

With make

PREFIX=~/.local make install

Usage

maman URL [LIMIT] [MIME_TYPES]

LIMIT must be an integer or 0 is the default, meaning no limit.

Environment variables

Defaults

  • MAMAN_ENV=development
  • REDIS_URL="redis://127.0.0.1/"

Others

  • RUST_LOG=maman=info

LICENSE

The MIT License

Copyright (c) 2016-2018 Laurent Arnoud laurent@spkdev.net


Build Version Documentation License Project status Dependency status

Dependencies

~21–29MB
~535K SLoC