#request-response #data-processing #web-framework #spider #web-crawling

no-std dyer

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, flexible, comprehensive features without compromising speed

9 stable releases (3 major)

3.3.2 Jul 27, 2022
3.2.0 May 19, 2022
3.1.0 Mar 14, 2022
3.0.1 Dec 1, 2021
0.3.0 Mar 23, 2021

#896 in Web programming

Download history 12506/week @ 2023-10-29 12417/week @ 2023-11-05 12542/week @ 2023-11-12 12473/week @ 2023-11-19 12473/week @ 2023-11-26 12403/week @ 2023-12-03 12538/week @ 2023-12-10 12531/week @ 2023-12-17 12090/week @ 2023-12-24 12318/week @ 2023-12-31 12308/week @ 2024-01-07 12311/week @ 2024-01-14 12275/week @ 2024-01-21 12231/week @ 2024-01-28 12193/week @ 2024-02-04 11427/week @ 2024-02-11

49,425 downloads per month
Used in aurion_rs

MIT license

1.5MB
5.5K SLoC

dyer

API Document crates.io Cookbook

Overview

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, interoperable, comprehensive features without compromising speed.

Features

  • asynchronous, concurrent streaming and I/O, make the best of thread pool, network, and system resource.
  • Event-driven, once you set the initials and recursive generator, dyer will handle the rest of it interoperably.
  • User-friendly and flexible, dyer offers high-level, flexible, easy to use wrappers and APIs what does a lot for you.

Feature Flag

To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features:

  • xpath: Enable parse the html response with xpath
  • compression: Enable HTTP Compression: br, deflate, gzip
  • proxy: Enable use proxies
  • full: Enable all features

Guide

Get started by installing dyer-cli and looking over the examples.

The Cookbook gives a detailed view of dyer.

Dependencies

~5–18MB
~232K SLoC