Lib.rs

› Network programming
#web-crawling #data-processing #web-framework #spider #async-io #thread-pool

no-std dyer

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, flexible, comprehensive features without compromising speed

by Bruce Yuan, pc153educon

9 stable releases (3 major)

3.3.2 Jul 27, 2022
3.2.0 May 19, 2022
3.1.0 Mar 14, 2022
3.0.1 Dec 1, 2021
0.3.0 Mar 23, 2021

#7 in #web-crawling


Used in aurion_rs

MIT license

1.5MB
5.5K SLoC

Contains (WOFF font, 99KB) fontawesome-webfont.woff, (WOFF font, 78KB) fontawesome-webfont.woff2, (WOFF font, 45KB) open-sans-v17-all-charsets-300.woff2, (WOFF font, 41KB) open-sans-v17-all-charsets-300italic.woff2, (WOFF font, 45KB) open-sans-v17-all-charsets-600.woff2, (WOFF font, 43KB) open-sans-v17-all-charsets-600italic.woff2 and 7 more.

dyer

API Document crates.io Cookbook

Overview

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, interoperable, comprehensive features without compromising speed.

Features

  • asynchronous, concurrent streaming and I/O, make the best of thread pool, network, and system resource.
  • Event-driven, once you set the initials and recursive generator, dyer will handle the rest of it interoperably.
  • User-friendly and flexible, dyer offers high-level, flexible, easy to use wrappers and APIs what does a lot for you.

Feature Flag

To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features:

  • xpath: Enable parse the html response with xpath
  • compression: Enable HTTP Compression: br, deflate, gzip
  • proxy: Enable use proxies
  • full: Enable all features

Guide

Get started by installing dyer-cli and looking over the examples.

The Cookbook gives a detailed view of dyer.

Dependencies

~5–16MB
~228K SLoC