Lib.rs

› Network programming
#data-processing #web-framework #spider #async-io #thread-pool #web-crawler #web-crawling

no-std dyer

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, flexible, comprehensive features without compromising speed

by Bruce Yuan, pc153educon

9 stable releases (3 major)

3.3.2 Jul 27, 2022
3.2.0 May 19, 2022
3.1.0 Mar 14, 2022
3.0.1 Dec 1, 2021
0.3.0 Mar 23, 2021

#7 in #spider

Download history 11621/week @ 2024-03-08 11575/week @ 2024-03-15 11319/week @ 2024-03-22 11640/week @ 2024-03-29 11488/week @ 2024-04-05 11541/week @ 2024-04-12 11591/week @ 2024-04-19 11786/week @ 2024-04-26 11778/week @ 2024-05-03 11665/week @ 2024-05-10 11648/week @ 2024-05-17 11717/week @ 2024-05-24 11758/week @ 2024-05-31 11604/week @ 2024-06-07 2786/week @ 2024-06-14

27,977 downloads per month
Used in aurion_rs

MIT license

1.5MB
5.5K SLoC

Contains (WOFF font, 99KB) fontawesome-webfont.woff, (WOFF font, 78KB) fontawesome-webfont.woff2, (WOFF font, 45KB) open-sans-v17-all-charsets-300.woff2, (WOFF font, 41KB) open-sans-v17-all-charsets-300italic.woff2, (WOFF font, 45KB) open-sans-v17-all-charsets-600.woff2, (WOFF font, 43KB) open-sans-v17-all-charsets-600italic.woff2 and 7 more.

dyer

API Document crates.io Cookbook

Overview

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, interoperable, comprehensive features without compromising speed.

Features

  • asynchronous, concurrent streaming and I/O, make the best of thread pool, network, and system resource.
  • Event-driven, once you set the initials and recursive generator, dyer will handle the rest of it interoperably.
  • User-friendly and flexible, dyer offers high-level, flexible, easy to use wrappers and APIs what does a lot for you.

Feature Flag

To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features:

  • xpath: Enable parse the html response with xpath
  • compression: Enable HTTP Compression: br, deflate, gzip
  • proxy: Enable use proxies
  • full: Enable all features

Guide

Get started by installing dyer-cli and looking over the examples.

The Cookbook gives a detailed view of dyer.

Dependencies

~5–17MB
~227K SLoC