Cargo Features
[dependencies]
spider = { version = "2.13.97", default-features = false, features = ["regex", "glob", "ua_generator", "jemalloc", "decentralized", "control", "time", "adblock", "sync", "flexbuffers", "serde", "fs", "full_resources", "socks", "reqwest_json", "sitemap", "cache_request", "cache", "cache_mem", "cache_openai", "cache_chrome_hybrid", "chrome", "chrome_headed", "chrome_cpu", "chrome_screenshot", "chrome_store_page", "chrome_intercept", "chrome_headless_new", "cookies", "cron", "smart", "encoding", "headers", "real_browser", "openai", "openai_slim_fit", "decentralized_headers", "spoof", "reqwest_rustls_tls", "reqwest_native_tls", "reqwest_native_tls_alpn", "reqwest_native_tls_vendored", "reqwest_native_tls_manual_roots", "reqwest_native_tls_webpki_roots", "reqwest_native_tls_native_roots", "reqwest_hickory_dns", "reqwest_multipart", "tokio_io_std", "tracing", "string_interner_buffer_backend", "string_interner_bucket_backend", "cowboy", "inline-more"] }- default = cookies, encoding, reqwest_native_tls_native_roots, string_interner_buffer_backend, sync, ua_generator
-
These default features are set whenever
spideris added withoutdefault-features = false - regex
-
Affects
configuration::AllowList,spider::black_list… - glob
-
Enables itertools ^0.12
Affects
features::glob… - ua_generator default
-
Enables ua_generator
Affects
configuration::get_ua… - jemalloc = tikv-jemallocator
- decentralized = flexbuffers, serde
-
Affects
page::Page,page::build,utils::fetch_page,utils::FetchPageResult,utils::fetch_page_and_headers… - control
-
Affects
utils::Handler,utils::pause,utils::resume,utils::shutdown,utils::reset,website::Website.crawl_id… - time
- adblock
-
Enables adblock of spider_chrome
- sync default
- flexbuffers decentralized?
-
Enables flexbuffers
- serde chrome_store_page? decentralized? openai?
-
Enables serde and serde_regex, serde of hashbrown and serde of smallvec and string-interner
string-interner:
Enable this if you need
Serdeserialization and deserialization support.Enabled by default.
- fs
-
Affects
utils::fetch_page_html,utils::fetch_page_html… - full_resources
- socks
- reqwest_json
- sitemap
-
Enables sitemap
Affects
configuration::Configuration.sitemap_url,configuration::Configuration.ignore_sitemap… - cache_request cache? cache_chrome_hybrid? cache_mem?
-
Enables http-cache-reqwest and reqwest-middleware
Affects
spider::Client,spider::ClientBuilder,configuration::Configuration.cache… - cache = cache_request
-
Enables manager-cacache of http-cache-reqwest
- cache_mem = cache_request
-
Enables manager-moka of http-cache-reqwest
- cache_openai
-
Enables moka
Affects
openai_common::AICache,utils::openai_request… - cache_chrome_hybrid = cache_request, chrome
-
Enables http, http-cache, and http-cache-semantics
Affects
utils::HttpRequestLike,utils::HttpResponseLike,utils::put_hybrid_cache,utils::cache_chrome_response,utils::convert_headers… - chrome cache_chrome_hybrid? chrome_cpu? chrome_headed? chrome_headless_new? chrome_intercept? chrome_screenshot? chrome_stealth? chrome_store_page? openai? smart?
-
Enables base64 and spider_chrome
Affects
configuration::Configuration.stealth_mode,configuration::Configuration.viewport,configuration::Configuration.timezone_id,configuration::Configuration.locale,configuration::Configuration.evaluate_on_new_document,configuration::Configuration.dismiss_dialogs,configuration::Configuration.wait_for,configuration::Configuration.screenshot,configuration::Configuration.fingerprint,configuration::Configuration.chrome_connection_url,configuration::Configuration.execution_scripts,configuration::Configuration.automation_scripts,configuration::Configuration.chrome_intercept,chrome::setup_auth_challenge_response,chrome::setup_chrome_interception_base,chrome_common::eval_execution_scripts,chrome_common::eval_automation_scripts,features::chrome,page::Page.screenshot_bytes,page::Page.screenshot_bytes… - chrome_headed = chrome
-
Affects
chrome::get_browser_config… - chrome_cpu = chrome
-
Affects
chrome::CHROME_ARGS… - chrome_stealth chrome_screenshot = chrome
- chrome_store_page = chrome, serde
- chrome_intercept openai? smart? = chrome
- chrome_headless_new = chrome
-
Affects
configuration::Configuration.cookie_str,page::Page.cookies,page::Page.cookies,utils::PageResponse.cookies,utils::get_cookies,utils::convert_headers… - cron
-
Enables async-trait, async_job, chrono, and cron ^0.12
Affects
configuration::Configuration.cron_str,configuration::Configuration.cron_type,website::run_cron… - smart = chrome, chrome_intercept
-
Enables aho-corasick and rand
- encoding default
-
Affects
page::get_html_encoded… - headers
-
Enables httpdate
Affects
page::Page.headers,page::Page.headers,utils::PageResponse.headers,utils::FetchPageResult,utils::fetch_page_and_headers,utils::convert_headers… - real_browser
-
Enables rand and statrs ^0.17
Affects
chrome::default_viewport,chrome::CHROME_ARGS,features::chrome_mouse_movements,features::chrome_viewport,chrome::CHROME_ARGS… - openai = chrome, chrome_intercept, serde
-
Enables async-openai, serde_json, and tiktoken-rs ^0.5
Affects
features::openai,page::Page.openai_credits_used,page::Page.extra_ai_data,page::Page.openai_credits_used,page::Page.extra_ai_data,utils::PageResponse.openai_credits_used,utils::PageResponse.extra_ai_data,utils::JsonResponse,utils::handle_openai_credits,utils::handle_extra_ai_data,utils::handle_ai_data,utils::openai_request_base,utils::openai_request,utils::clean_html_base,utils::clean_html_slim,utils::clean_html_full,utils::clean_html,utils::run_openai_request,utils::openai_request,utils::clean_html… - openai_slim_fit
-
Affects
utils::clean_html… - decentralized_headers
-
Enables const_format and itertools ^0.12
Affects
features::decentralized_headers… - spoof
-
Enables fastrand
Affects
spoof_referrer::spoof_referrer… - reqwest_rustls_tls
-
Enables rustls-tls of reqwest
- reqwest_native_tls
-
Enables native-tls of reqwest
reqwest:
Enables native-tls specific functionality not available by default.
- reqwest_native_tls_alpn
-
Enables native-tls-alpn of reqwest
- reqwest_native_tls_vendored
-
Enables native-tls-vendored of reqwest
- reqwest_native_tls_manual_roots
-
Enables rustls-tls-manual-roots of reqwest
- reqwest_native_tls_webpki_roots
-
Enables rustls-tls-webpki-roots of reqwest
- reqwest_native_tls_native_roots default
-
Enables rustls-tls-native-roots of reqwest
- reqwest_hickory_dns
-
Enables hickory-dns of reqwest
- reqwest_multipart
- tokio_io_std
-
tokio:
stdin, stdout, stderr
- tracing
-
Enables tracing, tracing of tokio
Affects
utils::emit_log,utils::emit_log_shutdown… - string_interner_buffer_backend default
-
Enables string-interner
- string_interner_string_backend string_interner_bucket_backend
-
Enables string-interner
- cowboy
-
Allows full concurrency.
- inline-more
-
Enables usage of
#[inline]on far more functions than by default in this crate. This may lead to a performance increase but often comes at a compile time cost.
Features from optional dependencies
In crates that don't use the dep: syntax, optional dependencies automatically become Cargo features. These features may have been created by mistake, and this functionality may be removed in the future.
- tikv-jemallocator not win not android not musl jemalloc?