Lib.rs

Cargo Features

[dependencies]
spider = { version = "1.93.38", default-features = false, features = ["regex", "glob", "ua_generator", "jemalloc", "decentralized", "control", "time", "sync", "flexbuffers", "serde", "fs", "full_resources", "socks", "reqwest_json", "sitemap", "js", "budget", "cache", "cache_mem", "cache_openai", "chrome", "chrome_headed", "chrome_cpu", "chrome_screenshot", "chrome_store_page", "chrome_intercept", "chrome_headless_new", "cookies", "cron", "napi", "napi_rustls_tls", "smart", "encoding", "headers", "real_browser", "openai", "openai_slim_fit", "decentralized_headers", "spoof"] }

default = cookies, sync, ua_generator

These default features are set whenever spider is added without default-features = false somewhere in the dependency tree.

Enables native-tls-alpn of reqwest

regex

Enables regex

Affects spider::black_list…

glob

Enables itertools and regex

Affects features::glob…

ua_generator default

Enables ua_generator

Affects configuration::get_ua…

jemalloc = tikv-jemallocator

decentralized = flexbuffers, serde

Affects page::Page, page::build, utils::fetch_page, utils::FetchPageResult, utils::fetch_page_and_headers…

control

Affects utils::Handler, utils::pause, utils::resume, utils::shutdown, utils::reset, website::Website.crawl_id…

time

sync default

Enables io-std and sync of tokio

flexbuffers decentralized?

Enables flexbuffers

serde chrome_store_page? decentralized? openai?

Enables serde, serde of compact_str and hashbrown

fs

Enables fs of tokio

Affects utils::fetch_page_html…

full_resources

socks

Enables socks of reqwest

reqwest_json

Enables json of reqwest

sitemap

Enables sitemap

Affects configuration::Configuration.sitemap_url, configuration::Configuration.ignore_sitemap…

js

Enables jsdom

budget

Affects configuration::Configuration.depth, configuration::Configuration.depth_distance, configuration::Configuration.budget, configuration::Configuration.wild_card_budgeting…

cache cache_mem?

Enables http-cache-reqwest and reqwest-middleware

Affects spider::Client, spider::ClientBuilder, configuration::Configuration.cache…

cache_mem = cache

Enables manager-moka of http-cache-reqwest

cache_openai

Enables moka

Affects openai_common::AICache, utils::openai_request…

chrome chrome_cpu? chrome_headed? chrome_headless_new? chrome_intercept? chrome_screenshot? chrome_stealth? chrome_store_page? openai? smart?

Enables chromiumoxide

Affects configuration::Configuration.stealth_mode, configuration::Configuration.chrome_intercept, configuration::Configuration.viewport, configuration::Configuration.chrome_intercept_block_visuals, configuration::Configuration.timezone_id, configuration::Configuration.locale, configuration::Configuration.evaluate_on_new_document, configuration::Configuration.wait_for, configuration::Configuration.screenshot, configuration::Configuration.fingerprint, configuration::Configuration.chrome_connection_url, features::chrome, page::Page.screenshot_bytes, page::Page.screenshot_bytes, utils::PageResponse.screenshot_bytes, utils::wait_for_event, utils::wait_for_selector, utils::create_output_path, utils::page_wait, utils::handle_extra_ai_data…

chrome_headed = chrome

Affects chrome::get_browser_config…

chrome_cpu = chrome

Affects chrome::CHROME_ARGS…

chrome_stealth chrome_screenshot = chrome

chrome_store_page = chrome, serde

chrome_intercept openai? = chrome

chrome_headless_new = chrome

cookies default

Enables cookies of reqwest

Affects configuration::Configuration.cookie_str…

cron

Enables async-trait, async_job, chrono, and cron

Affects configuration::Configuration.cron_str, configuration::Configuration.cron_type, website::run_cron…

napi napi_rustls_tls?

Enables napi

Affects website::WebsiteBuilderError…

napi_rustls_tls = napi

Enables rustls-tls of reqwest

smart = chrome

Enables regex

encoding

Enables encoding_rs

headers

Affects page::Page.headers, page::Page.headers, utils::PageResponse.headers, utils::FetchPageResult, utils::fetch_page_and_headers…

real_browser

Affects chrome::CHROME_ARGS, chrome::CHROME_ARGS…

openai = chrome, chrome_intercept, serde

Enables async-openai ^0.20.0, lol_html, serde_json, and tiktoken-rs

Affects features::openai, page::Page.openai_credits_used, page::Page.extra_ai_data, page::Page.openai_credits_used, page::Page.extra_ai_data, utils::PageResponse.openai_credits_used, utils::PageResponse.extra_ai_data, utils::handle_openai_credits, utils::handle_extra_ai_data, utils::handle_ai_data, utils::openai_request_base, utils::openai_request, utils::clean_html_base, utils::clean_html_slim, utils::clean_html, utils::JsonResponse, utils::openai_request, utils::clean_html…

openai_slim_fit

Affects utils::clean_html…

decentralized_headers

Enables const_format and itertools

Affects features::decentralized_headers…

spoof

Enables fastrand

Affects spoof_referrer::spoof_referrer…

Features from optional dependencies

In crates that don't use the dep: syntax, optional dependencies automatically become Cargo features. These features may have been created by mistake, and this functionality may be removed in the future.

tikv-jemallocator not win not android not musl jemalloc?