Lib.rs

Cargo Features

[dependencies]
tokenizers = { version = "0.23.1", default-features = false, features = ["esaxx_fast", "progressbar", "http", "unstable_wasm", "rustls-tls", "onig"] }

default = esaxx_fast, onig, progressbar

These default features are set whenever tokenizers is added without default-features = false somewhere in the dependency tree.

esaxx_fast default

Enables cpp of esaxx-rs

progressbar default = indicatif

http = hf-hub

unstable_wasm = fancy-regex

Enables wasm_js of getrandom ^0.3

getrandom:

Optional backend: wasm_js

This flag enables the wasm_js backend and uses it by default on wasm32 where the target_os is unknown. The getrandom_backend cfg may override this.

WARNING: We strongly recommend against enabling this feature in libraries (except for tests) since it is known to break non-Web WASM builds and further since the usage of wasm-bindgen causes significant bloat to Cargo.lock (on all targets).

The only exception to this rule: if your crate already unconditionally depends on wasm-bindgen or js-sys on "unknown" WASM targets then it's acceptable to enable this feature unconditionally.

rustls-tls

Enables rustls-tls of optional hf-hub ^0.4.1

Features from optional dependencies

In crates that don't use the dep: syntax, optional dependencies automatically become Cargo features. These features may have been created by mistake, and this functionality may be removed in the future.

onig default

indicatif progressbar

hf-hub http?

Enables hf-hub ^0.4.1

fancy-regex unstable_wasm?

Enables fancy-regex ^0.17