Cargo Features
AlephAlphaTokenizer has no features set by default.
[dependencies]
aleph-alpha-tokenizer = { version = "0.3.1", features = ["huggingface"] }- huggingface = tokenizers
-
Enable this to allow using the tokenizer as a
tokenizer::Model
Features from optional dependencies
In crates that don't use the dep: syntax, optional dependencies automatically become Cargo features. These features may have been created by mistake, and this functionality [may be removed in the future](https://rust-lang.github.io/rfcs/3491-remove-implicit-features.html).
- tokenizers huggingface?
-
Enables tokenizers ^0.10.1