Cargo Features

AlephAlphaTokenizer has no features set by default.

[dependencies]
aleph-alpha-tokenizer = { version = "0.3.1", features = ["huggingface"] }
huggingface = tokenizers

Enable this to allow using the tokenizer as a tokenizer::Model

Features from optional dependencies

In crates that don't use the dep: syntax, optional dependencies automatically become Cargo features. These features may have been created by mistake, and this functionality may be removed in the future.

tokenizers huggingface?

Enables tokenizers ^0.10.1