Tokenizers is used at run time in 8 crates (of which 2 optionally, 7 directly).

Number of dependers Tokenizers version Downloads/month
1 0.13.2 9.0K
5 0.11.3 2.3K
2 0.10.1 100
Depender (with downloads and own rev deps) Tokenizers version
modeling ^0.11.0
cracken ~0.11.0
sbert ^0.10
1 wonnx-preprocessing ^0.11.3
dataflow_nlp ^0.11
pyke-diffusers optional ^0.13
aleph-alpha-tokenizer optional ^0.10.1