| 1 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | 2 | 2 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 5 | 5 | 5 | 5 | 5 | 6 | 6 | 7 | 7 |
10
|
11 | 12 | 13 | 13 |
14
|
15
|
31
|
32
|
31
|
38
|
44
|
43
|
42
|
52
|
49
|
53
|
51
|
48
|
51
|
54
|
57
|
58
|
| 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Tokenizers is used at run time in 133 crates (of which 15 optionally). It is a direct run-time dependency in 87 crates. It's used only as a dev dependency in 5 crates (of which 4 directly).
| Number of dependers | Tokenizers version | Downloads/month |
|---|---|---|
| 63 | 0.21.1 | 51K |
| 0 | 0.21.0 | 52K |
| 10 | 0.20.4 | 16K |
| 1 | 0.20.3 | 25K |
| 20 | 0.19.1 | 23K |
| 20 | 0.15.2 | 37K |
| 9 | 0.14.1 | 3.7K |
| 10 | 0.13.4 | 4.8K |
| 4 | 0.11.3 | 0 |
| 1 | 0.10.1 | 54 |
| Depender (with downloads and own rev deps) | Tokenizers version | |||
|---|---|---|---|---|
| 13K | 1 | toktrie_hf_tokenizers | >=0.20.0, <1.0.0 | |
| 32K | 15 | text-splitter | optional | ^0.21 |
| 6.1K | 12 | fastembed | ^0.21 | |
| 3.7K | 2 | gibberish-or-not | ^0.14.1 | |
| 5.0K | 5 | lancedb | optional | ^0.19.1 |
| 3.0K | 19 | rust-bert | optional | ^0.20 |
| 700 | aleph-alpha-client | ^0.21.0 | ||
| 600 | r2md | ^0.21.0 | ||
| 550 | coze | ^0.15.2 | ||
| 500 | gline-rs | ^0.21.0 | ||
| 500 | 2 | claude-tokenizer | ^0.20.0 | |
| 450 | 4 | candle-datasets | ^0.21.0 | |
| 440 | rai-models | ^0.15.0 | ||
| 420 | 4 | candle-examples | ^0.21.0 | |
| 390 | 1 | kalosm-llama | ^0.21.0 | |
| 340 | valentinus | >=0.13.4 | ||
| 330 | 1 | rwhisper | ^0.21.0 | |
| 280 | mpnet-rs | ^0.15.1 | ||
| 270 | 1 | rbert | ^0.21.0 | |
| 220 | tkn-cli | ^0.21.1 | ||
| 180 | bundle_repo | ^0.21 | ||
| 170 | 1 | kalosm-ocr | ^0.21.0 | |
| 170 | llms-from-scratch-rs | ^0.21.0 | ||
| 150 | 1 | rwuerstchen | ^0.21.0 | |
| 150 | 4 | alith-models | ^0.21.1 | |
| 140 | simple-whisper | ^0.19.1 | ||
| 120 | embed_anything | ^0.21.1 | ||
| 120 | gte-rs | ^0.21.0 | ||
| 110 | ct2rs | ^0.21.0 | ||
| 110 | lsp-ai | ^0.14.1 | ||
| 110 | 1 | alith-inference | ^0.21.1 | |
| 100 | rphi | ^0.19.1 | ||
| gpt_sovits_rs | ^0.20.0 | |||
| smartgpt | ^0.13.3 | |||
| glimpse | ^0.21.0 | |||
| rusty-dawg | ^0.20.0 | |||
| ragtime | ^0.19 | |||
| oxidized-transformers | ^0.15 | |||
| outlines-core | =0.20.3 | |||
| reco-forge | ^0.15.0 | |||
| usls | ^0.15.2 | |||
| 3 | llm_models | ^0.21.0 | ||
| edge-transformers | ^0.13 | |||
| promptbox | ^0.15.0 | |||
| musicgpt | ^0.19.1 | |||
| enfer_core | ^0.15.0 | |||
| bm42 | ^0.19 | |||
| dataflow_nlp | ^0.11 | |||
| fashion-clip-rs | ^0.15 | |||
| 2 | diffuse_rs_common | ^0.21.0 | ||
| 2 | diffusion_rs_common | ^0.21.0 | ||
| sbv2_core | ^0.20.0 | |||
| moshi-backend | ^0.15.2 | |||
| moz_genai | ^0.15.0 | |||
| aiochatgpt | ^0.13.2 | |||
| 1 | diffusion_rs_core | ^0.21.0 | ||
| callm | ^0.19 | |||
| bge | ^0.15.2 | |||
| 4 | sapiens | ^0.14.1 | ||
| 1 | clip_txt | ^0.13.3 | ||
| spoilers | ^0 | |||
| aleph-alpha-api | ^0.15.0 | |||
| cracken | ~0.11.0 | |||
| rsllm | ^0.19.1 | |||
| sbert | ^0.11 | |||
| 1 | stable-diffusion | ^0.15.0 | ||
| bridge-common | ^0.19.1 | |||
| candle_embed | ^0.19.1 | |||
| 1 | diffuse_rs_core | ^0.21.0 | ||
| glowrs | ^0.19.1 | |||
| 1 | wonnx-preprocessing | ^0.13.3 | ||
| heapswap | ^0.15.1 | |||
| modeling | ^0.11.0 | |||
| prefrontal | ^0.15 | |||
| token-counter | ^0.19.1 | |||
| wimbd | ^0.15.1 | |||
| anonymize_rs | ^0.13.3 | |||
| candle-chat-types | ^0.13.4 | |||
| codegeex4-candle | ^0.19.1 | |||
| inference_core | ^0.15.0 | |||
| rmistral | ^0.13.4 | |||
| yolors | ^0.15.0 | |||
| 260 | code-splitter | optional | ^0.19.1 | |
| 180 | 2 | beet_ml | optional | ^0.19.1 |
| 130 | sentinellm_ai | optional | ^0.20 | |
| aleph-alpha-tokenizer | optional | ^0.10.1 | ||
| norma | optional | ^0.20.0 | ||
| 507K | 55 | ort | dev | >=0.13.4 |
| 700 | 3 | general-sam | dev | ^0.21.1 |
| 100 | mtc-token-healing | dev | ^0.21.1 | |
| nano-vectordb-rs | dev | ^0.21.0 | ||