1 unstable release
| 0.0.1 | Aug 11, 2024 |
|---|---|
| 0.0.0 |
|
#9 in #bpe
285 downloads per month
34KB
857 lines
Tokeneer
Another tokenizer crate.
Dependencies
~2.4–3.5MB
~61K SLoC
| 0.0.1 | Aug 11, 2024 |
|---|---|
| 0.0.0 |
|
#9 in #bpe
285 downloads per month
34KB
857 lines
Another tokenizer crate.
~2.4–3.5MB
~61K SLoC