2 releases
| 0.0.2 | Sep 19, 2024 |
|---|---|
| 0.0.1 | Aug 11, 2024 |
| 0.0.0 |
|
#717 in Algorithms
242 downloads per month
36KB
895 lines
Tokeneer
Another tokenizer crate.
Dependencies
~2.5–3.5MB
~63K SLoC
| 0.0.2 | Sep 19, 2024 |
|---|---|
| 0.0.1 | Aug 11, 2024 |
| 0.0.0 |
|
#717 in Algorithms
242 downloads per month
36KB
895 lines
Another tokenizer crate.
~2.5–3.5MB
~63K SLoC