2 releases
| 0.0.2 | Sep 19, 2024 |
|---|---|
| 0.0.1 | Aug 11, 2024 |
| 0.0.0 |
|
#1002 in Algorithms
46 downloads per month
36KB
895 lines
Tokeneer
Another tokenizer crate.
Dependencies
~2.4–3.5MB
~62K SLoC
| 0.0.2 | Sep 19, 2024 |
|---|---|
| 0.0.1 | Aug 11, 2024 |
| 0.0.0 |
|
#1002 in Algorithms
46 downloads per month
36KB
895 lines
Another tokenizer crate.
~2.4–3.5MB
~62K SLoC