3 unstable releases
Uses new Rust 2024
| 0.1.0 | Feb 23, 2025 |
|---|---|
| 0.0.2 | Sep 19, 2024 |
| 0.0.1 | Aug 11, 2024 |
| 0.0.0 |
|
#734 in Algorithms
36KB
893 lines
Tokeneer
Another tokenizer crate.
Dependencies
~2.3–3.5MB
~61K SLoC
Uses new Rust 2024
| 0.1.0 | Feb 23, 2025 |
|---|---|
| 0.0.2 | Sep 19, 2024 |
| 0.0.1 | Aug 11, 2024 |
| 0.0.0 |
|
#734 in Algorithms
36KB
893 lines
Another tokenizer crate.
~2.3–3.5MB
~61K SLoC