1 unstable release
Uses new Rust 2024
| new 0.1.0 | May 20, 2025 |
|---|
#2020 in Text processing
1.5MB
285 lines
token-dict
basic dictionary based tokenization
Uses new Rust 2024
| new 0.1.0 | May 20, 2025 |
|---|
#2020 in Text processing
1.5MB
285 lines
basic dictionary based tokenization