1 unstable release
0.1.0 | Jan 2, 2025 |
---|
#805 in Machine learning
214 downloads per month
16KB
438 lines
Tocken
Tokenizer implemented in Rust.
This tokenizer is based on Lucene's EnglishAnalyzer.
Usage
Dependencies
~10MB
~168K SLoC
0.1.0 | Jan 2, 2025 |
---|
#805 in Machine learning
214 downloads per month
16KB
438 lines
Tokenizer implemented in Rust.
This tokenizer is based on Lucene's EnglishAnalyzer.
~10MB
~168K SLoC