Lib.rs

› Text processing
#tokenizer #tokenize #great-tokenizer

nightly great-tokenizer

Owned by daiyan1145.

2 unstable releases

0.2.0 Aug 28, 2024
0.1.0 Aug 28, 2024

#1650 in Text processing

Download history 147/week @ 2024-08-22 86/week @ 2024-08-29 22/week @ 2024-09-12 11/week @ 2024-09-19 15/week @ 2024-09-26 5/week @ 2024-10-03

53 downloads per month

MIT license

16KB
407 lines

A tokenizer

Dependencies

~6–13MB
~160K SLoC