Lib.rs

› Text processing
#tokenizer #tokenize #great-tokenizer

nightly great-tokenizer

Owned by daiyan1145.

2 unstable releases

0.2.0 Aug 28, 2024
0.1.0 Aug 28, 2024

#1796 in Text processing

MIT license

16KB
407 lines

A tokenizer

Dependencies

~6–13MB
~160K SLoC