Lib.rs

› Text processing
#tokenizer #tokenize #text-tokenizer

text-tokenizer

Custom text tokenizer

by merl-twin

12 releases

0.5.10 Nov 22, 2024
0.5.9 Nov 20, 2024
0.5.6 Sep 6, 2024
0.5.3 Aug 7, 2024
0.4.1 Dec 20, 2023

#520 in Text processing

MIT license

490KB
10K SLoC

Custom text tokenizer

Dependencies

~1MB
~18K SLoC