Lib.rs

› Text processing
#tokenizer #tokenize #text-tokenizer

text-tokenizer

Custom text tokenizer

by merl-twin

12 releases

0.5.10 Nov 22, 2024
0.5.9 Nov 20, 2024
0.5.6 Sep 6, 2024
0.5.3 Aug 7, 2024
0.4.1 Dec 20, 2023

#466 in Text processing

Download history 547/week @ 2024-11-19 13/week @ 2024-11-26 28/week @ 2024-12-03 12/week @ 2024-12-10

893 downloads per month

MIT license

490KB
10K SLoC

Custom text tokenizer

Dependencies

~1MB
~18K SLoC