Lib.rs

› Text processing
#tokenizer #tokenize #text-tokenizer

text-tokenizer

Custom text tokenizer

by merl-twin

12 releases

0.5.10 Nov 22, 2024
0.5.9 Nov 20, 2024
0.5.6 Sep 6, 2024
0.5.3 Aug 7, 2024
0.4.1 Dec 20, 2023

#611 in Text processing

Download history 294/week @ 2024-08-31 86/week @ 2024-09-07 31/week @ 2024-09-14 3/week @ 2024-09-21 38/week @ 2024-09-28 460/week @ 2024-11-16 95/week @ 2024-11-23 35/week @ 2024-11-30

590 downloads per month

MIT license

490KB
10K SLoC

Custom text tokenizer

Dependencies

~1MB
~18K SLoC