Lib.rs

› Text processing
#tokenizer #tokenize #text-tokenizer

text-tokenizer

Custom text tokenizer

by merl-twin

11 releases

new 0.5.9 Nov 20, 2024
0.5.8 Nov 20, 2024
0.5.6 Sep 6, 2024
0.5.3 Aug 7, 2024
0.4.1 Dec 20, 2023

#558 in Text processing

Download history 153/week @ 2024-07-31 149/week @ 2024-08-07 102/week @ 2024-08-28 278/week @ 2024-09-04 22/week @ 2024-09-11 12/week @ 2024-09-18 18/week @ 2024-09-25 20/week @ 2024-10-02 68/week @ 2024-11-13

71 downloads per month

MIT license

490KB
10K SLoC

Custom text tokenizer

Dependencies

~1MB
~18K SLoC