Lib.rs

› Science › Machine learning
#tokenizer #tokenize #text-tokenizer

text-tokenizer

Custom text tokenizer

by merl-twin

8 releases

0.5.6 Sep 6, 2024
0.5.5 Sep 6, 2024
0.5.3 Aug 7, 2024
0.5.1 Jul 30, 2024
0.4.1 Dec 20, 2023

#181 in Machine learning

Download history 102/week @ 2024-07-22 140/week @ 2024-07-29 247/week @ 2024-08-05 1/week @ 2024-08-12 380/week @ 2024-09-02

389 downloads per month

MIT license

400KB
5K SLoC

Custom text tokenizer

Dependencies

~1MB
~18K SLoC