Lib.rs

› Text processing
#tokenizer #tokenize #text-tokenizer

text-tokenizer

Custom text tokenizer

by merl-twin

1 unstable release

0.4.1 Dec 20, 2023

#931 in Text processing

27 downloads per month

MIT license

355KB
5K SLoC

Custom text tokenizer

Dependencies

~3–4MB
~65K SLoC