Lib.rs

›
#tokenizer #bpe #nlp #word-piece #tiktoken

tokin

Experimental fast tokenizer

Owned by Mik.

1 unstable release

0.1.0 Oct 16, 2024

#10 in #bpe

Download history 132/week @ 2024-10-13 12/week @ 2024-10-20 1/week @ 2024-10-27

145 downloads per month

MIT license

21KB
496 lines

tokin

Experimental Rust Tokenizer

Dependencies

~7–23MB
~277K SLoC