Lib.rs

› Science › Machine learning | Hardware support
#tokenize #version #python

bytepiece

Rust version of bytepiece tokenizer

by SunDoge

2 unstable releases

0.2.0 Sep 26, 2023
0.1.0 Sep 21, 2023

#1478 in Hardware support

Apache-2.0

12KB
230 lines

bytepiece

Dependencies

~4–6MB
~110K SLoC