Lib.rs

› Science › Machine learning
#tokenizer #version #python

bytepiece

Rust version of bytepiece tokenizer

by SunDoge

2 unstable releases

0.2.0 Sep 26, 2023
0.1.0 Sep 21, 2023

#503 in Machine learning

Apache-2.0

12KB
230 lines

bytepiece

Dependencies

~4.5–6.5MB
~118K SLoC