Lib.rs

› Parser implementations | Programming languages › sentencepiece
#sentence-piece #bindings #tokenizer

sys sentencepiece-sys

by Daniël de Kok and 2 contributors

19 releases (10 breaking)

0.11.2 Jul 22, 2023
0.11.1 Mar 19, 2023
0.10.0 Oct 11, 2022
0.8.3 Jul 30, 2022
0.1.3 Feb 7, 2020

#110 in #tokenizer

Download history 246/week @ 2024-07-22 189/week @ 2024-07-29 439/week @ 2024-08-05 783/week @ 2024-08-12 452/week @ 2024-08-19 359/week @ 2024-08-26 495/week @ 2024-09-02 310/week @ 2024-09-09 1033/week @ 2024-09-16 406/week @ 2024-09-23 361/week @ 2024-09-30 349/week @ 2024-10-07 313/week @ 2024-10-14 336/week @ 2024-10-21 800/week @ 2024-10-28 615/week @ 2024-11-04

2,078 downloads per month
Used in 10 crates (via sentencepiece)

Apache-2.0

2MB
25K SLoC

C++ 24K SLoC // 0.1% comments Bitbake 371 SLoC // 0.5% comments Rust 216 SLoC // 0.0% comments Shell 5 SLoC

Binding for the sentencepiece tokenizer

No runtime deps