Lib.rs

› Parser implementations | Programming languages › sentencepiece
#sentence-piece #bindings #tokenizer

sys sentencepiece-sys

by Daniël de Kok and 2 contributors

19 releases (10 breaking)

0.11.2 Jul 22, 2023
0.11.1 Mar 19, 2023
0.10.0 Oct 11, 2022
0.8.3 Jul 30, 2022
0.1.3 Feb 7, 2020

#127 in #tokenizer

Download history 567/week @ 2024-11-15 628/week @ 2024-11-22 922/week @ 2024-11-29 1150/week @ 2024-12-06 806/week @ 2024-12-13 378/week @ 2024-12-20 601/week @ 2024-12-27 1372/week @ 2025-01-03 1427/week @ 2025-01-10 1347/week @ 2025-01-17 2630/week @ 2025-01-24 1539/week @ 2025-01-31 1573/week @ 2025-02-07 771/week @ 2025-02-14 1600/week @ 2025-02-21 892/week @ 2025-02-28

5,056 downloads per month
Used in 12 crates (via sentencepiece)

Apache-2.0

2MB
25K SLoC

C++ 24K SLoC // 0.1% comments Bitbake 371 SLoC // 0.5% comments Rust 216 SLoC // 0.0% comments Shell 5 SLoC

Binding for the sentencepiece tokenizer

No runtime deps