Lib.rs

› Parser implementations | Programming languages › sentencepiece
#sentence-piece #bindings #tokenizer

sys sentencepiece-sys

by Daniël de Kok and 2 contributors

19 releases (10 breaking)

0.11.2 Jul 22, 2023
0.11.1 Mar 19, 2023
0.10.0 Oct 11, 2022
0.8.3 Jul 30, 2022
0.1.3 Feb 7, 2020

#2140 in Parser implementations

Download history 109/week @ 2024-03-14 65/week @ 2024-03-21 119/week @ 2024-03-28 92/week @ 2024-04-04 110/week @ 2024-04-11 157/week @ 2024-04-18 169/week @ 2024-04-25 353/week @ 2024-05-02 272/week @ 2024-05-09 364/week @ 2024-05-16 279/week @ 2024-05-23 370/week @ 2024-05-30 1554/week @ 2024-06-06 1935/week @ 2024-06-13 2538/week @ 2024-06-20 3322/week @ 2024-06-27

9,435 downloads per month
Used in 7 crates (via sentencepiece)

Apache-2.0

2MB
25K SLoC

C++ 24K SLoC // 0.1% comments Bitbake 371 SLoC // 0.5% comments Rust 216 SLoC // 0.0% comments Shell 5 SLoC

Binding for the sentencepiece tokenizer

No runtime deps