#tokenizer #sentencepiece

sys sentencepiece-sys

Binding for the sentencepiece tokenizer

18 releases (10 breaking)

0.11.1 Mar 19, 2023
0.10.0 Oct 11, 2022
0.8.3 Jul 30, 2022
0.7.1 Jun 17, 2021
0.1.3 Feb 7, 2020

#678 in Text processing

Download history 66/week @ 2022-12-09 28/week @ 2022-12-16 62/week @ 2022-12-23 29/week @ 2022-12-30 27/week @ 2023-01-06 18/week @ 2023-01-13 53/week @ 2023-01-20 45/week @ 2023-01-27 70/week @ 2023-02-03 42/week @ 2023-02-10 93/week @ 2023-02-17 86/week @ 2023-02-24 34/week @ 2023-03-03 70/week @ 2023-03-10 99/week @ 2023-03-17 64/week @ 2023-03-24

275 downloads per month
Used in 6 crates (via sentencepiece)

Apache-2.0

2MB
25K SLoC

C++ 24K SLoC // 0.1% comments Rust 1K SLoC // 0.0% comments Shell 5 SLoC

sentencepiece

This Rust crate is a binding for the sentencepiece unsupervised text tokenizer. The crate documentation is available online.

libsentencepiece dependency

This crate depends on the sentencepiece C++ library. By default, this dependency is treated as follows:

  • If sentencepiece could be found with pkg-config, the crate will link against the library found through pkg-config. Warning: dynamic linking only works correctly with sentencepiece 0.1.95 or later, due to a bug in earlier versions.
  • Otherwise, the crate's build script will do a static build of the sentencepiece library. This requires that cmake is available.

If you wish to override this behavior, the sentencepiece-sys crate offers two features:

  • system: always attempt to link to the sentencepiece library found with pkg-config.
  • static: always do a static build of the sentencepiece library and link against that.

No runtime deps