18 releases
new 0.9.0 | Aug 11, 2022 |
---|---|
0.8.2 | Jul 30, 2022 |
0.8.1 | Jan 4, 2022 |
0.8.0 | Jul 11, 2021 |
0.1.3 | Feb 7, 2020 |
481 downloads per month
Used in 5 crates
(2 directly)
2MB
26K
SLoC
sentencepiece
This Rust crate is a binding for the sentencepiece unsupervised text tokenizer. The crate documentation is available online.
libsentencepiece
dependency
This crate depends on the sentencepiece
C++ library. By default,
this dependency is treated as follows:
- If
sentencepiece
could be found withpkg-config
, the crate will link against the library found throughpkg-config
. Warning: dynamic linking only works correctly with sentencepiece 0.1.95 or later, due to a bug in earlier versions. - Otherwise, the crate's build script will do a static build of the
sentencepiece
library. This requires thatcmake
is available.
If you wish to override this behavior, the sentencepiece-sys
crate
offers two features:
system
: always attempt to link to thesentencepiece
library found withpkg-config
.static
: always do a static build of thesentencepiece
library and link against that.