sentencepiece-model

SentencePiece model parser generated from the SentencePiece protobuf definition.

use sentencepiece_model::SentencePieceModel;

let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer()?.unk_id(), 2);

Usage

[dependencies]
sentencepiece-model = "0.1"

sentencepiece-model uses prost-build and protox to generate Rust code from the SentencePiece protobuf definition at build time. protoc is not required.

lib.rs:

SentencePiece model parser generated from the SentencePiece protobuf definition.

See SentencePieceModel for the entry point for parsing and accessing sentencepiece models.

use sentencepiece_model::SentencePieceModel;

let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer().unwrap().unk_id(), 2);