24 stable releases (5 major)

6.2.1 Feb 19, 2021
6.1.0 Nov 16, 2020
5.0.1 Oct 1, 2020
4.0.0 Aug 24, 2020
1.0.0 Feb 15, 2020

#8 in #tokenizer

Download history 254/week @ 2020-11-16 279/week @ 2020-11-23 255/week @ 2020-11-30 202/week @ 2020-12-07 92/week @ 2020-12-14 132/week @ 2020-12-21 74/week @ 2020-12-28 374/week @ 2021-01-04 128/week @ 2021-01-11 139/week @ 2021-01-18 173/week @ 2021-01-25 134/week @ 2021-02-01 113/week @ 2021-02-08 222/week @ 2021-02-15 176/week @ 2021-02-22 206/week @ 2021-03-01

718 downloads per month
Used in 3 crates (2 directly)

Apache-2.0

705KB
14K SLoC

rust-tokenizers

Rust-tokenizer is a drop-in replacement for the tokenization methods from the Transformers library It includes a broad range of tokenizers for state-of-the-art transformers architectures, including:

  • Sentence Piece (unigram model)
  • BERT
  • ALBERT
  • DistilBERT
  • RoBERTa
  • GPT
  • GPT2
  • ProphetNet
  • CTRL

The wordpiece based tokenizers include both single-threaded and multi-threaded processing. The Byte-Pair-Encoding tokenizers favor the use of a shared cache and are only available as single-threaded tokenizers Using the tokenizers requires downloading manually the tokenizers required files (vocabulary or merge files). These can be found in the Transformers library.

Usage example

let vocab = Arc::new(rust_tokenizers::BertVocab::from_file(&vocab_path));

let test_sentence = Example::new_from_string("This is a sample sentence to be tokenized");
let bert_tokenizer: BertTokenizer = BertTokenizer::from_existing_vocab(vocab.clone());

println!("{:?}", bert_tokenizer.encode(&test_sentence.sentence_1,
                                       None,
                                       128,
                                       &TruncationStrategy::LongestFirst,
                                       0));

Dependencies

~9MB
~214K SLoC