22 stable releases (5 major)

6.1.0 Nov 16, 2020
5.0.1 Oct 1, 2020
4.0.0 Aug 24, 2020
3.1.6 Jul 25, 2020
1.0.0 Feb 15, 2020

#42 in Machine learning

Download history 367/week @ 2020-08-09 183/week @ 2020-08-16 253/week @ 2020-08-23 210/week @ 2020-08-30 409/week @ 2020-09-06 232/week @ 2020-09-13 177/week @ 2020-09-20 183/week @ 2020-09-27 215/week @ 2020-10-04 207/week @ 2020-10-11 343/week @ 2020-10-18 254/week @ 2020-10-25 230/week @ 2020-11-01 278/week @ 2020-11-08 269/week @ 2020-11-15 247/week @ 2020-11-22

1,047 downloads per month
Used in 3 crates (2 directly)

Apache-2.0

660KB
13K SLoC

rust-tokenizers

Rust-tokenizer is a drop-in replacement for the tokenization methods from the Transformers library It includes a broad range of tokenizers for state-of-the-art transformers architectures, including:

  • Sentence Piece (unigram model)
  • BERT
  • DistilBERT
  • RoBERTa
  • GPT
  • GPT2
  • CTRL

The wordpiece based tokenizers include both single-threaded and multi-threaded processing. The Byte-Pair-Encoding tokenizers favor the use of a shared cache and are only available as single-threaded tokenizers Using the tokenizers requires downloading manually the tokenizers required files (vocabulary or merge files). These can be found in the Transformers library.

Usage example

let vocab = Arc::new(rust_tokenizers::BertVocab::from_file(&vocab_path));

let test_sentence = Example::new_from_string("This is a sample sentence to be tokenized");
let bert_tokenizer: BertTokenizer = BertTokenizer::from_existing_vocab(vocab.clone());

println!("{:?}", bert_tokenizer.encode(&test_sentence.sentence_1,
                                       None,
                                       128,
                                       &TruncationStrategy::LongestFirst,
                                       0));

Dependencies

~8.5MB
~210K SLoC