1 unstable release
0.1.3 | Feb 15, 2023 |
---|---|
0.1.2 |
|
0.1.1 |
|
0.1.0 |
|
#16 in #bert
37 downloads per month
Used in bert_create_pretraining
225KB
373 lines
bert_tokenizer
The crate provides the port of the original BERT tokenizer from the Google BERT repository.
License
MIT license. See LICENSE file for full license.
lib.rs
:
This crate is a Rust port of Google's BERT GoogleBERT WordPiece tokenizer.
Dependencies
~2.5MB
~54K SLoC