1 unstable release
| 0.1.3 | Feb 15, 2023 |
|---|---|
| 0.1.2 |
|
| 0.1.1 |
|
| 0.1.0 |
|
#15 in #bert
Used in bert_create_pretraining
225KB
373 lines
bert_tokenizer
The crate provides the port of the original BERT tokenizer from the Google BERT repository.
License
MIT license. See LICENSE file for full license.
lib.rs:
This crate is a Rust port of Google's BERT GoogleBERT WordPiece tokenizer.
Dependencies
~2MB
~60K SLoC