1 unstable release
0.0.0 | Jun 29, 2021 |
---|
#103 in #compact
2KB
This crate is a fork of: https://github.com/huggingface/tokenizers.
We removed all the cli parts, the multithreaded features and every non-BERT-related part to make it more compact and to allow it to work on wasm.