1 unstable release
| 0.1.1 | Feb 26, 2024 |
|---|
#18 in #bpe
17KB
365 lines
Minimal, fast, multi-threaded implementation of the Byte Pair Encoding (BPE) for LLM tokenization
Dependencies
~9–19MB
~296K SLoC
| 0.1.1 | Feb 26, 2024 |
|---|
#18 in #bpe
17KB
365 lines
Minimal, fast, multi-threaded implementation of the Byte Pair Encoding (BPE) for LLM tokenization
~9–19MB
~296K SLoC