1 unstable release
0.1.1 | Feb 26, 2024 |
---|
#465 in Machine learning
17KB
365 lines
Minimal, fast, multi-threaded implementation of the Byte Pair Encoding (BPE) for LLM tokenization
Dependencies
~9–18MB
~263K SLoC
0.1.1 | Feb 26, 2024 |
---|
#465 in Machine learning
17KB
365 lines
Minimal, fast, multi-threaded implementation of the Byte Pair Encoding (BPE) for LLM tokenization
~9–18MB
~263K SLoC