Experimental fast tokenizer
Owned by Mik.
#14 in #bpe
21KB 496 lines
Experimental Rust Tokenizer
~6–23MB ~278K SLoC