2 releases

0.0.2 Sep 19, 2024
0.0.1 Aug 11, 2024
0.0.0 Aug 7, 2024

#1002 in Algorithms

46 downloads per month

MIT license

36KB
895 lines

Tokeneer

CI Latest version Documentation license GitHub repo size GitHub code size in bytes

GitHub Issues GitHub Pull Requests GitHub contributors GitHub commit activity

Another tokenizer crate.

Dependencies

~2.4–3.5MB
~62K SLoC