34 releases (7 breaking)

0.7.3 Oct 24, 2022
0.6.2 Oct 8, 2022
0.2.1 Jul 24, 2022
0.0.11 Jan 5, 2022
0.0.6 Dec 30, 2021

#111 in #tokenizer

Download history 1/week @ 2024-06-18 33/week @ 2024-07-02 1/week @ 2024-07-23 3/week @ 2024-09-17 80/week @ 2024-09-24

83 downloads per month
Used in 2 crates (via morsels_indexer)

MIT license

155KB
2.5K SLoC

Morsels Language Package

This is a tokenizer crate for the morsels_indexer cli tool, not intended to be used directly.

Dependencies

~3.5MB
~44K SLoC