34 releases (7 breaking)

0.7.3 Oct 24, 2022
0.6.2 Oct 8, 2022
0.2.1 Jul 24, 2022
0.0.11 Jan 5, 2022
0.0.6 Dec 30, 2021

#120 in Internationalization (i18n)

Download history 11/week @ 2022-12-02 14/week @ 2022-12-09 38/week @ 2022-12-16 16/week @ 2022-12-23 3/week @ 2022-12-30 7/week @ 2023-01-06 12/week @ 2023-01-13 20/week @ 2023-01-20 16/week @ 2023-01-27 45/week @ 2023-02-03 46/week @ 2023-02-10 97/week @ 2023-02-17 112/week @ 2023-02-24 8/week @ 2023-03-03 12/week @ 2023-03-10 7/week @ 2023-03-17

141 downloads per month
Used in 2 crates (via morsels_indexer)

MIT license

260KB
7K SLoC

Morsels Language Package

This is a tokenizer crate for the morsels_indexer cli tool, not intended to be used directly.

Dependencies

~220–480KB
~10K SLoC