Lib.rs

› Text processing › morsels_common
#ascii #tokenizer #morsels #basic #language #morsels-indexer #search

morsels_lang_ascii

Basic ascii tokenizer for morsels

by Ang Ze Yu

34 releases (7 breaking)

0.7.3 Oct 24, 2022
0.6.2 Oct 8, 2022
0.2.1 Jul 24, 2022
0.0.11 Jan 5, 2022
0.0.6 Dec 30, 2021

#1942 in Text processing

Download history 10/week @ 2024-06-16 3/week @ 2024-06-23 145/week @ 2024-06-30 6/week @ 2024-07-14 6/week @ 2024-07-21 6/week @ 2024-07-28 4/week @ 2024-08-04 1/week @ 2024-08-11 2/week @ 2024-09-08 72/week @ 2024-09-22 2/week @ 2024-09-29

76 downloads per month
Used in 4 crates (3 directly)

MIT license

145KB
2.5K SLoC

Morsels Language Package

This is a tokenizer crate for the morsels_indexer cli tool, not intended to be used directly.

Dependencies

~205–740KB
~12K SLoC