Lib.rs

› Text processing › Ellie-Engine
#ellie #language #tokenizer #position #file #items #list

no-std ellie_tokenizer

Tokenizer for ellie language

by Ahmetcan Aksu, Behemehal

5 releases

0.6.2 Nov 8, 2023
0.6.1 Nov 8, 2023
0.6.0 Sep 23, 2023
0.5.1 Jul 31, 2023
0.5.0 Jul 30, 2023

#1330 in Text processing

Download history 20/week @ 2024-02-26 4/week @ 2024-03-04 15/week @ 2024-03-11 9/week @ 2024-03-18 5/week @ 2024-03-25 106/week @ 2024-04-01

136 downloads per month
Used in 4 crates (3 directly)

GPL-2.0 license

460KB
11K SLoC

Ellie-Tokenizer

Tokenizer is the first stage of the pipeline. It takes a text and returns a list of items and their positions.

lib.rs:

Ellie Tokenizer This is the code tokenizer for the Ellie Language.

Copyright (c) 2020 Behemehal. See license file for details

Dependencies

~3–5MB
~91K SLoC