6 releases
| 0.7.3 | Jul 3, 2024 |
|---|---|
| 0.6.2 | Nov 8, 2023 |
| 0.6.0 | Sep 23, 2023 |
| 0.5.1 | Jul 31, 2023 |
#695 in Development tools
Used in 4 crates
(3 directly)
460KB
11K
SLoC
Ellie-Tokenizer
Tokenizer is the first stage of the pipeline. It takes a text and returns a list of items and their positions.
lib.rs:
Ellie Tokenizer This is the code tokenizer for the Ellie Language.
Copyright (c) 2020 Behemehal. See license file for details
Dependencies
~3–5MB
~95K SLoC