25 releases (8 major breaking)
9.0.0 | Aug 16, 2023 |
---|---|
8.1.0 | Aug 9, 2023 |
7.0.1 | Aug 7, 2023 |
6.0.5 | Aug 1, 2023 |
0.3.1 | Mar 30, 2022 |
#100 in Hardware support
92 downloads per month
120KB
3K
SLoC
jayce
jayce is a tokenizer 🌌
Example
use jayce::{duos, Tokenizer};
use regex::Regex;
const SOURCE: &str = "Excalibur = 5000$; // Your own language!";
lazy_static::lazy_static! (
static ref DUOS: Vec<(&'static str, Regex)> = duos![
"whitespace", r"^[^\S\n]+",
"comment_line", r"^//(.*)",
"comment_block", r"^/\*(.|\n)*?\*/",
"newline", r"^\n",
"price", r"^[0-9]+\$",
"semicolon", r"^;",
"operator", r"^=",
"name", r"^[a-zA-Z_]+"
];
);
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut tokenizer = Tokenizer::new(SOURCE, &DUOS);
while let Some(token) = tokenizer.next()? {
println!("{:?}", token);
}
Ok(())
}
Result
Token { kind: "name", value: "Excalibur", pos: (1, 1) }
Token { kind: "whitespace", value: " ", pos: (1, 10) }
Token { kind: "operator", value: "=", pos: (1, 11) }
Token { kind: "whitespace", value: " ", pos: (1, 12) }
Token { kind: "price", value: "5000$", pos: (1, 13) }
Token { kind: "semicolon", value: ";", pos: (1, 18) }
Token { kind: "whitespace", value: " ", pos: (1, 19) }
Token { kind: "comment_line", value: "// Your own language!", pos: (1, 20) }
Info
next
possible Result
Ok(Some(token))
Match is foundOk(None)
End of sourceErr(error)
An error occurs
tokenize_all
possible Result
Ok(Vec<Tokens>)
Tokens are foundErr(error)
An error occurs
Performances
initialization in 1.83 nanoseconds
tokenization of 29 639 tokens in 4.6 milliseconds
7.0.2
is442%
faster than version4.0.1
from making everything precompiled
9.0.0
is30%
slower than version8.1.0
to support custom whitespaces & comments
Dependencies
~2.1–3.5MB
~58K SLoC