25 releases (8 major breaking)

9.0.0 Aug 16, 2023
8.1.0 Aug 9, 2023
7.0.1 Aug 7, 2023
6.0.5 Aug 1, 2023
0.3.1 Mar 30, 2022

#100 in Hardware support

Download history 112/week @ 2023-08-04 47/week @ 2023-08-11 9/week @ 2023-08-18 25/week @ 2023-08-25 2/week @ 2023-09-01 55/week @ 2023-09-08 7/week @ 2023-09-15 16/week @ 2023-09-22 4/week @ 2023-09-29 4/week @ 2023-10-06 6/week @ 2023-10-13 6/week @ 2023-10-20 51/week @ 2023-10-27 7/week @ 2023-11-03 4/week @ 2023-11-10 30/week @ 2023-11-17

92 downloads per month

MIT license

120KB
3K SLoC

jayce

jayce is a tokenizer 🌌

Example
use jayce::{duos, Tokenizer};
use regex::Regex;

const SOURCE: &str = "Excalibur = 5000$; // Your own language!";

lazy_static::lazy_static! (
    static ref DUOS: Vec<(&'static str, Regex)> = duos![
        "whitespace", r"^[^\S\n]+",
        "comment_line", r"^//(.*)",
        "comment_block", r"^/\*(.|\n)*?\*/",
        "newline", r"^\n",

        "price", r"^[0-9]+\$",
        "semicolon", r"^;",
        "operator", r"^=",
        "name", r"^[a-zA-Z_]+"
    ];
);

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut tokenizer = Tokenizer::new(SOURCE, &DUOS);

    while let Some(token) = tokenizer.next()? {
        println!("{:?}", token);
    }

    Ok(())
}
Result
Token { kind: "name", value: "Excalibur", pos: (1, 1) }
Token { kind: "whitespace", value: " ", pos: (1, 10) }
Token { kind: "operator", value: "=", pos: (1, 11) }
Token { kind: "whitespace", value: " ", pos: (1, 12) }
Token { kind: "price", value: "5000$", pos: (1, 13) }
Token { kind: "semicolon", value: ";", pos: (1, 18) }
Token { kind: "whitespace", value: " ", pos: (1, 19) }
Token { kind: "comment_line", value: "// Your own language!", pos: (1, 20) }
Info

next possible Result

  1. Ok(Some(token)) Match is found
  2. Ok(None) End of source
  3. Err(error) An error occurs

tokenize_all possible Result

  1. Ok(Vec<Tokens>) Tokens are found
  2. Err(error) An error occurs
Performances

initialization in 1.83 nanoseconds

tokenization of 29 639 tokens in 4.6 milliseconds

7.0.2 is 442% faster than version 4.0.1 from making everything precompiled

9.0.0 is 30% slower than version 8.1.0 to support custom whitespaces & comments

Dependencies

~2.1–3.5MB
~58K SLoC