#lexer #tokenizer #parser #char #lexing

char-lex

Create easy enum based lexers

11 releases (6 stable)

1.0.5 May 1, 2020
1.0.4 Apr 30, 2020
0.2.0 Apr 29, 2020
0.1.3 Apr 29, 2020

#690 in Text processing

26 downloads per month

MIT/Apache

9KB
179 lines

Discontinued

Use as you please...

CHAR-LEX

Char-Lex is a crate for easely creating a char based lexer from multiple custom enums!

Example

use char_lex::prelude::*;

#[token]
#[derive(Debug, PartialEq)]
enum Token {
    Whitespace = [' ', '\t', '\r', '\n'],

    Digit(Digit),
}

#[token]
#[derive(Debug, PartialEq)]
enum Digit {
    Zero = '0',
    One = '1',
    Two = '2',
    Three = '3',
    Four = '4',
    Five = '5',
    Six = '6',
    Seven = '7',
    Eight = '8',
    Nine = '9',
}

fn main() {
    let lexer: Lexer<Token, Token> = Lexer::new("189");
    let tokens: Vec<Token> = lexer.collect();

    assert_eq!(
        vec![
            Token::Digit(Digit::One),
            Token::Digit(Digit::Eight),
            Token::Digit(Digit::Nine)
        ],
        tokens
    );
}

Tokens with the TokenTrait can also be wrapped in anything that implements the TokenWrapper<T> trait!

Example

#[derive(Debug, PartialEq)]
struct Wrapper {
    token: Token,
    character: char,
}

impl TokenWrapper<Token> for Wrapper {
    fn wrap(token: Token, context: Context) -> Self {
        Self {
            token,
            character: context.character,
        }
    }
}

fn main() {
    let lexer: Lexer<Token, Wrapper> = Lexer::new("1");
    let tokens: Vec<Wrapper> = lexer.collect();

    assert_eq!(
        vec![Wrapper {
            token: Token::Digit(Digit::One),
            character: '1'
        }],
        tokens
    );
}

Dependencies

~92KB