Language Pack - Scanner

Language Pack - Scanner, is scanning/lexing Rust library, useful for interpreter/compiler applications. Library is intended to be highly customizable and easy to learn, it recognizes every character aviable on the keyboard and keywords you can configure, and returns an array of tokens, you can use for building AST.

installation

Run the following Cargo command in your project directory:

cargo add lp-pack-scanner

Or add the following line to your Cargo.toml:

lp-pack-scanner = "0.4.0"

Example

use std::collections::HashMap;
use lp_pack_scanner::{Config, Scanner, Token};

fn main() {
    let tokens = Scanner::scan(Config {
        input: "let x = 5".to_string(),
        keywords: HashMap::from([("let", lp_pack_scanner::Token::Variable)]),
        allow_unknown: Some(true),
        errors: None,
        comments: None,
        benchmark: false,
    });
    println!("{:?}", tokens);
}

/*
[OutputToken { token: Variable, literal: None, lexeme: "let", line: 1 }, OutputToken { token: Identifier, literal: None, lexeme: "x", line: 1 }, OutputToken { token: Equal, literal: None, lexeme: "=", line: 1 }, OutputToken { token: Number, literal: Number(5.0), lexeme: "5", line: 1 }, OutputToken { token: EndOfFile, literal: None, lexeme: "", line: 1 }]
*/

Documentation

soon