1 unstable release
0.3.0 | Dec 1, 2022 |
---|
20KB
611 lines
power-tokenizer
this is both a cli and a library.
tokenizes a string of characters. based on ECMAScript.
installing
assumes you have cargo/rustup installed.
cargo install power-tokenizer
running
power-tokenizer "const one_plus_one = 1 + 1"
expected output:
Const
Identifier(
[
'o',
'n',
'e',
'_',
'p',
'l',
'u',
's',
'_',
'o',
'n',
'e',
],
)
Assign
Identifier(
[
'1',
],
)
Plus
Identifier(
[
'1',
],
)
EndOfFile