bin+lib powr-tokenizer

converts JS/TS code into tokens. part of the power project.

1 unstable release

0.3.0 Dec 1, 2022

SSPL-1.0

20KB
611 lines

power-tokenizer

this is both a cli and a library.

tokenizes a string of characters. based on ECMAScript.

installing

assumes you have cargo/rustup installed.

cargo install power-tokenizer

running

power-tokenizer "const one_plus_one = 1 + 1"

expected output:

Const
Identifier(
    [
        'o',
        'n',
        'e',
        '_',
        'p',
        'l',
        'u',
        's',
        '_',
        'o',
        'n',
        'e',
    ],
)
Assign
Identifier(
    [
        '1',
    ],
)
Plus
Identifier(
    [
        '1',
    ],
)
EndOfFile

No runtime deps