#tokenizer #transform #operator #string #vector #string-code

code-tokenizer

A string/code tokenizer. Transforms a string with a vector of operators into a vector of keywords.

1 unstable release

0.1.0 Feb 6, 2020

#121 in #operator

MIT license

5KB
75 lines

Code Tokenizer

A string/code tokenizer. Transforms a string with a vector of operators into a vector of keywords.

No runtime deps