#compiler #lexer #dfa #tokenizer #flex #generate

bin+lib rust-lexer

A compiler that generates a Lexer using DFAs (inspired by flex)

2 unstable releases

0.2.0 Jul 9, 2023
0.1.0 Jul 4, 2023

#1058 in Development tools

MIT license

56KB
1.5K SLoC

rflex

This is a compiler which takes in a .tk regex specification and outputs a tokenizer. The usage is rflex [input_file] [output_file]. See example.tk and tokenizer.rs for an example input and output. Much of the design was based on the textbook: Compilers, Principles, Designs and Tools.

TODO

Make generator break when it encounters the dead state. Make generator handle variable types! It's really annoying otherwise.

No runtime deps