#lexer #ruleset #generation #rules #framework #processing #tokens

tuckey

An easy-to-use, rusty lexer/parser generation framework

8 releases

0.1.7 Mar 25, 2024
0.1.6 Jan 31, 2024
0.1.1 Dec 18, 2023

#66 in Parser tooling

Download history 4/week @ 2024-01-24 15/week @ 2024-01-31 5/week @ 2024-02-21 3/week @ 2024-02-28 100/week @ 2024-03-20 23/week @ 2024-03-27 15/week @ 2024-04-03 96/week @ 2024-04-10 7/week @ 2024-04-17

121 downloads per month

MIT license

47KB
1.5K SLoC

Tuckey

An evolution of tuck5's basic architecture as a more traditional lexer-parser generator (removing the tags and iterative processing in favor of something closer to traditional pattern matching).

You are meant to interact with Tuckey through the meta system - you provide a string from which Tuckey builds a set of rules which is used to process some characters into tokens. So:

                        |  Text  |
Rule Text => Ruleset => |   \/   |
                        | Tokens |

For example, to process only positive integers without leading zeroes, you can use these rules:

nonzero = 1..9
digit = 0..9
posInt = nonzero:first & digit*

Which transforms the string 120 into these tokens:

{
    first:
    "1",
    {
        "2",
        "3",
        "4",
    }
}

From there, the token containing the first digit can be accessed with token.get_first_child("first").unwrap().

TODO: Add meta syntax guide

Dependencies

~0.4–0.9MB
~20K SLoC