Alkale

This is the repository for Alkale, a Rust library to assist in making hand-written LL(1) tokenizers.

Goals

Alkale has three specific goals in mind for its design.

Goal 1: Handle Sources

Alkale should natively handle common code sources, strings and files in particular.

General-purpose parsers usually need to either operate on files' bytes alone, or read entire files into memory, neither of which are ideal. Because Alkale doesn't need to support extensive lookahead, it can directly read characters from file buffers and treat them the same as if a regular string was being tokenized.

Goal 2: Provide Span Information

Span information is annoying to keep track of manually, so Alkale will automatically keeps track of spans for its tokens.

Due to the avoidance of in-memory source loading, Alkale's spans store index, line, and column information. This may lead to higher-than-average memory usage for non-iterator tokenizers. An iterator-like tokenizer that creates tokens as they're needed will avoid this problem.

Goal 3: Include Many Built-Ins

Many aspects of tokenizers are extremely common and repetitive. Think things such as string parsing, number tokenization, error recovery, etc. These common elements should come pre-packaged with Alkale by default. You may find a list of these in COMMON.md.

Because I have roots in esolangs, there may be some odd built-ins to assist with non-standard languages.

Structure

The core of Alkale operates on the TokenizerContext type. It is created using a BufReader<File> (for convience), or more generally, any type that implements IntoIterator<Item = char>.

The TokenizerContext provides LL(1) access into the underlying string with the next and peek methods, as well as tons of helper methods. Other methods range from peek_is, a general-purpose method to check if the next character is equal to some characters— all the way to try_parse_simple_string, which attempts to parse an entire rust-like string with character escaping and everything.