#forth #tokenizer

bin+lib rust-forth-tokenizer

A Forth tokenizer written in Rust

7 releases

✓ Uses Rust 2018 edition

new 0.1.1 Sep 19, 2019
0.1.0 Sep 10, 2019
0.0.6 Sep 7, 2019
0.0.1 Aug 28, 2019

#178 in Embedded development

44 downloads per month
Used in 2 crates (1 directly)

MIT license

465KB
13K SLoC

Forth 12K SLoC // 0.2% comments Rust 354 SLoC // 0.1% comments

rust-forth-tokenizer

A Forth tokenizer written in Rust

Usage:

main() { use rust_forth_tokenizer::ForthToken; use rust_forth_tokenizer::ForthTokenizer;

 let tokenizer = ForthTokenizer::new("word : wordname 1 2 3 ; definition");
 // The code also supports the regular for loop iterator syntax
 let collected: Vec<_> = tokenizer.into_iter().collect();
 assert_eq!(
        &collected,
        &vec![
            ForthToken::Command("word"),
            ForthToken::Colon,
            ForthToken::Command("wordname"),
            ForthToken::Number(1),
            ForthToken::Number(2),
            ForthToken::Number(3),
            ForthToken::SemiColon,
            ForthToken::Command("definition"),
        ]
    );    

}

No runtime deps