#forth #tokenizer

bin+lib rust-forth-tokenizer

A Forth tokenizer written in Rust

9 releases

✓ Uses Rust 2018 edition

0.2.0 Oct 5, 2019
0.1.2 Sep 26, 2019
0.0.6 Sep 7, 2019
0.0.1 Aug 28, 2019

#290 in Embedded development

Download history 19/week @ 2019-10-29 11/week @ 2019-11-05 3/week @ 2019-11-12 27/week @ 2019-11-19 18/week @ 2019-11-26 27/week @ 2019-12-03 61/week @ 2019-12-10 1/week @ 2019-12-17 9/week @ 2019-12-24 10/week @ 2020-01-07 32/week @ 2020-01-14 1/week @ 2020-01-21 1/week @ 2020-01-28 2/week @ 2020-02-04

128 downloads per month
Used in 2 crates (via rust-forth-compiler)

MIT license

445KB
13K SLoC

Forth 12K SLoC // 0.2% comments Rust 354 SLoC // 0.1% comments

rust-forth-tokenizer

A Forth tokenizer written in Rust

Usage:

main() { use rust_forth_tokenizer::ForthToken; use rust_forth_tokenizer::ForthTokenizer;

 let tokenizer = ForthTokenizer::new("word : wordname 1 2 3 ; definition");
 // The code also supports the regular for loop iterator syntax
 let collected: Vec<_> = tokenizer.into_iter().collect();
 assert_eq!(
        &collected,
        &vec![
            ForthToken::Command("word"),
            ForthToken::Colon,
            ForthToken::Command("wordname"),
            ForthToken::Number(1),
            ForthToken::Number(2),
            ForthToken::Number(3),
            ForthToken::SemiColon,
            ForthToken::Command("definition"),
        ]
    );    

}

No runtime deps