#rustc #lexer #script #commit #at

rustc-ap-rustc_lexer

Automatically published version of the package rustc_lexer in the rust-lang/rust repository from commit 9a27044f42ace9eb652781b53f598e25d4e7e918 The publishing script for this crate lives at: https://github.com/alexcrichton/rustc-auto-publish

192 major breaking releases

727.0.0 Jul 6, 2021
726.0.0 Jun 29, 2021
725.0.0 Jun 22, 2021
724.0.0 Jun 15, 2021
535.0.0 Jul 22, 2019
Download history 11575/week @ 2021-04-07 10426/week @ 2021-04-14 11257/week @ 2021-04-21 11416/week @ 2021-04-28 10398/week @ 2021-05-05 10974/week @ 2021-05-12 10534/week @ 2021-05-19 11319/week @ 2021-05-26 11445/week @ 2021-06-02 11898/week @ 2021-06-09 12178/week @ 2021-06-16 11561/week @ 2021-06-23 13224/week @ 2021-06-30 13594/week @ 2021-07-07 12971/week @ 2021-07-14 12321/week @ 2021-07-21

48,295 downloads per month
Used in 60 crates (6 directly)

MIT/Apache

63KB
1.5K SLoC


lib.rs:

Low-level Rust lexer.

The idea with rustc_lexer is to make a reusable library, by separating out pure lexing and rustc-specific concerns, like spans, error reporting, and interning. So, rustc_lexer operates directly on &str, produces simple tokens which are a pair of type-tag and a bit of original text, and does not report errors, instead storing them as flags on the token.

Tokens produced by this lexer are not yet ready for parsing the Rust syntax. For that see rustc_parse::lexer, which converts this basic token stream into wide tokens used by actual parser.

The purpose of this crate is to convert raw sources into a labeled sequence of well-known token types, so building an actual Rust token stream will be easier.

The main entity of this crate is the [TokenKind] enum which represents common lexeme types.

Dependencies