#line-column #tokenizer #column #generic #line

generic_tokenizer

A generic tokenizer that tracks line and column numbers as it goes

1 unstable release

0.1.0 Oct 27, 2024

#23 in #line-column

Download history 68/week @ 2024-10-21 61/week @ 2024-10-28 13/week @ 2024-11-04

142 downloads per month
Used in 2 crates (via toy-json-formatter)

MIT license

11KB
291 lines

Generic Tokenizer

This crate provides a generic tokenizer that includes line and column numbers. This is useful for things like higher-level tokenizers, parsers, and language servers that need to track line and column numbers as they go.

No runtime deps