#line-column #tokenizer #column #generic #line

generic_tokenizer

A generic tokenizer that tracks line and column numbers as it goes

1 unstable release

0.1.0 Oct 27, 2024

#26 in #line-column


Used in 2 crates (via toy-json-formatter)

MIT license

11KB
291 lines

Generic Tokenizer

This crate provides a generic tokenizer that includes line and column numbers. This is useful for things like higher-level tokenizers, parsers, and language servers that need to track line and column numbers as they go.

No runtime deps