|0.1.0||Jan 12, 2023|
Used in summavy
An API to interface a tokenizer with tantivy.
The API will be kept stable in order to not break support for existing tokenizers.
Tokenizer are in charge of chopping text into a stream of tokens ready for indexing. This is an seperate crate from tantivy, so implementors don't need to update for each new tantivy version.