|0.2.0||Sep 1, 2023|
|0.1.1||Jun 23, 2023|
|0.1.0||Jun 9, 2023|
#147 in Database implementations
47,056 downloads per month
Used in 18 crates (4 directly)
An API to interface a tokenizer with tantivy.
The API will be kept stable in order to not break support for existing tokenizers.
Tokenizer are in charge of chopping text into a stream of tokens ready for indexing. This is an seperate crate from tantivy, so implementors don't need to update for each new tantivy version.