Lib.rs

›
#tantivy #tokenizer #japanese

tantivy-tokenizer-tiny-segmenter

A Japanese tokenizer for Tantivy, based on TinySegmenter

by Daisuke Aritomo

3 releases (breaking)

0.3.0 Nov 7, 2019
0.2.0 Mar 20, 2019
0.1.0 Feb 14, 2019

#43 in #tantivy

21 downloads per month

MIT license

15KB
57 lines

tantivy-tokenizer-tiny-segmenter

A Japanese tokenizer for Tantivy based on TinySegmenter. Compatible with Tantivy 0.10.

See examples/basic.rs for basic usage.

Dependencies

~19–30MB
~415K SLoC