#tantivy #bridge #adapter #tokenizer #jieba #jieba-rs

tantivy-jieba

A library that bridges between tantivy and jieba-rs

12 releases (breaking)

0.11.0 Apr 28, 2024
0.10.0 Oct 13, 2023
0.9.0 Jul 3, 2023
0.7.0 Jan 5, 2023
0.1.1 Feb 13, 2019

#190 in Database interfaces

Download history 29/week @ 2024-02-21 35/week @ 2024-02-28 12/week @ 2024-03-06 20/week @ 2024-03-13 1266/week @ 2024-03-20 2524/week @ 2024-03-27 1448/week @ 2024-04-03 1524/week @ 2024-04-10 1576/week @ 2024-04-17 2120/week @ 2024-04-24 1024/week @ 2024-05-01 3068/week @ 2024-05-08 2224/week @ 2024-05-15 2167/week @ 2024-05-22 1929/week @ 2024-05-29 1421/week @ 2024-06-05

8,311 downloads per month

MIT license

8KB
104 lines

tantivy-jieba

Crates.io version docs.rs Changelog FOSSA Status

An adapter that bridges between tantivy and jieba-rs.

Usage

Add dependency tantivy-jieba to your Cargo.toml.

Example

use tantivy::tokenizer::*;
let mut tokenizer = tantivy_jieba::JiebaTokenizer {};
let mut token_stream = tokenizer.token_stream("测试");
assert_eq!(token_stream.next().unwrap().text, "测试");
assert!(token_stream.next().is_none());

Register tantivy tokenizer

use tantivy::schema::Schema;
use tantivy::tokenizer::*;
use tantivy::Index;
let tokenizer = tantivy_jieba::JiebaTokenizer {};
let index = Index::create_in_ram(schema);
index.tokenizers()
     .register("jieba", tokenizer);

License

FOSSA Status

Dependencies

~9.5MB
~95K SLoC