2 releases

Uses old Rust 2015

0.1.1 Jul 21, 2018
0.1.0 Nov 1, 2015

#200 in Internationalization (i18n)

Download history 285/week @ 2024-03-25 446/week @ 2024-04-01 368/week @ 2024-04-08 388/week @ 2024-04-15 286/week @ 2024-04-22 373/week @ 2024-04-29 324/week @ 2024-05-06 487/week @ 2024-05-13 320/week @ 2024-05-20 222/week @ 2024-05-27 362/week @ 2024-06-03 277/week @ 2024-06-10 325/week @ 2024-06-17 235/week @ 2024-06-24 276/week @ 2024-07-01 441/week @ 2024-07-08

1,313 downloads per month
Used in 2 crates

MIT license

36KB
178 lines

rust-tinysegmenter

Crates.io

Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.

Install

Adding the following to the Cargo.toml in your project:

[dependencies]
tinysegmenter = "0.1"

and import using extern crate:

extern crate tinysegmenter;

Usage

let tokens = tinysegmenter::tokenize("私の名前は中野です");
println!("{}", &tokens.join("|")); // 私|の|名前|は|中野|です

License

Copyright (c) 2015 woxtu

Licensed under the MIT license.

Dependencies

~22KB