2 releases

Uses old Rust 2015

0.1.1 Jul 21, 2018
0.1.0 Nov 1, 2015

#229 in Internationalization (i18n)

Download history 745/week @ 2024-07-28 314/week @ 2024-08-04 448/week @ 2024-08-11 409/week @ 2024-08-18 226/week @ 2024-08-25 315/week @ 2024-09-01 212/week @ 2024-09-08 237/week @ 2024-09-15 314/week @ 2024-09-22 291/week @ 2024-09-29 305/week @ 2024-10-06 477/week @ 2024-10-13 310/week @ 2024-10-20 433/week @ 2024-10-27 283/week @ 2024-11-03 401/week @ 2024-11-10

1,454 downloads per month
Used in 2 crates

MIT license

36KB
178 lines

rust-tinysegmenter

Crates.io

Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.

Install

Adding the following to the Cargo.toml in your project:

[dependencies]
tinysegmenter = "0.1"

and import using extern crate:

extern crate tinysegmenter;

Usage

let tokens = tinysegmenter::tokenize("私の名前は中野です");
println!("{}", &tokens.join("|")); // 私|の|名前|は|中野|です

License

Copyright (c) 2015 woxtu

Licensed under the MIT license.

Dependencies

~22KB