2 releases
Uses old Rust 2015
| 0.1.1 | Jul 21, 2018 |
|---|---|
| 0.1.0 | Nov 1, 2015 |
710 downloads per month
Used in 2 crates
36KB
178 lines
rust-tinysegmenter
Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.
Install
Adding the following to the Cargo.toml in your project:
[dependencies]
tinysegmenter = "0.1"
and import using extern crate:
extern crate tinysegmenter;
Usage
let tokens = tinysegmenter::tokenize("私の名前は中野です");
println!("{}", &tokens.join("|")); // 私|の|名前|は|中野|です
License
Copyright (c) 2015 woxtu
Licensed under the MIT license.
Dependencies
~27KB