8 stable releases

new 1.4.0 Nov 9, 2024
1.3.2 Nov 12, 2021
1.2.0 Aug 2, 2021
1.1.2 Jul 20, 2021

#240 in Text processing

Download history 1/week @ 2024-09-17 19/week @ 2024-09-24 3/week @ 2024-10-29 1216/week @ 2024-11-05

1,219 downloads per month
Used in nlpo3-cli

Apache-2.0

84KB
1.5K SLoC

--- SPDX-FileCopyrightText: 2024 PyThaiNLP Project SPDX-License-Identifier: Apache-2.0

nlpO3

crates.io

Thai natural language processing library in Rust, with Python and Node bindings. Formerly oxidized-thainlp.

cargo install nlpo3
pip install nlpo3

Table of contents

Features

  • Thai word tokenizer
    • Use maximal-matching dictionary-based tokenization algorithm and honor Thai Character Cluster boundaries
      • 2.5x faster than similar pure Python implementation (PyThaiNLP's newmm)
    • Load a dictionary from a plain text file (one word per line) or from Vec<String>

Dictionary file

  • For the interest of library size, nlpO3 does not assume what dictionary the user would like to use, and it does not come with a dictionary.
  • A dictionary is needed for the dictionary-based word tokenizer.
  • For tokenization dictionary, try

Usage

Node.js binding

See nlpo3-nodejs.

Python binding

PyPI

Example:

from nlpo3 import load_dict, segment

load_dict("path/to/dict.file", "dict_name")
segment("สวัสดีครับ", "dict_name")

See more at nlpo3-python.

Rust library

crates.io

Install

cargo install nlpo3

In Cargo.toml:

[dependencies]
# ...
nlpo3 = "1.4.0"

Example

Create a tokenizer using a dictionary from file, then use it to tokenize a string (safe mode = true, and parallel mode = false):

use nlpo3::tokenizer::newmm::NewmmTokenizer;
use nlpo3::tokenizer::tokenizer_trait::Tokenizer;

let tokenizer = NewmmTokenizer::new("path/to/dict.file");
let tokens = tokenizer.segment("ห้องสมุดประชาชน", true, false).unwrap();

Create a tokenizer using a dictionary from a vector of Strings:

let words = vec!["ปาลิเมนต์".to_string(), "คอนสติติวชั่น".to_string()];
let tokenizer = NewmmTokenizer::from_word_list(words);

Add words to an existing tokenizer:

tokenizer.add_word(&["มิวเซียม"]);

Remove words from an existing tokenizer:

tokenizer.remove_word(&["กระเพรา", "ชานชลา"]);

Command-line interface

crates.io

Example:

echo "ฉันกินข้าว" | nlpo3 segment

See more at nlpo3-cli.

Build

Requirements

Steps

Generic test:

cargo test

Build API document and open it to check:

cargo doc --open

Build (remove --release to keep debug information):

cargo build --release

Check target/ for build artifacts.

Development

Development document:

Issues:

License

nlpO3 is copyrighted by its authors and licensed under terms of the Apache Software License 2.0 (Apache-2.0) - see file LICENSE for details.

Dependencies

~3.5–5MB
~84K SLoC