8 stable releases
new 1.4.0 | Nov 9, 2024 |
---|---|
1.3.2 | Nov 12, 2021 |
1.2.0 | Aug 2, 2021 |
1.1.2 | Jul 20, 2021 |
#240 in Text processing
1,219 downloads per month
Used in nlpo3-cli
84KB
1.5K
SLoC
--- SPDX-FileCopyrightText: 2024 PyThaiNLP Project SPDX-License-Identifier: Apache-2.0
nlpO3
Thai natural language processing library in Rust, with Python and Node bindings. Formerly oxidized-thainlp.
cargo install nlpo3
pip install nlpo3
Table of contents
Features
- Thai word tokenizer
- Use maximal-matching dictionary-based tokenization algorithm
and honor Thai Character Cluster boundaries
- 2.5x faster than similar pure Python implementation (PyThaiNLP's newmm)
- Load a dictionary from a plain text file (one word per line)
or from
Vec<String>
- Use maximal-matching dictionary-based tokenization algorithm
and honor Thai Character Cluster boundaries
Dictionary file
- For the interest of library size, nlpO3 does not assume what dictionary the user would like to use, and it does not come with a dictionary.
- A dictionary is needed for the dictionary-based word tokenizer.
- For tokenization dictionary, try
- words_th.tx from PyThaiNLP
- ~62,000 words
- CC0-1.0
- word break dictionary from libthai
- consists of dictionaries in different categories, with a make script
- LGPL-2.1
- words_th.tx from PyThaiNLP
Usage
Node.js binding
See nlpo3-nodejs.
Python binding
Example:
from nlpo3 import load_dict, segment
load_dict("path/to/dict.file", "dict_name")
segment("สวัสดีครับ", "dict_name")
See more at nlpo3-python.
Rust library
Install
cargo install nlpo3
In Cargo.toml
:
[dependencies]
# ...
nlpo3 = "1.4.0"
Example
Create a tokenizer using a dictionary from file, then use it to tokenize a string (safe mode = true, and parallel mode = false):
use nlpo3::tokenizer::newmm::NewmmTokenizer;
use nlpo3::tokenizer::tokenizer_trait::Tokenizer;
let tokenizer = NewmmTokenizer::new("path/to/dict.file");
let tokens = tokenizer.segment("ห้องสมุดประชาชน", true, false).unwrap();
Create a tokenizer using a dictionary from a vector of Strings:
let words = vec!["ปาลิเมนต์".to_string(), "คอนสติติวชั่น".to_string()];
let tokenizer = NewmmTokenizer::from_word_list(words);
Add words to an existing tokenizer:
tokenizer.add_word(&["มิวเซียม"]);
Remove words from an existing tokenizer:
tokenizer.remove_word(&["กระเพรา", "ชานชลา"]);
Command-line interface
Example:
echo "ฉันกินข้าว" | nlpo3 segment
See more at nlpo3-cli.
Build
Requirements
Steps
Generic test:
cargo test
Build API document and open it to check:
cargo doc --open
Build (remove --release
to keep debug information):
cargo build --release
Check target/
for build artifacts.
Development
Development document:
Issues:
- Please report issues at https://github.com/PyThaiNLP/nlpo3/issues
License
nlpO3 is copyrighted by its authors and licensed under terms of the Apache Software License 2.0 (Apache-2.0) - see file LICENSE for details.
Dependencies
~3.5–5MB
~84K SLoC