#language #language-detection #command-line-tool #text #offset #confidence #lingua-rs

app lingua-cli

Lingua-cli is a command line tool for language classification, using the lingua-rs library

3 releases (breaking)

0.4.0 Jun 10, 2024
0.3.0 Jun 4, 2024
0.2.1 Jun 3, 2024

#161 in Command line utilities

Download history 246/week @ 2024-06-02 224/week @ 2024-06-09 25/week @ 2024-06-16 1/week @ 2024-06-23

496 downloads per month

GPL-3.0-only

27KB
243 lines

Crate GitHub release Project Status: Active – The project has reached a stable, usable state and is being actively developed. Technology Readiness Level 7/9 - Release Candidate - Technology ready enough and in initial use by end-users in intended scholarly environments. Further validation in progress.

Lingua-cli

This is a small command-line tool for language detection, it is a simple wrapper around the lingua-rs library for Rust, read there for extensive documentation. A distinguishing feature is that this library works better for short texts thanmany other libraries

Installation

Ensure you have Rust's package manager cargo, then download, isntall and compile lingua-cli in one go as follows:

$ cargo install lingua-cli

Usage

Pass text as parameter

$ lingua-cli bonjour à tous

Pass text via standard input:

$ echo "bonjour à tous" | lingua-cli

Constrain the languages you want to detect using -l with iso-639-1 languages codes. Constraining the list improves accuracy. Do -L to see a list of supported languages.

$ echo "bonjour à tous" | lingua-cli -l "fr,de,es,nl,en"

To classify input line-by-line, pass -n.

$ echo -e "bonjour à tous\nhola a todos\nhallo allemaal" | lingua-cli -n -l "fr,de,es,nl,en"

fr      0.9069164472389637      bonjour à tous
es      0.918273871035807       hola a todos
nl      0.988293648761749       hallo allemaal

Output is TSV and consists of an iso-639-1 language code, confidence score, and in line-by-line mode, a copy of the line.

You can also classified mixed text using the --multi option. This will then output UTF-8 byte offsets:

$ lingua-cli --multi -l fr,de,en < /tmp/test.txt
0       23      fr      Parlez-vous français? 
23      73      de      Ich spreche ein bisschen spreche Französisch ja. 
73      110     en      A little bit is better than nothing.

Dependencies

~17MB
~426K SLoC