Lib.rs

› Internationalization (i18n) | Text processing
#nlp #language-detection #language #scripting-language #text

whatlang

Fast and lightweight language identification library for Rust

by Serhii Potapov and 19 contributors

33 releases

0.16.4 Jan 4, 2024
0.16.3 Aug 16, 2023
0.16.2 Oct 23, 2022
0.16.0 May 7, 2022
0.1.1 Dec 15, 2016

#7 in Internationalization (i18n)

Download history 5068/week @ 2024-07-27 3553/week @ 2024-08-03 4506/week @ 2024-08-10 3466/week @ 2024-08-17 4324/week @ 2024-08-24 4545/week @ 2024-08-31 3960/week @ 2024-09-07 3421/week @ 2024-09-14 4034/week @ 2024-09-21 4218/week @ 2024-09-28 4116/week @ 2024-10-05 3885/week @ 2024-10-12 4385/week @ 2024-10-19 4071/week @ 2024-10-26 5059/week @ 2024-11-02 4180/week @ 2024-11-09

18,291 downloads per month
Used in 33 crates (17 directly)

MIT license

670KB
18K SLoC

Whatlang - rust library for natural language detection

Whatlang

Natural language detection for Rust with focus on simplicity and performance.

Try online demo.

Build Status License Documentation

Stand With Ukraine

Content

Features

  • Supports 69 languages
  • 100% written in Rust
  • Lightweight, fast and simple
  • Recognizes not only a language, but also a script (Latin, Cyrillic, etc)
  • Provides reliability information

Get started

Example:

use whatlang::{detect, Lang, Script};

fn main() {
    let text = "Ĉu vi ne volas eklerni Esperanton? Bonvolu! Estas unu de la plej bonaj aferoj!";

    let info = detect(text).unwrap();
    assert_eq!(info.lang(), Lang::Epo);
    assert_eq!(info.script(), Script::Latin);
    assert_eq!(info.confidence(), 1.0);
    assert!(info.is_reliable());
}

For more details (e.g. how to blacklist some languages) please check the documentation.

Who uses Whatlang?

Whatlang is used within the following big projects as direct or indirect dependency for language recognition. You're gonna be in a great company using Whatlang:

  • Sonic - fast, lightweight and schema-less search backend in Rust.
  • Meilisearch - an open-source, easy-to-use, blazingly fast, and hyper-relevant search engine built in Rust.

Feature toggles

Feature Description
enum-map Lang and Script implement Enum trait from enum-map
arbitrary Support Arbitrary
serde Implements Serialize and Deserialize for Lang and Script
dev Enables whatlang::dev module which provides some internal API.
It exists for profiling purposes and normal users are discouraged to to rely on this API.

How does it work?

How does the language recognition work?

The algorithm is based on the trigram language models, which is a particular case of n-grams. To understand the idea, please check the original whitepaper Cavnar and Trenkle '94: N-Gram-Based Text Categorization'.

How is is_reliable calculated?

It is based on the following factors:

  • How many unique trigrams are in the given text
  • How big is the difference between the first and the second(not returned) detected languages? This metric is called rate in the code base.

Therefore, it can be presented as 2d space with threshold functions, that splits it into "Reliable" and "Not reliable" areas. This function is a hyperbola and it looks like the following one:

Language recognition whatlang rust

For more details, please check a blog article Introduction to Rust Whatlang Library and Natural Language Identification Algorithms.

Make tasks

  • make bench - run performance benchmarks
  • make doc - generate and open doc
  • make test - run tests
  • make watch - watch changes and run tests

Comparison with alternatives

Whatlang CLD2 CLD3
Implementation language Rust C++ C++
Languages 68 83 107
Algorithm trigrams quadgrams neural network
Supported Encoding UTF-8 UTF-8 ?
HTML support no yes ?

Ports and clones

Donations

You can support the project by donating NEAR tokens.

Our NEAR wallet address is whatlang.near

Derivation

Whatlang is a derivative work from Franc (JavaScript, MIT) by Titus Wormer.

License

MIT © Sergey Potapov

Contributors

Dependencies

~2MB
~29K SLoC