#language #nlp

bin+lib whichlang

A blazingly fast and lightweight language detection library for Rust

1 unstable release

0.1.0 May 10, 2023

#201 in Internationalization (i18n)

Download history 676/week @ 2024-02-26 775/week @ 2024-03-04 678/week @ 2024-03-11 869/week @ 2024-03-18 829/week @ 2024-03-25 1062/week @ 2024-04-01 482/week @ 2024-04-08 734/week @ 2024-04-15 700/week @ 2024-04-22 468/week @ 2024-04-29 543/week @ 2024-05-06 515/week @ 2024-05-13 371/week @ 2024-05-20 561/week @ 2024-05-27 333/week @ 2024-06-03 509/week @ 2024-06-10

1,786 downloads per month
Used in 2 crates

MIT license

745KB
4.5K SLoC

Whichlang

This is a language detection library, aiming for both precision and performance.

Features

  • No dependency
  • Throughput above 100 MB/s for short and long strings.
  • Good accuracy (99.5% on my validation dataset, but it really depends on the size of your input.)

How does it work?

It uses a multiclass logistic regression model over:

  • 2, 3, 4-grams of letters on ASCII
  • codepoint / 128
  • a slightly smarter projection of codepoints over a given class.

We use the hashing trick and project these features over a space of size 4_096.

The logistic regression is trained in the python notebook attached, and used to generate weight.rs.

No runtime deps