#unidecode #emoji #unicode #ascii #transliteration

deunicode

Convert Unicode strings to pure ASCII by intelligently transliterating them. Suppors Emoji and Chinese.

9 stable releases

1.3.3 Dec 15, 2022
1.3.2 Aug 21, 2022
1.3.1 Jul 24, 2021
1.2.0 Mar 24, 2021
0.4.0 May 5, 2018

#21 in Text processing

Download history 50150/week @ 2022-11-29 48516/week @ 2022-12-06 47868/week @ 2022-12-13 48508/week @ 2022-12-20 32358/week @ 2022-12-27 51817/week @ 2023-01-03 57390/week @ 2023-01-10 58838/week @ 2023-01-17 60087/week @ 2023-01-24 61871/week @ 2023-01-31 60924/week @ 2023-02-07 65446/week @ 2023-02-14 66048/week @ 2023-02-21 72190/week @ 2023-02-28 84038/week @ 2023-03-07 59403/week @ 2023-03-14

293,725 downloads per month
Used in 218 crates (28 directly)

BSD-3-Clause

175KB
146 lines

deunicode

Documentation

The deunicode library transliterates Unicode strings such as "Æneid" into pure ASCII ones such as "AEneid."

It started as a Rust port of Text::Unidecode Perl module, and was extended to support emoji.

This is a fork of unidecode crate. This fork uses a compact representation of Unicode data to minimize memory overhead and executable size (about 70K codepoints mapped to 240K ASCII characters, using 450KB or memory, 160KB gzipped).

Examples

extern crate deunicode;
use deunicode::deunicode;

assert_eq!(deunicode("Æneid"), "AEneid");
assert_eq!(deunicode("étude"), "etude");
assert_eq!(deunicode("北亰"), "Bei Jing");
assert_eq!(deunicode("ᔕᓇᓇ"), "shanana");
assert_eq!(deunicode("げんまい茶"), "genmaiCha");
assert_eq!(deunicode("🦄☣"), "unicorn biohazard");

Guarantees and Warnings

Here are some guarantees you have when calling deunicode():

  • The String returned will be valid ASCII; the decimal representation of every char in the string will be between 0 and 127, inclusive.
  • Every ASCII character (0x00 - 0x7F) is mapped to itself.
  • All Unicode characters will translate to printable ASCII characters (\n or characters in the range 0x20 - 0x7E).

There are, however, some things you should keep in mind:

  • Some transliterations do produce \n characters.
  • Some Unicode characters transliterate to an empty string, either on purpose or because deunicode does not know about the character.
  • Some Unicode characters are unknown and transliterate to "[?]" (or a custom placeholder, or None if you use a chars iterator).
  • Many Unicode characters transliterate to multi-character strings. For example, "北" is transliterated as "Bei".
  • Transliteration is context-free and not sophisticated enough to produce proper Chinese or Japanese. Han characters used in multiple languages are mapped to a single Mandarin pronounciation, and will be mostly illegible to Japanese readers. Transliteration can't handle cases where a single character has multiple possible pronounciations.

Unicode data

For a detailed explanation on the rationale behind the original dataset, refer to this article written by Burke in 2001.

No runtime deps

Features

  • alloc