#unidecode #emoji #unicode #ascii #transliteration

deunicode

Convert Unicode strings to pure ASCII by intelligently transliterating them. Suppors Emoji and Chinese.

11 releases (7 stable)

new 1.3.1 Jul 24, 2021
1.2.1 Apr 12, 2021
1.2.0 Mar 24, 2021
1.1.1 Apr 23, 2020
0.4.0 May 5, 2018

#19 in Text processing

Download history 26333/week @ 2021-04-08 28406/week @ 2021-04-15 27064/week @ 2021-04-22 21915/week @ 2021-04-29 27320/week @ 2021-05-06 29817/week @ 2021-05-13 30600/week @ 2021-05-20 28304/week @ 2021-05-27 37244/week @ 2021-06-03 35038/week @ 2021-06-10 30952/week @ 2021-06-17 30821/week @ 2021-06-24 35742/week @ 2021-07-01 26125/week @ 2021-07-08 36623/week @ 2021-07-15 20756/week @ 2021-07-22

129,948 downloads per month
Used in 132 crates (14 directly)

BSD-3-Clause

165KB
131 lines

deunicode

Documentation

The deunicode library transliterates Unicode strings such as "Æneid" into pure ASCII ones such as "AEneid."

It started as a Rust port of Text::Unidecode Perl module, and was extended to support emoji.

This is a fork of unidecode crate. This fork uses a compact representation of Unicode data to minimize memory overhead and executable size (about 70K codepoints mapped to 240K ASCII characters, using 450KB or memory, 160KB gzipped).

Examples

extern crate deunicode;
use deunicode::deunicode;

assert_eq!(deunicode("Æneid"), "AEneid");
assert_eq!(deunicode("étude"), "etude");
assert_eq!(deunicode("北亰"), "Bei Jing");
assert_eq!(deunicode("ᔕᓇᓇ"), "shanana");
assert_eq!(deunicode("げんまい茶"), "genmaiCha");
assert_eq!(deunicode("🦄☣"), "unicorn biohazard");

Guarantees and Warnings

Here are some guarantees you have when calling deunicode():

  • The String returned will be valid ASCII; the decimal representation of every char in the string will be between 0 and 127, inclusive.
  • Every ASCII character (0x00 - 0x7F) is mapped to itself.
  • All Unicode characters will translate to printable ASCII characters (\n or characters in the range 0x20 - 0x7E).

There are, however, some things you should keep in mind:

  • Some transliterations do produce \n characters.
  • Some Unicode characters transliterate to an empty string, either on purpose or because deunicode does not know about the character.
  • Some Unicode characters are unknown and transliterate to "[?]" (or a custom placeholder, or None if you use a chars iterator).
  • Many Unicode characters transliterate to multi-character strings. For example, "北" is transliterated as "Bei".
  • Transliteration is context-free and not sophisticated enough to produce proper Chinese or Japanese. Han characters used in multiple languages are mapped to a single Mandarin pronounciation, and will be mostly illegible to Japanese readers. Transliteration can't handle cases where a single character has multiple possible pronounciations.

Unicode data

For a detailed explanation on the rationale behind the original dataset, refer to this article written by Burke in 2001.

No runtime deps