#unicode #unicode-characters #ascii #emoji #transliteration #unidecode

no-std deunicode

Convert Unicode strings to pure ASCII by intelligently transliterating them. Suppors Emoji and Chinese.

10 stable releases

1.4.3 Feb 16, 2024
1.4.2 Dec 10, 2023
1.4.1 Oct 15, 2023
1.3.3 Dec 15, 2022
0.4.0 May 5, 2018

#19 in Text processing

Download history 148005/week @ 2023-11-26 136583/week @ 2023-12-03 143498/week @ 2023-12-10 126491/week @ 2023-12-17 70933/week @ 2023-12-24 104383/week @ 2023-12-31 127159/week @ 2024-01-07 130873/week @ 2024-01-14 143959/week @ 2024-01-21 138193/week @ 2024-01-28 132833/week @ 2024-02-04 146123/week @ 2024-02-11 141999/week @ 2024-02-18 144347/week @ 2024-02-25 152417/week @ 2024-03-03 58294/week @ 2024-03-10

507,430 downloads per month
Used in 465 crates (40 directly)

BSD-3-Clause

180KB
163 lines

deunicode

Documentation

The deunicode library transliterates Unicode strings such as "Æneid" into pure ASCII ones such as "AEneid." Includes support for emoji. It's compatible with no-std Rust environments.

This is a maintained alternative to the unidecode crate, which started as a Rust port of Text::Unidecode Perl module.

Deunicode is quite fast, and uses a compact representation of Unicode data to minimize memory overhead and executable size (about 70K codepoints mapped to 240K ASCII characters, using 450KB or memory, 160KB gzipped).

Examples

use deunicode::deunicode;

assert_eq!(deunicode("Æneid"), "AEneid");
assert_eq!(deunicode("étude"), "etude");
assert_eq!(deunicode("北亰"), "Bei Jing");
assert_eq!(deunicode("ᔕᓇᓇ"), "shanana");
assert_eq!(deunicode("げんまい茶"), "genmaiCha");
assert_eq!(deunicode("🦄☣"), "unicorn biohazard");

Guarantees and Warnings

Here are some guarantees you have when calling deunicode():

  • The String returned will be valid ASCII; the decimal representation of every char in the string will be between 0 and 127, inclusive.
  • Every ASCII character (0x00 - 0x7F) is mapped to itself.
  • All Unicode characters will translate to printable ASCII characters (\n or characters in the range 0x20 - 0x7E).

There are, however, some things you should keep in mind:

  • Some transliterations do produce \n characters.
  • Some Unicode characters transliterate to an empty string, either on purpose or because deunicode does not know about the character.
  • Some Unicode characters are unknown and transliterate to "[?]" (or a custom placeholder, or None if you use a chars iterator).
  • Many Unicode characters transliterate to multi-character strings. For example, "北" is transliterated as "Bei".
  • The transliteration is context-free, and not sophisticated enough to produce proper Chinese or Japanese. Han characters used in multiple languages are mapped to a single Mandarin pronounciation, and will be mostly illegible to Japanese readers. Transliteration can't handle cases where a single character has multiple possible pronounciations.

Unicode data

For a detailed explanation on the rationale behind the original dataset, refer to this article written by Burke in 2001.

No runtime deps

Features