#unicode-characters #unicode #security #confusable #homoglyphs #moderation

decancer

A library that removes common unicode confusables/homoglyphs from strings

28 stable releases

3.2.8 Oct 29, 2024
3.2.3 Jul 1, 2024
3.1.2 Mar 30, 2024
2.0.2 Feb 11, 2024
1.4.1 Jul 16, 2022

#80 in Text processing

Download history 196/week @ 2024-08-17 332/week @ 2024-08-24 245/week @ 2024-08-31 216/week @ 2024-09-07 150/week @ 2024-09-14 266/week @ 2024-09-21 244/week @ 2024-09-28 279/week @ 2024-10-05 176/week @ 2024-10-12 116/week @ 2024-10-19 224/week @ 2024-10-26 269/week @ 2024-11-02 711/week @ 2024-11-09 641/week @ 2024-11-16 1581/week @ 2024-11-23 1018/week @ 2024-11-30

4,093 downloads per month
Used in 12 crates (4 directly)

MIT license

90KB
2.5K SLoC

decancer npm crates.io npm downloads crates.io downloads codacy ko-fi

A library that removes common unicode confusables/homoglyphs from strings.

  • Its core is written in Rust and utilizes a form of Binary Search to ensure speed!
  • By default, it's capable of filtering 221,529 (19.88%) different unicode codepoints like:
  • Unlike other packages, this package is unicode bidi-aware where it also interprets right-to-left characters in the same way as it were to be rendered by an application!
  • Its behavior is also highly customizable to your liking!

Installation

In your Cargo.toml:

decancer = "3.2.8"

Examples

For more information, please read the documentation.

let mut cured = decancer::cure!(r"vEⓡ𝔂 𝔽𝕌Ňℕy ţ乇𝕏𝓣 wWiIiIIttHh l133t5p3/-\|<").unwrap();

assert_eq!(cured, "very funny text with leetspeak");

// WARNING: it's NOT recommended to coerce this output to a Rust string
//          and process it manually from there, as decancer has its own
//          custom comparison measures, including leetspeak matching!
assert_ne!(cured.as_str(), "very funny text with leetspeak");

assert!(cured.contains("funny"));

cured.censor("funny", '*');
assert_eq!(cured, "very ***** text with leetspeak");

cured.censor_multiple(["very", "text"], '-');
assert_eq!(cured, "---- ***** ---- with leetspeak");

Donations

If you want to support my eyes for manually looking at thousands of unicode characters, consider donating! ❤

ko-fi

Dependencies

~0–790KB
~14K SLoC