#unicode #general #category #no-std

no-std unicode-general-category

Fast lookup of the Unicode General Category property for char

4 releases (breaking)

0.4.0 Mar 26, 2021
0.3.0 Nov 18, 2020
0.2.0 Dec 13, 2019
0.1.0 Dec 13, 2019

#95 in Text processing

Download history 5847/week @ 2021-08-10 5800/week @ 2021-08-17 5133/week @ 2021-08-24 5124/week @ 2021-08-31 6367/week @ 2021-09-07 6046/week @ 2021-09-14 5717/week @ 2021-09-21 5986/week @ 2021-09-28 7018/week @ 2021-10-05 4925/week @ 2021-10-12 5241/week @ 2021-10-19 5694/week @ 2021-10-26 5459/week @ 2021-11-02 4758/week @ 2021-11-09 5576/week @ 2021-11-16 3769/week @ 2021-11-23

20,429 downloads per month
Used in 46 crates (5 directly)

Apache-2.0

210KB
4K SLoC

unicode-general-category


Fast lookup of the Unicode General Category property for char in Rust using Unicode 13.0 data. This crate is no-std compatible.

Usage

use unicode_general_category::{get_general_category, GeneralCategory};

fn main() {
    assert_eq!(get_general_category('A'), GeneralCategory::UppercaseLetter);
}

Performance & Implementation Notes

ucd-generate is used to generate tables.rs. A build script (build.rs) compiles this into a two level look up table. The look up time is constant as it is just indexing into two arrays.

The two level approach maps a code point to a block, then to a position within a block. The allows the second level of block to be deduplicated, saving space. The code is parameterised over the block size, which must be a power of 2. The value in the build script is optimal for the data set.

This approach trades off some space for faster lookups. The tables take up about 45KiB. Benchmarks showed this approach to be ~5–10× faster than the typical binary search approach.

It's possible there are further optimisations that could be made to eliminate some runs of repeated values in the first level array.

No runtime deps

%4]`