#unicode #grapheme #codepoint #text #reader

unicode_reader

Adaptors which wrap byte-oriented readers and yield the UTF-8 data as Unicode code points or grapheme clusters

5 releases (3 stable)

1.0.2 Nov 19, 2021
1.0.1 Jun 24, 2020
1.0.0 Jun 14, 2019
0.1.1 Feb 7, 2018
0.1.0 Sep 20, 2016

#208 in Text processing

Download history 102/week @ 2021-10-01 230/week @ 2021-10-08 149/week @ 2021-10-15 247/week @ 2021-10-22 374/week @ 2021-10-29 409/week @ 2021-11-05 414/week @ 2021-11-12 502/week @ 2021-11-19 643/week @ 2021-11-26 325/week @ 2021-12-03 232/week @ 2021-12-10 137/week @ 2021-12-17 40/week @ 2021-12-24 127/week @ 2021-12-31 120/week @ 2022-01-07 209/week @ 2022-01-14

532 downloads per month
Used in 13 crates (9 directly)

MIT/Apache

20KB
243 lines

unicode_reader

Build Status

Documentation

Adaptors which wrap byte-oriented readers and yield the UTF-8 data as Unicode code points or grapheme clusters.

Unlike other Unicode parsers which work on strings (for instance, unicode_segmentation, upon which this is built), this crate works on streams and doesn't require reading the entire data into memory. Instead it yields the graphemes or code points as it reads them.

extern crate unicode_reader;
use unicode_reader::Graphemes;

use std::io::Cursor;

fn main() {
    let input = Cursor::new("He\u{302}\u{320}llo");
    let mut graphemes = Graphemes::from(input);
    assert_eq!("H",                 graphemes.next().unwrap().unwrap());
    assert_eq!("e\u{302}\u{320}",   graphemes.next().unwrap().unwrap()); // note 3 characters
    assert_eq!("l",                 graphemes.next().unwrap().unwrap());
    assert_eq!("l",                 graphemes.next().unwrap().unwrap());
    assert_eq!("o",                 graphemes.next().unwrap().unwrap());
    assert!(graphemes.next().is_none());

    let greek_bytes = vec![0xCE, 0xA7, 0xCE, 0xB1, 0xCE, 0xAF, 0xCF, 0x81, 0xCE, 0xB5,
                           0xCF, 0x84, 0xCE, 0xB5];
    let mut codepoints = CodePoints::from(Cursor::new(greek_bytes));
    assert_eq!(vec!['Χ', 'α', 'ί', 'ρ', 'ε', 'τ', 'ε'],
                codepoints.map(|r| r.unwrap())
                          .collect::<Vec<char>>());
}

Dependencies

~545KB

7a