10 releases (5 stable)

1.4.0 Nov 16, 2023
1.3.0 Sep 23, 2023
1.2.1 Apr 18, 2023
1.0.0-alpha1 Aug 5, 2022
0.0.1 Apr 29, 2021

#8 in Internationalization (i18n)

Download history 1597/week @ 2023-12-25 2893/week @ 2024-01-01 3307/week @ 2024-01-08 3397/week @ 2024-01-15 3743/week @ 2024-01-22 3658/week @ 2024-01-29 4186/week @ 2024-02-05 4402/week @ 2024-02-12 5021/week @ 2024-02-19 3833/week @ 2024-02-26 4359/week @ 2024-03-04 4774/week @ 2024-03-11 3550/week @ 2024-03-18 3354/week @ 2024-03-25 3057/week @ 2024-04-01 2339/week @ 2024-04-08

12,780 downloads per month
Used in 50 crates (11 directly)

Custom license

2.5MB
27K SLoC

icu_segmenter crates.io

Segment strings by lines, graphemes, words, and sentences.

This module is published as its own crate (icu_segmenter) and as part of the icu crate. See the latter for more details on the ICU4X project.

This module contains segmenter implementation for the following rules.

Examples

Line Break

Find line break opportunities:

use icu::segmenter::LineSegmenter;

let segmenter = LineSegmenter::new_auto();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(&breakpoints, &[0, 6, 13, 17, 23, 29, 36]);

See LineSegmenter for more examples.

Grapheme Cluster Break

Find all grapheme cluster boundaries:

use icu::segmenter::GraphemeClusterSegmenter;

let segmenter = GraphemeClusterSegmenter::new();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(
    &breakpoints,
    &[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
        19, 21, 22, 23, 24, 25, 28, 29, 30, 31, 34, 35, 36
    ]
);

See GraphemeClusterSegmenter for more examples.

Word Break

Find all word boundaries:

use icu::segmenter::WordSegmenter;

let segmenter = WordSegmenter::new_auto();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(
    &breakpoints,
    &[0, 5, 6, 11, 12, 13, 16, 17, 22, 23, 28, 29, 35, 36]
);

See WordSegmenter for more examples.

Sentence Break

Segment the string into sentences:

use icu::segmenter::SentenceSegmenter;

let segmenter = SentenceSegmenter::new();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(&breakpoints, &[0, 13, 36]);

See SentenceSegmenter for more examples.

More Information

For more information on development, authorship, contributing etc. please visit ICU4X home page.

Dependencies