4 releases (2 stable)

Uses old Rust 2015

1.0.1 Aug 30, 2016
1.0.0 Aug 28, 2016
0.1.1 Nov 18, 2015
0.1.0 Nov 18, 2015

#1334 in Text processing

Download history 3294/week @ 2023-11-29 3149/week @ 2023-12-06 2420/week @ 2023-12-13 554/week @ 2023-12-20 357/week @ 2023-12-27 1667/week @ 2024-01-03 1874/week @ 2024-01-10 2932/week @ 2024-01-17 2318/week @ 2024-01-24 2417/week @ 2024-01-31 2335/week @ 2024-02-07 3762/week @ 2024-02-14 4337/week @ 2024-02-21 5443/week @ 2024-02-28 5771/week @ 2024-03-06 4410/week @ 2024-03-13

20,871 downloads per month

MIT/Apache

13KB
233 lines

N-grams

Build Status Coverage Status

Documentation

This crate takes a sequence of tokens and generates an n-gram for it. For more information about n-grams, check wikipedia: https://en.wikipedia.org/wiki/N-gram

Note: The canonical version of this crate is hosted on Gitlab

Usage

Probably the easiest way to use it is to use the iterator adaptor. If your tokens are strings (&str, String, char, or Vec), you don't have to do anything other than generate the token stream:

use ngrams::Ngram;
let grams: Vec<_> = "one two three".split(' ').ngrams(2).collect();
// => vec![
//        vec!["\u{2060}", "one"],
//        vec!["one", "two"],
//        vec!["two", "three"],
//        vec!["three", "\u{2060}"],
//    ]

(re: the "\u{2060}": We use the unicode WORD JOINER symbol as padding on the beginning and end of the token stream.)

If your token type isn't one of the listed types, you can still use the iterator adaptor by implementing the ngram::Pad trait for your type.

Dependencies

~125KB