Rust TF-IDF

Library to calculate TF-IDF (Term Frequency - Inverse Document Frequency) for generic documents. The library provides strategies to act on objects that implement certain document traits (NaiveDocument, ProcessedDocument, ExpandableDocument).

For more information on the strategies that were implemented, check out Wikipedia.

Document Types

A document is defined as a collection of terms. The documents don't make assumptions about the term types (the terms are not normalized in any way).

These document types are of my design. The terminology isn't standard, but they are fairly straight forward to understand.

• NaiveDocument - A document is 'naive' if it only knows if a term is contained within it or not, but does not know HOW MANY of the instances of the term it contains.

• ProcessedDocument - A document is 'processed' if it knows how many instances of each term is contained within it.

• ExpandableDocument - A document is 'expandable' if provides a way to access each term contained within it.

Example

The most simple way to calculate the TfIdf of a document is with the default implementation. Note, the library provides implementation of ProcessedDocument, for a Vec<(T, usize)>.

use tfidf::{TfIdf, TfIdfDefault};

let mut docs = Vec::new();
let doc1 = vec![("a", 3), ("b", 2), ("c", 4)];
let doc2 = vec![("a", 2), ("d", 5)];

docs.push(doc1);
docs.push(doc2);

assert_eq!(0f64, TfIdfDefault::tfidf("a", &docs[0], docs.iter()));
assert!(TfIdfDefault::tfidf("c", &docs[0], docs.iter()) > 0.5);

You can also roll your own strategies to calculate tf-idf using some strategies included in the library.

use tfidf::{TfIdf, ProcessedDocument};
use tfidf::tf::{RawFrequencyTf};
use tfidf::idf::{InverseFrequencySmoothIdf};

#[derive(Copy, Clone)] struct MyTfIdfStrategy;

impl<T> TfIdf<T> for MyTfIdfStrategy where T : ProcessedDocument {
  type Tf = RawFrequencyTf;
  type Idf = InverseFrequencySmoothIdf; 
}

let mut docs = Vec::new();
let doc1 = vec![("a", 3), ("b", 2), ("c", 4)];
let doc2 = vec![("a", 2), ("d", 5)];

docs.push(doc1);
docs.push(doc2);

assert!(MyTfIdfStrategy::tfidf("a", &docs[0], docs.iter()) > 0f64);
assert!(MyTfIdfStrategy::tfidf("c", &docs[0], docs.iter()) > 0f64);

License

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.