3 stable releases
1.1.1 | Nov 24, 2023 |
---|---|
1.1.0 | Nov 10, 2023 |
1.0.0 | Oct 30, 2020 |
#449 in Text processing
28 downloads per month
51KB
994 lines
Ferret: Copy-Detection in Text and Code
Ferret is a copy-detection tool, locating duplicate text or code in multiple text documents or source files. Ferret is designed to detect copying ( collusion ) within a given set of files.
As a library, Ferret can be used to analyse program code or natural language texts into trigrams, and compare pairs of documents for similarity.
Features:
- compares text documents containing natural language or computer language
- computes a similarity measure based on the trigrams found within pairs of documents
- many major programming languages are recognised and tokenised appropriately
- outputs for analysis include:
- pairwise comparisons ordered by similarity, including trigram counts
- counts of unique trigrams within each file / group
- reverse index from trigrams to list of documents they are found in
- XML detailed comparison of a pair of documents
Command line use
$ ferret --help
Usage: ferret [-ghluvx] filename [filenames...]
-g, --group Use subdirectory names to group files
-h, --help Show help information
-l, --list-trigrams
Output list of trigrams found
-u, --unique-counts
Output counts of unique trigrams
-v, --version Version number
-x, --xml-report filename1 filename2 outfile : Create XML report
Library use
Take some files and find the two most similar:
use ferret::documents::Documents;
fn main() {
let files = ["txt1.txt".to_string(), "txt2.txt".to_string(), "txt3.txt".to_string()];
let docs = Documents::new(&files[..]);
let results = docs.sorted_results(false);
println!("Most similar pair: {}", results[0]);
}
Take a file, and read it trigram-by-trigram:
use ferret::trigram_reader::TrigramReader;
use std::path::PathBuf;
fn main() {
let path = PathBuf::from(r"test.rb");
let mut reader = TrigramReader::new(&path);
while reader.read_trigram () {
println!("Trigram {}", reader.last_trigram ());
}
}