#duplicate #similarity #similar #compare

bin+lib near-facsimile

Find similar or identical text files in a directory

8 releases (5 stable)

1.0.5 Jan 26, 2023
1.0.2 Oct 25, 2022
1.0.1 Aug 24, 2022
0.8.2 Aug 17, 2022

#389 in Text processing

Apache-2.0

34KB
496 lines

near-facsimile

Find similar or identical text files in a directory.

Rust tests dependency status

  • reasonable facsimile, noun

    : a copy that is not exact but is fairly close

    The house is a reasonable facsimile of his original home.

    —sometimes used in a joking way

    I can speak French, or at least a reasonable facsimile of French.

    “reasonable facsimile,” Merriam-Webster.com Dictionary

Installation

  • On Fedora, RHEL, or CentOS, install this program from the Copr repository:

    1. Enable the repository:

      # dnf copr enable mareksu/near-facsimile
      
    2. Install the package:

      # dnf install near-facsimile
      
  • On macOS, use the Homebrew package manager:

    1. Install the Homebrew package manager as described at https://brew.sh/.

    2. Install the package:

      $ brew install msuchane/repo/near-facsimile
      
  • To install from source on any system, use the Cargo package manager:

    1. Install Cargo as described at https://rustup.rs/.

    2. Install the package:

      $ cargo install near-facsimile
      

Usage

  • Recursively examine a directory of text files:

    dir-with-text]$ near-facsimile
    

    The program continually prints out file information to the terminal.

    Finally, it can save all statistics sorted by file similarity as a CSV or JSON file.

Options

The following options are available:

Specifying the documentation directory

$ near-facsimile --path <path-to-directory>

Saving the the results as a CSV table or a JSON file

Optionally, you can save the file similarities above the set threshold as CSV or JSON:

$ near-facsimile --csv <path-to-file>
$ near-facsimile --json <path-to-file>

Setting the lowest reported similarity threshold

The tool only reports files that are similar over a certain threshold. By default, the threshold is 85.0, or 85% similar.

$ near-facsimile --threshold=<85.0>

Disregarding certain lines in files

You can configure the file comparison such that it skips all lines that match your regular expressions. The comparison is the calculated from the remaining lines, which match none of the regular expressions.

For example, skip all lines that start with //:

$ near-facsimile --skip-lines '^//'

Switching to a faster, less accurate comparison

By default, the tool uses the Levenshtein metric, which is accurate but rather slow. You can instead compare files using the Jaro metric, which finishes in around half the time, but produces less accurate statistics.

$ near-facsimile --fast

If you specify the --fast option twice (-ff), the tool uses the relatively rudimentary but very fast trigram comparison instead:

$ near-facsimile --fast --fast

Dependencies

~13–20MB
~351K SLoC