#duplicate #dupes

app fddf

Fast, parallelized duplicate file finder

11 stable releases

1.7.0 Apr 12, 2020
1.6.1 Nov 23, 2019
1.6.0 Oct 26, 2019
1.4.1 Nov 13, 2018
1.3.1 Jun 21, 2017

#985 in Filesystem

Download history 20/week @ 2023-08-07 25/week @ 2023-08-14 4/week @ 2023-08-21 15/week @ 2023-08-28 32/week @ 2023-09-04 10/week @ 2023-09-11 7/week @ 2023-09-18 7/week @ 2023-09-25 5/week @ 2023-10-02 2/week @ 2023-10-09 8/week @ 2023-10-16 7/week @ 2023-10-23 23/week @ 2023-10-30 5/week @ 2023-11-06 18/week @ 2023-11-13 19/week @ 2023-11-20

65 downloads per month

MIT/Apache

17KB
315 lines

fddf

Fast data dupe finder

This is a small Rust command-line program to find duplicate files in a directory recursively. It uses a thread pool to calculate file hashes in parallel.

Duplicates are found by checking size, then (Blake3) hashes of parts of files of same size, then a byte-for-byte comparison.

Build/install

Directly from crates.io with cargo install fddf.

From checkout:

cargo build --release
cargo run --release

Minimum supported Rust version is 1.39.0.

Usage

fddf [-s] [-t] [-S] [-m SIZE] [-M SIZE] [-v] <rootdir>

-s: report dupe groups in a single line
-t: produce a grand total
-S: don't scan recursively for each directory given
-f: check for files with given pattern only
-F: check for files with given regular expression only
-m: minimum size (default 1 byte)
-M: maximum size (default unlimited)
-v: verbose operation

By default, zero-length files are ignored, since there is no meaningful data to be duplicated. Pass -m 0 to include them.

PRs welcome!

Dependencies

~8MB
~153K SLoC