#edit #distance #lzjd #lempel #ziv

bin+lib malwaredb-lzjd

Rust implementation of the LZJD algorithm by Edward Raff -- https://github.com/EdwardRaff/jLZJD

2 releases

0.1.2 Sep 17, 2023
0.1.1 Sep 17, 2023
0.1.0 Aug 22, 2023

#125 in Compression

Download history 28/week @ 2023-08-17 117/week @ 2023-08-24 107/week @ 2023-08-31 180/week @ 2023-09-07 151/week @ 2023-09-14 119/week @ 2023-09-21

565 downloads per month
Used in 3 crates (2 directly)

GPL-3.0 license

555 lines



Rust implementation of Lempel-Ziv Jaccard Distance (LZJD) algorithm based on jLZJD

Main differences:

  • Rust instead of Java
  • Can use any hasher (executable uses CRC32) instead of just Murmur3
  • Does not allocate memory for every unique hash, instead keeps k=1024 smallest
  • Based on Vec instead of IntSetNoRemove, which is more like HashMap
  • Hash files are considerably smaller if small sequences have been digested

This fork has minor changes:

  • Update to Rust edition 2021.
  • Remove dependencies preventing it from working on non-x86 hardware.
    lzjd [FLAGS] [OPTIONS] <INPUT>...

    -c, --compare        compare SDBFs in file, or two SDBF files
    -r, --deep           generate SDBFs from directories and files
    -g, --gen-compare    compare all pairs in source data
    -h, --help           Prints help information
    -V, --version        Prints version information

    -o, --output <FILE>            send output to files
    -t, --threshold <THRESHOLD>    only show results >= threshold [default: 1]

    <INPUT>...    Sets the input file to use

See also:


~78K SLoC