ripuniq: fast line deduplicator

functions similarly to uniq, with the main exception that it does not require its input to be sorted (and unlike sort -u, the order of the input lines will actually be preserved).

ripuniq only reads lines from stdin, so it should always be used with < inputfile or as part of a pipeline.

ripuniq has one command-line option, -z. it changes the line seperator from '\n' to '\0'.

ripuniq is heavily optimized, up to 15x faster than sort -u.

installation

static binaries for x86_64 linux systems available from the releases page.

for other platforms, just run cargo install ripuniq. this will take a bit longer and result in a larger binary, but should work basically everywhere, since ripuniq does not have any system dependancies other than those needed by the rust runtime (which will already be installed as dependancies of cargo)