hist

A high-throughput CLI to count unique lines.

This is a standalone tool with equivalent functionality to sort | uniq -c | sort -n.

There is also support for deduplicating an input stream (i.e. only printing unique lines).

Installation

cargo install hist-rs

Usage

# count unique lines in a file
hist <file>

# count unique lines from stdin
/bin/cat <file> | hist

# skip counting and just write unique lines (in the same order as they appear)
hist <file> -u

# exclude lines matching a pattern while counting
hist <file> -e <pattern>

# include lines matching a pattern while counting
hist <file> -i <pattern>

# only output lines with abundance greater than or equal to a threshold
hist <file> -m <threshold>

# only output lines with abundance less than or equal to a threshold
hist <file> -M <threshold>

# sort output by the key (default: by abundance)
hist <file> -n

# sort output in descending order (default: ascending)
hist <file> -d

# show the last-k entries and a count of the other entries (lines + number of elements)
hist <file> -k <k>

# create histogram but skip sorting by value
hist <file> -S

# perform pattern substitution on incoming lines (regex compatible pattern matching)
hist <file> -s <pattern> <replacement>

# perform multiple pattern substitutions on incoming lines (can be used multiple times)
hist <file> -s <pattern> <replacement> -s <pattern> <replacement>

Benchmarks

Benchmarks (sort | uniq -c | sort -n)

I use nucgen to generate a random 1M line FASTQ file and pipe it into different tools to compare their throughput with hyperfine.

I am measuring the performance of equivalent sort <file | uniq -c | sort -n functionality.

Tools compared:

• hist
• cuniq
• huniq
• sortuniq
• awk
• Naive Implementation (coreutils sort <file | uniq -c | sort -n)
• Naive implementation (rust-coreutils sort <file | uniq -c | sort -n)
• Naive no cache (LC_ALL=C)
• Naive no cache size hints (LC_ALL=C; size hints for sort)

For the specific commands used please check the justfile.

Benchmark Table (sort | uniq -c | sort -n)

Measured on MacBook M3 Pro

Benchmarks (deduplicate stream)

I use nucgen to generate a random 1M line FASTQ file and pipe it into different tools to compare their throughput with hyperfine.

I am measuring the performance of deduplicating an input stream (i.e. only printing unique lines).

Tools compared:

• hist
• huniq
• runiq
• uq
• ripuniq
• unic
• awk

For the specific commands used please check the justfile.

Benchmark Table (deduplicate stream)

Measured on MacBook M3 Pro