detchar

A simple CLI to detect character encodings in files; similar to chardet.

Implemented as a very, very thin wrapper over chardetng.

The example text files in ./data are from this kaggle dataset.

Multithreading

chardetng has a feature which parallelises elimination of possible encodings for each text file. This can be enabled by compiling detchar with the multithreading feature.

However, this is disabled by default, because for large numbers of files it is generally more effective to just parallelise over files, using e.g. GNU parallel:

cat my_file_list.txt | parallel detchar