#unicode #encoding #binary #text #library


Fast inspection of binary buffers to guess/determine the encoding

5 releases

Uses old Rust 2015

0.2.4 Nov 4, 2018
0.2.3 Oct 7, 2018
0.2.2 Aug 21, 2018
0.2.1 Jun 3, 2018
0.2.0 Jun 3, 2018

#221 in Encoding

Download history 4643/week @ 2021-08-15 5384/week @ 2021-08-22 5254/week @ 2021-08-29 4747/week @ 2021-09-05 5395/week @ 2021-09-12 4885/week @ 2021-09-19 5636/week @ 2021-09-26 5540/week @ 2021-10-03 5021/week @ 2021-10-10 4056/week @ 2021-10-17 5508/week @ 2021-10-24 4954/week @ 2021-10-31 5787/week @ 2021-11-07 5547/week @ 2021-11-14 4185/week @ 2021-11-21 6905/week @ 2021-11-28

23,145 downloads per month
Used in 66 crates (14 directly)


150 lines


Crates.io Documentation

A simple library for fast inspection of binary buffers to guess the type of content.

This is mainly intended to quickly determine whether a given buffer contains "binary" or "text" data. Programs like grep or git diff use similar mechanisms to decide whether to treat some files as "binary data" or not.

The analysis is based on a very simple heuristic: Searching for NULL bytes (indicating "binary" content) and the detection of special byte order marks (indicating a particular kind of textual encoding). Note that this analysis can fail. For example, even if unlikely, UTF-8-encoded text can legally contain NULL bytes. Conversely, some particular binary formats (like binary PGM) may not contain NULL bytes. Also, for performance reasons, only the first 1024 bytes are checked for the NULL-byte (if no BOM was detected).

If this library reports a certain type of encoding (say UTF_16LE), there is no guarantee that the binary buffer can actually be decoded as UTF-16LE.


use content_inspector::{ContentType, inspect};

assert_eq!(ContentType::UTF_8, inspect(b"Hello"));
assert_eq!(ContentType::BINARY, inspect(b"\xFF\xE0\x00\x10\x4A\x46\x49\x46\x00"));


CLI example

This crate also comes with a small example command-line program (see examples/inspect.rs) that demonstrates the usage:

> inspect
USAGE: inspect FILE [FILE...]

> inspect testdata/*
testdata/create_text_files.py: UTF-8
testdata/file_sources.md: UTF-8
testdata/test.jpg: binary
testdata/test.pdf: binary
testdata/test.png: binary
testdata/text_UTF-16BE-BOM.txt: UTF-16BE
testdata/text_UTF-16LE-BOM.txt: UTF-16LE
testdata/text_UTF-32BE-BOM.txt: UTF-32BE
testdata/text_UTF-32LE-BOM.txt: UTF-32LE
testdata/text_UTF-8-BOM.txt: UTF-8-BOM
testdata/text_UTF-8.txt: UTF-8

If you only want to detect whether something is a binary or text file, this is about a factor of 250 faster than file --mime ....


Licensed under either of

at your option.