4 releases
new 0.0.3 | Sep 15, 2024 |
---|---|
0.0.2 | Sep 7, 2024 |
0.0.1 | Sep 3, 2024 |
0.0.0 | Sep 1, 2024 |
#17 in Authentication
532 downloads per month
45KB
1K
SLoC
filepack
filepack
is a command-line file hashing and verification utility written in
Rust.
It is an alternative to .sfv
files and tools like shasum
. Files are hashed
using BLAKE3, a fast, cryptographic
hash function.
A manifest named filepack.json
containing the hashes of files in a directory
can be created with:
filepack create path/to/directory
Which will write the manifest to path/to/directory/filepack.json
.
Files can later be verified with:
filepack verify path/to/directory
To protect against accidental or malicious corruption, as long as the manifest has not been tampered with.
If you run filepack
a lot, you might want to alias fp=filepack
.
filepack
is currently unstable: the interface and file format may change at
any time. Additionally, the code has not been extensively reviewed and should
be considered experimental.
Installation
filepack
is written in Rust and can be built
from source and installed from a checked-out copy of this repo with:
cargo install --path .
Or from crates.io with:
cargo install filepack
See rustup.rs for installation instructions for Rust.
Pre-Built Binaries
Pre-built binaries for Linux, MacOS, and Windows can be found on the releases page.
You can use the following command on Linux, MacOS, or Windows to download the
latest release, just replace DEST
with the directory where you'd like to put
filepack
:
curl --proto '=https' --tlsv1.2 -sSf https://filepack.com/install.sh | bash -s -- --to DEST
For example, to install filepack
to ~/bin
:
# create ~/bin
mkdir -p ~/bin
# download and extract filepack to ~/bin/filepack
curl --proto '=https' --tlsv1.2 -sSf https://filepack.com/install.sh | bash -s -- --to ~/bin
# add `~/bin` to the paths that your shell searches for executables
# this line should be added to your shell's initialization file,
# e.g. `~/.bashrc` or `~/.zshrc`
export PATH="$PATH:$HOME/bin"
# filepack should now be executable
filepack --help
Note that install.sh
may fail on GitHub Actions or in other environments
where many machines share IP addresses. install.sh
calls GitHub APIs in order
to determine the latest version of filepack
to install, and those API calls
are rate-limited on a per-IP basis. To make install.sh
more reliable in such
circumstances, pass a specific tag to install with --tag
.
Manifest format
filepack
manifests are named filepack.json
. filepack create
and
filepack verify
both expect the manifest to be in the root of the directory
containing the files to be verified.
The manifest contains a JSON object,
currently with a single key, files
, whose value is an object mapping strings
containing paths to manifest entries. Manifest entries are objects with the key
hash
, whose value is a hex-encoded BLAKE3 hash of the file, and size
, whose
value is the integer length of the file in bytes.
Manifests are encoded as UTF-8, so paths must be valid Unicode.
Paths are /
-separated, even on Windows, and may not contain backslashes.
Paths are relative, meaning that they cannot begin with a /
or a Windows
drive prefix, such as C:
, and cannot contain the path components .
or
..
, and may not end with a slash.
Filepack currently has no way of tracking empty directories, which are an error when creating or verifying a manifest.
An example manifest for a directory containing the files README.md
and
src/main.c
, pretty-printed for legibility:
{
"files": {
"README.md": {
"hash": "5a9a6d96244ec398545fc0c98c2cb7ed52511b025c19e9ad1e3c1ef4ac8575ad",
"size": 1573
},
"src/main.c": {
"hash": "38abf296dc2a90f66f7870fe0ce584af3859668cf5140c7557a76786189dcf0f",
"size": 4491
}
}
}
Lints
filepack create
supports optional lints that can be enabled with:
filepack create --deny all
These lints cover issues such as non-portable paths which are illegal on
Windows filesystems, paths which would conflict on case-insensitive
filesystems, and inclusion of junk files such as .DS_Store
.
Lofty Ambitions
filepack
has lofty ambitions!
-
Definition of a "root" hash, likely just the hash of the manifest itself, so that as long as the root hash is received from a trusted source the manifest itself does not need to be trusted.
-
Creation and verification of signatures over the root hash, so that developers and packagers can vouch for the correctness of the contents of a manifest, and users can verify that a manifest was signed by a trusted public key.
-
Portability lints, so packagers can ensure that the files in a manifest can be used in other environments, for example case-insensitive and Windows file systems.
-
Semantic, manchine-readable metadata about what a package is. For example, human-readable title, suggested filename-safe slug, description, or year of publication, to aid package indexing and search.
Suggestions for new features are most welcome!
Alternatives and Prior Art
filepack
serves the same purpose as programs like shasum
, which hash files
and output a text file containing file hashes and paths, which can later be
used with the same program to verify that the files have not changed.
They output hashes and paths one per line, separated by whitespace, and mainly differ in which hash function they use.
Some examples, with links to implementations and the hash functions they use:
Binary | Hash Function |
---|---|
cksfv |
CRC-32 |
shasum |
SHA-1 and SHA-2 |
sha3sum |
SHA-3 |
b2sum |
BLAKE2 |
b3sum |
BLAKE3 |
CRC-32 is not a cryptographic hash function and cannot be used to detect intentional modifications. Similarly, SHA-1 was thought to be a cryptographic hash function, but is now known to be insecure.
filepack
and b3sum
both use BLAKE3, a fast, general-purpose cryptographic
hash function.
Dependencies
~6–13MB
~170K SLoC