28 releases (18 stable)
|new 1.6.1||Oct 21, 2021|
|1.5.2||Jul 15, 2021|
|1.4.0||Feb 8, 2021|
|1.3.3||Oct 20, 2020|
|0.1.9||Jul 20, 2020|
#109 in Command line utilities
63 downloads per month
rocfl is a command line utility for interacting with
OCFL repositories on the local filesystem or in
S3. Its goal is to provide a logical view of OCFL objects and make
them easy to interact with in a unix-like way.
rocfl supports the following extensions:
- 0005-mutable-head: Only read is supported; not write.
Additionally, it uses the following extensions for write support that have not been specified:
rocfl-staging: By default, new object versions are staged in this extension's directory the contents of which are OCFL repository like.
rocfl-locks: This extension contains object file locks that provide limited concurrent modification protection.
The releases page has pre-built binaries that should work on most common OSes and architectures. You do not need to install Rust to use them.
- Download and unzip the appropriate binary
./rocfl --helpto verify it works
Alternatively, you can build
rocfl from source as follows:
- Install Rust, and make
cargois on your
cargo install rocfl
- Verify the install:
If you want to build a binary that does not include the S3
integration, which adds a large number of dependencies, then you can
do so by running:
cargo install rocfl --no-default-features.
rocfl supports optional configuration that makes it less verbose to
rocfl expects to find its configuration file in the following,
OS dependent location:
You can easily edit the configuration by executing
which will create/open the config file for editing.
The config file may contain any number of sections structured as follows:
[repo-name] # The name to attribute new OCFL versions to author_name = "My Name" # The URI address to associate with the above name author_address = "mailto:firstname.lastname@example.org" # The absolute path to the OCFL storage root root = "/path/to/storage/root" # The absolute path to the OCFL staging root staging_root = "/path/to/staging/root" # The next properties only apply when using S3 # The AWS region your bucket is in region = "aws-region" # The AWS credentials profile to load credentials from. If not specified, "default" is used. profile = "aws-credentials-profile" # The URL to the S3 endpoint. This is only needed if you are using a non-standard region endpoint = "https://s3-endpoint" # The S3 bucket the OCFL repository is in bucket = "s3-bucket"
repo-name is the arbitrary name assigned to the configuration. This
is the value that you pass
rocfl when invoking the
[global] section may be used to provide default values
across all configurations. This is primarily useful for defining
All of these properties correspond to values that can be specified in
rocfl resolves the configuration by first
global config, then overlays the repository specific
config, and finally applies any values specified directly as command
Refer to the command line argument documentation for more information on the configuration properties.
The following is an overview of the features that
For a detailed description of all of the options available, consult
the builtin help by executing
rocfl --help or
rocfl <COMMAND> --help.
validate command validates either an entire OCFL repository or
specific objects within the repository. Any issues that are identified
are reported with their corresponding validation
Validate an entire repository:
Validate an entire repository without performing content fixity checks and only reporting errors:
rocfl -q validate -n
Validate a specific object:
rocfl validate urn:example:rocfl:object-1
Validate multiple objects at specific paths relative the repository root:
rocfl validate -p object-1/ object-2/
ls command either lists all of the objects in a repository or
list the files in an OCFL object.
ls will list objects as soon as they're found so long as it does not
need to sort the objects or display them in a formatted table. Object
lists are not sorted by default, and the formatted table can be
rocfl must scan the repository to locate objects. This can be slow
when operating on large repositories. The scan can be avoided when
listing an object's contents if the repository uses a supported
storage layout extension that is defined in the repository's
When listing files, only files in the most recent version are
returned. Previous versions can be queried with the
The following command lists all of the object IDs in a repository that's rooted in the current working directory:
This lists the same objects but with additional details, current version and updated date:
rocfl ls -l
-p flag additionally provides the path from the storage
root to the object:
rocfl ls -lp
A subset of objects can be listed by providing a glob pattern to match on:
rocfl ls -lo '*object*'
The contents of an object's current state are displayed by invoking
ls on a specific object ID:
rocfl ls urn:example:rocfl:object-1
-l flag, additional details are displayed. In this case,
the version and date indicate when the individual file was last
rocfl ls -l urn:example:rocfl:object-1
-p flag can also be used here to display the paths to the
physical files on disk relative the storage root:
rocfl ls -p urn:example:rocfl:object-1
The contents of previous versions are displayed by using the
option. The following command displays the files that were in the
first version of the object:
rocfl ls -v1 urn:example:rocfl:object-1
An object's contents can be filtered by specifying a glob pattern to match on:
rocfl ls urn:example:rocfl:object-1 '*.txt'
The output is sorted by name by default, but can also be sorted version or updated date:
rocfl ls -l -s version urn:example:rocfl:object-1
Paths within in an object can be interpreted as containing logical
directories by using the
-D flag. For example, the following will
list the logical files and logical directories that are direct
children of the logical directory
rocfl ls -D urn:example:rocfl:object-1 sub/dir
log command displays the version metadata for all versions of an
object. It can also be executed on a file within an object, in which
case only versions that affected the specified file are displayed.
Show all of the versions of an object in ascending order:
rocfl log urn:example:rocfl:object-1
Only display the five most recent versions:
rocfl log -rn5 urn:example:rocfl:object-1
Show all of the versions, but formatted so each version is on a single line:
rocfl log -c urn:example:rocfl:object-1
Show all of the versions that affected a specific file:
rocfl log urn:example:rocfl:object-1 file1.txt
show command displays everything that changed in an object
within a specific version. If no version is specified, the most recent
changes are shown.
Show the changes in the most recent version:
rocfl show urn:example:rocfl:object-1
Show the changes in the first version:
rocfl show urn:example:rocfl:object-1 v1
Don't show the version metadata; only show the files that changed:
rocfl show -m urn:example:rocfl:object-1
diff command displays the files that changed between two
Show the changes between the second and fourth versions:
rocfl diff v2 v4
cat command writes the contents of a file to
Display the contents of the head version of a file:
rocfl cat urn:example:rocfl:object-1 file1.txt
Display the contents of a file from a specific version of the object:
rocfl cat -v1 urn:example:rocfl:object-1 file1.txt
status command shows objects that have staged changes pending
commit, as well as what an object's pending changes are.
List all of the objects with staged changes:
List all of the file level changes to an object:
rocfl status urn:example:rocfl:object-1
Staged changes can also be examined using the more featureful
cat commands by adding the
rocfl supports updating OCFL objects by staging changes to objects
in a local staging repository. For filesystem based repositories, the
staging repository is located within the storage root's extensions
directory, and for S3 it's in the user's home application data.
The intended workflow is to accumulate a collection of updates to an object, and then commit all of the changes to the object a single new OCFL version.
init command creates new OCFL repositories.
Create a new repository using the default layout, 0004-hashed-n-tuple-storage-layout:
rocfl -r /var/tmp/ocfl-repo-1 init
The default layout configuration can be changed by passing a config file that contains the desired configuration:
rocfl -r /var/tmp/ocfl-repo-2 init -l 0003-hash-and-id-n-tuple-storage-layout -c my-config.json
new command stages new OCFL objects. New objects will not exist
in the main repository until they have been committed. When invoked
with no options, the object is created using all of the OCFL spec
Create a new object with non-standard settings:
rocfl new urn:example:rocfl:object-1 -d sha256 -c data -z 6
cp command copies files from the local filesystem into a
staged object, or copies logical paths within an object to a new
location within the same object. This command attempts to mimic the
behavior of GNU
cp as closely as possible.
Copy a directory into the object's root:
rocfl cp -r urn:example:rocfl:object-1 /path/to/src -- /
Copy several files into a logical directory within the object:
rocfl cp urn:example:rocfl:object-1 /path/to/files/* -- sub/dir
Copy several existing files internally to a new location:
rocfl cp -i urn:example:rocfl:object-1 'internal/*.txt' -- new-location
Copy an entire logical directory from an old version to a new location in the staged version:
rocfl cp -ir -v2 urn:example:rocfl:object-1 src/dir -- dst/dir
mv command moves files from the local filesystem into a staged
object, or moves logical paths within an object to a new location
within the same object. This command attempts to mimic the behavior of
mv as closely as possible.
Move a directory into the object's root:
rocfl mv urn:example:rocfl:object-1 /path/to/src -- /
Move several files into a logical directory within the object:
rocfl mv urn:example:rocfl:object-1 /path/to/files/* -- sub/dir
Move an existing file internally to a new location:
rocfl mv -i urn:example:rocfl:object-1 internal/file.txt -- new/location.txt
rm command removes files from an object. If the removed files
were new to the staged version, then they are permanently removed and
will not appear anywhere in the object. Otherwise, references to the
files are removed from the staged version, but the files still exist
in prior versions.
Recursively remove a logical directory:
rocfl rm -r urn:example:rocfl:object-1 path/to/dir
Remove several individual files:
rocfl rm urn:example:rocfl:object-1 path/to/file1.txt path/to/file2.txt
Or with a glob:
rocfl rm urn:example:rocfl:object-1 'path/to/*.txt'
reset command unstages changes made to an object. Additions are
removed and files that were deleted or modified are reverted to their
Reset a file to its previous state:
rocfl reset urn:example:rocfl:object-1 file.txt
Reset an entire object to its previous state, removing all staged changes:
rocfl reset urn:example:rocfl:object-1
commit command moves an object's staged changes into the OCFL
object as a new version.
Commit changes to an object:
rocfl commit urn:example:rocfl:object-1 -n "My Name" -a "mailto:email@example.com" -m "commit message"
This can be simplified if you define your name and address in rocfl configuration. In which case, you can simply execute:
rocfl commit urn:example:rocfl:object-1 -m "commit message"
And your name and address will be automatically added to the version metadata.
In order to commit an object to a repository without a defined storage layout, the location to store the object with the repository must be manually specified as follows:
rocfl commit urn:example:rocfl:object-1 -m "commit message" -r relative/path/to/object/root
purge command permanently removes an object from the main OCFL
repository. This is not an operation that stages changes. However,
it will ask for confirmation before deleting an object.
To connect to an OCFL repository in S3, you first need to create an
IAM user with access to the S3 bucket, and then setup a local
~/.aws/credentials file or environment variables as described
Credential profiles can be specified using the
Then, when you invoke
rocfl you must specify the bucket the
repository is in as well as the bucket region. For example:
rocfl -R us-east-2 -b example-ocfl-repo ls
You can specify a sub directory, or prefix, that the repository is rooted in within the bucket like this:
rocfl -R us-east-2 -b example-ocfl-repo -r ocfl-root ls
Changes to objects are staged locally and are only pushed to S3 when the staged version is committed. By default, changes are staged in the following location:
This location can be changed by setting the
All of these properties can define defined in
file, and activated by invoking
rocfl using the
NAME as follows:
rocfl -n NAME ls
rocfl supports all of the same operations on S3 as it does the
local filesystem, this does come with a couple of caveats:
- Scanning for objects in S3 is very slow, and using a defined storage layout extension is key to improving performance.
rocfldoes not provide any strong concurrency guarantees when modifying objects in S3. A file lock is used to guard changes in the staging repository, but there is an unchecked race condition if multiple processes attempt to commit changes to the same object from different staging locations.
The following features are planned:
- Export objects to a location outside the repository
- Index objects when the storage layout is unknown
- Unsafe mutating operations such as
rewritethat can be used to change an object's history