24 releases (14 stable)
|1.5.0||Apr 8, 2021|
|1.4.0||Feb 8, 2021|
|1.3.3||Oct 20, 2020|
|1.3.1||Aug 6, 2020|
|0.1.9||Jul 20, 2020|
#55 in Command line utilities
101 downloads per month
rocfl is a command line utility for interacting with
OCFL repositories on the local filesystem or in
S3. Its goal is to provide a logical view of OCFL objects and make
them easy to interact with in a unix-like way.
rocfl supports the following extensions:
- 0005-mutable-head: Only read is supported; not write.
Additionally, it uses the following extensions for write support that have not been specified:
rocfl-staging: By default, new object versions are staged in this extension's directory the contents of which are OCFL repository like.
rocfl-locks: This extension contains object file locks that provide limited concurrent modification protection.
The releases page has
pre-built binaries that should work on most common OSes and
architectures. You do not need to install Rust to use them. Simply
download the appropriate binary, rename it to
rocfl, and execute
Alternatively, you can build
rocfl from source as follows:
- Install Rust, and make
cargois on your
cargo install rocfl
- Verify the install:
If you want to build a binary that does not include the S3
integration, which adds a large number of dependencies, then you can
do so by running:
cargo install rocfl --no-default-features.
rocfl supports optional configuration that makes it less verbose to
rocfl expects to find its configuration file in the following,
OS dependent location:
You can easily edit the configuration by executing
which will create/open the config file for editing.
The config file may contain any number of sections structured as follows:
[repo-name] # The name to attribute new OCFL versions to author_name = "My Name" # The URI address to associate with the above name author_address = "mailto:firstname.lastname@example.org" # The absolute path to the OCFL storage root root = "/path/to/storage/root" # The absolute path to the OCFL staging root staging_root = "/path/to/staging/root" # The next properties only apply when using S3 # The AWS region your bucket is in region = "aws-region" # The URL to the S3 endpoint. This is only needed if you are using a non-standard region endpoint = "https://s3-endpoint" # The S3 bucket the OCFL repository is in bucket = "s3-bucket"
repo-name is the arbitrary name assigned to the configuration. This
is the value that you pass
rocfl when invoking the
[global] section may be used to provide default values
across all configurations. This is primarily useful for defining
All of these properties correspond to values that can be specified in
rocfl resolves the configuration by first
global config, then overlays the repository specific
config, and finally applies any values specified directly as command
Refer to the command line argument documentation for more information on the configuration properties.
The following is an overview of the features that
For a detailed description of all of the options available, consult
the builtin help by executing
rocfl --help or
rocfl <COMMAND> --help.
ls command can be used to either list the objects in a
repository or list the files in an OCFL object.
rocfl must scan the repository to locate objects. This can be slow
when operating on large repositories. The scan can be avoided when
listing an object's contents if the repository uses a supported
storage layout extension that is defined in the repository's
When listing files, only files in the most recent version are
returned. Previous versions can be queried with the
The following command lists all of the object IDs in a repository that's rooted in the current working directory:
This lists the same objects but with additional details, current version and updated date:
rocfl ls -l
-p flag additionally provides the path from the storage
root to the object:
rocfl ls -lp
A subset of objects can be listed by providing a glob pattern to match on:
rocfl ls -lo '*object*'
The contents of an object's current state are displayed by invoking
ls on a specific object ID:
rocfl ls uri:object-1
-l flag, additional details are displayed. In this case,
the version and date indicate when the individual file was last
rocfl ls -l uri:object-1
-p flag can also be used here to display the paths to the
physical files on disk relative the storage root:
rocfl ls -p uri:object-1
The contents of previous versions are displayed by using the
option. The following command displays the files that were in the
first version of the object:
rocfl ls -v1 uri:object-1
An object's contents can be filtered by specifying a glob pattern to match on:
rocfl ls uri:object-1 '*.txt'
The output is sorted by name by default, but can also be sorted version or updated date:
rocfl ls -l -s version uri:object-1
Paths within in an object can be interpreted as containing logical
directories by using the
-D flag. For example, the following will
list the logical files and logical directories that are direct
children of the logical directory
rocfl ls -D uri:object-1 sub/dir
log command displays the version metadata for all versions of an
object. It can also be executed on a file within an object, in which
case only versions that affected the specified file are displayed.
Show all of the versions of an object in ascending order:
rocfl log uri:object-1
Only display the five most recent versions:
rocfl log -rn5 uri:object-1
Show all of the versions, but formatted so each version is on a single line:
rocfl log -c uri:object-1
Show all of the versions that affected a specific file:
rocfl log uri:object-1 file1.txt
show command displays everything that changed in an object
within a specific version. If no version is specified, the most recent
changes are shown.
Show the changes in the most recent version:
rocfl show uri:object-1
Show the changes in the first version:
rocfl show uri:object-1 v1
Don't show the version metadata; only show the files that changed:
rocfl show -m uri:object-1
diff command displays the files that changed between two
Show the changes between the second and fourth versions:
rocfl diff v2 v4
cat command writes the contents of a file to
Display the contents of the head version of a file:
rocfl cat uri:object-1 file1.txt
Display the contents of a file from a specific version of the object:
rocfl cat -v1 uri:object-1 file1.txt
status command shows objects that have staged changes pending
commit, as well as what an object's pending changes are.
List all of the objects with staged changes:
List all of the file level changes to an object:
rocfl status uri:object-1
Staged changes can also be examined using the more featureful
cat commands by adding the
rocfl supports updating OCFL objects by staging changes to objects
in a local staging repository. For filesystem based repositories, the
staging repository is located within the storage root's extensions
directory, and for S3 it's in the user's home application data.
The intended workflow is to accumulate a collection of updates to an object, and then commit all of the changes to the object a single new OCFL version.
init command creates new OCFL repositories.
Create a new repository using the default layout, 0004-hashed-n-tuple-storage-layout:
rocfl -r /var/tmp/ocfl-repo-1 init
The default layout configuration can be changed by passing a config file that contains the desired configuration:
rocfl -r /var/tmp/ocfl-repo-2 init -l 0003-hash-and-id-n-tuple-storage-layout -c my-config.json
new command stages new OCFL objects. New objects will not exist
in the main repository until they have been committed. When invoked
with no options, the object is created using all of the OCFL spec
Create a new object with non-standard settings:
rocfl new uri:object-1 -d sha256 -c data -z 6
cp command copies files from the local filesystem into a
staged object, or copies logical paths within an object to a new
location within the same object. This command attempts to mimic the
behavior of GNU
cp as closely as possible.
Copy a directory into the object's root:
rocfl cp -r uri:object-1 /path/to/src -- /
Copy several files into a logical directory within the object:
rocfl cp uri:object-1 /path/to/files/* -- sub/dir
Copy several existing files internally to a new location:
rocfl cp -i uri:object-1 'internal/*.txt' -- new-location
Copy an entire logical directory from an old version to a new location in the staged version:
rocfl cp -ir -v2 uri:object-1 src/dir -- dst/dir
mv command moves files from the local filesystem into a staged
object, or moves logical paths within an object to a new location
within the same object. This command attempts to mimic the behavior of
mv as closely as possible.
Move a directory into the object's root:
rocfl mv uri:object-1 /path/to/src -- /
Move several files into a logical directory within the object:
rocfl mv uri:object-1 /path/to/files/* -- sub/dir
Move an existing file internally to a new location:
rocfl mv -i uri:object-1 internal/file.txt -- new/location.txt
rm command removes files from an object. If the removed files
were new to the staged version, then they are permanently removed and
will not appear anywhere in the object. Otherwise, references to the
files are removed from the staged version, but the files still exist
in prior versions.
Recursively remove a logical directory:
rocfl rm -r uri:object-1 path/to/dir
Remove several individual files:
rocfl rm uri:object-1 path/to/file1.txt path/to/file2.txt
Or with a glob:
rocfl rm uri:object-1 'path/to/*.txt'
reset command unstages changes made to an object. Additions are
removed and files that were deleted or modified are reverted to their
Reset a file to its previous state:
rocfl reset uri:object-1 file.txt
Reset an entire object to its previous state, removing all staged changes:
rocfl reset uri:object-1
commit command moves an object's staged changes into the OCFL
object as a new version.
Commit changes to an object:
rocfl commit uri:object-1 -n "My Name" -a "mailto:email@example.com" -m "commit message"
This can be simplified if you define your name and address in rocfl configuration. In which case, you can simply execute:
rocfl commit uri:object-1 -m "commit message"
And your name and address will be automatically added to the version metadata.
In order to commit an object to a repository without a defined storage layout, the location to store the object with the repository must be manually specified as follows:
rocfl commit uri:object-1 -m "commit message" -r relative/path/to/object/root
purge command permanently removes an object from the main OCFL
repository. This is not an operation that stages changes. However,
it will ask for confirmation before deleting an object.
To connect to an OCFL repository in S3, you first need to create an
IAM user with access to the S3 bucket, and then setup a local
~/.aws/credentials file or environment variables as described
Then, when you invoke
rocfl you must specify the bucket the
repository is in as well as the bucket region. For example:
rocfl -R us-east-2 -b example-ocfl-repo ls
You can specify a sub directory, or prefix, that the repository is rooted in within the bucket like this:
rocfl -R us-east-2 -b example-ocfl-repo -r ocfl-root ls
Changes to objects are staged locally and are only pushed to S3 when the staged version is committed. By default, changes are staged in the following location:
This location can be changed by setting the
All of these properties can define defined in
file, and activated by invoking
rocfl using the
NAME as follows:
rocfl -n NAME ls
rocfl supports all of the same operations on S3 as it does the
local filesystem, this does come with a couple of caveats:
- Scanning for objects in S3 is very slow, and using a defined storage layout extension is key to improving performance.
rocfldoes not provide any strong concurrency guarantees when modifying objects in S3. A file lock is used to guard changes in the staging repository, but there is an unchecked race condition if multiple processes attempt to commit changes to the same object from different staging locations.
The following features are planned:
- Export objects to a location outside the repository
- Index objects when the storage layout is unknown
- Validate objects
- Validate repositories
- Unsafe mutating operations such as
rewritethat can be used to change an object's history