fstore

fstore is a tool that lets you attach tags and descriptions to files, then later query and retrieve files based on their tags. This is designed for use with large archives of files such as documents or pictures.

Motivation

Directory trees are not a good solution for organizing large archives of files. There is never a clear logic behind why one set of directories should be higher in the directory tree than another set of directories. You can easily end up with deep directory trees, which are difficult to navigate and make it difficult to retrieve a file that you are looking for. fstore is written to solve exactly this problem. Associating arbitrary tags with files, is a much more flexible and powerful way of organizing files. This allows you to search for files using any combination of tags. This also lets you have a relatively flat directory structure and make it easy to find the files you're looking for.

While there are other tools providing similar features, fstore is designed to be simple, reliable and suitable for long term archiving. All your tags and metadata are stored in plain text files. If you move or copy a directory, the plain text file(s) containing the metadata for that directory and all the files within get moved or copied with it. Because fstore uses plain text files, it is perfect for long term archives, as it gives you full ownership of your data.

Installation

Fstore is written in Rust. It is not available as a crate yet, but you can clone this repo and install it using cargo by running:

cargo install --path /path/to/fstore/repo/

Usage

.fstore files contain all the metadata, i.e. tags and descriptions for your files and directories. You can place a .fstore file anywhere in your file system, in whichever directory you wish to organize using tags. Each .fstore file should contain the metadata for the directory containing that .fstore file and the files that are immediate children of that directory, i.e. siblings of the .fstore file. It should not contain the metadata for any of the nested directories. The nested directories should use their own .fstore files. It is important to keep the metadata decentralized in this way, so that when you move or copy the directories, you don't invalidate the metadata.

By design, fstore never modifies the .fstore files. .fstore files are meant to be authored by the user, and only consumed and queried by fstore. As an Emacs user myself, I wrote this major mode which provides autocompletion, file preview etc. and makes authoring .fstore files a breeze (I haven't written plugins for any other editor but if you like fstore, feel free to contribute!). Further details of .fstore files are discussed later.

fstore CLI tool

This query will recursively traverse the directory structure from the working directory and print a list of files that are tagged with "my-tag".

fstore query my-tag

You can get fancy with your queries and compose Boolean expressions using tags. The symbols used for composing Boolean expressions are: & for AND, | for OR, ! for NOT, and () for nesting expressions. -q is an alias for the query command. This command will traverse the directories and output a list of files that satisfy the provided query string. This includes files that have "my-tag" and "other-tag" and do not have "exclude-tag" OR have both "tag1" and "tag2".

fstore -q 'my-tag & other-tag & !exclude-tag | (tag1 & tag2)'

Below command will traverse the directories and check to make sure all .fstore files are valid, i.e. the metadata contained within them has not been invalidated due to a renaming, moving or deleting files.

fstore check

Below command will produce a list of tags for the given directory or file, and a description. The description is just a string that was authored by the user to describe the file.

fstore whatis path/to/my/file

If you wish to modify the metadata, the edit command opens up the .fstore in the given directory in your default editor. If no directory is provided, the current working directory is assumed.

ftore edit path/to/directory
# OR
fstore edit # Edit working directory

When you start tagging a large collection of existing files, you won't be able to author the metadata for all of them in one sitting. It is often useful to see a list of files that are not tracked, i.e. are not assigned any metadata. This command will produce a list of untracked files.

fstore untracked

This command will traverse the directories recursively and produce a list of all tags. As this command walks the directories recursively, if a directory doesn't contain a .fstore file, it is ignored. It is assumed that you don't wish to track the files in that directory and they are not reported as untracked.

fstore tags

Most fstore subcommands recursively traverse the directory from the current working directory and produce the output that you asked for. If you wish to produce to same output from a different path instead of the current working directory, you can override it by providing a --path | -p flag.

fstore --path different/starting/directory

Bash autocompletion

When searching for files, you may not remember the exact tags you're supposed to search for. Having autocompletion for tags and commands can be very helpful. To enable tab-autocompletion in bash, add this to your bash profile:

complete -o default -C 'fstore --bash-complete' fstore

Interactive mode with TUI

If you really don't know what tags to query, interactive mode can be very helpful. It recursively traverses all directories, loads all metadata and starts an interactive session with a TUI. You'll see a list of all tracked files, and a union of all tags these files have. You can figure out what to search for by looking through the list of tags, or start typing something like filter tag1 & tag.. and let the autocompletion help you find the right tag. If you hit return, the list of files is filtered down to only the ones that satisfy the query. Similarly you should also see the collection of tags shrink. You can iteratively refine your search until you find the file you are looking for.

Other commands you can use in interactive mode are:

• reset to remove the current filter
• whatis <index> to see the tags and description of the file in the current list. You choose the file by it's index rather than name or path.
• open <index> to open the file with the given index in your default application.
• quit or exit will exit out of the interactive mode.

.fstore files

The format of a .fstore file should be a header, followed by content under that header, followed by another header and so on till the end of the file. Supported headers are: desc for description, tags for tags and path for filepaths and globs. Headers should be in their own line, wrapped in [] brackets, similar to TOML or INI files. So a typical .fstore file might look like:

[desc]
This is the description of this directory.
The description can span multiple lines and paragraphs.

[tags]
tag1 tag2 tag3 tag4
tag5
tag6

[path]
my_file_1.pdf
[tags]
ftag1 pdf document

[path]
my_file_2.pdf
[desc]
This is my second file.
[tags]
ftag2 pdf document

Tags and description headers that occur at the start of the .fstore file are associated with the directory itself. Tags and description headers that occur after a path header are associated with that specific file or glob. path doesn't need to be one specific file. Instead it can be a glob, in which case, the provided tags and description are applied to all files that match the glob. Globs can be used to avoid repitition when you want to associate the same set of tags with many files. As such, multiple globs can match a single file. The tags associated with that file will be the union of tags associated with the globs that match the file. The descriptions are concatenated.

Graph mode

In theory, the tags can be used to render a zettelkasten style graph, where the files are represented as nodes and any two files that share a tag are connected by an edge. This in combination with filtering, can be a powerful, visual way of searching through a large collection of files. This is not implemented yet. At this stage, this is just an idea in the making.