AllMyToes

Provides thumbnails by using the freedesktop-specified thumbnail data-base (aka XDG standard).

Thumbnails are shared with other programs via a common cache.

The (expensive) creation of thumbnails is only done if a thumbnails has not been created yet or if the original file has changed.

AllMyToes can create thumbnails for many image formats. (See “Decoding” support of the image crate.)

Additionally, AllMyToes can create thumbnails for other kind of files using other programs to create the base image. (See section Provider.)

AllMyToes is both, a non-interactive program for the shell and scripts, and a Rust library. It's abbreviated “AMT” (or “amt”) in code or structured data.

[[TOC]]

Usage

Call allmytoes with a file as parameter. The program will print the path to a thumbnail to stdout.

> allmytoes some_image.jpg
/home/me/.cache/thumbnails/large/b7931d1d6e0439c1a6e2e6b02c5b21a6.png

If the thumbnail already exists, AllMyToes will just return the path to the existing thumbnail file. If not, AllMyToes will create the image’s thumbnail first.

Thumbnail Size

The latest freedesktop.org specification (0.9.0) defines four different thumbnail sizes: normal, large, x-large, and xx-large with maximum edge lengths of 128 px, 256 px, 512 px, and 1024 px respectively.

By default, AllMyToes returns the path to the large (256 px) thumbnail. The size to be returned can be chosen with the -s (--size) option by giving a value out of {n, l, x, xx}.

For example, to get a xx-large thumbnail, the call would look like this:

> allmytoes -sxx some_image.jpg
/home/me/.cache/thumbnails/xx-large/ad0779df58de36f038bdc4040a322bfe.png

Small Input Images

If the input image is smaller than the requested thumbnail size, the thumbnail will have the same dimensions as the input image. A thumbnail will never be bigger than the original image.

If a thumb for the requested size does not already exist, AllMyToes will check if the input image is smaller than the requested thumb size. If so, AllMyToes will determine the smallest thumb size that still covers the input image dimensions (is bigger or equal than the input image). If that “feasible” thumb size is different from the requested thumb size, AllMyToes will start the job all over with that smaller, “feasible” thumb size. AllMyToes does that to avoid unnecessary, duplicated thumbnails for small images.

That implies that for small input images, the returned thumbnail path might not correspond to the requested thumb size, because creating and keeping thumbnails for the (bigger) requested size would be a waste of computation time and disk space.

If – for whatever reason – one must get a result path to the thumbnail of the requested size, one can use the -F (--force-size) switch. However, one should do that only if there is a really good reason.

Show Extensive Information

Calling AllMyToes with the --extensive (or -e) option makes it print out more information than just the path to the thumb. The provided information is

• The path to the thumb
• The URI-hash of the input file that is used for the XDG-thumb name
• All meta data (key-value pairs) stored in the thumb (as “tEXt” entries)

It does not matter if the thumb did already exist or if it has just been created by AllMyToes.

All data is printed as a list of key-value pairs, each tuple in one line. The key is separated from the value by “: ”. Key-value pairs which show meta data from the thumbnail file are preceded with a colon and are printed in alphabetical order.

Example:

$ allmytoes --extensive some_image.jpg 
Thumb path: /home/dude/.cache/thumbnails/large/c000276b8378a33e16e3fac005eebc02.png
Thumb hash: c000276b8378a33e16e3fac005eebc02
:Software: allmytoes
:Software::Version: 0.1.0
:Thumb::Image::Dimensions: 4608x3456
:Thumb::Image::Height: 3456
:Thumb::Image::Reorientation: Rotated 180 degrees, not flipped
:Thumb::Image::Width: 4608
:Thumb::MTime: 1693737198
:Thumb::Mimetype: image/jpeg
:Thumb::Size: 7777807
:Thumb::URI: file:///path/to/some_image.jpg

What meta-data is available in the thumb of course depends on the software which created the thumb. So, the lines starting with a colon (:) may vary.

Logging

By default, AllMyToes will print warnings and errors to stderr. A different log-level can be set via the environment variable RUST_LOG. For example, do a

export RUST_LOG=trace

to get all log entries, down to the most detailed “trace”-level.

Provider

AllMyToes can create thumbnails for non-image files if a “provider” is defined. A provider is a shell-command or a script that provides an image for a certain set of MIME types.

AllMyToes has inbuilt support for some file formats:

The exact MIME-types and the provider definitions can be found in /conf/provider.yaml.

Defining Provider

Users can use their own provider configuration in a YAML file to add support for more MIME types or to implement more fancy thumbnails.

To do so, copy /conf/provider.yaml to ~/.config/allmytoes/provider.yaml (or $XDG_CONFIG_HOME/allmytoes/provider.yaml if $XDG_CONFIG_HOME is defined) and adapt it. You can also use the -p (--provider-config-file) option to explicitly choose another provider configuration.

❗ Be aware that the provider-feature is pretty new and the format of the provider configuration will likely change here and there before it stabilizes.

The provider configuration YAML file has a list of dictionaries on the top level. Each list entry defines a provider for one or more MIME types. For each entry, there are two mandatory and two optional keys.

Key mimes

The mime type list defines for which mime types the provider is executed. MIME types are specifies as <type>/<subtype>, like video/x-matroska. If AllMyToes is called for a file for which no provider can be found, the MIME type is printed in an error message. MIME types for which a provider is found are printed in a debug message and also annotated as meta-data in the thumbnail itself. (Which you can see if you use the --extensive (-e) option.)

Key commmands

Each provider-entry must have a list of commands. Commands are shell commands (or scripts) that take the input file and return an image back to AllMyToes. AllMyToes takes that image and creates the XDG-conforming thumbnail from it. Commands provide that image by copying it to a defined location.

When one of the MIME types from the mimes list matches the input file, AllMyToes executes the first command from the commands list. If the command succeeds (exit code 0), AllMyToes stops and uses the thumbnail which has been provided by that command.

If the command does not succeed (exit code ≠ 0), AllMyToes tries the next command, and so forth.

This allows to have commands which use different programs to provide the thumb. If a user does not have the programs used in the first command, another option to calculate the thumb may work. If a command returns 0 but still does not provide a valid thumb, AllMyToes returns with an error.

A command provides the thumb by copying it to a certain path in a temporary directory, which is deleted by AllMyToes afterwards.

Commands need certain information to do their job, for example the input file and the path to copy the thumbnail to. Therefore, commands can use certain variables.

If the created image (at %(outfile)s) is bigger than %(size)s, AllMyToes will scale the image down accordingly. AllMyToes also does other necessary conversions like turning the image into a 8-bit per channel RGBA PNG image and add all necessary meta chunks. So, the commands can provide their thumbnails in any supported image format. AllMyToes will take care to turn the image into a freedesktop.org-conform thumbnail afterwards.

Key meta

freedesktop.org-thumbnails have “meta data chunks”, meta-data attributes in key-value pairs. (See explanation of meta-data chunks.) Providers can also provide such meta-data entries. Some are even recommended by the standard (Thumb::Document::Pages for paper-oriented documents and Thumb::Movie::Length for the length of video files in seconds).

The meta key in the provider specification contains a dictionary, mapping meta-data keys to lists of commands. For each key, the list of commands are executed like for the thumb image top down until one of them exits successfully with 0. The value for the meta-chunk has to be returned via stdout.

With the current implementation, the thumb-generation fails if none of the commands for a meta-chunk is successful. The commands for a meta-chunk can use the same variables as the commands for the image creation.

The specification of meta-chunks is optional.

Key revision

The revision key is intended as a “version” for the provider. It's optional and as of now, it does not have any effect other than being added as a meta-chunk to the thumbnail.

The idea is that this revision number can be used in the future to re-create thumbs if a provider is available with a higher revision.

Notes on commands

One should consider a few things for both, commands for the images in the commands key, and for commands under a meta key.

First, commands can be multi-line shell scripts, simply by using multi-line strings in the YAML file. This example for an SVG-provider shows how multi-line commands may look like:

- mimes:
  - image/svg+xml
  commands:
    - |
      which inkscape || exit 1
      w=$(inkscape -W "%(file)s" | sed 's/\..*//')
      h=$(inkscape -H "%(file)s" | sed 's/\..*//')
      test $w -gt $h && size_arg="-w %(size)s" || size_arg="-h %(size)s"
      inkscape --export-area-page --export-type=png $size_arg -o "%(tmpdir)s/outfile.png" "%(file)s" && \
        mv "%(tmpdir)s/outfile.png" "%(outfile)s"
    - |
      convert -background none -resize %(size)sx%(size)s "%(file)s" "%(tmpdir)s/outfile.png" && \
        mv "%(tmpdir)s/outfile.png" "%(outfile)s"

Secondly, commands should be POSIX shell compatible, not - for example - depend on BASH. This assures that the provider commands will work on most machines and accounts.

Third, commands should fail with a return code > 0 if some environment preconditions are not fulfilled. Keep in mind that a pipe of commands exits with the return code of the last command, no matter if an earlier command did fail.

So, it's a good idea to chain programs with && or to extend program calls with || exit 1 to make a provider command fail properly. The availability of required programs that are used in pipes can for example be tested with which <program> || exit 1.

Also remember to quote the input file (like "%(file)s") as the path may contain spaces. The %(outfile)s (and the %(tmpdir)s) should not contain any spaces, but it doesn't hurt to also quote them.

Example

Let's look at an example and specify a provider that provides thumbnails for PDF and postscript files. Both can be turned into a thumbnail image with the same command, which is why it's possible to handle both in one provider.

Be aware that the real default configuration for AllMyToes uses a different provider definition for these formats to implement more specific meta-chunk commands. This definition serves only as an example.

PDF files have a MIME type application/pdf, postscript files have application/postscript.

- mimes: [application/pdf, application/postscript]
  commands:
    - evince-thumbnailer -s %(size)s "%(file)s" %(outfile)s
    - convert -thumbnail %(size)sx%(size)s -background white %(file)s[0] PNG:%(outfile)s
  meta:
    'Thumb::Document::Pages':
      - |
        which exiftool || exit 1
        exiftool "%(file)s" | awk '/^Page Count/ {print $4}' | sed 's/[^0-9]*//g'
      - | 
        which gs || exit 1
        gs -o /dev/null -sDEVICE=bbox "%(file)s" 2>&1 | grep HiResBoundingBox | wc -l
  revision: 1

First, the MIME types for which this provider shall be used are defined.

The commands section first tries to create the thumbnail (in %(outfile)s) with evince-thumbnailer. If evince-thumbnailer exists, the task of providing the image is done and the second command is not executed.

If evince-thumbnailer is not available, the first command fails and AllMyToes will execute the second command which uses convert from ImageMagick. If also convert is not available, the thumb creation has failed and AllMyToes returns with an error.

The evince-thumbnailer is tried first because it's faster and we prefer that one.

If one of the two commands was successful, AllMyToes evaluates the meta section. In this example, we have only one meta-chunk with the meta-key Thumb::Document::Pages. This meta-chunk has also two commands and again, AllMyToes will first try the first and execute the second one only if the first one fails.

The first command depends on exiftool, the second depends on Ghostscript (gs). Because both of them are used in a pipe, we first check if they exist explicitly and exit with 1 otherwise. The example shows how a multi-line script can be used as a command.

Both of the meta-data commands return the value for the Thumb::Document::Pages meta data chunk as output on stdout.

Check the default provider configuration for more examples.

How is this useful?

My personal motivation was the use of thumbnails for image previews in terminal-based file managers like joshuto and ranger. But AllMyToes can be useful in any situation where one wants to show a size-limited image in scripted environments, maybe as icons in desktop notifications, other kind of image-overlays in terminal-based applications, or desktop widgets; wherever loading full sized-images would consume unnecessary time and CPU power.

Installation

Linux X86-64 Binary (Experimental)

Download the latest release as binary from the release page. Make the downloaded file executable (chmod +x allmytoes) and place it in a directory which is part of your PATH (for example mv allmytoes ~/.local/bin).

Latest release from source via cargo

cargo install allmytoes

Bleeding edge from source via cargo

cargo install --git https://gitlab.com/allmytoes/allmytoes.git

From repo clone

git clone https://gitlab.com/allmytoes/allmytoes.git
cd allmytoes
cargo build --release

The built binary will be at target/release/allmytoes.

Some Specs

• Freedesktop.org Thumbnail Specification: AllMyToes strives for compliance to the thumbnail freedesktop.org specification. All relevant parts of the standard should be implemented by now.
• AllMyToes will not create the cache directory ($XDG_CACHE_HOME or ~/.cache). In case the cache directory does not exist, AllMyToes will terminate with an error. (Might be changed later.)
• AllMyToes will create the thumbnail directory beneath the cache directory and the size-specific sub-directories if they do not exist.
• AllMyToes will only accept regular files and symlinks to regular files as input. Directories may get supported in the future. There are no plans for other file types.

Thumbnail Meta-Data Chunks

Thumbnails - all in PNG format - have certain meta-data entries, each of them a PNG “tEXt” chunk with a key and a value. The freedesktop.org thumbnail standard describes some mandatory and some optional keys. (See the “Thumbnail Creation section” in the standard.) AllMyToes implements all of them. The following table lists all meta-data entries created by AllMyToes.

† Only for image input files

Roadmap

Major milestones planned:

• Comply to mandatory requirements from the freedesktop.org specification and have basic robustness (see %1). AllMyToes will go to version 0.1.0 then.
• Support for non-image file formats (videos, fonts, documents,...) by configuring other programs providing previews. (#13)
• Support for shared repositories.
• Support for other thumbnail repository locations depending on configured path patterns. (Not part of the freedesktop.org standard.)

See also my development board.

Testing

Few things are tested by standard Rust unit test. Those tests can be run with cargo test.

The majority of AllMyToes’ functionality is tested with end-to-end tests by a BDD test framework, Python Behave. Those tests use real images on the file system and an allmytoes debug build, started as a sub-process.

The BDD tests live in the test sub-directory. The test-definitions can be found as feature-files in test/features.

Running the BDD tests

Have a Python >= 3.10 environment with the required dependencies. E.g., use a virtualenv and then pip install -r test/requirements.txt.

Have a debug build of AllMyToes. target/debug/allmytoes will be the binary under test.

Then, to run the tests, cd into the test directory and run behave.

Notes on missing Tests / untested things

• evaluation of XDG_CACHE_HOME
• MIME type probing for wrong and missing file extensions (if this fails on CI, check if the MIME-db is installed on the used docker img)
• --force-creation
• Return error when input file is not readable, no matter if valid thumb exists.
• Updating existing but outdated thumbs
• Error on non-existing, non-readable, not-decodable input-file
• Deny to process files which are neither a regular nor a symlink to a regular file
• All tEXt meta-data (mtime and URI are tested implicitly, of course)
• #61

You may prefer to use this instead of AllMyToes

• Tumbler is a much more mature and feature-rich tool for the same purpose, but needs to run as a service and uses DBUS for communication.

  I did not use Tumbler as I wanted to have something more simple for my scripting.

AllMyToes as a Rust Library

AllMyToes can be used as a Rust library to obtain a thumbnail for a given image. You can find the crate on crates.io.

AllMyToes has a very small interface. There's one struct for the configuration (AMT) that also provides the one function (get) to get a thumb. Then, there's one enumeration to specify the thumbnail size (ThumbSize), one struct for the result (Thumb), and one enumeration for the possible errors (ToeErrorType).

Example

use std::path::Path;
use allmytoes::{AMTConfiguration, AMT, ThumbSize};

fn main() {
    // The configuration for allmytoes
    // Usually, the defaults are fine.
    let configuration = AMTConfiguration::default();

    // An instance of allmytoes that can be used to provide thumbnails
    let amt = AMT::new(&configuration);

    // The file for which we want a thumbnail as a `std::path::Path`.
    let input_file = Path::new("/tmp/image.jpg");
    
    // The size of the thumbnail we want
    let thumb_size = ThumbSize::Large;

    // Get a thumbnail
    match amt.get(
        input_file,
        thumb_size,
    ) {
        Ok(thumb) => {
            println!("The thumb is here: {}", thumb.path)
        }
        Err(error) => println!(
            "Error '{:?}' occurred when trying to provide the thumb. ({})",
            error,
            error.msg(),
        ),
    }
}

License: GPL 3

AllMyToes is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

AllMyToes is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with AllMyToes. If not, see https://www.gnu.org/licenses/.