5 releases (3 unstable)
1.0.0-rc.1 | Feb 26, 2024 |
---|---|
1.0.0-rc.0 | Feb 18, 2024 |
0.5.1 | Feb 16, 2024 |
0.5.0 | Feb 15, 2024 |
0.4.0 | Feb 1, 2024 |
#450 in Development tools
35 downloads per month
110KB
2K
SLoC
🚀 Read the full Documentation 🚀 | ✨ See usage Examples ✨
Click on the animation to see it on asciinema.org!
Overview
Kerblam! has the following commands:
- 🪄
kerblam new
can be used to create a new kerblam! project. Kerblam! asks you if you want to use some common programming languages and sets up a proper.gitignore
for you. - 🚀
kerblam run
executes the analysis for you, by choosing yourmakefile
s and containers appropriately and building container images as needed. Optionally, allows test data or alternative data to be used instead of real data, in order to test your pipelines. - 🎁
kerblam package
packages your pipelines and exports a container image for execution later plus a tarball with all the data a reproducer needs to run the analysis again. - ♻️
kerblam replay
replays packaged pipelines from the tarball made bykerblam package
. It automatically tries to fetch the same container that was made when the tarball was packaged. - 📦
kerblam data
fetches remote data and saves it locally, manages local data and can clean it up, preserving only files that must be preserved. It also shows you how much local data is on the disk, how much data is remote and how much disk space you can free without losing anything important. It can also export the important data to share it with colleagues quickly. - ✂️
kerblam ignore
can edit your.gitignore
file by adding files, folders and GitHub's recommended ignores for specific languages in just one command.
Kerblam! is not and does not want to be:
- 🚱 A pipeline manager like
snakemake
andnextflow
;- It supports and helps you execute pipelines written in other formats, but it does not interfere from then on;
- You can however use
snakemake
,nextflow
or any other program in conjunction with Kerblam! by writing shell pipelines.
- ♻️ A replacement for any of the tools it leverages (e.g.
git
,docker
orpodman
,pre-commit
, etc...); - 🔎 Something that insulates you from the nuances of writing good, correct
pipelines and container recipies.
Specifically, Kerblam! will never:
- Parse your
.gitignore
,.dockerignore
, pipes or container recipies to check for errors or potential issues; - Edit code for you (with the exception of a tiny bit of wrapping to allow
kerblam package
to work); - Handle any errors produced by the pipelines or containers.
- Parse your
- 🌍 A tool that covers every edge case. Kerblam! will never have a wall of options for you to choose from. If you need more advanced control on what is done, you should directly use the tools that Kerblam! leverages.
Opinions
Kerblam! wants to streamline and standardize data analysis project as much as possible. For this reason, projects are opinionated:
- The folder structure of your project adheres to the Kerblam! standard,
although you may configure it in
kerblam.toml
. Read more about it here. - You use
make
or bash scripts as your pipeline manager.- Kerblam! natively uses
make
, but nothing stops you writing shell files that execute other tools, likesnakemake
.
- Kerblam! natively uses
- You use
docker
orpodman
as your virtualisation service. - You use
git
as your version control system. - You generally execute your pipelines in a container, and not in your development environment.
- Most of your input data is remotely downloadable, especially for large and bulky files.
If you don't like this setup, Kerblam! is probably not for you.
[!TIP] If you wish to learn more on why these design choices were made, please take a look at the kerblam! philosophy.
Documentation
The full Kerblam! documentation is online at kerblam.dev 🚀. Please take a look there for more information on what Kerblam! can do. For example, you might find the tutorial interesting.
Installation
In short, use a unix-compatible OS and either:
# Install a prebuilt binary
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/MrHedmad/kerblam/releases/latest/download/kerblam-installer.sh | sh
or, alternatively
# Install from source with rustc and cargo
cargo install kerblam
You will need git
, make
and docker
or podman
installed for
Kerblam! to work.
Contributing
To contribute, please take a look at the contributing guide.
Code is not the only thing that you can contribute. Written a guide? Considered a new feature? Wrote some docstrings? Found a bug? All of these are meaningful and important contributions. For this reason, all contributors are listed in the contributing guide.
If you use Kerblam! or want to add your opinion to the direction it is taking, take a look at the issues labelled with RFC. They are requests for comments where you can say your opinion on new features.
Thank you for taking an interest in Kerblam! Any help is greatly appreciated.
Licensing and citation
Kerblam! is licensed under the MIT License. If you wish to cite Kerblam!, please provide a link to this repository or use the Zenodo DOI 10.5281/zenodo.10664806
Naming
This project is named after the fictitious online shop/delivery company in S11E07 of Doctor Who. Kerblam! might be referred to as Kerblam!, Kerblam or Kerb!am, interchangeably, although Kerblam! is preferred. The Kerblam! logo is written in the Kwark Font by tup wanders.
And remember! If you want it...
Dependencies
~15–32MB
~510K SLoC