3 stable releases
1.1.0 | Sep 10, 2022 |
---|---|
1.0.1 | Sep 5, 2022 |
1.0.0 | Sep 4, 2022 |
#1483 in Development tools
53 downloads per month
120KB
2.5K
SLoC
monorail
Transform any git repository into a monorepo.
monorail
transforms any git repo into a trunk-based development monorepo. It uses a file describing the various directory paths and relationship between those paths, extracts changes from git since the last tag, and derives what has changed. These changes can then be fed to other programs (e.g. monorail-bash
) that can act on those changes. While monorail
currently supports only git
as the VCS backend, support for others could be added.
monorail
boils down to:
- A
Monorail.toml
file that describes your repository layout - the
monorail inspect change
command, which readsMonorail.toml
, analyzesgit
state between refs (usually between the most recent git annotated tag and HEAD), and returns what has changed - the
monorail-bash
program, which executes user-definedbash
scripts either directly, or informed bymonorail inspect change
output - the
monorail release
command, which creates annotated tags (essentially the "checkpoint" formonorail
change detection)
See the tutorial below for a practical walkthrough of how monorail
and monorail-bash
work.
Installation
Ensure the following are installed and available on your system path:
monorail
and all extensions can be installed from source by cloning the repository and executing the following, from the root of the repo:
./install.sh
By default, it will place these in /usr/local/bin
, if another location is preferred, use ./install.sh <destination>
.
Note that while monorail
can be installed from crates.io via cargo install
, cargo
does not support installing additional resources such as the script entrypoint for monorail-bash
. The install.sh
script handles both the binary and extension installation.
Commands
Use monorail help
Configuration
Create a Monorail.toml
configuration file in the root of the repository you want monorail
and extensions to use, referring to the Monorail.reference.toml
file for an annotated example.
Vocabulary
- project: a path to be developed/deployed/tested as a single unit
- group: a path containing a collection of projects and related configuration
- depend: a path that can be declared by a project as a dependency, causing any change on that path to be considered a change for projects that do
- link: a path that can be declared as an automatic dependency for all projects in a group, causing any change on that path to be considered a change for all projects
- target: an umbrella term for a path that can have changes detected, and commands run against it. One of group, project, depend, or link.
- extension: runs user-defined code written in a supported language; e.g.
bash
- command: a function, defined by a target, that can be invoked in an executor-dependent fashion
Tutorial
NOTE: this tutorial assumes a UNIX-like environment.
In this tutorial, you'll learn:
- how to declare a group, project, depend, and link
- how to inspect changes
- how to define commands
- how to execute commands
- how to release
First, create a fresh git
repository, and another to act as a remote:
NOTE: This assumes a init.defaultBranch
of master
or empty string, the default for git
. If yours is something else, change the master
in the git push
command to that.
git init monorail-tutorial
git init monorail-tutorial-remote
REMOTE_TRUNK=$(git -C monorail-tutorial-remote branch --show-current)
git -C monorail-tutorial-remote checkout -b x
pushd monorail-tutorial
git remote add origin ../monorail-tutorial-remote
git commit --allow-empty -m "HEAD"
git push --set-upstream origin $REMOTE_TRUNK
popd
NOTE: the commit is to create a valid HEAD reference, and the branch checkout in the remote is to avoid receive.denyCurrentBranch
errors from git
when we push during the tutorial.
To get started, generate a directory structure with the following shell commands:
cd monorail-tutorial
mkdir -p group1/project1
touch Monorail.toml
... which yields the following directory structure:
├── Monorail.toml
└── group1
└── project1
NOTE: the remainder of this tutorial will apply updates to the Monorail.toml
file with heredoc strings, for convenience.
Execute the following to specify the new group and project in Monorail.toml
, as well as an extension
to be used later in the tutorial:
cat <<EOF > Monorail.toml
[vcs]
use = "git"
[vcs.git]
trunk = "$(git branch --show-current)"
[extension]
use = "bash"
[[group]]
path = "group1"
[[group.project]]
path = "project1"
EOF
Recap
Your monorail
config file (default: Monorail.toml
) describes your existing repository layout in terms of monorail
concepts. A project
is a path to be developed/deployed/tested as a unit, e.g. a backend service, web app, etc. A group
is a set of related projects, and defines what can be shared amongst projects (more on sharing later).
Finally, many of monorail
s capabilities are path-based. Our definition of group.path
(relative to the repository root) and project.path
(relative to the specified group.path
) declare where these objects live in our repository.
Inspecting changes
monorail
will detect changes since the last release; see: Releasing. For git
, this means uncommitted, committed, and pushed files since the last annotated tag created by monorail release
.
Inspect showing no changes
Begin by viewing the output of inspect change
:
monorail inspect change | jq .
{
"group": {
"group1": {
"change": {
"file": [],
"project": [],
"link": [],
"depend": []
}
}
}
}
As expected there are no changes, and monorail
is able to interrogate git
and use the Monorail.toml
config successfully. The meaning of the file
, project
, link
, and depend
fields will be explained as the tutorial progresses.
Inspect showing a change
To show what an actual change looks like in monorail
output, create a new file in project1
:
touch group1/project1/foo.txt
View changes again:
monorail inspect change | jq .
{
"group": {
"group1": {
"change": {
"file": [
{
"name": "group1/project1/foo.txt",
"project": "group1/project1",
"action": "use",
"reason": "project_match"
}
],
"project": [
"group1/project1"
],
"link": [],
"depend": []
}
}
}
}
monorail
has identified that the newly added file represents a meaningful change, based on our configuration in Monorail.toml
.
The change.file
array contains a list of objects containing metadata about the change detected. It contains the name
of the file (a path relative to the repository root), the project
the file belongs to, the action
taken by monorail
during change detection (e.g. it was use
d), and the reason
that action
was taken (e.g. it matched a declared project).
The change.project
array contains a list of paths relative to the repo root for projects detected as changed. This list is de-duped across all change.file
entries; a project will appear at most once in this list.
Inspect showing a change, after commit or push
This understanding of what has changed persists between commits and pushes. Commit your changes:
git add * && git commit -am "x"
Then, view changes again:
monorail inspect change | jq .
{
"group": {
"group1": {
"change": {
"file": [
{
"name": "group1/project1/foo.txt",
"project": "group1/project1",
"action": "use",
"reason": "project_match"
}
],
"project": [
"group1/project1"
],
"link": [],
"depend": []
}
}
}
}
monorail
still knows about the change to the project group1/project1
.
Push, and view changes again:
git push && monorail inspect change | jq .
The output remains the same.
Declaring dependencies and links
monorail
allows for projects to depend on paths outside of the path
each project has declared. This allows for reference paths containing utility code, serialization files (e.g. protobuf definitions), configuration, etc. When these paths have changes, it triggers projects that depend on them to be changed.
Dependencies
Begin by creating a directory to be used as a dependency, and a directory to hold a new project, project2
:
mkdir -p group1/common/library1
mkdir group1/project2
Execute the following to adjust the [[group]]
section of Monorail.toml
to add library1
as a depend-able path, specify project2
, and make project2
depend on library1
:
cat <<EOF > Monorail.toml
[vcs]
use = "git"
[vcs.git]
trunk = "$(git branch --show-current)"
[extension]
use = "bash"
[[group]]
path = "group1"
depend = [
"common/library1"
]
[[group.project]]
path = "project1"
[[group.project]]
path = "project2"
depend = [
"common/library1"
]
EOF
The depend
declaration in the group
section indicates that this path can be depended on. The project.depend
is where you specify zero or more of these paths your project does depend on.
To trigger change detection, create a file in library1:
touch group1/common/library1/foo.proto
and then monorail inspect change | jq .
:
{
"group": {
"group1": {
"change": {
"file": [
{
"name": "group1/common/library1/foo.proto",
"project": null,
"action": "use",
"reason": "project_depend_effect"
},
{
"name": "group1/project1/foo.txt",
"project": "group1/project1",
"action": "use",
"reason": "project_match"
}
],
"project": [
"group1/project2",
"group1/project1"
],
"link": [],
"depend": [
"group1/common/library1"
]
}
}
}
}
Our original file entry is still there, but another for the newly-created file has appeared. It has a project
of null
because it does not lie in the path of a project, and a reason
that indicates it is being used due to a project depending on a path containing the file (project_depend_effect
).
An entry of group1/project2
has appeared in group.project
, indicating that this project is now part of the set of projects that have changed. We didn't change any files in project2
(indeed, none exist!), but did modify a path that project2
depends on.
Furthermore, an entry has appeared in group.depend
for our library path.
Links
A link
works similarly to a depend
, but applies to all projects in a group without them opting-in. To demonstrate, we will create a third project and a contrived Lockfile
to link all projects to:
mkdir group1/project3
touch group1/Lockfile
Execute the following to adjust the [[group]]
section of Monorail.toml
to specify this new project, as well as a group link
:
cat <<EOF > Monorail.toml
[vcs]
use = "git"
[vcs.git]
trunk = "$(git branch --show-current)"
[extension]
use = "bash"
[[group]]
path = "group1"
depend = [
"common/library1"
]
link = [
"Lockfile"
]
[[group.project]]
path = "project1"
[[group.project]]
path = "project2"
depend = [
"common/library1"
]
[[group.project]]
path = "project3"
EOF
Executing monocle inspect change | jq .
yields:
{
"group": {
"group1": {
"change": {
"file": [
{
"name": "group1/Lockfile",
"project": null,
"action": "use",
"reason": "group_link_effect"
},
{
"name": "group1/common/library1/foo.proto",
"project": null,
"action": "use",
"reason": "project_depend_effect"
},
{
"name": "group1/project1/foo.txt",
"project": "group1/project1",
"action": "use",
"reason": "project_match"
}
],
"project": [
"group1/project3",
"group1/project2",
"group1/project1"
],
"link": [
"group1/Lockfile"
],
"depend": [
"group1/common/library1"
]
}
}
}
}
Again, our original changes to project1
and library1
remain. A new change.file
for Lockfile
has appeared, a new change.project
for the project3
has been added, and change.link
now has a path to the Lockfile
we changed.
Note that project3
did not need to explicitly depend on Lockfile
; simply being a member of group1
does this.
Defining commands
Commands are run by extensions, which are "runners" for user-defined code. We have already specified the following in Monorail.toml
, so we will proceed with monorail-bash
:
[extension]
use = "bash"
Commands are stored in a file on a per-target basis, the path to which is defined in Monorail.toml
. In our case, that path will be support/script/monorail-exec.sh
(the default) relative to group1/project1
.
Create the path to this file with:
mkdir -p group1/project1/support/script
In the group1/project1/support/script/monorail-exec.sh
file, we will define a script containing three commands:
cat <<"EOF" > group1/project1/support/script/monorail-exec.sh
#!/usr/bin/env bash
function command1() {
echo "Hello, from command1"
echo "The calling environment is inherited: ${SOME_EXTRA_VAR}"
echo "some data" > side_effects.txt
}
function command2() {
echo "Hello, from command2"
cat side_effects.txt
}
function setup() {
echo "Installing everything you need"
}
EOF
Command names can be named any valid UTF-8 string, and are free to do anything a normal bash
script can do: source other scripts, call external build tools, perform network requests, etc. One of the benefits of monorail
is that it does not limit the build tooling you can use.
Executing commands
With the command script defined, it can be called with monorail-bash exec
. This can be done in one of two ways:
- implicitly, from the change detection output of
monorail
- explicitly, by specifying a list of targets
Implicit
When done implicitly, monorail-bash exec
uses the same processes that power monorail inspect change
to derive changed targets and execute commands against them. To illustrate this use the following (SOME_EXTRA_VAR
just shows that parent shell values can be passed to commands):
SOME_EXTRA_VAR=foo monorail-bash -v exec -c command1 -c command2
Sep 10 07:34:07 monorail-bash : 'monorail' path: monorail
Sep 10 07:34:07 monorail-bash : 'jq' path: jq
Sep 10 07:34:07 monorail-bash : 'git' path: git
Sep 10 07:34:07 monorail-bash : use libgit2 status: false
Sep 10 07:34:07 monorail-bash : 'monorail' config: Monorail.toml
Sep 10 07:34:07 monorail-bash : working directory: /Users/patrick/lab/github.com/pnordahl/monorail-tutorial
Sep 10 07:34:07 monorail-bash : command: command1
Sep 10 07:34:07 monorail-bash : command: command2
Sep 10 07:34:07 monorail-bash : start:
Sep 10 07:34:07 monorail-bash : end:
Sep 10 07:34:07 monorail-bash : target (inferred): group1/Lockfile
Sep 10 07:34:07 monorail-bash : target (inferred): group1/common/library1
Sep 10 07:34:07 monorail-bash : target (inferred): group1/project2
Sep 10 07:34:07 monorail-bash : target (inferred): group1/project3
Sep 10 07:34:07 monorail-bash : target (inferred): group1/project1
Sep 10 07:34:07 monorail-bash : NOTE: Ignoring command for non-directory target; command: command1, target: group1/Lockfile
Sep 10 07:34:07 monorail-bash : NOTE: Script not found; command: command1, target: group1/common/library1
Sep 10 07:34:07 monorail-bash : NOTE: Script not found; command: command1, target: group1/project2
Sep 10 07:34:07 monorail-bash : NOTE: Script not found; command: command1, target: group1/project3
Sep 10 07:34:07 monorail-bash : Executing command; command: command1, target: group1/project1
Hello, from command1
The calling environment is inherited: foo
Sep 10 07:34:07 monorail-bash : NOTE: Ignoring command for non-directory target; command: command2, target: group1/Lockfile
Sep 10 07:34:07 monorail-bash : NOTE: Script not found; command: command2, target: group1/common/library1
Sep 10 07:34:07 monorail-bash : NOTE: Script not found; command: command2, target: group1/project2
Sep 10 07:34:07 monorail-bash : NOTE: Script not found; command: command2, target: group1/project3
Sep 10 07:34:07 monorail-bash : Executing command; command: command2, target: group1/project1
Hello, from command2
some data
The majority of this output is workflow and debugging information, but it's worth noting a few key pieces.
- commands were executed in the order specified by the
-c
options. - paths we specified as
depend
andlink
entries could have specified their own implementations of thecommand1
andcommand2
commands - paths that point to individual files cannot specify a command implementation (e.g. our Lockfile)
- paths without implementations for
command1
and/orcommand2
were noted and ignored
Executing arbitrary bash functions against the changes detected by monorail
has a number of applications, including:
- executing commands against all projects/dependencies/links you've modified, without specifically targeting them;
monorail-bash
ensures that for each changed target, the requested commands are executed sequentially - running specific commands against all changed targets as part of CI (e.g.
check
,build
,test
,deploy
, etc.)
Explicit
Manually selecting targets gives one the ability to execute commands independent of VCS change detection. Applications include:
- getting new developers up to speed working on a codebase, as one can define all setup code in a set of commands and execute it against the target, e.g. a
bootstrap
command - run any command for any target in the entire repo, as desired
To illustrate manually selecting targets, we will run the setup
command we defined but did not execute previously. Execute the following (removing the -v
to cut down on the visual noise):
monorail-bash exec -t group1/project1 -c setup
Installing everything you need
For more information, execute monorail-bash -h
, and monorail-bash exec -h
.
Releasing
monorail
uses the backend VCS native mechanisms, e.g. tags in git
as "release" markers. This creates a "checkpoint" for change detection. Without release tags, monorail
is forced to search git
history back to the first commit. This would be ineffecient and make change detection useless as all targets would be considered changed over a long enough timeline.
When a release is performed, it applies to all targets that were changed since the previous release (or the first commit of the repository, if no releases yet exist).
First, let's commit and push our current changes:
git add * && git commit -am "update commands" && git push
Assuming that we have committed all that we intend to, and target commands have been run to our satisfaction (e.g. CI has passed for the merge of our branch), we can dry-run a patch release with:
monorail release --dry-run -t patch | jq .
{
"id": "v0.0.1",
"targets": [
"group1/Lockfile",
"group1/common/library1",
"group1/project1",
"group1/project2",
"group1/project3"
],
"dry_run": true
}
monorail
creates releases with an id
appropriate to the conventions of the chosen VCS; in this case, that is the git
semver tagging format. It also embeds the list of targets included as part of this release in the targets
array; in the case of git
, it will embed this list of targets in the release message.
Now, run a real release:
monorail release -t patch | jq .
{
"id": "v0.0.1",
"targets": [
"group1/Lockfile",
"group1/common/library1",
"group1/project1",
"group1/project2",
"group1/project3"
],
"dry_run": false
}
To show that the release cleared out monorail
s view of changes, execute:
monorail inspect change | jq .
{
"group": {
"group1": {
"change": {
"file": [],
"project": [],
"link": [],
"depend": []
}
}
}
}
Finally, our newly-pushed tag is now in the remote. To see this, execute:
git -C ../monorail-tutorial-remote show -s --format=%B v0.0.1
... which outputs
tag v0.0.1
Tagger: you <email@domain.com>
group1/Lockfile
group1/common/library1
group1/project1
group1/project2
group1/project3
This concludes the tutorial. Now that you have seen how the core concepts of monorail
and extensions work, you're ready to use it in real projects. Experiment with repository layouts, commands, CI, and working on a trunk-based development workflow that works for your teams.
Refer to Monorail.reference.toml
for more specific information about the various configuration available for monorail
and extensions.
Invariants
In order to work, monorail
requires the following invariants be satisfied:
- groups may not reference paths in other groups
- targets may not reference paths in other targets
- only paths specified in a group's
link
, ordepend
configuration may be shared between projects of that group
If any of these are violated, the behavior of monorail
commands that analyze changes to the repository is undefined.
Development setup
This will build the project and run the tests:
cargo build
cargo test -- --nocapture
You can use install.sh
to build a release binary of monorail
and copy it, along with extensions, into your PATH.
Dependencies
~16MB
~346K SLoC