Literate programming toolsuite

This is a suite of programs to use for literate programming. They try to make this process as smooth as possible.

Basic design decisions

This software is written in rust. It uses it's own tools to gererate the rust sourcecode.

Every logical part is seperated in its own subcrate. Most of those subcrates define a cmdline util of the same name in this crate. In everey subcrate folder we have a <name>.adoc file which is the main source for that subcrate.

The subrates are:

asciidoctrine:: An extensible asciidoctrine/asciidoctrine.adoc#, asciidoc interpreter. It isn't very mature right now. lisi:: A flexible lisi/lisi.adoc#,literate programming tool for both tangle and weave. It has some basic functionality right now. ansicht:: A ansicht/ansicht.adoc#,visual programming tool primerily meant to use in conjunction with literate programming tools. Currently just a draft. dolmetscher:: A <<dolmetscher/dolmetscher.adoc#,translation helper tool>>. Currently just a draft.

I think asciidoc is superior to markdown so we use this as our basic input format.

Install

To install and use this software there are some different options:

//Install from your distro package manager:: Download precompiled binaries:: For windows you can download https://nightly.link/kober-systems/literate_programming_toolsuite/actions/runs/8063422942/lisi-Windows.zip[latest Windows binaries] built through github actions Use the docker image:: You can use the precompiled image lammermann/lisi from Dockerhub. If you want to modify the code for the Dockerfile itself you can find the code in a dockerfile, seperate section. Install from cargo:: run [[install_with_cargo]]cargo install --force --path lisi Build from source:: This makes most sense if you want to contribute to the project or otherwise modify the code for personal use. You can the find instructions build_instructions, later in this document. NixOS/nix:: install::: [[install_with_nix]]nix-env -i -f lisi.nix build locally::: nix-build lisi.nix

[[build_instructions]]

Build from source

If you want to modify how the programs work to suit your usecase you will need build and compile the code.

There are a few steps to build the toolsuite.

[source, sh, save] .build.sh

set -e # <4>

[check_generated_sources_are_vcs_clean](#check_generated_sources_are_vcs_clean) # <1>

[generate_sources](#generate_sources)
[build_and_test](#build_and_test)
[update_docs](#update_docs)

[ask_for_vcs_check_in](#ask_for_vcs_check_in) # <2>

[publish_on_crates_io](#publish_on_crates_io)

[ask_for_local_install](#ask_for_local_install) # <3>

<1> If we have edited some of the generated sources by hand we may don't want them to be overwritten by our build process before we merged them back into the asciidoc source. <2> Once our build succeds there is a good chance we want to save the changes to our vcs. <3> If everything went good we may want to install our new version of the toolsuite locally. <4> When a step fails we want the script to end

Prerequisites

There must be some programms installed on your computer to release a version of the literate programming toolsuite.

git:: It is currently published on github so we need git. rust, cargo, etc:: These porgramms are written in rust therefore we need the rust toolchain. asciidoctor:: Right now we need asciidcotor to build the docs. In future versions it is planned to use our own asciidoctrine programm. There are some additional dependencies for our doc generation process. asciidcotor-diagram::: For our diagrams. ditaa::: For our diagrams. pygments::: For source code highlighting.

Check if generated files are untouched in VCS

Maybe we have edited some of our generated source files by hand. In that case we would like to be warned before generating the files. This way we could check if we want to put our changes into our asciidoc sources or don't care if they are overwritten.

We assume that all changed files that are not asciidoc files are potentially in danger to be overwritten.

[[check_generated_sources_are_vcs_clean]] [source, sh]

changed_files=`git diff --name-only | grep --invert-match '\.adoc$'`
if [[ ! -z "$changed_files" ]]; then
  while true; do
      echo "some files are modified"
      echo "$changed_files"
      read -p "Do you wish continue anyway? [yes|no] " yn
      case $yn in
          [Yy]* ) break;;
          [Nn]* ) exit;;
          * ) echo "Please answer yes or no.";;
      esac
  done
fi

Generate source files

[[generate_sources]] [source, sh]

echo "Start generating source files ..."

cd asciidoctrine/
lisi -o ../docs/asciidoctrine/asciidoctrine.lisi.html asciidoctrine.adoc \
  || echo "lisi is currenty not installed"
cd ..

cd lisi
lisi -o /dev/null lisi.adoc || echo "lisi is currenty not installed" # <1>
# The new generated source must be able to
# generate itself
cargo run --manifest-path ../Cargo.toml --bin lisi -- -o lisi.html lisi.adoc
cd ..

cargo run --bin lisi -- -o /dev/null README.adoc # <2>

echo "Generating source files done!"

<1> We use a preinstalled version of lisi to build the sources. This helps us if theres a bug in our generated sources. If we have no version of lisi installed yet theres no problem the script will just give us a warning and generate the sources in the next step. <2> Since lisi is currently unable to evaluate scripts with user cmdline input we need to refresh the build script regulary.

Build and test

[[build_and_test]] [source, sh]

cargo test

Build websites for github pages

TODO later we want to do this with asciidoctrine alone.

[[update_docs]] [source, sh]

echo "Start generating html files ..."

asciidoctor \
            [asciidoctor-styles](#asciidoctor-styles)
            -D docs \
            README.adoc -o index.html
asciidoctor \
            [asciidoctor-styles](#asciidoctor-styles)
            -D docs/lisi \
            lisi/lisi.adoc
asciidoctor \
            [asciidoctor-styles](#asciidoctor-styles)
            -D docs/asciidoctrine \
            asciidoctrine/asciidoctrine.adoc
asciidoctor \
            [asciidoctor-styles](#asciidoctor-styles)
            -D docs/ansicht \
            ansicht/ansicht.adoc
asciidoctor \
            [asciidoctor-styles](#asciidoctor-styles)
            -D docs/dolmetscher \
            dolmetscher/dolmetscher.adoc

echo "Generating html files done!"

We have some general styles that should be equal in all of our files:

[[asciidoctor-styles]] [source, sh]

-r asciidoctor-diagram \
-a source-highlighter=pygments \
-a toc=left \
-a icons=font \
-a toclevels=4 \
-a data-uri \
-a reproducible \

Hacking Guide

If you want to modify the code to fit your own needs you could follow this describes my process of doing it:

Changing the sources:: This is a literate program so the source of truth here is the asciidoc document. However I often consider the literate document and the generated source code as to differnt views on the same programm. To keep them both in sync I make sure to commit any changes I make at the generated source code to the VCS before I regenerate the source code from the literate source. Than I can view the diff between the generated source and my own modified version and change the literate sources accodingly until the generated code does not differ anymore from the one in VCS. + When the two sources are in sync, than I can modify the literate sources however I like and direkty regenerate the source code. Compile:: I compile and test in a loop during the whole modification process. For this I use watchexec as my own poor mans ci. Commit:: When the bug is fixed or the feature is implemented etc I commit my modifications to VCS. Check:: After some time I reach a point where I want to release. Here I do the following: Check the literate and generated sources are in sync. Rerender the docs. Push to github. Let the ci do his work.

Poor mans ci

Whenever the rust files change we want to rebuild and test the program.

[source, sh, save] .auto_build_loop.sh

watchexec -w . -c -e rs,toml --no-vcs-ignore -- "[build_and_test](#build_and_test)"

Whenever the asciidoc files change we want to regenerate the source files. But before we can do that we have to make sure the aktual source files and the generated source files are in sync.

TODO This isn't easy because when the asciidoc files are moified they generate "dirty" source files (from the perspective of the VCS). So we have to stash the changes somehow so that it is seen, that the sources are in sync.

Ask for checkin into the VCS

[[ask_for_vcs_check_in]] [source, sh]

while true; do
    git diff; # <1>
    read -p "Do you wish to commit your changes to git? [yes|no] " yn
    case $yn in
        [Yy]* )
          git add .; # <2>
          git commit; # <2>
          break;;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac
done

<1> Before we commit everything we should do a last review. <2> Normally we know what we do and can just add everything and go on, but if we saw something in the commit that we don't want to include we should stop before we submit the commit (by letting the commit message empty or by changing the included chunks in another shell).

Install the tools on our computer

[[ask_for_local_install]] [source, sh]

while true; do
    read -p "Do you wish to install this program? [yes|no] " yn
    case $yn in
        [Yy]* )
          [install_with_nix](#install_with_nix);
          break;;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac
done

Review process

Whenever the code is updated on the master branch or via pull request it will be reviewed. The main part of that review follows the guidelines documented in this section.

TODO github actions

Release

When everything goes well it's time to think about releasing.

[[publish_on_crates_io]] [source, sh]

cargo publish --dry-run -p asciidoctrine

while true; do
    read -p "Do you wish to publish asciidoctrine? [yes|no] " yn
    case $yn in
        [Yy]* )
          cargo login;
          cargo publish --dry-run -p asciidoctrine;
          break;;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac
done

cargo publish --dry-run -p lisi

while true; do
    read -p "Do you wish to publish lisi? [yes|no] " yn
    case $yn in
        [Yy]* )
          cargo login;
          cargo publish --dry-run -p lisi;
          break;;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac
done

[[dockerfile]]

Dockerfile

To use lisi in ci-scripts (at least thats my usecase) it is very handy to have a docker image at hand. However when it comes to docker images size is key. For this reason we use to multiple different stages in our Dockerfile:

• one that builds our software and has all the build dependencies
• one that only holds our final binaries and minimal runtime dependencies to enable a small image size.

[source, Dockerfile, save] .Dockerfile

[docker_build_step](#docker_build_step)

[docker_final_image_step](#docker_final_image_step)

To build binaries that later have minimal runtime dependencies we use the musl target which lets us build statically compiled binaries. To do this we use the rust docker image based on alpine

[[docker_build_step]] [source, Dockerfile]

FROM rust:alpine AS builder

RUN apk --no-cache add g++ # <1>

WORKDIR /home/rust/
COPY . .
RUN cargo test
RUN cargo build --release

RUN strip target/release/lisi # <2>

<1> We can only compile on this system if we have g++ installed for some weird reason I don't understand (see the related bug here). <2> After building the binaries we can shrink down the size significantly by striping them.

After we build our program we take a fresh image based on alpine (becase it's small) and copy only our binaries over.

[[docker_final_image_step]] [source, Dockerfile]

FROM alpine:latest
WORKDIR /home/lisi
COPY --from=builder /home/rust/target/release/lisi .
ENV PATH="${PATH}:/home/lisi"

TODO github actions for docker image