10 stable releases
1.5.4 | Oct 30, 2024 |
---|---|
1.5.3 | Dec 21, 2023 |
1.5.1 | Mar 3, 2021 |
1.4.1 | Dec 30, 2020 |
1.1.2 | Mar 15, 2018 |
#55 in Debugging
47 downloads per month
150KB
4K
SLoC
cntr
Say no to $ apt install vim
in containers!
cntr
is a replacement for docker exec
that brings all your developers tools with you.
This is done by mounting the file system from one container or the host into the target container
by creating a nested container with the help of a FUSE filesystem.
This allows to ship minimal runtime image in production and limit the surface for exploits.
Cntr was also published in Usenix ATC 2018. See bibtex for citation.
Demo
In this two minute recording you learn all the basics of cntr:
Features
- For convenience cntr supports container names/identifier for the following container engines natively:
- docker
- podman
- LXC
- LXD
- rkt
- systemd-nspawn
- containerd
- For other container engines cntr also takes process ids (PIDs) instead of container names.
Installation
Cntr can be only supports linux.
Pre-build static-linked binary
For linux x86_64 we build static binaries for every release. More platforms can added on request. See the release tab for pre-build tarballs. At runtime only commandline utils of the container engine in questions are required.
Build from source
All you need for compilation is rust + cargo. Checkout rustup.rs on how to get a working rust toolchain. Then run:
Either:
$ cargo install cntr
Or the latest master:
$ cargo install --git https://github.com/Mic92/cntr
For offline builds we also provided a tarball with all dependencies bundled here for compilation with cargo-vendor.
Usage
At a high-level cntr provides two subcommands: attach
and exec
:
attach
: Allows you to attach to a container with your own native shell/commands. Cntr will mount the container at/var/lib/cntr
. The container itself will run unaffected as the mount changes are not visible to container processes.- Example:
cntr attach <container_id>
wherecontainer_id
can be a container identifier or process id (see examples below).
- Example:
exec
: Once you are in the container, you can also run commands from the container filesystem itself. Since those might need their native mount layout at/
instead of/var/lib/cntr
, cntr providesexec
subcommand to chroot to container again and also resets the environment variables that might have been changed by the shell.- Example:
cntr exec <command>
wherecommand
is an executable in the container
- Example:
Note: Cntr needs to run on the same host as the container. It does not work if the container is running in a virtual machine while cntr is running on the hypervisor.
$ cntr --help
Cntr 1.5.1
Jörg Thalheim <joerg@thalheim.io>
Enter or executed in container
USAGE:
cntr <SUBCOMMAND>
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
attach Enter container
exec Execute command in container filesystem
help Prints this message or the help of the given subcommand(s)
$ cntr attach --help
cntr-attach 1.5.1
Jörg Thalheim <joerg@thalheim.io>
Enter container
USAGE:
cntr attach [OPTIONS] <id> [command]...
FLAGS:
-h, --help Prints help information
OPTIONS:
--effective-user <EFFECTIVE_USER> effective username that should be owner of new created files on the host
-t, --type <TYPE> Container types to try (sperated by ','). [default: all but command]
[possible values: process_id, rkt, podman, docker, nspawn, lxc, lxd,
containerd, command]
ARGS:
<id> container id, container name or process id
<command>... Command and its arguments to execute after attach. Consider prepending it with '-- ' to prevent
parsing of '-x'-like flags. [default: $SHELL]
$ cntr exec --help
cntr-exec 1.5.1
Jörg Thalheim <joerg@thalheim.io>
Execute command in container filesystem
USAGE:
cntr exec [command]...
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
ARGS:
<command>... Command and its arguments to execute after attach. Consider prepending it with '-- ' to prevent
parsing of '-x'-like flags. [default: $SHELL]
Docker
1: Find out the container name/container id:
$ docker run --name boxbusy -ti busybox
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55a93d71b53b busybox "sh" 22 seconds ago Up 20 seconds boxbusy
Either provide a container id...
$ cntr attach 55a93d71b53b
[root@55a93d71b53b:/var/lib/cntr]# echo "I am in a container!"
[root@55a93d71b53b:/var/lib/cntr]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
40: eth0@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
[root@55a93d71b53b:/var/lib/cntr]# vim etc/resolv.conf
...or the container name.
Use cntr exec
to execute container native commands (while running in the cntr shell).
$ cntr attach boxbusy
[root@55a93d71b53b:/var/lib/cntr]# cntr exec -- sh -c 'busybox | head -1'
You can also use Dockerfile from this repo to build a docker container with cntr:
$ docker build -f Dockerfile . -t cntr
# boxbusy here is the name of the target container to attach to
$ docker run --pid=host --privileged=true -v /var/run/docker.sock:/var/run/docker.sock -ti --rm cntr attach boxbusy /bin/sh
Podman
See docker usage, just replace docker
with the podman
command.
LXD
1: Create a container and start it
$ lxc image import images:/alpine/edge
$ lxc launch images:alpine/edge
$ lxc list
+-----------------+---------+------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-----------------+---------+------+------+------------+-----------+
| amazed-sailfish | RUNNING | | | PERSISTENT | 0 |
+-----------------+---------+------+------+------------+-----------+
2: Attach to the container with cntr
$ cntr attach amazed-sailfish
$ cat etc/hostname
amazed-sailfish
LXC
1: Create a container and start it
$ lxc-create --name ubuntu -t download -- -d ubuntu -r xenial -a amd64
$ lxc-start --name ubuntu -F
...
Ubuntu 16.04.4 LTS ubuntu console
ubuntu login:
$ lxc-ls
ubuntu
2: Attach to container with cntr:
$ cntr attach ubuntu
[root@ubuntu2:/var/lib/cntr]# cat etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
rkt
1: Find out the container uuid:
$ rkt run --interactive=true docker://busybox
$ rkt list
UUID APP IMAGE NAME STATE CREATED STARTED NETWORKS
c2d2e87e busybox registry-1.docker.io/library/busybox:latest running 6 minutes ago 6 minutes ago default:ip4=172.16.28.3
2: Attach with cntr
# make sure your container is still running!
$ cntr attach c2d2e87e
# Finally not the old ugly top!
[gen0@rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017:/var/lib/cntr]# htop
...
With cntr you can also debug stage1 of rkt - even there is no support from rkt itself.
$ ps aux | grep stage1
joerg 13546 0.0 0.0 120808 1608 pts/12 S+ 11:10 0:00 grep --binary-files=without-match --directories=skip --color=auto stage1
root 22232 0.0 0.0 54208 2656 pts/7 S+ 10:54 0:00 stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=c2d2e87e-e798-4341-ae93-26f6cbb7c017 --machine=rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017 --directory=stage1/rootfs --capability=CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FSETID,CAP_FOWNER,CAP_KILL,CAP_MKNOD,CAP_NET_RAW,CAP_NET_BIND_SERVICE,CAP_SETUID,CAP_SETGID,CAP_SETPCAP,CAP_SETFCAP,CAP_SYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0
Therefore we use the process id instead of the container uuid:
$ cntr attach 22232
# new and exiting territory!
[root@turingmachine:/var/lib/cntr]# mount | grep pods
sysfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
systemd-nspawn
1: Start container
$ wget https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-root.tar.xz
$ mkdir /var/lib/machines/ubuntu
$ tar -xf ubuntu-16.04-server-cloudimg-amd64-root.tar.xz -C /var/lib/machines/ubuntu
$ systemd-nspawn -b -M ubuntu
$ machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
ubuntu container systemd-nspawn ubuntu 16.04 -
2: Attach
$ cntr attach ubuntu
Generic process id
The minimal information needed by cntr is the process id of a container process you want to attach to.
# Did you now chromium uses namespaces too?
$ ps aux | grep 'chromium --type=renderer'
joerg 17498 11.7 1.0 1394504 174256 ? Sl 15:16 0:08 /usr/bin/chromium
In this case 17498 is the pid we are looking for.
$ cntr attach 17498
# looks quite similar to our system, but with less users
[joerg@turingmachine cntr]$ ls -la /
total 240
drwxr-xr-x 23 nobody nogroup 23 Mar 13 15:05 .
drwxr-xr-x 23 nobody nogroup 23 Mar 13 15:05 ..
drwxr-xr-x 2 nobody nogroup 3 Mar 13 15:14 bin
drwxr-xr-x 4 nobody nogroup 16384 Jan 1 1970 boot
drwxr-xr-x 24 nobody nogroup 4120 Mar 13 14:56 dev
drwxr-xr-x 52 nobody nogroup 125 Mar 13 15:14 etc
drwxr-xr-x 3 nobody nogroup 3 Jan 8 16:17 home
drwxr-xr-x 8 nobody nogroup 8 Feb 9 22:10 mnt
dr-xr-xr-x 306 nobody nogroup 0 Mar 13 09:38 proc
drwx------ 22 nobody nogroup 43 Mar 13 15:09 root
...
Containerd
For containerd integration the ctr
binary is required. You can get a binary by running:
$ GOPATH=$(mktemp -d)
$ go get github.com/containerd/containerd/cmd/ctr
$ $GOPATH/bin/ctr --help
Put the resulting ctr
binary in your $PATH
1: Start container
$ ctr images pull docker.io/library/busybox:latest
$ ctr run docker.io/library/busybox:latest boxbusy
$ ctr tasks lists
TASK PID STATUS
boxbusy 24310 RUNNING
2: Attach
$ cntr attach boxbusy
It's also possible to run cntr from a container itself. This repository contains a example Dockerfile for that:
$ docker build -f Dockerfile.example . -t cntr
$ docker save cntr > cntr.tar
$ ctr images import --base-name cntr ./cntr.tar
In this example we attach to containerd by process id. The process id of a task is given in ctr tasks list
.
$ ctr run --privileged --with-ns pid:/proc/1/ns/pid --tty docker.io/library/cntr:latest cntr /usr/bin/cntr attach 31523 /bin/sh
To resolve containerd names one also would need to add the ctr
binary (~12mb) to the Dockerfile.
Additional Config
ZFS
cntr
requires POSIX ACLs be enabled under ZFS. By default, Linux ZFS doesn't have POSIX ACLs enabled. This results in
the following error when trying to attach
:
unable to move container mounts to new mountpoint: EOPNOTSUPP: Operation not supported on transport endpoint
To enable POSIX ACLs on the ZFS dataset:
$ zfs set acltype=posixacl zpool/media
$ zfs set xattr=sa zpool/media # optional, but encouraged for best performance
How it works
Cntr is container-agnostic: Instead of interfacing with container engines, it implements the underlying operating system API. It treats every container as a group of processes, that it can inherit properties from.
Cntr inherits the following container properties:
- Namespaces (mount, uts, pid, net, cgroup, ipc)
- Cgroups
- Apparamor/selinux
- Capabilities
- User/group ids
- Environment variables
- The following files: /etc/passwd, /etc/hostname, /etc/hosts, /etc/resolv.conf
Under the hood it spawns a shell or user defined program that inherits the full context of the container and mount itself as a fuse filesystem.
We extensively evaluated the correctness and performance of cntr's filesystem using xfstests and a wide range of filesystem performance benchmarks (iozone, pgbench, dbench, fio, fs-mark, postmark, ...)
Related projects
- nsenter
- Only covers linux namespaces and the user is limited to tools installed in the containers
- toolbox
- Does attach from a container to the host, this is the opposite of what Cntr is doing
Bibtex
We published a paper with all technical details about Cntr in Usenix ATC 2018.
@inproceedings{cntr-atc18,
author = {J{\"o}rg Thalheim and Pramod Bhatotia and Pedro Fonseca and Baris Kasikci},
title = {Cntr: Lightweight {OS} Containers},
booktitle = {2018 {USENIX} Annual Technical Conference ({USENIX} {ATC} 18)},
year = {2018},
}
Dependencies
~4–9.5MB
~102K SLoC