detached b5ce612

docs(readme): document sandboxes, configuration and integration tests

commit
b5ce612a68e5937ac9fd519760af51b6d3d964e7
author
brice · · 2026-10-12 15:35 UTC
parent
ea1f07e
signature
GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email
3 files · +192 -2
M README.md +84 -2
@@ -5,9 +5,91 @@ The CI runner of [ermite](https://git.bnema.dev/bnema/ermite). It runs workflows
55 - **`vm`**: a Firecracker micro-VM per job, destroyed at the end. Needs `/dev/kvm`.
66 - **`container`**: a container per job with its own Docker daemon, for hosts without KVM. Uses the sysbox runtime when available, else a privileged container.
77
8-`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when KVM is usable, else `container`.
8+`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when `/dev/kvm` opens read-write and Firecracker, the guest kernel and the base image exist, else `container`. The chosen mode and the reasons are logged at start.
99
10-Status: early development.
10+Status: early development. This tree has the job sandboxes (phase D of the CI-1 plan) with a stub executor; the host loop that fetches jobs from ermite comes later.
11+
12+## Sandboxes
13+
14+Both modes run the same executable, the runner's own, attached **read-only** to the sandbox (never baked into the image, so a runner upgrade needs no image rebuild): `ermite-runner init` is the entrypoint (PID 1 of a VM), then `ermite-runner exec` connects back to the host over a framed protobuf stream (`internal/execstream/exec.proto`; vsock in a VM, a Unix socket in a container). The executable must be static: build with `CGO_ENABLED=0`.
15+
16+### vm
17+
18+- Firecracker is driven through its HTTP API on a unix socket with `net/http`, not through `firecracker-go-sdk` (its last release, v1.0.0 from 2022, predates Firecracker 1.17 and pulls CNI, go-openapi and netlink for a handful of PUT requests).
19+- Disks: the runner executable (read-only boot root, ext4 built with `mkfs.ext4 -d`, cached by the executable's hash), the shared read-only base image, and a per-job sparse scratch disk (`ERMITE_RUNNER_JOB_DISK`). The guest init builds an overlayfs root (lower = base, upper = scratch) and keeps dockerd's data root on the scratch disk.
20+- The jailer runs each VM as uid/gid `ERMITE_RUNNER_UID_BASE` + slot in its own chroot under the state directory. The kernel, the base image and the state directory must be on the same filesystem (files are hard-linked into the jail) and the kernel and base image must be world-readable.
21+- One TAP (`ertap<slot>`) per VM with a /30 from `ERMITE_RUNNER_NETWORK` (default `10.200.0.0/16`), vsock for the executor stream, IPv6 disabled in the guest.
22+- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires (the plan runs the host process as root under systemd).
23+
24+### container
25+
26+- Docker Engine API via the maintained client `github.com/moby/moby/client` (`DOCKER_HOST` honoured). Runtime `sysbox-runc` when the daemon lists it, else a **privileged** container, which is host-root equivalent: it is logged at start and its network policy is best effort.
27+- A **rootless** Docker daemon is refused with a clear error (a nested dockerd needs a rootful daemon or sysbox).
28+- Per job: a bridge network (`erbr<slot>`), a scratch volume, a Unix socket in a bind-mounted per-job directory of the state directory. Every Docker object carries the label `dev.bnema.ermite-runner.job`; leftovers are reaped by label at start.
29+- The Docker daemon must run on the same host as the runner (host paths are bind-mounted).
30+
31+### Network policy (both modes)
32+
33+One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address, where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. The golden rulesets are in `internal/netpolicy/testdata`.
34+
35+### Cleanup
36+
37+Everything a job creates (process, TAP or bridge, nftables rules, files, containers, networks, volumes) is removed by `Stop`, which also runs with a cancelled context, and by `Reap` at start for what a crash left behind. A VM leaves a state record (`<state dir>/vms/<id>.json`) before its first resource; one runner instance per state directory (flock).
38+
39+## Configuration (sandbox part)
40+
41+| Variable | Default | |
42+|---|---|---|
43+| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
44+| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | absolute path |
45+| `ERMITE_RUNNER_CAPACITY` | `1` | parallel jobs (slots) |
46+| `ERMITE_RUNNER_IMAGE` | | VM: image directory (`vmlinux`, `base.ext4`; default `<state dir>/image`); container: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) |
47+| `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T` |
48+| `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |
49+| `ERMITE_RUNNER_NETWORK` | `10.200.0.0/16` | pool of /30 subnets, one per slot |
50+| `ERMITE_RUNNER_FIRECRACKER`, `_JAILER`, `_KERNEL` | from `PATH` | binaries and kernel overrides |
51+| `ERMITE_RUNNER_JAILER_MODE` | `on` | `off` only for development and tests |
52+
53+## Images and kernel
54+
55+```sh
56+scripts/build-kernel.sh tmp/image # downloads Linux 6.18 LTS from kernel.org, builds vmlinux (needs bc, flex, bison, pahole, libelf, openssl headers)
57+scripts/build-image.sh tmp/image # builds the OCI image with docker or podman, exports tmp/image/base.ext4 (no root needed)
58+```
59+
60+The kernel configuration is `images/kernel/config-6.18` (+ per-architecture fragments), checked after `olddefconfig`: the build fails if a listed option did not survive. arm64 builds (`ARCH_NAME=arm64`) but is not tested in CI-1.
61+
62+## Tests
63+
64+```sh
65+GOWORK=off go vet ./... && GOWORK=off go test -race ./...
66+```
67+
68+Unit tests need nothing special (the drivers are tested with a fake Firecracker and a fake Docker engine). Integration tests are opt-in and use the real stub executor: output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
69+
70+**VM** (`ERMITE_RUNNER_IT_VM=1`): needs `/dev/kvm`, Firecracker, the kernel and base image from the scripts above.
71+
72+```sh
73+# Unprivileged: inside a user + network namespace (TAP, nftables and DNS on the TAP address work there).
74+# Jailer off: it needs real root. Everything else is the production path.
75+FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
76+
77+# Privileged, with the jailer (the production configuration):
78+sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
79+ ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
80+ "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
81+```
82+
83+**Container** (`ERMITE_RUNNER_IT_CONTAINER=1`): needs a rootful Docker daemon, `CAP_NET_ADMIN` and the OCI image (`scripts/build-image.sh`).
84+
85+```sh
86+sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:dev \
87+ DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
88+```
89+
90+`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
91+
92+`ermite-runner sandbox-check [script]` runs one stub job through the selected mode and prints its output: the smoke test of an installation.
1193
1294 ## License
1395
@@ -5,9 +5,91 @@ The CI runner of [ermite](https://git.bnema.dev/bnema/ermite). It runs workflows
5 - **`vm`**: a Firecracker micro-VM per job, destroyed at the end. Needs `/dev/kvm`.5 - **`vm`**: a Firecracker micro-VM per job, destroyed at the end. Needs `/dev/kvm`.
6 - **`container`**: a container per job with its own Docker daemon, for hosts without KVM. Uses the sysbox runtime when available, else a privileged container.6 - **`container`**: a container per job with its own Docker daemon, for hosts without KVM. Uses the sysbox runtime when available, else a privileged container.
7 7
8-`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when KVM is usable, else `container`.8+`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when `/dev/kvm` opens read-write and Firecracker, the guest kernel and the base image exist, else `container`. The chosen mode and the reasons are logged at start.
9 9
10-Status: early development.10+Status: early development. This tree has the job sandboxes (phase D of the CI-1 plan) with a stub executor; the host loop that fetches jobs from ermite comes later.
11+
12+## Sandboxes
13+
14+Both modes run the same executable, the runner's own, attached **read-only** to the sandbox (never baked into the image, so a runner upgrade needs no image rebuild): `ermite-runner init` is the entrypoint (PID 1 of a VM), then `ermite-runner exec` connects back to the host over a framed protobuf stream (`internal/execstream/exec.proto`; vsock in a VM, a Unix socket in a container). The executable must be static: build with `CGO_ENABLED=0`.
15+
16+### vm
17+
18+- Firecracker is driven through its HTTP API on a unix socket with `net/http`, not through `firecracker-go-sdk` (its last release, v1.0.0 from 2022, predates Firecracker 1.17 and pulls CNI, go-openapi and netlink for a handful of PUT requests).
19+- Disks: the runner executable (read-only boot root, ext4 built with `mkfs.ext4 -d`, cached by the executable's hash), the shared read-only base image, and a per-job sparse scratch disk (`ERMITE_RUNNER_JOB_DISK`). The guest init builds an overlayfs root (lower = base, upper = scratch) and keeps dockerd's data root on the scratch disk.
20+- The jailer runs each VM as uid/gid `ERMITE_RUNNER_UID_BASE` + slot in its own chroot under the state directory. The kernel, the base image and the state directory must be on the same filesystem (files are hard-linked into the jail) and the kernel and base image must be world-readable.
21+- One TAP (`ertap<slot>`) per VM with a /30 from `ERMITE_RUNNER_NETWORK` (default `10.200.0.0/16`), vsock for the executor stream, IPv6 disabled in the guest.
22+- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires (the plan runs the host process as root under systemd).
23+
24+### container
25+
26+- Docker Engine API via the maintained client `github.com/moby/moby/client` (`DOCKER_HOST` honoured). Runtime `sysbox-runc` when the daemon lists it, else a **privileged** container, which is host-root equivalent: it is logged at start and its network policy is best effort.
27+- A **rootless** Docker daemon is refused with a clear error (a nested dockerd needs a rootful daemon or sysbox).
28+- Per job: a bridge network (`erbr<slot>`), a scratch volume, a Unix socket in a bind-mounted per-job directory of the state directory. Every Docker object carries the label `dev.bnema.ermite-runner.job`; leftovers are reaped by label at start.
29+- The Docker daemon must run on the same host as the runner (host paths are bind-mounted).
30+
31+### Network policy (both modes)
32+
33+One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address, where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. The golden rulesets are in `internal/netpolicy/testdata`.
34+
35+### Cleanup
36+
37+Everything a job creates (process, TAP or bridge, nftables rules, files, containers, networks, volumes) is removed by `Stop`, which also runs with a cancelled context, and by `Reap` at start for what a crash left behind. A VM leaves a state record (`<state dir>/vms/<id>.json`) before its first resource; one runner instance per state directory (flock).
38+
39+## Configuration (sandbox part)
40+
41+| Variable | Default | |
42+|---|---|---|
43+| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
44+| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | absolute path |
45+| `ERMITE_RUNNER_CAPACITY` | `1` | parallel jobs (slots) |
46+| `ERMITE_RUNNER_IMAGE` | | VM: image directory (`vmlinux`, `base.ext4`; default `<state dir>/image`); container: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) |
47+| `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T` |
48+| `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |
49+| `ERMITE_RUNNER_NETWORK` | `10.200.0.0/16` | pool of /30 subnets, one per slot |
50+| `ERMITE_RUNNER_FIRECRACKER`, `_JAILER`, `_KERNEL` | from `PATH` | binaries and kernel overrides |
51+| `ERMITE_RUNNER_JAILER_MODE` | `on` | `off` only for development and tests |
52+
53+## Images and kernel
54+
55+```sh
56+scripts/build-kernel.sh tmp/image # downloads Linux 6.18 LTS from kernel.org, builds vmlinux (needs bc, flex, bison, pahole, libelf, openssl headers)
57+scripts/build-image.sh tmp/image # builds the OCI image with docker or podman, exports tmp/image/base.ext4 (no root needed)
58+```
59+
60+The kernel configuration is `images/kernel/config-6.18` (+ per-architecture fragments), checked after `olddefconfig`: the build fails if a listed option did not survive. arm64 builds (`ARCH_NAME=arm64`) but is not tested in CI-1.
61+
62+## Tests
63+
64+```sh
65+GOWORK=off go vet ./... && GOWORK=off go test -race ./...
66+```
67+
68+Unit tests need nothing special (the drivers are tested with a fake Firecracker and a fake Docker engine). Integration tests are opt-in and use the real stub executor: output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
69+
70+**VM** (`ERMITE_RUNNER_IT_VM=1`): needs `/dev/kvm`, Firecracker, the kernel and base image from the scripts above.
71+
72+```sh
73+# Unprivileged: inside a user + network namespace (TAP, nftables and DNS on the TAP address work there).
74+# Jailer off: it needs real root. Everything else is the production path.
75+FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
76+
77+# Privileged, with the jailer (the production configuration):
78+sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
79+ ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
80+ "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
81+```
82+
83+**Container** (`ERMITE_RUNNER_IT_CONTAINER=1`): needs a rootful Docker daemon, `CAP_NET_ADMIN` and the OCI image (`scripts/build-image.sh`).
84+
85+```sh
86+sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:dev \
87+ DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
88+```
89+
90+`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
91+
92+`ermite-runner sandbox-check [script]` runs one stub job through the selected mode and prints its output: the smoke test of an installation.
11 93
12 ## License94 ## License
13 95
M go.mod +33 -0
@@ -1,3 +1,36 @@
11 module bnema.dev/ermite-runner
22
33 go 1.27.1
4+
5+require (
6+ github.com/bnema/zerowrap v1.4.1
7+ github.com/containerd/errdefs v1.0.0
8+ github.com/moby/moby/api v1.56.1
9+ github.com/moby/moby/client v0.6.2
10+ golang.org/x/sys v0.49.0
11+ google.golang.org/protobuf v1.36.12
12+)
13+
14+require (
15+ github.com/Microsoft/go-winio v0.6.2 // indirect
16+ github.com/cespare/xxhash/v2 v2.3.0 // indirect
17+ github.com/containerd/errdefs/pkg v0.3.0 // indirect
18+ github.com/distribution/reference v0.6.0 // indirect
19+ github.com/docker/go-connections v0.8.2 // indirect
20+ github.com/docker/go-units v0.5.0 // indirect
21+ github.com/felixge/httpsnoop v1.0.4 // indirect
22+ github.com/go-logr/logr v1.4.4 // indirect
23+ github.com/go-logr/stdr v1.2.2 // indirect
24+ github.com/mattn/go-colorable v0.1.15 // indirect
25+ github.com/mattn/go-isatty v0.0.22 // indirect
26+ github.com/moby/docker-image-spec v1.3.1 // indirect
27+ github.com/opencontainers/go-digest v1.0.0 // indirect
28+ github.com/opencontainers/image-spec v1.1.1 // indirect
29+ github.com/rs/zerolog v1.35.1 // indirect
30+ go.opentelemetry.io/auto/sdk v1.2.1 // indirect
31+ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 // indirect
32+ go.opentelemetry.io/otel v1.45.0 // indirect
33+ go.opentelemetry.io/otel/metric v1.45.0 // indirect
34+ go.opentelemetry.io/otel/trace v1.45.0 // indirect
35+ gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
36+)
@@ -1,3 +1,36 @@
1 module bnema.dev/ermite-runner1 module bnema.dev/ermite-runner
2 2
3 go 1.27.13 go 1.27.1
4+
5+require (
6+ github.com/bnema/zerowrap v1.4.1
7+ github.com/containerd/errdefs v1.0.0
8+ github.com/moby/moby/api v1.56.1
9+ github.com/moby/moby/client v0.6.2
10+ golang.org/x/sys v0.49.0
11+ google.golang.org/protobuf v1.36.12
12+)
13+
14+require (
15+ github.com/Microsoft/go-winio v0.6.2 // indirect
16+ github.com/cespare/xxhash/v2 v2.3.0 // indirect
17+ github.com/containerd/errdefs/pkg v0.3.0 // indirect
18+ github.com/distribution/reference v0.6.0 // indirect
19+ github.com/docker/go-connections v0.8.2 // indirect
20+ github.com/docker/go-units v0.5.0 // indirect
21+ github.com/felixge/httpsnoop v1.0.4 // indirect
22+ github.com/go-logr/logr v1.4.4 // indirect
23+ github.com/go-logr/stdr v1.2.2 // indirect
24+ github.com/mattn/go-colorable v0.1.15 // indirect
25+ github.com/mattn/go-isatty v0.0.22 // indirect
26+ github.com/moby/docker-image-spec v1.3.1 // indirect
27+ github.com/opencontainers/go-digest v1.0.0 // indirect
28+ github.com/opencontainers/image-spec v1.1.1 // indirect
29+ github.com/rs/zerolog v1.35.1 // indirect
30+ go.opentelemetry.io/auto/sdk v1.2.1 // indirect
31+ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 // indirect
32+ go.opentelemetry.io/otel v1.45.0 // indirect
33+ go.opentelemetry.io/otel/metric v1.45.0 // indirect
34+ go.opentelemetry.io/otel/trace v1.45.0 // indirect
35+ gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
36+)
A go.sum +75 -0 generated file: collapsed

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 4

Tags 11

↑↓ moveenter openesc close