1# ermite-runner
2
3The CI runner of [ermite](https://git.bnema.dev/bnema/ermite). It runs workflows compatible with GitHub Actions (`.github/workflows/*.yml`), one job per sandbox:
4
5- **`vm`**: a Firecracker micro-VM per job, destroyed at the end. Needs `/dev/kvm`.
6- **`container`**: a container per job with its own Docker daemon, for hosts without KVM. Uses the sysbox runtime when available, else a privileged container.
7
8`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when `/dev/kvm` opens read-write and Firecracker, the guest kernel and the base image exist, else `container`. The chosen mode and the reasons are logged at start.
9
10Status: early development. This tree has the job sandboxes (phase D of the CI-1 plan) with a stub executor; the host loop that fetches jobs from ermite comes later.
11
12## Sandboxes
13
14Both modes run the same executable, the runner's own, attached **read-only** to the sandbox (never baked into the image, so a runner upgrade needs no image rebuild): `ermite-runner init` is the entrypoint (PID 1 of a VM), then `ermite-runner exec` connects back to the host over a framed protobuf stream (`internal/execstream/exec.proto`; vsock in a VM, a Unix socket in a container). The executable must be static: build with `CGO_ENABLED=0`.
15
16### vm
17
18- Firecracker is driven through its HTTP API on a unix socket with `net/http`, not through `firecracker-go-sdk` (its last release, v1.0.0 from 2022, predates Firecracker 1.17 and pulls CNI, go-openapi and netlink for a handful of PUT requests).
19- Disks: the runner executable (read-only boot root, ext4 built with `mkfs.ext4 -d`, cached by the executable's hash), the shared read-only base image, a per-job sparse scratch disk (`ERMITE_RUNNER_JOB_DISK`) and, when there are externals (below), a fourth read-only disk with the host's Node.js builds, cached by their content. The guest init builds an overlayfs root (lower = base, upper = scratch), mounts the externals disk (`/dev/vdd`) at `/home/runner/externals` and keeps dockerd's data root on the scratch disk. The scratch disk and `eth0` are rate limited (defaults 400 MiB/s and 20000 IOPS, 1 Gbit/s each way; `ERMITE_RUNNER_JOB_DISK_BANDWIDTH` in MiB/s, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` in Mbit/s; 0 disables).
20- The jailer runs each VM as uid/gid `ERMITE_RUNNER_UID_BASE` + slot in its own chroot under the state directory, in a new PID namespace and in a cgroup v2 with `memory.max` (guest memory + 256 MiB for the VMM) and `pids.max`. The kernel, the base image and the state directory must be on the same filesystem (files are hard-linked into the jail) and the kernel and base image must be world-readable.
21- **The state directory and every parent of it must be owned by root and not writable by group or others.** The jailer runs as root and builds the chroots under `<state dir>/jail`; the driver checks that `jail` itself is a real directory, root-owned and not group/other-writable, but it does not walk the parents: a user who can rename or replace a parent directory can still redirect root's file operations. `/var/lib/ermite-runner` (the default, created by the unit's `StateDirectory`) is fine; do not point `ERMITE_RUNNER_STATE_DIR` into a home directory or `/tmp`.
22- One TAP (`ertap<slot>`) per VM with a /30 from `ERMITE_RUNNER_NETWORK` (default `10.200.0.0/16`), vsock for the executor stream, IPv6 disabled in the guest.
23- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires (the plan runs the host process as root under systemd).
24
25### container
26
27- Docker Engine API via the maintained client `github.com/moby/moby/client` (`DOCKER_HOST` honoured). Runtime `sysbox-runc` when the daemon lists it, else a **privileged** container, which is host-root equivalent. That is allowed by default (ADR 0025) but never silent: the runner logs a warning at start and at the start of every job (`privileged container: host-root equivalent, network policy best effort`), `Driver.Isolation()` reports `container_privileged` (or `container_sysbox`, `vm`) for the registration with ermite, and `ERMITE_RUNNER_DENY_PRIVILEGED=1` refuses to start without sysbox.
28- A **rootless** Docker daemon is refused with a clear error (a nested dockerd needs a rootful daemon or sysbox).
29- Limits per job: memory with swap equal to memory (no swap), `ERMITE_RUNNER_JOB_PIDS` processes (default 32768), CPUs.
30- Disk: the job's writable space is enforced like in `vm`: a sparse ext4 file of `ERMITE_RUNNER_JOB_DISK`, **loop-mounted on the host** (so this mode needs root, like the rest of it) and bind-mounted at `/scratch`. The nested dockerd's data root, `/home/runner/work` and `/tmp` are directories of that disk; the container's own writable layer stays small. A job that fills its disk gets "no space left on device" instead of filling the host.
31- Per job: a bridge network (`erbr<slot>`), the scratch disk, a Unix socket in a bind-mounted per-job directory of the state directory (the job sees only that directory, not the scratch file). Every Docker object carries the label `dev.bnema.ermite-runner.job`; leftovers are reaped by label at start. The state directory path must keep the socket path under 107 bytes.
32- The Docker daemon must run on the same host as the runner (host paths are bind-mounted).
33
34### Externals (Node.js)
35
36The host provides Node.js to the jobs **outside the image**, like its own executable, so that a runner upgrade or a new Node.js version never needs an image rebuild. The builds are the ones of `actions/runner` (`Misc/externals.sh`), pinned in `internal/externals/table.go` with the SHA-256 of each archive:
37
38| Name | Node.js | Source | Architectures |
39| --- | --- | --- | --- |
40| `node20` | 20.20.2 | `nodejs.org/dist/v20.20.2/node-v20.20.2-linux-<arch>.tar.gz` | x64, arm64 |
41| `node24` | 24.21.0 | `nodejs.org/dist/v24.21.0/node-v24.21.0-linux-<arch>.tar.gz` | x64, arm64 |
42| `node20_alpine` | 20.20.2 | `github.com/actions/alpine_nodejs` release `v20.20.2` | x64 |
43| `node24_alpine` | 24.21.0 | `github.com/actions/alpine_nodejs` release `v24.21.0` | x64 |
44
45- **Node versions of actions** follow github.com today (`RequireNode24` and `Node24ByDefault` of the executor's `ActionsConfig`, both set by the sandbox through `DefaultActionsConfig`): `using: node12`, `node16` and `node20` actions run on `node24`, whatever `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24` and `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION` say, and `Complete job` ends with `##[warning]Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: <actions, sorted ignoring case>. For more information see: …` (recordings `act-node-*`). Both flags off give the older phases of the runner.
46- **Download.** `ermite-runner run` (and `sandbox-check`) fetch what is missing at start, into `$ERMITE_RUNNER_STATE_DIR/externals/<name>-<version>-<arch>/`, in the background; the start waits for them at most 2 minutes (a runner with a network normally takes its first job with everything in place, one without never waits long). The archive is the `.tar.gz` (the Go standard library has no xz); its SHA-256 is checked **before** anything is unpacked. Redirects must stay on https.
47- **Extraction** keeps the top directory out and refuses absolute names, `..`, devices and fifos, symbolic links that are absolute or climb out of the tree, entries below a link, hard links to anything but an unpacked file, and archives over 200,000 entries or 2 GiB; writes go through `os.Root`. The tree is built in a temporary directory and renamed into place, so a directory under its final name is always complete.
48- **Only pinned versions are kept**: at every start everything else in `externals/` (other versions, other architectures, interrupted downloads) is removed.
49- **Layout in the sandbox** (GitHub's): `/home/runner/externals/node20/bin/node`, `/node24/bin/node`, and the alpine ones for job containers, read-only. In `container` mode they are read-only bind mounts of the host directories; in `vm` mode one ext4 disk (`mkfs.ext4 -d`) named after the content of the set (`cache/extdisk-<key>.ext4`), so a new Node.js builds a new disk and leaves the executable's disk and the base image alone. Superseded disks are removed an hour after the last build. The set is read at every job start: a download that finishes later is used by the next job.
50- **When a build cannot be provided** (offline, HTTP error, wrong checksum, unsafe archive, no build for the architecture) the runner **still runs**: it logs why and attaches in its place a directory with the file `UNAVAILABLE` (the reason) and a `bin/node` script, `cat "$(dirname "$0")/../UNAVAILABLE" >&2; exit 1`, so a job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: download …: SHA-256 mismatch …` rather than `No such file or directory`. These directories (`externals/.unavailable/<name>-<reason hash>`) are built once, complete, and renamed into place; a file is never rewritten in place, so a reader (the build of a VM's externals disk, a job running the stub) never sees a half-written one. A directory handed out to a sandbox in the last hour is kept by the pruning, since a running container may have it bind-mounted.
51- **Retries.** A download that failed for a reason that may pass (network error, HTTP status) is retried every 5 minutes by a timer, whether jobs come or not, until the runner stops. A verdict (wrong checksum, unsafe archive, no build for the architecture) is not retried before the next start.
52- **Archive layouts.** The nodejs.org archives have one top directory (`node-vX-linux-x64/`), which is dropped; the `actions/alpine_nodejs` archives start at `./bin/node` and are unpacked as they are. The alpine checksums are the digests of the GitHub release assets.
53- `ERMITE_RUNNER_NODE=0` turns all of this off: nothing is downloaded or attached.
54
55### One runner per host
56
57Reaping removes every process, device, firewall rule and Docker object the runner owns, so two runners on one host would destroy each other's jobs. Both modes take an exclusive lock at a fixed host path, `/run/ermite-runner.lock` (`ERMITE_RUNNER_LOCK_FILE` overrides it, for tests), and refuse to start when it is held, whatever their state directories or isolation modes. The kernel drops the lock when the process dies. A runner in a container or in another mount namespace must share that path (or it is not protected): do not run it so.
58
59### Network policy (both modes)
60
61One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
62
63**Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
64
65### Guest forge access (opt-in, staging only)
66
67Off by default. `ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the forge **the way the runner does**, for a staging host where ermite is only reachable through an overlay (NetBird) and the runner itself uses a local route, e.g. `/etc/hosts` `ermite.bnema.dev -> 127.0.0.1` and a reverse proxy (Gordon) on `*:8088` with `ERMITE_RUNNER_URL=http://ermite.bnema.dev:8088`. Jobs otherwise cannot `git clone ${{ github.server_url }}/...`: the public URL (`https://ermite.bnema.dev`) is TLS-terminated off-host, and the private ranges are denied.
68
69The host name and port come from `ERMITE_RUNNER_URL` (port defaulted from the scheme: 443 or 80). The runner **refuses to start** when the URL host is an IP literal (sandboxes are pointed at the host by DNS), when the URL has a path, when the name does not resolve on the host, or when the URL is plain `http://` without `ERMITE_RUNNER_INSECURE_HTTP=1` (same rule as for the runner itself: tokens then travel in clear). Then, for every sandbox in both modes:
70
71- the DNS forwarder answers `A` queries for exactly that host name (case-insensitive, trailing dot ignored) with the sandbox's **gateway address**, and `AAAA` with an empty `NOERROR` answer; every other name is forwarded as before;
72- the nftables input chain accepts TCP from the sandbox's own address to its own gateway address on that one port, before the sandbox's catch-all drop (one extra line per sandbox; see `internal/netpolicy/testdata/guest_forge.nft`). Nothing else on the host opens; the forward chain (private-range denies, source check, no sandbox-to-sandbox) is unchanged;
73- the `github.server_url`, `github.api_url` and `github.graphql_url` of the job's contexts, and so `GITHUB_SERVER_URL`, `GITHUB_API_URL` and `GITHUB_GRAPHQL_URL`, are rewritten to the origin of `ERMITE_RUNNER_URL` when their origin is the server's public one (`github.server_url`); paths are kept (`/-/github/api`). Other values, and the URLs inside `github.event`, stay as the server recorded them;
74- `guest forge access: sandboxes may reach <origin> on the host` is logged at start and at the start of every job, like the privileged-container warning.
75
76**The runner does not proxy.** The traffic reaches the host's own listener on that port, so a service must listen on it on an address the gateway reaches (`*`, as Gordon does; a listener bound to `127.0.0.1` receives nothing). A shared reverse proxy on that port routes by `Host` header, so **a job can reach every app that proxy serves on that port**, not only the forge, by sending another `Host` header: use this on staging hosts only, with a proxy whose other apps you accept to expose to job code. That traffic comes from a private address (`10.200.x.x`), so anything behind the proxy that trusts private sources (admin pages, metrics, rate-limit exemptions) trusts job code too. Jobs also see the forge over plain HTTP in that setup.
77
78### Cancellation
79
80The host sends a `Cancel{reason}` frame (`execstream.Drive` takes a cancel channel, not a context). The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later, with the timers stopped as soon as the shell exits and nothing signalled afterwards (the group id could be reused). Background children that keep the output pipes open are killed when the shell exits, after a short drain. Init forwards SIGTERM and SIGINT to the executor.
81
82### Annotations, summaries and masks
83
84The executor ports the issue handling of the official runner: `::notice`, `::warning` and `::error` (with `title`, `file`, `line`, `endLine`, `col`, `endColumn`), problem matchers (`add-matcher`, `remove-matcher`, .NET regular expressions through `regexp2`, 1 s timeout, `GITHUB_ACTIONS_RUNNER_ISSUE_MATCHER_TIMEOUT`, removed after 3 timeouts) and every `##[error]` the runner itself writes become annotations, at most 10 per level per step, messages cut at 4096 characters; `GITHUB_STEP_SUMMARY` is read after each step (1 MiB). `::add-mask::` adds to the masker at run time, with the encodings of `ValueEncoders.cs`. The executor sends them to the host as `StepAnnotations` and `StepSummary` frames; the host checks the limits again, sends the annotations with the `UpdateJob` that completes the step (the list replaces the step's stored one) and the leftovers with `CompleteJob`, and the summaries with `UploadSummary`. Both are sent only when the server lists `ci.annotations` and `ci.summaries` in `GetVersion` (read once per job). Step `timeout-minutes` is enforced like `StepsRunner` does. Attribution for the ported code and tests: [THIRD_PARTY.md](THIRD_PARTY.md).
85
86### Cleanup
87
88Everything a job creates (process, TAP or bridge, nftables rules, files, loop mounts, containers, networks, cgroups) is removed by `Stop`, which also runs with a cancelled context, aborts a `Start` in progress, and by `Reap` at start for what a crash left behind. A VM leaves a state record (`<state dir>/vms/<id>.json`) before its first resource. **The network policy and the TAP or bridge are never removed while the VMM or the container may still be running**: when a process cannot be killed or a container cannot be removed, `Stop` returns an error wrapping `sandbox.ErrLeaked`, the sandbox's slot stays taken, and `Driver.Reap` (which returns `sandbox.ErrBusy` while a sandbox is running) finishes the job and frees the slot. The host should stop taking jobs on a leaked slot until then. `Sandbox.ID()` names a sandbox's files and Docker objects.
89
90## Running jobs
91
92```
93printf %s "$TOKEN" | ermite-runner register --token-stdin   # once
94ermite-runner run                                           # as a service
95```
96
97`register` picks the isolation mode like `run` will, registers the runner with
98ermite (name, labels, version, isolation, architecture) and stores the runner
99token in `$ERMITE_RUNNER_STATE_DIR/runner.json` (mode 0600, in a directory made
1000700; a file readable by others is refused). `--replace` removes a runner of the
101same name first.
102
103**The token of `register`** (scope `runner:register`: it reads the code and the
104secrets of every repository, see `runner.proto`) is looked up in this order:
105standard input with `--token-stdin`, then `ERMITE_RUNNER_REGISTER_TOKEN`, then
106`--token`. Prefer the first two: **the arguments of a process are visible to
107every user of the host** (`ps`, `/proc/<pid>/cmdline`), the environment and
108standard input are not.
109
110**The server URL** must be `https://`. Plain `http://` is accepted for
111`localhost`, `127.0.0.0/8` and `::1` only; `ERMITE_RUNNER_INSECURE_HTTP=1` allows
112any other `http://` URL (tests, a trusted network) and sends every token in
113clear. The client never follows a redirect, so a token cannot be carried to
114another host.
115
116`run` reaps what a previous run left, then polls `FetchJob` whenever a slot is
117free (`ERMITE_RUNNER_CAPACITY`), with a `poll_id` that is kept when a response
118is lost and obeying `retry_after`. For each job it sends the acknowledging
119heartbeat first, then one every 30 s (a failed one is retried every 5 s), opens
120`WatchJob` (a cancel request reaches the executor, which interrupts, terminates
121and kills like the official runner), forwards the log with `AppendLog`
122(batches of at most 1,000 lines and 64 KiB per step, numbered offsets, retried
123with the same lines), sends step snapshots with `UpdateJob` and ends with an
124idempotent `CompleteJob`. When ermite says the job ended (`JobEnded`) or revokes
125the assignment (`unauthenticated`) the sandbox is destroyed and nothing more is
126sent. On SIGINT or SIGTERM the running jobs are cancelled with
127`runner_shutdown`, reported, and the sandboxes removed. The job token stays in
128the host process; the sandbox receives the workflow and its contexts only.
129
130| Variable | Meaning |
131| --- | --- |
132| `ERMITE_RUNNER_URL` | base URL of ermite, as in a browser (the runner adds `/-/api` itself) |
133| `ERMITE_RUNNER_NAME` | runner name (default: the host name) |
134| `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |
135| `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` (default) or `self-hosted`: the `runner.environment` value the jobs see. ermite's runners stand in for GitHub-hosted ones, so workflows written for github.com (`runner.environment == 'github-hosted'`) behave the same; `self-hosted` also makes "Set up job" print the runner and machine name, as GitHub does for self-hosted runners |
136| `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` (see above) |
137| `ERMITE_RUNNER_INSECURE_HTTP` | `1` allows a non-loopback `http://` URL (tests only); also required for `ERMITE_RUNNER_GUEST_FORGE` with an `http://` URL |
138| `ERMITE_RUNNER_GUEST_FORGE` | `1` lets sandboxes reach the forge as the runner does (staging only; see "Guest forge access") |
139| `ERMITE_RUNNER_LOG_LEVEL` | `debug`, `info` (default), `warn`, `error` |
140| `ERMITE_RUNNER_LOG_FORMAT` | `console` (default, colours) or `json` |
141
142## Installing the service
143
144`contrib/ermite-runner.service` is a systemd unit. The runner runs as **root**
145on the host (TAP devices, nftables, loop mounts, the jailer, containers); the
146jobs never do: they run as user `runner` in a micro-VM or a container.
147
148```sh
149sudo install -m 0755 ermite-runner /usr/local/bin/ermite-runner     # a static build: CGO_ENABLED=0
150sudo install -m 0644 contrib/ermite-runner.service /etc/systemd/system/
151sudo install -d -m 0700 /etc/ermite-runner
152sudo install -m 0600 /dev/stdin /etc/ermite-runner/env <<'ENV'
153ERMITE_RUNNER_URL=https://ermite.example.org
154ERMITE_RUNNER_NAME=igor
155ERMITE_RUNNER_LABELS=ubuntu-latest,ubuntu-24.04
156ERMITE_RUNNER_ISOLATION=auto
157ERMITE_RUNNER_CAPACITY=2
158ERMITE_RUNNER_JOB_CPUS=4
159ERMITE_RUNNER_JOB_MEMORY=8G
160ERMITE_RUNNER_JOB_DISK=64G
161ERMITE_RUNNER_LOG_FORMAT=json
162ENV
163# Register once, in the same environment as the service (the settings, the state
164# directory), with the token on standard input so that it is not in any
165# command line:
166printf %s "$TOKEN" | sudo systemd-run --pipe --wait -p EnvironmentFile=/etc/ermite-runner/env \
167  -p StateDirectory=ermite-runner -p StateDirectoryMode=0700 \
168  /usr/local/bin/ermite-runner register --token-stdin
169sudo systemctl enable --now ermite-runner
170journalctl -u ermite-runner -f
171```
172
173`systemd-run` reads the same `EnvironmentFile` as the unit, so `register` and
174`run` agree on the URL, the name, the labels, the isolation mode and the state
175directory (`/var/lib/ermite-runner`, the default of `ERMITE_RUNNER_STATE_DIR`).
176
177**Logs.** The runner writes to standard error, which journald collects. Set
178`ERMITE_RUNNER_LOG_FORMAT=json` under journald: one JSON event per line, with
179`job_id`, `run_id` and `attempt` on every line about a job, `warn` for what is
180retried and `error` for what is refused or leaked; the colours of the `console`
181format are noise in a journal. `ERMITE_RUNNER_LOG_LEVEL=debug` adds detail.
182Tokens are never logged.
183
184**Stopping.** `systemctl stop` sends SIGTERM: running jobs are cancelled with
185`runner_shutdown`, reported to ermite, and their sandboxes removed before the
186service exits. The budget, with the defaults, is **90 s**: 30 s for the executor
187to stop its steps (SIGINT, SIGTERM after 7.5 s, SIGKILL 2.5 s later, then the
188post-steps), 30 s to destroy the sandbox, 30 s to flush the log and report; jobs
189do this in parallel, so it does not grow with `ERMITE_RUNNER_CAPACITY`. The unit
190sets `TimeoutStopSec=120`, 30 s of margin (a test in `internal/host` fails if the
191two drift apart). A job whose first heartbeat is still pending when the signal
192arrives is not started: ermite gives it to another runner after 30 s. A crash or
193a power cut leaves sandboxes behind; the next start reaps them. Only one runner
194may live on a host (a host-wide lock enforces it), because reaping removes
195everything labelled as the runner's own.
196
197**Sandboxing of the unit.** The runner has to be root (TAP devices, nftables,
198loop mounts, the jailer, Docker), so the unit only takes away what it does not
199use, and only with options that **leave it in the host's mount namespace**:
200`NoNewPrivileges`, `ProtectClock`, `RestrictSUIDSGID`, `LockPersonality`,
201`RestrictRealtime`, `SystemCallArchitectures=native` and `RestrictAddressFamilies`
202(UNIX, INET, INET6, NETLINK, VSOCK).
203
204**Deliberately absent**, and why:
205
206- **Every option that gives the unit a mount namespace of its own:** `PrivateTmp`,
207  `ProtectHome`, `ProtectSystem`, `ProtectKernelModules`, `ProtectKernelLogs`,
208  `ProtectKernelTunables`, `ProtectControlGroups`, `PrivateDevices`,
209  `ReadOnlyPaths`, `ReadWritePaths`, `InaccessiblePaths`, `TemporaryFileSystem`,
210  `BindPaths`. In container mode the runner loop-mounts the scratch disk of each
211  job and the Docker daemon, which lives in the host's namespace, binds that
212  directory into the container; a mount made inside the unit's namespace is
213  invisible to it and the container would get an empty directory. (`ProtectKernelModules`
214  and `ProtectKernelLogs` look harmless but are implemented with mounts too.) The
215  micro-VM mode would tolerate them, but one unit serves both modes, and a second
216  unit for VM-only hardening is not worth maintaining. A test in `internal/host`
217  fails if the unit sets any of them.
218- `Delegate=`: the runner manages no cgroup of its own (the jailer writes the
219  VMs' cgroups as root, Docker has its own).
220- `RestrictNamespaces=`, `PrivateNetwork=`, `CapabilityBoundingSet=`,
221  `SystemCallFilter=`: network namespaces and devices (TAP, veth), mounts,
222  `net.ipv4.ip_forward`, capabilities handed to the jailer and Docker.
223- `MemoryDenyWriteExecute=`: Firecracker maps guest memory writable and executable.
224- `UMask=`: the drivers create directories that the job's user must enter, with
225  explicit modes; the state directory is made 0700 by the runner.
226
227The boundary of a job is its micro-VM or container, not this unit. Checked on a
228workstation (a user manager, no root): the micro-VM job test (`TestVMRunsAJob`:
229user and network namespaces, Firecracker, TAP, vsock) passes under the options
230that remain. Not checked, because they need root on a real host: the jailer, loop
231mounts, nftables in the host namespace, Docker.
232
233## The image
234
235A job runs in the image named by `ERMITE_RUNNER_IMAGE` (container: an OCI
236reference; vm: the directory with `vmlinux` and `base.ext4`, both built from
237`images/ubuntu-24.04/Dockerfile`). The runner binary and Node.js (see "Externals") are attached to the sandbox
238by the host and are never part of the image. What the executor needs from the
239image:
240
241- a user **`runner`** with uid and gid 1001 and the home `/home/runner`; the
242  executor starts as root (init needs it to start `dockerd`) and runs every step
243  as this user, with HOME, USER and LOGNAME set. Without such a user the steps
244  run as root (a test image only);
245- the directories `/home/runner/work` and `/opt/hostedtoolcache` owned by
246  `runner` (`RUNNER_TOOL_CACHE`, which the executor reads from its environment
247  like the official runner; each job gets `/home/runner/work/<repo>/<repo>` and
248  `/home/runner/work/_temp`, cleaned at the start of the job);
249- **`bash`** and **`sh`** (the default shell is `bash -e {0}` when bash exists,
250  else `sh -e {0}`; like on hosted runners `shell: bash` adds
251  `--noprofile --norc -eo pipefail`), `python3` and a `python` command for
252  `shell: python`; `pwsh` is not part of the CI-1 image (a step that asks for it
253  fails with `pwsh: command not found`);
254- `dockerd` (and passwordless `sudo` for `runner`) for jobs that use Docker: the
255  job gets its own daemon with its data root on the scratch disk;
256- what workflows commonly use: git, curl, jq, build-essential, ca-certificates,
257  tzdata, zip/unzip. Full parity with the hosted images (`actions/runner-images`)
258  is a later phase.
259
260## Job containers and service containers
261
262`container:` and `services:` work as on GitHub-hosted Linux runners
263(`internal/executor/containers_*.go`, a port of `ContainerOperationProvider`,
264`ContainerInfo`, `DockerCommandManager` and `ContainerStepHost` of
265actions/runner, with their line numbers in the comments). The containers are
266created in the **sandbox's own `dockerd`**, so nothing reaches the host: in `vm`
267mode (the mode of hosts without Docker Engine) the daemon runs inside the
268micro-VM, in `container` mode it is the nested daemon of the job.
269
270- **Initialize containers** (a step after "Set up job"): docker version check
271  (API ≥ 1.35), cleanup of labelled leftovers, network `github_network_<32 hex>`,
272  for each container `docker pull` (3 tries, random 1–10 s back off),
273  `docker create`/`start`, `docker port` and the health checks of the services
274  (exponential back off 2–32 s, "`<alias>` service is healthy."). Registry
275  `credentials:` go to a private `docker --config` directory under RUNNER_TEMP
276  (password through stdin, masked in every log, directory removed after the pull).
277- **Stop containers** (post step, `always()`): service logs, `docker rm --force`,
278  `docker network rm`. It runs after a failure, a cancellation, a step timeout
279  and the job timeout.
280- **Steps in the job container**: `run:` steps and node actions go through
281  `docker exec -i --workdir … -e NAME… <id> …` (values are passed in the
282  environment of the docker process, translated to `/__w`, `/__t`, `/__e`,
283  `/github/home`, `/github/workflow`, never on the command line). The shell is
284  looked up by the container: `sh -e {0}` unless the step says otherwise. Node
285  actions use the externals mounted read-only at `/__e`, `node<N>_alpine` when
286  `/etc/*release` says alpine. Docker actions run as siblings on the job network.
287- Contexts: `job.container.{id,network}`, `job.services.<alias>.{id,network,ports}`
288  (host ports from `docker port`).
289- **Untrusted input.** `options:` is split into arguments like .NET splits a
290  command line and passed to `docker create` as an argument vector, never through
291  a shell, so a workflow cannot inject a second command; it can still pass any
292  `docker create` flag (`--privileged`, `--pid host`, `--device`, …) and bind
293  volumes (`volumes: /:/host`), exactly as on a self-hosted GitHub runner. That
294  is contained by the sandbox (its own daemon, its own disk), not by the runner:
295  **host paths in `volumes:` and `options:` are paths of the sandbox**. A job
296  that mounts `/` exposes the sandbox's filesystem to its own container only.
297  The runner warns, as GitHub does, about a volume that mounts `/`.
298- Not ported: container hooks (`ACTIONS_RUNNER_CONTAINER_HOOKS`), the
299  "already running in a container" refusal (the sandbox's daemon is local to the
300  executor).
301
302## Job limits
303
304Each job gets its own resources, set for the runner as a whole:
305
306| Variable | Default | Effect |
307|---|---|---|
308| `ERMITE_RUNNER_CAPACITY` | `1` | jobs in parallel: one sandbox, one uid, one `/30` network and one disk per slot. The runner polls ermite only when a slot is free. |
309| `ERMITE_RUNNER_JOB_CPUS` | `2` | vCPUs of a micro-VM; CPU quota of a container |
310| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | guest memory (`M`, `G`, `T`; at least `256M`); a jailed VMM gets this plus 256 MiB of cgroup headroom |
311| `ERMITE_RUNNER_JOB_DISK` | `64G` | size of the sparse scratch disk (at least `1G`): the overlay upper layer of a VM and the Docker data root of a container. Deleted with the job |
312| `ERMITE_RUNNER_JOB_PIDS` | `32768` | process limit of a container |
313| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits (MiB/s, IOPS, Mbit/s); `0` disables |
314
315The job timeout is not a runner setting: it is the workflow's `timeout-minutes`
316(360 by default), which ermite sends with the job and the executor enforces from
317the moment it starts.
318
319## Configuration (sandbox part)
320
321| Variable | Default | |
322|---|---|---|
323| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
324| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | absolute path |
325| `ERMITE_RUNNER_CAPACITY` | `1` | parallel jobs (slots) |
326| `ERMITE_RUNNER_IMAGE` | | VM: image directory (`vmlinux`, `base.ext4`; default `<state dir>/image`); container: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) |
327| `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T`. A warning is logged at start when the state directory has less free space than capacity x disk (disks are sparse: it is only a warning) |
328| `ERMITE_RUNNER_JOB_PIDS` | `32768` | container process limit |
329| `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses container mode without sysbox |
330| `ERMITE_RUNNER_GUEST_FORGE` | `0` | `1` lets sandboxes reach the host and port of `ERMITE_RUNNER_URL` on the host (staging only, see "Guest forge access") |
331| `ERMITE_RUNNER_NODE` | `1` | `0` turns the Node.js externals off (see "Externals") |
332| `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | host-wide single-instance lock (tests only) |
333| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits in MiB/s, IOPS and Mbit/s; 0 disables |
334| `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |
335| `ERMITE_RUNNER_NETWORK` | `10.200.0.0/16` | pool of /30 subnets, one per slot |
336| `ERMITE_RUNNER_FIRECRACKER`, `_JAILER`, `_KERNEL` | from `PATH` | binaries and kernel overrides |
337| `ERMITE_RUNNER_JAILER_MODE` | `on` | `off` only for development and tests |
338
339## Images and kernel
340
341```sh
342scripts/build-kernel.sh tmp/image   # downloads Linux 6.18 LTS from kernel.org, builds vmlinux (needs bc, flex, bison, pahole, libelf, openssl headers)
343scripts/build-image.sh tmp/image    # builds the OCI image with docker or podman, exports tmp/image/base.ext4 (no root needed)
344```
345
346`scripts/build-image.sh` tags the OCI image `ermite-runner-ubuntu-24.04:latest`, the default of `ERMITE_RUNNER_IMAGE` in container mode, and needs e2fsprogs >= 1.47.1 (the runner itself needs >= 1.43 and checks it at start).
347
348The kernel configuration is `images/kernel/config-6.18` (+ per-architecture fragments), checked after `olddefconfig`: the build fails if a listed option did not survive. arm64 builds (`ARCH_NAME=arm64`) but is not tested in CI-1.
349
350## Tests
351
352```sh
353GOWORK=off go vet ./... && GOWORK=off go test -race ./...
354```
355
356Unit tests need nothing special (the drivers are tested with a fake Firecracker and a fake Docker engine). Integration tests are opt-in and use the real stub executor: output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
357
358**VM** (`ERMITE_RUNNER_IT_VM=1`): needs `/dev/kvm`, Firecracker, the kernel and base image from the scripts above.
359
360```sh
361# Unprivileged: inside a user + network namespace (TAP, nftables and DNS on the TAP address work there).
362# Jailer off: it needs real root. Everything else is the production path.
363FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
364
365# Privileged, with the jailer (the production configuration):
366sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
367  ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
368  "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
369```
370
371**Container** (`ERMITE_RUNNER_IT_CONTAINER=1`): needs a rootful Docker daemon, root (loop mounts, `CAP_NET_ADMIN`) and the OCI image (`scripts/build-image.sh`).
372
373```sh
374sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:latest \
375  DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
376```
377
378**Jobs end to end** (`ERMITE_RUNNER_IT_E2E=1`, `internal/e2e`): the real executor in the real image, started through the container driver. They also run on a workstation with a *rootless* Docker daemon (the test-only option `UnsafeAllowRootlessForTests`; the nested `dockerd` of the image cannot start there, and the jobs do not need it). `mount` and `umount` are replaced by stand-ins, since loop mounts need root.
379
380```sh
381IMAGE_TAG=ermite-runner-ubuntu-24.04:dev scripts/build-image.sh tmp/image     # or: docker build -t ermite-runner-ubuntu-24.04:dev images/ubuntu-24.04
382ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-24.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
383```
384
385- `TestContainerRunsAJobAsTheRunnerUser` and `TestContainerCancelStopsTheJobAndOrphans` drive a job over the stream: steps run as `runner` (1001), file ownership, `GITHUB_PATH` and `GITHUB_ENV`, `python`, cancellation by SIGINT, the `always()` step, and the orphan cleanup of "Complete job".
386- `TestChainCorpus` runs the whole chain: a fake `RunnerService` (`internal/host/hosttest`, which checks the offsets, limits and tokens of `runner.proto`) hands out 22 jobs of the ermite-conformance corpus, the real host loop starts them in containers (two at a time), and what the fake stored (steps, conclusions, outputs, log lines) is compared with the recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
387- `TestVMRunsAJob` (`internal/sandbox/integration`, run by `scripts/it-vm-userns.sh`) does the first of these in a micro-VM.
388- `TestVMNodeExternals` (same place) downloads the pinned `node24` (or takes it from `ERMITE_RUNNER_IT_NODE_DIR`, a directory with `node-v24.21.0-linux-x64.tar.gz`, for an offline host; the checksum and the extraction are the real ones), boots a micro-VM and runs `/home/runner/externals/node24/bin/node -e 'console.log(process.version)'` in it, checking that the mount is read-only.
389
390Checked as root on a real KVM host (Arch Linux, kernel 6.18 LTS, Firecracker and jailer 1.17, rootful podman as the Docker API): the jailer (uid drop, chroot, `memory.max` and `pids.max` cgroups, PID namespace), the VM and container integration scenarios, the firewall policy and NAT, the nested `dockerd` on the loop-mounted scratch disk (full disk gives `ENOSPC`), and `internal/e2e` including `TestChainCorpus`. Not checked: sysbox, and a real Docker Engine (podman's compatibility API is not a drop-in, see below).
391
392**Jailer notes** (found on that host): Firecracker 1.17's jailer needs `--chroot-base-dir` to exist (the driver creates `<state dir>/jail`), starts Firecracker as `/firecracker --id <id> --start-time-us ...` (the `/proc` scan that finds a VMM by id knows that form), and leaves `<state dir>/jail/<exec name>` and the cgroup parent `/sys/fs/cgroup/firecracker` behind (only the per-VM directories are the runner's). The jailed API socket path (`<state dir>/jail/firecracker/<id>/root/run/firecracker.socket`) must fit in 107 bytes; the driver checks it at start.
393
394**Host firewalls.** The runner's table has its own `input` hook at priority `filter`; another table with a later hook (firewalld at `filter + 10`) still sees a guest's packet to the gateway after the runner accepted it. A firewalld zone that rejects `udp/tcp 53` (the `public` zone with a DNS reject rule, for instance) therefore breaks the guests' DNS (the symptom is `dns silent` in `TestVMNetworkPolicy`) while NAT keeps working. On such a host, allow DNS from `ertap*` and `erbr*` in that firewall, or run the policy test in a private network namespace (as `scripts/it-vm-userns.sh` does).
395
396**Podman as the Docker API.** `DOCKER_HOST` pointing at rootful podman's compatibility socket gets as far as creating the network, then fails: the compatibility `POST /networks/create` cannot disable netavark's DNS (`dns_enabled` is always true) and `HostConfig.Dns` equal to the network gateway is refused when the container starts (`netavark: error while applying dns entries: DNS server 10.200.0.1 is the gateway of network ...; using it would cause aardvark-dns to forward queries to itself`); the option `com.docker.network.bridge.enable_ip_masquerade` is ignored with a warning, and netavark (`firewall_driver = "nftables"` by default) adds its own rules unless it is configured with `firewall_driver = "none"`. The integration tests passed with a small test-only proxy that rewrites that one request to the libpod endpoint with `dns_enabled=false`, and a private `CONTAINERS_CONF`. Production container mode supports Docker Engine (or sysbox) only.
397
398`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
399
400The firewall policy has a packet-level test (`ERMITE_RUNNER_IT_NET=1`, run by `scripts/it-vm-userns.sh`): two guests and an outside host on veth pairs in separate network namespaces, with a forged source and a forged "reply" between guests.
401
402Protobuf: `go generate ./internal/execstream` (buf and protoc-gen-go are pinned as `tool` directives in go.mod); CI lints the proto and fails when the committed generated code differs.
403
404`ermite-runner sandbox-check [script]` runs one stub job through the selected mode and prints its output: the smoke test of an installation.
405
406## License
407
408MIT. See [LICENSE](LICENSE).