detached 3ad0272

docs(readme): rewrite README with a short overview and docs/ (#43)

commit
3ad02724ad0b7cf6e40f9912e7ed98e7613c7273
author
brice · · 2026-10-18 17:43 UTC
committer
bnema · · 2026-10-18 17:43 UTC
parent
edcdd4c
signature
SSH UnverifiedSHA256:VOCA5lYARHcbKDZ7GhTV45VmcjWnJ6ZQIBazj17LivA The key is unknown
9 files · +666 -408
Split
UnifiedSplit
M README.md +96 -408
@@ -1,447 +1,135 @@
1-# ermite-runner
1+<h1 align="center">ermite-runner</h1>
22
3-> Read-only mirror. Development happens at git.bnema.dev; issues and pull requests are closed here.
4-
5-The CI runner of [ermite](https://github.com/bnema/ermite). It runs workflows compatible with GitHub Actions (`.github/workflows/*.yml`), one job per sandbox:
6-
7-- **`vm`**: a Firecracker micro-VM per job, destroyed at the end. Needs `/dev/kvm`.
8-- **`container`**: a container per job with its own Docker daemon, for hosts without KVM. Uses the sysbox runtime when available, else a privileged container.
9-
10-`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when `/dev/kvm` opens read-write and Firecracker, the guest kernel and the base image exist, else `container`. The chosen mode and the reasons are logged at start.
11-
12-Status: early development. This tree has the job sandboxes (phase D of the CI-1 plan) with a stub executor; the host loop that fetches jobs from ermite comes later.
13-
14-## Sandboxes
15-
16-Both modes run the same executable, the runner's own, attached **read-only** to the sandbox (never baked into the image, so a runner upgrade needs no image rebuild): `ermite-runner init` is the entrypoint (PID 1 of a VM), then `ermite-runner exec` connects back to the host over a framed protobuf stream (`internal/execstream/exec.proto`; vsock in a VM, a Unix socket in a container). The executable must be static: build with `CGO_ENABLED=0`.
17-
18-### vm
19-
20-- Firecracker is driven through its HTTP API on a unix socket with `net/http`, not through `firecracker-go-sdk` (its last release, v1.0.0 from 2022, predates Firecracker 1.17 and pulls CNI, go-openapi and netlink for a handful of PUT requests).
21-- Disks: the runner executable (read-only boot root, ext4 built with `mkfs.ext4 -d`, cached by the executable's hash), the shared read-only base image, a per-job sparse scratch disk (`ERMITE_RUNNER_JOB_DISK`) and, when there are externals (below), a fourth read-only disk with the host's Node.js builds, cached by their content. The guest init builds an overlayfs root (lower = base, upper = scratch), mounts the externals disk (`/dev/vdd`) at `/home/runner/externals` and keeps dockerd's data root on the scratch disk. The scratch disk and `eth0` are rate limited (defaults 400 MiB/s and 20000 IOPS, 1 Gbit/s each way; `ERMITE_RUNNER_JOB_DISK_BANDWIDTH` in MiB/s, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` in Mbit/s; 0 disables).
22-- The jailer runs each VM as uid/gid `ERMITE_RUNNER_UID_BASE` + slot in its own chroot under the state directory, in a new PID namespace and in a cgroup v2 with `memory.max` (guest memory + 256 MiB for the VMM) and `pids.max`. The kernel, the base image and the state directory must be on the same filesystem (files are hard-linked into the jail) and the kernel and base image must be world-readable.
23-- **The state directory and every parent of it must be owned by root and not writable by group or others.** The jailer runs as root and builds the chroots under `<state dir>/jail`; the driver checks that `jail` itself is a real directory, root-owned and not group/other-writable, but it does not walk the parents: a user who can rename or replace a parent directory can still redirect root's file operations. `/var/lib/ermite-runner` (the default, created by the unit's `StateDirectory`) is fine; do not point `ERMITE_RUNNER_STATE_DIR` into a home directory or `/tmp`.
24-- One TAP (`ertap<slot>`) per VM with a /30 from `ERMITE_RUNNER_NETWORK` (default `10.200.0.0/16`), vsock for the executor stream, IPv6 disabled in the guest.
25-- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires (the plan runs the host process as root under systemd).
26-
27-### container
28-
29-- Docker Engine API via the maintained client `github.com/moby/moby/client` (`DOCKER_HOST` honoured). Runtime `sysbox-runc` when the daemon lists it, else a **privileged** container, which is host-root equivalent. That is allowed by default (ADR 0025) but never silent: the runner logs a warning at start and at the start of every job (`privileged container: host-root equivalent, network policy best effort`), `Driver.Isolation()` reports `container_privileged` (or `container_sysbox`, `vm`) for the registration with ermite, and `ERMITE_RUNNER_DENY_PRIVILEGED=1` refuses to start without sysbox.
30-- A **rootless** Docker daemon is refused with a clear error (a nested dockerd needs a rootful daemon or sysbox).
31-- Limits per job: memory with swap equal to memory (no swap), `ERMITE_RUNNER_JOB_PIDS` processes (default 32768), CPUs.
32-- Disk: the job's writable space is enforced like in `vm`: a sparse ext4 file of `ERMITE_RUNNER_JOB_DISK`, **loop-mounted on the host** (so this mode needs root, like the rest of it) and bind-mounted at `/scratch`. The nested dockerd's data root, `/home/runner/work` and `/tmp` are directories of that disk; the container's own writable layer stays small. A job that fills its disk gets "no space left on device" instead of filling the host.
33-- Per job: a bridge network (`erbr<slot>`), the scratch disk, a Unix socket in a bind-mounted per-job directory of the state directory (the job sees only that directory, not the scratch file). Every Docker object carries the label `dev.bnema.ermite-runner.job`; leftovers are reaped by label at start. The state directory path must keep the socket path under 107 bytes.
34-- The Docker daemon must run on the same host as the runner (host paths are bind-mounted).
35-
36-### Externals (Node.js)
37-
38-The host provides Node.js to the jobs **outside the image**, like its own executable, so that a runner upgrade or a new Node.js version never needs an image rebuild. The builds are the ones of `actions/runner` (`Misc/externals.sh`), pinned in `internal/externals/table.go` with the SHA-256 of each archive:
39-
40-| Name | Node.js | Source | Architectures |
41-| --- | --- | --- | --- |
42-| `node20` | 20.20.2 | `nodejs.org/dist/v20.20.2/node-v20.20.2-linux-<arch>.tar.gz` | x64, arm64 |
43-| `node24` | 24.21.0 | `nodejs.org/dist/v24.21.0/node-v24.21.0-linux-<arch>.tar.gz` | x64, arm64 |
44-| `node20_alpine` | 20.20.2 | `github.com/actions/alpine_nodejs` release `v20.20.2` | x64 |
45-| `node24_alpine` | 24.21.0 | `github.com/actions/alpine_nodejs` release `v24.21.0` | x64 |
46-
47-- **Node versions of actions** follow github.com today (`RequireNode24` and `Node24ByDefault` of the executor's `ActionsConfig`, both set by the sandbox through `DefaultActionsConfig`): `using: node12`, `node16` and `node20` actions run on `node24`, whatever `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24` and `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION` say, and `Complete job` ends with `##[warning]Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: <actions, sorted ignoring case>. For more information see: …` (recordings `act-node-*`). Both flags off give the older phases of the runner.
48-- **Download.** `ermite-runner run` (and `sandbox-check`) fetch what is missing at start, into `$ERMITE_RUNNER_STATE_DIR/externals/<name>-<version>-<arch>/`, in the background; the start waits for them at most 2 minutes (a runner with a network normally takes its first job with everything in place, one without never waits long). The archive is the `.tar.gz` (the Go standard library has no xz); its SHA-256 is checked **before** anything is unpacked. Redirects must stay on https.
49-- **Extraction** keeps the top directory out and refuses absolute names, `..`, devices and fifos, symbolic links that are absolute or climb out of the tree, entries below a link, hard links to anything but an unpacked file, and archives over 200,000 entries or 2 GiB; writes go through `os.Root`. The tree is built in a temporary directory and renamed into place, so a directory under its final name is always complete.
50-- **Only pinned versions are kept**: at every start everything else in `externals/` (other versions, other architectures, interrupted downloads) is removed.
51-- **Layout in the sandbox** (GitHub's): `/home/runner/externals/node20/bin/node`, `/node24/bin/node`, and the alpine ones for job containers, read-only. In `container` mode they are read-only bind mounts of the host directories; in `vm` mode one ext4 disk (`mkfs.ext4 -d`) named after the content of the set (`cache/extdisk-<key>.ext4`), so a new Node.js builds a new disk and leaves the executable's disk and the base image alone. Superseded disks are removed an hour after the last build. The set is read at every job start: a download that finishes later is used by the next job.
52-- **When a build cannot be provided** (offline, HTTP error, wrong checksum, unsafe archive, no build for the architecture) the runner **still runs**: it logs why and attaches in its place a directory with the file `UNAVAILABLE` (the reason) and a `bin/node` script, `cat "$(dirname "$0")/../UNAVAILABLE" >&2; exit 1`, so a job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: download …: SHA-256 mismatch …` rather than `No such file or directory`. These directories (`externals/.unavailable/<name>-<reason hash>`) are built once, complete, and renamed into place; a file is never rewritten in place, so a reader (the build of a VM's externals disk, a job running the stub) never sees a half-written one. A directory handed out to a sandbox in the last hour is kept by the pruning, since a running container may have it bind-mounted.
53-- **Retries.** A download that failed for a reason that may pass (network error, HTTP status) is retried every 5 minutes by a timer, whether jobs come or not, until the runner stops. A verdict (wrong checksum, unsafe archive, no build for the architecture) is not retried before the next start.
54-- **Archive layouts.** The nodejs.org archives have one top directory (`node-vX-linux-x64/`), which is dropped; the `actions/alpine_nodejs` archives start at `./bin/node` and are unpacked as they are. The alpine checksums are the digests of the GitHub release assets.
55-- `ERMITE_RUNNER_NODE=0` turns all of this off: nothing is downloaded or attached.
56-
57-### One runner per host
58-
59-Reaping removes every process, device, firewall rule and Docker object the runner owns, so two runners on one host would destroy each other's jobs. Both modes take an exclusive lock at a fixed host path, `/run/ermite-runner.lock` (`ERMITE_RUNNER_LOCK_FILE` overrides it, for tests), and refuse to start when it is held, whatever their state directories or isolation modes. The kernel drops the lock when the process dies. A runner in a container or in another mount namespace must share that path (or it is not protected): do not run it so.
60-
61-### Network policy (both modes)
62-
63-One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below; the cache has no rule, see "Cache"), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
64-
65-**Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
66-
67-### Guest forge access (opt-in, staging only)
68-
69-Off by default. `ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the forge **the way the runner does**, for a staging host where ermite is only reachable through an overlay network and the runner itself uses a local route, e.g. `/etc/hosts` `forge.example -> 127.0.0.1` and a reverse proxy on `*:8088` with `ERMITE_RUNNER_URL=http://forge.example:8088`. Jobs otherwise cannot `git clone ${{ github.server_url }}/...`: the public URL (`https://forge.example`) is TLS-terminated off-host, and the private ranges are denied.
70-
71-The host name and port come from `ERMITE_RUNNER_URL` (port defaulted from the scheme: 443 or 80). The runner **refuses to start** when the URL host is an IP literal (sandboxes are pointed at the host by DNS), when the URL has a path, when the name does not resolve on the host, or when the URL is plain `http://` without `ERMITE_RUNNER_INSECURE_HTTP=1` (same rule as for the runner itself: tokens then travel in clear). Then, for every sandbox in both modes:
72-
73-- the DNS forwarder answers `A` queries for exactly that host name (case-insensitive, trailing dot ignored) with the sandbox's **gateway address**, and `AAAA` with an empty `NOERROR` answer; every other name is forwarded as before;
74-- the nftables input chain accepts TCP from the sandbox's own address to its own gateway address on that one port, before the sandbox's catch-all drop (one extra line per sandbox; see `internal/netpolicy/testdata/guest_forge.nft`). Nothing else on the host opens; the forward chain (private-range denies, source check, no sandbox-to-sandbox) is unchanged;
75-- the `github.server_url`, `github.api_url` and `github.graphql_url` of the job's contexts, and so `GITHUB_SERVER_URL`, `GITHUB_API_URL` and `GITHUB_GRAPHQL_URL`, are rewritten to the origin of `ERMITE_RUNNER_URL` when their origin is the server's public one (`github.server_url`); paths are kept (`/-/github/api`). Other values, and the URLs inside `github.event`, stay as the server recorded them;
76-- `guest forge access: sandboxes may reach <origin> on the host` is logged at start and at the start of every job, like the privileged-container warning.
77-
78-**The runner does not proxy.** The traffic reaches the host's own listener on that port, so a service must listen on it on an address the gateway reaches (`*`; a listener bound to `127.0.0.1` receives nothing). A shared reverse proxy on that port routes by `Host` header, so **a job can reach every app that proxy serves on that port**, not only the forge, by sending another `Host` header: use this on staging hosts only, with a proxy whose other apps you accept to expose to job code. That traffic comes from a private address (`10.200.x.x`), so anything behind the proxy that trusts private sources (admin pages, metrics, rate-limit exemptions) trusts job code too. Jobs also see the forge over plain HTTP in that setup.
79-
80-### Cache (GitHub Actions cache, API v1)
3+<p align="center">
4+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="License: MIT"></a>
5+ <a href="https://github.com/bnema/ermite-runner/commits/main"><img src="https://badgen.net/github/last-commit/bnema/ermite-runner/main?icon=github" alt="Last commit"></a>
6+ <a href="https://github.com/bnema/ermite-runner/stargazers"><img src="https://badgen.net/github/stars/bnema/ermite-runner?icon=github" alt="GitHub stars"></a>
7+</p>
818
82-The runner serves the cache of `actions/cache`, `setup-go`/`setup-node` with `cache:` and everything else built on `@actions/cache`. The toolkit uses **cache API v1** for any `GITHUB_SERVER_URL` that is not `github.com`, `*.ghe.com` or `*.localhost` (`toolkit/packages/cache/src/internal/config.ts:1-20`), so that is what is served: `GET _apis/artifactcache/cache?keys=&version=`, `POST caches` (reserve), `PATCH caches/{id}` (chunks with `Content-Range`, in any order), `POST caches/{id}` (commit), and the archive by a plain `GET` of the `archiveLocation`. `ACTIONS_CACHE_SERVICE_V2` is never set.
9+<p align="center">The CI runner of <a href="https://github.com/bnema/ermite">ermite</a>: runs GitHub Actions workflows, one microVM per job.</p>
8310
84-- **Where.** One service per runner, reachable **only from inside the sandbox**: on its loopback, `127.0.0.1:9089`, and on the address of the sandbox's own Docker bridge (`docker0`, typically `172.17.0.1:9089`) for the steps that run in a container; it listens on no network of the host and on no host port, and the host firewall has no rule for it (nothing to open on the host). A small relay in the sandbox (`internal/guest/relay.go`, started with the job and ended with it) listens on those addresses and pipes each TCP connection, in both directions with half-close, to a new connection on the host channel the executor already uses, and the runner serves HTTP on the other end:
85- - **VM:** the vsock port 9089 (the relay connects to the host, CID 2). Firecracker hands the connection to the VM's own Unix socket `<vm dir>/v.sock_9089`, next to the one for the executor (`_1024`); with the jailer both are in the VM's chroot.
86- - **Container:** `cache.sock` next to `exec.sock` in the per-job directory that is bind-mounted as `/run/ermite`. That directory exists for that container only.
87-
88- Each sandbox has its own listener and handler on the runner, and the runner accepts on it only the token made for that sandbox's job, so one sandbox can neither reach another's channel nor use a token on it that was not made for its job. The connection caps apply to the channel (32 per sandbox, 128 in all; each relay allows 32 open connections). Node actions on the sandbox get `ACTIONS_CACHE_URL=http://127.0.0.1:9089/` (the toolkit appends `_apis/…`), `ACTIONS_RUNTIME_URL` (same value), `ACTIONS_RUNTIME_TOKEN` and `ACTIONS_CACHE_MODE`; `run:` steps get none, as on the official runner. The token is masked. **Containers:** a job container and a docker action have a network namespace of their own, whose loopback is not the sandbox's, so node steps in a job container and docker actions get the same variables with the bridge address instead (`http://<docker0>:9089/`, read when the job starts: the nested daemon picks it). A container on any bridge network of that daemon reaches a local address of the sandbox through `INPUT`, which Docker does not filter: no `--add-host`, no name, no host rule. Without a Docker bridge those steps get no runtime variables. A relay that cannot start (port taken) leaves that job without a cache; the job runs.
89-- **Storage.** `<state dir>/cache/data/<id>` plus `index.json` (mode 0700/0600), written atomically (temporary file, fsync, rename, directory fsync). Total size is capped by `ERMITE_RUNNER_CACHE_MAX` (default 10G; `500M`, `20G`…): the least recently used entries are evicted first, both on commit and when a reservation needs room. A reservation is dropped after an hour without activity, and in any case at the expiry of the token that made it or an hour after it was made, whichever comes first (one with a chunk in flight is reclaimed when that chunk ends; every transfer's deadline is capped at the token's expiry). A restart keeps the entries and drops unfinished uploads.
90-- **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref only and reads its read refs, in priority order; keys are tried in the order the toolkit sends them in each ref (the primary key exactly, restore keys as prefixes, newest first). Another repository's entries are never visible. A second reservation of the same (repository, ref, key, version) while one exists or is in flight is a `409`.
91-- **Tokens.** `ACTIONS_RUNTIME_TOKEN` is a JWT (three base64url segments, HS256) whose claims are the job id, repository, write ref, read refs, expiry (job timeout plus 15 min), `iat`, `exp` and `scp` (`Actions.GenericRead:…`, plus the job's `Actions.Results:<run>:<job>` when it has the results service, see Artifacts). The key is derived for tokens from a random key made at start and kept in memory only (a restart ends every job anyway), so no URL signature can pass for a token. The comparison is constant time; everything wrong (no token, bad signature, expired, ids of another job or ref, a token without a cache scope) is a bare `403`. The token is not an ermite credential and ermite refuses it. The archive URL carries its own HMAC (entry, repository, issuing job, expiry) because the toolkit downloads without the bearer token; a sandbox's channel serves only URLs issued to its own job, so a URL taken out of one sandbox is refused on every other's.
92-- **Off.** `ERMITE_RUNNER_CACHE=0`.
93-- **Action archives.** The same channel serves the tarballs of actions. When the server resolves an action to a commit, the host downloads the archive once into `<state dir>/actions-cache/` (keyed by forge, repository and commit sha, never by ref) and points the sandbox at a signed URL on the cache channel (bound to the job, the key and the expiry); the sandbox extracts it as it would a download, and "Download action repository" is printed on a hit as on a miss. Concurrent jobs share one download; each entry stores its sha256, which is checked on every read (a corrupt entry is dropped and fetched again); least recently used entries are evicted above `ERMITE_RUNNER_ACTIONS_CACHE_MAX` (default `2G`, `0` turns it off), and an entry just handed out is kept for 30 minutes, and downloads in progress are charged against the cap as they are written (an archive that does not fit is not cached and the sandbox downloads it). The sandbox never writes to this cache. Whatever cannot be cached (no cache channel for the job, no sha, a failed host download) is downloaded by the sandbox as before.
94-
95-**Scopes from the server.** The server sends `Job.cache {repository, write_ref, read_refs}` (feature `ci.cache`). The runner checks it (`internal/host/cache.go`): the repository must be the job's own, and the scope always carries the job's canonical `owner/name`; refs are full `refs/…` names, empty ones are dropped, at most 8 read refs; an empty `write_ref` makes the job read-only (reserve answers `403 cache write denied:`). A scope that fails the checks disables the cache for that job with a log line; the job runs. **Old server fallback:** with no `Job.cache` the scope is `github.repository` and `Job.ref`, for reading and writing, with no base or default branch fallback; a job without either gets no cache.
11+> Read-only mirror. Development happens at git.bnema.dev; issues and pull requests are closed here.
9612
97-**Limits.** Every entry and reservation is charged at least 4 KiB, so tiny entries cannot pile up; uploads that give no `cacheSize` are charged for the bytes they write, before the bytes are accepted. At most 256 uploads in progress (32 per job); an entry being downloaded is never evicted (its bytes stay charged), and a reservation that cannot be given room answers `503`. At most 32 connections per sandbox and 128 in all; an upload or download that makes no progress for 60 s is cut. **Bandwidth:** a VM's cache traffic does not cross `eth0`, so the runner limits it itself: one limit per sandbox, shared by all its connections and both directions (request bodies and replies or archives), at the sandbox's `ERMITE_RUNNER_JOB_NET_BANDWIDTH` (Mbit/s, same default; `0` is unlimited). A transfer over the limit is slowed, not cut. Containers have no per-job network rate limit, so their cache channel is unlimited.
13+---
9814
99-### Artifacts (artifacts v4)
15+ermite-runner fetches jobs from an ermite server and runs workflows written for GitHub Actions (`.github/workflows/*.yml`). Each job runs in its own sandbox, destroyed when the job ends.
10016
101-`actions/upload-artifact@v4+` and `actions/download-artifact@v4+` (and anything else bundling `@actions/artifact` v2) work when the server has the feature `ci.artifacts`. **ermite stores the artifacts** (cross-runner downloads, retention, quota, the run page and `ermite run download`); the runner keeps no byte of them on disk. It serves GitHub's results service on the same channel and port as the cache (`internal/resultssvc`), and translates it to ermite's `JobArtifactService` with the job token, which never enters the sandbox:
17+## Isolation modes
10218
103-- **Protocol.** Twirp JSON, `POST /twirp/github.actions.results.api.v1.ArtifactService/{CreateArtifact,FinalizeArtifact,ListArtifacts,GetSignedArtifactURL,DeleteArtifact}`, authenticated by `ACTIONS_RUNTIME_TOKEN` (the job's token with its `Actions.Results:<run id>:<job>` scope, which the toolkit decodes for its ids). The bytes go through Azure block blob requests on signed URLs: `PUT ?comp=block` (each 8 MiB block streamed to ermite as it arrives, answered `201` once ermite has stored it), `PUT ?comp=blocklist` (the commit), single-shot `PUT`, and `GET` for downloads (streamed from ermite; a stream that breaks is resumed at the byte reached, up to 3 times). Signed URLs (`/_apis/results/blob/…?se=&sig=`) are bound to the operation, the artifact, the job and the token's expiry, and use the host the step used, so they work from the sandbox and from containers alike.
104-- **Isolation.** Run and job ids come from the job the channel serves, never from request bodies; ermite checks the run again. A job reaches the artifacts of its own run only.
105-- **Errors.** ermite's message reaches the step log. Quota and size limits are `412 failed_precondition` (`413` on a block), not retried; ermite unavailable is `503`, which the toolkit retries.
106-- **Variables.** Node actions and docker actions get `ACTIONS_RESULTS_URL` (the same origin as `ACTIONS_CACHE_URL`, the bridge address in containers), as `NodeScriptActionHandler.cs`/`ContainerActionHandler.cs` set it. A job with no cache scope still gets the channel for artifacts, without `ACTIONS_CACHE_URL`.
107-- **`GITHUB_SERVER_URL` of the artifact steps.** This is the one place where the runner changes an action's environment. `@actions/artifact` refuses to run on GitHub Enterprise Server, which it recognises **only** by `GITHUB_SERVER_URL` (`isGhes()`, `toolkit/packages/artifact/src/internal/shared/config.ts`): any host but `github.com`, `*.ghe.com` and `*.localhost` fails with "not currently supported on GHES". So in the process of a **node** step (main, pre, post, also inside composites) of `actions/upload-artifact` or `actions/download-artifact` (any subpath, such as `actions/upload-artifact/merge`; `upload-pages-artifact` uses it), or of an action whose script contains that toolkit message, `GITHUB_SERVER_URL` is `https://ermite.ghe.localhost`. Nothing resolves or connects to that host: the toolkit only reads it. The `github` context, `${{ github.server_url }}` and every other step keep the real URL. Changing it globally would break git, curl and checkout; forking the actions would break SHA pins.
108-- **Not covered.** Artifact actions pinned below `upload-artifact@v4.3.1` / `download-artifact@v4.1.2` (their toolkit treats `*.localhost` as GHES), artifacts v3 (`@actions/artifact` v1), and the `artifact-url` output, which points at the fake host (the run page in ermite lists the artifacts). Downloads of another run (`run-id` with `github-token`) need ermite's REST endpoints.
109-- **Off.** `ERMITE_RUNNER_ARTIFACTS=0`, or a server without `ci.artifacts`: no `ACTIONS_RESULTS_URL`, no override, and the artifact actions fail with the toolkit's GHES message, as before. The service needs the cache channel: `ERMITE_RUNNER_CACHE=0` turns it off too.
19+| Mode | Sandbox | Needs |
20+| --- | --- | --- |
21+| `vm` | A Firecracker microVM per job | A usable `/dev/kvm` |
22+| `container` | A container per job with its own Docker daemon | A rootful Docker Engine |
11023
111-### Cancellation
24+`ERMITE_RUNNER_ISOLATION=auto` is the default. It picks `vm` when `/dev/kvm` is usable and Firecracker, the guest kernel and the base image exist. Otherwise it picks `container`. The chosen mode and the reasons are logged at start.
11225
113-The host sends a `Cancel{reason}` frame (`execstream.Drive` takes a cancel channel, not a context). The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later, with the timers stopped as soon as the shell exits and nothing signalled afterwards (the group id could be reused). Background children that keep the output pipes open are killed when the shell exits, after a short drain. Init forwards SIGTERM and SIGINT to the executor.
26+Details: [docs/sandboxes.md](docs/sandboxes.md).
11427
115-### Annotations, summaries and masks
28+## Requirements
11629
117-The executor ports the issue handling of the official runner: `::notice`, `::warning` and `::error` (with `title`, `file`, `line`, `endLine`, `col`, `endColumn`), problem matchers (`add-matcher`, `remove-matcher`, .NET regular expressions through `regexp2`, 1 s timeout, `GITHUB_ACTIONS_RUNNER_ISSUE_MATCHER_TIMEOUT`, removed after 3 timeouts) and every `##[error]` the runner itself writes become annotations, at most 10 per level per step, messages cut at 4096 characters; `GITHUB_STEP_SUMMARY` is read after each step (1 MiB). `::add-mask::` adds to the masker at run time, with the encodings of `ValueEncoders.cs`. The executor sends them to the host as `StepAnnotations` and `StepSummary` frames; the host checks the limits again, sends the annotations with the `UpdateJob` that completes the step (the list replaces the step's stored one) and the leftovers with `CompleteJob`, and the summaries with `UploadSummary`. Both are sent only when the server lists `ci.annotations` and `ci.summaries` in `GetVersion` (read once per job). Step `timeout-minutes` is enforced like `StepsRunner` does. Attribution for the ported code and tests: [THIRD_PARTY.md](THIRD_PARTY.md).
30+- A Linux host. The runner runs as root.
31+- `ip`, `nft` and `mkfs.ext4` (e2fsprogs 1.43 or newer).
32+- Go, to build the runner.
33+- For `vm`: `/dev/kvm`, `firecracker` and `jailer` in `PATH`, a guest kernel and a base image.
34+- For `container`: a rootful Docker Engine and the job image. [sysbox](https://github.com/nestybox/sysbox) is optional: without it the runner uses a privileged container.
11835
119-### Cleanup
36+[docs/images.md](docs/images.md) explains how to build the kernel and the images.
12037
121-Everything a job creates (process, TAP or bridge, nftables rules, files, loop mounts, containers, networks, cgroups) is removed by `Stop`, which also runs with a cancelled context, aborts a `Start` in progress, and by `Reap` at start for what a crash left behind. A VM leaves a state record (`<state dir>/vms/<id>.json`) before its first resource. **The network policy and the TAP or bridge are never removed while the VMM or the container may still be running**: when a process cannot be killed or a container cannot be removed, `Stop` returns an error wrapping `sandbox.ErrLeaked`, the sandbox's slot stays taken, and `Driver.Reap` (which returns `sandbox.ErrBusy` while a sandbox is running) finishes the job and frees the slot. The host should stop taking jobs on a leaked slot until then. `Sandbox.ID()` names a sandbox's files and Docker objects.
38+## Build
12239
123-## Running jobs
124-
125-```
126-printf %s "$TOKEN" | ermite-runner register --token-stdin # once
127-ermite-runner run # as a service
128-```
129-
130-`register` picks the isolation mode like `run` will, registers the runner with
131-ermite (name, labels, version, isolation, architecture) and stores the runner
132-token in `$ERMITE_RUNNER_STATE_DIR/runner.json` (mode 0600, in a directory made
133-0700; a file readable by others is refused). `--replace` removes a runner of the
134-same name first.
135-
136-**The token of `register`** (scope `runner:register`: it reads the code and the
137-secrets of every repository, see `runner.proto`) is looked up in this order:
138-standard input with `--token-stdin`, then `ERMITE_RUNNER_REGISTER_TOKEN`, then
139-`--token`. Prefer the first two: **the arguments of a process are visible to
140-every user of the host** (`ps`, `/proc/<pid>/cmdline`), the environment and
141-standard input are not.
142-
143-**The server URL** must be `https://`. Plain `http://` is accepted for
144-`localhost`, `127.0.0.0/8` and `::1` only; `ERMITE_RUNNER_INSECURE_HTTP=1` allows
145-any other `http://` URL (tests, a trusted network) and sends every token in
146-clear. The client never follows a redirect, so a token cannot be carried to
147-another host.
148-
149-`run` reaps what a previous run left, then polls `FetchJob` whenever a slot is
150-free (`ERMITE_RUNNER_CAPACITY`), with a `poll_id` that is kept when a response
151-is lost and obeying `retry_after`. For each job it sends the acknowledging
152-heartbeat first, then one every 30 s (a failed one is retried every 5 s), opens
153-`WatchJob` (a cancel request reaches the executor, which interrupts, terminates
154-and kills like the official runner), forwards the log with `AppendLog`
155-(batches of at most 1,000 lines and 64 KiB per step, numbered offsets, retried
156-with the same lines), sends step snapshots with `UpdateJob` and ends with an
157-idempotent `CompleteJob`. When ermite says the job ended (`JobEnded`) or revokes
158-the assignment (`unauthenticated`) the sandbox is destroyed and nothing more is
159-sent. On SIGINT or SIGTERM the running jobs are cancelled with
160-`runner_shutdown`, reported, and the sandboxes removed. The job token stays in
161-the host process; the sandbox receives the workflow and its contexts only.
162-
163-| Variable | Meaning |
164-| --- | --- |
165-| `ERMITE_RUNNER_URL` | base URL of ermite, as in a browser (the runner adds `/-/api` itself) |
166-| `ERMITE_RUNNER_NAME` | runner name (default: the host name) |
167-| `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |
168-| `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` (default) or `self-hosted`: the `runner.environment` value the jobs see. ermite's runners stand in for GitHub-hosted ones, so workflows written for github.com (`runner.environment == 'github-hosted'`) behave the same; `self-hosted` also makes "Set up job" print the runner and machine name, as GitHub does for self-hosted runners |
169-| `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` (see above) |
170-| `ERMITE_RUNNER_INSECURE_HTTP` | `1` allows a non-loopback `http://` URL (tests only); also required for `ERMITE_RUNNER_GUEST_FORGE` with an `http://` URL |
171-| `ERMITE_RUNNER_GUEST_FORGE` | `1` lets sandboxes reach the forge as the runner does (staging only; see "Guest forge access") |
172-| `ERMITE_RUNNER_LOG_LEVEL` | `debug`, `info` (default), `warn`, `error` |
173-| `ERMITE_RUNNER_LOG_FORMAT` | `console` (default, colours) or `json` |
174-
175-## Installing the service
176-
177-`contrib/ermite-runner.service` is a systemd unit. The runner runs as **root**
178-on the host (TAP devices, nftables, loop mounts, the jailer, containers); the
179-jobs never do: they run as user `runner` in a micro-VM or a container.
40+The binary must be static:
18041
18142 ```sh
182-sudo install -m 0755 ermite-runner /usr/local/bin/ermite-runner # a static build: CGO_ENABLED=0
183-sudo install -m 0644 contrib/ermite-runner.service /etc/systemd/system/
184-sudo install -d -m 0700 /etc/ermite-runner
185-sudo install -m 0600 /dev/stdin /etc/ermite-runner/env <<'ENV'
186-ERMITE_RUNNER_URL=https://ermite.example.org
187-ERMITE_RUNNER_NAME=runner-1
188-ERMITE_RUNNER_LABELS=ubuntu-latest,ubuntu-24.04
189-ERMITE_RUNNER_ISOLATION=auto
190-ERMITE_RUNNER_CAPACITY=2
191-ERMITE_RUNNER_JOB_CPUS=4
192-ERMITE_RUNNER_JOB_MEMORY=8G
193-ERMITE_RUNNER_JOB_DISK=64G
194-ERMITE_RUNNER_LOG_FORMAT=json
195-ENV
196-# Register once, in the same environment as the service (the settings, the state
197-# directory), with the token on standard input so that it is not in any
198-# command line:
199-printf %s "$TOKEN" | sudo systemd-run --pipe --wait -p EnvironmentFile=/etc/ermite-runner/env \
200- -p StateDirectory=ermite-runner -p StateDirectoryMode=0700 \
201- /usr/local/bin/ermite-runner register --token-stdin
202-sudo systemctl enable --now ermite-runner
203-journalctl -u ermite-runner -f
43+CGO_ENABLED=0 go build -trimpath -o ermite-runner .
20444 ```
20545
206-`systemd-run` reads the same `EnvironmentFile` as the unit, so `register` and
207-`run` agree on the URL, the name, the labels, the isolation mode and the state
208-directory (`/var/lib/ermite-runner`, the default of `ERMITE_RUNNER_STATE_DIR`).
209-
210-**Logs.** The runner writes to standard error, which journald collects. Set
211-`ERMITE_RUNNER_LOG_FORMAT=json` under journald: one JSON event per line, with
212-`job_id`, `run_id` and `attempt` on every line about a job, `warn` for what is
213-retried and `error` for what is refused or leaked; the colours of the `console`
214-format are noise in a journal. `ERMITE_RUNNER_LOG_LEVEL=debug` adds detail.
215-Tokens are never logged.
216-
217-**Stopping.** `systemctl stop` sends SIGTERM: running jobs are cancelled with
218-`runner_shutdown`, reported to ermite, and their sandboxes removed before the
219-service exits. The budget, with the defaults, is **90 s**: 30 s for the executor
220-to stop its steps (SIGINT, SIGTERM after 7.5 s, SIGKILL 2.5 s later, then the
221-post-steps), 30 s to destroy the sandbox, 30 s to flush the log and report; jobs
222-do this in parallel, so it does not grow with `ERMITE_RUNNER_CAPACITY`. The unit
223-sets `TimeoutStopSec=120`, 30 s of margin (a test in `internal/host` fails if the
224-two drift apart). A job whose first heartbeat is still pending when the signal
225-arrives is not started: ermite gives it to another runner after 30 s. A crash or
226-a power cut leaves sandboxes behind; the next start reaps them. Only one runner
227-may live on a host (a host-wide lock enforces it), because reaping removes
228-everything labelled as the runner's own.
229-
230-**Sandboxing of the unit.** The runner has to be root (TAP devices, nftables,
231-loop mounts, the jailer, Docker), so the unit only takes away what it does not
232-use, and only with options that **leave it in the host's mount namespace**:
233-`NoNewPrivileges`, `ProtectClock`, `RestrictSUIDSGID`, `LockPersonality`,
234-`RestrictRealtime`, `SystemCallArchitectures=native` and `RestrictAddressFamilies`
235-(UNIX, INET, INET6, NETLINK, VSOCK).
236-
237-**Deliberately absent**, and why:
238-
239-- **Every option that gives the unit a mount namespace of its own:** `PrivateTmp`,
240- `ProtectHome`, `ProtectSystem`, `ProtectKernelModules`, `ProtectKernelLogs`,
241- `ProtectKernelTunables`, `ProtectControlGroups`, `PrivateDevices`,
242- `ReadOnlyPaths`, `ReadWritePaths`, `InaccessiblePaths`, `TemporaryFileSystem`,
243- `BindPaths`. In container mode the runner loop-mounts the scratch disk of each
244- job and the Docker daemon, which lives in the host's namespace, binds that
245- directory into the container; a mount made inside the unit's namespace is
246- invisible to it and the container would get an empty directory. (`ProtectKernelModules`
247- and `ProtectKernelLogs` look harmless but are implemented with mounts too.) The
248- micro-VM mode would tolerate them, but one unit serves both modes, and a second
249- unit for VM-only hardening is not worth maintaining. A test in `internal/host`
250- fails if the unit sets any of them.
251-- `Delegate=`: the runner manages no cgroup of its own (the jailer writes the
252- VMs' cgroups as root, Docker has its own).
253-- `RestrictNamespaces=`, `PrivateNetwork=`, `CapabilityBoundingSet=`,
254- `SystemCallFilter=`: network namespaces and devices (TAP, veth), mounts,
255- `net.ipv4.ip_forward`, capabilities handed to the jailer and Docker.
256-- `MemoryDenyWriteExecute=`: Firecracker maps guest memory writable and executable.
257-- `UMask=`: the drivers create directories that the job's user must enter, with
258- explicit modes; the state directory is made 0700 by the runner.
259-
260-The boundary of a job is its micro-VM or container, not this unit. Checked on a
261-workstation (a user manager, no root): the micro-VM job test (`TestVMRunsAJob`:
262-user and network namespaces, Firecracker, TAP, vsock) passes under the options
263-that remain. Not checked, because they need root on a real host: the jailer, loop
264-mounts, nftables in the host namespace, Docker.
265-
266-## The image
267-
268-A job runs in the image named by `ERMITE_RUNNER_IMAGE` (container: an OCI
269-reference; vm: the directory with `vmlinux` and `base.ext4`, both built from
270-`images/ubuntu-24.04/Dockerfile`). The runner binary and Node.js (see "Externals") are attached to the sandbox
271-by the host and are never part of the image. What the executor needs from the
272-image:
273-
274-- a user **`runner`** with uid and gid 1001 and the home `/home/runner`; the
275- executor starts as root (init needs it to start `dockerd`) and runs every step
276- as this user, with HOME, USER and LOGNAME set. Without such a user the steps
277- run as root (a test image only);
278-- the directories `/home/runner/work` and `/opt/hostedtoolcache` owned by
279- `runner` (`RUNNER_TOOL_CACHE`, which the executor reads from its environment
280- like the official runner; each job gets `/home/runner/work/<repo>/<repo>` and
281- `/home/runner/work/_temp`, cleaned at the start of the job);
282-- **`bash`** and **`sh`** (the default shell is `bash -e {0}` when bash exists,
283- else `sh -e {0}`; like on hosted runners `shell: bash` adds
284- `--noprofile --norc -eo pipefail`), `python3` and a `python` command for
285- `shell: python`; `pwsh` is not part of the CI-1 image (a step that asks for it
286- fails with `pwsh: command not found`);
287-- `dockerd` (and passwordless `sudo` for `runner`) for jobs that use Docker: the
288- job gets its own daemon with its data root on the scratch disk;
289-- what workflows commonly use: git, curl, jq, build-essential, ca-certificates,
290- tzdata, zip/unzip. Full parity with the hosted images (`actions/runner-images`)
291- is a later phase.
292-
293-## Job containers and service containers
294-
295-`container:` and `services:` work as on GitHub-hosted Linux runners
296-(`internal/executor/containers_*.go`, a port of `ContainerOperationProvider`,
297-`ContainerInfo`, `DockerCommandManager` and `ContainerStepHost` of
298-actions/runner, with their line numbers in the comments). The containers are
299-created in the **sandbox's own `dockerd`**, so nothing reaches the host: in `vm`
300-mode (the mode of hosts without Docker Engine) the daemon runs inside the
301-micro-VM, in `container` mode it is the nested daemon of the job.
302-
303-- **Initialize containers** (a step after "Set up job"): docker version check
304- (API ≥ 1.35), cleanup of labelled leftovers, network `github_network_<32 hex>`,
305- for each container `docker pull` (3 tries, random 1–10 s back off),
306- `docker create`/`start`, `docker port` and the health checks of the services
307- (exponential back off 2–32 s, "`<alias>` service is healthy."). Registry
308- `credentials:` go to a private `docker --config` directory under RUNNER_TEMP
309- (password through stdin, masked in every log, directory removed after the pull).
310-- **Stop containers** (post step, `always()`): service logs, `docker rm --force`,
311- `docker network rm`. It runs after a failure, a cancellation, a step timeout
312- and the job timeout.
313-- **Steps in the job container**: `run:` steps and node actions go through
314- `docker exec -i --workdir … -e NAME… <id> …` (values are passed in the
315- environment of the docker process, translated to `/__w`, `/__t`, `/__e`,
316- `/github/home`, `/github/workflow`, never on the command line). The shell is
317- looked up by the container: `sh -e {0}` unless the step says otherwise. Node
318- actions use the externals mounted read-only at `/__e`, `node<N>_alpine` when
319- `/etc/*release` says alpine. Docker actions run as siblings on the job network.
320-- Contexts: `job.container.{id,network}`, `job.services.<alias>.{id,network,ports}`
321- (host ports from `docker port`).
322-- **Untrusted input.** `options:` is split into arguments like .NET splits a
323- command line and passed to `docker create` as an argument vector, never through
324- a shell, so a workflow cannot inject a second command; it can still pass any
325- `docker create` flag (`--privileged`, `--pid host`, `--device`, …) and bind
326- volumes (`volumes: /:/host`), exactly as on a self-hosted GitHub runner. That
327- is contained by the sandbox (its own daemon, its own disk), not by the runner:
328- **host paths in `volumes:` and `options:` are paths of the sandbox**. A job
329- that mounts `/` exposes the sandbox's filesystem to its own container only.
330- The runner warns, as GitHub does, about a volume that mounts `/`.
331-- Not ported: container hooks (`ACTIONS_RUNNER_CONTAINER_HOOKS`), the
332- "already running in a container" refusal (the sandbox's daemon is local to the
333- executor).
334-
335-## Job limits
336-
337-Each job gets its own resources, set for the runner as a whole:
338-
339-| Variable | Default | Effect |
340-|---|---|---|
341-| `ERMITE_RUNNER_CAPACITY` | `1` | jobs in parallel: one sandbox, one uid, one `/30` network and one disk per slot. The runner polls ermite only when a slot is free. |
342-| `ERMITE_RUNNER_JOB_CPUS` | `2` | vCPUs of a micro-VM; CPU quota of a container |
343-| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | guest memory (`M`, `G`, `T`; at least `256M`); a jailed VMM gets this plus 256 MiB of cgroup headroom |
344-| `ERMITE_RUNNER_JOB_DISK` | `64G` | size of the sparse scratch disk (at least `1G`): the overlay upper layer of a VM and the Docker data root of a container. Deleted with the job |
345-| `ERMITE_RUNNER_JOB_PIDS` | `32768` | process limit of a container |
346-| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits (MiB/s, IOPS, Mbit/s); `0` disables |
347-
348-The job timeout is not a runner setting: it is the workflow's `timeout-minutes`
349-(360 by default), which ermite sends with the job and the executor enforces from
350-the moment it starts.
351-
352-## Configuration (sandbox part)
353-
354-| Variable | Default | |
355-|---|---|---|
356-| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
357-| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | absolute path |
358-| `ERMITE_RUNNER_CAPACITY` | `1` | parallel jobs (slots) |
359-| `ERMITE_RUNNER_IMAGE` | | VM: image directory (`vmlinux`, `base.ext4`; default `<state dir>/image`); container: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) |
360-| `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T`. A warning is logged at start when the state directory has less free space than capacity x disk (disks are sparse: it is only a warning) |
361-| `ERMITE_RUNNER_JOB_PIDS` | `32768` | container process limit |
362-| `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses container mode without sysbox |
363-| `ERMITE_RUNNER_GUEST_FORGE` | `0` | `1` lets sandboxes reach the host and port of `ERMITE_RUNNER_URL` on the host (staging only, see "Guest forge access") |
364-| `ERMITE_RUNNER_CACHE` | `1` | `0` turns the cache service off (see "Cache") |
365-| `ERMITE_RUNNER_CACHE_MAX` | `10G` | size cap of the cache, LRU eviction |
366-| `ERMITE_RUNNER_ACTIONS_CACHE_MAX` | `2G` | size cap of the action archive cache; `0` turns it off |
367-| `ERMITE_RUNNER_ARTIFACTS` | `1` | `0` turns the results service (artifacts v4) off (see "Artifacts") |
368-| `ERMITE_RUNNER_NODE` | `1` | `0` turns the Node.js externals off (see "Externals") |
369-| `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | host-wide single-instance lock (tests only) |
370-| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits in MiB/s, IOPS and Mbit/s; 0 disables |
371-| `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |
372-| `ERMITE_RUNNER_NETWORK` | `10.200.0.0/16` | pool of /30 subnets, one per slot |
373-| `ERMITE_RUNNER_FIRECRACKER`, `_JAILER`, `_KERNEL` | from `PATH` | binaries and kernel overrides |
374-| `ERMITE_RUNNER_JAILER_MODE` | `on` | `off` only for development and tests |
375-
376-## Images and kernel
46+## Register and run
37747
378-```sh
379-scripts/build-kernel.sh tmp/image # downloads Linux 6.18 LTS from kernel.org, builds vmlinux (needs bc, flex, bison, pahole, libelf, openssl headers)
380-scripts/build-image.sh tmp/image # builds the OCI image with docker or podman, exports tmp/image/base.ext4 (no root needed)
381-```
48+1. On the ermite server, create a registration token:
38249
383-`scripts/build-image.sh` tags the OCI image `ermite-runner-ubuntu-24.04:latest`, the default of `ERMITE_RUNNER_IMAGE` in container mode, and needs e2fsprogs >= 1.47.1 (the runner itself needs >= 1.43 and checks it at start).
50+ ```sh
51+ ermite token create my-runner --preset runner
52+ ```
38453
385-The kernel configuration is `images/kernel/config-6.18` (+ per-architecture fragments), checked after `olddefconfig`: the build fails if a listed option did not survive. arm64 builds (`ARCH_NAME=arm64`) but is not tested in CI-1.
54+2. Install the binary, the unit and the settings:
38655
387-`CONFIG_BINFMT_MISC` is built in (the guest has no modules) and the guest init mounts it at `/proc/sys/fs/binfmt_misc`, as systemd does on GitHub's image, so `docker/setup-qemu-action` (`tonistiigi/binfmt --install`) can register QEMU handlers and run foreign-architecture containers. The handlers belong to the job's VM and disappear with it. With a kernel built before this option, the init logs the failed mount and jobs run without it.
56+ ```sh
57+ sudo install -m 0755 ermite-runner /usr/local/bin/ermite-runner
58+ sudo install -m 0644 contrib/ermite-runner.service /etc/systemd/system/
59+ sudo install -d -m 0700 /etc/ermite-runner
60+ sudo install -m 0600 /dev/stdin /etc/ermite-runner/env <<'ENV'
61+ ERMITE_RUNNER_URL=https://ermite.example.com
62+ ERMITE_RUNNER_NAME=runner-1
63+ ERMITE_RUNNER_LABELS=ubuntu-latest,ubuntu-24.04
64+ ERMITE_RUNNER_LOG_FORMAT=json
65+ ENV
66+ ```
38867
389-## Tests
68+3. Register once. The token goes through standard input, so it is in no command line:
39069
391-```sh
392-GOWORK=off go vet ./... && GOWORK=off go test -race ./...
393-```
70+ ```sh
71+ printf %s "$TOKEN" | sudo systemd-run --pipe --wait \
72+ -p EnvironmentFile=/etc/ermite-runner/env \
73+ -p StateDirectory=ermite-runner -p StateDirectoryMode=0700 \
74+ /usr/local/bin/ermite-runner register --token-stdin
75+ ```
39476
395-Unit tests need nothing special (the drivers are tested with a fake Firecracker and a fake Docker engine). Integration tests are opt-in and use the real stub executor: output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
77+4. Start the service:
39678
397-**VM** (`ERMITE_RUNNER_IT_VM=1`): needs `/dev/kvm`, Firecracker, the kernel and base image from the scripts above.
79+ ```sh
80+ sudo systemctl enable --now ermite-runner
81+ journalctl -u ermite-runner -f
82+ ```
39883
399-```sh
400-# Unprivileged: inside a user + network namespace (TAP, nftables and DNS on the TAP address work there).
401-# Jailer off: it needs real root. Everything else is the production path.
402-FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
403-
404-# Privileged, with the jailer (the production configuration):
405-sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
406- ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
407- "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
408-```
409-
410-**Container** (`ERMITE_RUNNER_IT_CONTAINER=1`): needs a rootful Docker daemon, root (loop mounts, `CAP_NET_ADMIN`) and the OCI image (`scripts/build-image.sh`).
411-
412-```sh
413-sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:latest \
414- DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
415-```
84+`ermite-runner run` polls the server for jobs and runs each one in a sandbox, up to `ERMITE_RUNNER_CAPACITY` jobs at a time. On SIGTERM it cancels the running jobs, reports them and removes their sandboxes.
41685
417-**Jobs end to end** (`ERMITE_RUNNER_IT_E2E=1`, `internal/e2e`): the real executor in the real image, started through the container driver. They also run on a workstation with a *rootless* Docker daemon (the test-only option `UnsafeAllowRootlessForTests`; the nested `dockerd` of the image cannot start there, and the jobs do not need it). `mount` and `umount` are replaced by stand-ins, since loop mounts need root.
86+To check an installation without a server, run `ermite-runner sandbox-check`. It runs one stub script in the selected mode.
41887
419-```sh
420-IMAGE_TAG=ermite-runner-ubuntu-24.04:dev scripts/build-image.sh tmp/image # or: docker build -t ermite-runner-ubuntu-24.04:dev images/ubuntu-24.04
421-ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-24.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
422-```
423-
424-- `TestContainerRunsAJobAsTheRunnerUser` and `TestContainerCancelStopsTheJobAndOrphans` drive a job over the stream: steps run as `runner` (1001), file ownership, `GITHUB_PATH` and `GITHUB_ENV`, `python`, cancellation by SIGINT, the `always()` step, and the orphan cleanup of "Complete job".
425-- `TestChainCorpus` runs the whole chain: a fake `RunnerService` (`internal/host/hosttest`, which checks the offsets, limits and tokens of `runner.proto`) hands out 22 jobs of the ermite-conformance corpus, the real host loop starts them in containers (two at a time), and what the fake stored (steps, conclusions, outputs, log lines) is compared with the recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
426-- `TestVMRunsAJob` (`internal/sandbox/integration`, run by `scripts/it-vm-userns.sh`) does the first of these in a micro-VM.
427-- `TestVMNodeExternals` (same place) downloads the pinned `node24` (or takes it from `ERMITE_RUNNER_IT_NODE_DIR`, a directory with `node-v24.21.0-linux-x64.tar.gz`, for an offline host; the checksum and the extraction are the real ones), boots a micro-VM and runs `/home/runner/externals/node24/bin/node -e 'console.log(process.version)'` in it, checking that the mount is read-only.
88+More on the service, logs and shutdown: [docs/service.md](docs/service.md).
42889
429-Checked as root on a real KVM host (Arch Linux, kernel 6.18 LTS, Firecracker and jailer 1.17, rootful podman as the Docker API): the jailer (uid drop, chroot, `memory.max` and `pids.max` cgroups, PID namespace), the VM and container integration scenarios, the firewall policy and NAT, the nested `dockerd` on the loop-mounted scratch disk (full disk gives `ENOSPC`), and `internal/e2e` including `TestChainCorpus`. Not checked: sysbox, and a real Docker Engine (podman's compatibility API is not a drop-in, see below).
90+## Commands
43091
431-**Jailer notes** (found on that host): Firecracker 1.17's jailer needs `--chroot-base-dir` to exist (the driver creates `<state dir>/jail`), starts Firecracker as `/firecracker --id <id> --start-time-us ...` (the `/proc` scan that finds a VMM by id knows that form), and leaves `<state dir>/jail/<exec name>` and the cgroup parent `/sys/fs/cgroup/firecracker` behind (only the per-VM directories are the runner's). The jailed API socket path (`<state dir>/jail/firecracker/<id>/root/run/firecracker.socket`) must fit in 107 bytes; the driver checks it at start.
432-
433-**Host firewalls.** The runner's table has its own `input` hook at priority `filter`; another table with a later hook (firewalld at `filter + 10`) still sees a guest's packet to the gateway after the runner accepted it. A firewalld zone that rejects `udp/tcp 53` (the `public` zone with a DNS reject rule, for instance) therefore breaks the guests' DNS (the symptom is `dns silent` in `TestVMNetworkPolicy`) while NAT keeps working. On such a host, allow DNS from `ertap*` and `erbr*` in that firewall, or run the policy test in a private network namespace (as `scripts/it-vm-userns.sh` does).
92+| Command | Purpose |
93+| --- | --- |
94+| `ermite-runner register [--token-stdin] [--token T] [--replace]` | Register with the server and store the runner token |
95+| `ermite-runner run` | Fetch and run jobs |
96+| `ermite-runner sandbox-check [script]` | Run one stub script in a sandbox |
97+| `ermite-runner version` | Print the version |
98+
99+## Key settings
100+
101+| Variable | Default | Meaning |
102+| --- | --- | --- |
103+| `ERMITE_RUNNER_URL` | | Base URL of the ermite server. Must be `https://` |
104+| `ERMITE_RUNNER_NAME` | host name | Runner name |
105+| `ERMITE_RUNNER_LABELS` | | Extra labels, comma separated |
106+| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
107+| `ERMITE_RUNNER_CAPACITY` | `1` | Jobs in parallel |
108+| `ERMITE_RUNNER_JOB_CPUS` | `2` | CPUs per job |
109+| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | Memory per job |
110+| `ERMITE_RUNNER_JOB_DISK` | `64G` | Scratch disk per job |
111+| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | State directory |
434112
435-**Podman as the Docker API.** `DOCKER_HOST` pointing at rootful podman's compatibility socket gets as far as creating the network, then fails: the compatibility `POST /networks/create` cannot disable netavark's DNS (`dns_enabled` is always true) and `HostConfig.Dns` equal to the network gateway is refused when the container starts (`netavark: error while applying dns entries: DNS server 10.200.0.1 is the gateway of network ...; using it would cause aardvark-dns to forward queries to itself`); the option `com.docker.network.bridge.enable_ip_masquerade` is ignored with a warning, and netavark (`firewall_driver = "nftables"` by default) adds its own rules unless it is configured with `firewall_driver = "none"`. The integration tests passed with a small test-only proxy that rewrites that one request to the libpod endpoint with `dns_enabled=false`, and a private `CONTAINERS_CONF`. Production container mode supports Docker Engine (or sysbox) only.
113+All settings: [docs/configuration.md](docs/configuration.md).
436114
437-`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
115+## Security notes
438116
439-The firewall policy has a packet-level test (`ERMITE_RUNNER_IT_NET=1`, run by `scripts/it-vm-userns.sh`): two guests and an outside host on veth pairs in separate network namespaces, with a forged source and a forged "reply" between guests.
117+- Run only one runner per host.
118+- The registration token can read the code and secrets of every repository. Pass it through standard input.
119+- Without sysbox, `container` mode uses a privileged container, which is equivalent to host root. Set `ERMITE_RUNNER_DENY_PRIVILEGED=1` to refuse it.
120+- The state directory and all its parents must be owned by root and not writable by others.
440121
441-Protobuf: `go generate ./internal/execstream` (buf and protoc-gen-go are pinned as `tool` directives in go.mod); CI lints the proto and fails when the committed generated code differs.
122+## Documentation
442123
443-`ermite-runner sandbox-check [script]` runs one stub job through the selected mode and prints its output: the smoke test of an installation.
124+- [Configuration](docs/configuration.md): every environment variable
125+- [Sandboxes](docs/sandboxes.md): `vm` and `container` modes, network policy, cleanup
126+- [Service](docs/service.md): systemd unit, logs, shutdown
127+- [Images](docs/images.md): job image, guest kernel
128+- [Workflows](docs/workflows.md): job and service containers, cancellation, annotations
129+- [Cache and artifacts](docs/cache-and-artifacts.md): Actions cache and artifacts v4
130+- [Node.js externals](docs/externals.md): Node.js for JavaScript actions
131+- [Development](docs/development.md): build, tests
444132
445133 ## License
446134
447-MIT. See [LICENSE](LICENSE).
135+MIT. See [LICENSE](LICENSE). Third-party attributions: [THIRD_PARTY.md](THIRD_PARTY.md).
@@ -1,447 +1,135 @@
1-# ermite-runner1+<h1 align="center">ermite-runner</h1>
2 2
3-> Read-only mirror. Development happens at git.bnema.dev; issues and pull requests are closed here.3+<p align="center">
4-4+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="License: MIT"></a>
5-The CI runner of [ermite](https://github.com/bnema/ermite). It runs workflows compatible with GitHub Actions (`.github/workflows/*.yml`), one job per sandbox:5+ <a href="https://github.com/bnema/ermite-runner/commits/main"><img src="https://badgen.net/github/last-commit/bnema/ermite-runner/main?icon=github" alt="Last commit"></a>
6-6+ <a href="https://github.com/bnema/ermite-runner/stargazers"><img src="https://badgen.net/github/stars/bnema/ermite-runner?icon=github" alt="GitHub stars"></a>
7-- **`vm`**: a Firecracker micro-VM per job, destroyed at the end. Needs `/dev/kvm`.7+</p>
8-- **`container`**: a container per job with its own Docker daemon, for hosts without KVM. Uses the sysbox runtime when available, else a privileged container.
9-
10-`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when `/dev/kvm` opens read-write and Firecracker, the guest kernel and the base image exist, else `container`. The chosen mode and the reasons are logged at start.
11-
12-Status: early development. This tree has the job sandboxes (phase D of the CI-1 plan) with a stub executor; the host loop that fetches jobs from ermite comes later.
13-
14-## Sandboxes
15-
16-Both modes run the same executable, the runner's own, attached **read-only** to the sandbox (never baked into the image, so a runner upgrade needs no image rebuild): `ermite-runner init` is the entrypoint (PID 1 of a VM), then `ermite-runner exec` connects back to the host over a framed protobuf stream (`internal/execstream/exec.proto`; vsock in a VM, a Unix socket in a container). The executable must be static: build with `CGO_ENABLED=0`.
17-
18-### vm
19-
20-- Firecracker is driven through its HTTP API on a unix socket with `net/http`, not through `firecracker-go-sdk` (its last release, v1.0.0 from 2022, predates Firecracker 1.17 and pulls CNI, go-openapi and netlink for a handful of PUT requests).
21-- Disks: the runner executable (read-only boot root, ext4 built with `mkfs.ext4 -d`, cached by the executable's hash), the shared read-only base image, a per-job sparse scratch disk (`ERMITE_RUNNER_JOB_DISK`) and, when there are externals (below), a fourth read-only disk with the host's Node.js builds, cached by their content. The guest init builds an overlayfs root (lower = base, upper = scratch), mounts the externals disk (`/dev/vdd`) at `/home/runner/externals` and keeps dockerd's data root on the scratch disk. The scratch disk and `eth0` are rate limited (defaults 400 MiB/s and 20000 IOPS, 1 Gbit/s each way; `ERMITE_RUNNER_JOB_DISK_BANDWIDTH` in MiB/s, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` in Mbit/s; 0 disables).
22-- The jailer runs each VM as uid/gid `ERMITE_RUNNER_UID_BASE` + slot in its own chroot under the state directory, in a new PID namespace and in a cgroup v2 with `memory.max` (guest memory + 256 MiB for the VMM) and `pids.max`. The kernel, the base image and the state directory must be on the same filesystem (files are hard-linked into the jail) and the kernel and base image must be world-readable.
23-- **The state directory and every parent of it must be owned by root and not writable by group or others.** The jailer runs as root and builds the chroots under `<state dir>/jail`; the driver checks that `jail` itself is a real directory, root-owned and not group/other-writable, but it does not walk the parents: a user who can rename or replace a parent directory can still redirect root's file operations. `/var/lib/ermite-runner` (the default, created by the unit's `StateDirectory`) is fine; do not point `ERMITE_RUNNER_STATE_DIR` into a home directory or `/tmp`.
24-- One TAP (`ertap<slot>`) per VM with a /30 from `ERMITE_RUNNER_NETWORK` (default `10.200.0.0/16`), vsock for the executor stream, IPv6 disabled in the guest.
25-- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires (the plan runs the host process as root under systemd).
26-
27-### container
28-
29-- Docker Engine API via the maintained client `github.com/moby/moby/client` (`DOCKER_HOST` honoured). Runtime `sysbox-runc` when the daemon lists it, else a **privileged** container, which is host-root equivalent. That is allowed by default (ADR 0025) but never silent: the runner logs a warning at start and at the start of every job (`privileged container: host-root equivalent, network policy best effort`), `Driver.Isolation()` reports `container_privileged` (or `container_sysbox`, `vm`) for the registration with ermite, and `ERMITE_RUNNER_DENY_PRIVILEGED=1` refuses to start without sysbox.
30-- A **rootless** Docker daemon is refused with a clear error (a nested dockerd needs a rootful daemon or sysbox).
31-- Limits per job: memory with swap equal to memory (no swap), `ERMITE_RUNNER_JOB_PIDS` processes (default 32768), CPUs.
32-- Disk: the job's writable space is enforced like in `vm`: a sparse ext4 file of `ERMITE_RUNNER_JOB_DISK`, **loop-mounted on the host** (so this mode needs root, like the rest of it) and bind-mounted at `/scratch`. The nested dockerd's data root, `/home/runner/work` and `/tmp` are directories of that disk; the container's own writable layer stays small. A job that fills its disk gets "no space left on device" instead of filling the host.
33-- Per job: a bridge network (`erbr<slot>`), the scratch disk, a Unix socket in a bind-mounted per-job directory of the state directory (the job sees only that directory, not the scratch file). Every Docker object carries the label `dev.bnema.ermite-runner.job`; leftovers are reaped by label at start. The state directory path must keep the socket path under 107 bytes.
34-- The Docker daemon must run on the same host as the runner (host paths are bind-mounted).
35-
36-### Externals (Node.js)
37-
38-The host provides Node.js to the jobs **outside the image**, like its own executable, so that a runner upgrade or a new Node.js version never needs an image rebuild. The builds are the ones of `actions/runner` (`Misc/externals.sh`), pinned in `internal/externals/table.go` with the SHA-256 of each archive:
39-
40-| Name | Node.js | Source | Architectures |
41-| --- | --- | --- | --- |
42-| `node20` | 20.20.2 | `nodejs.org/dist/v20.20.2/node-v20.20.2-linux-<arch>.tar.gz` | x64, arm64 |
43-| `node24` | 24.21.0 | `nodejs.org/dist/v24.21.0/node-v24.21.0-linux-<arch>.tar.gz` | x64, arm64 |
44-| `node20_alpine` | 20.20.2 | `github.com/actions/alpine_nodejs` release `v20.20.2` | x64 |
45-| `node24_alpine` | 24.21.0 | `github.com/actions/alpine_nodejs` release `v24.21.0` | x64 |
46-
47-- **Node versions of actions** follow github.com today (`RequireNode24` and `Node24ByDefault` of the executor's `ActionsConfig`, both set by the sandbox through `DefaultActionsConfig`): `using: node12`, `node16` and `node20` actions run on `node24`, whatever `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24` and `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION` say, and `Complete job` ends with `##[warning]Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: <actions, sorted ignoring case>. For more information see: …` (recordings `act-node-*`). Both flags off give the older phases of the runner.
48-- **Download.** `ermite-runner run` (and `sandbox-check`) fetch what is missing at start, into `$ERMITE_RUNNER_STATE_DIR/externals/<name>-<version>-<arch>/`, in the background; the start waits for them at most 2 minutes (a runner with a network normally takes its first job with everything in place, one without never waits long). The archive is the `.tar.gz` (the Go standard library has no xz); its SHA-256 is checked **before** anything is unpacked. Redirects must stay on https.
49-- **Extraction** keeps the top directory out and refuses absolute names, `..`, devices and fifos, symbolic links that are absolute or climb out of the tree, entries below a link, hard links to anything but an unpacked file, and archives over 200,000 entries or 2 GiB; writes go through `os.Root`. The tree is built in a temporary directory and renamed into place, so a directory under its final name is always complete.
50-- **Only pinned versions are kept**: at every start everything else in `externals/` (other versions, other architectures, interrupted downloads) is removed.
51-- **Layout in the sandbox** (GitHub's): `/home/runner/externals/node20/bin/node`, `/node24/bin/node`, and the alpine ones for job containers, read-only. In `container` mode they are read-only bind mounts of the host directories; in `vm` mode one ext4 disk (`mkfs.ext4 -d`) named after the content of the set (`cache/extdisk-<key>.ext4`), so a new Node.js builds a new disk and leaves the executable's disk and the base image alone. Superseded disks are removed an hour after the last build. The set is read at every job start: a download that finishes later is used by the next job.
52-- **When a build cannot be provided** (offline, HTTP error, wrong checksum, unsafe archive, no build for the architecture) the runner **still runs**: it logs why and attaches in its place a directory with the file `UNAVAILABLE` (the reason) and a `bin/node` script, `cat "$(dirname "$0")/../UNAVAILABLE" >&2; exit 1`, so a job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: download …: SHA-256 mismatch …` rather than `No such file or directory`. These directories (`externals/.unavailable/<name>-<reason hash>`) are built once, complete, and renamed into place; a file is never rewritten in place, so a reader (the build of a VM's externals disk, a job running the stub) never sees a half-written one. A directory handed out to a sandbox in the last hour is kept by the pruning, since a running container may have it bind-mounted.
53-- **Retries.** A download that failed for a reason that may pass (network error, HTTP status) is retried every 5 minutes by a timer, whether jobs come or not, until the runner stops. A verdict (wrong checksum, unsafe archive, no build for the architecture) is not retried before the next start.
54-- **Archive layouts.** The nodejs.org archives have one top directory (`node-vX-linux-x64/`), which is dropped; the `actions/alpine_nodejs` archives start at `./bin/node` and are unpacked as they are. The alpine checksums are the digests of the GitHub release assets.
55-- `ERMITE_RUNNER_NODE=0` turns all of this off: nothing is downloaded or attached.
56-
57-### One runner per host
58-
59-Reaping removes every process, device, firewall rule and Docker object the runner owns, so two runners on one host would destroy each other's jobs. Both modes take an exclusive lock at a fixed host path, `/run/ermite-runner.lock` (`ERMITE_RUNNER_LOCK_FILE` overrides it, for tests), and refuse to start when it is held, whatever their state directories or isolation modes. The kernel drops the lock when the process dies. A runner in a container or in another mount namespace must share that path (or it is not protected): do not run it so.
60-
61-### Network policy (both modes)
62-
63-One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below; the cache has no rule, see "Cache"), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
64-
65-**Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
66-
67-### Guest forge access (opt-in, staging only)
68-
69-Off by default. `ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the forge **the way the runner does**, for a staging host where ermite is only reachable through an overlay network and the runner itself uses a local route, e.g. `/etc/hosts` `forge.example -> 127.0.0.1` and a reverse proxy on `*:8088` with `ERMITE_RUNNER_URL=http://forge.example:8088`. Jobs otherwise cannot `git clone ${{ github.server_url }}/...`: the public URL (`https://forge.example`) is TLS-terminated off-host, and the private ranges are denied.
70-
71-The host name and port come from `ERMITE_RUNNER_URL` (port defaulted from the scheme: 443 or 80). The runner **refuses to start** when the URL host is an IP literal (sandboxes are pointed at the host by DNS), when the URL has a path, when the name does not resolve on the host, or when the URL is plain `http://` without `ERMITE_RUNNER_INSECURE_HTTP=1` (same rule as for the runner itself: tokens then travel in clear). Then, for every sandbox in both modes:
72-
73-- the DNS forwarder answers `A` queries for exactly that host name (case-insensitive, trailing dot ignored) with the sandbox's **gateway address**, and `AAAA` with an empty `NOERROR` answer; every other name is forwarded as before;
74-- the nftables input chain accepts TCP from the sandbox's own address to its own gateway address on that one port, before the sandbox's catch-all drop (one extra line per sandbox; see `internal/netpolicy/testdata/guest_forge.nft`). Nothing else on the host opens; the forward chain (private-range denies, source check, no sandbox-to-sandbox) is unchanged;
75-- the `github.server_url`, `github.api_url` and `github.graphql_url` of the job's contexts, and so `GITHUB_SERVER_URL`, `GITHUB_API_URL` and `GITHUB_GRAPHQL_URL`, are rewritten to the origin of `ERMITE_RUNNER_URL` when their origin is the server's public one (`github.server_url`); paths are kept (`/-/github/api`). Other values, and the URLs inside `github.event`, stay as the server recorded them;
76-- `guest forge access: sandboxes may reach <origin> on the host` is logged at start and at the start of every job, like the privileged-container warning.
77-
78-**The runner does not proxy.** The traffic reaches the host's own listener on that port, so a service must listen on it on an address the gateway reaches (`*`; a listener bound to `127.0.0.1` receives nothing). A shared reverse proxy on that port routes by `Host` header, so **a job can reach every app that proxy serves on that port**, not only the forge, by sending another `Host` header: use this on staging hosts only, with a proxy whose other apps you accept to expose to job code. That traffic comes from a private address (`10.200.x.x`), so anything behind the proxy that trusts private sources (admin pages, metrics, rate-limit exemptions) trusts job code too. Jobs also see the forge over plain HTTP in that setup.
79-
80-### Cache (GitHub Actions cache, API v1)
81 8
82-The runner serves the cache of `actions/cache`, `setup-go`/`setup-node` with `cache:` and everything else built on `@actions/cache`. The toolkit uses **cache API v1** for any `GITHUB_SERVER_URL` that is not `github.com`, `*.ghe.com` or `*.localhost` (`toolkit/packages/cache/src/internal/config.ts:1-20`), so that is what is served: `GET _apis/artifactcache/cache?keys=&version=`, `POST caches` (reserve), `PATCH caches/{id}` (chunks with `Content-Range`, in any order), `POST caches/{id}` (commit), and the archive by a plain `GET` of the `archiveLocation`. `ACTIONS_CACHE_SERVICE_V2` is never set.9+<p align="center">The CI runner of <a href="https://github.com/bnema/ermite">ermite</a>: runs GitHub Actions workflows, one microVM per job.</p>
83 10
84-- **Where.** One service per runner, reachable **only from inside the sandbox**: on its loopback, `127.0.0.1:9089`, and on the address of the sandbox's own Docker bridge (`docker0`, typically `172.17.0.1:9089`) for the steps that run in a container; it listens on no network of the host and on no host port, and the host firewall has no rule for it (nothing to open on the host). A small relay in the sandbox (`internal/guest/relay.go`, started with the job and ended with it) listens on those addresses and pipes each TCP connection, in both directions with half-close, to a new connection on the host channel the executor already uses, and the runner serves HTTP on the other end:11+> Read-only mirror. Development happens at git.bnema.dev; issues and pull requests are closed here.
85- - **VM:** the vsock port 9089 (the relay connects to the host, CID 2). Firecracker hands the connection to the VM's own Unix socket `<vm dir>/v.sock_9089`, next to the one for the executor (`_1024`); with the jailer both are in the VM's chroot.
86- - **Container:** `cache.sock` next to `exec.sock` in the per-job directory that is bind-mounted as `/run/ermite`. That directory exists for that container only.
87-
88- Each sandbox has its own listener and handler on the runner, and the runner accepts on it only the token made for that sandbox's job, so one sandbox can neither reach another's channel nor use a token on it that was not made for its job. The connection caps apply to the channel (32 per sandbox, 128 in all; each relay allows 32 open connections). Node actions on the sandbox get `ACTIONS_CACHE_URL=http://127.0.0.1:9089/` (the toolkit appends `_apis/…`), `ACTIONS_RUNTIME_URL` (same value), `ACTIONS_RUNTIME_TOKEN` and `ACTIONS_CACHE_MODE`; `run:` steps get none, as on the official runner. The token is masked. **Containers:** a job container and a docker action have a network namespace of their own, whose loopback is not the sandbox's, so node steps in a job container and docker actions get the same variables with the bridge address instead (`http://<docker0>:9089/`, read when the job starts: the nested daemon picks it). A container on any bridge network of that daemon reaches a local address of the sandbox through `INPUT`, which Docker does not filter: no `--add-host`, no name, no host rule. Without a Docker bridge those steps get no runtime variables. A relay that cannot start (port taken) leaves that job without a cache; the job runs.
89-- **Storage.** `<state dir>/cache/data/<id>` plus `index.json` (mode 0700/0600), written atomically (temporary file, fsync, rename, directory fsync). Total size is capped by `ERMITE_RUNNER_CACHE_MAX` (default 10G; `500M`, `20G`…): the least recently used entries are evicted first, both on commit and when a reservation needs room. A reservation is dropped after an hour without activity, and in any case at the expiry of the token that made it or an hour after it was made, whichever comes first (one with a chunk in flight is reclaimed when that chunk ends; every transfer's deadline is capped at the token's expiry). A restart keeps the entries and drops unfinished uploads.
90-- **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref only and reads its read refs, in priority order; keys are tried in the order the toolkit sends them in each ref (the primary key exactly, restore keys as prefixes, newest first). Another repository's entries are never visible. A second reservation of the same (repository, ref, key, version) while one exists or is in flight is a `409`.
91-- **Tokens.** `ACTIONS_RUNTIME_TOKEN` is a JWT (three base64url segments, HS256) whose claims are the job id, repository, write ref, read refs, expiry (job timeout plus 15 min), `iat`, `exp` and `scp` (`Actions.GenericRead:…`, plus the job's `Actions.Results:<run>:<job>` when it has the results service, see Artifacts). The key is derived for tokens from a random key made at start and kept in memory only (a restart ends every job anyway), so no URL signature can pass for a token. The comparison is constant time; everything wrong (no token, bad signature, expired, ids of another job or ref, a token without a cache scope) is a bare `403`. The token is not an ermite credential and ermite refuses it. The archive URL carries its own HMAC (entry, repository, issuing job, expiry) because the toolkit downloads without the bearer token; a sandbox's channel serves only URLs issued to its own job, so a URL taken out of one sandbox is refused on every other's.
92-- **Off.** `ERMITE_RUNNER_CACHE=0`.
93-- **Action archives.** The same channel serves the tarballs of actions. When the server resolves an action to a commit, the host downloads the archive once into `<state dir>/actions-cache/` (keyed by forge, repository and commit sha, never by ref) and points the sandbox at a signed URL on the cache channel (bound to the job, the key and the expiry); the sandbox extracts it as it would a download, and "Download action repository" is printed on a hit as on a miss. Concurrent jobs share one download; each entry stores its sha256, which is checked on every read (a corrupt entry is dropped and fetched again); least recently used entries are evicted above `ERMITE_RUNNER_ACTIONS_CACHE_MAX` (default `2G`, `0` turns it off), and an entry just handed out is kept for 30 minutes, and downloads in progress are charged against the cap as they are written (an archive that does not fit is not cached and the sandbox downloads it). The sandbox never writes to this cache. Whatever cannot be cached (no cache channel for the job, no sha, a failed host download) is downloaded by the sandbox as before.
94-
95-**Scopes from the server.** The server sends `Job.cache {repository, write_ref, read_refs}` (feature `ci.cache`). The runner checks it (`internal/host/cache.go`): the repository must be the job's own, and the scope always carries the job's canonical `owner/name`; refs are full `refs/…` names, empty ones are dropped, at most 8 read refs; an empty `write_ref` makes the job read-only (reserve answers `403 cache write denied:`). A scope that fails the checks disables the cache for that job with a log line; the job runs. **Old server fallback:** with no `Job.cache` the scope is `github.repository` and `Job.ref`, for reading and writing, with no base or default branch fallback; a job without either gets no cache.
96 12
97-**Limits.** Every entry and reservation is charged at least 4 KiB, so tiny entries cannot pile up; uploads that give no `cacheSize` are charged for the bytes they write, before the bytes are accepted. At most 256 uploads in progress (32 per job); an entry being downloaded is never evicted (its bytes stay charged), and a reservation that cannot be given room answers `503`. At most 32 connections per sandbox and 128 in all; an upload or download that makes no progress for 60 s is cut. **Bandwidth:** a VM's cache traffic does not cross `eth0`, so the runner limits it itself: one limit per sandbox, shared by all its connections and both directions (request bodies and replies or archives), at the sandbox's `ERMITE_RUNNER_JOB_NET_BANDWIDTH` (Mbit/s, same default; `0` is unlimited). A transfer over the limit is slowed, not cut. Containers have no per-job network rate limit, so their cache channel is unlimited.13+---
98 14
99-### Artifacts (artifacts v4)15+ermite-runner fetches jobs from an ermite server and runs workflows written for GitHub Actions (`.github/workflows/*.yml`). Each job runs in its own sandbox, destroyed when the job ends.
100 16
101-`actions/upload-artifact@v4+` and `actions/download-artifact@v4+` (and anything else bundling `@actions/artifact` v2) work when the server has the feature `ci.artifacts`. **ermite stores the artifacts** (cross-runner downloads, retention, quota, the run page and `ermite run download`); the runner keeps no byte of them on disk. It serves GitHub's results service on the same channel and port as the cache (`internal/resultssvc`), and translates it to ermite's `JobArtifactService` with the job token, which never enters the sandbox:17+## Isolation modes
102 18
103-- **Protocol.** Twirp JSON, `POST /twirp/github.actions.results.api.v1.ArtifactService/{CreateArtifact,FinalizeArtifact,ListArtifacts,GetSignedArtifactURL,DeleteArtifact}`, authenticated by `ACTIONS_RUNTIME_TOKEN` (the job's token with its `Actions.Results:<run id>:<job>` scope, which the toolkit decodes for its ids). The bytes go through Azure block blob requests on signed URLs: `PUT ?comp=block` (each 8 MiB block streamed to ermite as it arrives, answered `201` once ermite has stored it), `PUT ?comp=blocklist` (the commit), single-shot `PUT`, and `GET` for downloads (streamed from ermite; a stream that breaks is resumed at the byte reached, up to 3 times). Signed URLs (`/_apis/results/blob/…?se=&sig=`) are bound to the operation, the artifact, the job and the token's expiry, and use the host the step used, so they work from the sandbox and from containers alike.19+| Mode | Sandbox | Needs |
104-- **Isolation.** Run and job ids come from the job the channel serves, never from request bodies; ermite checks the run again. A job reaches the artifacts of its own run only.20+| --- | --- | --- |
105-- **Errors.** ermite's message reaches the step log. Quota and size limits are `412 failed_precondition` (`413` on a block), not retried; ermite unavailable is `503`, which the toolkit retries.21+| `vm` | A Firecracker microVM per job | A usable `/dev/kvm` |
106-- **Variables.** Node actions and docker actions get `ACTIONS_RESULTS_URL` (the same origin as `ACTIONS_CACHE_URL`, the bridge address in containers), as `NodeScriptActionHandler.cs`/`ContainerActionHandler.cs` set it. A job with no cache scope still gets the channel for artifacts, without `ACTIONS_CACHE_URL`.22+| `container` | A container per job with its own Docker daemon | A rootful Docker Engine |
107-- **`GITHUB_SERVER_URL` of the artifact steps.** This is the one place where the runner changes an action's environment. `@actions/artifact` refuses to run on GitHub Enterprise Server, which it recognises **only** by `GITHUB_SERVER_URL` (`isGhes()`, `toolkit/packages/artifact/src/internal/shared/config.ts`): any host but `github.com`, `*.ghe.com` and `*.localhost` fails with "not currently supported on GHES". So in the process of a **node** step (main, pre, post, also inside composites) of `actions/upload-artifact` or `actions/download-artifact` (any subpath, such as `actions/upload-artifact/merge`; `upload-pages-artifact` uses it), or of an action whose script contains that toolkit message, `GITHUB_SERVER_URL` is `https://ermite.ghe.localhost`. Nothing resolves or connects to that host: the toolkit only reads it. The `github` context, `${{ github.server_url }}` and every other step keep the real URL. Changing it globally would break git, curl and checkout; forking the actions would break SHA pins.
108-- **Not covered.** Artifact actions pinned below `upload-artifact@v4.3.1` / `download-artifact@v4.1.2` (their toolkit treats `*.localhost` as GHES), artifacts v3 (`@actions/artifact` v1), and the `artifact-url` output, which points at the fake host (the run page in ermite lists the artifacts). Downloads of another run (`run-id` with `github-token`) need ermite's REST endpoints.
109-- **Off.** `ERMITE_RUNNER_ARTIFACTS=0`, or a server without `ci.artifacts`: no `ACTIONS_RESULTS_URL`, no override, and the artifact actions fail with the toolkit's GHES message, as before. The service needs the cache channel: `ERMITE_RUNNER_CACHE=0` turns it off too.
110 23
111-### Cancellation24+`ERMITE_RUNNER_ISOLATION=auto` is the default. It picks `vm` when `/dev/kvm` is usable and Firecracker, the guest kernel and the base image exist. Otherwise it picks `container`. The chosen mode and the reasons are logged at start.
112 25
113-The host sends a `Cancel{reason}` frame (`execstream.Drive` takes a cancel channel, not a context). The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later, with the timers stopped as soon as the shell exits and nothing signalled afterwards (the group id could be reused). Background children that keep the output pipes open are killed when the shell exits, after a short drain. Init forwards SIGTERM and SIGINT to the executor.26+Details: [docs/sandboxes.md](docs/sandboxes.md).
114 27
115-### Annotations, summaries and masks28+## Requirements
116 29
117-The executor ports the issue handling of the official runner: `::notice`, `::warning` and `::error` (with `title`, `file`, `line`, `endLine`, `col`, `endColumn`), problem matchers (`add-matcher`, `remove-matcher`, .NET regular expressions through `regexp2`, 1 s timeout, `GITHUB_ACTIONS_RUNNER_ISSUE_MATCHER_TIMEOUT`, removed after 3 timeouts) and every `##[error]` the runner itself writes become annotations, at most 10 per level per step, messages cut at 4096 characters; `GITHUB_STEP_SUMMARY` is read after each step (1 MiB). `::add-mask::` adds to the masker at run time, with the encodings of `ValueEncoders.cs`. The executor sends them to the host as `StepAnnotations` and `StepSummary` frames; the host checks the limits again, sends the annotations with the `UpdateJob` that completes the step (the list replaces the step's stored one) and the leftovers with `CompleteJob`, and the summaries with `UploadSummary`. Both are sent only when the server lists `ci.annotations` and `ci.summaries` in `GetVersion` (read once per job). Step `timeout-minutes` is enforced like `StepsRunner` does. Attribution for the ported code and tests: [THIRD_PARTY.md](THIRD_PARTY.md).30+- A Linux host. The runner runs as root.
31+- `ip`, `nft` and `mkfs.ext4` (e2fsprogs 1.43 or newer).
32+- Go, to build the runner.
33+- For `vm`: `/dev/kvm`, `firecracker` and `jailer` in `PATH`, a guest kernel and a base image.
34+- For `container`: a rootful Docker Engine and the job image. [sysbox](https://github.com/nestybox/sysbox) is optional: without it the runner uses a privileged container.
118 35
119-### Cleanup36+[docs/images.md](docs/images.md) explains how to build the kernel and the images.
120 37
121-Everything a job creates (process, TAP or bridge, nftables rules, files, loop mounts, containers, networks, cgroups) is removed by `Stop`, which also runs with a cancelled context, aborts a `Start` in progress, and by `Reap` at start for what a crash left behind. A VM leaves a state record (`<state dir>/vms/<id>.json`) before its first resource. **The network policy and the TAP or bridge are never removed while the VMM or the container may still be running**: when a process cannot be killed or a container cannot be removed, `Stop` returns an error wrapping `sandbox.ErrLeaked`, the sandbox's slot stays taken, and `Driver.Reap` (which returns `sandbox.ErrBusy` while a sandbox is running) finishes the job and frees the slot. The host should stop taking jobs on a leaked slot until then. `Sandbox.ID()` names a sandbox's files and Docker objects.38+## Build
122 39
123-## Running jobs40+The binary must be static:
124-
125-```
126-printf %s "$TOKEN" | ermite-runner register --token-stdin # once
127-ermite-runner run # as a service
128-```
129-
130-`register` picks the isolation mode like `run` will, registers the runner with
131-ermite (name, labels, version, isolation, architecture) and stores the runner
132-token in `$ERMITE_RUNNER_STATE_DIR/runner.json` (mode 0600, in a directory made
133-0700; a file readable by others is refused). `--replace` removes a runner of the
134-same name first.
135-
136-**The token of `register`** (scope `runner:register`: it reads the code and the
137-secrets of every repository, see `runner.proto`) is looked up in this order:
138-standard input with `--token-stdin`, then `ERMITE_RUNNER_REGISTER_TOKEN`, then
139-`--token`. Prefer the first two: **the arguments of a process are visible to
140-every user of the host** (`ps`, `/proc/<pid>/cmdline`), the environment and
141-standard input are not.
142-
143-**The server URL** must be `https://`. Plain `http://` is accepted for
144-`localhost`, `127.0.0.0/8` and `::1` only; `ERMITE_RUNNER_INSECURE_HTTP=1` allows
145-any other `http://` URL (tests, a trusted network) and sends every token in
146-clear. The client never follows a redirect, so a token cannot be carried to
147-another host.
148-
149-`run` reaps what a previous run left, then polls `FetchJob` whenever a slot is
150-free (`ERMITE_RUNNER_CAPACITY`), with a `poll_id` that is kept when a response
151-is lost and obeying `retry_after`. For each job it sends the acknowledging
152-heartbeat first, then one every 30 s (a failed one is retried every 5 s), opens
153-`WatchJob` (a cancel request reaches the executor, which interrupts, terminates
154-and kills like the official runner), forwards the log with `AppendLog`
155-(batches of at most 1,000 lines and 64 KiB per step, numbered offsets, retried
156-with the same lines), sends step snapshots with `UpdateJob` and ends with an
157-idempotent `CompleteJob`. When ermite says the job ended (`JobEnded`) or revokes
158-the assignment (`unauthenticated`) the sandbox is destroyed and nothing more is
159-sent. On SIGINT or SIGTERM the running jobs are cancelled with
160-`runner_shutdown`, reported, and the sandboxes removed. The job token stays in
161-the host process; the sandbox receives the workflow and its contexts only.
162-
163-| Variable | Meaning |
164-| --- | --- |
165-| `ERMITE_RUNNER_URL` | base URL of ermite, as in a browser (the runner adds `/-/api` itself) |
166-| `ERMITE_RUNNER_NAME` | runner name (default: the host name) |
167-| `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |
168-| `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` (default) or `self-hosted`: the `runner.environment` value the jobs see. ermite's runners stand in for GitHub-hosted ones, so workflows written for github.com (`runner.environment == 'github-hosted'`) behave the same; `self-hosted` also makes "Set up job" print the runner and machine name, as GitHub does for self-hosted runners |
169-| `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` (see above) |
170-| `ERMITE_RUNNER_INSECURE_HTTP` | `1` allows a non-loopback `http://` URL (tests only); also required for `ERMITE_RUNNER_GUEST_FORGE` with an `http://` URL |
171-| `ERMITE_RUNNER_GUEST_FORGE` | `1` lets sandboxes reach the forge as the runner does (staging only; see "Guest forge access") |
172-| `ERMITE_RUNNER_LOG_LEVEL` | `debug`, `info` (default), `warn`, `error` |
173-| `ERMITE_RUNNER_LOG_FORMAT` | `console` (default, colours) or `json` |
174-
175-## Installing the service
176-
177-`contrib/ermite-runner.service` is a systemd unit. The runner runs as **root**
178-on the host (TAP devices, nftables, loop mounts, the jailer, containers); the
179-jobs never do: they run as user `runner` in a micro-VM or a container.
180 41
181 ```sh42 ```sh
182-sudo install -m 0755 ermite-runner /usr/local/bin/ermite-runner # a static build: CGO_ENABLED=043+CGO_ENABLED=0 go build -trimpath -o ermite-runner .
183-sudo install -m 0644 contrib/ermite-runner.service /etc/systemd/system/
184-sudo install -d -m 0700 /etc/ermite-runner
185-sudo install -m 0600 /dev/stdin /etc/ermite-runner/env <<'ENV'
186-ERMITE_RUNNER_URL=https://ermite.example.org
187-ERMITE_RUNNER_NAME=runner-1
188-ERMITE_RUNNER_LABELS=ubuntu-latest,ubuntu-24.04
189-ERMITE_RUNNER_ISOLATION=auto
190-ERMITE_RUNNER_CAPACITY=2
191-ERMITE_RUNNER_JOB_CPUS=4
192-ERMITE_RUNNER_JOB_MEMORY=8G
193-ERMITE_RUNNER_JOB_DISK=64G
194-ERMITE_RUNNER_LOG_FORMAT=json
195-ENV
196-# Register once, in the same environment as the service (the settings, the state
197-# directory), with the token on standard input so that it is not in any
198-# command line:
199-printf %s "$TOKEN" | sudo systemd-run --pipe --wait -p EnvironmentFile=/etc/ermite-runner/env \
200- -p StateDirectory=ermite-runner -p StateDirectoryMode=0700 \
201- /usr/local/bin/ermite-runner register --token-stdin
202-sudo systemctl enable --now ermite-runner
203-journalctl -u ermite-runner -f
204 ```44 ```
205 45
206-`systemd-run` reads the same `EnvironmentFile` as the unit, so `register` and46+## Register and run
207-`run` agree on the URL, the name, the labels, the isolation mode and the state
208-directory (`/var/lib/ermite-runner`, the default of `ERMITE_RUNNER_STATE_DIR`).
209-
210-**Logs.** The runner writes to standard error, which journald collects. Set
211-`ERMITE_RUNNER_LOG_FORMAT=json` under journald: one JSON event per line, with
212-`job_id`, `run_id` and `attempt` on every line about a job, `warn` for what is
213-retried and `error` for what is refused or leaked; the colours of the `console`
214-format are noise in a journal. `ERMITE_RUNNER_LOG_LEVEL=debug` adds detail.
215-Tokens are never logged.
216-
217-**Stopping.** `systemctl stop` sends SIGTERM: running jobs are cancelled with
218-`runner_shutdown`, reported to ermite, and their sandboxes removed before the
219-service exits. The budget, with the defaults, is **90 s**: 30 s for the executor
220-to stop its steps (SIGINT, SIGTERM after 7.5 s, SIGKILL 2.5 s later, then the
221-post-steps), 30 s to destroy the sandbox, 30 s to flush the log and report; jobs
222-do this in parallel, so it does not grow with `ERMITE_RUNNER_CAPACITY`. The unit
223-sets `TimeoutStopSec=120`, 30 s of margin (a test in `internal/host` fails if the
224-two drift apart). A job whose first heartbeat is still pending when the signal
225-arrives is not started: ermite gives it to another runner after 30 s. A crash or
226-a power cut leaves sandboxes behind; the next start reaps them. Only one runner
227-may live on a host (a host-wide lock enforces it), because reaping removes
228-everything labelled as the runner's own.
229-
230-**Sandboxing of the unit.** The runner has to be root (TAP devices, nftables,
231-loop mounts, the jailer, Docker), so the unit only takes away what it does not
232-use, and only with options that **leave it in the host's mount namespace**:
233-`NoNewPrivileges`, `ProtectClock`, `RestrictSUIDSGID`, `LockPersonality`,
234-`RestrictRealtime`, `SystemCallArchitectures=native` and `RestrictAddressFamilies`
235-(UNIX, INET, INET6, NETLINK, VSOCK).
236-
237-**Deliberately absent**, and why:
238-
239-- **Every option that gives the unit a mount namespace of its own:** `PrivateTmp`,
240- `ProtectHome`, `ProtectSystem`, `ProtectKernelModules`, `ProtectKernelLogs`,
241- `ProtectKernelTunables`, `ProtectControlGroups`, `PrivateDevices`,
242- `ReadOnlyPaths`, `ReadWritePaths`, `InaccessiblePaths`, `TemporaryFileSystem`,
243- `BindPaths`. In container mode the runner loop-mounts the scratch disk of each
244- job and the Docker daemon, which lives in the host's namespace, binds that
245- directory into the container; a mount made inside the unit's namespace is
246- invisible to it and the container would get an empty directory. (`ProtectKernelModules`
247- and `ProtectKernelLogs` look harmless but are implemented with mounts too.) The
248- micro-VM mode would tolerate them, but one unit serves both modes, and a second
249- unit for VM-only hardening is not worth maintaining. A test in `internal/host`
250- fails if the unit sets any of them.
251-- `Delegate=`: the runner manages no cgroup of its own (the jailer writes the
252- VMs' cgroups as root, Docker has its own).
253-- `RestrictNamespaces=`, `PrivateNetwork=`, `CapabilityBoundingSet=`,
254- `SystemCallFilter=`: network namespaces and devices (TAP, veth), mounts,
255- `net.ipv4.ip_forward`, capabilities handed to the jailer and Docker.
256-- `MemoryDenyWriteExecute=`: Firecracker maps guest memory writable and executable.
257-- `UMask=`: the drivers create directories that the job's user must enter, with
258- explicit modes; the state directory is made 0700 by the runner.
259-
260-The boundary of a job is its micro-VM or container, not this unit. Checked on a
261-workstation (a user manager, no root): the micro-VM job test (`TestVMRunsAJob`:
262-user and network namespaces, Firecracker, TAP, vsock) passes under the options
263-that remain. Not checked, because they need root on a real host: the jailer, loop
264-mounts, nftables in the host namespace, Docker.
265-
266-## The image
267-
268-A job runs in the image named by `ERMITE_RUNNER_IMAGE` (container: an OCI
269-reference; vm: the directory with `vmlinux` and `base.ext4`, both built from
270-`images/ubuntu-24.04/Dockerfile`). The runner binary and Node.js (see "Externals") are attached to the sandbox
271-by the host and are never part of the image. What the executor needs from the
272-image:
273-
274-- a user **`runner`** with uid and gid 1001 and the home `/home/runner`; the
275- executor starts as root (init needs it to start `dockerd`) and runs every step
276- as this user, with HOME, USER and LOGNAME set. Without such a user the steps
277- run as root (a test image only);
278-- the directories `/home/runner/work` and `/opt/hostedtoolcache` owned by
279- `runner` (`RUNNER_TOOL_CACHE`, which the executor reads from its environment
280- like the official runner; each job gets `/home/runner/work/<repo>/<repo>` and
281- `/home/runner/work/_temp`, cleaned at the start of the job);
282-- **`bash`** and **`sh`** (the default shell is `bash -e {0}` when bash exists,
283- else `sh -e {0}`; like on hosted runners `shell: bash` adds
284- `--noprofile --norc -eo pipefail`), `python3` and a `python` command for
285- `shell: python`; `pwsh` is not part of the CI-1 image (a step that asks for it
286- fails with `pwsh: command not found`);
287-- `dockerd` (and passwordless `sudo` for `runner`) for jobs that use Docker: the
288- job gets its own daemon with its data root on the scratch disk;
289-- what workflows commonly use: git, curl, jq, build-essential, ca-certificates,
290- tzdata, zip/unzip. Full parity with the hosted images (`actions/runner-images`)
291- is a later phase.
292-
293-## Job containers and service containers
294-
295-`container:` and `services:` work as on GitHub-hosted Linux runners
296-(`internal/executor/containers_*.go`, a port of `ContainerOperationProvider`,
297-`ContainerInfo`, `DockerCommandManager` and `ContainerStepHost` of
298-actions/runner, with their line numbers in the comments). The containers are
299-created in the **sandbox's own `dockerd`**, so nothing reaches the host: in `vm`
300-mode (the mode of hosts without Docker Engine) the daemon runs inside the
301-micro-VM, in `container` mode it is the nested daemon of the job.
302-
303-- **Initialize containers** (a step after "Set up job"): docker version check
304- (API ≥ 1.35), cleanup of labelled leftovers, network `github_network_<32 hex>`,
305- for each container `docker pull` (3 tries, random 1–10 s back off),
306- `docker create`/`start`, `docker port` and the health checks of the services
307- (exponential back off 2–32 s, "`<alias>` service is healthy."). Registry
308- `credentials:` go to a private `docker --config` directory under RUNNER_TEMP
309- (password through stdin, masked in every log, directory removed after the pull).
310-- **Stop containers** (post step, `always()`): service logs, `docker rm --force`,
311- `docker network rm`. It runs after a failure, a cancellation, a step timeout
312- and the job timeout.
313-- **Steps in the job container**: `run:` steps and node actions go through
314- `docker exec -i --workdir … -e NAME… <id> …` (values are passed in the
315- environment of the docker process, translated to `/__w`, `/__t`, `/__e`,
316- `/github/home`, `/github/workflow`, never on the command line). The shell is
317- looked up by the container: `sh -e {0}` unless the step says otherwise. Node
318- actions use the externals mounted read-only at `/__e`, `node<N>_alpine` when
319- `/etc/*release` says alpine. Docker actions run as siblings on the job network.
320-- Contexts: `job.container.{id,network}`, `job.services.<alias>.{id,network,ports}`
321- (host ports from `docker port`).
322-- **Untrusted input.** `options:` is split into arguments like .NET splits a
323- command line and passed to `docker create` as an argument vector, never through
324- a shell, so a workflow cannot inject a second command; it can still pass any
325- `docker create` flag (`--privileged`, `--pid host`, `--device`, …) and bind
326- volumes (`volumes: /:/host`), exactly as on a self-hosted GitHub runner. That
327- is contained by the sandbox (its own daemon, its own disk), not by the runner:
328- **host paths in `volumes:` and `options:` are paths of the sandbox**. A job
329- that mounts `/` exposes the sandbox's filesystem to its own container only.
330- The runner warns, as GitHub does, about a volume that mounts `/`.
331-- Not ported: container hooks (`ACTIONS_RUNNER_CONTAINER_HOOKS`), the
332- "already running in a container" refusal (the sandbox's daemon is local to the
333- executor).
334-
335-## Job limits
336-
337-Each job gets its own resources, set for the runner as a whole:
338-
339-| Variable | Default | Effect |
340-|---|---|---|
341-| `ERMITE_RUNNER_CAPACITY` | `1` | jobs in parallel: one sandbox, one uid, one `/30` network and one disk per slot. The runner polls ermite only when a slot is free. |
342-| `ERMITE_RUNNER_JOB_CPUS` | `2` | vCPUs of a micro-VM; CPU quota of a container |
343-| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | guest memory (`M`, `G`, `T`; at least `256M`); a jailed VMM gets this plus 256 MiB of cgroup headroom |
344-| `ERMITE_RUNNER_JOB_DISK` | `64G` | size of the sparse scratch disk (at least `1G`): the overlay upper layer of a VM and the Docker data root of a container. Deleted with the job |
345-| `ERMITE_RUNNER_JOB_PIDS` | `32768` | process limit of a container |
346-| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits (MiB/s, IOPS, Mbit/s); `0` disables |
347-
348-The job timeout is not a runner setting: it is the workflow's `timeout-minutes`
349-(360 by default), which ermite sends with the job and the executor enforces from
350-the moment it starts.
351-
352-## Configuration (sandbox part)
353-
354-| Variable | Default | |
355-|---|---|---|
356-| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
357-| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | absolute path |
358-| `ERMITE_RUNNER_CAPACITY` | `1` | parallel jobs (slots) |
359-| `ERMITE_RUNNER_IMAGE` | | VM: image directory (`vmlinux`, `base.ext4`; default `<state dir>/image`); container: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) |
360-| `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T`. A warning is logged at start when the state directory has less free space than capacity x disk (disks are sparse: it is only a warning) |
361-| `ERMITE_RUNNER_JOB_PIDS` | `32768` | container process limit |
362-| `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses container mode without sysbox |
363-| `ERMITE_RUNNER_GUEST_FORGE` | `0` | `1` lets sandboxes reach the host and port of `ERMITE_RUNNER_URL` on the host (staging only, see "Guest forge access") |
364-| `ERMITE_RUNNER_CACHE` | `1` | `0` turns the cache service off (see "Cache") |
365-| `ERMITE_RUNNER_CACHE_MAX` | `10G` | size cap of the cache, LRU eviction |
366-| `ERMITE_RUNNER_ACTIONS_CACHE_MAX` | `2G` | size cap of the action archive cache; `0` turns it off |
367-| `ERMITE_RUNNER_ARTIFACTS` | `1` | `0` turns the results service (artifacts v4) off (see "Artifacts") |
368-| `ERMITE_RUNNER_NODE` | `1` | `0` turns the Node.js externals off (see "Externals") |
369-| `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | host-wide single-instance lock (tests only) |
370-| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits in MiB/s, IOPS and Mbit/s; 0 disables |
371-| `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |
372-| `ERMITE_RUNNER_NETWORK` | `10.200.0.0/16` | pool of /30 subnets, one per slot |
373-| `ERMITE_RUNNER_FIRECRACKER`, `_JAILER`, `_KERNEL` | from `PATH` | binaries and kernel overrides |
374-| `ERMITE_RUNNER_JAILER_MODE` | `on` | `off` only for development and tests |
375-
376-## Images and kernel
377 47
378-```sh48+1. On the ermite server, create a registration token:
379-scripts/build-kernel.sh tmp/image # downloads Linux 6.18 LTS from kernel.org, builds vmlinux (needs bc, flex, bison, pahole, libelf, openssl headers)
380-scripts/build-image.sh tmp/image # builds the OCI image with docker or podman, exports tmp/image/base.ext4 (no root needed)
381-```
382 49
383-`scripts/build-image.sh` tags the OCI image `ermite-runner-ubuntu-24.04:latest`, the default of `ERMITE_RUNNER_IMAGE` in container mode, and needs e2fsprogs >= 1.47.1 (the runner itself needs >= 1.43 and checks it at start).50+ ```sh
51+ ermite token create my-runner --preset runner
52+ ```
384 53
385-The kernel configuration is `images/kernel/config-6.18` (+ per-architecture fragments), checked after `olddefconfig`: the build fails if a listed option did not survive. arm64 builds (`ARCH_NAME=arm64`) but is not tested in CI-1.54+2. Install the binary, the unit and the settings:
386 55
387-`CONFIG_BINFMT_MISC` is built in (the guest has no modules) and the guest init mounts it at `/proc/sys/fs/binfmt_misc`, as systemd does on GitHub's image, so `docker/setup-qemu-action` (`tonistiigi/binfmt --install`) can register QEMU handlers and run foreign-architecture containers. The handlers belong to the job's VM and disappear with it. With a kernel built before this option, the init logs the failed mount and jobs run without it.56+ ```sh
57+ sudo install -m 0755 ermite-runner /usr/local/bin/ermite-runner
58+ sudo install -m 0644 contrib/ermite-runner.service /etc/systemd/system/
59+ sudo install -d -m 0700 /etc/ermite-runner
60+ sudo install -m 0600 /dev/stdin /etc/ermite-runner/env <<'ENV'
61+ ERMITE_RUNNER_URL=https://ermite.example.com
62+ ERMITE_RUNNER_NAME=runner-1
63+ ERMITE_RUNNER_LABELS=ubuntu-latest,ubuntu-24.04
64+ ERMITE_RUNNER_LOG_FORMAT=json
65+ ENV
66+ ```
388 67
389-## Tests68+3. Register once. The token goes through standard input, so it is in no command line:
390 69
391-```sh70+ ```sh
392-GOWORK=off go vet ./... && GOWORK=off go test -race ./...71+ printf %s "$TOKEN" | sudo systemd-run --pipe --wait \
393-```72+ -p EnvironmentFile=/etc/ermite-runner/env \
73+ -p StateDirectory=ermite-runner -p StateDirectoryMode=0700 \
74+ /usr/local/bin/ermite-runner register --token-stdin
75+ ```
394 76
395-Unit tests need nothing special (the drivers are tested with a fake Firecracker and a fake Docker engine). Integration tests are opt-in and use the real stub executor: output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.77+4. Start the service:
396 78
397-**VM** (`ERMITE_RUNNER_IT_VM=1`): needs `/dev/kvm`, Firecracker, the kernel and base image from the scripts above.79+ ```sh
80+ sudo systemctl enable --now ermite-runner
81+ journalctl -u ermite-runner -f
82+ ```
398 83
399-```sh84+`ermite-runner run` polls the server for jobs and runs each one in a sandbox, up to `ERMITE_RUNNER_CAPACITY` jobs at a time. On SIGTERM it cancels the running jobs, reports them and removes their sandboxes.
400-# Unprivileged: inside a user + network namespace (TAP, nftables and DNS on the TAP address work there).
401-# Jailer off: it needs real root. Everything else is the production path.
402-FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
403-
404-# Privileged, with the jailer (the production configuration):
405-sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
406- ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
407- "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
408-```
409-
410-**Container** (`ERMITE_RUNNER_IT_CONTAINER=1`): needs a rootful Docker daemon, root (loop mounts, `CAP_NET_ADMIN`) and the OCI image (`scripts/build-image.sh`).
411-
412-```sh
413-sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:latest \
414- DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
415-```
416 85
417-**Jobs end to end** (`ERMITE_RUNNER_IT_E2E=1`, `internal/e2e`): the real executor in the real image, started through the container driver. They also run on a workstation with a *rootless* Docker daemon (the test-only option `UnsafeAllowRootlessForTests`; the nested `dockerd` of the image cannot start there, and the jobs do not need it). `mount` and `umount` are replaced by stand-ins, since loop mounts need root.86+To check an installation without a server, run `ermite-runner sandbox-check`. It runs one stub script in the selected mode.
418 87
419-```sh88+More on the service, logs and shutdown: [docs/service.md](docs/service.md).
420-IMAGE_TAG=ermite-runner-ubuntu-24.04:dev scripts/build-image.sh tmp/image # or: docker build -t ermite-runner-ubuntu-24.04:dev images/ubuntu-24.04
421-ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-24.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
422-```
423-
424-- `TestContainerRunsAJobAsTheRunnerUser` and `TestContainerCancelStopsTheJobAndOrphans` drive a job over the stream: steps run as `runner` (1001), file ownership, `GITHUB_PATH` and `GITHUB_ENV`, `python`, cancellation by SIGINT, the `always()` step, and the orphan cleanup of "Complete job".
425-- `TestChainCorpus` runs the whole chain: a fake `RunnerService` (`internal/host/hosttest`, which checks the offsets, limits and tokens of `runner.proto`) hands out 22 jobs of the ermite-conformance corpus, the real host loop starts them in containers (two at a time), and what the fake stored (steps, conclusions, outputs, log lines) is compared with the recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
426-- `TestVMRunsAJob` (`internal/sandbox/integration`, run by `scripts/it-vm-userns.sh`) does the first of these in a micro-VM.
427-- `TestVMNodeExternals` (same place) downloads the pinned `node24` (or takes it from `ERMITE_RUNNER_IT_NODE_DIR`, a directory with `node-v24.21.0-linux-x64.tar.gz`, for an offline host; the checksum and the extraction are the real ones), boots a micro-VM and runs `/home/runner/externals/node24/bin/node -e 'console.log(process.version)'` in it, checking that the mount is read-only.
428 89
429-Checked as root on a real KVM host (Arch Linux, kernel 6.18 LTS, Firecracker and jailer 1.17, rootful podman as the Docker API): the jailer (uid drop, chroot, `memory.max` and `pids.max` cgroups, PID namespace), the VM and container integration scenarios, the firewall policy and NAT, the nested `dockerd` on the loop-mounted scratch disk (full disk gives `ENOSPC`), and `internal/e2e` including `TestChainCorpus`. Not checked: sysbox, and a real Docker Engine (podman's compatibility API is not a drop-in, see below).90+## Commands
430 91
431-**Jailer notes** (found on that host): Firecracker 1.17's jailer needs `--chroot-base-dir` to exist (the driver creates `<state dir>/jail`), starts Firecracker as `/firecracker --id <id> --start-time-us ...` (the `/proc` scan that finds a VMM by id knows that form), and leaves `<state dir>/jail/<exec name>` and the cgroup parent `/sys/fs/cgroup/firecracker` behind (only the per-VM directories are the runner's). The jailed API socket path (`<state dir>/jail/firecracker/<id>/root/run/firecracker.socket`) must fit in 107 bytes; the driver checks it at start.92+| Command | Purpose |
432-93+| --- | --- |
433-**Host firewalls.** The runner's table has its own `input` hook at priority `filter`; another table with a later hook (firewalld at `filter + 10`) still sees a guest's packet to the gateway after the runner accepted it. A firewalld zone that rejects `udp/tcp 53` (the `public` zone with a DNS reject rule, for instance) therefore breaks the guests' DNS (the symptom is `dns silent` in `TestVMNetworkPolicy`) while NAT keeps working. On such a host, allow DNS from `ertap*` and `erbr*` in that firewall, or run the policy test in a private network namespace (as `scripts/it-vm-userns.sh` does).94+| `ermite-runner register [--token-stdin] [--token T] [--replace]` | Register with the server and store the runner token |
95+| `ermite-runner run` | Fetch and run jobs |
96+| `ermite-runner sandbox-check [script]` | Run one stub script in a sandbox |
97+| `ermite-runner version` | Print the version |
98+
99+## Key settings
100+
101+| Variable | Default | Meaning |
102+| --- | --- | --- |
103+| `ERMITE_RUNNER_URL` | | Base URL of the ermite server. Must be `https://` |
104+| `ERMITE_RUNNER_NAME` | host name | Runner name |
105+| `ERMITE_RUNNER_LABELS` | | Extra labels, comma separated |
106+| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
107+| `ERMITE_RUNNER_CAPACITY` | `1` | Jobs in parallel |
108+| `ERMITE_RUNNER_JOB_CPUS` | `2` | CPUs per job |
109+| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | Memory per job |
110+| `ERMITE_RUNNER_JOB_DISK` | `64G` | Scratch disk per job |
111+| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | State directory |
434 112
435-**Podman as the Docker API.** `DOCKER_HOST` pointing at rootful podman's compatibility socket gets as far as creating the network, then fails: the compatibility `POST /networks/create` cannot disable netavark's DNS (`dns_enabled` is always true) and `HostConfig.Dns` equal to the network gateway is refused when the container starts (`netavark: error while applying dns entries: DNS server 10.200.0.1 is the gateway of network ...; using it would cause aardvark-dns to forward queries to itself`); the option `com.docker.network.bridge.enable_ip_masquerade` is ignored with a warning, and netavark (`firewall_driver = "nftables"` by default) adds its own rules unless it is configured with `firewall_driver = "none"`. The integration tests passed with a small test-only proxy that rewrites that one request to the libpod endpoint with `dns_enabled=false`, and a private `CONTAINERS_CONF`. Production container mode supports Docker Engine (or sysbox) only.113+All settings: [docs/configuration.md](docs/configuration.md).
436 114
437-`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.115+## Security notes
438 116
439-The firewall policy has a packet-level test (`ERMITE_RUNNER_IT_NET=1`, run by `scripts/it-vm-userns.sh`): two guests and an outside host on veth pairs in separate network namespaces, with a forged source and a forged "reply" between guests.117+- Run only one runner per host.
118+- The registration token can read the code and secrets of every repository. Pass it through standard input.
119+- Without sysbox, `container` mode uses a privileged container, which is equivalent to host root. Set `ERMITE_RUNNER_DENY_PRIVILEGED=1` to refuse it.
120+- The state directory and all its parents must be owned by root and not writable by others.
440 121
441-Protobuf: `go generate ./internal/execstream` (buf and protoc-gen-go are pinned as `tool` directives in go.mod); CI lints the proto and fails when the committed generated code differs.122+## Documentation
442 123
443-`ermite-runner sandbox-check [script]` runs one stub job through the selected mode and prints its output: the smoke test of an installation.124+- [Configuration](docs/configuration.md): every environment variable
125+- [Sandboxes](docs/sandboxes.md): `vm` and `container` modes, network policy, cleanup
126+- [Service](docs/service.md): systemd unit, logs, shutdown
127+- [Images](docs/images.md): job image, guest kernel
128+- [Workflows](docs/workflows.md): job and service containers, cancellation, annotations
129+- [Cache and artifacts](docs/cache-and-artifacts.md): Actions cache and artifacts v4
130+- [Node.js externals](docs/externals.md): Node.js for JavaScript actions
131+- [Development](docs/development.md): build, tests
444 132
445 ## License133 ## License
446 134
447-MIT. See [LICENSE](LICENSE).135+MIT. See [LICENSE](LICENSE). Third-party attributions: [THIRD_PARTY.md](THIRD_PARTY.md).
A docs/cache-and-artifacts.md +81 -0
@@ -0,0 +1,81 @@
1+# Cache and artifacts
2+
3+The runner serves the GitHub Actions cache and artifacts v4 to jobs. Both use one channel per sandbox.
4+
5+## Channel
6+
7+- The services are reachable only from inside the sandbox.
8+- They listen on the sandbox loopback (`127.0.0.1:9089`) and on the address of its own Docker bridge (`docker0`, typically `172.17.0.1:9089`).
9+- Nothing listens on a host network or port. The host firewall needs no rule.
10+- A relay in the sandbox pipes each connection to the host over a channel that belongs to that sandbox:
11+ - `vm`: vsock port 9089.
12+ - `container`: `cache.sock` in the per-job directory.
13+- A sandbox cannot reach another sandbox's channel.
14+- Limits: 32 connections per sandbox, 128 in all. A transfer with no progress for 60 s is cut.
15+
16+### Variables
17+
18+- Node actions get `ACTIONS_CACHE_URL`, `ACTIONS_RUNTIME_URL`, `ACTIONS_RUNTIME_TOKEN`, `ACTIONS_CACHE_MODE` and `ACTIONS_RESULTS_URL`. `run:` steps get none, as on the official runner.
19+- In job containers and docker actions the same variables point to the Docker bridge address.
20+- The token is masked.
21+- If the relay cannot start, the job runs without a cache.
22+
23+## Cache (API v1)
24+
25+Works with `actions/cache`, `setup-go` and `setup-node` with `cache:`, and anything built on `@actions/cache`. The toolkit uses cache API v1 for any server that is not `github.com`, `*.ghe.com` or `*.localhost`. `ACTIONS_CACHE_SERVICE_V2` is never set.
26+
27+| Setting | Value |
28+| --- | --- |
29+| Storage | `<state dir>/cache`, mode 0700/0600, written atomically |
30+| Size cap | `ERMITE_RUNNER_CACHE_MAX`, default `10G`. Least recently used entries are evicted |
31+| Off | `ERMITE_RUNNER_CACHE=0` |
32+| Restart | Entries are kept. Unfinished uploads are dropped |
33+
34+- **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref and reads its read refs in priority order. Another repository's entries are never visible.
35+- **Server scope.** The server sends the repository, write ref and read refs (feature `ci.cache`). The runner checks them: the repository must be the job's own, refs are full `refs/...` names, at most 8 read refs. An empty write ref makes the job read-only. A scope that fails the checks disables the cache for that job.
36+- **Old servers.** Without a scope from the server, the scope is `github.repository` and the job ref, for reading and writing.
37+- **Duplicates.** A second reservation of the same entry while one exists is a `409`.
38+- **Reservations.** Dropped after an hour without activity, or at the expiry of the token that made them.
39+- **Limits.** Each entry is charged at least 4 KiB. At most 256 uploads in progress (32 per job). An entry being downloaded is never evicted. A reservation that cannot get room answers `503`.
40+- **Bandwidth.** A VM's cache traffic does not cross `eth0`. The runner limits it per sandbox at `ERMITE_RUNNER_JOB_NET_BANDWIDTH`. Containers have no limit.
41+
42+### Tokens
43+
44+- `ACTIONS_RUNTIME_TOKEN` is a JWT (HS256) bound to the job, repository, refs and an expiry (job timeout plus 15 min).
45+- The key is random, made at start and kept in memory only.
46+- Any invalid token gets a bare `403`.
47+- The token is not an ermite credential. The server refuses it.
48+- Archive URLs carry their own HMAC and are served only to the job they were issued to.
49+
50+### Action archives
51+
52+- The host downloads an action archive once into `<state dir>/actions-cache/`, keyed by forge, repository and commit sha.
53+- The sandbox gets a signed URL on the cache channel and extracts the archive.
54+- The SHA-256 is checked on every read. A corrupt entry is dropped and fetched again.
55+- Size cap: `ERMITE_RUNNER_ACTIONS_CACHE_MAX`, default `2G`. `0` turns it off.
56+- The sandbox never writes to this cache. Anything that cannot be cached is downloaded by the sandbox.
57+
58+## Artifacts (v4)
59+
60+`actions/upload-artifact@v4` and later, `actions/download-artifact@v4` and later, and anything bundling `@actions/artifact` v2 work when the server has the feature `ci.artifacts`.
61+
62+- **ermite stores the artifacts.** The runner keeps nothing on disk.
63+- The runner serves GitHub's results service (Twirp JSON plus Azure block blob requests on signed URLs) and translates it to the server with the job token.
64+- The job token never enters the sandbox.
65+- A job reaches only the artifacts of its own run.
66+- Quota and size limits answer `412` (`413` on a block) and are not retried. Server unavailable answers `503`, which the toolkit retries.
67+- Off: `ERMITE_RUNNER_ARTIFACTS=0` or a server without `ci.artifacts`. Artifacts need the cache service, so `ERMITE_RUNNER_CACHE=0` turns them off too.
68+
69+### `GITHUB_SERVER_URL` in artifact steps
70+
71+- `@actions/artifact` refuses to run on GitHub Enterprise Server, which it detects only from `GITHUB_SERVER_URL`.
72+- In node steps of `actions/upload-artifact` and `actions/download-artifact` (any subpath), and of actions that contain the toolkit's GHES message, `GITHUB_SERVER_URL` is `https://ermite.ghe.localhost`.
73+- Nothing resolves or connects to that host.
74+- The `github` context, `${{ github.server_url }}` and all other steps keep the real URL.
75+
76+### Not covered
77+
78+- `upload-artifact` below v4.3.1 and `download-artifact` below v4.1.2.
79+- Artifacts v3.
80+- The `artifact-url` output. It points at the fake host. The run page in ermite lists the artifacts.
81+- Downloads from another run (`run-id` with `github-token`).
@@ -0,0 +1,81 @@
1+# Cache and artifacts
2+
3+The runner serves the GitHub Actions cache and artifacts v4 to jobs. Both use one channel per sandbox.
4+
5+## Channel
6+
7+- The services are reachable only from inside the sandbox.
8+- They listen on the sandbox loopback (`127.0.0.1:9089`) and on the address of its own Docker bridge (`docker0`, typically `172.17.0.1:9089`).
9+- Nothing listens on a host network or port. The host firewall needs no rule.
10+- A relay in the sandbox pipes each connection to the host over a channel that belongs to that sandbox:
11+ - `vm`: vsock port 9089.
12+ - `container`: `cache.sock` in the per-job directory.
13+- A sandbox cannot reach another sandbox's channel.
14+- Limits: 32 connections per sandbox, 128 in all. A transfer with no progress for 60 s is cut.
15+
16+### Variables
17+
18+- Node actions get `ACTIONS_CACHE_URL`, `ACTIONS_RUNTIME_URL`, `ACTIONS_RUNTIME_TOKEN`, `ACTIONS_CACHE_MODE` and `ACTIONS_RESULTS_URL`. `run:` steps get none, as on the official runner.
19+- In job containers and docker actions the same variables point to the Docker bridge address.
20+- The token is masked.
21+- If the relay cannot start, the job runs without a cache.
22+
23+## Cache (API v1)
24+
25+Works with `actions/cache`, `setup-go` and `setup-node` with `cache:`, and anything built on `@actions/cache`. The toolkit uses cache API v1 for any server that is not `github.com`, `*.ghe.com` or `*.localhost`. `ACTIONS_CACHE_SERVICE_V2` is never set.
26+
27+| Setting | Value |
28+| --- | --- |
29+| Storage | `<state dir>/cache`, mode 0700/0600, written atomically |
30+| Size cap | `ERMITE_RUNNER_CACHE_MAX`, default `10G`. Least recently used entries are evicted |
31+| Off | `ERMITE_RUNNER_CACHE=0` |
32+| Restart | Entries are kept. Unfinished uploads are dropped |
33+
34+- **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref and reads its read refs in priority order. Another repository's entries are never visible.
35+- **Server scope.** The server sends the repository, write ref and read refs (feature `ci.cache`). The runner checks them: the repository must be the job's own, refs are full `refs/...` names, at most 8 read refs. An empty write ref makes the job read-only. A scope that fails the checks disables the cache for that job.
36+- **Old servers.** Without a scope from the server, the scope is `github.repository` and the job ref, for reading and writing.
37+- **Duplicates.** A second reservation of the same entry while one exists is a `409`.
38+- **Reservations.** Dropped after an hour without activity, or at the expiry of the token that made them.
39+- **Limits.** Each entry is charged at least 4 KiB. At most 256 uploads in progress (32 per job). An entry being downloaded is never evicted. A reservation that cannot get room answers `503`.
40+- **Bandwidth.** A VM's cache traffic does not cross `eth0`. The runner limits it per sandbox at `ERMITE_RUNNER_JOB_NET_BANDWIDTH`. Containers have no limit.
41+
42+### Tokens
43+
44+- `ACTIONS_RUNTIME_TOKEN` is a JWT (HS256) bound to the job, repository, refs and an expiry (job timeout plus 15 min).
45+- The key is random, made at start and kept in memory only.
46+- Any invalid token gets a bare `403`.
47+- The token is not an ermite credential. The server refuses it.
48+- Archive URLs carry their own HMAC and are served only to the job they were issued to.
49+
50+### Action archives
51+
52+- The host downloads an action archive once into `<state dir>/actions-cache/`, keyed by forge, repository and commit sha.
53+- The sandbox gets a signed URL on the cache channel and extracts the archive.
54+- The SHA-256 is checked on every read. A corrupt entry is dropped and fetched again.
55+- Size cap: `ERMITE_RUNNER_ACTIONS_CACHE_MAX`, default `2G`. `0` turns it off.
56+- The sandbox never writes to this cache. Anything that cannot be cached is downloaded by the sandbox.
57+
58+## Artifacts (v4)
59+
60+`actions/upload-artifact@v4` and later, `actions/download-artifact@v4` and later, and anything bundling `@actions/artifact` v2 work when the server has the feature `ci.artifacts`.
61+
62+- **ermite stores the artifacts.** The runner keeps nothing on disk.
63+- The runner serves GitHub's results service (Twirp JSON plus Azure block blob requests on signed URLs) and translates it to the server with the job token.
64+- The job token never enters the sandbox.
65+- A job reaches only the artifacts of its own run.
66+- Quota and size limits answer `412` (`413` on a block) and are not retried. Server unavailable answers `503`, which the toolkit retries.
67+- Off: `ERMITE_RUNNER_ARTIFACTS=0` or a server without `ci.artifacts`. Artifacts need the cache service, so `ERMITE_RUNNER_CACHE=0` turns them off too.
68+
69+### `GITHUB_SERVER_URL` in artifact steps
70+
71+- `@actions/artifact` refuses to run on GitHub Enterprise Server, which it detects only from `GITHUB_SERVER_URL`.
72+- In node steps of `actions/upload-artifact` and `actions/download-artifact` (any subpath), and of actions that contain the toolkit's GHES message, `GITHUB_SERVER_URL` is `https://ermite.ghe.localhost`.
73+- Nothing resolves or connects to that host.
74+- The `github` context, `${{ github.server_url }}` and all other steps keep the real URL.
75+
76+### Not covered
77+
78+- `upload-artifact` below v4.3.1 and `download-artifact` below v4.1.2.
79+- Artifacts v3.
80+- The `artifact-url` output. It points at the fake host. The run page in ermite lists the artifacts.
81+- Downloads from another run (`run-id` with `github-token`).
A docs/configuration.md +98 -0
@@ -0,0 +1,98 @@
1+# Configuration
2+
3+The runner reads environment variables only. Under systemd they come from `/etc/ermite-runner/env`.
4+
5+Sizes accept `M`, `G` and `T`.
6+
7+## Server and identity
8+
9+| Variable | Default | Meaning |
10+| --- | --- | --- |
11+| `ERMITE_RUNNER_URL` | | Base URL of the server, as in a browser. The runner adds `/-/api` itself |
12+| `ERMITE_RUNNER_NAME` | host name | Runner name: 1 to 64 letters, digits, `.`, `_` and `-` |
13+| `ERMITE_RUNNER_LABELS` | | Extra labels, comma separated, same characters as the name |
14+| `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` | `github-hosted` or `self-hosted`. Sets the `runner.environment` value that jobs see. `self-hosted` also makes "Set up job" print the runner and machine name |
15+| `ERMITE_RUNNER_REGISTER_TOKEN` | | Token for `register` |
16+| `ERMITE_RUNNER_INSECURE_HTTP` | `0` | `1` allows a non-loopback `http://` URL. Every token then travels in clear. For tests and trusted networks |
17+
18+### Server URL
19+
20+- The URL must be `https://`.
21+- Plain `http://` is accepted for `localhost`, `127.0.0.0/8` and `::1`.
22+- The client never follows a redirect, so a token cannot be carried to another host.
23+
24+### Registration token
25+
26+`register` reads the token from the first source that is set:
27+
28+1. Standard input, with `--token-stdin`.
29+2. `ERMITE_RUNNER_REGISTER_TOKEN`.
30+3. `--token`.
31+
32+Prefer the first two. The arguments of a process are visible to every user of the host.
33+
34+The token has the scope `runner:register`. It can read the code and the secrets of every repository.
35+
36+`register` stores the runner token in `$ERMITE_RUNNER_STATE_DIR/runner.json`. The file is mode 0600 in a directory made 0700. A file readable by others is refused. `--replace` removes a runner of the same name first.
37+
38+## Sandboxes
39+
40+| Variable | Default | Meaning |
41+| --- | --- | --- |
42+| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
43+| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | Absolute path. See the ownership rules in [sandboxes](sandboxes.md#state-directory) |
44+| `ERMITE_RUNNER_CAPACITY` | `1` | Jobs in parallel. One sandbox, one uid, one `/30` network and one disk per slot. The runner polls the server only when a slot is free |
45+| `ERMITE_RUNNER_IMAGE` | | `vm`: image directory with the kernel and `base.ext4` (default `<state dir>/image`). `container`: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) |
46+| `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses `container` mode without sysbox |
47+| `ERMITE_RUNNER_NETWORK` | `10.200.0.0/16` | Pool of `/30` subnets, one per slot |
48+| `ERMITE_RUNNER_UID_BASE` | `900000` | First uid of the jailer pool. A VM runs as this value plus its slot |
49+| `ERMITE_RUNNER_FIRECRACKER` | from `PATH` | Firecracker binary |
50+| `ERMITE_RUNNER_JAILER` | from `PATH` | Jailer binary |
51+| `ERMITE_RUNNER_KERNEL` | `<image dir>/vmlinux` (`Image` on arm64) | Guest kernel |
52+| `ERMITE_RUNNER_JAILER_MODE` | `on` | `off` only for development and tests. It removes the chroot, uid and namespace confinement of the VMM |
53+| `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | Single-instance lock. Change it for tests only |
54+
55+`DOCKER_HOST` is honoured in `container` mode.
56+
57+## Job limits
58+
59+| Variable | Default | Meaning |
60+| --- | --- | --- |
61+| `ERMITE_RUNNER_JOB_CPUS` | `2` | vCPUs of a VM. CPU quota of a container |
62+| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | Guest memory, at least `256M`. A jailed VMM gets this plus 256 MiB |
63+| `ERMITE_RUNNER_JOB_DISK` | `64G` | Sparse scratch disk, at least `1G`. Deleted with the job |
64+| `ERMITE_RUNNER_JOB_PIDS` | `32768` | Process limit of a container |
65+| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH` | `400` | VM disk limit in MiB/s. `0` disables |
66+| `ERMITE_RUNNER_JOB_DISK_IOPS` | `20000` | VM disk limit in IOPS. `0` disables |
67+| `ERMITE_RUNNER_JOB_NET_BANDWIDTH` | `1000` | VM network limit in Mbit/s, each way. `0` disables |
68+
69+- The runner logs a warning at start when the state directory has less free space than capacity times disk size. Disks are sparse, so it is only a warning.
70+- The job timeout is the workflow's `timeout-minutes` (360 by default).
71+
72+## Services
73+
74+| Variable | Default | Meaning |
75+| --- | --- | --- |
76+| `ERMITE_RUNNER_CACHE` | `1` | `0` turns the Actions cache off. See [cache and artifacts](cache-and-artifacts.md) |
77+| `ERMITE_RUNNER_CACHE_MAX` | `10G` | Size cap of the cache. Least recently used entries are evicted |
78+| `ERMITE_RUNNER_ACTIONS_CACHE_MAX` | `2G` | Size cap of the action archive cache. `0` turns it off |
79+| `ERMITE_RUNNER_ARTIFACTS` | `1` | `0` turns artifacts v4 off. Artifacts need the cache service |
80+| `ERMITE_RUNNER_NODE` | `1` | `0` turns the Node.js externals off. See [externals](externals.md) |
81+| `ERMITE_RUNNER_GUEST_FORGE` | `0` | `1` lets sandboxes reach the server host and port. Staging only. See [sandboxes](sandboxes.md#guest-forge-access) |
82+
83+## Logging
84+
85+| Variable | Default | Meaning |
86+| --- | --- | --- |
87+| `ERMITE_RUNNER_LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error` |
88+| `ERMITE_RUNNER_LOG_FORMAT` | `console` | `console` (colours) or `json` |
89+
90+Tokens are never logged.
91+
92+## Test-only
93+
94+| Variable | Meaning |
95+| --- | --- |
96+| `ERMITE_RUNNER_ALLOW_ROOT_STEPS` | `1` runs steps as root in an image without the `runner` user. A runner never sets it |
97+
98+Integration test variables are listed in [development](development.md).
@@ -0,0 +1,98 @@
1+# Configuration
2+
3+The runner reads environment variables only. Under systemd they come from `/etc/ermite-runner/env`.
4+
5+Sizes accept `M`, `G` and `T`.
6+
7+## Server and identity
8+
9+| Variable | Default | Meaning |
10+| --- | --- | --- |
11+| `ERMITE_RUNNER_URL` | | Base URL of the server, as in a browser. The runner adds `/-/api` itself |
12+| `ERMITE_RUNNER_NAME` | host name | Runner name: 1 to 64 letters, digits, `.`, `_` and `-` |
13+| `ERMITE_RUNNER_LABELS` | | Extra labels, comma separated, same characters as the name |
14+| `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` | `github-hosted` or `self-hosted`. Sets the `runner.environment` value that jobs see. `self-hosted` also makes "Set up job" print the runner and machine name |
15+| `ERMITE_RUNNER_REGISTER_TOKEN` | | Token for `register` |
16+| `ERMITE_RUNNER_INSECURE_HTTP` | `0` | `1` allows a non-loopback `http://` URL. Every token then travels in clear. For tests and trusted networks |
17+
18+### Server URL
19+
20+- The URL must be `https://`.
21+- Plain `http://` is accepted for `localhost`, `127.0.0.0/8` and `::1`.
22+- The client never follows a redirect, so a token cannot be carried to another host.
23+
24+### Registration token
25+
26+`register` reads the token from the first source that is set:
27+
28+1. Standard input, with `--token-stdin`.
29+2. `ERMITE_RUNNER_REGISTER_TOKEN`.
30+3. `--token`.
31+
32+Prefer the first two. The arguments of a process are visible to every user of the host.
33+
34+The token has the scope `runner:register`. It can read the code and the secrets of every repository.
35+
36+`register` stores the runner token in `$ERMITE_RUNNER_STATE_DIR/runner.json`. The file is mode 0600 in a directory made 0700. A file readable by others is refused. `--replace` removes a runner of the same name first.
37+
38+## Sandboxes
39+
40+| Variable | Default | Meaning |
41+| --- | --- | --- |
42+| `ERMITE_RUNNER_ISOLATION` | `auto` | `auto`, `vm` or `container` |
43+| `ERMITE_RUNNER_STATE_DIR` | `/var/lib/ermite-runner` | Absolute path. See the ownership rules in [sandboxes](sandboxes.md#state-directory) |
44+| `ERMITE_RUNNER_CAPACITY` | `1` | Jobs in parallel. One sandbox, one uid, one `/30` network and one disk per slot. The runner polls the server only when a slot is free |
45+| `ERMITE_RUNNER_IMAGE` | | `vm`: image directory with the kernel and `base.ext4` (default `<state dir>/image`). `container`: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) |
46+| `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses `container` mode without sysbox |
47+| `ERMITE_RUNNER_NETWORK` | `10.200.0.0/16` | Pool of `/30` subnets, one per slot |
48+| `ERMITE_RUNNER_UID_BASE` | `900000` | First uid of the jailer pool. A VM runs as this value plus its slot |
49+| `ERMITE_RUNNER_FIRECRACKER` | from `PATH` | Firecracker binary |
50+| `ERMITE_RUNNER_JAILER` | from `PATH` | Jailer binary |
51+| `ERMITE_RUNNER_KERNEL` | `<image dir>/vmlinux` (`Image` on arm64) | Guest kernel |
52+| `ERMITE_RUNNER_JAILER_MODE` | `on` | `off` only for development and tests. It removes the chroot, uid and namespace confinement of the VMM |
53+| `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | Single-instance lock. Change it for tests only |
54+
55+`DOCKER_HOST` is honoured in `container` mode.
56+
57+## Job limits
58+
59+| Variable | Default | Meaning |
60+| --- | --- | --- |
61+| `ERMITE_RUNNER_JOB_CPUS` | `2` | vCPUs of a VM. CPU quota of a container |
62+| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | Guest memory, at least `256M`. A jailed VMM gets this plus 256 MiB |
63+| `ERMITE_RUNNER_JOB_DISK` | `64G` | Sparse scratch disk, at least `1G`. Deleted with the job |
64+| `ERMITE_RUNNER_JOB_PIDS` | `32768` | Process limit of a container |
65+| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH` | `400` | VM disk limit in MiB/s. `0` disables |
66+| `ERMITE_RUNNER_JOB_DISK_IOPS` | `20000` | VM disk limit in IOPS. `0` disables |
67+| `ERMITE_RUNNER_JOB_NET_BANDWIDTH` | `1000` | VM network limit in Mbit/s, each way. `0` disables |
68+
69+- The runner logs a warning at start when the state directory has less free space than capacity times disk size. Disks are sparse, so it is only a warning.
70+- The job timeout is the workflow's `timeout-minutes` (360 by default).
71+
72+## Services
73+
74+| Variable | Default | Meaning |
75+| --- | --- | --- |
76+| `ERMITE_RUNNER_CACHE` | `1` | `0` turns the Actions cache off. See [cache and artifacts](cache-and-artifacts.md) |
77+| `ERMITE_RUNNER_CACHE_MAX` | `10G` | Size cap of the cache. Least recently used entries are evicted |
78+| `ERMITE_RUNNER_ACTIONS_CACHE_MAX` | `2G` | Size cap of the action archive cache. `0` turns it off |
79+| `ERMITE_RUNNER_ARTIFACTS` | `1` | `0` turns artifacts v4 off. Artifacts need the cache service |
80+| `ERMITE_RUNNER_NODE` | `1` | `0` turns the Node.js externals off. See [externals](externals.md) |
81+| `ERMITE_RUNNER_GUEST_FORGE` | `0` | `1` lets sandboxes reach the server host and port. Staging only. See [sandboxes](sandboxes.md#guest-forge-access) |
82+
83+## Logging
84+
85+| Variable | Default | Meaning |
86+| --- | --- | --- |
87+| `ERMITE_RUNNER_LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error` |
88+| `ERMITE_RUNNER_LOG_FORMAT` | `console` | `console` (colours) or `json` |
89+
90+Tokens are never logged.
91+
92+## Test-only
93+
94+| Variable | Meaning |
95+| --- | --- |
96+| `ERMITE_RUNNER_ALLOW_ROOT_STEPS` | `1` runs steps as root in an image without the `runner` user. A runner never sets it |
97+
98+Integration test variables are listed in [development](development.md).
A docs/development.md +65 -0
@@ -0,0 +1,65 @@
1+# Development
2+
3+## Build and check
4+
5+```sh
6+CGO_ENABLED=0 go build -trimpath -o ermite-runner .
7+GOWORK=off go vet ./... && GOWORK=off go test -race ./...
8+```
9+
10+- The binary must be static: the sandbox has no libc of ours.
11+- Unit tests need nothing special. The drivers are tested with a fake Firecracker and a fake Docker engine.
12+- Protobuf: `go generate ./internal/execstream`. `buf` and `protoc-gen-go` are pinned as `tool` directives in `go.mod`. CI lints the proto and fails when the committed generated code differs.
13+- `ermite-runner sandbox-check [script]` runs one stub job in the selected mode. It is the smoke test of an installation.
14+
15+## Integration tests
16+
17+They are opt-in. They run the real stub executor. Output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
18+
19+| Variable | Enables |
20+| --- | --- |
21+| `ERMITE_RUNNER_IT_VM=1` | VM tests. Need `/dev/kvm`, Firecracker, the kernel and the base image |
22+| `ERMITE_RUNNER_IT_CONTAINER=1` | Container tests. Need a rootful Docker daemon, root and the OCI image |
23+| `ERMITE_RUNNER_IT_E2E=1` | End-to-end job tests in `internal/e2e` |
24+| `ERMITE_RUNNER_IT_NET=1` | Packet-level firewall test |
25+| `ERMITE_RUNNER_IT_DOCKER_API=1` | Docker Engine calls against any daemon, including rootless |
26+| `ERMITE_RUNNER_IT_NODE_DIR` | Directory with `node-v24.21.0-linux-x64.tar.gz`, for offline hosts |
27+| `ERMITE_RUNNER_TEST_IMAGE` | Image used by the container and end-to-end tests |
28+
29+Build the kernel and images first: see [images](images.md).
30+
31+### VM
32+
33+```sh
34+# Unprivileged, in a user and network namespace. The jailer is off.
35+FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
36+
37+# Privileged, with the jailer (the production configuration)
38+sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
39+ ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
40+ "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
41+```
42+
43+### Container
44+
45+```sh
46+sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:latest \
47+ DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
48+```
49+
50+### End to end
51+
52+The real executor runs in the real image, started by the container driver. It also works with a rootless Docker daemon, where `mount` and `umount` are replaced by stand-ins.
53+
54+```sh
55+IMAGE_TAG=ermite-runner-ubuntu-24.04:dev scripts/build-image.sh tmp/image
56+ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-24.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
57+```
58+
59+- `TestChainCorpus` runs jobs of the ermite-conformance corpus through a fake server and compares the result with recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
60+
61+## Notes
62+
63+- **Docker API.** Production `container` mode supports Docker Engine and sysbox. Podman's compatibility API is not a drop-in: it cannot disable DNS on the network that the runner creates.
64+- **Jailer.** The jailer needs `--chroot-base-dir` to exist. The driver creates `<state dir>/jail`. It leaves `<state dir>/jail/<exec name>` and `/sys/fs/cgroup/firecracker` behind.
65+- **Host firewalls.** A firewall that rejects `53` on `ertap*` or `erbr*` breaks guest DNS. The symptom is `dns silent` in `TestVMNetworkPolicy`.
@@ -0,0 +1,65 @@
1+# Development
2+
3+## Build and check
4+
5+```sh
6+CGO_ENABLED=0 go build -trimpath -o ermite-runner .
7+GOWORK=off go vet ./... && GOWORK=off go test -race ./...
8+```
9+
10+- The binary must be static: the sandbox has no libc of ours.
11+- Unit tests need nothing special. The drivers are tested with a fake Firecracker and a fake Docker engine.
12+- Protobuf: `go generate ./internal/execstream`. `buf` and `protoc-gen-go` are pinned as `tool` directives in `go.mod`. CI lints the proto and fails when the committed generated code differs.
13+- `ermite-runner sandbox-check [script]` runs one stub job in the selected mode. It is the smoke test of an installation.
14+
15+## Integration tests
16+
17+They are opt-in. They run the real stub executor. Output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
18+
19+| Variable | Enables |
20+| --- | --- |
21+| `ERMITE_RUNNER_IT_VM=1` | VM tests. Need `/dev/kvm`, Firecracker, the kernel and the base image |
22+| `ERMITE_RUNNER_IT_CONTAINER=1` | Container tests. Need a rootful Docker daemon, root and the OCI image |
23+| `ERMITE_RUNNER_IT_E2E=1` | End-to-end job tests in `internal/e2e` |
24+| `ERMITE_RUNNER_IT_NET=1` | Packet-level firewall test |
25+| `ERMITE_RUNNER_IT_DOCKER_API=1` | Docker Engine calls against any daemon, including rootless |
26+| `ERMITE_RUNNER_IT_NODE_DIR` | Directory with `node-v24.21.0-linux-x64.tar.gz`, for offline hosts |
27+| `ERMITE_RUNNER_TEST_IMAGE` | Image used by the container and end-to-end tests |
28+
29+Build the kernel and images first: see [images](images.md).
30+
31+### VM
32+
33+```sh
34+# Unprivileged, in a user and network namespace. The jailer is off.
35+FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
36+
37+# Privileged, with the jailer (the production configuration)
38+sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
39+ ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
40+ "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
41+```
42+
43+### Container
44+
45+```sh
46+sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:latest \
47+ DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
48+```
49+
50+### End to end
51+
52+The real executor runs in the real image, started by the container driver. It also works with a rootless Docker daemon, where `mount` and `umount` are replaced by stand-ins.
53+
54+```sh
55+IMAGE_TAG=ermite-runner-ubuntu-24.04:dev scripts/build-image.sh tmp/image
56+ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-24.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
57+```
58+
59+- `TestChainCorpus` runs jobs of the ermite-conformance corpus through a fake server and compares the result with recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
60+
61+## Notes
62+
63+- **Docker API.** Production `container` mode supports Docker Engine and sysbox. Podman's compatibility API is not a drop-in: it cannot disable DNS on the network that the runner creates.
64+- **Jailer.** The jailer needs `--chroot-base-dir` to exist. The driver creates `<state dir>/jail`. It leaves `<state dir>/jail/<exec name>` and `/sys/fs/cgroup/firecracker` behind.
65+- **Host firewalls.** A firewall that rejects `53` on `ertap*` or `erbr*` breaks guest DNS. The symptom is `dns silent` in `TestVMNetworkPolicy`.
A docs/externals.md +60 -0
@@ -0,0 +1,60 @@
1+# Node.js externals
2+
3+The host provides Node.js to jobs outside the image. A runner upgrade or a new Node.js version needs no image rebuild.
4+
5+## Builds
6+
7+The builds are the ones of `actions/runner`. Each archive is pinned with its SHA-256 in `internal/externals/table.go`.
8+
9+| Name | Node.js | Source | Architectures |
10+| --- | --- | --- | --- |
11+| `node20` | 20.20.2 | `nodejs.org/dist/v20.20.2/node-v20.20.2-linux-<arch>.tar.gz` | x64, arm64 |
12+| `node24` | 24.21.0 | `nodejs.org/dist/v24.21.0/node-v24.21.0-linux-<arch>.tar.gz` | x64, arm64 |
13+| `node20_alpine` | 20.20.2 | `github.com/actions/alpine_nodejs` release `v20.20.2` | x64 |
14+| `node24_alpine` | 24.21.0 | `github.com/actions/alpine_nodejs` release `v24.21.0` | x64 |
15+
16+## Which Node.js an action uses
17+
18+- As on github.com, actions with `using: node12`, `node16` or `node20` run on `node24`.
19+- `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24` and `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION` have no effect.
20+- "Complete job" prints a warning that lists the actions forced from Node.js 20 to 24.
21+
22+## Download
23+
24+- `ermite-runner run` and `sandbox-check` fetch missing builds at start, in the background, into `$ERMITE_RUNNER_STATE_DIR/externals/<name>-<version>-<arch>/`.
25+- Start waits for them for at most 2 minutes.
26+- The SHA-256 of the `.tar.gz` is checked **before** anything is unpacked. Redirects must stay on https.
27+- Extraction refuses:
28+ - absolute names and `..`;
29+ - devices and fifos;
30+ - symbolic links that are absolute or leave the tree;
31+ - entries below a link;
32+ - hard links to anything but an unpacked file;
33+ - archives over 200,000 entries or 2 GiB.
34+- Each build is unpacked in a temporary directory and renamed into place. A directory under its final name is complete.
35+- At every start, everything in `externals/` that is not a pinned build is removed.
36+
37+## In the sandbox
38+
39+Read-only, as on GitHub: `/home/runner/externals/node20/bin/node`, `node24/bin/node` and the alpine ones.
40+
41+| Mode | How |
42+| --- | --- |
43+| `container` | Read-only bind mounts of the host directories |
44+| `vm` | One read-only ext4 disk, built from the set (`cache/extdisk-<key>.ext4`). Old disks are removed an hour after the last build |
45+
46+The set is read at every job start. A download that finishes later is used by the next job.
47+
48+## When a build is unavailable
49+
50+Causes: offline, HTTP error, wrong checksum, unsafe archive, or no build for the architecture.
51+
52+- The runner still runs and logs why.
53+- The sandbox gets a stub `bin/node` and a file `UNAVAILABLE` with the reason.
54+- A job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: ...`.
55+- Failures that may pass (network error, HTTP status) are retried every 5 minutes until the runner stops.
56+- Other failures (wrong checksum, unsafe archive, no build for the architecture) are not retried before the next start.
57+
58+## Turn off
59+
60+`ERMITE_RUNNER_NODE=0` turns the externals off. Nothing is downloaded or attached.
@@ -0,0 +1,60 @@
1+# Node.js externals
2+
3+The host provides Node.js to jobs outside the image. A runner upgrade or a new Node.js version needs no image rebuild.
4+
5+## Builds
6+
7+The builds are the ones of `actions/runner`. Each archive is pinned with its SHA-256 in `internal/externals/table.go`.
8+
9+| Name | Node.js | Source | Architectures |
10+| --- | --- | --- | --- |
11+| `node20` | 20.20.2 | `nodejs.org/dist/v20.20.2/node-v20.20.2-linux-<arch>.tar.gz` | x64, arm64 |
12+| `node24` | 24.21.0 | `nodejs.org/dist/v24.21.0/node-v24.21.0-linux-<arch>.tar.gz` | x64, arm64 |
13+| `node20_alpine` | 20.20.2 | `github.com/actions/alpine_nodejs` release `v20.20.2` | x64 |
14+| `node24_alpine` | 24.21.0 | `github.com/actions/alpine_nodejs` release `v24.21.0` | x64 |
15+
16+## Which Node.js an action uses
17+
18+- As on github.com, actions with `using: node12`, `node16` or `node20` run on `node24`.
19+- `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24` and `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION` have no effect.
20+- "Complete job" prints a warning that lists the actions forced from Node.js 20 to 24.
21+
22+## Download
23+
24+- `ermite-runner run` and `sandbox-check` fetch missing builds at start, in the background, into `$ERMITE_RUNNER_STATE_DIR/externals/<name>-<version>-<arch>/`.
25+- Start waits for them for at most 2 minutes.
26+- The SHA-256 of the `.tar.gz` is checked **before** anything is unpacked. Redirects must stay on https.
27+- Extraction refuses:
28+ - absolute names and `..`;
29+ - devices and fifos;
30+ - symbolic links that are absolute or leave the tree;
31+ - entries below a link;
32+ - hard links to anything but an unpacked file;
33+ - archives over 200,000 entries or 2 GiB.
34+- Each build is unpacked in a temporary directory and renamed into place. A directory under its final name is complete.
35+- At every start, everything in `externals/` that is not a pinned build is removed.
36+
37+## In the sandbox
38+
39+Read-only, as on GitHub: `/home/runner/externals/node20/bin/node`, `node24/bin/node` and the alpine ones.
40+
41+| Mode | How |
42+| --- | --- |
43+| `container` | Read-only bind mounts of the host directories |
44+| `vm` | One read-only ext4 disk, built from the set (`cache/extdisk-<key>.ext4`). Old disks are removed an hour after the last build |
45+
46+The set is read at every job start. A download that finishes later is used by the next job.
47+
48+## When a build is unavailable
49+
50+Causes: offline, HTTP error, wrong checksum, unsafe archive, or no build for the architecture.
51+
52+- The runner still runs and logs why.
53+- The sandbox gets a stub `bin/node` and a file `UNAVAILABLE` with the reason.
54+- A job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: ...`.
55+- Failures that may pass (network error, HTTP status) are retried every 5 minutes until the runner stops.
56+- Other failures (wrong checksum, unsafe archive, no build for the architecture) are not retried before the next start.
57+
58+## Turn off
59+
60+`ERMITE_RUNNER_NODE=0` turns the externals off. Nothing is downloaded or attached.
A docs/images.md +34 -0
@@ -0,0 +1,34 @@
1+# Images and kernel
2+
3+A job runs in the image named by `ERMITE_RUNNER_IMAGE`.
4+
5+| Mode | Value | Contents |
6+| --- | --- | --- |
7+| `container` | OCI reference | Default `ermite-runner-ubuntu-24.04:latest` |
8+| `vm` | Directory | `vmlinux` (`Image` on arm64) and `base.ext4`. Default `<state dir>/image` |
9+
10+Both come from `images/ubuntu-24.04/Dockerfile`. The runner executable and Node.js are attached by the host. They are never part of the image.
11+
12+## Build
13+
14+```sh
15+scripts/build-kernel.sh tmp/image # vm only: builds the guest kernel
16+scripts/build-image.sh tmp/image # builds the OCI image and tmp/image/base.ext4
17+```
18+
19+- `build-kernel.sh` downloads Linux 6.18 LTS from kernel.org and checks the tarball against the published sha256 sums. It needs `bc`, `flex`, `bison`, `pahole`, libelf and openssl headers.
20+- `build-image.sh` needs Docker or Podman (rootless is fine) and e2fsprogs 1.47.1 or newer. No root is needed.
21+- The OCI image is tagged `ermite-runner-ubuntu-24.04:latest`. Set `IMAGE_TAG` to change it.
22+- The kernel config is `images/kernel/config-6.18` plus per-architecture fragments. The build fails if a listed option does not survive `olddefconfig`.
23+- arm64 builds with `ARCH_NAME=arm64`.
24+- `CONFIG_BINFMT_MISC` is built in and mounted by the guest init, so `docker/setup-qemu-action` can run foreign-architecture containers.
25+
26+## What the image must contain
27+
28+- User `runner`, uid and gid 1001, home `/home/runner`. Steps run as this user. Without it, steps run as root (test images only).
29+- `/home/runner/work` and `/opt/hostedtoolcache`, owned by `runner`.
30+- `bash` and `sh`. `python3` and a `python` command for `shell: python`.
31+- `dockerd` and passwordless `sudo` for `runner`, for jobs that use Docker.
32+- Common tools: git, curl, jq, build-essential, ca-certificates, tzdata, zip and unzip.
33+
34+`pwsh` is not in the image. A step that asks for it fails with `pwsh: command not found`.
@@ -0,0 +1,34 @@
1+# Images and kernel
2+
3+A job runs in the image named by `ERMITE_RUNNER_IMAGE`.
4+
5+| Mode | Value | Contents |
6+| --- | --- | --- |
7+| `container` | OCI reference | Default `ermite-runner-ubuntu-24.04:latest` |
8+| `vm` | Directory | `vmlinux` (`Image` on arm64) and `base.ext4`. Default `<state dir>/image` |
9+
10+Both come from `images/ubuntu-24.04/Dockerfile`. The runner executable and Node.js are attached by the host. They are never part of the image.
11+
12+## Build
13+
14+```sh
15+scripts/build-kernel.sh tmp/image # vm only: builds the guest kernel
16+scripts/build-image.sh tmp/image # builds the OCI image and tmp/image/base.ext4
17+```
18+
19+- `build-kernel.sh` downloads Linux 6.18 LTS from kernel.org and checks the tarball against the published sha256 sums. It needs `bc`, `flex`, `bison`, `pahole`, libelf and openssl headers.
20+- `build-image.sh` needs Docker or Podman (rootless is fine) and e2fsprogs 1.47.1 or newer. No root is needed.
21+- The OCI image is tagged `ermite-runner-ubuntu-24.04:latest`. Set `IMAGE_TAG` to change it.
22+- The kernel config is `images/kernel/config-6.18` plus per-architecture fragments. The build fails if a listed option does not survive `olddefconfig`.
23+- arm64 builds with `ARCH_NAME=arm64`.
24+- `CONFIG_BINFMT_MISC` is built in and mounted by the guest init, so `docker/setup-qemu-action` can run foreign-architecture containers.
25+
26+## What the image must contain
27+
28+- User `runner`, uid and gid 1001, home `/home/runner`. Steps run as this user. Without it, steps run as root (test images only).
29+- `/home/runner/work` and `/opt/hostedtoolcache`, owned by `runner`.
30+- `bash` and `sh`. `python3` and a `python` command for `shell: python`.
31+- `dockerd` and passwordless `sudo` for `runner`, for jobs that use Docker.
32+- Common tools: git, curl, jq, build-essential, ca-certificates, tzdata, zip and unzip.
33+
34+`pwsh` is not in the image. A step that asks for it fails with `pwsh: command not found`.
A docs/sandboxes.md +121 -0
@@ -0,0 +1,121 @@
1+# Sandboxes
2+
3+Each job runs in its own sandbox. Both modes attach the runner's own executable read-only to the sandbox. It must be a static build (`CGO_ENABLED=0`). The runner refuses to start otherwise.
4+
5+Inside the sandbox, `ermite-runner init` is the entrypoint (PID 1 in a VM). `ermite-runner exec` runs the job and talks to the host over a framed protobuf stream: vsock in a VM, a Unix socket in a container.
6+
7+## vm
8+
9+- Firecracker runs the job in a microVM. The VM is destroyed when the job ends.
10+- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires.
11+- Disks:
12+ - The runner executable is the read-only boot root.
13+ - The shared base image is read-only.
14+ - A sparse scratch disk per job (`ERMITE_RUNNER_JOB_DISK`) holds the writable layer and the Docker data root.
15+ - A read-only disk holds the Node.js externals, when there are any.
16+- The scratch disk and `eth0` are rate limited. See [job limits](configuration.md#job-limits).
17+- Network: one TAP device (`ertap<slot>`) per VM with a `/30` from `ERMITE_RUNNER_NETWORK`. IPv6 is disabled in the guest.
18+- The jailer runs each VM:
19+ - as uid and gid `ERMITE_RUNNER_UID_BASE` plus the slot;
20+ - in its own chroot under the state directory;
21+ - in a new PID namespace;
22+ - in a cgroup v2 with `memory.max` (guest memory plus 256 MiB) and `pids.max`.
23+- The kernel, the base image and the state directory must be on the same filesystem. The kernel and the base image must be world-readable.
24+- The jailed API socket path must fit in 107 bytes. The driver checks it at start.
25+
26+## container
27+
28+- Uses the Docker Engine API (`DOCKER_HOST` is honoured). The Docker daemon must run on the same host as the runner.
29+- Runtime: `sysbox-runc` when the daemon lists it. Otherwise a **privileged** container.
30+- A rootless Docker daemon is refused. A nested `dockerd` needs a rootful daemon or sysbox.
31+- Limits per job: memory (no swap), CPUs and `ERMITE_RUNNER_JOB_PIDS` processes.
32+- Disk:
33+ - A sparse ext4 file of `ERMITE_RUNNER_JOB_DISK`, loop-mounted on the host and bind-mounted at `/scratch`. This is why the mode needs root.
34+ - The nested `dockerd` data root, `/home/runner/work` and `/tmp` live on it.
35+ - A job that fills its disk gets "no space left on device".
36+- Per job: a bridge network (`erbr<slot>`) and a Unix socket in a per-job directory of the state directory. The job sees only that directory.
37+- Every Docker object carries the label `dev.bnema.ermite-runner.job`. Leftovers are reaped by label at start.
38+- The state directory path must keep the socket path under 107 bytes.
39+
40+### Privileged container warning
41+
42+- A privileged container is equivalent to host root.
43+- It is allowed by default. The runner logs a warning at start and at the start of every job.
44+- The isolation reported at registration is `container_privileged`, `container_sysbox` or `vm`.
45+- `ERMITE_RUNNER_DENY_PRIVILEGED=1` refuses to start without sysbox.
46+- Network policy is best effort in this mode.
47+
48+## State directory
49+
50+- The state directory and every parent must be owned by root and not writable by group or others.
51+- The jailer runs as root and builds chroots under `<state dir>/jail`. The driver checks `jail` itself. It does not check the parents.
52+- A user who can rename or replace a parent can redirect root's file operations.
53+- The default `/var/lib/ermite-runner` is safe. The unit creates it with `StateDirectory`.
54+- Do not point `ERMITE_RUNNER_STATE_DIR` into a home directory or `/tmp`.
55+
56+## One runner per host
57+
58+- Cleanup removes every process, device, firewall rule and Docker object the runner owns. Two runners on one host would destroy each other's jobs.
59+- The runner takes an exclusive lock on `/run/ermite-runner.lock` and refuses to start when it is held. `ERMITE_RUNNER_LOCK_FILE` overrides the path for tests.
60+- The kernel drops the lock when the process dies.
61+- A runner in a container or another mount namespace must share that path. Do not run it that way.
62+
63+## Network policy
64+
65+Both modes use one nftables table, `inet ermite_runner`. It is replaced atomically and deleted with the last sandbox. Nothing else is touched.
66+
67+From a sandbox:
68+
69+| Traffic | Result |
70+| --- | --- |
71+| DNS to its own gateway address | Allowed. The runner's DNS forwarder answers, using the resolvers of `/etc/resolv.conf` |
72+| One TCP port to the gateway, with guest forge access on | Allowed |
73+| Other traffic to the host | Denied |
74+| RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, documentation, benchmarking, IETF-assigned, multicast and reserved ranges | Denied |
75+| IPv6 | Denied |
76+| Traffic between sandboxes | Denied |
77+| The rest | NATed |
78+
79+- Source address and destination checks run before any "established" accept. A guest cannot forge another's flow.
80+- The DNS forwarder allows 32 in-flight queries per listener (extra UDP is dropped), 16 TCP connections, and 30 s per TCP connection.
81+- Golden rulesets: `internal/netpolicy/testdata`.
82+
83+Host side effects:
84+
85+- The runner turns `net.ipv4.ip_forward` on and leaves it on when it exits.
86+- Forwarding is host-wide. If the machine must not route for other networks, restrict it with your own firewall rules.
87+- Another firewall with a later hook (for example firewalld) can still block guest DNS. Allow DNS from `ertap*` and `erbr*` there.
88+
89+## Guest forge access
90+
91+Off by default. Use it on staging hosts only.
92+
93+`ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the server the way the runner does. It is for hosts where the server is reachable only through a local route, such as an `/etc/hosts` entry plus a reverse proxy on a local port. Jobs cannot otherwise run `git clone` against `github.server_url`.
94+
95+The host name and port come from `ERMITE_RUNNER_URL`. The runner refuses to start when:
96+
97+- the URL host is an IP literal;
98+- the URL has a path;
99+- the name does not resolve on the host;
100+- the URL is `http://` and `ERMITE_RUNNER_INSECURE_HTTP=1` is not set.
101+
102+For every sandbox:
103+
104+- The DNS forwarder answers `A` queries for that host name with the sandbox's gateway address. `AAAA` gets an empty answer.
105+- The firewall accepts TCP from the sandbox to its gateway on that one port.
106+- `github.server_url`, `github.api_url` and `github.graphql_url` (and the matching `GITHUB_*` variables) are rewritten to the origin of `ERMITE_RUNNER_URL`.
107+- The runner logs `guest forge access: sandboxes may reach <origin> on the host` at start and at every job start.
108+
109+Security notes:
110+
111+- The runner does not proxy. A service must listen on that port on an address the gateway reaches (`*`, not `127.0.0.1`).
112+- A shared reverse proxy routes by `Host` header. A job can reach every app that proxy serves on that port.
113+- Traffic comes from a private address. Anything behind the proxy that trusts private sources trusts job code too.
114+- Jobs see the server over plain HTTP in this setup.
115+
116+## Cleanup
117+
118+- `Stop` removes everything a job creates: processes, TAP or bridge, firewall rules, files, loop mounts, containers, networks and cgroups. It also runs with a cancelled context.
119+- At start, the runner reaps what a crash left behind.
120+- The network policy and the TAP or bridge are never removed while the VMM or container may still run.
121+- If a process cannot be killed or a container cannot be removed, the slot stays taken until the next reap. The host stops taking jobs on that slot.
@@ -0,0 +1,121 @@
1+# Sandboxes
2+
3+Each job runs in its own sandbox. Both modes attach the runner's own executable read-only to the sandbox. It must be a static build (`CGO_ENABLED=0`). The runner refuses to start otherwise.
4+
5+Inside the sandbox, `ermite-runner init` is the entrypoint (PID 1 in a VM). `ermite-runner exec` runs the job and talks to the host over a framed protobuf stream: vsock in a VM, a Unix socket in a container.
6+
7+## vm
8+
9+- Firecracker runs the job in a microVM. The VM is destroyed when the job ends.
10+- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires.
11+- Disks:
12+ - The runner executable is the read-only boot root.
13+ - The shared base image is read-only.
14+ - A sparse scratch disk per job (`ERMITE_RUNNER_JOB_DISK`) holds the writable layer and the Docker data root.
15+ - A read-only disk holds the Node.js externals, when there are any.
16+- The scratch disk and `eth0` are rate limited. See [job limits](configuration.md#job-limits).
17+- Network: one TAP device (`ertap<slot>`) per VM with a `/30` from `ERMITE_RUNNER_NETWORK`. IPv6 is disabled in the guest.
18+- The jailer runs each VM:
19+ - as uid and gid `ERMITE_RUNNER_UID_BASE` plus the slot;
20+ - in its own chroot under the state directory;
21+ - in a new PID namespace;
22+ - in a cgroup v2 with `memory.max` (guest memory plus 256 MiB) and `pids.max`.
23+- The kernel, the base image and the state directory must be on the same filesystem. The kernel and the base image must be world-readable.
24+- The jailed API socket path must fit in 107 bytes. The driver checks it at start.
25+
26+## container
27+
28+- Uses the Docker Engine API (`DOCKER_HOST` is honoured). The Docker daemon must run on the same host as the runner.
29+- Runtime: `sysbox-runc` when the daemon lists it. Otherwise a **privileged** container.
30+- A rootless Docker daemon is refused. A nested `dockerd` needs a rootful daemon or sysbox.
31+- Limits per job: memory (no swap), CPUs and `ERMITE_RUNNER_JOB_PIDS` processes.
32+- Disk:
33+ - A sparse ext4 file of `ERMITE_RUNNER_JOB_DISK`, loop-mounted on the host and bind-mounted at `/scratch`. This is why the mode needs root.
34+ - The nested `dockerd` data root, `/home/runner/work` and `/tmp` live on it.
35+ - A job that fills its disk gets "no space left on device".
36+- Per job: a bridge network (`erbr<slot>`) and a Unix socket in a per-job directory of the state directory. The job sees only that directory.
37+- Every Docker object carries the label `dev.bnema.ermite-runner.job`. Leftovers are reaped by label at start.
38+- The state directory path must keep the socket path under 107 bytes.
39+
40+### Privileged container warning
41+
42+- A privileged container is equivalent to host root.
43+- It is allowed by default. The runner logs a warning at start and at the start of every job.
44+- The isolation reported at registration is `container_privileged`, `container_sysbox` or `vm`.
45+- `ERMITE_RUNNER_DENY_PRIVILEGED=1` refuses to start without sysbox.
46+- Network policy is best effort in this mode.
47+
48+## State directory
49+
50+- The state directory and every parent must be owned by root and not writable by group or others.
51+- The jailer runs as root and builds chroots under `<state dir>/jail`. The driver checks `jail` itself. It does not check the parents.
52+- A user who can rename or replace a parent can redirect root's file operations.
53+- The default `/var/lib/ermite-runner` is safe. The unit creates it with `StateDirectory`.
54+- Do not point `ERMITE_RUNNER_STATE_DIR` into a home directory or `/tmp`.
55+
56+## One runner per host
57+
58+- Cleanup removes every process, device, firewall rule and Docker object the runner owns. Two runners on one host would destroy each other's jobs.
59+- The runner takes an exclusive lock on `/run/ermite-runner.lock` and refuses to start when it is held. `ERMITE_RUNNER_LOCK_FILE` overrides the path for tests.
60+- The kernel drops the lock when the process dies.
61+- A runner in a container or another mount namespace must share that path. Do not run it that way.
62+
63+## Network policy
64+
65+Both modes use one nftables table, `inet ermite_runner`. It is replaced atomically and deleted with the last sandbox. Nothing else is touched.
66+
67+From a sandbox:
68+
69+| Traffic | Result |
70+| --- | --- |
71+| DNS to its own gateway address | Allowed. The runner's DNS forwarder answers, using the resolvers of `/etc/resolv.conf` |
72+| One TCP port to the gateway, with guest forge access on | Allowed |
73+| Other traffic to the host | Denied |
74+| RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, documentation, benchmarking, IETF-assigned, multicast and reserved ranges | Denied |
75+| IPv6 | Denied |
76+| Traffic between sandboxes | Denied |
77+| The rest | NATed |
78+
79+- Source address and destination checks run before any "established" accept. A guest cannot forge another's flow.
80+- The DNS forwarder allows 32 in-flight queries per listener (extra UDP is dropped), 16 TCP connections, and 30 s per TCP connection.
81+- Golden rulesets: `internal/netpolicy/testdata`.
82+
83+Host side effects:
84+
85+- The runner turns `net.ipv4.ip_forward` on and leaves it on when it exits.
86+- Forwarding is host-wide. If the machine must not route for other networks, restrict it with your own firewall rules.
87+- Another firewall with a later hook (for example firewalld) can still block guest DNS. Allow DNS from `ertap*` and `erbr*` there.
88+
89+## Guest forge access
90+
91+Off by default. Use it on staging hosts only.
92+
93+`ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the server the way the runner does. It is for hosts where the server is reachable only through a local route, such as an `/etc/hosts` entry plus a reverse proxy on a local port. Jobs cannot otherwise run `git clone` against `github.server_url`.
94+
95+The host name and port come from `ERMITE_RUNNER_URL`. The runner refuses to start when:
96+
97+- the URL host is an IP literal;
98+- the URL has a path;
99+- the name does not resolve on the host;
100+- the URL is `http://` and `ERMITE_RUNNER_INSECURE_HTTP=1` is not set.
101+
102+For every sandbox:
103+
104+- The DNS forwarder answers `A` queries for that host name with the sandbox's gateway address. `AAAA` gets an empty answer.
105+- The firewall accepts TCP from the sandbox to its gateway on that one port.
106+- `github.server_url`, `github.api_url` and `github.graphql_url` (and the matching `GITHUB_*` variables) are rewritten to the origin of `ERMITE_RUNNER_URL`.
107+- The runner logs `guest forge access: sandboxes may reach <origin> on the host` at start and at every job start.
108+
109+Security notes:
110+
111+- The runner does not proxy. A service must listen on that port on an address the gateway reaches (`*`, not `127.0.0.1`).
112+- A shared reverse proxy routes by `Host` header. A job can reach every app that proxy serves on that port.
113+- Traffic comes from a private address. Anything behind the proxy that trusts private sources trusts job code too.
114+- Jobs see the server over plain HTTP in this setup.
115+
116+## Cleanup
117+
118+- `Stop` removes everything a job creates: processes, TAP or bridge, firewall rules, files, loop mounts, containers, networks and cgroups. It also runs with a cancelled context.
119+- At start, the runner reaps what a crash left behind.
120+- The network policy and the TAP or bridge are never removed while the VMM or container may still run.
121+- If a process cannot be killed or a container cannot be removed, the slot stays taken until the next reap. The host stops taking jobs on that slot.
A docs/service.md +65 -0
@@ -0,0 +1,65 @@
1+# Service
2+
3+`contrib/ermite-runner.service` is a systemd unit. For the commands to install it, see the [README](../README.md#register-and-run).
4+
5+## Privileges
6+
7+- The runner runs as root on the host. It needs TAP devices, nftables, loop mounts, the jailer and Docker.
8+- Jobs never run as root on the host. They run as user `runner` in a microVM or a container.
9+
10+## Registration
11+
12+- Register in the same environment as the service, so `register` and `run` agree on the URL, name, labels, isolation and state directory.
13+- `systemd-run` with the same `EnvironmentFile` does this.
14+- `register` picks the isolation mode like `run` will. It sends the name, labels, version, isolation and architecture.
15+
16+## Logs
17+
18+- The runner writes to standard error, which journald collects.
19+- Set `ERMITE_RUNNER_LOG_FORMAT=json` under journald. Every line about a job has `job_id`, `run_id` and `attempt`.
20+- `warn` marks what is retried. `error` marks what is refused or leaked.
21+- `ERMITE_RUNNER_LOG_LEVEL=debug` adds detail.
22+- Tokens are never logged.
23+
24+## Stopping
25+
26+`systemctl stop` sends SIGTERM. The runner:
27+
28+1. Cancels running jobs with `runner_shutdown`.
29+2. Reports them to the server.
30+3. Removes their sandboxes.
31+
32+Budget with the defaults:
33+
34+| Step | Time |
35+| --- | --- |
36+| Executor stops its steps and runs post-steps | 30 s |
37+| Destroy the sandbox | 30 s |
38+| Flush the log and report | 30 s |
39+| **Total** | **90 s** |
40+
41+- Jobs stop in parallel. The total does not grow with `ERMITE_RUNNER_CAPACITY`.
42+- The unit sets `TimeoutStopSec=120`.
43+- A job whose first heartbeat is pending at SIGTERM is not started. The server gives it to another runner after 30 s.
44+- A crash or power cut leaves sandboxes behind. The next start reaps them.
45+
46+## Unit hardening
47+
48+The unit sets only options that keep it in the host's mount namespace:
49+
50+- `NoNewPrivileges`
51+- `ProtectClock`
52+- `RestrictSUIDSGID`
53+- `LockPersonality`
54+- `RestrictRealtime`
55+- `SystemCallArchitectures=native`
56+- `RestrictAddressFamilies` (UNIX, INET, INET6, NETLINK, VSOCK)
57+
58+Do not add:
59+
60+- Options that give the unit its own mount namespace: `PrivateTmp`, `ProtectHome`, `ProtectSystem`, `ProtectKernelModules`, `ProtectKernelLogs`, `ProtectKernelTunables`, `ProtectControlGroups`, `PrivateDevices`, `ReadOnlyPaths`, `ReadWritePaths`, `InaccessiblePaths`, `TemporaryFileSystem`, `BindPaths`. In `container` mode the runner loop-mounts the job disk and the Docker daemon, in the host namespace, must see it. A test fails if the unit sets one of them.
61+- `PrivateNetwork`, `RestrictNamespaces`, `CapabilityBoundingSet`, `SystemCallFilter`: the runner creates network devices, mounts filesystems and hands capabilities to the jailer and Docker.
62+- `MemoryDenyWriteExecute`: Firecracker maps guest memory writable and executable.
63+- `Delegate` and `UMask`: not needed.
64+
65+The security boundary of a job is its microVM or container, not the unit.
@@ -0,0 +1,65 @@
1+# Service
2+
3+`contrib/ermite-runner.service` is a systemd unit. For the commands to install it, see the [README](../README.md#register-and-run).
4+
5+## Privileges
6+
7+- The runner runs as root on the host. It needs TAP devices, nftables, loop mounts, the jailer and Docker.
8+- Jobs never run as root on the host. They run as user `runner` in a microVM or a container.
9+
10+## Registration
11+
12+- Register in the same environment as the service, so `register` and `run` agree on the URL, name, labels, isolation and state directory.
13+- `systemd-run` with the same `EnvironmentFile` does this.
14+- `register` picks the isolation mode like `run` will. It sends the name, labels, version, isolation and architecture.
15+
16+## Logs
17+
18+- The runner writes to standard error, which journald collects.
19+- Set `ERMITE_RUNNER_LOG_FORMAT=json` under journald. Every line about a job has `job_id`, `run_id` and `attempt`.
20+- `warn` marks what is retried. `error` marks what is refused or leaked.
21+- `ERMITE_RUNNER_LOG_LEVEL=debug` adds detail.
22+- Tokens are never logged.
23+
24+## Stopping
25+
26+`systemctl stop` sends SIGTERM. The runner:
27+
28+1. Cancels running jobs with `runner_shutdown`.
29+2. Reports them to the server.
30+3. Removes their sandboxes.
31+
32+Budget with the defaults:
33+
34+| Step | Time |
35+| --- | --- |
36+| Executor stops its steps and runs post-steps | 30 s |
37+| Destroy the sandbox | 30 s |
38+| Flush the log and report | 30 s |
39+| **Total** | **90 s** |
40+
41+- Jobs stop in parallel. The total does not grow with `ERMITE_RUNNER_CAPACITY`.
42+- The unit sets `TimeoutStopSec=120`.
43+- A job whose first heartbeat is pending at SIGTERM is not started. The server gives it to another runner after 30 s.
44+- A crash or power cut leaves sandboxes behind. The next start reaps them.
45+
46+## Unit hardening
47+
48+The unit sets only options that keep it in the host's mount namespace:
49+
50+- `NoNewPrivileges`
51+- `ProtectClock`
52+- `RestrictSUIDSGID`
53+- `LockPersonality`
54+- `RestrictRealtime`
55+- `SystemCallArchitectures=native`
56+- `RestrictAddressFamilies` (UNIX, INET, INET6, NETLINK, VSOCK)
57+
58+Do not add:
59+
60+- Options that give the unit its own mount namespace: `PrivateTmp`, `ProtectHome`, `ProtectSystem`, `ProtectKernelModules`, `ProtectKernelLogs`, `ProtectKernelTunables`, `ProtectControlGroups`, `PrivateDevices`, `ReadOnlyPaths`, `ReadWritePaths`, `InaccessiblePaths`, `TemporaryFileSystem`, `BindPaths`. In `container` mode the runner loop-mounts the job disk and the Docker daemon, in the host namespace, must see it. A test fails if the unit sets one of them.
61+- `PrivateNetwork`, `RestrictNamespaces`, `CapabilityBoundingSet`, `SystemCallFilter`: the runner creates network devices, mounts filesystems and hands capabilities to the jailer and Docker.
62+- `MemoryDenyWriteExecute`: Firecracker maps guest memory writable and executable.
63+- `Delegate` and `UMask`: not needed.
64+
65+The security boundary of a job is its microVM or container, not the unit.
A docs/workflows.md +46 -0
@@ -0,0 +1,46 @@
1+# Workflow features
2+
3+## Job containers and service containers
4+
5+`container:` and `services:` work as on GitHub-hosted Linux runners. The containers are created in the **sandbox's own `dockerd`**, so nothing reaches the host.
6+
7+- **Initialize containers** runs after "Set up job":
8+ - checks the Docker API version (1.35 or newer);
9+ - removes labelled leftovers;
10+ - creates the network `github_network_<32 hex>`;
11+ - pulls (3 tries) and starts each container;
12+ - waits for service health checks.
13+- Registry `credentials:` go to a private `docker --config` directory under `RUNNER_TEMP`. The password goes through stdin and is masked. The directory is removed after the pull.
14+- **Stop containers** is a post step (`always()`). It prints service logs and removes containers and the network. It runs after a failure, a cancellation and a timeout.
15+- `run:` steps and node actions in the job container go through `docker exec`. Values are passed in the environment of the docker process, never on the command line.
16+- Node actions in a job container use the externals mounted at `/__e`. `node<N>_alpine` is used when `/etc/*release` says alpine.
17+- Docker actions run as siblings on the job network.
18+- Contexts `job.container.{id,network}` and `job.services.<alias>.{id,network,ports}` are set.
19+- Not supported: container hooks (`ACTIONS_RUNNER_CONTAINER_HOOKS`) and the "already running in a container" refusal.
20+
21+Security notes:
22+
23+- `options:` is split into arguments and passed to `docker create` as an argument vector, never through a shell. A workflow cannot inject a second command.
24+- A workflow can still pass any `docker create` flag (`--privileged`, `--pid host`, `--device`) and bind volumes, as on a self-hosted GitHub runner.
25+- The sandbox contains this, not the runner. **Host paths in `volumes:` and `options:` are paths of the sandbox.**
26+- The runner warns about a volume that mounts `/`.
27+
28+## Cancellation
29+
30+- The host sends a cancel request to the executor.
31+- The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later.
32+- Background children that keep the output pipes open are killed when the shell exits.
33+- Init forwards SIGTERM and SIGINT to the executor.
34+- Step `timeout-minutes` is enforced.
35+
36+## Annotations, summaries and masks
37+
38+- `::notice`, `::warning` and `::error` become annotations, with `title`, `file`, `line`, `endLine`, `col` and `endColumn`.
39+- Problem matchers are supported (`add-matcher`, `remove-matcher`). Each has a 1 s timeout (`GITHUB_ACTIONS_RUNNER_ISSUE_MATCHER_TIMEOUT`) and is removed after 3 timeouts.
40+- Every `##[error]` the runner writes becomes an annotation.
41+- Limits: 10 annotations per level per step, messages cut at 4096 characters.
42+- `GITHUB_STEP_SUMMARY` is read after each step, up to 1 MiB.
43+- `::add-mask::` masks values at run time.
44+- Annotations and summaries are sent only when the server lists `ci.annotations` and `ci.summaries`.
45+
46+Attribution for the ported code: [THIRD_PARTY.md](../THIRD_PARTY.md).
@@ -0,0 +1,46 @@
1+# Workflow features
2+
3+## Job containers and service containers
4+
5+`container:` and `services:` work as on GitHub-hosted Linux runners. The containers are created in the **sandbox's own `dockerd`**, so nothing reaches the host.
6+
7+- **Initialize containers** runs after "Set up job":
8+ - checks the Docker API version (1.35 or newer);
9+ - removes labelled leftovers;
10+ - creates the network `github_network_<32 hex>`;
11+ - pulls (3 tries) and starts each container;
12+ - waits for service health checks.
13+- Registry `credentials:` go to a private `docker --config` directory under `RUNNER_TEMP`. The password goes through stdin and is masked. The directory is removed after the pull.
14+- **Stop containers** is a post step (`always()`). It prints service logs and removes containers and the network. It runs after a failure, a cancellation and a timeout.
15+- `run:` steps and node actions in the job container go through `docker exec`. Values are passed in the environment of the docker process, never on the command line.
16+- Node actions in a job container use the externals mounted at `/__e`. `node<N>_alpine` is used when `/etc/*release` says alpine.
17+- Docker actions run as siblings on the job network.
18+- Contexts `job.container.{id,network}` and `job.services.<alias>.{id,network,ports}` are set.
19+- Not supported: container hooks (`ACTIONS_RUNNER_CONTAINER_HOOKS`) and the "already running in a container" refusal.
20+
21+Security notes:
22+
23+- `options:` is split into arguments and passed to `docker create` as an argument vector, never through a shell. A workflow cannot inject a second command.
24+- A workflow can still pass any `docker create` flag (`--privileged`, `--pid host`, `--device`) and bind volumes, as on a self-hosted GitHub runner.
25+- The sandbox contains this, not the runner. **Host paths in `volumes:` and `options:` are paths of the sandbox.**
26+- The runner warns about a volume that mounts `/`.
27+
28+## Cancellation
29+
30+- The host sends a cancel request to the executor.
31+- The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later.
32+- Background children that keep the output pipes open are killed when the shell exits.
33+- Init forwards SIGTERM and SIGINT to the executor.
34+- Step `timeout-minutes` is enforced.
35+
36+## Annotations, summaries and masks
37+
38+- `::notice`, `::warning` and `::error` become annotations, with `title`, `file`, `line`, `endLine`, `col` and `endColumn`.
39+- Problem matchers are supported (`add-matcher`, `remove-matcher`). Each has a 1 s timeout (`GITHUB_ACTIONS_RUNNER_ISSUE_MATCHER_TIMEOUT`) and is removed after 3 timeouts.
40+- Every `##[error]` the runner writes becomes an annotation.
41+- Limits: 10 annotations per level per step, messages cut at 4096 characters.
42+- `GITHUB_STEP_SUMMARY` is read after each step, up to 1 MiB.
43+- `::add-mask::` masks values at run time.
44+- Annotations and summaries are sent only when the server lists `ci.annotations` and `ci.summaries`.
45+
46+Attribution for the ported code: [THIRD_PARTY.md](../THIRD_PARTY.md).

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 4

Tags 11

↑↓ moveenter openesc close