sudo -E ermite-runner register --token <token with runner:register>
109
+
sudo systemctl enable --now ermite-runner
110
+
```
111
+
112
+
`systemctl stop` sends SIGTERM: running jobs are cancelled with
113
+
`runner_shutdown`, reported to ermite, and their sandboxes removed before the
114
+
service exits (the unit allows 120 s). A crash or a power cut leaves sandboxes
115
+
behind; the next start reaps them. Only one runner may live on a host (a
116
+
host-wide lock enforces it), because reaping removes everything labelled as the
117
+
runner's own.
118
+
119
+
## The image
120
+
121
+
A job runs in the image named by `ERMITE_RUNNER_IMAGE` (container: an OCI
122
+
reference; vm: the directory with `vmlinux` and `base.ext4`, both built from
123
+
`images/ubuntu-24.04/Dockerfile`). The runner binary is attached to the sandbox
124
+
by the host and is never part of the image. What the executor needs from the
125
+
image:
126
+
127
+
- a user **`runner`** with uid and gid 1001 and the home `/home/runner`; the
128
+
executor starts as root (init needs it to start `dockerd`) and runs every step
129
+
as this user, with HOME, USER and LOGNAME set. Without such a user the steps
130
+
run as root (a test image only);
131
+
- the directories `/home/runner/work` and `/opt/hostedtoolcache` owned by
132
+
`runner` (`RUNNER_TOOL_CACHE`, which the executor reads from its environment
133
+
like the official runner; each job gets `/home/runner/work/<repo>/<repo>` and
134
+
`/home/runner/work/_temp`, cleaned at the start of the job);
135
+
- **`bash`** and **`sh`** (the default shell is `bash -e {0}` when bash exists,
136
+
else `sh -e {0}`; like on hosted runners `shell: bash` adds
137
+
`--noprofile --norc -eo pipefail`), `python3` and a `python` command for
138
+
`shell: python`; `pwsh` is not part of the CI-1 image (a step that asks for it
139
+
fails with `pwsh: command not found`);
140
+
-`dockerd` (and passwordless `sudo` for `runner`) for jobs that use Docker: the
141
+
job gets its own daemon with its data root on the scratch disk;
142
+
- what workflows commonly use: git, curl, jq, build-essential, ca-certificates,
143
+
tzdata, zip/unzip. Full parity with the hosted images (`actions/runner-images`)
144
+
is a later phase.
145
+
146
+
## Job limits
147
+
148
+
Each job gets its own resources, set for the runner as a whole:
149
+
150
+
| Variable | Default | Effect |
151
+
|---|---|---|
152
+
| `ERMITE_RUNNER_CAPACITY` | `1` | jobs in parallel: one sandbox, one uid, one `/30` network and one disk per slot. The runner polls ermite only when a slot is free. |
153
+
| `ERMITE_RUNNER_JOB_CPUS` | `2` | vCPUs of a micro-VM; CPU quota of a container |
154
+
| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | guest memory (`M`, `G`, `T`; at least `256M`); a jailed VMM gets this plus 256 MiB of cgroup headroom |
155
+
| `ERMITE_RUNNER_JOB_DISK` | `64G` | size of the sparse scratch disk (at least `1G`): the overlay upper layer of a VM and the Docker data root of a container. Deleted with the job |
156
+
| `ERMITE_RUNNER_JOB_PIDS` | `32768` | process limit of a container |
DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
141
218
```
142
219
220
+
**Jobs end to end** (`ERMITE_RUNNER_IT_E2E=1`, `internal/e2e`): the real executor in the real image, started through the container driver. They also run on a workstation with a *rootless* Docker daemon (the test-only option `AllowRootless`; the nested `dockerd` of the image cannot start there, and the jobs do not need it). `mount` and `umount` are replaced by stand-ins, since loop mounts need root.
ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-24.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
225
+
```
226
+
227
+
-`TestContainerRunsAJobAsTheRunnerUser` and `TestContainerCancelStopsTheJobAndOrphans` drive a job over the stream: steps run as `runner` (1001), file ownership, `GITHUB_PATH` and `GITHUB_ENV`, `python`, cancellation by SIGINT, the `always()` step, and the orphan cleanup of "Complete job".
228
+
-`TestChainCorpus` runs the whole chain: a fake `RunnerService` (`internal/host/hosttest`, which checks the offsets, limits and tokens of `runner.proto`) hands out 22 jobs of the ermite-conformance corpus, the real host loop starts them in containers (two at a time), and what the fake stored (steps, conclusions, outputs, log lines) is compared with the recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
229
+
-`TestVMRunsAJob` (`internal/sandbox/integration`, run by `scripts/it-vm-userns.sh`) does the first of these in a micro-VM.
230
+
231
+
Not covered anywhere yet, because they need root on a real host: the jailer (uid drop, chroot, cgroups), a rootful Docker daemon or sysbox with the nested `dockerd`, and the firewall of container mode.
232
+
143
233
`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
144
234
145
235
The firewall policy has a packet-level test (`ERMITE_RUNNER_IT_NET=1`, run by `scripts/it-vm-userns.sh`): two guests and an outside host on veth pairs in separate network namespaces, with a forged source and a forged "reply" between guests.
@@ -83,6 +83,83 @@ the host process; the sandbox receives the workflow and its contexts only.
83
| `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |
83
| `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |
84
| `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` |
84
| `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` |
85
85
86
+
## Installing the service
87
+
88
+
`contrib/ermite-runner.service` is a systemd unit. The runner runs as **root**
89
+
on the host (TAP devices, nftables, loop mounts, the jailer, containers); the
90
+
jobs never do: they run as user `runner` in a micro-VM or a container.
sudo -E ermite-runner register --token <token with runner:register>
109
+
sudo systemctl enable --now ermite-runner
110
+
```
111
+
112
+
`systemctl stop` sends SIGTERM: running jobs are cancelled with
113
+
`runner_shutdown`, reported to ermite, and their sandboxes removed before the
114
+
service exits (the unit allows 120 s). A crash or a power cut leaves sandboxes
115
+
behind; the next start reaps them. Only one runner may live on a host (a
116
+
host-wide lock enforces it), because reaping removes everything labelled as the
117
+
runner's own.
118
+
119
+
## The image
120
+
121
+
A job runs in the image named by `ERMITE_RUNNER_IMAGE` (container: an OCI
122
+
reference; vm: the directory with `vmlinux` and `base.ext4`, both built from
123
+
`images/ubuntu-24.04/Dockerfile`). The runner binary is attached to the sandbox
124
+
by the host and is never part of the image. What the executor needs from the
125
+
image:
126
+
127
+
- a user **`runner`** with uid and gid 1001 and the home `/home/runner`; the
128
+
executor starts as root (init needs it to start `dockerd`) and runs every step
129
+
as this user, with HOME, USER and LOGNAME set. Without such a user the steps
130
+
run as root (a test image only);
131
+
- the directories `/home/runner/work` and `/opt/hostedtoolcache` owned by
132
+
`runner` (`RUNNER_TOOL_CACHE`, which the executor reads from its environment
133
+
like the official runner; each job gets `/home/runner/work/<repo>/<repo>` and
134
+
`/home/runner/work/_temp`, cleaned at the start of the job);
135
+
- **`bash`** and **`sh`** (the default shell is `bash -e {0}` when bash exists,
136
+
else `sh -e {0}`; like on hosted runners `shell: bash` adds
137
+
`--noprofile --norc -eo pipefail`), `python3` and a `python` command for
138
+
`shell: python`; `pwsh` is not part of the CI-1 image (a step that asks for it
139
+
fails with `pwsh: command not found`);
140
+
-`dockerd` (and passwordless `sudo` for `runner`) for jobs that use Docker: the
141
+
job gets its own daemon with its data root on the scratch disk;
142
+
- what workflows commonly use: git, curl, jq, build-essential, ca-certificates,
143
+
tzdata, zip/unzip. Full parity with the hosted images (`actions/runner-images`)
144
+
is a later phase.
145
+
146
+
## Job limits
147
+
148
+
Each job gets its own resources, set for the runner as a whole:
149
+
150
+
| Variable | Default | Effect |
151
+
|---|---|---|
152
+
| `ERMITE_RUNNER_CAPACITY` | `1` | jobs in parallel: one sandbox, one uid, one `/30` network and one disk per slot. The runner polls ermite only when a slot is free. |
153
+
| `ERMITE_RUNNER_JOB_CPUS` | `2` | vCPUs of a micro-VM; CPU quota of a container |
154
+
| `ERMITE_RUNNER_JOB_MEMORY` | `4G` | guest memory (`M`, `G`, `T`; at least `256M`); a jailed VMM gets this plus 256 MiB of cgroup headroom |
155
+
| `ERMITE_RUNNER_JOB_DISK` | `64G` | size of the sparse scratch disk (at least `1G`): the overlay upper layer of a VM and the Docker data root of a container. Deleted with the job |
156
+
| `ERMITE_RUNNER_JOB_PIDS` | `32768` | process limit of a container |
DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
217
DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
141
```
218
```
142
219
220
+
**Jobs end to end** (`ERMITE_RUNNER_IT_E2E=1`, `internal/e2e`): the real executor in the real image, started through the container driver. They also run on a workstation with a *rootless* Docker daemon (the test-only option `AllowRootless`; the nested `dockerd` of the image cannot start there, and the jobs do not need it). `mount` and `umount` are replaced by stand-ins, since loop mounts need root.
ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-24.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
225
+
```
226
+
227
+
-`TestContainerRunsAJobAsTheRunnerUser` and `TestContainerCancelStopsTheJobAndOrphans` drive a job over the stream: steps run as `runner` (1001), file ownership, `GITHUB_PATH` and `GITHUB_ENV`, `python`, cancellation by SIGINT, the `always()` step, and the orphan cleanup of "Complete job".
228
+
-`TestChainCorpus` runs the whole chain: a fake `RunnerService` (`internal/host/hosttest`, which checks the offsets, limits and tokens of `runner.proto`) hands out 22 jobs of the ermite-conformance corpus, the real host loop starts them in containers (two at a time), and what the fake stored (steps, conclusions, outputs, log lines) is compared with the recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
229
+
-`TestVMRunsAJob` (`internal/sandbox/integration`, run by `scripts/it-vm-userns.sh`) does the first of these in a micro-VM.
230
+
231
+
Not covered anywhere yet, because they need root on a real host: the jailer (uid drop, chroot, cgroups), a rootful Docker daemon or sysbox with the nested `dockerd`, and the firewall of container mode.
232
+
143
`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
233
`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
144
234
145
The firewall policy has a packet-level test (`ERMITE_RUNNER_IT_NET=1`, run by `scripts/it-vm-userns.sh`): two guests and an outside host on veth pairs in separate network namespaces, with a forged source and a forged "reply" between guests.
235
The firewall policy has a packet-level test (`ERMITE_RUNNER_IT_NET=1`, run by `scripts/it-vm-userns.sh`): two guests and an outside host on veth pairs in separate network namespaces, with a forged source and a forged "reply" between guests.