sandboxes.md Raw History Folder

Sandboxes

Each job runs in its own sandbox. Both modes attach the runner's own executable read-only to the sandbox. It must be a static build (CGO_ENABLED=0). The runner refuses to start otherwise.

Inside the sandbox, ermite-runner init is the entrypoint (PID 1 in a VM). ermite-runner exec runs the job and talks to the host over a framed protobuf stream: vsock in a VM, a Unix socket in a container.

vm

Firecracker runs the job in a microVM. The VM is destroyed when the job ends.
With the jailer (default), needs root. Without it (ERMITE_RUNNER_JAILER_MODE=off), CAP_NET_ADMIN, /dev/kvm and /dev/net/tun are enough, as in the unprivileged container.
Disks:
- The runner executable is the read-only boot root.
- The shared base image is read-only.
- A sparse scratch disk per job (ERMITE_RUNNER_JOB_DISK) holds the writable layer and the Docker data root.
- A read-only disk holds the Node.js externals, when there are any.
The scratch disk and eth0 are rate limited. See job limits.
Network: one TAP device (ertap<slot>) per VM with a /30 from ERMITE_RUNNER_NETWORK. IPv6 is disabled in the guest.
The jailer runs each VM:
- as uid and gid ERMITE_RUNNER_UID_BASE plus the slot;
- in its own chroot under the state directory;
- in a new PID namespace;
- in a cgroup v2 with memory.max (guest memory plus 256 MiB) and pids.max.
The kernel, the base image and the state directory must be on the same filesystem. The kernel and the base image must be world-readable.
The jailed API socket path must fit in 107 bytes. The driver checks it at start.

container

Uses the Docker Engine API (DOCKER_HOST is honoured). The Docker daemon must run on the same host as the runner.
Runtime: sysbox-runc when the daemon lists it. Otherwise a privileged container.
A rootless Docker daemon is refused. A nested dockerd needs a rootful daemon or sysbox.
Limits per job: memory (no swap), CPUs and ERMITE_RUNNER_JOB_PIDS processes.
Disk:
- A sparse ext4 file of ERMITE_RUNNER_JOB_DISK, loop-mounted on the host and bind-mounted at /scratch. This is why the mode needs root.
- The nested dockerd data root, /home/runner/work and /tmp live on it.
- A job that fills its disk gets "no space left on device".
Per job: a bridge network (erbr<slot>) and a Unix socket in a per-job directory of the state directory. The job sees only that directory.
Every Docker object carries the label dev.bnema.ermite-runner.job. Leftovers are reaped by label at start.
The state directory path must keep the socket path under 107 bytes.

Privileged container warning

A privileged container is equivalent to host root.
It is allowed by default. The runner logs a warning at start and at the start of every job.
The isolation reported at registration is container_privileged, container_sysbox or vm.
ERMITE_RUNNER_DENY_PRIVILEGED=1 refuses to start without sysbox.
Network policy is best effort in this mode.

State directory

The state directory and every parent must be owned by root and not writable by group or others.
The jailer runs as root and builds chroots under <state dir>/jail. The driver checks jail itself. It does not check the parents.
A user who can rename or replace a parent can redirect root's file operations.
The default /var/lib/ermite-runner is safe. The unit creates it with StateDirectory.
Do not point ERMITE_RUNNER_STATE_DIR into a home directory or /tmp.

One runner per host

Cleanup removes every process, device, firewall rule and Docker object the runner owns. Two runners on one host would destroy each other's jobs.
The runner takes an exclusive lock on /run/ermite-runner.lock and refuses to start when it is held. ERMITE_RUNNER_LOCK_FILE overrides the path for tests.
The kernel drops the lock when the process dies.
A runner in a container or another mount namespace must share that path. Do not run it that way.

Network policy

Both modes use one nftables table, inet ermite_runner. It is replaced atomically and deleted with the last sandbox. Nothing else is touched.

From a sandbox:

Traffic	Result
DNS to its own gateway address	Allowed. The runner's DNS forwarder answers, using the resolvers of `/etc/resolv.conf`
One TCP port to the gateway, with guest forge access on	Allowed
Other traffic to the host	Denied
RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, documentation, benchmarking, IETF-assigned, multicast and reserved ranges	Denied
IPv6	Denied
Traffic between sandboxes	Denied
The rest	NATed

Source address and destination checks run before any "established" accept. A guest cannot forge another's flow.
The DNS forwarder allows 32 in-flight queries per listener (extra UDP is dropped), 16 TCP connections, and 30 s per TCP connection.
Upstream queries use UDP for 1 s, then TCP on the same resolver (also when UDP is truncated). One query takes at most 6 s. A UDP answer larger than the client accepts (512 bytes without EDNS) is replaced by an empty reply with TC=1, so the client retries over TCP.
Golden rulesets: internal/netpolicy/testdata.

Host side effects:

The runner turns net.ipv4.ip_forward on and leaves it on when it exits.
Forwarding is host-wide. If the machine must not route for other networks, restrict it with your own firewall rules.
Another firewall with a later hook (for example firewalld) can still block guest DNS. Allow DNS from ertap* and erbr* there.

Guest forge access

Off by default. Without a relay target, jobs reach whatever listens on that gateway port, so keep it to hosts where that is acceptable. With ERMITE_RUNNER_GUEST_FORGE_TARGET, jobs reach only the relayed target.

ERMITE_RUNNER_GUEST_FORGE=1 lets sandboxes reach the server the way the runner does. It is for hosts where the server is reachable only through a local route, such as an /etc/hosts entry plus a reverse proxy on a local port. Jobs cannot otherwise run git clone against github.server_url.

The host name and port come from ERMITE_RUNNER_URL. The runner refuses to start when:

the URL host is an IP literal;
the URL has a path;
the name does not resolve on the host;
the URL is http:// and ERMITE_RUNNER_INSECURE_HTTP=1 is not set.

For every sandbox:

The DNS forwarder answers A queries for that host name with the sandbox's gateway address. AAAA gets an empty answer.
The firewall accepts TCP from the sandbox to its gateway on that one port.
github.server_url, github.api_url and github.graphql_url (and the matching GITHUB_* variables) are rewritten to the origin of ERMITE_RUNNER_URL.
The runner logs guest forge access: sandboxes may reach <origin> on the host at start and at every job start.

Security notes:

Without ERMITE_RUNNER_GUEST_FORGE_TARGET, the runner does not proxy. A service must listen on that port on an address the gateway reaches (*, not 127.0.0.1).
With ERMITE_RUNNER_GUEST_FORGE_TARGET=host:port, the runner listens on each sandbox gateway on that port and relays TCP to the target. Use it when the server is only reachable from the runner, for example on a container network. See container.
A shared reverse proxy routes by Host header. A job can reach every app that proxy serves on that port.
Traffic comes from a private address. Anything behind the proxy that trusts private sources trusts job code too.
Jobs see the server over plain HTTP in this setup.

Cleanup

Stop removes everything a job creates: processes, TAP or bridge, firewall rules, files, loop mounts, containers, networks and cgroups. It also runs with a cancelled context.
At start, the runner reaps what a crash left behind.
The network policy and the TAP or bridge are never removed while the VMM or container may still run.
If a process cannot be killed or a container cannot be removed, the slot stays taken until the next reap. The host stops taking jobs on that slot.

source

1# Sandboxes
2
3Each job runs in its own sandbox. Both modes attach the runner's own executable read-only to the sandbox. It must be a static build (`CGO_ENABLED=0`). The runner refuses to start otherwise.
4
5Inside the sandbox, `ermite-runner init` is the entrypoint (PID 1 in a VM). `ermite-runner exec` runs the job and talks to the host over a framed protobuf stream: vsock in a VM, a Unix socket in a container.
6
7## vm
8
9- Firecracker runs the job in a microVM. The VM is destroyed when the job ends.
10- With the jailer (default), needs root. Without it (`ERMITE_RUNNER_JAILER_MODE=off`), `CAP_NET_ADMIN`, `/dev/kvm` and `/dev/net/tun` are enough, as in the [unprivileged container](container.md).
11- Disks:
12  - The runner executable is the read-only boot root.
13  - The shared base image is read-only.
14  - A sparse scratch disk per job (`ERMITE_RUNNER_JOB_DISK`) holds the writable layer and the Docker data root.
15  - A read-only disk holds the Node.js externals, when there are any.
16- The scratch disk and `eth0` are rate limited. See [job limits](configuration.md#job-limits).
17- Network: one TAP device (`ertap<slot>`) per VM with a `/30` from `ERMITE_RUNNER_NETWORK`. IPv6 is disabled in the guest.
18- The jailer runs each VM:
19  - as uid and gid `ERMITE_RUNNER_UID_BASE` plus the slot;
20  - in its own chroot under the state directory;
21  - in a new PID namespace;
22  - in a cgroup v2 with `memory.max` (guest memory plus 256 MiB) and `pids.max`.
23- The kernel, the base image and the state directory must be on the same filesystem. The kernel and the base image must be world-readable.
24- The jailed API socket path must fit in 107 bytes. The driver checks it at start.
25
26## container
27
28- Uses the Docker Engine API (`DOCKER_HOST` is honoured). The Docker daemon must run on the same host as the runner.
29- Runtime: `sysbox-runc` when the daemon lists it. Otherwise a **privileged** container.
30- A rootless Docker daemon is refused. A nested `dockerd` needs a rootful daemon or sysbox.
31- Limits per job: memory (no swap), CPUs and `ERMITE_RUNNER_JOB_PIDS` processes.
32- Disk:
33  - A sparse ext4 file of `ERMITE_RUNNER_JOB_DISK`, loop-mounted on the host and bind-mounted at `/scratch`. This is why the mode needs root.
34  - The nested `dockerd` data root, `/home/runner/work` and `/tmp` live on it.
35  - A job that fills its disk gets "no space left on device".
36- Per job: a bridge network (`erbr<slot>`) and a Unix socket in a per-job directory of the state directory. The job sees only that directory.
37- Every Docker object carries the label `dev.bnema.ermite-runner.job`. Leftovers are reaped by label at start.
38- The state directory path must keep the socket path under 107 bytes.
39
40### Privileged container warning
41
42- A privileged container is equivalent to host root.
43- It is allowed by default. The runner logs a warning at start and at the start of every job.
44- The isolation reported at registration is `container_privileged`, `container_sysbox` or `vm`.
45- `ERMITE_RUNNER_DENY_PRIVILEGED=1` refuses to start without sysbox.
46- Network policy is best effort in this mode.
47
48## State directory
49
50- The state directory and every parent must be owned by root and not writable by group or others.
51- The jailer runs as root and builds chroots under `<state dir>/jail`. The driver checks `jail` itself. It does not check the parents.
52- A user who can rename or replace a parent can redirect root's file operations.
53- The default `/var/lib/ermite-runner` is safe. The unit creates it with `StateDirectory`.
54- Do not point `ERMITE_RUNNER_STATE_DIR` into a home directory or `/tmp`.
55
56## One runner per host
57
58- Cleanup removes every process, device, firewall rule and Docker object the runner owns. Two runners on one host would destroy each other's jobs.
59- The runner takes an exclusive lock on `/run/ermite-runner.lock` and refuses to start when it is held. `ERMITE_RUNNER_LOCK_FILE` overrides the path for tests.
60- The kernel drops the lock when the process dies.
61- A runner in a container or another mount namespace must share that path. Do not run it that way.
62
63## Network policy
64
65Both modes use one nftables table, `inet ermite_runner`. It is replaced atomically and deleted with the last sandbox. Nothing else is touched.
66
67From a sandbox:
68
69| Traffic | Result |
70| --- | --- |
71| DNS to its own gateway address | Allowed. The runner's DNS forwarder answers, using the resolvers of `/etc/resolv.conf` |
72| One TCP port to the gateway, with guest forge access on | Allowed |
73| Other traffic to the host | Denied |
74| RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, documentation, benchmarking, IETF-assigned, multicast and reserved ranges | Denied |
75| IPv6 | Denied |
76| Traffic between sandboxes | Denied |
77| The rest | NATed |
78
79- Source address and destination checks run before any "established" accept. A guest cannot forge another's flow.
80- The DNS forwarder allows 32 in-flight queries per listener (extra UDP is dropped), 16 TCP connections, and 30 s per TCP connection.
81- Upstream queries use UDP for 1 s, then TCP on the same resolver (also when UDP is truncated). One query takes at most 6 s. A UDP answer larger than the client accepts (512 bytes without EDNS) is replaced by an empty reply with TC=1, so the client retries over TCP.
82- Golden rulesets: `internal/netpolicy/testdata`.
83
84Host side effects:
85
86- The runner turns `net.ipv4.ip_forward` on and leaves it on when it exits.
87- Forwarding is host-wide. If the machine must not route for other networks, restrict it with your own firewall rules.
88- Another firewall with a later hook (for example firewalld) can still block guest DNS. Allow DNS from `ertap*` and `erbr*` there.
89
90## Guest forge access
91
92Off by default. Without a relay target, jobs reach whatever listens on that gateway port, so keep it to hosts where that is acceptable. With `ERMITE_RUNNER_GUEST_FORGE_TARGET`, jobs reach only the relayed target.
93
94`ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the server the way the runner does. It is for hosts where the server is reachable only through a local route, such as an `/etc/hosts` entry plus a reverse proxy on a local port. Jobs cannot otherwise run `git clone` against `github.server_url`.
95
96The host name and port come from `ERMITE_RUNNER_URL`. The runner refuses to start when:
97
98- the URL host is an IP literal;
99- the URL has a path;
100- the name does not resolve on the host;
101- the URL is `http://` and `ERMITE_RUNNER_INSECURE_HTTP=1` is not set.
102
103For every sandbox:
104
105- The DNS forwarder answers `A` queries for that host name with the sandbox's gateway address. `AAAA` gets an empty answer.
106- The firewall accepts TCP from the sandbox to its gateway on that one port.
107- `github.server_url`, `github.api_url` and `github.graphql_url` (and the matching `GITHUB_*` variables) are rewritten to the origin of `ERMITE_RUNNER_URL`.
108- The runner logs `guest forge access: sandboxes may reach <origin> on the host` at start and at every job start.
109
110Security notes:
111
112- Without `ERMITE_RUNNER_GUEST_FORGE_TARGET`, the runner does not proxy. A service must listen on that port on an address the gateway reaches (`*`, not `127.0.0.1`).
113- With `ERMITE_RUNNER_GUEST_FORGE_TARGET=host:port`, the runner listens on each sandbox gateway on that port and relays TCP to the target. Use it when the server is only reachable from the runner, for example on a container network. See [container](container.md#reaching-the-server).
114- A shared reverse proxy routes by `Host` header. A job can reach every app that proxy serves on that port.
115- Traffic comes from a private address. Anything behind the proxy that trusts private sources trusts job code too.
116- Jobs see the server over plain HTTP in this setup.
117
118## Cleanup
119
120- `Stop` removes everything a job creates: processes, TAP or bridge, firewall rules, files, loop mounts, containers, networks and cgroups. It also runs with a cancelled context.
121- At start, the runner reaps what a crash left behind.
122- The network policy and the TAP or bridge are never removed while the VMM or container may still run.
123- If a process cannot be killed or a container cannot be removed, the slot stays taken until the next reap. The host stops taking jobs on that slot.

sandboxes.md RawHistoryFolder