Merge pull request 'feat(sandbox): micro-VM and container job sandboxes' (#1) from feat/sandbox into main

commit: 9b7473e4042aeb3c0d3627bdf6132c58f9e375af
author: bnema · 10d ago · 2026-10-12 18:16 UTC
merge of: 885c03f + ca0f9b5 · diff against the first parent
signature: SSH UnverifiedSHA256:VOCA5lYARHcbKDZ7GhTV45VmcjWnJ6ZQIBazj17LivA The key is unknown

70 files · +12120 -6

M .forgejo/workflows/ci.yml +7 -0


@@ -38,4 +38,11 @@ jobs:
38	38		go mod tidy
39	39		git diff --exit-code -- go.mod go.sum
40	40		- run: go vet ./...
	41	+	- name: Protobuf lint and drift
	42	+	# buf and protoc-gen-go are pinned as `tool` directives in go.mod: the
	43	+	# generated code is reproducible and must match the committed one.
	44	+	run: \|
	45	+	go tool buf lint
	46	+	go generate ./internal/execstream
	47	+	git diff --exit-code -- internal/execstream/execpb
41	48		- run: go test -race ./...

M README.md +106 -2


@@ -5,9 +5,113 @@ The CI runner of [ermite](https://git.bnema.dev/bnema/ermite). It runs workflows
5	5		- `vm`: a Firecracker micro-VM per job, destroyed at the end. Needs `/dev/kvm`.
6	6		- `container`: a container per job with its own Docker daemon, for hosts without KVM. Uses the sysbox runtime when available, else a privileged container.
7	7
8		-	`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when KVM is usable, else `container`.
	8	+	`ERMITE_RUNNER_ISOLATION=auto` (default) picks `vm` when `/dev/kvm` opens read-write and Firecracker, the guest kernel and the base image exist, else `container`. The chosen mode and the reasons are logged at start.
9	9
10		-	Status: early development.
	10	+	Status: early development. This tree has the job sandboxes (phase D of the CI-1 plan) with a stub executor; the host loop that fetches jobs from ermite comes later.
	11	+
	12	+	## Sandboxes
	13	+
	14	+	Both modes run the same executable, the runner's own, attached read-only to the sandbox (never baked into the image, so a runner upgrade needs no image rebuild): `ermite-runner init` is the entrypoint (PID 1 of a VM), then `ermite-runner exec` connects back to the host over a framed protobuf stream (`internal/execstream/exec.proto`; vsock in a VM, a Unix socket in a container). The executable must be static: build with `CGO_ENABLED=0`.
	15	+
	16	+	### vm
	17	+
	18	+	- Firecracker is driven through its HTTP API on a unix socket with `net/http`, not through `firecracker-go-sdk` (its last release, v1.0.0 from 2022, predates Firecracker 1.17 and pulls CNI, go-openapi and netlink for a handful of PUT requests).
	19	+	- Disks: the runner executable (read-only boot root, ext4 built with `mkfs.ext4 -d`, cached by the executable's hash), the shared read-only base image, and a per-job sparse scratch disk (`ERMITE_RUNNER_JOB_DISK`). The guest init builds an overlayfs root (lower = base, upper = scratch) and keeps dockerd's data root on the scratch disk. The scratch disk and `eth0` are rate limited (defaults 400 MiB/s and 20000 IOPS, 1 Gbit/s each way; `ERMITE_RUNNER_JOB_DISK_BANDWIDTH` in MiB/s, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` in Mbit/s; 0 disables).
	20	+	- The jailer runs each VM as uid/gid `ERMITE_RUNNER_UID_BASE` + slot in its own chroot under the state directory, in a new PID namespace and in a cgroup v2 with `memory.max` (guest memory + 256 MiB for the VMM) and `pids.max`. The kernel, the base image and the state directory must be on the same filesystem (files are hard-linked into the jail) and the kernel and base image must be world-readable.
	21	+	- One TAP (`ertap<slot>`) per VM with a /30 from `ERMITE_RUNNER_NETWORK` (default `10.200.0.0/16`), vsock for the executor stream, IPv6 disabled in the guest.
	22	+	- Needs root, or `CAP_NET_ADMIN` and the rights the jailer requires (the plan runs the host process as root under systemd).
	23	+
	24	+	### container
	25	+
	26	+	- Docker Engine API via the maintained client `github.com/moby/moby/client` (`DOCKER_HOST` honoured). Runtime `sysbox-runc` when the daemon lists it, else a privileged container, which is host-root equivalent. That is allowed by default (ADR 0025) but never silent: the runner logs a warning at start and at the start of every job (`privileged container: host-root equivalent, network policy best effort`), `Driver.Isolation()` reports `container_privileged` (or `container_sysbox`, `vm`) for the registration with ermite, and `ERMITE_RUNNER_DENY_PRIVILEGED=1` refuses to start without sysbox.
	27	+	- A rootless Docker daemon is refused with a clear error (a nested dockerd needs a rootful daemon or sysbox).
	28	+	- Limits per job: memory with swap equal to memory (no swap), `ERMITE_RUNNER_JOB_PIDS` processes (default 32768), CPUs.
	29	+	- Disk: the job's writable space is enforced like in `vm`: a sparse ext4 file of `ERMITE_RUNNER_JOB_DISK`, loop-mounted on the host (so this mode needs root, like the rest of it) and bind-mounted at `/scratch`. The nested dockerd's data root, `/home/runner/work` and `/tmp` are directories of that disk; the container's own writable layer stays small. A job that fills its disk gets "no space left on device" instead of filling the host.
	30	+	- Per job: a bridge network (`erbr<slot>`), the scratch disk, a Unix socket in a bind-mounted per-job directory of the state directory (the job sees only that directory, not the scratch file). Every Docker object carries the label `dev.bnema.ermite-runner.job`; leftovers are reaped by label at start. The state directory path must keep the socket path under 107 bytes.
	31	+	- The Docker daemon must run on the same host as the runner (host paths are bind-mounted).
	32	+
	33	+	### One runner per host
	34	+
	35	+	Reaping removes every process, device, firewall rule and Docker object the runner owns, so two runners on one host would destroy each other's jobs. Both modes take an exclusive lock at a fixed host path, `/run/ermite-runner.lock` (`ERMITE_RUNNER_LOCK_FILE` overrides it, for tests), and refuse to start when it is held, whatever their state directories or isolation modes. The kernel drops the lock when the process dies. A runner in a container or in another mount namespace must share that path (or it is not protected): do not run it so.
	36	+
	37	+	### Network policy (both modes)
	38	+
	39	+	One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address, where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
	40	+
	41	+	Side effect on the host: the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
	42	+
	43	+	### Cancellation
	44	+
	45	+	The host sends a `Cancel{reason}` frame (`execstream.Drive` takes a cancel channel, not a context). The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later, with the timers stopped as soon as the shell exits and nothing signalled afterwards (the group id could be reused). Background children that keep the output pipes open are killed when the shell exits, after a short drain. Init forwards SIGTERM and SIGINT to the executor.
	46	+
	47	+	### Cleanup
	48	+
	49	+	Everything a job creates (process, TAP or bridge, nftables rules, files, loop mounts, containers, networks, cgroups) is removed by `Stop`, which also runs with a cancelled context, aborts a `Start` in progress, and by `Reap` at start for what a crash left behind. A VM leaves a state record (`<state dir>/vms/<id>.json`) before its first resource. The network policy and the TAP or bridge are never removed while the VMM or the container may still be running: when a process cannot be killed or a container cannot be removed, `Stop` returns an error wrapping `sandbox.ErrLeaked`, the sandbox's slot stays taken, and `Driver.Reap` (which returns `sandbox.ErrBusy` while a sandbox is running) finishes the job and frees the slot. The host should stop taking jobs on a leaked slot until then. `Sandbox.ID()` names a sandbox's files and Docker objects.
	50	+
	51	+	## Configuration (sandbox part)
	52	+
	53	+	\| Variable \| Default \| \|
	54	+	\|---\|---\|---\|
	55	+	\| `ERMITE_RUNNER_ISOLATION` \| `auto` \| `auto`, `vm` or `container` \|
	56	+	\| `ERMITE_RUNNER_STATE_DIR` \| `/var/lib/ermite-runner` \| absolute path \|
	57	+	\| `ERMITE_RUNNER_CAPACITY` \| `1` \| parallel jobs (slots) \|
	58	+	\| `ERMITE_RUNNER_IMAGE` \| \| VM: image directory (`vmlinux`, `base.ext4`; default `<state dir>/image`); container: OCI reference (default `ermite-runner-ubuntu-24.04:latest`) \|
	59	+	\| `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` \| `2` / `4G` / `64G` \| per job; sizes accept `M`, `G`, `T`. A warning is logged at start when the state directory has less free space than capacity x disk (disks are sparse: it is only a warning) \|
	60	+	\| `ERMITE_RUNNER_JOB_PIDS` \| `32768` \| container process limit \|
	61	+	\| `ERMITE_RUNNER_DENY_PRIVILEGED` \| `0` \| `1` refuses container mode without sysbox \|
	62	+	\| `ERMITE_RUNNER_LOCK_FILE` \| `/run/ermite-runner.lock` \| host-wide single-instance lock (tests only) \|
	63	+	\| `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` \| `400`, `20000`, `1000` \| VM rate limits in MiB/s, IOPS and Mbit/s; 0 disables \|
	64	+	\| `ERMITE_RUNNER_UID_BASE` \| `900000` \| jailer uid pool start \|
	65	+	\| `ERMITE_RUNNER_NETWORK` \| `10.200.0.0/16` \| pool of /30 subnets, one per slot \|
	66	+	\| `ERMITE_RUNNER_FIRECRACKER`, `_JAILER`, `_KERNEL` \| from `PATH` \| binaries and kernel overrides \|
	67	+	\| `ERMITE_RUNNER_JAILER_MODE` \| `on` \| `off` only for development and tests \|
	68	+
	69	+	## Images and kernel
	70	+
	71	+	```sh
	72	+	scripts/build-kernel.sh tmp/image # downloads Linux 6.18 LTS from kernel.org, builds vmlinux (needs bc, flex, bison, pahole, libelf, openssl headers)
	73	+	scripts/build-image.sh tmp/image # builds the OCI image with docker or podman, exports tmp/image/base.ext4 (no root needed)
	74	+	```
	75	+
	76	+	`scripts/build-image.sh` tags the OCI image `ermite-runner-ubuntu-24.04:latest`, the default of `ERMITE_RUNNER_IMAGE` in container mode, and needs e2fsprogs >= 1.47.1 (the runner itself needs >= 1.43 and checks it at start).
	77	+
	78	+	The kernel configuration is `images/kernel/config-6.18` (+ per-architecture fragments), checked after `olddefconfig`: the build fails if a listed option did not survive. arm64 builds (`ARCH_NAME=arm64`) but is not tested in CI-1.
	79	+
	80	+	## Tests
	81	+
	82	+	```sh
	83	+	GOWORK=off go vet ./... && GOWORK=off go test -race ./...
	84	+	```
	85	+
	86	+	Unit tests need nothing special (the drivers are tested with a fake Firecracker and a fake Docker engine). Integration tests are opt-in and use the real stub executor: output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
	87	+
	88	+	VM (`ERMITE_RUNNER_IT_VM=1`): needs `/dev/kvm`, Firecracker, the kernel and base image from the scripts above.
	89	+
	90	+	```sh
	91	+	# Unprivileged: inside a user + network namespace (TAP, nftables and DNS on the TAP address work there).
	92	+	# Jailer off: it needs real root. Everything else is the production path.
	93	+	FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
	94	+
	95	+	# Privileged, with the jailer (the production configuration):
	96	+	sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
	97	+	ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
	98	+	"$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
	99	+	```
	100	+
	101	+	Container (`ERMITE_RUNNER_IT_CONTAINER=1`): needs a rootful Docker daemon, root (loop mounts, `CAP_NET_ADMIN`) and the OCI image (`scripts/build-image.sh`).
	102	+
	103	+	```sh
	104	+	sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-24.04:latest \
	105	+	DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
	106	+	```
	107	+
	108	+	`ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container` exercises the real Docker Engine calls (labels, networks, volumes, reaping) against any daemon, including a rootless one; it does not test isolation.
	109	+
	110	+	The firewall policy has a packet-level test (`ERMITE_RUNNER_IT_NET=1`, run by `scripts/it-vm-userns.sh`): two guests and an outside host on veth pairs in separate network namespaces, with a forged source and a forged "reply" between guests.
	111	+
	112	+	Protobuf: `go generate ./internal/execstream` (buf and protoc-gen-go are pinned as `tool` directives in go.mod); CI lints the proto and fails when the committed generated code differs.
	113	+
	114	+	`ermite-runner sandbox-check [script]` runs one stub job through the selected mode and prints its output: the smoke test of an installation.
11	115
12	116		## License
13	117

A buf.gen.yaml +5 -0

@@ -0,0 +1,5 @@
	1	+	version: v2
	2	+	plugins:
	3	+	- local: [go, tool, protoc-gen-go]
	4	+	out: ../..
	5	+	opt: module=bnema.dev/ermite-runner

A buf.yaml +12 -0

@@ -0,0 +1,12 @@
	1	+	version: v2
	2	+	modules:
	3	+	- path: internal/execstream
	4	+	lint:
	5	+	use:
	6	+	- STANDARD
	7	+	except:
	8	+	# The proto is internal to this module and lives next to its Go package.
	9	+	- PACKAGE_DIRECTORY_MATCH
	10	+	breaking:
	11	+	use:
	12	+	- FILE

M go.mod +103 -0

@@ -1,3 +1,106 @@
1	1		module bnema.dev/ermite-runner
2	2
3	3		go 1.27.1
	4	+
	5	+	require (
	6	+	github.com/bnema/zerowrap v1.4.1
	7	+	github.com/containerd/errdefs v1.0.0
	8	+	github.com/moby/moby/api v1.56.1
	9	+	github.com/moby/moby/client v0.6.2
	10	+	golang.org/x/sys v0.49.0
	11	+	google.golang.org/protobuf v1.36.12
	12	+	)
	13	+
	14	+	require (
	15	+	buf.build/gen/go/bufbuild/bufplugin/protocolbuffers/go v1.36.12-20260722160903-4d94f3df3a7b.2 // indirect
	16	+	buf.build/gen/go/bufbuild/protodescriptor/protocolbuffers/go v1.36.12-20250109164928-1da0de137947.2 // indirect
	17	+	buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.12-20260825204119-511051f7f437.2 // indirect
	18	+	buf.build/gen/go/bufbuild/registry/connectrpc/go v1.20.0-20260831211851-b4432e12a6e7.1 // indirect
	19	+	buf.build/gen/go/bufbuild/registry/protocolbuffers/go v1.36.12-20260831211851-b4432e12a6e7.2 // indirect
	20	+	buf.build/gen/go/pluginrpc/pluginrpc/protocolbuffers/go v1.36.12-20241007202033-cf42259fcbfc.2 // indirect
	21	+	buf.build/go/app v0.2.1-0.20260824172350-b0e76892c61a // indirect
	22	+	buf.build/go/bufplugin v0.10.0 // indirect
	23	+	buf.build/go/bufprivateusage v0.1.0 // indirect
	24	+	buf.build/go/interrupt v1.1.0 // indirect
	25	+	buf.build/go/protovalidate v1.4.0 // indirect
	26	+	buf.build/go/protoyaml v0.7.0 // indirect
	27	+	buf.build/go/spdx v0.2.0 // indirect
	28	+	buf.build/go/standard v0.1.1-0.20260325175353-2b287e071df5 // indirect
	29	+	cel.dev/cel-go v0.32.0 // indirect
	30	+	cel.dev/expr v0.25.3 // indirect
	31	+	connectrpc.com/connect v1.20.0 // indirect
	32	+	connectrpc.com/otelconnect v0.9.0 // indirect
	33	+	github.com/Microsoft/go-winio v0.6.2 // indirect
	34	+	github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
	35	+	github.com/bufbuild/buf v1.73.0 // indirect
	36	+	github.com/bufbuild/protocompile v0.14.2-0.20260910151042-7436f7c76201 // indirect
	37	+	github.com/bufbuild/protoplugin v0.0.0-20260414125817-25d1d281b46b // indirect
	38	+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
	39	+	github.com/cli/browser v1.3.0 // indirect
	40	+	github.com/containerd/errdefs/pkg v0.3.0 // indirect
	41	+	github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
	42	+	github.com/distribution/reference v0.6.0 // indirect
	43	+	github.com/docker/cli v29.8.0+incompatible // indirect
	44	+	github.com/docker/docker-credential-helpers v0.9.9 // indirect
	45	+	github.com/docker/go-connections v0.8.2 // indirect
	46	+	github.com/docker/go-units v0.5.0 // indirect
	47	+	github.com/felixge/httpsnoop v1.1.0 // indirect
	48	+	github.com/go-logr/logr v1.4.4 // indirect
	49	+	github.com/go-logr/stdr v1.2.2 // indirect
	50	+	github.com/gofrs/flock v0.13.1 // indirect
	51	+	github.com/google/go-containerregistry v0.22.1 // indirect
	52	+	github.com/google/uuid v1.6.0 // indirect
	53	+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	54	+	github.com/jdx/go-netrc v1.0.0 // indirect
	55	+	github.com/klauspost/compress v1.20.0 // indirect
	56	+	github.com/klauspost/pgzip v1.2.6 // indirect
	57	+	github.com/mattn/go-colorable v0.1.15 // indirect
	58	+	github.com/mattn/go-isatty v0.0.24 // indirect
	59	+	github.com/moby/docker-image-spec v1.3.1 // indirect
	60	+	github.com/opencontainers/go-digest v1.0.0 // indirect
	61	+	github.com/opencontainers/image-spec v1.1.1 // indirect
	62	+	github.com/petermattis/goid v0.0.0-20260820044319-269ab09b5261 // indirect
	63	+	github.com/quic-go/qpack v0.6.0 // indirect
	64	+	github.com/quic-go/quic-go v0.62.0 // indirect
	65	+	github.com/rivo/uniseg v0.4.7 // indirect
	66	+	github.com/rs/cors v1.11.1 // indirect
	67	+	github.com/rs/zerolog v1.35.1 // indirect
	68	+	github.com/russross/blackfriday/v2 v2.1.0 // indirect
	69	+	github.com/segmentio/asm v1.2.1 // indirect
	70	+	github.com/segmentio/encoding v0.5.4 // indirect
	71	+	github.com/sirupsen/logrus v1.10.2 // indirect
	72	+	github.com/spf13/cobra v1.10.2 // indirect
	73	+	github.com/spf13/pflag v1.0.10 // indirect
	74	+	github.com/tetratelabs/wazero v1.12.0 // indirect
	75	+	github.com/tidwall/btree v1.8.1 // indirect
	76	+	go.lsp.dev/jsonrpc2 v0.10.0 // indirect
	77	+	go.lsp.dev/pkg v0.0.0-20210717090340-384b27a52fb2 // indirect
	78	+	go.lsp.dev/protocol v0.12.0 // indirect
	79	+	go.lsp.dev/uri v0.3.0 // indirect
	80	+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
	81	+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 // indirect
	82	+	go.opentelemetry.io/otel v1.46.0 // indirect
	83	+	go.opentelemetry.io/otel/metric v1.46.0 // indirect
	84	+	go.opentelemetry.io/otel/trace v1.46.0 // indirect
	85	+	go.uber.org/multierr v1.11.0 // indirect
	86	+	go.uber.org/zap v1.28.0 // indirect
	87	+	go.yaml.in/yaml/v3 v3.0.5 // indirect
	88	+	golang.org/x/crypto v0.56.0 // indirect
	89	+	golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect
	90	+	golang.org/x/mod v0.40.0 // indirect
	91	+	golang.org/x/net v0.58.0 // indirect
	92	+	golang.org/x/sync v0.23.0 // indirect
	93	+	golang.org/x/term v0.45.0 // indirect
	94	+	golang.org/x/text v0.41.0 // indirect
	95	+	google.golang.org/genproto/googleapis/api v0.0.0-20260904194346-d0f1323225a4 // indirect
	96	+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260904194346-d0f1323225a4 // indirect
	97	+	gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
	98	+	gopkg.in/yaml.v3 v3.0.1 // indirect
	99	+	mvdan.cc/xurls/v2 v2.6.0 // indirect
	100	+	pluginrpc.com/pluginrpc v0.5.0 // indirect
	101	+	)
	102	+
	103	+	tool (
	104	+	github.com/bufbuild/buf/cmd/buf
	105	+	google.golang.org/protobuf/cmd/protoc-gen-go
	106	+	)

A go.sum +233 -0 generated file: collapsed

load diff

A images/kernel/config-6.18 +163 -0

@@ -0,0 +1,163 @@
	1	+	# Minimal Linux 6.18 guest configuration for Firecracker micro-VMs running
	2	+	# Docker (ermite-runner). This is a configuration fragment: scripts/build-kernel.sh
	3	+	# merges it over `make tinyconfig` and resolves dependencies with olddefconfig,
	4	+	# then fails if an option listed here did not survive. Per-architecture options
	5	+	# are in config-6.18.x86_64 and config-6.18.arm64, merged after this file.
	6	+
	7	+	# --- Core ---
	8	+	CONFIG_64BIT=y
	9	+	CONFIG_SMP=y
	10	+	CONFIG_NR_CPUS=64
	11	+	CONFIG_EXPERT=y
	12	+	CONFIG_PRINTK=y
	13	+	CONFIG_PRINTK_TIME=y
	14	+	CONFIG_KALLSYMS=y
	15	+	CONFIG_BUG=y
	16	+	CONFIG_TTY=y
	17	+	CONFIG_UNIX98_PTYS=y
	18	+	CONFIG_MULTIUSER=y
	19	+	CONFIG_FUTEX=y
	20	+	CONFIG_EPOLL=y
	21	+	CONFIG_EVENTFD=y
	22	+	CONFIG_SIGNALFD=y
	23	+	CONFIG_TIMERFD=y
	24	+	CONFIG_AIO=y
	25	+	CONFIG_FHANDLE=y
	26	+	CONFIG_FILE_LOCKING=y
	27	+	CONFIG_ADVISE_SYSCALLS=y
	28	+	CONFIG_MEMBARRIER=y
	29	+	CONFIG_POSIX_TIMERS=y
	30	+	CONFIG_HIGH_RES_TIMERS=y
	31	+	CONFIG_NO_HZ_IDLE=y
	32	+	CONFIG_BINFMT_ELF=y
	33	+	CONFIG_BINFMT_SCRIPT=y
	34	+	CONFIG_SYSVIPC=y
	35	+	CONFIG_POSIX_MQUEUE=y
	36	+	CONFIG_KEYS=y
	37	+	CONFIG_SECCOMP=y
	38	+	CONFIG_SECCOMP_FILTER=y
	39	+	CONFIG_BPF_SYSCALL=y
	40	+	CONFIG_BPF_JIT=y
	41	+	CONFIG_INOTIFY_USER=y
	42	+	CONFIG_FANOTIFY=y
	43	+	CONFIG_SHMEM=y
	44	+	CONFIG_MODULES=n
	45	+
	46	+	# --- Guest basics (x86-64 only options live in config-6.18.x86_64) ---
	47	+	CONFIG_SERIAL_8250=y
	48	+	CONFIG_SERIAL_8250_CONSOLE=y
	49	+	CONFIG_HW_RANDOM=y
	50	+	CONFIG_HW_RANDOM_VIRTIO=y
	51	+
	52	+	# --- virtio (Firecracker uses virtio-mmio) ---
	53	+	CONFIG_VIRTIO_MENU=y
	54	+	CONFIG_VIRTIO=y
	55	+	CONFIG_VIRTIO_MMIO=y
	56	+	CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
	57	+	CONFIG_VIRTIO_BLK=y
	58	+	CONFIG_VIRTIO_NET=y
	59	+	CONFIG_VSOCKETS=y
	60	+	CONFIG_VIRTIO_VSOCKETS=y
	61	+
	62	+	# --- Block, filesystems ---
	63	+	CONFIG_BLOCK=y
	64	+	CONFIG_BLK_DEV=y
	65	+	CONFIG_PROC_FS=y
	66	+	CONFIG_PROC_SYSCTL=y
	67	+	CONFIG_SYSFS=y
	68	+	CONFIG_TMPFS=y
	69	+	CONFIG_TMPFS_POSIX_ACL=y
	70	+	CONFIG_TMPFS_XATTR=y
	71	+	CONFIG_DEVTMPFS=y
	72	+	CONFIG_DEVTMPFS_MOUNT=y
	73	+	CONFIG_SYSCTL=y
	74	+	CONFIG_EXT4_FS=y
	75	+	CONFIG_EXT4_FS_POSIX_ACL=y
	76	+	CONFIG_EXT4_FS_SECURITY=y
	77	+	CONFIG_OVERLAY_FS=y
	78	+	CONFIG_FUSE_FS=y
	79	+	CONFIG_CRYPTO=y
	80	+	CONFIG_CRYPTO_CRC32C=y
	81	+
	82	+	# --- Namespaces and cgroups v2 (Docker, runc) ---
	83	+	CONFIG_NAMESPACES=y
	84	+	CONFIG_UTS_NS=y
	85	+	CONFIG_IPC_NS=y
	86	+	CONFIG_USER_NS=y
	87	+	CONFIG_PID_NS=y
	88	+	CONFIG_NET_NS=y
	89	+	CONFIG_TIME_NS=y
	90	+	CONFIG_CGROUPS=y
	91	+	CONFIG_CGROUP_PIDS=y
	92	+	CONFIG_CGROUP_SCHED=y
	93	+	CONFIG_FAIR_GROUP_SCHED=y
	94	+	CONFIG_CFS_BANDWIDTH=y
	95	+	CONFIG_CPUSETS=y
	96	+	CONFIG_MEMCG=y
	97	+	CONFIG_BLK_CGROUP=y
	98	+	CONFIG_CGROUP_DEVICE=y
	99	+	CONFIG_CGROUP_FREEZER=y
	100	+	CONFIG_CGROUP_CPUACCT=y
	101	+	CONFIG_CGROUP_BPF=y
	102	+	CONFIG_HUGETLBFS=y
	103	+	CONFIG_HUGETLB_PAGE=y
	104	+	CONFIG_CGROUP_HUGETLB=y
	105	+
	106	+	# --- Network ---
	107	+	CONFIG_NET=y
	108	+	CONFIG_NETDEVICES=y
	109	+	CONFIG_NET_CORE=y
	110	+	CONFIG_INET=y
	111	+	CONFIG_IP_PNP=y
	112	+	CONFIG_IPV6=y
	113	+	CONFIG_UNIX=y
	114	+	CONFIG_PACKET=y
	115	+	CONFIG_TUN=y
	116	+	CONFIG_VETH=y
	117	+	CONFIG_BRIDGE=y
	118	+	CONFIG_BRIDGE_NETFILTER=y
	119	+	CONFIG_VLAN_8021Q=y
	120	+	CONFIG_INET_DIAG=y
	121	+	CONFIG_INET_UDP_DIAG=y
	122	+	CONFIG_UNIX_DIAG=y
	123	+	CONFIG_NETLINK_DIAG=y
	124	+	CONFIG_SYN_COOKIES=y
	125	+
	126	+	# --- netfilter: nftables, iptables compatibility, NAT ---
	127	+	CONFIG_NETFILTER=y
	128	+	CONFIG_NETFILTER_ADVANCED=y
	129	+	CONFIG_NETFILTER_NETLINK=y
	130	+	CONFIG_NF_CONNTRACK=y
	131	+	CONFIG_NF_NAT=y
	132	+	CONFIG_NF_TABLES=y
	133	+	CONFIG_NF_TABLES_INET=y
	134	+	CONFIG_NF_TABLES_IPV4=y
	135	+	CONFIG_NFT_CT=y
	136	+	CONFIG_NFT_NAT=y
	137	+	CONFIG_NFT_MASQ=y
	138	+	CONFIG_NFT_REJECT=y
	139	+	CONFIG_NFT_LIMIT=y
	140	+	CONFIG_NFT_LOG=y
	141	+	CONFIG_NFT_COMPAT=y
	142	+	CONFIG_NETFILTER_XTABLES=y
	143	+	CONFIG_NETFILTER_XTABLES_LEGACY=y
	144	+	CONFIG_NETFILTER_XT_NAT=y
	145	+	CONFIG_NETFILTER_XT_MARK=y
	146	+	CONFIG_NETFILTER_XT_CONNMARK=y
	147	+	CONFIG_NETFILTER_XT_TARGET_MASQUERADE=y
	148	+	CONFIG_NETFILTER_XT_TARGET_MARK=y
	149	+	CONFIG_NETFILTER_XT_TARGET_CHECKSUM=y
	150	+	CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
	151	+	CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
	152	+	CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
	153	+	CONFIG_NETFILTER_XT_MATCH_COMMENT=y
	154	+	CONFIG_NETFILTER_XT_MATCH_MARK=y
	155	+	CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
	156	+	CONFIG_IP_NF_IPTABLES_LEGACY=y
	157	+	CONFIG_IP_NF_IPTABLES=y
	158	+	CONFIG_IP_NF_FILTER=y
	159	+	CONFIG_IP_NF_MANGLE=y
	160	+	CONFIG_IP_NF_RAW=y
	161	+	CONFIG_IP_NF_NAT=y
	162	+	CONFIG_IP_NF_TARGET_MASQUERADE=y
	163	+	CONFIG_IP_NF_TARGET_REJECT=y

A images/kernel/config-6.18.arm64 +11 -0

@@ -0,0 +1,11 @@
	1	+	# arm64 specific guest options, merged after config-6.18. Not exercised in CI-1
	2	+	# (no arm64 host); kept so that the build is reproducible there.
	3	+	CONFIG_ARM64=y
	4	+	CONFIG_ARM64_VA_BITS_48=y
	5	+	CONFIG_ARM_GIC_V3=y
	6	+	CONFIG_ARM_PSCI_FW=y
	7	+	CONFIG_ARM_ARCH_TIMER=y
	8	+	CONFIG_SERIAL_AMBA_PL011=y
	9	+	CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
	10	+	CONFIG_RTC_CLASS=y
	11	+	CONFIG_RTC_DRV_PL031=y

A images/kernel/config-6.18.x86_64 +11 -0

@@ -0,0 +1,11 @@
	1	+	# x86-64 specific guest options, merged after config-6.18.
	2	+	CONFIG_HYPERVISOR_GUEST=y
	3	+	CONFIG_PARAVIRT=y
	4	+	CONFIG_KVM_GUEST=y
	5	+	CONFIG_IA32_EMULATION=y
	6	+	CONFIG_SERIO=y
	7	+	CONFIG_SERIO_I8042=y
	8	+	CONFIG_ACPI=y
	9	+	CONFIG_X86_MPPARSE=y
	10	+	CONFIG_X86_LOCAL_APIC=y
	11	+	CONFIG_X86_IO_APIC=y

A images/ubuntu-24.04/Dockerfile +43 -0

@@ -0,0 +1,43 @@
	1	+	# SPDX-License-Identifier: MIT
	2	+	#
	3	+	# ermite-runner-ubuntu-24.04: the minimal CI-1 runner image. The same
	4	+	# filesystem is used as an OCI image (container isolation) and exported to an
	5	+	# ext4 base image (vm isolation) by scripts/build-image.sh. It holds the OS
	6	+	# only: the runner executable is attached by the host at start (never baked
	7	+	# in), so a runner upgrade does not need an image rebuild.
	8	+	FROM ubuntu:24.04
	9	+
	10	+	ARG DEBIAN_FRONTEND=noninteractive
	11	+
	12	+	RUN apt-get update \
	13	+	&& apt-get install -y --no-install-recommends \
	14	+	bash build-essential ca-certificates curl e2fsprogs git gnupg iproute2 \
	15	+	iptables jq kmod procps python3 sudo tzdata unzip xz-utils zip \
	16	+	&& rm -rf /var/lib/apt/lists/*
	17	+
	18	+	# Docker Engine from Docker's apt repository (the job gets its own daemon).
	19	+	RUN install -m 0755 -d /etc/apt/keyrings \
	20	+	&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \
	21	+	&& chmod a+r /etc/apt/keyrings/docker.asc \
	22	+	&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" \
	23	+	> /etc/apt/sources.list.d/docker.list \
	24	+	&& apt-get update \
	25	+	&& apt-get install -y --no-install-recommends \
	26	+	docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin \
	27	+	&& rm -rf /var/lib/apt/lists/*
	28	+
	29	+	# Like GitHub-hosted runners: user "runner" (uid 1001), passwordless sudo,
	30	+	# member of "docker", with the job layout directories.
	31	+	RUN useradd --create-home --uid 1001 --shell /bin/bash runner \
	32	+	&& usermod -aG docker runner \
	33	+	&& echo 'runner ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/runner \
	34	+	&& chmod 0440 /etc/sudoers.d/runner \
	35	+	&& mkdir -p /home/runner/work /opt/hostedtoolcache /opt/ermite \
	36	+	&& chown -R runner:runner /home/runner /opt/hostedtoolcache
	37	+
	38	+	ENV RUNNER_TOOL_CACHE=/opt/hostedtoolcache \
	39	+	LANG=C.UTF-8
	40	+
	41	+	# The runner executable is bind-mounted (container) or provided by the init
	42	+	# disk (vm) here; its init subcommand starts the job's own Docker daemon.
	43	+	ENTRYPOINT ["/opt/ermite/ermite-runner", "init"]

A internal/cmdrun/cmdrun.go +75 -0

@@ -0,0 +1,75 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package cmdrun runs host commands (ip, nft, mkfs.ext4) behind a small
	4	+	// interface so that callers can be tested with a fake.
	5	+	package cmdrun
	6	+
	7	+	import (
	8	+	"bytes"
	9	+	"context"
	10	+	"fmt"
	11	+	"io"
	12	+	"os/exec"
	13	+	"strings"
	14	+	)
	15	+
	16	+	// Runner runs a command to completion and returns its combined output.
	17	+	type Runner interface {
	18	+	Run(ctx context.Context, stdin io.Reader, name string, args ...string) ([]byte, error)
	19	+	}
	20	+
	21	+	// Exec runs real commands.
	22	+	type Exec struct{}
	23	+
	24	+	// Run implements Runner. A non-zero exit becomes an error carrying the command
	25	+	// line and its output.
	26	+	func (Exec) Run(ctx context.Context, stdin io.Reader, name string, args ...string) ([]byte, error) {
	27	+	cmd := exec.CommandContext(ctx, name, args...)
	28	+	cmd.Stdin = stdin
	29	+	var out bytes.Buffer
	30	+	cmd.Stdout = &out
	31	+	cmd.Stderr = &out
	32	+	if err := cmd.Run(); err != nil {
	33	+	return out.Bytes(), fmt.Errorf("%s %s: %w: %s", name, strings.Join(args, " "), err, strings.TrimSpace(out.String()))
	34	+	}
	35	+	return out.Bytes(), nil
	36	+	}
	37	+
	38	+	// Call is one recorded invocation of a Fake.
	39	+	type Call struct {
	40	+	Name string
	41	+	Args []string
	42	+	Stdin string
	43	+	}
	44	+
	45	+	// String renders the call as a shell-like line.
	46	+	func (c Call) String() string { return strings.TrimSpace(c.Name + " " + strings.Join(c.Args, " ")) }
	47	+
	48	+	// Fake records calls and answers them with Handle (nil means success).
	49	+	type Fake struct {
	50	+	Calls []Call
	51	+	Handle func(Call) ([]byte, error)
	52	+	}
	53	+
	54	+	// Run implements Runner.
	55	+	func (f *Fake) Run(_ context.Context, stdin io.Reader, name string, args ...string) ([]byte, error) {
	56	+	c := Call{Name: name, Args: append([]string(nil), args...)}
	57	+	if stdin != nil {
	58	+	b, _ := io.ReadAll(stdin)
	59	+	c.Stdin = string(b)
	60	+	}
	61	+	f.Calls = append(f.Calls, c)
	62	+	if f.Handle != nil {
	63	+	return f.Handle(c)
	64	+	}
	65	+	return nil, nil
	66	+	}
	67	+
	68	+	// Lines returns the recorded calls as strings.
	69	+	func (f *Fake) Lines() []string {
	70	+	out := make([]string, len(f.Calls))
	71	+	for i, c := range f.Calls {
	72	+	out[i] = c.String()
	73	+	}
	74	+	return out
	75	+	}

A internal/dnsfwd/dnsfwd.go +361 -0

@@ -0,0 +1,361 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package dnsfwd is a small DNS forwarder for job sandboxes (stdlib only). It
	4	+	// listens on the gateway address of each sandbox (UDP and TCP, port 53 in
	5	+	// production) and relays queries unchanged to the host's resolvers; the network
	6	+	// policy lets guests reach nothing else on the host. It does not parse queries
	7	+	// beyond what framing needs, so it supports every record type and EDNS.
	8	+	//
	9	+	// Abuse limits, per listener (one listener per guest, so one guest cannot
	10	+	// starve another): at most MaxInflight concurrent upstream exchanges (UDP
	11	+	// queries beyond that are dropped, never queued), at most MaxTCPConns open TCP
	12	+	// connections, an absolute lifetime of TCPLifetime per TCP connection (a client
	13	+	// cannot hold a slot by trickling bytes), a deadline on every upstream
	14	+	// exchange, and a cap on queries per TCP connection.
	15	+	package dnsfwd
	16	+
	17	+	import (
	18	+	"context"
	19	+	"encoding/binary"
	20	+	"errors"
	21	+	"fmt"
	22	+	"io"
	23	+	"net"
	24	+	"net/netip"
	25	+	"strconv"
	26	+	"sync"
	27	+	"sync/atomic"
	28	+	"time"
	29	+
	30	+	"github.com/bnema/zerowrap"
	31	+	)
	32	+
	33	+	const (
	34	+	// TCPLifetime is the absolute lifetime of a TCP connection, set at accept.
	35	+	TCPLifetime = 30 * time.Second
	36	+	// UpstreamTimeout is the deadline of one upstream attempt.
	37	+	UpstreamTimeout = 3 * time.Second
	38	+	// TCPIdleTimeout closes TCP client connections that stay silent.
	39	+	TCPIdleTimeout = 10 * time.Second
	40	+
	41	+	// MaxInflight bounds concurrent upstream exchanges per listener.
	42	+	MaxInflight = 32
	43	+	// MaxTCPConns bounds open TCP connections per listener.
	44	+	MaxTCPConns = 16
	45	+	// TCPMaxQueries caps queries per TCP connection.
	46	+	TCPMaxQueries = 64
	47	+
	48	+	maxUDP = 4096 // largest datagram relayed (EDNS0 sizes up to 4096)
	49	+	hdrLen = 12
	50	+	)
	51	+
	52	+	// Forwarder relays DNS queries. The zero value is not usable; use New.
	53	+	type Forwarder struct {
	54	+	upstreams func() []string
	55	+	port int
	56	+	// Lifetime overrides TCPLifetime; set before Listen (tests).
	57	+	Lifetime time.Duration
	58	+
	59	+	mu sync.Mutex
	60	+	listeners map[netip.Addr]*listener
	61	+	}
	62	+
	63	+	type listener struct {
	64	+	udp net.PacketConn
	65	+	tcp net.Listener
	66	+	cancel context.CancelFunc
	67	+	done sync.WaitGroup
	68	+	// inflight and conns are the per-listener limits (counting semaphores).
	69	+	inflight chan struct{}
	70	+	conns chan struct{}
	71	+	// Dropped counts UDP queries dropped because the listener was saturated.
	72	+	dropped atomic.Int64
	73	+	}
	74	+
	75	+	// New returns a Forwarder. upstreams returns the "host:port" resolvers to try
	76	+	// in order (see Resolvers.Addrs); port is the port to listen on (53, or another
	77	+	// port in tests).
	78	+	func New(upstreams func() []string, port int) *Forwarder {
	79	+	return &Forwarder{
	80	+	upstreams: upstreams,
	81	+	port: port,
	82	+	Lifetime: TCPLifetime,
	83	+	listeners: map[netip.Addr]*listener{},
	84	+	}
	85	+	}
	86	+
	87	+	// Listen starts serving on addr (UDP and TCP). It returns once both sockets are
	88	+	// bound. Listening twice on the same address is an error.
	89	+	func (f *Forwarder) Listen(ctx context.Context, addr netip.Addr) error {
	90	+	f.mu.Lock()
	91	+	defer f.mu.Unlock()
	92	+	if _, ok := f.listeners[addr]; ok {
	93	+	return fmt.Errorf("dnsfwd: already listening on %s", addr)
	94	+	}
	95	+	hp := net.JoinHostPort(addr.String(), strconv.Itoa(f.port))
	96	+	var lc net.ListenConfig
	97	+	pc, err := lc.ListenPacket(ctx, "udp", hp)
	98	+	if err != nil {
	99	+	return fmt.Errorf("dnsfwd: listen udp %s: %w", hp, err)
	100	+	}
	101	+	// With port 0 (tests) UDP and TCP must share the port the kernel picked.
	102	+	if f.port == 0 {
	103	+	hp = pc.LocalAddr().String()
	104	+	}
	105	+	tl, err := lc.Listen(ctx, "tcp", hp)
	106	+	if err != nil {
	107	+	_ = pc.Close()
	108	+	return fmt.Errorf("dnsfwd: listen tcp %s: %w", hp, err)
	109	+	}
	110	+	lctx, cancel := context.WithCancel(context.WithoutCancel(ctx))
	111	+	l := &listener{udp: pc, tcp: tl, cancel: cancel, inflight: make(chan struct{}, MaxInflight), conns: make(chan struct{}, MaxTCPConns)}
	112	+	f.listeners[addr] = l
	113	+	log := zerowrap.FromCtx(ctx)
	114	+	l.done.Add(2)
	115	+	go func() { defer l.done.Done(); f.serveUDP(lctx, l, log) }()
	116	+	go func() { defer l.done.Done(); f.serveTCP(lctx, l, log) }()
	117	+	return nil
	118	+	}
	119	+
	120	+	// Addr returns the bound UDP address for addr, useful when port is 0.
	121	+	func (f *Forwarder) Addr(addr netip.Addr) (net.Addr, bool) {
	122	+	f.mu.Lock()
	123	+	defer f.mu.Unlock()
	124	+	l, ok := f.listeners[addr]
	125	+	if !ok {
	126	+	return nil, false
	127	+	}
	128	+	return l.udp.LocalAddr(), true
	129	+	}
	130	+
	131	+	// Close stops listening on addr and waits for its goroutines. Unknown
	132	+	// addresses are ignored.
	133	+	func (f *Forwarder) Close(addr netip.Addr) {
	134	+	f.mu.Lock()
	135	+	l, ok := f.listeners[addr]
	136	+	delete(f.listeners, addr)
	137	+	f.mu.Unlock()
	138	+	if !ok {
	139	+	return
	140	+	}
	141	+	l.cancel()
	142	+	_ = l.udp.Close()
	143	+	_ = l.tcp.Close()
	144	+	l.done.Wait()
	145	+	}
	146	+
	147	+	// CloseAll stops every listener.
	148	+	func (f *Forwarder) CloseAll() {
	149	+	f.mu.Lock()
	150	+	addrs := make([]netip.Addr, 0, len(f.listeners))
	151	+	for a := range f.listeners {
	152	+	addrs = append(addrs, a)
	153	+	}
	154	+	f.mu.Unlock()
	155	+	for _, a := range addrs {
	156	+	f.Close(a)
	157	+	}
	158	+	}
	159	+
	160	+	// Dropped returns how many UDP queries the listener on addr dropped because it
	161	+	// was saturated.
	162	+	func (f *Forwarder) Dropped(addr netip.Addr) int64 {
	163	+	f.mu.Lock()
	164	+	defer f.mu.Unlock()
	165	+	if l, ok := f.listeners[addr]; ok {
	166	+	return l.dropped.Load()
	167	+	}
	168	+	return 0
	169	+	}
	170	+
	171	+	func (f Forwarder) serveUDP(ctx context.Context, l listener, log zerowrap.Logger) {
	172	+	pc := l.udp
	173	+	buf := make([]byte, maxUDP)
	174	+	for {
	175	+	n, from, err := pc.ReadFrom(buf)
	176	+	if err != nil {
	177	+	if ctx.Err() == nil && !errors.Is(err, net.ErrClosed) {
	178	+	log.Warn().Err(err).Msg("dns forwarder: udp read")
	179	+	}
	180	+	return
	181	+	}
	182	+	if n < hdrLen {
	183	+	continue
	184	+	}
	185	+	select {
	186	+	case l.inflight <- struct{}{}:
	187	+	default:
	188	+	// Saturated: drop. The client retries; blocking here would let one
	189	+	// guest's flood stall the read loop.
	190	+	l.dropped.Add(1)
	191	+	continue
	192	+	}
	193	+	q := append([]byte(nil), buf[:n]...)
	194	+	go func() {
	195	+	defer func() { <-l.inflight }()
	196	+	resp := f.exchange(ctx, q, false)
	197	+	if len(resp) > 0 {
	198	+	_, _ = pc.WriteTo(resp, from)
	199	+	}
	200	+	}()
	201	+	}
	202	+	}
	203	+
	204	+	func (f Forwarder) serveTCP(ctx context.Context, l listener, log zerowrap.Logger) {
	205	+	var wg sync.WaitGroup
	206	+	defer wg.Wait()
	207	+	for {
	208	+	c, err := l.tcp.Accept()
	209	+	if err != nil {
	210	+	if ctx.Err() == nil && !errors.Is(err, net.ErrClosed) {
	211	+	log.Warn().Err(err).Msg("dns forwarder: tcp accept")
	212	+	}
	213	+	return
	214	+	}
	215	+	select {
	216	+	case l.conns <- struct{}{}:
	217	+	default:
	218	+	_ = c.Close() // too many open connections on this listener
	219	+	continue
	220	+	}
	221	+	wg.Add(1)
	222	+	go func() {
	223	+	defer wg.Done()
	224	+	defer func() { <-l.conns }()
	225	+	defer func() { _ = c.Close() }()
	226	+	stop := context.AfterFunc(ctx, func() { _ = c.Close() })
	227	+	defer stop()
	228	+	// Absolute deadline from accept: neither idle nor slow-loris
	229	+	// connections keep their slot past TCPLifetime.
	230	+	deadline := time.Now().Add(f.Lifetime)
	231	+	_ = c.SetDeadline(deadline)
	232	+	f.serveConn(ctx, l, c, deadline)
	233	+	}()
	234	+	}
	235	+	}
	236	+
	237	+	func (f Forwarder) serveConn(ctx context.Context, l listener, c net.Conn, deadline time.Time) {
	238	+	for range TCPMaxQueries {
	239	+	// The idle timeout can only shorten the absolute deadline.
	240	+	dl := time.Now().Add(TCPIdleTimeout)
	241	+	if deadline.Before(dl) {
	242	+	dl = deadline
	243	+	}
	244	+	_ = c.SetReadDeadline(dl)
	245	+	var lb [2]byte
	246	+	if _, err := io.ReadFull(c, lb[:]); err != nil {
	247	+	return
	248	+	}
	249	+	n := int(binary.BigEndian.Uint16(lb[:]))
	250	+	if n < hdrLen {
	251	+	return
	252	+	}
	253	+	q := make([]byte, n)
	254	+	if _, err := io.ReadFull(c, q); err != nil {
	255	+	return
	256	+	}
	257	+	select {
	258	+	case l.inflight <- struct{}{}:
	259	+	default:
	260	+	return // saturated: close, the client retries
	261	+	}
	262	+	resp := f.exchange(ctx, q, true)
	263	+	<-l.inflight
	264	+	if len(resp) == 0 \|\| len(resp) > 0xffff {
	265	+	return
	266	+	}
	267	+	out := make([]byte, 2+len(resp))
	268	+	binary.BigEndian.PutUint16(out, uint16(len(resp)))
	269	+	copy(out[2:], resp)
	270	+	if _, err := c.Write(out); err != nil {
	271	+	return
	272	+	}
	273	+	}
	274	+	}
	275	+
	276	+	// exchange relays q to the upstreams in order and returns the first answer, or
	277	+	// a SERVFAIL when none answers. The transport towards the upstream matches the
	278	+	// client's: a client that fell back to TCP gets a complete answer.
	279	+	func (f *Forwarder) exchange(ctx context.Context, q []byte, overTCP bool) []byte {
	280	+	for _, up := range f.upstreams() {
	281	+	var resp []byte
	282	+	var err error
	283	+	if overTCP {
	284	+	resp, err = exchangeTCP(ctx, up, q)
	285	+	} else {
	286	+	resp, err = exchangeUDP(ctx, up, q)
	287	+	}
	288	+	if err == nil && len(resp) >= hdrLen && resp[0] == q[0] && resp[1] == q[1] {
	289	+	return resp
	290	+	}
	291	+	}
	292	+	return servfail(q)
	293	+	}
	294	+
	295	+	func exchangeUDP(ctx context.Context, up string, q []byte) ([]byte, error) {
	296	+	ctx, cancel := context.WithTimeout(ctx, UpstreamTimeout)
	297	+	defer cancel()
	298	+	var d net.Dialer
	299	+	c, err := d.DialContext(ctx, "udp", up)
	300	+	if err != nil {
	301	+	return nil, err
	302	+	}
	303	+	defer func() { _ = c.Close() }()
	304	+	dl, _ := ctx.Deadline()
	305	+	_ = c.SetDeadline(dl)
	306	+	if _, err := c.Write(q); err != nil {
	307	+	return nil, err
	308	+	}
	309	+	buf := make([]byte, maxUDP)
	310	+	for {
	311	+	n, err := c.Read(buf)
	312	+	if err != nil {
	313	+	return nil, err
	314	+	}
	315	+	// Ignore datagrams that do not answer this query (ID mismatch).
	316	+	if n >= hdrLen && buf[0] == q[0] && buf[1] == q[1] {
	317	+	return append([]byte(nil), buf[:n]...), nil
	318	+	}
	319	+	}
	320	+	}
	321	+
	322	+	func exchangeTCP(ctx context.Context, up string, q []byte) ([]byte, error) {
	323	+	ctx, cancel := context.WithTimeout(ctx, UpstreamTimeout)
	324	+	defer cancel()
	325	+	var d net.Dialer
	326	+	c, err := d.DialContext(ctx, "tcp", up)
	327	+	if err != nil {
	328	+	return nil, err
	329	+	}
	330	+	defer func() { _ = c.Close() }()
	331	+	dl, _ := ctx.Deadline()
	332	+	_ = c.SetDeadline(dl)
	333	+	out := make([]byte, 2+len(q))
	334	+	binary.BigEndian.PutUint16(out, uint16(len(q)))
	335	+	copy(out[2:], q)
	336	+	if _, err := c.Write(out); err != nil {
	337	+	return nil, err
	338	+	}
	339	+	var l [2]byte
	340	+	if _, err := io.ReadFull(c, l[:]); err != nil {
	341	+	return nil, err
	342	+	}
	343	+	resp := make([]byte, binary.BigEndian.Uint16(l[:]))
	344	+	if _, err := io.ReadFull(c, resp); err != nil {
	345	+	return nil, err
	346	+	}
	347	+	return resp, nil
	348	+	}
	349	+
	350	+	// servfail builds a SERVFAIL reply for q: same ID and opcode, QR and RA set,
	351	+	// RD copied, no records.
	352	+	func servfail(q []byte) []byte {
	353	+	if len(q) < hdrLen {
	354	+	return nil
	355	+	}
	356	+	r := make([]byte, hdrLen)
	357	+	r[0], r[1] = q[0], q[1]
	358	+	r[2] = 0x80 \| (q[2] & 0x79) // QR, opcode, RD kept; AA/TC cleared
	359	+	r[3] = 0x80 \| 0x02 // RA, RCODE=SERVFAIL
	360	+	return r
	361	+	}

A internal/dnsfwd/dnsfwd_test.go +442 -0

@@ -0,0 +1,442 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package dnsfwd
	4	+
	5	+	import (
	6	+	"context"
	7	+	"encoding/binary"
	8	+	"errors"
	9	+	"io"
	10	+	"net"
	11	+	"net/netip"
	12	+	"os"
	13	+	"path/filepath"
	14	+	"reflect"
	15	+	"strings"
	16	+	"sync/atomic"
	17	+	"syscall"
	18	+	"testing"
	19	+	"time"
	20	+	)
	21	+
	22	+	// query builds a minimal query for name with the given ID.
	23	+	func query(id uint16, name string) []byte {
	24	+	b := make([]byte, hdrLen)
	25	+	binary.BigEndian.PutUint16(b, id)
	26	+	b[2] = 0x01 // RD
	27	+	b[5] = 1 // QDCOUNT
	28	+	for _, l := range strings.Split(name, ".") {
	29	+	b = append(b, byte(len(l)))
	30	+	b = append(b, l...)
	31	+	}
	32	+	return append(b, 0, 0, 1, 0, 1) // root, A, IN
	33	+	}
	34	+
	35	+	// answer turns a query into a response carrying a recognisable payload.
	36	+	func answer(q []byte) []byte {
	37	+	r := append([]byte(nil), q...)
	38	+	r[2] \|= 0x80
	39	+	return append(r, "ANSWER"...)
	40	+	}
	41	+
	42	+	// fakeUpstream serves UDP and TCP on one port of 127.0.0.1 and returns it.
	43	+	func fakeUpstream(t *testing.T, handle func(q []byte) []byte) string {
	44	+	t.Helper()
	45	+	pc, err := net.ListenPacket("udp", "127.0.0.1:0")
	46	+	if err != nil {
	47	+	t.Skip("no loopback:", err)
	48	+	}
	49	+	addr := pc.LocalAddr().String()
	50	+	tl, err := net.Listen("tcp", addr)
	51	+	if err != nil {
	52	+	t.Fatal(err)
	53	+	}
	54	+	t.Cleanup(func() { _ = pc.Close(); _ = tl.Close() })
	55	+	go func() {
	56	+	buf := make([]byte, 4096)
	57	+	for {
	58	+	n, from, err := pc.ReadFrom(buf)
	59	+	if err != nil {
	60	+	return
	61	+	}
	62	+	if r := handle(append([]byte(nil), buf[:n]...)); r != nil {
	63	+	_, _ = pc.WriteTo(r, from)
	64	+	}
	65	+	}
	66	+	}()
	67	+	go func() {
	68	+	for {
	69	+	c, err := tl.Accept()
	70	+	if err != nil {
	71	+	return
	72	+	}
	73	+	go func() {
	74	+	defer func() { _ = c.Close() }()
	75	+	var l [2]byte
	76	+	if _, err := io.ReadFull(c, l[:]); err != nil {
	77	+	return
	78	+	}
	79	+	q := make([]byte, binary.BigEndian.Uint16(l[:]))
	80	+	if _, err := io.ReadFull(c, q); err != nil {
	81	+	return
	82	+	}
	83	+	r := handle(q)
	84	+	out := make([]byte, 2+len(r))
	85	+	binary.BigEndian.PutUint16(out, uint16(len(r)))
	86	+	copy(out[2:], r)
	87	+	_, _ = c.Write(out)
	88	+	}()
	89	+	}
	90	+	}()
	91	+	return addr
	92	+	}
	93	+
	94	+	func start(t testing.T, ups ...string) (Forwarder, string) {
	95	+	t.Helper()
	96	+	f := New(func() []string { return ups }, 0)
	97	+	ip := netip.MustParseAddr("127.0.0.1")
	98	+	if err := f.Listen(context.Background(), ip); err != nil {
	99	+	t.Fatal(err)
	100	+	}
	101	+	t.Cleanup(f.CloseAll)
	102	+	a, _ := f.Addr(ip)
	103	+	return f, a.String()
	104	+	}
	105	+
	106	+	func TestUDPAndTCPForwarding(t *testing.T) {
	107	+	up := fakeUpstream(t, answer)
	108	+	_, addr := start(t, up)
	109	+
	110	+	c, err := net.Dial("udp", addr)
	111	+	if err != nil {
	112	+	t.Fatal(err)
	113	+	}
	114	+	defer func() { _ = c.Close() }()
	115	+	_ = c.SetDeadline(time.Now().Add(5 * time.Second))
	116	+	q := query(0x1234, "example.test")
	117	+	_, _ = c.Write(q)
	118	+	buf := make([]byte, 512)
	119	+	n, err := c.Read(buf)
	120	+	if err != nil {
	121	+	t.Fatal(err)
	122	+	}
	123	+	if string(buf[n-6:n]) != "ANSWER" \|\| binary.BigEndian.Uint16(buf) != 0x1234 {
	124	+	t.Fatalf("udp answer: %x", buf[:n])
	125	+	}
	126	+
	127	+	tc, err := net.Dial("tcp", addr)
	128	+	if err != nil {
	129	+	t.Fatal(err)
	130	+	}
	131	+	defer func() { _ = tc.Close() }()
	132	+	_ = tc.SetDeadline(time.Now().Add(5 * time.Second))
	133	+	// Two queries on one connection.
	134	+	for _, id := range []uint16{7, 8} {
	135	+	q := query(id, "tcp.test")
	136	+	framed := append([]byte{byte(len(q) >> 8), byte(len(q))}, q...)
	137	+	_, _ = tc.Write(framed)
	138	+	var l [2]byte
	139	+	if _, err := io.ReadFull(tc, l[:]); err != nil {
	140	+	t.Fatal(err)
	141	+	}
	142	+	r := make([]byte, binary.BigEndian.Uint16(l[:]))
	143	+	if _, err := io.ReadFull(tc, r); err != nil {
	144	+	t.Fatal(err)
	145	+	}
	146	+	if binary.BigEndian.Uint16(r) != id \|\| string(r[len(r)-6:]) != "ANSWER" {
	147	+	t.Fatalf("tcp answer for %d: %x", id, r)
	148	+	}
	149	+	}
	150	+	}
	151	+
	152	+	func TestFailoverToSecondUpstream(t *testing.T) {
	153	+	dead := "127.0.0.1:1" // nothing listens: UDP read fails with ICMP refused
	154	+	live := fakeUpstream(t, answer)
	155	+	_, addr := start(t, dead, live)
	156	+	c, _ := net.Dial("udp", addr)
	157	+	defer func() { _ = c.Close() }()
	158	+	_ = c.SetDeadline(time.Now().Add(8 * time.Second))
	159	+	_, _ = c.Write(query(1, "a.test"))
	160	+	buf := make([]byte, 512)
	161	+	n, err := c.Read(buf)
	162	+	if err != nil \|\| string(buf[n-6:n]) != "ANSWER" {
	163	+	t.Fatalf("n=%d err=%v", n, err)
	164	+	}
	165	+	}
	166	+
	167	+	func TestServfailWhenNoUpstreamAnswers(t *testing.T) {
	168	+	_, addr := start(t) // no upstream at all
	169	+	c, _ := net.Dial("udp", addr)
	170	+	defer func() { _ = c.Close() }()
	171	+	_ = c.SetDeadline(time.Now().Add(5 * time.Second))
	172	+	_, _ = c.Write(query(0xbeef, "a.test"))
	173	+	buf := make([]byte, 512)
	174	+	n, err := c.Read(buf)
	175	+	if err != nil {
	176	+	t.Fatal(err)
	177	+	}
	178	+	if n != hdrLen \|\| binary.BigEndian.Uint16(buf) != 0xbeef \|\| buf[3]&0x0f != 2 \|\| buf[2]&0x80 == 0 {
	179	+	t.Fatalf("not a SERVFAIL: %x", buf[:n])
	180	+	}
	181	+	}
	182	+
	183	+	func TestMismatchedUpstreamIDIgnored(t *testing.T) {
	184	+	// Upstream answers with a wrong ID: treated as no answer, client gets SERVFAIL.
	185	+	up := fakeUpstream(t, func(q []byte) []byte {
	186	+	r := answer(q)
	187	+	r[0] ^= 0xff
	188	+	return r
	189	+	})
	190	+	f, addr := start(t, up)
	191	+	_ = f
	192	+	c, _ := net.Dial("udp", addr)
	193	+	defer func() { _ = c.Close() }()
	194	+	_ = c.SetDeadline(time.Now().Add(2UpstreamTimeout + 2time.Second))
	195	+	_, _ = c.Write(query(5, "a.test"))
	196	+	buf := make([]byte, 512)
	197	+	n, err := c.Read(buf)
	198	+	if err != nil {
	199	+	t.Fatal(err)
	200	+	}
	201	+	if buf[3]&0x0f != 2 {
	202	+	t.Fatalf("want SERVFAIL, got %x", buf[:n])
	203	+	}
	204	+	}
	205	+
	206	+	func TestShortPacketsIgnored(t *testing.T) {
	207	+	up := fakeUpstream(t, answer)
	208	+	_, addr := start(t, up)
	209	+	c, _ := net.Dial("udp", addr)
	210	+	defer func() { _ = c.Close() }()
	211	+	_, _ = c.Write([]byte{1, 2, 3}) // dropped
	212	+	_, _ = c.Write(query(9, "a.test"))
	213	+	_ = c.SetDeadline(time.Now().Add(5 * time.Second))
	214	+	buf := make([]byte, 512)
	215	+	n, err := c.Read(buf)
	216	+	if err != nil \|\| binary.BigEndian.Uint16(buf) != 9 {
	217	+	t.Fatalf("n=%d err=%v", n, err)
	218	+	}
	219	+	}
	220	+
	221	+	func TestListenTwiceAndClose(t *testing.T) {
	222	+	f := New(func() []string { return nil }, 0)
	223	+	ip := netip.MustParseAddr("127.0.0.1")
	224	+	if err := f.Listen(context.Background(), ip); err != nil {
	225	+	t.Skip(err)
	226	+	}
	227	+	if err := f.Listen(context.Background(), ip); err == nil {
	228	+	t.Fatal("second Listen on the same address must fail")
	229	+	}
	230	+	a, _ := f.Addr(ip)
	231	+	f.Close(ip)
	232	+	f.Close(ip) // idempotent
	233	+	if _, ok := f.Addr(ip); ok {
	234	+	t.Fatal("still listening after Close")
	235	+	}
	236	+	if pc, err := net.ListenPacket("udp", a.String()); err != nil {
	237	+	t.Fatalf("port not released: %v", err)
	238	+	} else {
	239	+	_ = pc.Close()
	240	+	}
	241	+	}
	242	+
	243	+	func TestServfailShape(t *testing.T) {
	244	+	if servfail([]byte{1}) != nil {
	245	+	t.Fatal("short query must give nil")
	246	+	}
	247	+	q := query(0xabcd, "x.test")
	248	+	q[2] = 0x79 // opcode bits + RD + AA
	249	+	r := servfail(q)
	250	+	if r[0] != 0xab \|\| r[1] != 0xcd \|\| r[2]&0x80 == 0 \|\| r[3] != 0x82 {
	251	+	t.Fatalf("bad servfail %x", r)
	252	+	}
	253	+	}
	254	+
	255	+	func TestParseResolvConf(t *testing.T) {
	256	+	in := `# comment
	257	+	nameserver 127.0.0.53
	258	+	nameserver 127.0.0.53
	259	+	nameserver 192.0.2.1 # trailing
	260	+	nameserver 2001:db8::1
	261	+	nameserver fe80::1%eth0
	262	+	nameserver 0.0.0.0
	263	+	nameserver not-an-ip
	264	+	search example.test
	265	+	options edns0
	266	+	nameserver
	267	+	`
	268	+	got := ParseResolvConf(strings.NewReader(in))
	269	+	want := []string{"127.0.0.53:53", "192.0.2.1:53", "[2001:db8::1]:53", "[fe80::1%eth0]:53"}
	270	+	if !reflect.DeepEqual(got, want) {
	271	+	t.Fatalf("got %v want %v", got, want)
	272	+	}
	273	+	}
	274	+
	275	+	func TestResolversReload(t *testing.T) {
	276	+	p := filepath.Join(t.TempDir(), "resolv.conf")
	277	+	if err := os.WriteFile(p, []byte("nameserver 192.0.2.1\n"), 0o644); err != nil {
	278	+	t.Fatal(err)
	279	+	}
	280	+	r := NewResolvers(p)
	281	+	if got := r.Addrs(); !reflect.DeepEqual(got, []string{"192.0.2.1:53"}) {
	282	+	t.Fatal(got)
	283	+	}
	284	+	if err := os.WriteFile(p, []byte("nameserver 192.0.2.99\nnameserver 192.0.2.1\n"), 0o644); err != nil {
	285	+	t.Fatal(err)
	286	+	}
	287	+	later := time.Now().Add(2 * time.Second)
	288	+	_ = os.Chtimes(p, later, later)
	289	+	if got := r.Addrs(); !reflect.DeepEqual(got, []string{"192.0.2.99:53", "192.0.2.1:53"}) {
	290	+	t.Fatal(got)
	291	+	}
	292	+	// Unreadable file keeps the last known list.
	293	+	_ = os.Remove(p)
	294	+	if got := r.Addrs(); len(got) != 2 {
	295	+	t.Fatalf("lost the last known list: %v", got)
	296	+	}
	297	+	if got := NewResolvers(filepath.Join(t.TempDir(), "missing")).Addrs(); len(got) != 0 {
	298	+	t.Fatal(got)
	299	+	}
	300	+	}
	301	+
	302	+	// S2: a listener holds at most MaxTCPConns connections; extra ones are closed
	303	+	// at once, and a free slot is usable again after a close.
	304	+	func TestTCPConnectionCapPerListener(t *testing.T) {
	305	+	up := fakeUpstream(t, answer)
	306	+	_, addr := start(t, up)
	307	+	var held []net.Conn
	308	+	for range MaxTCPConns {
	309	+	c, err := net.Dial("tcp", addr)
	310	+	if err != nil {
	311	+	t.Fatal(err)
	312	+	}
	313	+	held = append(held, c)
	314	+	defer func() { _ = c.Close() }()
	315	+	}
	316	+	// Let the accept loop register all of them.
	317	+	time.Sleep(200 * time.Millisecond)
	318	+	extra, err := net.Dial("tcp", addr)
	319	+	if err != nil {
	320	+	t.Fatal(err)
	321	+	}
	322	+	defer func() { _ = extra.Close() }()
	323	+	_ = extra.SetReadDeadline(time.Now().Add(3 * time.Second))
	324	+	if _, err := extra.Read(make([]byte, 1)); err != io.EOF && !errors.Is(err, syscall.ECONNRESET) {
	325	+	t.Fatalf("the connection over the cap must be closed by the server, got %v", err)
	326	+	}
	327	+	// Free one slot: a new connection is served.
	328	+	_ = held[0].Close()
	329	+	time.Sleep(200 * time.Millisecond)
	330	+	c, err := net.Dial("tcp", addr)
	331	+	if err != nil {
	332	+	t.Fatal(err)
	333	+	}
	334	+	defer func() { _ = c.Close() }()
	335	+	_ = c.SetDeadline(time.Now().Add(5 * time.Second))
	336	+	q := query(3, "slot.test")
	337	+	_, _ = c.Write(append([]byte{byte(len(q) >> 8), byte(len(q))}, q...))
	338	+	var l [2]byte
	339	+	if _, err := io.ReadFull(c, l[:]); err != nil {
	340	+	t.Fatalf("a freed slot must serve again: %v", err)
	341	+	}
	342	+	}
	343	+
	344	+	// S2: the lifetime is absolute from accept: a client that keeps the connection
	345	+	// alive by trickling bytes still loses it.
	346	+	func TestTCPAbsoluteDeadline(t *testing.T) {
	347	+	up := fakeUpstream(t, answer)
	348	+	f := New(func() []string { return []string{up} }, 0)
	349	+	f.Lifetime = 600 * time.Millisecond
	350	+	ip := netip.MustParseAddr("127.0.0.1")
	351	+	if err := f.Listen(context.Background(), ip); err != nil {
	352	+	t.Skip(err)
	353	+	}
	354	+	t.Cleanup(f.CloseAll)
	355	+	la, _ := f.Addr(ip)
	356	+	addr := la.String()
	357	+	c, err := net.Dial("tcp", addr)
	358	+	if err != nil {
	359	+	t.Fatal(err)
	360	+	}
	361	+	defer func() { _ = c.Close() }()
	362	+	start := time.Now()
	363	+	closed := make(chan struct{})
	364	+	go func() {
	365	+	_, _ = io.Copy(io.Discard, c) // returns when the server closes
	366	+	close(closed)
	367	+	}()
	368	+	// Trickle one byte every 100 ms, never finishing a frame: only an idle
	369	+	// timeout (10 s) would let this live; the absolute deadline must not.
	370	+	go func() {
	371	+	for i := 0; i < 40; i++ {
	372	+	if _, err := c.Write([]byte{0}); err != nil {
	373	+	return
	374	+	}
	375	+	time.Sleep(100 * time.Millisecond)
	376	+	}
	377	+	}()
	378	+	select {
	379	+	case <-closed:
	380	+	if d := time.Since(start); d > 3*time.Second {
	381	+	t.Fatalf("closed after %v, want about %v", d, f.Lifetime)
	382	+	}
	383	+	case <-time.After(5 * time.Second):
	384	+	t.Fatal("a trickling client kept its connection past the absolute deadline")
	385	+	}
	386	+	}
	387	+
	388	+	// S2/S9: a saturated listener drops UDP queries instead of blocking its read
	389	+	// loop, and another listener (another guest) keeps answering.
	390	+	func TestUDPSaturationDropsAndIsolatesListeners(t *testing.T) {
	391	+	release := make(chan struct{})
	392	+	slow := fakeUpstream(t, func(q []byte) []byte { <-release; return answer(q) })
	393	+	fast := fakeUpstream(t, answer)
	394	+	var useSlow atomic.Bool
	395	+	useSlow.Store(true)
	396	+	f := New(func() []string {
	397	+	if useSlow.Load() {
	398	+	return []string{slow}
	399	+	}
	400	+	return []string{fast}
	401	+	}, 0)
	402	+	a1, a2 := netip.MustParseAddr("127.0.0.1"), netip.MustParseAddr("127.0.0.2")
	403	+	if err := f.Listen(context.Background(), a1); err != nil {
	404	+	t.Skip(err)
	405	+	}
	406	+	if err := f.Listen(context.Background(), a2); err != nil {
	407	+	t.Skip(err)
	408	+	}
	409	+	t.Cleanup(f.CloseAll)
	410	+	t.Cleanup(func() { close(release) })
	411	+	l1, _ := f.Addr(a1)
	412	+	flood, err := net.Dial("udp", l1.String())
	413	+	if err != nil {
	414	+	t.Fatal(err)
	415	+	}
	416	+	defer func() { _ = flood.Close() }()
	417	+	for i := 0; i < MaxInflight*3; i++ {
	418	+	_, _ = flood.Write(query(uint16(i), "flood.test"))
	419	+	}
	420	+	deadline := time.Now().Add(3 * time.Second)
	421	+	for f.Dropped(a1) == 0 && time.Now().Before(deadline) {
	422	+	time.Sleep(20 * time.Millisecond)
	423	+	}
	424	+	if f.Dropped(a1) == 0 {
	425	+	t.Fatal("a saturated listener must drop queries")
	426	+	}
	427	+	// The other listener is unaffected: point the resolver at the fast upstream.
	428	+	useSlow.Store(false)
	429	+	l2, _ := f.Addr(a2)
	430	+	c, err := net.Dial("udp", l2.String())
	431	+	if err != nil {
	432	+	t.Fatal(err)
	433	+	}
	434	+	defer func() { _ = c.Close() }()
	435	+	_ = c.SetDeadline(time.Now().Add(4 * time.Second))
	436	+	_, _ = c.Write(query(77, "other-guest.test"))
	437	+	buf := make([]byte, 512)
	438	+	n, err := c.Read(buf)
	439	+	if err != nil \|\| string(buf[n-6:n]) != "ANSWER" {
	440	+	t.Fatalf("another listener must not be starved by the flood: n=%d err=%v", n, err)
	441	+	}
	442	+	}

A internal/dnsfwd/resolvers.go +86 -0

@@ -0,0 +1,86 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package dnsfwd
	4	+
	5	+	import (
	6	+	"bufio"
	7	+	"net"
	8	+	"net/netip"
	9	+	"os"
	10	+	"strings"
	11	+	"sync"
	12	+	"time"
	13	+	)
	14	+
	15	+	// Resolvers reads the upstream nameservers from a resolv.conf-format file and
	16	+	// re-reads it when it changes. Entries such as the systemd-resolved stub
	17	+	// (127.0.0.53) are kept on purpose: the forwarder runs on the host, where the
	18	+	// stub is reachable, and that stub is the only resolver that knows the host's
	19	+	// split-DNS configuration.
	20	+	type Resolvers struct {
	21	+	path string
	22	+
	23	+	mu sync.Mutex
	24	+	mtime time.Time
	25	+	size int64
	26	+	addrs []string
	27	+	}
	28	+
	29	+	// NewResolvers returns a Resolvers reading path (default /etc/resolv.conf).
	30	+	func NewResolvers(path string) *Resolvers {
	31	+	if path == "" {
	32	+	path = "/etc/resolv.conf"
	33	+	}
	34	+	return &Resolvers{path: path}
	35	+	}
	36	+
	37	+	// Addrs returns the upstream "host:53" addresses, in file order. When the file
	38	+	// cannot be read the last known list is returned.
	39	+	func (r *Resolvers) Addrs() []string {
	40	+	r.mu.Lock()
	41	+	defer r.mu.Unlock()
	42	+	st, err := os.Stat(r.path)
	43	+	if err != nil {
	44	+	return r.addrs
	45	+	}
	46	+	if st.ModTime().Equal(r.mtime) && st.Size() == r.size && r.addrs != nil {
	47	+	return r.addrs
	48	+	}
	49	+	f, err := os.Open(r.path)
	50	+	if err != nil {
	51	+	return r.addrs
	52	+	}
	53	+	defer func() { _ = f.Close() }()
	54	+	r.addrs = ParseResolvConf(f)
	55	+	r.mtime, r.size = st.ModTime(), st.Size()
	56	+	return r.addrs
	57	+	}
	58	+
	59	+	// ParseResolvConf extracts "nameserver" entries as "ip:53" strings. Comments,
	60	+	// unknown directives and invalid addresses are ignored; duplicates removed.
	61	+	// IPv6 addresses with a zone (fe80::1%eth0) are kept with their zone.
	62	+	func ParseResolvConf(f interface{ Read([]byte) (int, error) }) []string {
	63	+	var out []string
	64	+	seen := map[string]bool{}
	65	+	sc := bufio.NewScanner(f)
	66	+	for sc.Scan() {
	67	+	line := strings.TrimSpace(sc.Text())
	68	+	if i := strings.IndexAny(line, "#;"); i >= 0 {
	69	+	line = strings.TrimSpace(line[:i])
	70	+	}
	71	+	fields := strings.Fields(line)
	72	+	if len(fields) < 2 \|\| fields[0] != "nameserver" {
	73	+	continue
	74	+	}
	75	+	a, err := netip.ParseAddr(fields[1])
	76	+	if err != nil \|\| a.IsUnspecified() {
	77	+	continue
	78	+	}
	79	+	s := net.JoinHostPort(a.String(), "53")
	80	+	if !seen[s] {
	81	+	seen[s] = true
	82	+	out = append(out, s)
	83	+	}
	84	+	}
	85	+	return out
	86	+	}

A internal/execstream/exec.proto +64 -0

@@ -0,0 +1,64 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Internal wire format between the host process of ermite-runner and the
	4	+	// executor running inside a sandbox (vsock in a micro-VM, Unix socket in a
	5	+	// container). Frames are length-prefixed on the stream (see frame.go); this file
	6	+	// defines the payload. It is internal to ermite-runner: no compatibility is
	7	+	// promised between runner versions, which is safe because the host attaches its
	8	+	// own executable to every sandbox.
	9	+
	10	+	syntax = "proto3";
	11	+
	12	+	package ermite.runner.exec.v1;
	13	+
	14	+	option go_package = "bnema.dev/ermite-runner/internal/execstream/execpb";
	15	+
	16	+	// Frame is the only message on the stream.
	17	+	message Frame {
	18	+	oneof body {
	19	+	Hello hello = 1; // executor -> host, first frame
	20	+	Run run = 2; // host -> executor, after Hello
	21	+	Output output = 3; // executor -> host
	22	+	Exit exit = 4; // executor -> host, last frame
	23	+	Cancel cancel = 5; // host -> executor
	24	+	}
	25	+	}
	26	+
	27	+	message Hello {
	28	+	string version = 1; // runner version of the executor
	29	+	string sandbox = 2; // "vm" or "container"
	30	+	string hostname = 3;
	31	+	}
	32	+
	33	+	// Run asks the executor to run a script. CI-1 stub executor: the script is run
	34	+	// with `sh -c`; an empty script means the built-in diagnostics script.
	35	+	message Run {
	36	+	string script = 1;
	37	+	map<string, string> env = 2;
	38	+	}
	39	+
	40	+	enum StreamKind {
	41	+	STREAM_KIND_UNSPECIFIED = 0;
	42	+	STREAM_KIND_STDOUT = 1;
	43	+	STREAM_KIND_STDERR = 2;
	44	+	}
	45	+
	46	+	message Output {
	47	+	StreamKind stream = 1;
	48	+	bytes data = 2;
	49	+	int64 unix_nano = 3;
	50	+	}
	51	+
	52	+	message Exit {
	53	+	int32 code = 1;
	54	+	string error = 2; // set when the executor failed to run the script at all
	55	+	}
	56	+
	57	+	// Cancel asks the executor to stop the running script. The executor escalates
	58	+	// like the official runner: SIGINT to the script's process group at once,
	59	+	// SIGTERM 7.5 s later, SIGKILL 2.5 s after that.
	60	+	message Cancel {
	61	+	// reason is for the job log and diagnostics ("cancelled by the user",
	62	+	// "job timeout", ...); it never changes the behaviour.
	63	+	string reason = 1;
	64	+	}

A internal/execstream/execpb/exec.pb.go +595 -0 generated file: collapsed

load diff

A internal/execstream/frame.go +147 -0

@@ -0,0 +1,147 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package execstream is the framed protobuf stream between the host process of
	4	+	// ermite-runner and the executor running inside a sandbox.
	5	+	//
	6	+	// Wire format: each frame is a 4-byte big-endian length followed by that many
	7	+	// bytes of a serialized execpb.Frame (see exec.proto). A frame is at most
	8	+	// MaxFrame bytes; longer output is split by the sender. The stream is
	9	+	// transport-agnostic: vsock in a micro-VM, a Unix socket in a container.
	10	+	package execstream
	11	+
	12	+	//go:generate go tool buf generate ../.. --template ../../buf.gen.yaml
	13	+
	14	+	import (
	15	+	"bufio"
	16	+	"encoding/binary"
	17	+	"errors"
	18	+	"fmt"
	19	+	"io"
	20	+	"sync"
	21	+
	22	+	"google.golang.org/protobuf/proto"
	23	+
	24	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	25	+	)
	26	+
	27	+	// MaxFrame is the largest accepted serialized frame.
	28	+	const MaxFrame = 1 << 20
	29	+
	30	+	// ChunkSize is the largest output payload placed in one Output frame.
	31	+	const ChunkSize = 32 * 1024
	32	+
	33	+	// ErrFrameTooLarge is returned for a frame above MaxFrame, on either side.
	34	+	var ErrFrameTooLarge = errors.New("execstream: frame too large")
	35	+
	36	+	// Conn reads and writes frames on a byte stream. Send is safe for concurrent
	37	+	// use; Recv must be called from one goroutine.
	38	+	type Conn struct {
	39	+	rw io.ReadWriter
	40	+	br *bufio.Reader
	41	+	wmu sync.Mutex
	42	+	}
	43	+
	44	+	// NewConn wraps a byte stream.
	45	+	func NewConn(rw io.ReadWriter) *Conn {
	46	+	return &Conn{rw: rw, br: bufio.NewReaderSize(rw, 64*1024)}
	47	+	}
	48	+
	49	+	// Send writes one frame.
	50	+	func (c Conn) Send(f execpb.Frame) error {
	51	+	b, err := proto.Marshal(f)
	52	+	if err != nil {
	53	+	return fmt.Errorf("execstream: marshal: %w", err)
	54	+	}
	55	+	if len(b) > MaxFrame {
	56	+	return ErrFrameTooLarge
	57	+	}
	58	+	buf := make([]byte, 4+len(b))
	59	+	binary.BigEndian.PutUint32(buf, uint32(len(b)))
	60	+	copy(buf[4:], b)
	61	+	c.wmu.Lock()
	62	+	defer c.wmu.Unlock()
	63	+	_, err = c.rw.Write(buf)
	64	+	return err
	65	+	}
	66	+
	67	+	// Recv reads one frame. It returns io.EOF only on a clean end of stream between
	68	+	// frames; a stream cut inside a frame yields io.ErrUnexpectedEOF.
	69	+	func (c Conn) Recv() (execpb.Frame, error) {
	70	+	var hdr [4]byte
	71	+	if _, err := io.ReadFull(c.br, hdr[:]); err != nil {
	72	+	return nil, err
	73	+	}
	74	+	n := binary.BigEndian.Uint32(hdr[:])
	75	+	if n > MaxFrame {
	76	+	return nil, ErrFrameTooLarge
	77	+	}
	78	+	b := make([]byte, n)
	79	+	if _, err := io.ReadFull(c.br, b); err != nil {
	80	+	if errors.Is(err, io.EOF) {
	81	+	err = io.ErrUnexpectedEOF
	82	+	}
	83	+	return nil, err
	84	+	}
	85	+	f := &execpb.Frame{}
	86	+	if err := proto.Unmarshal(b, f); err != nil {
	87	+	return nil, fmt.Errorf("execstream: unmarshal: %w", err)
	88	+	}
	89	+	return f, nil
	90	+	}
	91	+
	92	+	// Drive is the host side of a session: it waits for the executor's Hello, sends
	93	+	// run, passes every Output frame to onOutput, and returns the Exit frame.
	94	+	//
	95	+	// cancel, when non-nil, asks the executor to stop the script: the first value
	96	+	// received is sent as one Cancel frame (the executor escalates SIGINT, SIGTERM,
	97	+	// SIGKILL by itself); closing the channel without a value sends nothing. It is
	98	+	// deliberately not a context: cancelling the session's context would also abort
	99	+	// the reads that deliver the executor's Exit. Drive keeps waiting for Exit after
	100	+	// a Cancel; the caller bounds that wait by stopping the sandbox, which closes
	101	+	// the stream and makes Drive return an error.
	102	+	func Drive(rw io.ReadWriter, run execpb.Run, cancel <-chan execpb.Cancel, onOutput func(execpb.Output)) (execpb.Hello, *execpb.Exit, error) {
	103	+	c := NewConn(rw)
	104	+	first, err := c.Recv()
	105	+	if err != nil {
	106	+	return nil, nil, fmt.Errorf("execstream: waiting for hello: %w", err)
	107	+	}
	108	+	hello := first.GetHello()
	109	+	if hello == nil {
	110	+	return nil, nil, fmt.Errorf("execstream: first frame is not hello")
	111	+	}
	112	+	if err := c.Send(&execpb.Frame{Body: &execpb.Frame_Run{Run: run}}); err != nil {
	113	+	return hello, nil, fmt.Errorf("execstream: sending run: %w", err)
	114	+	}
	115	+
	116	+	done := make(chan struct{})
	117	+	defer close(done)
	118	+	if cancel != nil {
	119	+	go func() {
	120	+	select {
	121	+	case cn, ok := <-cancel:
	122	+	if ok {
	123	+	if cn == nil {
	124	+	cn = &execpb.Cancel{}
	125	+	}
	126	+	_ = c.Send(&execpb.Frame{Body: &execpb.Frame_Cancel{Cancel: cn}})
	127	+	}
	128	+	case <-done:
	129	+	}
	130	+	}()
	131	+	}
	132	+
	133	+	for {
	134	+	f, err := c.Recv()
	135	+	if err != nil {
	136	+	return hello, nil, fmt.Errorf("execstream: reading: %w", err)
	137	+	}
	138	+	switch b := f.Body.(type) {
	139	+	case *execpb.Frame_Output:
	140	+	if onOutput != nil {
	141	+	onOutput(b.Output)
	142	+	}
	143	+	case *execpb.Frame_Exit:
	144	+	return hello, b.Exit, nil
	145	+	}
	146	+	}
	147	+	}

A internal/execstream/frame_test.go +138 -0

@@ -0,0 +1,138 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package execstream
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"encoding/binary"
	8	+	"errors"
	9	+	"io"
	10	+	"net"
	11	+	"testing"
	12	+	"testing/iotest"
	13	+	"time"
	14	+
	15	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	16	+	)
	17	+
	18	+	func TestRoundTrip(t *testing.T) {
	19	+	var buf bytes.Buffer
	20	+	c := NewConn(&buf)
	21	+	in := []*execpb.Frame{
	22	+	{Body: &execpb.Frame_Hello{Hello: &execpb.Hello{Version: "v", Sandbox: "vm"}}},
	23	+	{Body: &execpb.Frame_Output{Output: &execpb.Output{Stream: execpb.StreamKind_STREAM_KIND_STDERR, Data: []byte("x\x00y")}}},
	24	+	{Body: &execpb.Frame_Exit{Exit: &execpb.Exit{Code: 3}}},
	25	+	}
	26	+	for _, f := range in {
	27	+	if err := c.Send(f); err != nil {
	28	+	t.Fatal(err)
	29	+	}
	30	+	}
	31	+	// One byte at a time: framing must not depend on read sizes.
	32	+	r := NewConn(struct {
	33	+	io.Reader
	34	+	io.Writer
	35	+	}{iotest.OneByteReader(&buf), io.Discard})
	36	+	for i := range in {
	37	+	out, err := r.Recv()
	38	+	if err != nil {
	39	+	t.Fatalf("frame %d: %v", i, err)
	40	+	}
	41	+	if out.String() != in[i].String() {
	42	+	t.Fatalf("frame %d: got %v want %v", i, out, in[i])
	43	+	}
	44	+	}
	45	+	if _, err := r.Recv(); err != io.EOF {
	46	+	t.Fatalf("want io.EOF at end, got %v", err)
	47	+	}
	48	+	}
	49	+
	50	+	func TestRecvErrors(t *testing.T) {
	51	+	huge := make([]byte, 4)
	52	+	binary.BigEndian.PutUint32(huge, MaxFrame+1)
	53	+	if _, err := NewConn(bytes.NewBuffer(huge)).Recv(); !errors.Is(err, ErrFrameTooLarge) {
	54	+	t.Fatalf("oversize: %v", err)
	55	+	}
	56	+	trunc := []byte{0, 0, 0, 9, 1, 2}
	57	+	if _, err := NewConn(bytes.NewBuffer(trunc)).Recv(); !errors.Is(err, io.ErrUnexpectedEOF) {
	58	+	t.Fatalf("truncated body: %v", err)
	59	+	}
	60	+	if _, err := NewConn(bytes.NewBuffer([]byte{0, 0})).Recv(); !errors.Is(err, io.ErrUnexpectedEOF) {
	61	+	t.Fatalf("truncated header: %v", err)
	62	+	}
	63	+	garbage := []byte{0, 0, 0, 3, 0xff, 0xff, 0xff}
	64	+	if _, err := NewConn(bytes.NewBuffer(garbage)).Recv(); err == nil {
	65	+	t.Fatal("garbage payload accepted")
	66	+	}
	67	+	}
	68	+
	69	+	func TestSendTooLarge(t *testing.T) {
	70	+	c := NewConn(struct {
	71	+	io.Reader
	72	+	io.Writer
	73	+	}{bytes.NewReader(nil), io.Discard})
	74	+	err := c.Send(&execpb.Frame{Body: &execpb.Frame_Output{Output: &execpb.Output{Data: make([]byte, MaxFrame)}}})
	75	+	if !errors.Is(err, ErrFrameTooLarge) {
	76	+	t.Fatalf("got %v", err)
	77	+	}
	78	+	}
	79	+
	80	+	func TestDrive(t *testing.T) {
	81	+	host, guest := net.Pipe()
	82	+	defer func() { _ = host.Close() }()
	83	+	go func() {
	84	+	defer func() { _ = guest.Close() }()
	85	+	g := NewConn(guest)
	86	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Hello{Hello: &execpb.Hello{Sandbox: "container"}}})
	87	+	f, err := g.Recv()
	88	+	if err != nil \|\| f.GetRun() == nil \|\| f.GetRun().Script != "echo hi" {
	89	+	return
	90	+	}
	91	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Output{Output: &execpb.Output{Data: []byte("hi\n")}}})
	92	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Exit{Exit: &execpb.Exit{Code: 7}}})
	93	+	}()
	94	+	var got []byte
	95	+	hello, exit, err := Drive(host, &execpb.Run{Script: "echo hi"}, nil, func(o *execpb.Output) { got = append(got, o.Data...) })
	96	+	if err != nil {
	97	+	t.Fatal(err)
	98	+	}
	99	+	if hello.Sandbox != "container" \|\| exit.Code != 7 \|\| string(got) != "hi\n" {
	100	+	t.Fatalf("hello=%v exit=%v out=%q", hello, exit, got)
	101	+	}
	102	+	}
	103	+
	104	+	func TestDriveCancelSendsCancel(t *testing.T) {
	105	+	host, guest := net.Pipe()
	106	+	defer func() { _ = host.Close() }()
	107	+	gotCancel := make(chan bool, 1)
	108	+	go func() {
	109	+	defer func() { _ = guest.Close() }()
	110	+	g := NewConn(guest)
	111	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Hello{Hello: &execpb.Hello{}}})
	112	+	_, _ = g.Recv() // run
	113	+	f, err := g.Recv()
	114	+	gotCancel <- err == nil && f.GetCancel() != nil && f.GetCancel().Reason == "cancelled by the user"
	115	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Exit{Exit: &execpb.Exit{Code: 130}}})
	116	+	}()
	117	+	cancel := make(chan *execpb.Cancel, 1)
	118	+	time.AfterFunc(50*time.Millisecond, func() { cancel <- &execpb.Cancel{Reason: "cancelled by the user"} })
	119	+	_, exit, err := Drive(host, &execpb.Run{}, cancel, nil)
	120	+	if err != nil \|\| exit.Code != 130 {
	121	+	t.Fatalf("exit=%v err=%v", exit, err)
	122	+	}
	123	+	if !<-gotCancel {
	124	+	t.Fatal("guest did not receive a Cancel frame")
	125	+	}
	126	+	}
	127	+
	128	+	func TestDriveHelloMissing(t *testing.T) {
	129	+	host, guest := net.Pipe()
	130	+	defer func() { _ = host.Close() }()
	131	+	go func() {
	132	+	defer func() { _ = guest.Close() }()
	133	+	_ = NewConn(guest).Send(&execpb.Frame{Body: &execpb.Frame_Exit{Exit: &execpb.Exit{}}})
	134	+	}()
	135	+	if _, _, err := Drive(host, &execpb.Run{}, nil, nil); err == nil {
	136	+	t.Fatal("want error when the first frame is not hello")
	137	+	}
	138	+	}

A internal/guest/cmdline.go +43 -0

@@ -0,0 +1,43 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package guest is the code that runs inside a sandbox: `ermite-runner init`
	4	+	// (the guest PID 1) and `ermite-runner exec` (the executor).
	5	+	package guest
	6	+
	7	+	import "strings"
	8	+
	9	+	// ParseCmdline splits a kernel command line into key=value pairs; a bare word
	10	+	// has the value "". The last occurrence of a key wins. Values may be quoted
	11	+	// with double quotes, as the kernel allows.
	12	+	func ParseCmdline(s string) map[string]string {
	13	+	out := map[string]string{}
	14	+	for _, f := range splitCmdline(strings.TrimSpace(s)) {
	15	+	k, v, _ := strings.Cut(f, "=")
	16	+	out[k] = strings.Trim(v, `"`)
	17	+	}
	18	+	return out
	19	+	}
	20	+
	21	+	func splitCmdline(s string) []string {
	22	+	var out []string
	23	+	var b strings.Builder
	24	+	inQuote := false
	25	+	for _, r := range s {
	26	+	switch {
	27	+	case r == '"':
	28	+	inQuote = !inQuote
	29	+	b.WriteRune(r)
	30	+	case (r == ' ' \|\| r == '\t' \|\| r == '\n') && !inQuote:
	31	+	if b.Len() > 0 {
	32	+	out = append(out, b.String())
	33	+	b.Reset()
	34	+	}
	35	+	default:
	36	+	b.WriteRune(r)
	37	+	}
	38	+	}
	39	+	if b.Len() > 0 {
	40	+	out = append(out, b.String())
	41	+	}
	42	+	return out
	43	+	}

A internal/guest/connect.go +87 -0

@@ -0,0 +1,87 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package guest
	4	+
	5	+	import (
	6	+	"context"
	7	+	"fmt"
	8	+	"io"
	9	+	"net"
	10	+	"strconv"
	11	+	"strings"
	12	+	"time"
	13	+
	14	+	"golang.org/x/sys/unix"
	15	+	)
	16	+
	17	+	// Target is where the executor connects back to: "unix:/path" (container) or
	18	+	// "vsock:PORT" (vm; the host is CID 2).
	19	+	type Target struct {
	20	+	Network string // "unix" or "vsock"
	21	+	Addr string // socket path, or the vsock port
	22	+	}
	23	+
	24	+	// ParseTarget parses "unix:/path" or "vsock:port".
	25	+	func ParseTarget(s string) (Target, error) {
	26	+	kind, rest, ok := strings.Cut(s, ":")
	27	+	if !ok \|\| rest == "" {
	28	+	return Target{}, fmt.Errorf("connect target %q: want unix:PATH or vsock:PORT", s)
	29	+	}
	30	+	switch kind {
	31	+	case "unix":
	32	+	return Target{Network: "unix", Addr: rest}, nil
	33	+	case "vsock":
	34	+	if p, err := strconv.ParseUint(rest, 10, 32); err != nil \|\| p == 0 {
	35	+	return Target{}, fmt.Errorf("connect target %q: bad vsock port", s)
	36	+	}
	37	+	return Target{Network: "vsock", Addr: rest}, nil
	38	+	}
	39	+	return Target{}, fmt.Errorf("connect target %q: unknown kind %q", s, kind)
	40	+	}
	41	+
	42	+	func (t Target) String() string { return t.Network + ":" + t.Addr }
	43	+
	44	+	// Dial connects to the host, retrying until ctx ends or timeout elapses: the
	45	+	// host listens before the sandbox starts, but a vsock device may take a moment.
	46	+	func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCloser, error) {
	47	+	ctx, cancel := context.WithTimeout(ctx, timeout)
	48	+	defer cancel()
	49	+	var last error
	50	+	for {
	51	+	var c io.ReadWriteCloser
	52	+	var err error
	53	+	if t.Network == "unix" {
	54	+	var d net.Dialer
	55	+	c, err = d.DialContext(ctx, "unix", t.Addr)
	56	+	} else {
	57	+	c, err = dialVsock(t.Addr)
	58	+	}
	59	+	if err == nil {
	60	+	return c, nil
	61	+	}
	62	+	last = err
	63	+	select {
	64	+	case <-ctx.Done():
	65	+	return nil, fmt.Errorf("connect to %s: %w (last error: %v)", t, ctx.Err(), last)
	66	+	case <-time.After(100 * time.Millisecond):
	67	+	}
	68	+	}
	69	+	}
	70	+
	71	+	// vsockHostCID is the well-known CID of the host.
	72	+	const vsockHostCID = 2
	73	+
	74	+	func dialVsock(port string) (io.ReadWriteCloser, error) {
	75	+	p, _ := strconv.ParseUint(port, 10, 32)
	76	+	fd, err := unix.Socket(unix.AF_VSOCK, unix.SOCK_STREAM\|unix.SOCK_CLOEXEC, 0)
	77	+	if err != nil {
	78	+	return nil, err
	79	+	}
	80	+	// A blocking connect is fine: a Firecracker vsock either connects to the
	81	+	// host listener at once or fails.
	82	+	if err := unix.Connect(fd, &unix.SockaddrVM{CID: vsockHostCID, Port: uint32(p)}); err != nil {
	83	+	_ = unix.Close(fd)
	84	+	return nil, err
	85	+	}
	86	+	return newFDConn(fd)
	87	+	}

A internal/guest/exec.go +276 -0

@@ -0,0 +1,276 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package guest
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"fmt"
	9	+	"io"
	10	+	"os"
	11	+	"os/exec"
	12	+	"sync"
	13	+	"syscall"
	14	+	"time"
	15	+
	16	+	"bnema.dev/ermite-runner/internal/execstream"
	17	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	18	+	)
	19	+
	20	+	// DefaultScript is what the stub executor runs when the host sends no script.
	21	+	const DefaultScript = "uname -a; id; cat /etc/os-release; docker info"
	22	+
	23	+	// Cancellation timings, like the official runner's ProcessInvoker: SIGINT, then
	24	+	// SIGTERM after 7.5 s, then SIGKILL 2.5 s later.
	25	+	const (
	26	+	DefaultTermAfter = 7500 * time.Millisecond
	27	+	DefaultKillAfter = 2500 * time.Millisecond
	28	+	// DefaultWaitDelay bounds how long the executor waits for the output pipes
	29	+	// to close after the script's shell exited (background children may hold
	30	+	// them open).
	31	+	DefaultWaitDelay = 500 * time.Millisecond
	32	+	)
	33	+
	34	+	// ExecConfig configures the executor.
	35	+	type ExecConfig struct {
	36	+	Target Target
	37	+	Sandbox string // "vm" or "container", reported in Hello
	38	+	Version string
	39	+	// Shell runs the script as `Shell -c script`; default /bin/sh.
	40	+	Shell string
	41	+	// ConnectTimeout bounds the connection attempts (default 30 s).
	42	+	ConnectTimeout time.Duration
	43	+	// RunTimeout bounds the wait for the host's Run frame (default 60 s).
	44	+	RunTimeout time.Duration
	45	+	// TermAfter and KillAfter are the escalation delays after a Cancel (or after
	46	+	// the executor's own context ends): SIGINT at once, SIGTERM after TermAfter,
	47	+	// SIGKILL KillAfter later. WaitDelay is the pipe-drain grace after the shell
	48	+	// exits. Zero means the defaults above.
	49	+	TermAfter, KillAfter, WaitDelay time.Duration
	50	+
	51	+	// signal sends a signal to a process group; tests replace it to observe the
	52	+	// escalation. Default: syscall.Kill(-pgid, sig).
	53	+	signal func(pgid int, sig syscall.Signal)
	54	+	}
	55	+
	56	+	func (c *ExecConfig) defaults() {
	57	+	if c.Shell == "" {
	58	+	c.Shell = "/bin/sh"
	59	+	}
	60	+	if c.ConnectTimeout == 0 {
	61	+	c.ConnectTimeout = 30 * time.Second
	62	+	}
	63	+	if c.RunTimeout == 0 {
	64	+	c.RunTimeout = 60 * time.Second
	65	+	}
	66	+	if c.TermAfter == 0 {
	67	+	c.TermAfter = DefaultTermAfter
	68	+	}
	69	+	if c.KillAfter == 0 {
	70	+	c.KillAfter = DefaultKillAfter
	71	+	}
	72	+	if c.WaitDelay == 0 {
	73	+	c.WaitDelay = DefaultWaitDelay
	74	+	}
	75	+	if c.signal == nil {
	76	+	c.signal = func(pgid int, sig syscall.Signal) { _ = syscall.Kill(-pgid, sig) }
	77	+	}
	78	+	}
	79	+
	80	+	// Exec is `ermite-runner exec`, the CI-1 stub executor: it connects back to the
	81	+	// host, receives one Run, runs the script with the shell, streams stdout and
	82	+	// stderr as Output frames, and ends with an Exit frame. It returns the exit code
	83	+	// it reported. A Cancel frame, or the end of ctx, stops the script's process
	84	+	// group with escalating signals (see ExecConfig).
	85	+	func Exec(ctx context.Context, cfg ExecConfig) (int, error) {
	86	+	cfg.defaults()
	87	+	raw, err := cfg.Target.Dial(ctx, cfg.ConnectTimeout)
	88	+	if err != nil {
	89	+	return 1, err
	90	+	}
	91	+	defer func() { _ = raw.Close() }()
	92	+	return serve(ctx, raw, cfg)
	93	+	}
	94	+
	95	+	// frameWriter turns writes into Output frames of at most ChunkSize bytes. It is
	96	+	// the command's Stdout/Stderr, so no goroutine of ours sits between the pipe
	97	+	// and the stream.
	98	+	type frameWriter struct {
	99	+	conn *execstream.Conn
	100	+	kind execpb.StreamKind
	101	+	}
	102	+
	103	+	func (w frameWriter) Write(p []byte) (int, error) {
	104	+	n := 0
	105	+	for len(p) > 0 {
	106	+	chunk := p
	107	+	if len(chunk) > execstream.ChunkSize {
	108	+	chunk = chunk[:execstream.ChunkSize]
	109	+	}
	110	+	err := w.conn.Send(&execpb.Frame{Body: &execpb.Frame_Output{Output: &execpb.Output{
	111	+	Stream: w.kind, Data: append([]byte(nil), chunk...), UnixNano: time.Now().UnixNano(),
	112	+	}}})
	113	+	if err != nil {
	114	+	return n, err
	115	+	}
	116	+	n += len(chunk)
	117	+	p = p[len(chunk):]
	118	+	}
	119	+	return n, nil
	120	+	}
	121	+
	122	+	// escalator signals a process group and escalates after a cancel. Once finish
	123	+	// has been called (the shell was reaped) it never signals again: the group id
	124	+	// could have been reused by then.
	125	+	type escalator struct {
	126	+	cfg ExecConfig
	127	+	pgid int
	128	+
	129	+	mu sync.Mutex
	130	+	finished bool
	131	+	started bool
	132	+	timers []*time.Timer
	133	+	}
	134	+
	135	+	func (e *escalator) send(sig syscall.Signal) {
	136	+	e.mu.Lock()
	137	+	defer e.mu.Unlock()
	138	+	if !e.finished {
	139	+	e.cfg.signal(e.pgid, sig)
	140	+	}
	141	+	}
	142	+
	143	+	// cancel starts the escalation; only the first call does anything.
	144	+	func (e *escalator) cancel() {
	145	+	e.mu.Lock()
	146	+	if e.finished \|\| e.started {
	147	+	e.mu.Unlock()
	148	+	return
	149	+	}
	150	+	e.started = true
	151	+	e.cfg.signal(e.pgid, syscall.SIGINT)
	152	+	e.timers = append(e.timers,
	153	+	time.AfterFunc(e.cfg.TermAfter, func() { e.send(syscall.SIGTERM) }),
	154	+	time.AfterFunc(e.cfg.TermAfter+e.cfg.KillAfter, func() { e.send(syscall.SIGKILL) }),
	155	+	)
	156	+	e.mu.Unlock()
	157	+	}
	158	+
	159	+	// finish is called right after the shell was reaped: it kills what is left of
	160	+	// the group (background children), stops the timers and forbids further signals.
	161	+	func (e *escalator) finish() {
	162	+	e.mu.Lock()
	163	+	defer e.mu.Unlock()
	164	+	if e.finished {
	165	+	return
	166	+	}
	167	+	e.cfg.signal(e.pgid, syscall.SIGKILL)
	168	+	e.finished = true
	169	+	for _, t := range e.timers {
	170	+	t.Stop()
	171	+	}
	172	+	}
	173	+
	174	+	func serve(ctx context.Context, rw io.ReadWriteCloser, cfg ExecConfig) (int, error) {
	175	+	conn := execstream.NewConn(rw)
	176	+	host, _ := os.Hostname()
	177	+	if err := conn.Send(&execpb.Frame{Body: &execpb.Frame_Hello{Hello: &execpb.Hello{
	178	+	Version: cfg.Version, Sandbox: cfg.Sandbox, Hostname: host,
	179	+	}}}); err != nil {
	180	+	return 1, fmt.Errorf("send hello: %w", err)
	181	+	}
	182	+
	183	+	// Wait for Run, bounded: a host that never answers must not pin the sandbox.
	184	+	type recv struct {
	185	+	f *execpb.Frame
	186	+	err error
	187	+	}
	188	+	first := make(chan recv, 1)
	189	+	go func() { f, err := conn.Recv(); first <- recv{f, err} }()
	190	+	var run *execpb.Run
	191	+	select {
	192	+	case r := <-first:
	193	+	if r.err != nil {
	194	+	return 1, fmt.Errorf("waiting for run: %w", r.err)
	195	+	}
	196	+	if run = r.f.GetRun(); run == nil {
	197	+	return 1, errors.New("first frame from host is not run")
	198	+	}
	199	+	case <-time.After(cfg.RunTimeout):
	200	+	return 1, errors.New("timed out waiting for the host's run frame")
	201	+	case <-ctx.Done():
	202	+	return 1, ctx.Err()
	203	+	}
	204	+
	205	+	script := run.GetScript()
	206	+	if script == "" {
	207	+	script = DefaultScript
	208	+	}
	209	+	cmd := exec.Command(cfg.Shell, "-c", script)
	210	+	cmd.Env = os.Environ()
	211	+	for k, v := range run.GetEnv() {
	212	+	cmd.Env = append(cmd.Env, k+"="+v)
	213	+	}
	214	+	cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
	215	+	// Frame writers as stdout/stderr: exec copies the pipes for us, and
	216	+	// WaitDelay bounds the wait when a background child keeps a pipe open.
	217	+	cmd.Stdout = frameWriter{conn, execpb.StreamKind_STREAM_KIND_STDOUT}
	218	+	cmd.Stderr = frameWriter{conn, execpb.StreamKind_STREAM_KIND_STDERR}
	219	+	cmd.WaitDelay = cfg.WaitDelay
	220	+	if err := cmd.Start(); err != nil {
	221	+	return sendExit(conn, 127, fmt.Errorf("start %s: %w", cfg.Shell, err))
	222	+	}
	223	+	esc := &escalator{cfg: cfg, pgid: cmd.Process.Pid}
	224	+
	225	+	// Frames from the host while the script runs: a Cancel starts the escalation;
	226	+	// a lost host kills the script rather than let it run unobserved.
	227	+	go func() {
	228	+	for {
	229	+	f, err := conn.Recv()
	230	+	if err != nil {
	231	+	esc.send(syscall.SIGKILL)
	232	+	return
	233	+	}
	234	+	if f.GetCancel() != nil {
	235	+	esc.cancel()
	236	+	}
	237	+	}
	238	+	}()
	239	+	// The executor's own context (SIGTERM forwarded by init) cancels the same way.
	240	+	stopCtx := context.AfterFunc(ctx, esc.cancel)
	241	+	defer stopCtx()
	242	+
	243	+	waitErr := cmd.Wait()
	244	+	esc.finish()
	245	+
	246	+	code := 0
	247	+	var ee *exec.ExitError
	248	+	switch {
	249	+	case waitErr == nil, errors.Is(waitErr, exec.ErrWaitDelay):
	250	+	// ErrWaitDelay: the shell exited cleanly but a background child kept the
	251	+	// pipes open; the group was killed above.
	252	+	case errors.As(waitErr, &ee):
	253	+	code = ee.ExitCode()
	254	+	if code < 0 { // killed by a signal: 128+signal like a shell
	255	+	if ws, ok := ee.Sys().(syscall.WaitStatus); ok && ws.Signaled() {
	256	+	code = 128 + int(ws.Signal())
	257	+	} else {
	258	+	code = 1
	259	+	}
	260	+	}
	261	+	default:
	262	+	return sendExit(conn, 1, waitErr)
	263	+	}
	264	+	return sendExit(conn, code, nil)
	265	+	}
	266	+
	267	+	func sendExit(c *execstream.Conn, code int, cause error) (int, error) {
	268	+	e := &execpb.Exit{Code: int32(code)}
	269	+	if cause != nil {
	270	+	e.Error = cause.Error()
	271	+	}
	272	+	if err := c.Send(&execpb.Frame{Body: &execpb.Frame_Exit{Exit: e}}); err != nil {
	273	+	return code, fmt.Errorf("send exit: %w", err)
	274	+	}
	275	+	return code, nil
	276	+	}

A internal/guest/fdconn.go +20 -0

@@ -0,0 +1,20 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package guest
	4	+
	5	+	import (
	6	+	"io"
	7	+	"os"
	8	+
	9	+	"golang.org/x/sys/unix"
	10	+	)
	11	+
	12	+	// newFDConn wraps a connected stream socket (net.FileConn does not know
	13	+	// AF_VSOCK) as a ReadWriteCloser registered with the runtime poller.
	14	+	func newFDConn(fd int) (io.ReadWriteCloser, error) {
	15	+	if err := unix.SetNonblock(fd, true); err != nil {
	16	+	_ = unix.Close(fd)
	17	+	return nil, err
	18	+	}
	19	+	return os.NewFile(uintptr(fd), "vsock"), nil
	20	+	}

A internal/guest/guest_test.go +411 -0

@@ -0,0 +1,411 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package guest
	4	+
	5	+	import (
	6	+	"context"
	7	+	"fmt"
	8	+	"net"
	9	+	"os"
	10	+	"path/filepath"
	11	+	"reflect"
	12	+	"strconv"
	13	+	"strings"
	14	+	"sync"
	15	+	"sync/atomic"
	16	+	"syscall"
	17	+	"testing"
	18	+	"time"
	19	+
	20	+	"bnema.dev/ermite-runner/internal/execstream"
	21	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	22	+	)
	23	+
	24	+	func TestParseCmdline(t *testing.T) {
	25	+	got := ParseCmdline("console=ttyS0 ro ermite.connect=vsock:1024 x=\"a b\" ermite.gw=10.0.0.1 ermite.gw=10.0.0.9 -- init\n")
	26	+	want := map[string]string{"console": "ttyS0", "ro": "", "ermite.connect": "vsock:1024", "x": "a b", "ermite.gw": "10.0.0.9", "--": "", "init": ""}
	27	+	if !reflect.DeepEqual(got, want) {
	28	+	t.Fatalf("got %v", got)
	29	+	}
	30	+	if len(ParseCmdline("")) != 0 {
	31	+	t.Fatal("empty command line")
	32	+	}
	33	+	}
	34	+
	35	+	func TestParseTarget(t *testing.T) {
	36	+	for in, want := range map[string]Target{
	37	+	"unix:/run/ermite/exec.sock": {"unix", "/run/ermite/exec.sock"},
	38	+	"vsock:1024": {"vsock", "1024"},
	39	+	} {
	40	+	got, err := ParseTarget(in)
	41	+	if err != nil \|\| got != want \|\| got.String() != in {
	42	+	t.Errorf("%q: %v %v", in, got, err)
	43	+	}
	44	+	}
	45	+	for _, in := range []string{"", "unix", "unix:", "vsock:0", "vsock:abc", "tcp:1.2.3.4:5", "vsock:99999999999"} {
	46	+	if _, err := ParseTarget(in); err == nil {
	47	+	t.Errorf("%q accepted", in)
	48	+	}
	49	+	}
	50	+	}
	51	+
	52	+	// host plays the host process on a unix socket and runs one session.
	53	+	func host(t testing.T, script string, cancelAfter time.Duration, tweak func(ExecConfig)) (out, errOut string, exit execpb.Exit, hello execpb.Hello, code int, execErr error) {
	54	+	t.Helper()
	55	+	sock := filepath.Join(t.TempDir(), "s")
	56	+	ln, err := net.Listen("unix", sock)
	57	+	if err != nil {
	58	+	t.Skip(err)
	59	+	}
	60	+	defer func() { _ = ln.Close() }()
	61	+	type res struct {
	62	+	code int
	63	+	err error
	64	+	}
	65	+	done := make(chan res, 1)
	66	+	go func() {
	67	+	cfg := ExecConfig{Target: Target{"unix", sock}, Sandbox: "container", Version: "test", Shell: "/bin/sh", RunTimeout: 5 * time.Second}
	68	+	if tweak != nil {
	69	+	tweak(&cfg)
	70	+	}
	71	+	c, err := Exec(context.Background(), cfg)
	72	+	done <- res{c, err}
	73	+	}()
	74	+	conn, err := ln.Accept()
	75	+	if err != nil {
	76	+	t.Fatal(err)
	77	+	}
	78	+	defer func() { _ = conn.Close() }()
	79	+	var cancel chan *execpb.Cancel
	80	+	if cancelAfter > 0 {
	81	+	cancel = make(chan *execpb.Cancel, 1)
	82	+	time.AfterFunc(cancelAfter, func() { cancel <- &execpb.Cancel{Reason: "test"} })
	83	+	}
	84	+	var so, se strings.Builder
	85	+	hello, exit, err = execstream.Drive(conn, &execpb.Run{Script: script, Env: map[string]string{"STUB_VAR": "from-host"}}, cancel, func(o *execpb.Output) {
	86	+	if o.Stream == execpb.StreamKind_STREAM_KIND_STDERR {
	87	+	se.Write(o.Data)
	88	+	} else {
	89	+	so.Write(o.Data)
	90	+	}
	91	+	})
	92	+	if err != nil {
	93	+	t.Fatal(err)
	94	+	}
	95	+	r := <-done
	96	+	return so.String(), se.String(), exit, hello, r.code, r.err
	97	+	}
	98	+
	99	+	func TestExecStreamsOutputAndExitCode(t *testing.T) {
	100	+	out, errOut, exit, hello, code, err := host(t, `echo out-line; echo err-line >&2; echo "$STUB_VAR"; exit 3`, 0, nil)
	101	+	if err != nil {
	102	+	t.Fatal(err)
	103	+	}
	104	+	if out != "out-line\nfrom-host\n" \|\| errOut != "err-line\n" {
	105	+	t.Fatalf("out=%q err=%q", out, errOut)
	106	+	}
	107	+	if exit.Code != 3 \|\| code != 3 \|\| exit.Error != "" {
	108	+	t.Fatalf("exit=%v code=%d", exit, code)
	109	+	}
	110	+	if hello.Sandbox != "container" \|\| hello.Version != "test" \|\| hello.Hostname == "" {
	111	+	t.Fatalf("hello=%v", hello)
	112	+	}
	113	+	}
	114	+
	115	+	func TestExecDefaultScriptRunsDiagnostics(t *testing.T) {
	116	+	// The default script is the CI-1 stub: uname, id, os-release, docker info.
	117	+	// docker may be absent here: only the layout of the output is checked.
	118	+	out, errOut, exit, _, _, _ := host(t, "", 0, nil)
	119	+	if !strings.Contains(out, "Linux") \|\| !strings.Contains(out, "uid=") {
	120	+	t.Fatalf("uname/id missing: out=%q err=%q", out, errOut)
	121	+	}
	122	+	if exit == nil {
	123	+	t.Fatal("no exit frame")
	124	+	}
	125	+	if !strings.Contains(DefaultScript, "docker info") \|\| !strings.Contains(DefaultScript, "os-release") {
	126	+	t.Fatal("default script must run what the plan says")
	127	+	}
	128	+	}
	129	+
	130	+	func TestExecLargeOutputIsChunkedAndComplete(t *testing.T) {
	131	+	out, _, exit, _, _, _ := host(t, `head -c 300000 /dev/zero \| tr '\0' 'x'`, 0, nil)
	132	+	if len(out) != 300000 \|\| exit.Code != 0 {
	133	+	t.Fatalf("len=%d exit=%v", len(out), exit)
	134	+	}
	135	+	}
	136	+
	137	+	func TestExecCancelSendsSIGINT(t *testing.T) {
	138	+	start := time.Now()
	139	+	out, _, exit, _, _, _ := host(t, `trap 'echo got-int; exit 130' INT; echo ready; while :; do sleep 0.05; done`, 500*time.Millisecond, nil)
	140	+	if !strings.Contains(out, "got-int") \|\| exit.Code != 130 {
	141	+	t.Fatalf("out=%q exit=%v", out, exit)
	142	+	}
	143	+	if time.Since(start) > 5*time.Second {
	144	+	t.Fatal("cancellation too slow")
	145	+	}
	146	+	}
	147	+
	148	+	// sigRecorder replaces the signal function: it records (signal, time since
	149	+	// start) and still delivers the signal.
	150	+	type sigRecorder struct {
	151	+	mu sync.Mutex
	152	+	start time.Time
	153	+	got []string
	154	+	times []time.Duration
	155	+	}
	156	+
	157	+	func (r *sigRecorder) fn(pgid int, sig syscall.Signal) {
	158	+	r.mu.Lock()
	159	+	r.got = append(r.got, sig.String())
	160	+	r.times = append(r.times, time.Since(r.start))
	161	+	r.mu.Unlock()
	162	+	_ = syscall.Kill(-pgid, sig)
	163	+	}
	164	+
	165	+	// C1: a script that ignores SIGINT and SIGTERM is stopped by the escalation:
	166	+	// SIGINT at once, SIGTERM after TermAfter, SIGKILL KillAfter later.
	167	+	func TestExecCancelEscalatesToKill(t *testing.T) {
	168	+	rec := &sigRecorder{start: time.Now()}
	169	+	termAfter, killAfter := 400time.Millisecond, 300time.Millisecond
	170	+	_, _, exit, _, _, _ := host(t, `trap '' INT TERM; echo ready; while :; do sleep 0.05; done`, 300time.Millisecond, func(c ExecConfig) {
	171	+	c.TermAfter, c.KillAfter, c.signal = termAfter, killAfter, rec.fn
	172	+	})
	173	+	if exit.Code != 128+int32(syscall.SIGKILL) {
	174	+	t.Fatalf("a script ignoring INT and TERM must end by SIGKILL, exit=%v", exit)
	175	+	}
	176	+	rec.mu.Lock()
	177	+	defer rec.mu.Unlock()
	178	+	// SIGINT, SIGTERM, SIGKILL (from the escalation), then the final group kill
	179	+	// issued after the shell was reaped.
	180	+	if len(rec.got) < 3 \|\| rec.got[0] != "interrupt" \|\| rec.got[1] != "terminated" \|\| rec.got[2] != "killed" {
	181	+	t.Fatalf("signals: %v", rec.got)
	182	+	}
	183	+	cancelAt := rec.times[0]
	184	+	if d := rec.times[1] - cancelAt; d < termAfter-50time.Millisecond \|\| d > termAfter+400time.Millisecond {
	185	+	t.Errorf("SIGTERM %v after SIGINT, want about %v", d, termAfter)
	186	+	}
	187	+	if d := rec.times[2] - rec.times[1]; d < killAfter-50time.Millisecond \|\| d > killAfter+400time.Millisecond {
	188	+	t.Errorf("SIGKILL %v after SIGTERM, want about %v", d, killAfter)
	189	+	}
	190	+	}
	191	+
	192	+	// C1 + C9: a script that exits on SIGINT stops the escalation: the timers are
	193	+	// stopped when Wait returns, so no SIGTERM/SIGKILL follows, and nothing is
	194	+	// signalled after the shell was reaped except the final group sweep.
	195	+	func TestExecEscalationStopsWhenScriptExits(t *testing.T) {
	196	+	rec := &sigRecorder{start: time.Now()}
	197	+	_, _, exit, _, _, _ := host(t, `trap 'exit 130' INT; while :; do sleep 0.05; done`, 200time.Millisecond, func(c ExecConfig) {
	198	+	c.TermAfter, c.KillAfter, c.signal = 300time.Millisecond, 300time.Millisecond, rec.fn
	199	+	})
	200	+	if exit.Code != 130 {
	201	+	t.Fatalf("exit=%v", exit)
	202	+	}
	203	+	time.Sleep(1200 * time.Millisecond) // well past TermAfter+KillAfter
	204	+	rec.mu.Lock()
	205	+	defer rec.mu.Unlock()
	206	+	if len(rec.got) != 2 \|\| rec.got[0] != "interrupt" \|\| rec.got[1] != "killed" {
	207	+	t.Fatalf("want SIGINT then the final sweep only, got %v", rec.got)
	208	+	}
	209	+	}
	210	+
	211	+	// C9: after the shell was reaped the escalator never signals again, even if its
	212	+	// timers had fired concurrently.
	213	+	func TestEscalatorNeverSignalsAfterFinish(t *testing.T) {
	214	+	var n atomic.Int32
	215	+	e := &escalator{cfg: ExecConfig{TermAfter: time.Millisecond, KillAfter: time.Millisecond, signal: func(int, syscall.Signal) { n.Add(1) }}, pgid: 1 << 30}
	216	+	e.cancel()
	217	+	e.finish()
	218	+	after := n.Load()
	219	+	time.Sleep(50 * time.Millisecond)
	220	+	e.send(syscall.SIGTERM)
	221	+	e.cancel()
	222	+	if n.Load() != after {
	223	+	t.Fatalf("signalled %d times after finish", n.Load()-after)
	224	+	}
	225	+	}
	226	+
	227	+	// C2: a background child that keeps stdout open must not hang the executor: it
	228	+	// reports the script's exit promptly and the child is killed.
	229	+	func TestExecBackgroundChildDoesNotHoldTheSession(t *testing.T) {
	230	+	marker := filepath.Join(t.TempDir(), "child.pid")
	231	+	start := time.Now()
	232	+	out, _, exit, _, _, _ := host(t, `sleep 30 & echo $! > `+marker+`; echo parent-done`, 0, func(c ExecConfig) { c.WaitDelay = 300 time.Millisecond })
	233	+	if time.Since(start) > 5*time.Second {
	234	+	t.Fatalf("the session was held by a background child for %v", time.Since(start))
	235	+	}
	236	+	if exit == nil \|\| exit.Code != 0 \|\| !strings.Contains(out, "parent-done") {
	237	+	t.Fatalf("exit=%v out=%q", exit, out)
	238	+	}
	239	+	b, err := os.ReadFile(marker)
	240	+	if err != nil {
	241	+	t.Fatal(err)
	242	+	}
	243	+	pid, _ := strconv.Atoi(strings.TrimSpace(string(b)))
	244	+	deadline := time.Now().Add(3 * time.Second)
	245	+	for time.Now().Before(deadline) {
	246	+	if st, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid)); err != nil \|\| strings.Contains(string(st), ") Z") {
	247	+	return
	248	+	}
	249	+	time.Sleep(50 * time.Millisecond)
	250	+	}
	251	+	t.Fatalf("background child %d survived the job", pid)
	252	+	}
	253	+
	254	+	// C2: output written by the script before the shell exits is complete even when
	255	+	// the stream is large and a background child holds the pipe.
	256	+	func TestExecOutputCompleteWithBackgroundChild(t *testing.T) {
	257	+	out, _, exit, _, _, _ := host(t, `sleep 30 & head -c 200000 /dev/zero \| tr '\0' 'y'`, 0, func(c ExecConfig) { c.WaitDelay = 300 time.Millisecond })
	258	+	if exit.Code != 0 \|\| len(out) != 200000 {
	259	+	t.Fatalf("len=%d exit=%v", len(out), exit)
	260	+	}
	261	+	}
	262	+
	263	+	func TestExecKilledBySignalReports128PlusSignal(t *testing.T) {
	264	+	_, _, exit, _, _, _ := host(t, `kill -TERM $$`, 0, nil)
	265	+	if exit.Code != 128+int32(syscall.SIGTERM) {
	266	+	t.Fatalf("exit=%v", exit)
	267	+	}
	268	+	}
	269	+
	270	+	func TestExecMissingShell(t *testing.T) {
	271	+	sock := filepath.Join(t.TempDir(), "s")
	272	+	ln, err := net.Listen("unix", sock)
	273	+	if err != nil {
	274	+	t.Skip(err)
	275	+	}
	276	+	defer func() { _ = ln.Close() }()
	277	+	go func() {
	278	+	c, _ := ln.Accept()
	279	+	defer func() { _ = c.Close() }()
	280	+	_, exit, _ := execstream.Drive(c, &execpb.Run{Script: "true"}, nil, nil)
	281	+	if exit == nil \|\| exit.Code != 127 \|\| exit.Error == "" {
	282	+	t.Errorf("want 127 with an error message, got %v", exit)
	283	+	}
	284	+	}()
	285	+	code, _ := Exec(context.Background(), ExecConfig{Target: Target{"unix", sock}, Shell: "/nonexistent/sh", RunTimeout: 3 * time.Second})
	286	+	if code != 127 {
	287	+	t.Fatalf("code=%d", code)
	288	+	}
	289	+	}
	290	+
	291	+	func TestExecHostGoneKillsTheScript(t *testing.T) {
	292	+	sock := filepath.Join(t.TempDir(), "s")
	293	+	ln, err := net.Listen("unix", sock)
	294	+	if err != nil {
	295	+	t.Skip(err)
	296	+	}
	297	+	defer func() { _ = ln.Close() }()
	298	+	done := make(chan int, 1)
	299	+	go func() {
	300	+	c, _ := Exec(context.Background(), ExecConfig{Target: Target{"unix", sock}, Shell: "/bin/sh", RunTimeout: 3 * time.Second})
	301	+	done <- c
	302	+	}()
	303	+	conn, _ := ln.Accept()
	304	+	g := execstream.NewConn(conn)
	305	+	_, _ = g.Recv() // hello
	306	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Run{Run: &execpb.Run{Script: "sleep 60"}}})
	307	+	time.Sleep(200 * time.Millisecond)
	308	+	_ = conn.Close() // the host died
	309	+	select {
	310	+	case code := <-done:
	311	+	if code == 0 {
	312	+	t.Fatalf("code=%d", code)
	313	+	}
	314	+	case <-time.After(10 * time.Second):
	315	+	t.Fatal("the executor kept running after the host went away")
	316	+	}
	317	+	}
	318	+
	319	+	func TestExecConnectTimeout(t *testing.T) {
	320	+	start := time.Now()
	321	+	code, err := Exec(context.Background(), ExecConfig{Target: Target{"unix", filepath.Join(t.TempDir(), "nobody")}, ConnectTimeout: 300 * time.Millisecond})
	322	+	if code == 0 \|\| err == nil \|\| time.Since(start) > 3*time.Second {
	323	+	t.Fatalf("code=%d err=%v after %v", code, err, time.Since(start))
	324	+	}
	325	+	}
	326	+
	327	+	func TestExecRunTimeout(t *testing.T) {
	328	+	sock := filepath.Join(t.TempDir(), "s")
	329	+	ln, err := net.Listen("unix", sock)
	330	+	if err != nil {
	331	+	t.Skip(err)
	332	+	}
	333	+	defer func() { _ = ln.Close() }()
	334	+	go func() { c, _ := ln.Accept(); time.Sleep(2 * time.Second); _ = c.Close() }()
	335	+	code, err := Exec(context.Background(), ExecConfig{Target: Target{"unix", sock}, RunTimeout: 200 * time.Millisecond})
	336	+	if code == 0 \|\| err == nil \|\| !strings.Contains(err.Error(), "run frame") {
	337	+	t.Fatalf("code=%d err=%v", code, err)
	338	+	}
	339	+	}
	340	+
	341	+	func TestInitInContainerSkipsVMSetup(t *testing.T) {
	342	+	// In a container (not a VM's PID 1) Init must not mount anything: with a
	343	+	// command line that says "vm" but a pid that is not 1 it behaves as a
	344	+	// container, runs the payload and returns the executor's code.
	345	+	sock := filepath.Join(t.TempDir(), "s")
	346	+	ln, err := net.Listen("unix", sock)
	347	+	if err != nil {
	348	+	t.Skip(err)
	349	+	}
	350	+	defer func() { _ = ln.Close() }()
	351	+	go func() {
	352	+	c, err := ln.Accept()
	353	+	if err != nil {
	354	+	return
	355	+	}
	356	+	defer func() { _ = c.Close() }()
	357	+	_, _, _ = execstream.Drive(c, &execpb.Run{Script: "exit 0"}, nil, nil)
	358	+	}()
	359	+	code := Init(context.Background(), InitConfig{
	360	+	Cmdline: "ermite.sandbox=vm ermite.connect=unix:" + sock,
	361	+	Getpid: func() int { return 4242 },
	362	+	ExecArgs: []string{"/bin/sh", "-c", "exit 5"},
	363	+	SkipDocker: true,
	364	+	})
	365	+	// ExecArgs is a stand-in for the executor: its exit code is Init's.
	366	+	if code != 5 {
	367	+	t.Fatalf("code=%d", code)
	368	+	}
	369	+	}
	370	+
	371	+	func TestInitWithoutTargetFails(t *testing.T) {
	372	+	if code := Init(context.Background(), InitConfig{Cmdline: "ermite.sandbox=container", Getpid: func() int { return 77 }, ExecArgs: []string{"/bin/true"}, SkipDocker: true}); code == 0 {
	373	+	t.Fatal("init without a connect target must fail")
	374	+	}
	375	+	}
	376	+
	377	+	func TestWriteResolvConfNeedsGateway(t *testing.T) {
	378	+	if err := writeResolvConf(""); err == nil {
	379	+	t.Fatal("empty gateway accepted")
	380	+	}
	381	+	}
	382	+
	383	+	// C11: init forwards SIGTERM to the executor and finishes with the executor's
	384	+	// own exit code, instead of dying with its child still running.
	385	+	func TestInitForwardsSIGTERMToTheExecutor(t *testing.T) {
	386	+	sigs := make(chan os.Signal, 1)
	387	+	done := make(chan int, 1)
	388	+	marker := filepath.Join(t.TempDir(), "got-term")
	389	+	go func() {
	390	+	done <- Init(context.Background(), InitConfig{
	391	+	Cmdline: "ermite.sandbox=container ermite.connect=unix:/nonexistent",
	392	+	Getpid: func() int { return 4242 },
	393	+	SkipDocker: true,
	394	+	Signals: sigs,
	395	+	ExecArgs: []string{"/bin/sh", "-c", `trap 'echo term > ` + marker + `; exit 9' TERM; while :; do sleep 0.05; done`},
	396	+	})
	397	+	}()
	398	+	time.Sleep(500 * time.Millisecond)
	399	+	sigs <- syscall.SIGTERM
	400	+	select {
	401	+	case code := <-done:
	402	+	if code != 9 {
	403	+	t.Fatalf("init must return the executor's exit code, got %d", code)
	404	+	}
	405	+	case <-time.After(10 * time.Second):
	406	+	t.Fatal("init did not finish after SIGTERM")
	407	+	}
	408	+	if _, err := os.Stat(marker); err != nil {
	409	+	t.Fatal("the executor never received SIGTERM")
	410	+	}
	411	+	}

A internal/guest/init.go +259 -0

@@ -0,0 +1,259 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package guest
	4	+
	5	+	import (
	6	+	"context"
	7	+	"fmt"
	8	+	"os"
	9	+	"os/signal"
	10	+	"strings"
	11	+	"syscall"
	12	+	"time"
	13	+
	14	+	"github.com/bnema/zerowrap"
	15	+	"golang.org/x/sys/unix"
	16	+
	17	+	"bnema.dev/ermite-runner/internal/sandbox"
	18	+	)
	19	+
	20	+	// InitConfig configures Init. The zero value reads the real system; tests
	21	+	// replace the fields.
	22	+	type InitConfig struct {
	23	+	Version string
	24	+	// Cmdline is the kernel command line; empty reads /proc/cmdline.
	25	+	Cmdline string
	26	+	// Getpid reports the process id; Init only acts as a VM's PID 1 when it is 1.
	27	+	Getpid func() int
	28	+	// Signals delivers the termination signals to forward to the executor;
	29	+	// nil means "subscribe to SIGTERM and SIGINT". Tests inject their own.
	30	+	Signals <-chan os.Signal
	31	+	// SkipDocker does not start dockerd (tests, images without Docker).
	32	+	SkipDocker bool
	33	+	// ExecArgs is the executor command line; default
	34	+	// [GuestExecDir/GuestExecName exec].
	35	+	ExecArgs []string
	36	+	}
	37	+
	38	+	// Init is `ermite-runner init`: the entrypoint of every sandbox.
	39	+	//
	40	+	// In a micro-VM it is the kernel's init (PID 1, root = the read-only exec disk,
	41	+	// /dev/vdb = shared base image, /dev/vdc = per-job scratch disk): it mounts the
	42	+	// pseudo filesystems, sets the clock and the DNS server, builds an overlayfs
	43	+	// root (lower = base, upper = scratch) and pivots into it, then does what a
	44	+	// container does.
	45	+	//
	46	+	// In a container the root and the pseudo filesystems are ready: it skips all of
	47	+	// that and only prepares cgroups. Then, in both: start dockerd with its data
	48	+	// root on the scratch space, wait for its socket, run the executor
	49	+	// (`ermite-runner exec`) as a child, reap orphans, stop dockerd, and finish by
	50	+	// powering the VM off, or exiting with the executor's code in a container.
	51	+	func Init(ctx context.Context, cfg InitConfig) int {
	52	+	log := zerowrap.FromCtx(ctx)
	53	+	if cfg.Getpid == nil {
	54	+	cfg.Getpid = os.Getpid
	55	+	}
	56	+	if cfg.ExecArgs == nil {
	57	+	cfg.ExecArgs = []string{sandbox.GuestExecDir + "/" + sandbox.GuestExecName, "exec"}
	58	+	}
	59	+	cmdline := cfg.Cmdline
	60	+	if cmdline == "" {
	61	+	b, err := os.ReadFile("/proc/cmdline")
	62	+	if err != nil && cfg.Getpid() == 1 {
	63	+	// A VM's init starts with nothing mounted: /proc is needed to read
	64	+	// the kernel command line that says what we are. setupVM reuses it.
	65	+	if merr := unix.Mount("proc", "/proc", "proc", 0, ""); merr == nil {
	66	+	b, err = os.ReadFile("/proc/cmdline")
	67	+	}
	68	+	}
	69	+	if err == nil {
	70	+	cmdline = string(b)
	71	+	}
	72	+	}
	73	+	args := ParseCmdline(cmdline)
	74	+	inVM := args[sandbox.CmdlineSandbox] == string(sandbox.VM) && cfg.Getpid() == 1
	75	+
	76	+	mode := string(sandbox.Container)
	77	+	if inVM {
	78	+	mode = string(sandbox.VM)
	79	+	if err := setupVM(log, args); err != nil {
	80	+	log.Error().Err(err).Msg("init: vm setup failed")
	81	+	powerOff()
	82	+	return 1
	83	+	}
	84	+	}
	85	+	code := runPayload(ctx, cfg, mode, args)
	86	+	if inVM {
	87	+	powerOff()
	88	+	}
	89	+	return code
	90	+	}
	91	+
	92	+	// defaultPath is the PATH of a sandbox whose init was started by the kernel
	93	+	// with an empty environment.
	94	+	const defaultPath = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
	95	+
	96	+	func runPayload(ctx context.Context, cfg InitConfig, mode string, args map[string]string) int {
	97	+	log := zerowrap.FromCtx(ctx)
	98	+	if os.Getenv("PATH") == "" {
	99	+	_ = os.Setenv("PATH", defaultPath)
	100	+	}
	101	+	if os.Getenv("HOME") == "" {
	102	+	_ = os.Setenv("HOME", "/root")
	103	+	}
	104	+	if mode == string(sandbox.Container) {
	105	+	if err := nestCgroups(); err != nil {
	106	+	log.Warn().Err(err).Msg("init: cgroup setup (docker inside may not start)")
	107	+	}
	108	+	}
	109	+	target := args[sandbox.CmdlineConnect]
	110	+	if target == "" {
	111	+	target = os.Getenv("ERMITE_CONNECT")
	112	+	}
	113	+	if _, err := ParseTarget(target); err != nil {
	114	+	log.Error().Err(err).Msg("init: no executor target")
	115	+	return 1
	116	+	}
	117	+
	118	+	var dockerPid int
	119	+	if _, err := os.Stat("/usr/bin/dockerd"); err == nil && !cfg.SkipDocker {
	120	+	_ = os.MkdirAll(sandbox.GuestScratch+"/docker", 0o711)
	121	+	pid, err := spawn([]string{"/usr/bin/dockerd", "--data-root", sandbox.GuestScratch + "/docker",
	122	+	"--storage-driver", "overlay2", "--host", "unix:///var/run/docker.sock"}, nil)
	123	+	if err != nil {
	124	+	log.Error().Err(err).Msg("init: start dockerd")
	125	+	} else {
	126	+	dockerPid = pid
	127	+	}
	128	+	}
	129	+
	130	+	execPid, err := spawn(cfg.ExecArgs, append(os.Environ(),
	131	+	"ERMITE_CONNECT="+target, "ERMITE_SANDBOX="+mode, "ERMITE_VERSION="+cfg.Version))
	132	+	if err != nil {
	133	+	log.Error().Err(err).Msg("init: start executor")
	134	+	return 1
	135	+	}
	136	+	// SIGTERM/SIGINT (docker stop, or the host stopping the VM) go to the
	137	+	// executor, which cancels its script with the usual escalation and reports
	138	+	// its Exit; init then finishes normally instead of dying with children.
	139	+	sigs := cfg.Signals
	140	+	if sigs == nil {
	141	+	ch := make(chan os.Signal, 4)
	142	+	signal.Notify(ch, syscall.SIGTERM, syscall.SIGINT)
	143	+	defer signal.Stop(ch)
	144	+	sigs = ch
	145	+	}
	146	+	stopFwd := make(chan struct{})
	147	+	defer close(stopFwd)
	148	+	go func() {
	149	+	for {
	150	+	select {
	151	+	case sig := <-sigs:
	152	+	if s, ok := sig.(syscall.Signal); ok {
	153	+	log.Info().Str("signal", s.String()).Msg("init: forwarding the signal to the executor")
	154	+	_ = syscall.Kill(-execPid, s) // the executor leads its own process group
	155	+	_ = syscall.Kill(execPid, s)
	156	+	}
	157	+	case <-stopFwd:
	158	+	return
	159	+	}
	160	+	}
	161	+	}()
	162	+	code := reapUntil(execPid)
	163	+	if dockerPid != 0 {
	164	+	_ = syscall.Kill(dockerPid, syscall.SIGTERM)
	165	+	deadline := time.Now().Add(5 * time.Second)
	166	+	for time.Now().Before(deadline) {
	167	+	if pid, _ := wait4NoHang(dockerPid); pid == dockerPid {
	168	+	break
	169	+	}
	170	+	time.Sleep(100 * time.Millisecond)
	171	+	}
	172	+	_ = syscall.Kill(dockerPid, syscall.SIGKILL)
	173	+	}
	174	+	return code
	175	+	}
	176	+
	177	+	// spawn starts argv[0] as a child (own process group) without os/exec, so that
	178	+	// reapUntil is the only waiter and orphans can be reaped safely.
	179	+	func spawn(argv, env []string) (int, error) {
	180	+	if env == nil {
	181	+	env = os.Environ()
	182	+	}
	183	+	return syscall.ForkExec(argv[0], argv, &syscall.ProcAttr{
	184	+	Env: env,
	185	+	Files: []uintptr{0, 1, 2},
	186	+	Sys: &syscall.SysProcAttr{Setpgid: true},
	187	+	})
	188	+	}
	189	+
	190	+	// reapUntil waits for every child (orphans included) and returns when pid
	191	+	// exits, with its exit code (128+signal when killed).
	192	+	func reapUntil(pid int) int {
	193	+	for {
	194	+	var ws syscall.WaitStatus
	195	+	got, err := syscall.Wait4(-1, &ws, 0, nil)
	196	+	if err == syscall.EINTR {
	197	+	continue
	198	+	}
	199	+	if err != nil { // ECHILD: no child left, the executor included
	200	+	return 1
	201	+	}
	202	+	if got == pid {
	203	+	switch {
	204	+	case ws.Exited():
	205	+	return ws.ExitStatus()
	206	+	case ws.Signaled():
	207	+	return 128 + int(ws.Signal())
	208	+	}
	209	+	return 1
	210	+	}
	211	+	}
	212	+	}
	213	+
	214	+	func wait4NoHang(pid int) (int, error) {
	215	+	var ws syscall.WaitStatus
	216	+	return syscall.Wait4(pid, &ws, syscall.WNOHANG, nil)
	217	+	}
	218	+
	219	+	func powerOff() {
	220	+	unix.Sync()
	221	+	// On x86 Firecracker has no ACPI power off: a reboot makes the VMM exit
	222	+	// (kernel cmdline reboot=k). On arm64 PSCI handles power off.
	223	+	_ = unix.Reboot(unix.LINUX_REBOOT_CMD_RESTART)
	224	+	_ = unix.Reboot(unix.LINUX_REBOOT_CMD_POWER_OFF)
	225	+	}
	226	+
	227	+	// nestCgroups prepares cgroup v2 for a nested dockerd in a container: a cgroup
	228	+	// with member processes cannot delegate controllers to children, so move every
	229	+	// process to an "init" child and enable the controllers (like the upstream
	230	+	// docker-in-docker entrypoint). No-op without cgroup v2.
	231	+	func nestCgroups() error {
	232	+	const root = "/sys/fs/cgroup"
	233	+	ctl, err := os.ReadFile(root + "/cgroup.controllers")
	234	+	if err != nil {
	235	+	return nil
	236	+	}
	237	+	if err := os.MkdirAll(root+"/init", 0o755); err != nil {
	238	+	return err
	239	+	}
	240	+	procs, err := os.ReadFile(root + "/cgroup.procs")
	241	+	if err != nil {
	242	+	return err
	243	+	}
	244	+	for _, p := range strings.Fields(string(procs)) {
	245	+	// Processes may exit meanwhile: ignore per-process errors.
	246	+	_ = os.WriteFile(root+"/init/cgroup.procs", []byte(p), 0o644)
	247	+	}
	248	+	var enable []string
	249	+	for _, c := range strings.Fields(string(ctl)) {
	250	+	enable = append(enable, "+"+c)
	251	+	}
	252	+	if len(enable) == 0 {
	253	+	return nil
	254	+	}
	255	+	if err := os.WriteFile(root+"/cgroup.subtree_control", []byte(strings.Join(enable, " ")), 0o644); err != nil {
	256	+	return fmt.Errorf("enable controllers: %w", err)
	257	+	}
	258	+	return nil
	259	+	}

A internal/guest/vmsetup.go +203 -0

@@ -0,0 +1,203 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package guest
	4	+
	5	+	import (
	6	+	"fmt"
	7	+	"os"
	8	+	"strconv"
	9	+	"strings"
	10	+
	11	+	"github.com/bnema/zerowrap"
	12	+	"golang.org/x/sys/unix"
	13	+
	14	+	"bnema.dev/ermite-runner/internal/sandbox"
	15	+	)
	16	+
	17	+	// Devices of the micro-VM, in the order the host attaches them (virtio-blk).
	18	+	const (
	19	+	devBase = "/dev/vdb" // shared read-only base image (ext4)
	20	+	devScratch = "/dev/vdc" // per-job scratch disk (ext4)
	21	+	)
	22	+
	23	+	// KernelArg names of the clock and DNS settings (not part of the contract
	24	+	// constants because only init reads them).
	25	+	const (
	26	+	cmdlineTime = "ermite.time" // host unix time at boot, seconds
	27	+	)
	28	+
	29	+	// setupVM turns the minimal boot root (the exec disk) into the job's root:
	30	+	// overlayfs of base + scratch, with the exec disk bound read-only at
	31	+	// /opt/ermite, then pivot_root into it and mount the pseudo filesystems.
	32	+	func setupVM(log zerowrap.Logger, args map[string]string) error {
	33	+	// Pseudo filesystems of the boot root. Their mountpoints exist on the exec
	34	+	// disk (the root is read-only).
	35	+	for _, m := range []struct{ src, dst, typ string }{
	36	+	{"proc", "/proc", "proc"},
	37	+	{"sysfs", "/sys", "sysfs"},
	38	+	{"devtmpfs", "/dev", "devtmpfs"},
	39	+	} {
	40	+	// proc is mounted by Init (to read the command line) and devtmpfs by the
	41	+	// kernel (DEVTMPFS_MOUNT): mount only what is missing.
	42	+	if isMounted(m.dst) {
	43	+	continue
	44	+	}
	45	+	if err := mount(m.src, m.dst, m.typ, 0, ""); err != nil {
	46	+	return err
	47	+	}
	48	+	}
	49	+	// Firecracker has no RTC on x86: the guest clock starts at the epoch, which
	50	+	// breaks TLS. The host passes its time on the command line.
	51	+	if v, ok := args[cmdlineTime]; ok {
	52	+	if sec, err := strconv.ParseInt(v, 10, 64); err == nil && sec > 1_600_000_000 {
	53	+	if err := unix.ClockSettime(unix.CLOCK_REALTIME, &unix.Timespec{Sec: sec}); err != nil {
	54	+	log.Warn().Err(err).Msg("init: set clock")
	55	+	}
	56	+	}
	57	+	}
	58	+
	59	+	if err := mount(devBase, "/mnt/base", "ext4", unix.MS_RDONLY, ""); err != nil {
	60	+	return err
	61	+	}
	62	+	if err := mount(devScratch, "/mnt/scratch", "ext4", unix.MS_NOATIME, ""); err != nil {
	63	+	return err
	64	+	}
	65	+	for _, d := range []string{"/mnt/scratch/upper", "/mnt/scratch/work", "/mnt/scratch/docker"} {
	66	+	if err := os.MkdirAll(d, 0o755); err != nil {
	67	+	return err
	68	+	}
	69	+	}
	70	+	opts := "lowerdir=/mnt/base,upperdir=/mnt/scratch/upper,workdir=/mnt/scratch/work"
	71	+	if err := mount("overlay", "/mnt/newroot", "overlay", 0, opts); err != nil {
	72	+	return err
	73	+	}
	74	+	// The new root needs these mountpoints; they live in the (writable) upper.
	75	+	for _, d := range []string{"/mnt/newroot/proc", "/mnt/newroot/sys", "/mnt/newroot/dev",
	76	+	"/mnt/newroot/run", "/mnt/newroot/tmp", "/mnt/newroot/scratch", "/mnt/newroot/.oldroot",
	77	+	"/mnt/newroot" + sandbox.GuestExecDir} {
	78	+	if err := os.MkdirAll(d, 0o755); err != nil {
	79	+	return err
	80	+	}
	81	+	}
	82	+	// Scratch and the exec disk stay reachable after the pivot.
	83	+	if err := mount("/mnt/scratch", "/mnt/newroot"+sandbox.GuestScratch, "", unix.MS_BIND, ""); err != nil {
	84	+	return err
	85	+	}
	86	+	if err := mount("/", "/mnt/newroot"+sandbox.GuestExecDir, "", unix.MS_BIND, ""); err != nil {
	87	+	return err
	88	+	}
	89	+	if err := mount("", "/mnt/newroot"+sandbox.GuestExecDir, "", unix.MS_BIND\|unix.MS_REMOUNT\|unix.MS_RDONLY, ""); err != nil {
	90	+	return err
	91	+	}
	92	+	for _, d := range []string{"/proc", "/sys", "/dev"} {
	93	+	if err := mount(d, "/mnt/newroot"+d, "", unix.MS_MOVE, ""); err != nil {
	94	+	return err
	95	+	}
	96	+	}
	97	+	if err := unix.Chdir("/mnt/newroot"); err != nil {
	98	+	return err
	99	+	}
	100	+	if err := unix.PivotRoot(".", ".oldroot"); err != nil {
	101	+	return fmt.Errorf("pivot_root: %w", err)
	102	+	}
	103	+	if err := unix.Chroot("."); err != nil {
	104	+	return err
	105	+	}
	106	+	if err := unix.Chdir("/"); err != nil {
	107	+	return err
	108	+	}
	109	+	_ = unix.Unmount("/.oldroot", unix.MNT_DETACH)
	110	+
	111	+	for _, m := range []struct {
	112	+	src, dst, typ string
	113	+	flags uintptr
	114	+	data string
	115	+	}{
	116	+	{"devpts", "/dev/pts", "devpts", 0, "gid=5,mode=620,ptmxmode=666"},
	117	+	{"tmpfs", "/dev/shm", "tmpfs", unix.MS_NOSUID \| unix.MS_NODEV, ""},
	118	+	{"tmpfs", "/run", "tmpfs", unix.MS_NOSUID \| unix.MS_NODEV, "mode=755"},
	119	+	{"tmpfs", "/tmp", "tmpfs", unix.MS_NOSUID \| unix.MS_NODEV, "mode=1777"},
	120	+	{"cgroup2", "/sys/fs/cgroup", "cgroup2", unix.MS_NOSUID \| unix.MS_NODEV \| unix.MS_NOEXEC, ""},
	121	+	} {
	122	+	if err := os.MkdirAll(m.dst, 0o755); err != nil {
	123	+	return err
	124	+	}
	125	+	if err := mount(m.src, m.dst, m.typ, m.flags, m.data); err != nil {
	126	+	// devpts and shm failures are not fatal for the stub.
	127	+	log.Warn().Err(err).Str("target", m.dst).Msg("init: mount")
	128	+	}
	129	+	}
	130	+	_ = os.Chmod("/tmp", 0o1777)
	131	+
	132	+	gw := args[sandbox.CmdlineGateway]
	133	+	if err := writeResolvConf(gw); err != nil {
	134	+	return err
	135	+	}
	136	+	_ = os.WriteFile("/etc/hostname", []byte("ermite-job\n"), 0o644)
	137	+	_ = os.WriteFile("/etc/hosts", []byte("127.0.0.1 localhost ermite-job\n"), 0o644)
	138	+	_ = unix.Sethostname([]byte("ermite-job"))
	139	+	if err := loUp(); err != nil {
	140	+	log.Warn().Err(err).Msg("init: loopback")
	141	+	}
	142	+	// IPv6 is disabled on the kernel command line (ipv6.disable=1); this is a
	143	+	// second line of defence when the module is built in but not disabled.
	144	+	_ = os.WriteFile("/proc/sys/net/ipv6/conf/all/disable_ipv6", []byte("1\n"), 0o644)
	145	+	return nil
	146	+	}
	147	+
	148	+	// writeResolvConf points the guest at the gateway, where the runner's DNS
	149	+	// forwarder listens. /etc/resolv.conf may be a dangling symlink in the image
	150	+	// (systemd-resolved stub): replace it.
	151	+	func writeResolvConf(gw string) error {
	152	+	if gw == "" {
	153	+	return fmt.Errorf("kernel command line has no %s", sandbox.CmdlineGateway)
	154	+	}
	155	+	_ = os.Remove("/etc/resolv.conf")
	156	+	return os.WriteFile("/etc/resolv.conf", []byte("nameserver "+gw+"\noptions timeout:2 attempts:2\n"), 0o644)
	157	+	}
	158	+
	159	+	func mount(src, dst, typ string, flags uintptr, data string) error {
	160	+	if err := os.MkdirAll(dst, 0o755); err != nil && !os.IsExist(err) {
	161	+	// Read-only boot root: the mountpoint must already exist.
	162	+	if _, serr := os.Stat(dst); serr != nil {
	163	+	return fmt.Errorf("mountpoint %s: %w", dst, err)
	164	+	}
	165	+	}
	166	+	if err := unix.Mount(src, dst, typ, flags, data); err != nil {
	167	+	return fmt.Errorf("mount %s on %s (%s): %w", src, dst, strings.TrimSpace(typ), err)
	168	+	}
	169	+	return nil
	170	+	}
	171	+
	172	+	// loUp brings the loopback interface up (the kernel leaves it down).
	173	+	func loUp() error {
	174	+	fd, err := unix.Socket(unix.AF_INET, unix.SOCK_DGRAM\|unix.SOCK_CLOEXEC, 0)
	175	+	if err != nil {
	176	+	return err
	177	+	}
	178	+	defer func() { _ = unix.Close(fd) }()
	179	+	ifr, err := unix.NewIfreq("lo")
	180	+	if err != nil {
	181	+	return err
	182	+	}
	183	+	if err := unix.IoctlIfreq(fd, unix.SIOCGIFFLAGS, ifr); err != nil {
	184	+	return err
	185	+	}
	186	+	ifr.SetUint16(ifr.Uint16() \| unix.IFF_UP)
	187	+	return unix.IoctlIfreq(fd, unix.SIOCSIFFLAGS, ifr)
	188	+	}
	189	+
	190	+	// isMounted reports whether path is a mount point listed in /proc/self/mountinfo.
	191	+	func isMounted(path string) bool {
	192	+	b, err := os.ReadFile("/proc/self/mountinfo")
	193	+	if err != nil {
	194	+	return false
	195	+	}
	196	+	for _, line := range strings.Split(string(b), "\n") {
	197	+	f := strings.Fields(line)
	198	+	if len(f) > 4 && f[4] == path {
	199	+	return true
	200	+	}
	201	+	}
	202	+	return false
	203	+	}

A internal/netpolicy/netns_test.go +202 -0

@@ -0,0 +1,202 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package netpolicy
	4	+
	5	+	import (
	6	+	"context"
	7	+	"fmt"
	8	+	"os"
	9	+	"os/exec"
	10	+	"strings"
	11	+	"testing"
	12	+	"time"
	13	+
	14	+	"bnema.dev/ermite-runner/internal/cmdrun"
	15	+	)
	16	+
	17	+	// TestRulesetBetweenTwoGuestsAndAForgedSource loads the real ruleset into the
	18	+	// kernel and sends real packets: two guests (ertap0, ertap1) and an outside
	19	+	// host hang off veth pairs, each in its own network namespace. It needs
	20	+	// CAP_NET_ADMIN, which scripts/it-vm-userns.sh provides without root:
	21	+	//
	22	+	// ERMITE_RUNNER_IT_NET=1 unshare --user --map-root-user --net go test -run Netns ./internal/netpolicy
	23	+	func TestRulesetBetweenTwoGuestsAndAForgedSource(t *testing.T) {
	24	+	if os.Getenv("ERMITE_RUNNER_IT_NET") != "1" {
	25	+	t.Skip("set ERMITE_RUNNER_IT_NET=1 inside a user+network namespace (scripts/it-vm-userns.sh)")
	26	+	}
	27	+	for _, c := range []string{"ip", "nft", "nsenter", "unshare", "python3"} {
	28	+	if _, err := exec.LookPath(c); err != nil {
	29	+	t.Skipf("%s not found", c)
	30	+	}
	31	+	}
	32	+	sh := func(args ...string) string {
	33	+	t.Helper()
	34	+	out, err := exec.Command(args[0], args[1:]...).CombinedOutput()
	35	+	if err != nil {
	36	+	t.Fatalf("%s: %v: %s", strings.Join(args, " "), err, out)
	37	+	}
	38	+	return strings.TrimSpace(string(out))
	39	+	}
	40	+	if out, err := exec.Command("ip", "link", "add", "ertapprobe", "type", "dummy").CombinedOutput(); err != nil {
	41	+	t.Skipf("no CAP_NET_ADMIN: %v: %s", err, out)
	42	+	}
	43	+	sh("ip", "link", "del", "ertapprobe")
	44	+	sh("ip", "link", "set", "lo", "up")
	45	+
	46	+	// Three machines: guest A, guest B, an outside host.
	47	+	pid := map[string]int{}
	48	+	for _, n := range []string{"a", "b", "ext"} {
	49	+	c := exec.Command("unshare", "--net", "sleep", "120")
	50	+	if err := c.Start(); err != nil {
	51	+	t.Fatal(err)
	52	+	}
	53	+	pid[n] = c.Process.Pid
	54	+	t.Cleanup(func() { _ = c.Process.Kill(); _ = c.Wait() })
	55	+	}
	56	+	time.Sleep(300 * time.Millisecond)
	57	+	in := func(n string, args ...string) string {
	58	+	t.Helper()
	59	+	return sh(append([]string{"nsenter", fmt.Sprintf("--net=/proc/%d/ns/net", pid[n])}, args...)...)
	60	+	}
	61	+	link := func(host, n string) {
	62	+	sh("ip", "link", "add", host, "type", "veth", "peer", "name", "v0", "netns", fmt.Sprint(pid[n]))
	63	+	in(n, "ip", "link", "set", "lo", "up")
	64	+	in(n, "ip", "link", "set", "v0", "up")
	65	+	sh("sysctl", "-qw", "net.ipv4.conf."+host+".rp_filter=0")
	66	+	}
	67	+	link("ertap0", "a")
	68	+	link("ertap1", "b")
	69	+	link("extif", "ext")
	70	+	sh("sysctl", "-qw", "net.ipv4.conf.all.rp_filter=0")
	71	+	sh("sysctl", "-qw", "net.ipv4.ip_forward=1")
	72	+
	73	+	gA := guest(t, "ertap0", 0) // 10.200.0.1 / .2
	74	+	gB := guest(t, "ertap1", 1) // 10.200.0.5 / .6
	75	+	for _, g := range []Guest{gA, gB} {
	76	+	sh("ip", "addr", "add", g.Gateway.String()+"/30", "dev", g.Iface)
	77	+	sh("ip", "link", "set", g.Iface, "up")
	78	+	}
	79	+	sh("ip", "addr", "add", "203.0.114.1/24", "dev", "extif")
	80	+	sh("ip", "link", "set", "extif", "up")
	81	+	in("a", "ip", "addr", "add", gA.Addr.String()+"/30", "dev", "v0")
	82	+	in("a", "ip", "route", "add", "default", "via", gA.Gateway.String())
	83	+	in("b", "ip", "addr", "add", gB.Addr.String()+"/30", "dev", "v0")
	84	+	in("b", "ip", "route", "add", "default", "via", gB.Gateway.String())
	85	+	in("ext", "ip", "addr", "add", "203.0.114.2/24", "dev", "v0")
	86	+	in("ext", "ip", "route", "add", "default", "via", "203.0.114.1")
	87	+	// Guest A holds guest B's address too, to be able to forge it.
	88	+	in("a", "ip", "addr", "add", gB.Addr.String()+"/32", "dev", "lo")
	89	+
	90	+	m := NewManager(cmdrun.Exec{}, "nft")
	91	+	ctx := context.Background()
	92	+	t.Cleanup(func() { _ = m.Reset(ctx) })
	93	+	for _, g := range []Guest{gA, gB} {
	94	+	if err := m.Add(ctx, g); err != nil {
	95	+	t.Fatal(err)
	96	+	}
	97	+	}
	98	+
	99	+	// An outside UDP server: logs each payload with the source it saw, and
	100	+	// answers "pong".
	101	+	logf := t.TempDir() + "/ext.log"
	102	+	const server = `
	103	+	import socket,sys
	104	+	s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM); s.bind(("0.0.0.0",9999))
	105	+	f=open(sys.argv[1],"a",buffering=1)
	106	+	while True:
	107	+	d,a=s.recvfrom(100); f.write(d.decode()+" from "+a[0]+"\n"); s.sendto(b"pong",a)
	108	+	`
	109	+	srv := exec.Command("nsenter", fmt.Sprintf("--net=/proc/%d/ns/net", pid["ext"]), "python3", "-c", server, logf)
	110	+	if err := srv.Start(); err != nil {
	111	+	t.Fatal(err)
	112	+	}
	113	+	t.Cleanup(func() { _ = srv.Process.Kill(); _ = srv.Wait() })
	114	+	time.Sleep(500 * time.Millisecond)
	115	+
	116	+	// udp sends one datagram from machine n (optionally bound to src:sport) and
	117	+	// reports whether an answer came back.
	118	+	const client = `
	119	+	import socket,sys
	120	+	dst,payload,src,sport=sys.argv[1],sys.argv[2],sys.argv[3],int(sys.argv[4])
	121	+	s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM); s.settimeout(2)
	122	+	if src: s.bind((src,sport))
	123	+	s.sendto(payload.encode(),(dst,int(sys.argv[5]) if len(sys.argv)>5 else 9999))
	124	+	try: s.recv(10); print("answer")
	125	+	except Exception: print("silence")
	126	+	`
	127	+	udp := func(n, dst, payload, src string, sport int, dport int) string {
	128	+	return in(n, "python3", "-c", client, dst, payload, src, fmt.Sprint(sport), fmt.Sprint(dport))
	129	+	}
	130	+	seen := func() string { b, _ := os.ReadFile(logf); return string(b) }
	131	+
	132	+	// Control: a guest reaches the outside host and gets the reply (NAT and the
	133	+	// established path work, so the drops below are the policy, not a broken rig).
	134	+	if got := udp("a", "203.0.114.2", "ctl-a", "", 0, 9999); got != "answer" {
	135	+	t.Fatalf("control: guest A cannot reach the outside host: %s", got)
	136	+	}
	137	+
	138	+	// Guests cannot reach each other, directly or through the other's gateway.
	139	+	if got := udp("a", gB.Addr.String(), "a-to-b", "", 0, 9999); got != "silence" {
	140	+	t.Errorf("guest A reached guest B: %s", got)
	141	+	}
	142	+	// Forged reply (S1): B opens a flow to the outside host from port 5555 and
	143	+	// keeps listening. Guest A then forges a reply of that flow: source = the
	144	+	// outside host's address and port, destination = the host's masquerade
	145	+	// address and B's NAT port. Conntrack would map it onto B's flow and deliver
	146	+	// it to B as "established" traffic if an established-accept came before the
	147	+	// source check.
	148	+	in("a", "ip", "addr", "add", "203.0.114.2/32", "dev", "lo")
	149	+	bLog := t.TempDir() + "/b.log"
	150	+	const listener = `
	151	+	import socket,sys
	152	+	s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM); s.settimeout(6)
	153	+	s.bind(("10.200.0.6",5555)); s.sendto(b"open-flow",("203.0.114.2",9999))
	154	+	f=open(sys.argv[1],"a",buffering=1)
	155	+	try:
	156	+	while True:
	157	+	d,a=s.recvfrom(100); f.write(d.decode()+" from "+a[0]+"\n")
	158	+	except Exception: pass
	159	+	`
	160	+	bProc := exec.Command("nsenter", fmt.Sprintf("--net=/proc/%d/ns/net", pid["b"]), "python3", "-c", listener, bLog)
	161	+	if err := bProc.Start(); err != nil {
	162	+	t.Fatal(err)
	163	+	}
	164	+	time.Sleep(1500 * time.Millisecond)
	165	+	if got := udp("a", "203.0.114.1", "forged-reply-to-b", "203.0.114.2", 9999, 5555); got != "silence" {
	166	+	t.Errorf("unexpected answer to the forgery: %s", got)
	167	+	}
	168	+	_ = bProc.Wait()
	169	+	if b, _ := os.ReadFile(bLog); strings.Contains(string(b), "forged-reply-to-b") {
	170	+	t.Errorf("guest B received a reply forged by guest A through conntrack:\n%s", b)
	171	+	}
	172	+	// A forged source address on a plain forwarded packet is dropped as well.
	173	+	if got := udp("a", "203.0.114.2", "forged-src", gB.Addr.String(), 7777, 9999); got != "silence" {
	174	+	t.Errorf("a packet with another guest's source address got through: %s", got)
	175	+	}
	176	+	// S6: guest A cannot use guest B's DNS gateway, nor the host's other ports.
	177	+	if got := udp("a", gB.Gateway.String(), "dns-a-to-b-gw", "", 0, 53); got != "silence" {
	178	+	t.Errorf("guest A reached guest B's gateway on 53: %s", got)
	179	+	}
	180	+	// The outside host cannot open connections to a guest.
	181	+	if got := in("ext", "python3", "-c", client, gA.Addr.String(), "ext-to-a", "", "0", "9999"); got != "silence" {
	182	+	t.Errorf("the outside host reached guest A: %s", got)
	183	+	}
	184	+	// Special-use ranges (S10) are dropped even though the rig could route them.
	185	+	in("ext", "ip", "addr", "add", "198.51.100.7/32", "dev", "lo")
	186	+	sh("ip", "route", "add", "198.51.100.7/32", "via", "203.0.114.2")
	187	+	if got := udp("a", "198.51.100.7", "doc-range", "", 0, 9999); got != "silence" {
	188	+	t.Errorf("TEST-NET-2 reached: %s", got)
	189	+	}
	190	+
	191	+	out := seen()
	192	+	for _, bad := range []string{"forged-src", "a-to-b", "ext-to-a", "doc-range"} {
	193	+	if strings.Contains(out, bad) {
	194	+	t.Errorf("outside host saw %q:\n%s", bad, out)
	195	+	}
	196	+	}
	197	+	for _, good := range []string{"ctl-a", "open-flow"} {
	198	+	if !strings.Contains(out, good) {
	199	+	t.Errorf("outside host did not see %q:\n%s", good, out)
	200	+	}
	201	+	}
	202	+	}

A internal/netpolicy/netpolicy.go +268 -0

@@ -0,0 +1,268 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package netpolicy generates and applies the nftables policy of job sandboxes.
	4	+	//
	5	+	// All rules live in one dedicated table, "inet ermite_runner", that is replaced
	6	+	// atomically (one `nft -f -` transaction) whenever a sandbox is added or
	7	+	// removed, and deleted when none is left. Nothing outside that table is touched.
	8	+	//
	9	+	// Policy for every guest interface (a TAP in a micro-VM, a bridge in a
	10	+	// container):
	11	+	// - to the host itself (input hook): only DNS (udp/tcp 53) to the gateway
	12	+	// address of that guest, where the runner's DNS forwarder listens; every
	13	+	// other packet is dropped, and port 53 of a gateway is closed to every
	14	+	// other interface;
	15	+	// - IPv6 from the guest is dropped (the guest also disables IPv6);
	16	+	// - forwarded traffic must carry the guest's own source address;
	17	+	// - forwarded traffic to RFC 1918, CGNAT 100.64.0.0/10 (VPN overlays),
	18	+	// link-local 169.254.0.0/16 (cloud metadata 169.254.169.254), loopback,
	19	+	// "this network", multicast and reserved ranges is dropped;
	20	+	// - traffic towards a guest interface that does not belong to an established
	21	+	// connection is dropped, so guests cannot reach each other;
	22	+	// - everything else is NATed (masquerade) to the internet.
	23	+	package netpolicy
	24	+
	25	+	import (
	26	+	"context"
	27	+	"errors"
	28	+	"fmt"
	29	+	"net/netip"
	30	+	"sort"
	31	+	"strings"
	32	+	"sync"
	33	+
	34	+	"bnema.dev/ermite-runner/internal/cmdrun"
	35	+	)
	36	+
	37	+	// Table is the nftables table owned by the runner.
	38	+	const Table = "ermite_runner"
	39	+
	40	+	// IfacePrefixes are the interface name prefixes of sandboxes; wildcard rules
	41	+	// are generated for them. TAP devices of micro-VMs, bridges of containers.
	42	+	var IfacePrefixes = []string{"ertap", "erbr"}
	43	+
	44	+	// blockedV4 are the destinations a guest may never reach.
	45	+	var blockedV4 = []string{
	46	+	"0.0.0.0/8", // "this network"
	47	+	"10.0.0.0/8", // RFC 1918
	48	+	"100.64.0.0/10", // RFC 6598 shared address space (CGNAT, VPN overlays)
	49	+	"127.0.0.0/8", // loopback
	50	+	"169.254.0.0/16", // link-local, incl. the 169.254.169.254 metadata service
	51	+	"172.16.0.0/12", // RFC 1918
	52	+	"192.0.0.0/24", // IETF protocol assignments
	53	+	"192.0.2.0/24", // TEST-NET-1
	54	+	"192.168.0.0/16", // RFC 1918
	55	+	"198.18.0.0/15", // benchmarking
	56	+	"198.51.100.0/24", // TEST-NET-2
	57	+	"203.0.113.0/24", // TEST-NET-3
	58	+	"224.0.0.0/4", // multicast
	59	+	"240.0.0.0/4", // reserved, incl. limited broadcast
	60	+	}
	61	+
	62	+	// prefixSet renders IfacePrefixes as an nft anonymous set of interface wildcards.
	63	+	func prefixSet() string {
	64	+	q := make([]string, len(IfacePrefixes))
	65	+	for i, p := range IfacePrefixes {
	66	+	q[i] = `"` + p + `*"`
	67	+	}
	68	+	return "{ " + strings.Join(q, ", ") + " }"
	69	+	}
	70	+
	71	+	// Guest is one sandbox network attachment.
	72	+	type Guest struct {
	73	+	Iface string // host-side interface: TAP or bridge name
	74	+	Gateway netip.Addr // host address on Iface; the DNS forwarder listens here
	75	+	Addr netip.Addr // the guest's only allowed source address
	76	+	}
	77	+
	78	+	// Validate checks the fields used in generated rules.
	79	+	func (g Guest) Validate() error {
	80	+	if !validIface(g.Iface) {
	81	+	return fmt.Errorf("netpolicy: invalid interface name %q", g.Iface)
	82	+	}
	83	+	if !g.Gateway.Is4() \|\| !g.Addr.Is4() {
	84	+	return errors.New("netpolicy: guest and gateway must be IPv4 addresses")
	85	+	}
	86	+	return nil
	87	+	}
	88	+
	89	+	func validIface(s string) bool {
	90	+	if s == "" \|\| len(s) > 15 {
	91	+	return false
	92	+	}
	93	+	for _, r := range s {
	94	+	if (r < 'a' \|\| r > 'z') && (r < '0' \|\| r > '9') && r != '-' && r != '_' {
	95	+	return false
	96	+	}
	97	+	}
	98	+	return true
	99	+	}
	100	+
	101	+	// Net is the /30 given to one sandbox.
	102	+	type Net struct {
	103	+	Prefix netip.Prefix // the /30
	104	+	Gateway netip.Addr // host side (.1)
	105	+	Guest netip.Addr // guest side (.2)
	106	+	}
	107	+
	108	+	// Subnet returns the slot-th /30 of base (an IPv4 prefix of at most /30).
	109	+	func Subnet(base netip.Prefix, slot int) (Net, error) {
	110	+	if !base.Addr().Is4() \|\| base.Bits() > 30 \|\| slot < 0 {
	111	+	return Net{}, fmt.Errorf("netpolicy: bad base %s or slot %d", base, slot)
	112	+	}
	113	+	if slot >= 1<<(30-base.Bits()) {
	114	+	return Net{}, fmt.Errorf("netpolicy: slot %d does not fit in %s", slot, base)
	115	+	}
	116	+	b := base.Masked().Addr().As4()
	117	+	v := uint32(b[0])<<24 \| uint32(b[1])<<16 \| uint32(b[2])<<8 \| uint32(b[3])
	118	+	v += uint32(slot) * 4
	119	+	addr := func(off uint32) netip.Addr {
	120	+	x := v + off
	121	+	return netip.AddrFrom4([4]byte{byte(x >> 24), byte(x >> 16), byte(x >> 8), byte(x)})
	122	+	}
	123	+	return Net{Prefix: netip.PrefixFrom(addr(0), 30), Gateway: addr(1), Guest: addr(2)}, nil
	124	+	}
	125	+
	126	+	// Ruleset returns the nft script that atomically replaces the runner table with
	127	+	// the policy for guests. With no guest it only deletes the table. The output is
	128	+	// deterministic (guests sorted by interface).
	129	+	func Ruleset(guests []Guest) string {
	130	+	var b strings.Builder
	131	+	fmt.Fprintf(&b, "table inet %s\ndelete table inet %s\n", Table, Table)
	132	+	if len(guests) == 0 {
	133	+	return b.String()
	134	+	}
	135	+	gs := append([]Guest(nil), guests...)
	136	+	sort.Slice(gs, func(i, j int) bool { return gs[i].Iface < gs[j].Iface })
	137	+
	138	+	fmt.Fprintf(&b, "table inet %s {\n", Table)
	139	+	b.WriteString("\tset blocked_v4 {\n\t\ttype ipv4_addr\n\t\tflags interval\n\t\telements = { ")
	140	+	b.WriteString(strings.Join(blockedV4, ", "))
	141	+	b.WriteString(" }\n\t}\n")
	142	+
	143	+	b.WriteString("\tchain input {\n\t\ttype filter hook input priority filter; policy accept;\n")
	144	+	for _, p := range IfacePrefixes {
	145	+	fmt.Fprintf(&b, "\t\tiifname \"%s*\" meta nfproto ipv6 drop\n", p)
	146	+	}
	147	+	// The DNS forwarder of a guest answers that guest only: port 53 of a gateway
	148	+	// address is dropped from every other interface (other guests, the LAN).
	149	+	for _, g := range gs {
	150	+	for _, proto := range []string{"udp", "tcp"} {
	151	+	fmt.Fprintf(&b, "\t\tip daddr %s %s dport 53 iifname != \"%s\" drop\n", g.Gateway, proto, g.Iface)
	152	+	}
	153	+	}
	154	+	for _, g := range gs {
	155	+	for _, proto := range []string{"udp", "tcp"} {
	156	+	fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s %s dport 53 accept\n", g.Iface, g.Addr, g.Gateway, proto)
	157	+	}
	158	+	fmt.Fprintf(&b, "\t\tiifname \"%s\" drop\n", g.Iface)
	159	+	}
	160	+	// An interface with our prefix that no live guest owns (a leftover) is
	161	+	// dropped as well.
	162	+	for _, p := range IfacePrefixes {
	163	+	fmt.Fprintf(&b, "\t\tiifname \"%s*\" drop\n", p)
	164	+	}
	165	+	b.WriteString("\t}\n")
	166	+
	167	+	// Forward chain. Order matters: everything a guest sends is checked (family,
	168	+	// source address, destination ranges) BEFORE any "established" accept, or a
	169	+	// guest could forge the source of another guest's flow and have it accepted
	170	+	// as a reply.
	171	+	b.WriteString("\tchain forward {\n\t\ttype filter hook forward priority filter; policy accept;\n")
	172	+	for _, p := range IfacePrefixes {
	173	+	fmt.Fprintf(&b, "\t\tiifname \"%s*\" meta nfproto ipv6 drop\n", p)
	174	+	}
	175	+	for _, g := range gs {
	176	+	fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr != %s drop\n", g.Iface, g.Addr)
	177	+	fmt.Fprintf(&b, "\t\tiifname \"%s\" ip daddr @blocked_v4 drop\n", g.Iface)
	178	+	}
	179	+	// Replies from outside to a guest; never for packets that come from a guest.
	180	+	b.WriteString("\t\tiifname != " + prefixSet() + " oifname " + prefixSet() + " ct state established,related accept\n")
	181	+	for _, p := range IfacePrefixes {
	182	+	fmt.Fprintf(&b, "\t\toifname \"%s*\" drop\n", p)
	183	+	}
	184	+	for _, g := range gs {
	185	+	fmt.Fprintf(&b, "\t\tiifname \"%s\" accept\n", g.Iface)
	186	+	}
	187	+	for _, p := range IfacePrefixes {
	188	+	fmt.Fprintf(&b, "\t\tiifname \"%s*\" drop\n", p)
	189	+	}
	190	+	b.WriteString("\t}\n")
	191	+
	192	+	b.WriteString("\tchain postrouting {\n\t\ttype nat hook postrouting priority srcnat; policy accept;\n")
	193	+	for _, g := range gs {
	194	+	fmt.Fprintf(&b, "\t\tip saddr %s oifname != \"%s\" masquerade\n", g.Addr, g.Iface)
	195	+	}
	196	+	b.WriteString("\t}\n}\n")
	197	+	return b.String()
	198	+	}
	199	+
	200	+	// Manager applies the policy for the set of live guests.
	201	+	type Manager struct {
	202	+	run cmdrun.Runner
	203	+	nft string
	204	+	mu sync.Mutex
	205	+	guests map[string]Guest
	206	+	}
	207	+
	208	+	// NewManager returns a Manager that runs nft (path or name) through run.
	209	+	func NewManager(run cmdrun.Runner, nft string) *Manager {
	210	+	if nft == "" {
	211	+	nft = "nft"
	212	+	}
	213	+	return &Manager{run: run, nft: nft, guests: map[string]Guest{}}
	214	+	}
	215	+
	216	+	// Add installs (or replaces) the policy of g.
	217	+	func (m *Manager) Add(ctx context.Context, g Guest) error {
	218	+	if err := g.Validate(); err != nil {
	219	+	return err
	220	+	}
	221	+	m.mu.Lock()
	222	+	defer m.mu.Unlock()
	223	+	prev, had := m.guests[g.Iface]
	224	+	m.guests[g.Iface] = g
	225	+	if err := m.applyLocked(ctx); err != nil {
	226	+	if had {
	227	+	m.guests[g.Iface] = prev
	228	+	} else {
	229	+	delete(m.guests, g.Iface)
	230	+	}
	231	+	return err
	232	+	}
	233	+	return nil
	234	+	}
	235	+
	236	+	// Remove drops the policy of iface; the table disappears with the last guest.
	237	+	// Removing an unknown interface is not an error.
	238	+	func (m *Manager) Remove(ctx context.Context, iface string) error {
	239	+	m.mu.Lock()
	240	+	defer m.mu.Unlock()
	241	+	if _, ok := m.guests[iface]; !ok {
	242	+	return nil
	243	+	}
	244	+	delete(m.guests, iface)
	245	+	return m.applyLocked(ctx)
	246	+	}
	247	+
	248	+	// Reset deletes the runner table whatever its content: used at start to reap
	249	+	// the table of a previous run. Safe when the table does not exist.
	250	+	func (m *Manager) Reset(ctx context.Context) error {
	251	+	m.mu.Lock()
	252	+	defer m.mu.Unlock()
	253	+	m.guests = map[string]Guest{}
	254	+	return m.applyLocked(ctx)
	255	+	}
	256	+
	257	+	func (m *Manager) applyLocked(ctx context.Context) error {
	258	+	gs := make([]Guest, 0, len(m.guests))
	259	+	for _, g := range m.guests {
	260	+	gs = append(gs, g)
	261	+	}
	262	+	// `table X` creates the table if missing so that the following delete cannot
	263	+	// fail; the whole script is one transaction.
	264	+	if _, err := m.run.Run(ctx, strings.NewReader(Ruleset(gs)), m.nft, "-f", "-"); err != nil {
	265	+	return fmt.Errorf("netpolicy: apply ruleset: %w", err)
	266	+	}
	267	+	return nil
	268	+	}

A internal/netpolicy/netpolicy_test.go +234 -0

@@ -0,0 +1,234 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package netpolicy
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"flag"
	9	+	"io"
	10	+	"net/netip"
	11	+	"os"
	12	+	"path/filepath"
	13	+	"strings"
	14	+	"testing"
	15	+
	16	+	"bnema.dev/ermite-runner/internal/cmdrun"
	17	+	)
	18	+
	19	+	var update = flag.Bool("update", false, "rewrite golden files")
	20	+
	21	+	func golden(t *testing.T, name, got string) {
	22	+	t.Helper()
	23	+	p := filepath.Join("testdata", name)
	24	+	if *update {
	25	+	if err := os.MkdirAll("testdata", 0o755); err != nil {
	26	+	t.Fatal(err)
	27	+	}
	28	+	if err := os.WriteFile(p, []byte(got), 0o644); err != nil {
	29	+	t.Fatal(err)
	30	+	}
	31	+	}
	32	+	want, err := os.ReadFile(p)
	33	+	if err != nil {
	34	+	t.Fatal(err)
	35	+	}
	36	+	if string(want) != got {
	37	+	t.Fatalf("%s differs from golden (run with -update to refresh):\n%s", name, got)
	38	+	}
	39	+	}
	40	+
	41	+	func guest(t *testing.T, iface string, slot int) Guest {
	42	+	t.Helper()
	43	+	n, err := Subnet(netip.MustParsePrefix("10.200.0.0/16"), slot)
	44	+	if err != nil {
	45	+	t.Fatal(err)
	46	+	}
	47	+	return Guest{Iface: iface, Gateway: n.Gateway, Addr: n.Guest}
	48	+	}
	49	+
	50	+	func TestRulesetGolden(t *testing.T) {
	51	+	golden(t, "empty.nft", Ruleset(nil))
	52	+	golden(t, "one.nft", Ruleset([]Guest{guest(t, "ertap0", 0)}))
	53	+	// Order of input must not matter.
	54	+	two := Ruleset([]Guest{guest(t, "erbr1", 1), guest(t, "ertap0", 0)})
	55	+	if two != Ruleset([]Guest{guest(t, "ertap0", 0), guest(t, "erbr1", 1)}) {
	56	+	t.Fatal("ruleset depends on guest order")
	57	+	}
	58	+	golden(t, "two.nft", two)
	59	+	}
	60	+
	61	+	func TestRulesetPolicyInvariants(t *testing.T) {
	62	+	rs := Ruleset([]Guest{guest(t, "ertap0", 0)})
	63	+	for _, want := range []string{
	64	+	"169.254.0.0/16", "100.64.0.0/10", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
	65	+	`iifname "ertap0" drop`, "dport 53 accept", "masquerade", `iifname "ertap*" meta nfproto ipv6 drop`,
	66	+	} {
	67	+	if !strings.Contains(rs, want) {
	68	+	t.Errorf("ruleset lacks %q", want)
	69	+	}
	70	+	}
	71	+	// The drop for the host must come after the DNS accepts and the guest
	72	+	// table must never use a drop policy (it would break other tables' traffic).
	73	+	if strings.Contains(rs, "policy drop") {
	74	+	t.Error("base chains must keep policy accept")
	75	+	}
	76	+	if strings.Index(rs, "dport 53 accept") > strings.Index(rs, `iifname "ertap0" drop`) {
	77	+	t.Error("DNS accept must precede the catch-all drop")
	78	+	}
	79	+	}
	80	+
	81	+	// S1: nothing a guest sends may be accepted by a conntrack rule before the
	82	+	// per-guest checks: a forged "reply" from guest A to guest B's flow would
	83	+	// otherwise be accepted as established traffic.
	84	+	func TestForwardChainChecksGuestTrafficBeforeEstablished(t *testing.T) {
	85	+	rs := Ruleset([]Guest{guest(t, "ertap0", 0), guest(t, "erbr1", 1)})
	86	+	fwd := rs[strings.Index(rs, "chain forward"):strings.Index(rs, "chain postrouting")]
	87	+	lines := strings.Split(fwd, "\n")
	88	+	idx := func(sub string) int {
	89	+	for i, l := range lines {
	90	+	if strings.Contains(l, sub) {
	91	+	return i
	92	+	}
	93	+	}
	94	+	t.Fatalf("forward chain lacks %q:\n%s", sub, fwd)
	95	+	return -1
	96	+	}
	97	+	est := idx("ct state established,related accept")
	98	+	for _, must := range []string{
	99	+	`iifname "ertap" meta nfproto ipv6 drop`, `iifname "erbr" meta nfproto ipv6 drop`,
	100	+	`iifname "ertap0" ip saddr != 10.200.0.2 drop`, `iifname "ertap0" ip daddr @blocked_v4 drop`,
	101	+	`iifname "erbr1" ip saddr != 10.200.0.6 drop`, `iifname "erbr1" ip daddr @blocked_v4 drop`,
	102	+	} {
	103	+	if idx(must) > est {
	104	+	t.Errorf("%q must come before the established accept", must)
	105	+	}
	106	+	}
	107	+	// And the established accept itself never applies to packets that enter
	108	+	// through a sandbox interface.
	109	+	if !strings.Contains(lines[est], `iifname != { "ertap", "erbr" }`) {
	110	+	t.Errorf("established accept must exclude sandbox ingress: %s", lines[est])
	111	+	}
	112	+	}
	113	+
	114	+	// S6: port 53 of a gateway address is closed to every interface but its guest's.
	115	+	func TestInputChainClosesGatewayDNSToOtherInterfaces(t *testing.T) {
	116	+	rs := Ruleset([]Guest{guest(t, "ertap0", 0), guest(t, "ertap1", 1)})
	117	+	for _, want := range []string{
	118	+	`ip daddr 10.200.0.1 udp dport 53 iifname != "ertap0" drop`,
	119	+	`ip daddr 10.200.0.1 tcp dport 53 iifname != "ertap0" drop`,
	120	+	`ip daddr 10.200.0.5 udp dport 53 iifname != "ertap1" drop`,
	121	+	`ip daddr 10.200.0.5 tcp dport 53 iifname != "ertap1" drop`,
	122	+	} {
	123	+	if !strings.Contains(rs, want) {
	124	+	t.Errorf("lacks %q", want)
	125	+	}
	126	+	}
	127	+	}
	128	+
	129	+	// S10: documentation, test-net and benchmarking ranges are blocked too.
	130	+	func TestBlockedSetCoversSpecialUseRanges(t *testing.T) {
	131	+	rs := Ruleset([]Guest{guest(t, "ertap0", 0)})
	132	+	for _, r := range []string{"192.0.0.0/24", "192.0.2.0/24", "198.18.0.0/15", "198.51.100.0/24", "203.0.113.0/24", "224.0.0.0/4", "240.0.0.0/4"} {
	133	+	if !strings.Contains(rs, r) {
	134	+	t.Errorf("blocked set lacks %s", r)
	135	+	}
	136	+	}
	137	+	}
	138	+
	139	+	func TestSubnet(t *testing.T) {
	140	+	base := netip.MustParsePrefix("10.200.0.0/16")
	141	+	n, err := Subnet(base, 3)
	142	+	if err != nil {
	143	+	t.Fatal(err)
	144	+	}
	145	+	if n.Prefix.String() != "10.200.0.12/30" \|\| n.Gateway.String() != "10.200.0.13" \|\| n.Guest.String() != "10.200.0.14" {
	146	+	t.Fatalf("got %+v", n)
	147	+	}
	148	+	n, _ = Subnet(base, 64)
	149	+	if n.Gateway.String() != "10.200.1.1" {
	150	+	t.Fatalf("carry: %+v", n)
	151	+	}
	152	+	if _, err := Subnet(base, 1<<14); err == nil {
	153	+	t.Fatal("slot beyond the prefix accepted")
	154	+	}
	155	+	if _, err := Subnet(netip.MustParsePrefix("10.0.0.0/31"), 0); err == nil {
	156	+	t.Fatal("prefix longer than /30 accepted")
	157	+	}
	158	+	if _, err := Subnet(base, -1); err == nil {
	159	+	t.Fatal("negative slot accepted")
	160	+	}
	161	+	}
	162	+
	163	+	func TestGuestValidate(t *testing.T) {
	164	+	g := guest(t, "ertap0", 0)
	165	+	if err := g.Validate(); err != nil {
	166	+	t.Fatal(err)
	167	+	}
	168	+	for _, bad := range []string{"", "a b", `x"y`, "a;b", "waytoolonginterfacename", "x\ny"} {
	169	+	g.Iface = bad
	170	+	if g.Validate() == nil {
	171	+	t.Errorf("iface %q accepted", bad)
	172	+	}
	173	+	}
	174	+	g.Iface = "ok"
	175	+	g.Addr = netip.MustParseAddr("::1")
	176	+	if g.Validate() == nil {
	177	+	t.Error("IPv6 guest accepted")
	178	+	}
	179	+	}
	180	+
	181	+	func TestManager(t *testing.T) {
	182	+	f := &cmdrun.Fake{}
	183	+	m := NewManager(f, "nft")
	184	+	ctx := context.Background()
	185	+	if err := m.Add(ctx, guest(t, "ertap0", 0)); err != nil {
	186	+	t.Fatal(err)
	187	+	}
	188	+	if err := m.Add(ctx, guest(t, "ertap1", 1)); err != nil {
	189	+	t.Fatal(err)
	190	+	}
	191	+	if err := m.Remove(ctx, "ertap0"); err != nil {
	192	+	t.Fatal(err)
	193	+	}
	194	+	if err := m.Remove(ctx, "unknown"); err != nil {
	195	+	t.Fatal(err)
	196	+	}
	197	+	if len(f.Calls) != 3 {
	198	+	t.Fatalf("want 3 nft calls (unknown iface is a no-op), got %v", f.Lines())
	199	+	}
	200	+	last := f.Calls[2]
	201	+	if last.Name != "nft" \|\| strings.Join(last.Args, " ") != "-f -" {
	202	+	t.Fatalf("bad command %v", last)
	203	+	}
	204	+	if strings.Contains(last.Stdin, "ertap0") \|\| !strings.Contains(last.Stdin, "ertap1") {
	205	+	t.Fatalf("after removing ertap0 only ertap1 must remain:\n%s", last.Stdin)
	206	+	}
	207	+	if err := m.Remove(ctx, "ertap1"); err != nil {
	208	+	t.Fatal(err)
	209	+	}
	210	+	if got := f.Calls[3].Stdin; got != Ruleset(nil) {
	211	+	t.Fatalf("last removal must delete the table, got:\n%s", got)
	212	+	}
	213	+	if err := m.Reset(ctx); err != nil {
	214	+	t.Fatal(err)
	215	+	}
	216	+	}
	217	+
	218	+	func TestManagerRollsBackOnError(t *testing.T) {
	219	+	boom := errors.New("boom")
	220	+	f := &cmdrun.Fake{Handle: func(cmdrun.Call) ([]byte, error) { return nil, boom }}
	221	+	m := NewManager(f, "")
	222	+	if err := m.Add(context.Background(), guest(t, "ertap0", 0)); !errors.Is(err, boom) {
	223	+	t.Fatalf("got %v", err)
	224	+	}
	225	+	f.Handle = nil
	226	+	if err := m.Remove(context.Background(), "ertap0"); err != nil {
	227	+	t.Fatal(err)
	228	+	}
	229	+	if len(f.Calls) != 1 {
	230	+	t.Fatalf("a guest whose Add failed must not be tracked: %v", f.Lines())
	231	+	}
	232	+	}
	233	+
	234	+	var _ io.Reader = strings.NewReader("")

A internal/netpolicy/testdata/empty.nft +2 -0

@@ -0,0 +1,2 @@
	1	+	table inet ermite_runner
	2	+	delete table inet ermite_runner

A internal/netpolicy/testdata/one.nft +38 -0

@@ -0,0 +1,38 @@
	1	+	table inet ermite_runner
	2	+	delete table inet ermite_runner
	3	+	table inet ermite_runner {
	4	+	set blocked_v4 {
	5	+	type ipv4_addr
	6	+	flags interval
	7	+	elements = { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4 }
	8	+	}
	9	+	chain input {
	10	+	type filter hook input priority filter; policy accept;
	11	+	iifname "ertap*" meta nfproto ipv6 drop
	12	+	iifname "erbr*" meta nfproto ipv6 drop
	13	+	ip daddr 10.200.0.1 udp dport 53 iifname != "ertap0" drop
	14	+	ip daddr 10.200.0.1 tcp dport 53 iifname != "ertap0" drop
	15	+	iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 udp dport 53 accept
	16	+	iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 53 accept
	17	+	iifname "ertap0" drop
	18	+	iifname "ertap*" drop
	19	+	iifname "erbr*" drop
	20	+	}
	21	+	chain forward {
	22	+	type filter hook forward priority filter; policy accept;
	23	+	iifname "ertap*" meta nfproto ipv6 drop
	24	+	iifname "erbr*" meta nfproto ipv6 drop
	25	+	iifname "ertap0" ip saddr != 10.200.0.2 drop
	26	+	iifname "ertap0" ip daddr @blocked_v4 drop
	27	+	iifname != { "ertap", "erbr" } oifname { "ertap", "erbr" } ct state established,related accept
	28	+	oifname "ertap*" drop
	29	+	oifname "erbr*" drop
	30	+	iifname "ertap0" accept
	31	+	iifname "ertap*" drop
	32	+	iifname "erbr*" drop
	33	+	}
	34	+	chain postrouting {
	35	+	type nat hook postrouting priority srcnat; policy accept;
	36	+	ip saddr 10.200.0.2 oifname != "ertap0" masquerade
	37	+	}
	38	+	}

A internal/netpolicy/testdata/two.nft +47 -0

@@ -0,0 +1,47 @@
	1	+	table inet ermite_runner
	2	+	delete table inet ermite_runner
	3	+	table inet ermite_runner {
	4	+	set blocked_v4 {
	5	+	type ipv4_addr
	6	+	flags interval
	7	+	elements = { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4 }
	8	+	}
	9	+	chain input {
	10	+	type filter hook input priority filter; policy accept;
	11	+	iifname "ertap*" meta nfproto ipv6 drop
	12	+	iifname "erbr*" meta nfproto ipv6 drop
	13	+	ip daddr 10.200.0.5 udp dport 53 iifname != "erbr1" drop
	14	+	ip daddr 10.200.0.5 tcp dport 53 iifname != "erbr1" drop
	15	+	ip daddr 10.200.0.1 udp dport 53 iifname != "ertap0" drop
	16	+	ip daddr 10.200.0.1 tcp dport 53 iifname != "ertap0" drop
	17	+	iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 udp dport 53 accept
	18	+	iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 53 accept
	19	+	iifname "erbr1" drop
	20	+	iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 udp dport 53 accept
	21	+	iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 53 accept
	22	+	iifname "ertap0" drop
	23	+	iifname "ertap*" drop
	24	+	iifname "erbr*" drop
	25	+	}
	26	+	chain forward {
	27	+	type filter hook forward priority filter; policy accept;
	28	+	iifname "ertap*" meta nfproto ipv6 drop
	29	+	iifname "erbr*" meta nfproto ipv6 drop
	30	+	iifname "erbr1" ip saddr != 10.200.0.6 drop
	31	+	iifname "erbr1" ip daddr @blocked_v4 drop
	32	+	iifname "ertap0" ip saddr != 10.200.0.2 drop
	33	+	iifname "ertap0" ip daddr @blocked_v4 drop
	34	+	iifname != { "ertap", "erbr" } oifname { "ertap", "erbr" } ct state established,related accept
	35	+	oifname "ertap*" drop
	36	+	oifname "erbr*" drop
	37	+	iifname "erbr1" accept
	38	+	iifname "ertap0" accept
	39	+	iifname "ertap*" drop
	40	+	iifname "erbr*" drop
	41	+	}
	42	+	chain postrouting {
	43	+	type nat hook postrouting priority srcnat; policy accept;
	44	+	ip saddr 10.200.0.6 oifname != "erbr1" masquerade
	45	+	ip saddr 10.200.0.2 oifname != "ertap0" masquerade
	46	+	}
	47	+	}

A internal/sandbox/config.go +322 -0

@@ -0,0 +1,322 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"errors"
	7	+	"fmt"
	8	+	"net/netip"
	9	+	"os"
	10	+	"path/filepath"
	11	+	"runtime"
	12	+	"strconv"
	13	+	"strings"
	14	+	)
	15	+
	16	+	// Environment variables read by LoadConfig. The other ERMITE_RUNNER_* variables
	17	+	// (URL, NAME, LABELS, ...) belong to the host process, not to the sandboxes.
	18	+	const (
	19	+	EnvIsolation = "ERMITE_RUNNER_ISOLATION"
	20	+	EnvStateDir = "ERMITE_RUNNER_STATE_DIR"
	21	+	EnvCapacity = "ERMITE_RUNNER_CAPACITY"
	22	+	EnvImage = "ERMITE_RUNNER_IMAGE"
	23	+	EnvJobCPUs = "ERMITE_RUNNER_JOB_CPUS"
	24	+	EnvJobMemory = "ERMITE_RUNNER_JOB_MEMORY"
	25	+	EnvJobDisk = "ERMITE_RUNNER_JOB_DISK"
	26	+	EnvUIDBase = "ERMITE_RUNNER_UID_BASE"
	27	+	EnvNetwork = "ERMITE_RUNNER_NETWORK"
	28	+	EnvFirecracker = "ERMITE_RUNNER_FIRECRACKER"
	29	+	EnvJailer = "ERMITE_RUNNER_JAILER"
	30	+	EnvJailerMode = "ERMITE_RUNNER_JAILER_MODE"
	31	+	EnvKernel = "ERMITE_RUNNER_KERNEL"
	32	+
	33	+	EnvJobPids = "ERMITE_RUNNER_JOB_PIDS"
	34	+	EnvDenyPrivileged = "ERMITE_RUNNER_DENY_PRIVILEGED"
	35	+	EnvLockFile = "ERMITE_RUNNER_LOCK_FILE"
	36	+	EnvJobDiskBW = "ERMITE_RUNNER_JOB_DISK_BANDWIDTH"
	37	+	EnvJobDiskIOPS = "ERMITE_RUNNER_JOB_DISK_IOPS"
	38	+	EnvJobNetBandwidth = "ERMITE_RUNNER_JOB_NET_BANDWIDTH"
	39	+	)
	40	+
	41	+	// Defaults.
	42	+	const (
	43	+	DefaultStateDir = "/var/lib/ermite-runner"
	44	+	DefaultOCIImage = "ermite-runner-ubuntu-24.04:latest"
	45	+	DefaultUIDBase = 900000
	46	+	DefaultNetwork = "10.200.0.0/16"
	47	+	DefaultCPUs = 2
	48	+	DefaultMemoryMiB = 4096
	49	+	DefaultDiskMiB = 64 * 1024
	50	+	DefaultJobPids = 32768
	51	+	// DefaultLockFile is the single-instance lock of the host: reaping removes
	52	+	// everything the runner labels as its own, so two runners (even with
	53	+	// different state directories, or one in a container) must never coexist.
	54	+	DefaultLockFile = "/run/ermite-runner.lock"
	55	+	// Rate limits of a micro-VM, generous so that they only stop a runaway job
	56	+	// from starving the host: 400 MiB/s and 20000 IOPS per disk, 1 Gbit/s per
	57	+	// network direction. Zero disables a limit.
	58	+	DefaultDiskBandwidthMiB = 400
	59	+	DefaultDiskIOPS = 20000
	60	+	DefaultNetMbit = 1000
	61	+	)
	62	+
	63	+	// Config is the sandbox part of the runner configuration.
	64	+	type Config struct {
	65	+	Isolation Mode
	66	+	StateDir string
	67	+	Capacity int
	68	+
	69	+	// Image is ERMITE_RUNNER_IMAGE: a directory (VM: holds the kernel and
	70	+	// base.ext4) or an OCI reference (container). A value that is empty or does
	71	+	// not fit the selected mode falls back to the default for that mode.
	72	+	Image string
	73	+
	74	+	JobCPUs int
	75	+	JobMemoryMiB int
	76	+	JobDiskMiB int
	77	+
	78	+	// UIDBase+slot is the uid and gid the jailer drops to.
	79	+	UIDBase int
	80	+	// Network is the pool of /30 subnets, one per slot (VM TAPs, container
	81	+	// bridges).
	82	+	Network netip.Prefix
	83	+
	84	+	// Firecracker and Jailer are binary paths; empty means look in PATH.
	85	+	Firecracker string
	86	+	Jailer string
	87	+	// Kernel overrides <image dir>/vmlinux (Image on arm64).
	88	+	Kernel string
	89	+	// JailerOff runs Firecracker without the jailer (development and tests as
	90	+	// an unprivileged user). Never use it in production: it only drops the
	91	+	// chroot, uid and namespace confinement of the VMM process.
	92	+	JailerOff bool
	93	+
	94	+	// KVMPath is the KVM device; tests point it elsewhere to hide /dev/kvm.
	95	+	KVMPath string
	96	+	// Executable is attached read-only to every sandbox. Empty means
	97	+	// os.Executable(). It must be a static binary (CGO_ENABLED=0).
	98	+	Executable string
	99	+	// DNSPort is the port of the DNS forwarder; 53 in production. Tests that
	100	+	// cannot bind it use another port with no guest.
	101	+	DNSPort int
	102	+
	103	+	// JobPids is the process limit of a job (container PidsLimit, VM pids.max
	104	+	// is derived from the vCPU count and is not affected).
	105	+	JobPids int
	106	+	// DenyPrivileged refuses the container mode when it would have to run
	107	+	// privileged (no sysbox).
	108	+	DenyPrivileged bool
	109	+	// LockFile is the host-wide single-instance lock.
	110	+	LockFile string
	111	+	// Rate limits of micro-VMs; zero disables.
	112	+	DiskBandwidthMiB, DiskIOPS, NetMbit int
	113	+	}
	114	+
	115	+	// LoadConfig reads the sandbox settings from getenv (os.Getenv in production).
	116	+	func LoadConfig(getenv func(string) string) (Config, error) {
	117	+	c := Config{
	118	+	Isolation: Auto,
	119	+	StateDir: DefaultStateDir,
	120	+	Capacity: 1,
	121	+	JobCPUs: DefaultCPUs,
	122	+	JobMemoryMiB: DefaultMemoryMiB,
	123	+	JobDiskMiB: DefaultDiskMiB,
	124	+	UIDBase: DefaultUIDBase,
	125	+	KVMPath: "/dev/kvm",
	126	+	DNSPort: 53,
	127	+	JobPids: DefaultJobPids,
	128	+	LockFile: DefaultLockFile,
	129	+
	130	+	DiskBandwidthMiB: DefaultDiskBandwidthMiB,
	131	+	DiskIOPS: DefaultDiskIOPS,
	132	+	NetMbit: DefaultNetMbit,
	133	+	}
	134	+	var err error
	135	+	if v := getenv(EnvIsolation); v != "" {
	136	+	switch m := Mode(strings.ToLower(strings.TrimSpace(v))); m {
	137	+	case Auto, VM, Container:
	138	+	c.Isolation = m
	139	+	default:
	140	+	return c, fmt.Errorf("%s: %q is not auto, vm or container", EnvIsolation, v)
	141	+	}
	142	+	}
	143	+	if v := getenv(EnvStateDir); v != "" {
	144	+	c.StateDir = v
	145	+	}
	146	+	if !filepath.IsAbs(c.StateDir) {
	147	+	return c, fmt.Errorf("%s: %q must be an absolute path", EnvStateDir, c.StateDir)
	148	+	}
	149	+	if c.Capacity, err = intEnv(getenv, EnvCapacity, c.Capacity, 1, 1024); err != nil {
	150	+	return c, err
	151	+	}
	152	+	c.Image = getenv(EnvImage)
	153	+	if c.JobCPUs, err = intEnv(getenv, EnvJobCPUs, c.JobCPUs, 1, 128); err != nil {
	154	+	return c, err
	155	+	}
	156	+	if v := getenv(EnvJobMemory); v != "" {
	157	+	if c.JobMemoryMiB, err = ParseSizeMiB(v); err != nil \|\| c.JobMemoryMiB < 256 {
	158	+	return c, fmt.Errorf("%s: %q: need a size of at least 256M", EnvJobMemory, v)
	159	+	}
	160	+	}
	161	+	if v := getenv(EnvJobDisk); v != "" {
	162	+	if c.JobDiskMiB, err = ParseSizeMiB(v); err != nil \|\| c.JobDiskMiB < 1024 {
	163	+	return c, fmt.Errorf("%s: %q: need a size of at least 1G", EnvJobDisk, v)
	164	+	}
	165	+	}
	166	+	if c.UIDBase, err = intEnv(getenv, EnvUIDBase, c.UIDBase, 1000, 4_000_000_000); err != nil {
	167	+	return c, err
	168	+	}
	169	+	if c.UIDBase+c.Capacity > 4_294_967_294 {
	170	+	return c, fmt.Errorf("%s: %d + capacity overflows the uid range", EnvUIDBase, c.UIDBase)
	171	+	}
	172	+	net := DefaultNetwork
	173	+	if v := getenv(EnvNetwork); v != "" {
	174	+	net = v
	175	+	}
	176	+	if c.Network, err = netip.ParsePrefix(net); err != nil \|\| !c.Network.Addr().Is4() \|\| c.Network.Bits() > 30 {
	177	+	return c, fmt.Errorf("%s: %q is not an IPv4 prefix of at most /30", EnvNetwork, net)
	178	+	}
	179	+	if 1<<(30-c.Network.Bits()) < c.Capacity {
	180	+	return c, fmt.Errorf("%s: %s has fewer /30 subnets than the capacity %d", EnvNetwork, c.Network, c.Capacity)
	181	+	}
	182	+	if c.JobPids, err = intEnv(getenv, EnvJobPids, c.JobPids, 64, 4_194_304); err != nil {
	183	+	return c, err
	184	+	}
	185	+	switch strings.ToLower(strings.TrimSpace(getenv(EnvDenyPrivileged))) {
	186	+	case "", "0", "false", "no":
	187	+	case "1", "true", "yes":
	188	+	c.DenyPrivileged = true
	189	+	default:
	190	+	return c, fmt.Errorf("%s: %q is not 0 or 1", EnvDenyPrivileged, getenv(EnvDenyPrivileged))
	191	+	}
	192	+	if v := getenv(EnvLockFile); v != "" {
	193	+	if !filepath.IsAbs(v) {
	194	+	return c, fmt.Errorf("%s: %q must be an absolute path", EnvLockFile, v)
	195	+	}
	196	+	c.LockFile = v
	197	+	}
	198	+	if c.DiskBandwidthMiB, err = intEnv(getenv, EnvJobDiskBW, c.DiskBandwidthMiB, 0, 1_000_000); err != nil {
	199	+	return c, err
	200	+	}
	201	+	if c.DiskIOPS, err = intEnv(getenv, EnvJobDiskIOPS, c.DiskIOPS, 0, 100_000_000); err != nil {
	202	+	return c, err
	203	+	}
	204	+	if c.NetMbit, err = intEnv(getenv, EnvJobNetBandwidth, c.NetMbit, 0, 1_000_000); err != nil {
	205	+	return c, err
	206	+	}
	207	+	c.Firecracker = getenv(EnvFirecracker)
	208	+	c.Jailer = getenv(EnvJailer)
	209	+	c.Kernel = getenv(EnvKernel)
	210	+	switch strings.ToLower(getenv(EnvJailerMode)) {
	211	+	case "", "on":
	212	+	case "off":
	213	+	c.JailerOff = true
	214	+	default:
	215	+	return c, fmt.Errorf("%s: %q is not on or off", EnvJailerMode, getenv(EnvJailerMode))
	216	+	}
	217	+	return c, nil
	218	+	}
	219	+
	220	+	func intEnv(getenv func(string) string, key string, def, lo, hi int) (int, error) {
	221	+	v := getenv(key)
	222	+	if v == "" {
	223	+	return def, nil
	224	+	}
	225	+	n, err := strconv.Atoi(strings.TrimSpace(v))
	226	+	if err != nil \|\| n < lo \|\| n > hi {
	227	+	return 0, fmt.Errorf("%s: %q is not an integer in [%d, %d]", key, v, lo, hi)
	228	+	}
	229	+	return n, nil
	230	+	}
	231	+
	232	+	// ParseSizeMiB parses "4096", "512M", "4G", "64GiB", "1T" into MiB. A bare
	233	+	// number is MiB. Suffixes are binary (1G = 1024 MiB) and case-insensitive.
	234	+	func ParseSizeMiB(s string) (int, error) {
	235	+	s = strings.ToLower(strings.TrimSpace(s))
	236	+	s = strings.TrimSuffix(strings.TrimSuffix(s, "b"), "i")
	237	+	mult := 1
	238	+	if s != "" {
	239	+	switch s[len(s)-1] {
	240	+	case 'k':
	241	+	mult, s = 0, s[:len(s)-1]
	242	+	case 'm':
	243	+	mult, s = 1, s[:len(s)-1]
	244	+	case 'g':
	245	+	mult, s = 1024, s[:len(s)-1]
	246	+	case 't':
	247	+	mult, s = 1024*1024, s[:len(s)-1]
	248	+	}
	249	+	}
	250	+	n, err := strconv.Atoi(s)
	251	+	if err != nil \|\| n <= 0 \|\| mult == 0 {
	252	+	return 0, errors.New("invalid size")
	253	+	}
	254	+	if n > (1<<31-1)/mult {
	255	+	return 0, errors.New("size too large")
	256	+	}
	257	+	return n * mult, nil
	258	+	}
	259	+
	260	+	// ImageDir returns the VM image directory: Image when it is a directory path,
	261	+	// else <StateDir>/image.
	262	+	func (c Config) ImageDir() string {
	263	+	if c.Image != "" && looksLikePath(c.Image) {
	264	+	return c.Image
	265	+	}
	266	+	return filepath.Join(c.StateDir, "image")
	267	+	}
	268	+
	269	+	// OCIImage returns the container image reference: Image when it is not a
	270	+	// directory path, else DefaultOCIImage.
	271	+	func (c Config) OCIImage() string {
	272	+	if c.Image != "" && !looksLikePath(c.Image) {
	273	+	return c.Image
	274	+	}
	275	+	return DefaultOCIImage
	276	+	}
	277	+
	278	+	func looksLikePath(s string) bool {
	279	+	return strings.HasPrefix(s, "/") \|\| strings.HasPrefix(s, "./") \|\| strings.HasPrefix(s, "../")
	280	+	}
	281	+
	282	+	// KernelPath returns the guest kernel: Kernel, else <ImageDir>/vmlinux (x86-64)
	283	+	// or <ImageDir>/Image (arm64).
	284	+	func (c Config) KernelPath() string {
	285	+	if c.Kernel != "" {
	286	+	return c.Kernel
	287	+	}
	288	+	name := "vmlinux"
	289	+	if runtime.GOARCH == "arm64" {
	290	+	name = "Image"
	291	+	}
	292	+	return filepath.Join(c.ImageDir(), name)
	293	+	}
	294	+
	295	+	// BaseImagePath returns the shared read-only base disk.
	296	+	func (c Config) BaseImagePath() string { return filepath.Join(c.ImageDir(), "base.ext4") }
	297	+
	298	+	// ExecutablePath returns the binary attached to sandboxes.
	299	+	func (c Config) ExecutablePath() (string, error) {
	300	+	if c.Executable != "" {
	301	+	return c.Executable, nil
	302	+	}
	303	+	p, err := os.Executable()
	304	+	if err != nil {
	305	+	return "", fmt.Errorf("sandbox: locate own executable: %w", err)
	306	+	}
	307	+	return filepath.EvalSymlinks(p)
	308	+	}
	309	+
	310	+	// Resolve fills the zero resource fields of a JobSpec from the config.
	311	+	func (c Config) Resolve(s JobSpec) JobSpec {
	312	+	if s.CPUs <= 0 {
	313	+	s.CPUs = c.JobCPUs
	314	+	}
	315	+	if s.MemoryMiB <= 0 {
	316	+	s.MemoryMiB = c.JobMemoryMiB
	317	+	}
	318	+	if s.DiskMiB <= 0 {
	319	+	s.DiskMiB = c.JobDiskMiB
	320	+	}
	321	+	return s
	322	+	}

A internal/sandbox/container/container_test.go +1012 -0

@@ -0,0 +1,1012 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package container
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"context"
	8	+	"errors"
	9	+	"fmt"
	10	+	"net"
	11	+	"net/netip"
	12	+	"os"
	13	+	"path/filepath"
	14	+	"slices"
	15	+	"strings"
	16	+	"sync"
	17	+	"testing"
	18	+	"time"
	19	+
	20	+	"github.com/bnema/zerowrap"
	21	+	"github.com/moby/moby/api/types/container"
	22	+
	23	+	"bnema.dev/ermite-runner/internal/cmdrun"
	24	+	"bnema.dev/ermite-runner/internal/execstream"
	25	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	26	+	"bnema.dev/ermite-runner/internal/sandbox"
	27	+	"bnema.dev/ermite-runner/internal/sandbox/disk"
	28	+	"bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
	29	+	)
	30	+
	31	+	// fakeEngine plays the Docker daemon. When a container "starts", it plays the
	32	+	// executor inside it by connecting to the bind-mounted socket.
	33	+	type fakeEngine struct {
	34	+	mu sync.Mutex
	35	+	info daemonInfo
	36	+	infoErr error
	37	+	imageErr error
	38	+	networks map[string]networkSpec
	39	+	containers map[string]containerSpec
	40	+	failStart error
	41	+	noConnect bool
	42	+	exitOnRun bool
	43	+	removeErr error
	44	+	waiters map[string]chan exitResult
	45	+	seq int
	46	+	pulled []string
	47	+	}
	48	+
	49	+	func newFake() *fakeEngine {
	50	+	return &fakeEngine{
	51	+	info: daemonInfo{OSType: "linux", Runtimes: []string{"runc"}},
	52	+	networks: map[string]networkSpec{},
	53	+	containers: map[string]containerSpec{},
	54	+	waiters: map[string]chan exitResult{},
	55	+	}
	56	+	}
	57	+
	58	+	func (f *fakeEngine) Info(context.Context) (daemonInfo, error) { return f.info, f.infoErr }
	59	+	func (f *fakeEngine) EnsureImage(_ context.Context, ref string) error {
	60	+	f.pulled = append(f.pulled, ref)
	61	+	return f.imageErr
	62	+	}
	63	+	func (f *fakeEngine) CreateNetwork(_ context.Context, s networkSpec) (string, error) {
	64	+	f.mu.Lock()
	65	+	defer f.mu.Unlock()
	66	+	f.seq++
	67	+	id := fmt.Sprintf("net%d", f.seq)
	68	+	f.networks[id] = s
	69	+	return id, nil
	70	+	}
	71	+	func (f *fakeEngine) RemoveNetwork(_ context.Context, id string) error {
	72	+	f.mu.Lock()
	73	+	defer f.mu.Unlock()
	74	+	delete(f.networks, id)
	75	+	return nil
	76	+	}
	77	+	func (f *fakeEngine) ListNetworks(context.Context, string) ([]string, error) {
	78	+	f.mu.Lock()
	79	+	defer f.mu.Unlock()
	80	+	var out []string
	81	+	for id, n := range f.networks {
	82	+	if _, ok := n.Labels[sandbox.LabelJob]; ok {
	83	+	out = append(out, id)
	84	+	}
	85	+	}
	86	+	return out, nil
	87	+	}
	88	+	func (f *fakeEngine) CreateContainer(_ context.Context, s containerSpec) (string, error) {
	89	+	f.mu.Lock()
	90	+	defer f.mu.Unlock()
	91	+	f.seq++
	92	+	id := fmt.Sprintf("ctr%d", f.seq)
	93	+	f.containers[id] = s
	94	+	f.waiters[id] = make(chan exitResult, 1)
	95	+	return id, nil
	96	+	}
	97	+	func (f *fakeEngine) StartContainer(_ context.Context, id string) error {
	98	+	f.mu.Lock()
	99	+	spec := f.containers[id]
	100	+	fail, noConnect, exitOnRun := f.failStart, f.noConnect, f.exitOnRun
	101	+	w := f.waiters[id]
	102	+	f.mu.Unlock()
	103	+	if fail != nil {
	104	+	return fail
	105	+	}
	106	+	if exitOnRun {
	107	+	w <- exitResult{Code: 125}
	108	+	return nil
	109	+	}
	110	+	if noConnect {
	111	+	return nil
	112	+	}
	113	+	// The host side of the bind mount is the source of the run-dir mount.
	114	+	var sock string
	115	+	for _, b := range spec.Binds {
	116	+	if b.Target == sandbox.ContainerRunDir {
	117	+	sock = filepath.Join(b.Source, sandbox.ContainerSocket)
	118	+	}
	119	+	}
	120	+	go func() {
	121	+	c, err := net.Dial("unix", sock)
	122	+	if err != nil {
	123	+	return
	124	+	}
	125	+	defer func() { _ = c.Close() }()
	126	+	g := execstream.NewConn(c)
	127	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Hello{Hello: &execpb.Hello{Sandbox: "container"}}})
	128	+	if _, err := g.Recv(); err != nil {
	129	+	return
	130	+	}
	131	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Output{Output: &execpb.Output{Data: []byte("hello from fake container\n")}}})
	132	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Exit{Exit: &execpb.Exit{Code: 7}}})
	133	+	_, _ = g.Recv()
	134	+	}()
	135	+	return nil
	136	+	}
	137	+	func (f *fakeEngine) WaitExit(_ context.Context, id string) <-chan exitResult {
	138	+	f.mu.Lock()
	139	+	defer f.mu.Unlock()
	140	+	return f.waiters[id]
	141	+	}
	142	+	func (f *fakeEngine) RemoveContainer(_ context.Context, id string) error {
	143	+	f.mu.Lock()
	144	+	defer f.mu.Unlock()
	145	+	if f.removeErr != nil {
	146	+	return f.removeErr
	147	+	}
	148	+	delete(f.containers, id)
	149	+	return nil
	150	+	}
	151	+	func (f *fakeEngine) ListContainers(context.Context, string) ([]string, error) {
	152	+	f.mu.Lock()
	153	+	defer f.mu.Unlock()
	154	+	var out []string
	155	+	for id := range f.containers {
	156	+	out = append(out, id)
	157	+	}
	158	+	return out, nil
	159	+	}
	160	+	func (f *fakeEngine) LogsTail(context.Context, string, int) (string, error) {
	161	+	return "fake container log line", nil
	162	+	}
	163	+	func (f *fakeEngine) Close() error { return nil }
	164	+
	165	+	func (f *fakeEngine) objects() []string {
	166	+	f.mu.Lock()
	167	+	defer f.mu.Unlock()
	168	+	var out []string
	169	+	for id := range f.containers {
	170	+	out = append(out, "container "+id)
	171	+	}
	172	+	for id := range f.networks {
	173	+	out = append(out, "network "+id)
	174	+	}
	175	+	return out
	176	+	}
	177	+
	178	+	// fakeMounts plays mount/umount: it records mounts and writes a mountinfo file
	179	+	// that disk.Mounter reads.
	180	+	type fakeMounts struct {
	181	+	mu sync.Mutex
	182	+	mounted map[string]bool
	183	+	file string
	184	+	failUm error
	185	+	}
	186	+
	187	+	func (m *fakeMounts) sync() {
	188	+	var b strings.Builder
	189	+	for d := range m.mounted {
	190	+	fmt.Fprintf(&b, "40 28 7:0 / %s rw - ext4 /dev/loop0 rw\n", d)
	191	+	}
	192	+	_ = os.WriteFile(m.file, []byte(b.String()), 0o644)
	193	+	}
	194	+
	195	+	func (m *fakeMounts) handle(c cmdrun.Call) ([]byte, error) {
	196	+	m.mu.Lock()
	197	+	defer m.mu.Unlock()
	198	+	switch c.Name {
	199	+	case "mkfs.ext4":
	200	+	if len(c.Args) == 1 && c.Args[0] == "-V" {
	201	+	return []byte("mke2fs 1.47.4 (6-Mar-2025)\n"), nil
	202	+	}
	203	+	case "mount":
	204	+	m.mounted[c.Args[len(c.Args)-1]] = true
	205	+	m.sync()
	206	+	case "umount":
	207	+	if m.failUm != nil {
	208	+	return nil, m.failUm
	209	+	}
	210	+	delete(m.mounted, c.Args[0])
	211	+	m.sync()
	212	+	}
	213	+	return nil, nil
	214	+	}
	215	+
	216	+	func (m *fakeMounts) count() int { m.mu.Lock(); defer m.mu.Unlock(); return len(m.mounted) }
	217	+
	218	+	type cenv struct {
	219	+	cfg sandbox.Config
	220	+	eng *fakeEngine
	221	+	plumb *sandboxtest.Plumbing
	222	+	mounts *fakeMounts
	223	+	run *cmdrun.Fake
	224	+	}
	225	+
	226	+	func newCEnv(t testing.T) cenv {
	227	+	t.Helper()
	228	+	// A short directory: the executor socket path is limited to 107 bytes and
	229	+	// t.TempDir embeds the (long) test name.
	230	+	dir, err := os.MkdirTemp("", "erc")
	231	+	if err != nil {
	232	+	t.Fatal(err)
	233	+	}
	234	+	t.Cleanup(func() { _ = os.RemoveAll(dir) })
	235	+	exe := filepath.Join(dir, "runner")
	236	+	sandboxtest.WriteStaticELF(t, exe)
	237	+	fm := &fakeMounts{mounted: map[string]bool{}, file: filepath.Join(dir, "mountinfo")}
	238	+	fm.sync()
	239	+	return &cenv{
	240	+	eng: newFake(),
	241	+	plumb: sandboxtest.NewPlumbing(),
	242	+	mounts: fm,
	243	+	run: &cmdrun.Fake{Handle: fm.handle},
	244	+	cfg: sandbox.Config{
	245	+	Isolation: sandbox.Container, StateDir: filepath.Join(dir, "state"), Capacity: 2,
	246	+	JobCPUs: 2, JobMemoryMiB: 1024, JobDiskMiB: 1024, UIDBase: 900000,
	247	+	Network: netip.MustParsePrefix("10.200.0.0/16"), Executable: exe, DNSPort: 0,
	248	+	LockFile: filepath.Join(dir, "host.lock"), JobPids: 32768,
	249	+	},
	250	+	}
	251	+	}
	252	+
	253	+	func (e *cenv) opts() Options {
	254	+	return Options{engine: e.eng, Net: e.plumb, ConnectTimeout: 3 * time.Second, Run: e.run,
	255	+	Mounter: &disk.Mounter{Run: e.run, Mountinfo: e.mounts.file}}
	256	+	}
	257	+
	258	+	func (e cenv) driver(t testing.T) *Driver {
	259	+	t.Helper()
	260	+	d, err := New(context.Background(), e.cfg, e.opts())
	261	+	if err != nil {
	262	+	t.Fatal(err)
	263	+	}
	264	+	t.Cleanup(func() { _ = d.Close() })
	265	+	return d
	266	+	}
	267	+
	268	+	func withTimeout(o Options, d time.Duration) Options { o.ConnectTimeout = d; return o }
	269	+
	270	+	func (e *cenv) leftovers() []string {
	271	+	out := e.eng.objects()
	272	+	if n := e.mounts.count(); n != 0 {
	273	+	out = append(out, fmt.Sprintf("%d scratch mounts", n))
	274	+	}
	275	+	if n := e.plumb.Live(); n != 0 {
	276	+	out = append(out, fmt.Sprintf("%d attached guests", n))
	277	+	}
	278	+	ents, _ := os.ReadDir(filepath.Join(e.cfg.StateDir, "containers"))
	279	+	for _, d := range ents {
	280	+	out = append(out, "dir "+d.Name())
	281	+	}
	282	+	return out
	283	+	}
	284	+
	285	+	func stub(t *testing.T, conn sandbox.Conn) (string, int) {
	286	+	t.Helper()
	287	+	var out strings.Builder
	288	+	_, exit, err := execstream.Drive(conn, &execpb.Run{}, nil, func(o *execpb.Output) { out.Write(o.Data) })
	289	+	if err != nil {
	290	+	t.Fatal(err)
	291	+	}
	292	+	return out.String(), int(exit.Code)
	293	+	}
	294	+
	295	+	func TestRootlessDockerIsRefused(t *testing.T) {
	296	+	e := newCEnv(t)
	297	+	e.eng.info.Rootless = true
	298	+	_, err := New(context.Background(), e.cfg, e.opts())
	299	+	if err == nil \|\| !strings.Contains(err.Error(), "rootless") \|\| !strings.Contains(err.Error(), "rootful") {
	300	+	t.Fatalf("want a clear rootless error, got %v", err)
	301	+	}
	302	+	}
	303	+
	304	+	func TestRuntimeSelection(t *testing.T) {
	305	+	e := newCEnv(t)
	306	+	d := e.driver(t)
	307	+	if !d.Privileged() \|\| d.runtime != "" {
	308	+	t.Fatal("without sysbox the driver must run privileged")
	309	+	}
	310	+	e2 := newCEnv(t)
	311	+	e2.eng.info.Runtimes = []string{"runc", "sysbox-runc"}
	312	+	d2 := e2.driver(t)
	313	+	if d2.Privileged() \|\| d2.runtime != SysboxRuntime {
	314	+	t.Fatal("sysbox listed: must use it, unprivileged")
	315	+	}
	316	+	}
	317	+
	318	+	func TestNewErrors(t *testing.T) {
	319	+	e := newCEnv(t)
	320	+	e.eng.infoErr = errors.New("connection refused")
	321	+	if _, err := New(context.Background(), e.cfg, e.opts()); err == nil \|\| !strings.Contains(err.Error(), "DOCKER_HOST") {
	322	+	t.Fatalf("got %v", err)
	323	+	}
	324	+	e = newCEnv(t)
	325	+	e.eng.imageErr = errors.New("no such image")
	326	+	if _, err := New(context.Background(), e.cfg, e.opts()); err == nil {
	327	+	t.Fatal("missing image accepted")
	328	+	}
	329	+	e = newCEnv(t)
	330	+	e.eng.info.OSType = "windows"
	331	+	if _, err := New(context.Background(), e.cfg, e.opts()); err == nil {
	332	+	t.Fatal("windows daemon accepted")
	333	+	}
	334	+	e = newCEnv(t)
	335	+	e.cfg.Executable = "/bin/sh"
	336	+	if _, err := New(context.Background(), e.cfg, e.opts()); err == nil {
	337	+	t.Skip("/bin/sh is static")
	338	+	}
	339	+	}
	340	+
	341	+	func TestStartRunStop(t *testing.T) {
	342	+	e := newCEnv(t)
	343	+	d := e.driver(t)
	344	+	sb, err := d.New()
	345	+	if err != nil {
	346	+	t.Fatal(err)
	347	+	}
	348	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "job-7", CPUs: 3, MemoryMiB: 2048})
	349	+	if err != nil {
	350	+	t.Fatal(err)
	351	+	}
	352	+	out, code := stub(t, conn)
	353	+	if out != "hello from fake container\n" \|\| code != 7 {
	354	+	t.Fatalf("out=%q code=%d", out, code)
	355	+	}
	356	+
	357	+	// What was asked of Docker.
	358	+	e.eng.mu.Lock()
	359	+	var spec containerSpec
	360	+	for _, s := range e.eng.containers {
	361	+	spec = s
	362	+	}
	363	+	var netSpec networkSpec
	364	+	for _, n := range e.eng.networks {
	365	+	netSpec = n
	366	+	}
	367	+	e.eng.mu.Unlock()
	368	+	if !spec.Privileged \|\| spec.Runtime != "" {
	369	+	t.Errorf("privileged fallback expected: %+v", spec)
	370	+	}
	371	+	if spec.Labels[sandbox.LabelJob] != "job-7" \|\| netSpec.Labels[sandbox.LabelJob] != "job-7" {
	372	+	t.Errorf("every object must carry the job label: %v / %v", spec.Labels, netSpec.Labels)
	373	+	}
	374	+	if spec.MemoryMiB != 2048 \|\| spec.CPUs != 3 {
	375	+	t.Errorf("resources: %+v", spec)
	376	+	}
	377	+	if !slices.Equal(spec.Entrypoint, []string{"/opt/ermite/ermite-runner", "init"}) {
	378	+	t.Errorf("entrypoint: %v", spec.Entrypoint)
	379	+	}
	380	+	var roExe, runDir bool
	381	+	for _, b := range spec.Binds {
	382	+	if b.Target == "/opt/ermite/ermite-runner" && b.ReadOnly && b.Source == e.cfg.Executable {
	383	+	roExe = true
	384	+	}
	385	+	if b.Target == sandbox.ContainerRunDir && !b.ReadOnly {
	386	+	runDir = true
	387	+	}
	388	+	}
	389	+	if !roExe \|\| !runDir {
	390	+	t.Errorf("binds: %+v", spec.Binds)
	391	+	}
	392	+	var scratch, work, tmp bool
	393	+	for _, b := range spec.Binds {
	394	+	switch b.Target {
	395	+	case sandbox.GuestScratch:
	396	+	scratch = strings.HasSuffix(b.Source, "/scratch") && !b.ReadOnly
	397	+	case "/home/runner/work":
	398	+	work = strings.HasSuffix(b.Source, "/scratch/work")
	399	+	case "/tmp":
	400	+	tmp = strings.HasSuffix(b.Source, "/scratch/tmp")
	401	+	}
	402	+	}
	403	+	if !scratch \|\| !work \|\| !tmp {
	404	+	t.Errorf("scratch, workspace and /tmp must come from the loop-mounted disk: %+v", spec.Binds)
	405	+	}
	406	+	if spec.DNS.String() != "10.200.0.1" \|\| spec.IPv4.String() != "10.200.0.2" \|\| netSpec.Bridge != "erbr0" \|\| netSpec.Subnet.String() != "10.200.0.0/30" {
	407	+	t.Errorf("network: %+v %+v", spec, netSpec)
	408	+	}
	409	+	if !slices.Contains(spec.Env, "ERMITE_CONNECT=unix:/run/ermite/exec.sock") {
	410	+	t.Errorf("env: %v", spec.Env)
	411	+	}
	412	+	if e.plumb.Live() != 1 {
	413	+	t.Fatal("firewall policy not attached")
	414	+	}
	415	+
	416	+	if err := sb.Stop(context.Background()); err != nil {
	417	+	t.Fatal(err)
	418	+	}
	419	+	if left := e.leftovers(); len(left) != 0 {
	420	+	t.Fatalf("leftovers: %v", left)
	421	+	}
	422	+	if err := sb.Stop(context.Background()); err != nil {
	423	+	t.Fatalf("Stop must be idempotent: %v", err)
	424	+	}
	425	+	if d.slots.InUse() != 0 {
	426	+	t.Fatal("slot not released")
	427	+	}
	428	+	}
	429	+
	430	+	func TestSysboxJobIsNotPrivileged(t *testing.T) {
	431	+	e := newCEnv(t)
	432	+	e.eng.info.Runtimes = []string{"sysbox-runc", "runc"}
	433	+	d := e.driver(t)
	434	+	sb, _ := d.New()
	435	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	436	+	if err != nil {
	437	+	t.Fatal(err)
	438	+	}
	439	+	stub(t, conn)
	440	+	e.eng.mu.Lock()
	441	+	for _, s := range e.eng.containers {
	442	+	if s.Privileged \|\| s.Runtime != "sysbox-runc" {
	443	+	t.Errorf("sysbox job: %+v", s)
	444	+	}
	445	+	}
	446	+	e.eng.mu.Unlock()
	447	+	_ = sb.Stop(context.Background())
	448	+	}
	449	+
	450	+	func TestStartFailuresCleanEverything(t *testing.T) {
	451	+	cases := map[string]func(*cenv){
	452	+	"container cannot start": func(e *cenv) { e.eng.failStart = errors.New("oci runtime error") },
	453	+	"container exits early": func(e *cenv) { e.eng.exitOnRun = true },
	454	+	"executor never connects": func(e *cenv) {
	455	+	e.eng.noConnect = true
	456	+	},
	457	+	"firewall attach fails": func(e *cenv) { e.plumb.FailNext = errors.New("nft broke") },
	458	+	}
	459	+	for name, setup := range cases {
	460	+	t.Run(name, func(t *testing.T) {
	461	+	e := newCEnv(t)
	462	+	setup(e)
	463	+	d, err := New(context.Background(), e.cfg, withTimeout(e.opts(), 300*time.Millisecond))
	464	+	if err != nil {
	465	+	t.Fatal(err)
	466	+	}
	467	+	defer func() { _ = d.Close() }()
	468	+	sb, _ := d.New()
	469	+	_, err = sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	470	+	if err == nil {
	471	+	t.Fatal("Start succeeded")
	472	+	}
	473	+	if left := e.leftovers(); len(left) != 0 {
	474	+	t.Fatalf("leftovers: %v (err %v)", left, err)
	475	+	}
	476	+	if d.slots.InUse() != 0 {
	477	+	t.Fatal("slot leaked")
	478	+	}
	479	+	})
	480	+	}
	481	+	}
	482	+
	483	+	func TestFailedStartIncludesContainerLogs(t *testing.T) {
	484	+	e := newCEnv(t)
	485	+	e.eng.exitOnRun = true
	486	+	d := e.driver(t)
	487	+	sb, _ := d.New()
	488	+	_, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	489	+	if err == nil \|\| !strings.Contains(err.Error(), "fake container log line") {
	490	+	t.Fatalf("got %v", err)
	491	+	}
	492	+	}
	493	+
	494	+	func TestCleanupFailureKeepsSlotAndIsRetriedByReap(t *testing.T) {
	495	+	e := newCEnv(t)
	496	+	d := e.driver(t)
	497	+	sb, _ := d.New()
	498	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	499	+	if err != nil {
	500	+	t.Fatal(err)
	501	+	}
	502	+	stub(t, conn)
	503	+	e.eng.removeErr = errors.New("daemon busy")
	504	+	if err := sb.Stop(context.Background()); err == nil {
	505	+	t.Fatal("Stop must report the failed removal")
	506	+	}
	507	+	if d.slots.InUse() != 1 {
	508	+	t.Fatal("the slot must stay taken while its bridge and subnet still exist")
	509	+	}
	510	+	e.eng.removeErr = nil
	511	+	if err := d.Reap(context.Background()); err != nil {
	512	+	t.Fatal(err)
	513	+	}
	514	+	if left := e.leftovers(); len(left) != 0 {
	515	+	t.Fatalf("Reap left %v", left)
	516	+	}
	517	+	}
	518	+
	519	+	func TestReapRemovesLabelledObjectsOnly(t *testing.T) {
	520	+	e := newCEnv(t)
	521	+	d := e.driver(t)
	522	+	ctx := context.Background()
	523	+	// Leftovers of a crashed run, and an unrelated user's network.
	524	+	_, _ = e.eng.CreateContainer(ctx, containerSpec{Name: "old", Labels: map[string]string{sandbox.LabelJob: "x"}})
	525	+	_, _ = e.eng.CreateNetwork(ctx, networkSpec{Name: "old", Labels: map[string]string{sandbox.LabelJob: "x"}})
	526	+	_, _ = e.eng.CreateNetwork(ctx, networkSpec{Name: "mine", Labels: map[string]string{"app": "other"}})
	527	+	_ = os.MkdirAll(filepath.Join(e.cfg.StateDir, "containers", "er00112233"), 0o700)
	528	+	_ = os.MkdirAll(filepath.Join(e.cfg.StateDir, "containers", "keep-me"), 0o700)
	529	+
	530	+	if err := d.Reap(ctx); err != nil {
	531	+	t.Fatal(err)
	532	+	}
	533	+	left := e.leftovers()
	534	+	slices.Sort(left)
	535	+	if len(left) != 2 \|\| !strings.HasPrefix(left[0], "dir keep-me") \|\| !strings.HasPrefix(left[1], "network") {
	536	+	t.Fatalf("only the unrelated network and directory may remain: %v", left)
	537	+	}
	538	+	if _, err := os.Stat(filepath.Join(e.cfg.StateDir, "containers", "keep-me")); err != nil {
	539	+	t.Fatal("Reap removed a directory that is not ours")
	540	+	}
	541	+	if e.plumb.Resets != 1 {
	542	+	t.Fatal("nftables table not reset")
	543	+	}
	544	+	}
	545	+
	546	+	func TestCapacityAndSlotIsolation(t *testing.T) {
	547	+	e := newCEnv(t)
	548	+	d := e.driver(t)
	549	+	a, _ := d.New()
	550	+	b, _ := d.New()
	551	+	if _, err := d.New(); !errors.Is(err, sandbox.ErrCapacity) {
	552	+	t.Fatalf("got %v", err)
	553	+	}
	554	+	for _, sb := range []sandbox.Sandbox{a, b} {
	555	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	556	+	if err != nil {
	557	+	t.Fatal(err)
	558	+	}
	559	+	stub(t, conn)
	560	+	}
	561	+	ifaces := map[string]bool{}
	562	+	for iface := range e.plumb.Attached {
	563	+	ifaces[iface] = true
	564	+	}
	565	+	if !ifaces["erbr0"] \|\| !ifaces["erbr1"] {
	566	+	t.Fatalf("each job needs its own bridge: %v", ifaces)
	567	+	}
	568	+	_ = a.Stop(context.Background())
	569	+	_ = b.Stop(context.Background())
	570	+	if left := e.leftovers(); len(left) != 0 {
	571	+	t.Fatalf("leftovers: %v", left)
	572	+	}
	573	+	}
	574	+
	575	+	func TestStopWithCancelledContext(t *testing.T) {
	576	+	e := newCEnv(t)
	577	+	d := e.driver(t)
	578	+	sb, _ := d.New()
	579	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	580	+	if err != nil {
	581	+	t.Fatal(err)
	582	+	}
	583	+	stub(t, conn)
	584	+	ctx, cancel := context.WithCancel(context.Background())
	585	+	cancel()
	586	+	if err := sb.Stop(ctx); err != nil {
	587	+	t.Fatal(err)
	588	+	}
	589	+	if left := e.leftovers(); len(left) != 0 {
	590	+	t.Fatalf("leftovers: %v", left)
	591	+	}
	592	+	}
	593	+
	594	+	func TestDoubleStartRefused(t *testing.T) {
	595	+	e := newCEnv(t)
	596	+	d := e.driver(t)
	597	+	sb, _ := d.New()
	598	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	599	+	if err != nil {
	600	+	t.Fatal(err)
	601	+	}
	602	+	stub(t, conn)
	603	+	if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err == nil {
	604	+	t.Fatal("second Start accepted")
	605	+	}
	606	+	_ = sb.Stop(context.Background())
	607	+	}
	608	+
	609	+	func mustPrefix(s string) netip.Prefix { return netip.MustParsePrefix(s) }
	610	+
	611	+	func logger(buf *bytes.Buffer) context.Context {
	612	+	return zerowrap.WithCtx(context.Background(), zerowrap.New(zerowrap.Config{Level: "debug", Format: "json", Output: buf}))
	613	+	}
	614	+
	615	+	// S4: the privileged fallback is allowed by default, warned about at the start
	616	+	// of EVERY job, reported as the effective isolation, and refused on demand.
	617	+	func TestPrivilegedWarnedAtEveryJobAndReportedAsIsolation(t *testing.T) {
	618	+	e := newCEnv(t)
	619	+	var buf bytes.Buffer
	620	+	ctx := logger(&buf)
	621	+	d, err := New(ctx, e.cfg, e.opts())
	622	+	if err != nil {
	623	+	t.Fatal(err)
	624	+	}
	625	+	defer func() { _ = d.Close() }()
	626	+	if d.Isolation() != sandbox.IsolationContainerPrivileged \|\| d.Mode() != sandbox.Container {
	627	+	t.Fatalf("isolation %q", d.Isolation())
	628	+	}
	629	+	const msg = "privileged container: host-root equivalent, network policy best effort"
	630	+	if n := strings.Count(buf.String(), msg); n != 1 {
	631	+	t.Fatalf("one warning at driver start, got %d", n)
	632	+	}
	633	+	for i := range 3 {
	634	+	sb, _ := d.New()
	635	+	conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: fmt.Sprint("job", i)})
	636	+	if err != nil {
	637	+	t.Fatal(err)
	638	+	}
	639	+	stub(t, conn)
	640	+	_ = sb.Stop(ctx)
	641	+	}
	642	+	if n := strings.Count(buf.String(), msg); n != 4 {
	643	+	t.Fatalf("the warning must be logged at every job start (1 + 3), got %d:\n%s", n, buf.String())
	644	+	}
	645	+	if !strings.Contains(buf.String(), `"job":"job2"`) {
	646	+	t.Fatal("the warning must name the job")
	647	+	}
	648	+	}
	649	+
	650	+	func TestSysboxIsReportedAndSilent(t *testing.T) {
	651	+	e := newCEnv(t)
	652	+	e.eng.info.Runtimes = []string{"runc", "sysbox-runc"}
	653	+	var buf bytes.Buffer
	654	+	ctx := logger(&buf)
	655	+	d, err := New(ctx, e.cfg, e.opts())
	656	+	if err != nil {
	657	+	t.Fatal(err)
	658	+	}
	659	+	defer func() { _ = d.Close() }()
	660	+	if d.Isolation() != sandbox.IsolationContainerSysbox {
	661	+	t.Fatalf("isolation %q", d.Isolation())
	662	+	}
	663	+	sb, _ := d.New()
	664	+	conn, _ := sb.Start(ctx, sandbox.JobSpec{JobID: "j"})
	665	+	stub(t, conn)
	666	+	_ = sb.Stop(ctx)
	667	+	if strings.Contains(buf.String(), "host-root equivalent") {
	668	+	t.Fatal("no privileged warning with sysbox")
	669	+	}
	670	+	}
	671	+
	672	+	func TestDenyPrivileged(t *testing.T) {
	673	+	e := newCEnv(t)
	674	+	e.cfg.DenyPrivileged = true
	675	+	_, err := New(context.Background(), e.cfg, e.opts())
	676	+	if err == nil \|\| !strings.Contains(err.Error(), "ERMITE_RUNNER_DENY_PRIVILEGED") \|\| !strings.Contains(err.Error(), "sysbox") {
	677	+	t.Fatalf("got %v", err)
	678	+	}
	679	+	// The host lock must be free again after a refused start.
	680	+	if l, err := sandbox.AcquireHostLock(e.cfg.LockFile); err != nil {
	681	+	t.Fatalf("a failed New must release the host lock: %v", err)
	682	+	} else {
	683	+	_ = l.Release()
	684	+	}
	685	+	// With sysbox available the setting does nothing.
	686	+	e2 := newCEnv(t)
	687	+	e2.cfg.DenyPrivileged = true
	688	+	e2.eng.info.Runtimes = []string{"sysbox-runc"}
	689	+	d, err := New(context.Background(), e2.cfg, e2.opts())
	690	+	if err != nil {
	691	+	t.Fatalf("deny-privileged with sysbox must work: %v", err)
	692	+	}
	693	+	_ = d.Close()
	694	+	}
	695	+
	696	+	// S5: pids limit from the config and no swap.
	697	+	func TestHostConfigPidsAndSwap(t *testing.T) {
	698	+	hc := hostConfigFor(containerSpec{Network: "n", MemoryMiB: 2048, CPUs: 3, Pids: 4096, DNS: netip.MustParseAddr("10.200.0.1")})
	699	+	if hc.Memory != 2048<<20 \|\| hc.MemorySwap != hc.Memory {
	700	+	t.Errorf("memory %d swap %d: swap must equal memory (no swap)", hc.Memory, hc.MemorySwap)
	701	+	}
	702	+	if hc.PidsLimit == nil \|\| *hc.PidsLimit != 4096 {
	703	+	t.Errorf("pids limit %v", hc.PidsLimit)
	704	+	}
	705	+	if hc.NanoCPUs != 3_000_000_000 {
	706	+	t.Errorf("cpus %d", hc.NanoCPUs)
	707	+	}
	708	+	if hc.Sysctls["net.ipv6.conf.all.disable_ipv6"] != "1" {
	709	+	t.Error("IPv6 must be disabled in the container")
	710	+	}
	711	+	if hostConfigFor(containerSpec{Network: "n"}).PidsLimit != nil {
	712	+	t.Error("no pids limit configured: leave it unset")
	713	+	}
	714	+	}
	715	+
	716	+	func TestJobPidsFlowsFromConfig(t *testing.T) {
	717	+	e := newCEnv(t)
	718	+	e.cfg.JobPids = 777
	719	+	d := e.driver(t)
	720	+	sb, _ := d.New()
	721	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	722	+	if err != nil {
	723	+	t.Fatal(err)
	724	+	}
	725	+	stub(t, conn)
	726	+	defer func() { _ = sb.Stop(context.Background()) }()
	727	+	e.eng.mu.Lock()
	728	+	defer e.eng.mu.Unlock()
	729	+	for _, s := range e.eng.containers {
	730	+	if s.Pids != 777 {
	731	+	t.Fatalf("spec pids %d", s.Pids)
	732	+	}
	733	+	}
	734	+	}
	735	+
	736	+	// C4: the disk is enforced by a loop-mounted ext4 file of ERMITE_RUNNER_JOB_DISK,
	737	+	// mounted while the job runs and unmounted (before the files are removed) after.
	738	+	func TestScratchDiskIsLoopMountedAndSized(t *testing.T) {
	739	+	e := newCEnv(t)
	740	+	d := e.driver(t)
	741	+	sb, _ := d.New()
	742	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j", DiskMiB: 2048})
	743	+	if err != nil {
	744	+	t.Fatal(err)
	745	+	}
	746	+	stub(t, conn)
	747	+	if e.mounts.count() != 1 {
	748	+	t.Fatalf("mounts: %d", e.mounts.count())
	749	+	}
	750	+	var mkfs, mount string
	751	+	for _, l := range e.run.Lines() {
	752	+	if strings.HasPrefix(l, "mkfs.ext4") && strings.Contains(l, "scratch.ext4") {
	753	+	mkfs = l
	754	+	}
	755	+	if strings.HasPrefix(l, "mount ") {
	756	+	mount = l
	757	+	}
	758	+	}
	759	+	if !strings.Contains(mkfs, "ermite-scratch") \|\| !strings.Contains(mount, "loop") \|\| !strings.Contains(mount, "scratch.ext4") {
	760	+	t.Fatalf("mkfs=%q mount=%q", mkfs, mount)
	761	+	}
	762	+	// The scratch file has the configured size (sparse) and is not visible to the job.
	763	+	matches, _ := filepath.Glob(filepath.Join(e.cfg.StateDir, "containers", sb.ID(), "scratch.ext4"))
	764	+	if len(matches) != 1 {
	765	+	t.Fatal("no scratch file")
	766	+	}
	767	+	if st, _ := os.Stat(matches[0]); st.Size() != 2048<<20 {
	768	+	t.Fatalf("size %d", st.Size())
	769	+	}
	770	+	e.eng.mu.Lock()
	771	+	for _, s := range e.eng.containers {
	772	+	for _, b := range s.Binds {
	773	+	if strings.HasSuffix(b.Source, "scratch.ext4") \|\| b.Source == filepath.Join(e.cfg.StateDir, "containers", sb.ID()) {
	774	+	t.Errorf("the job must not see the disk file or the whole run directory: %+v", b)
	775	+	}
	776	+	}
	777	+	}
	778	+	e.eng.mu.Unlock()
	779	+	if err := sb.Stop(context.Background()); err != nil {
	780	+	t.Fatal(err)
	781	+	}
	782	+	// Order: umount comes before the directory removal (nothing below a mount is deleted).
	783	+	lines := e.run.Lines()
	784	+	if lines[len(lines)-1] != "umount "+filepath.Join(e.cfg.StateDir, "containers", sb.ID(), "scratch") {
	785	+	t.Fatalf("last command: %s", lines[len(lines)-1])
	786	+	}
	787	+	if left := e.leftovers(); len(left) != 0 {
	788	+	t.Fatalf("leftovers: %v", left)
	789	+	}
	790	+	}
	791	+
	792	+	// C5: when the container cannot be removed it may still run: keep the firewall
	793	+	// policy, the bridge, the mount and the files; report ErrLeaked; keep the slot;
	794	+	// let Reap finish.
	795	+	func TestStopKeepsPolicyWhenTheContainerMayStillRun(t *testing.T) {
	796	+	e := newCEnv(t)
	797	+	d := e.driver(t)
	798	+	sb, _ := d.New()
	799	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	800	+	if err != nil {
	801	+	t.Fatal(err)
	802	+	}
	803	+	stub(t, conn)
	804	+	e.eng.removeErr = errors.New("daemon busy")
	805	+	err = sb.Stop(context.Background())
	806	+	if !errors.Is(err, sandbox.ErrLeaked) {
	807	+	t.Fatalf("want ErrLeaked, got %v", err)
	808	+	}
	809	+	if e.plumb.Live() != 1 {
	810	+	t.Fatal("the firewall policy must stay while the container may run")
	811	+	}
	812	+	e.eng.mu.Lock()
	813	+	nets := len(e.eng.networks)
	814	+	e.eng.mu.Unlock()
	815	+	if nets != 1 \|\| e.mounts.count() != 1 {
	816	+	t.Fatalf("the bridge and the scratch mount must stay: nets=%d mounts=%d", nets, e.mounts.count())
	817	+	}
	818	+	if _, err := os.Stat(filepath.Join(e.cfg.StateDir, "containers", sb.ID(), "scratch.ext4")); err != nil {
	819	+	t.Fatal("the scratch file must stay")
	820	+	}
	821	+	if err := sb.Stop(context.Background()); !errors.Is(err, sandbox.ErrLeaked) {
	822	+	t.Fatalf("second Stop: %v", err)
	823	+	}
	824	+	// C8: the slot is taken, a new sandbox gets another slot, Reap is busy while it runs.
	825	+	other, err := d.New()
	826	+	if err != nil {
	827	+	t.Fatal(err)
	828	+	}
	829	+	if err := d.Reap(context.Background()); !errors.Is(err, sandbox.ErrBusy) {
	830	+	t.Fatalf("Reap while a sandbox exists: %v", err)
	831	+	}
	832	+	_ = other.Stop(context.Background())
	833	+	e.eng.removeErr = nil
	834	+	if err := d.Reap(context.Background()); err != nil {
	835	+	t.Fatal(err)
	836	+	}
	837	+	if left := e.leftovers(); len(left) != 0 {
	838	+	t.Fatalf("leftovers after Reap: %v", left)
	839	+	}
	840	+	if d.slots.InUse() != 0 {
	841	+	t.Fatal("Reap must free the leaked slot")
	842	+	}
	843	+	}
	844	+
	845	+	// A scratch disk that cannot be unmounted keeps its files: nothing is deleted
	846	+	// below a live mount.
	847	+	func TestStopNeverDeletesBelowALiveMount(t *testing.T) {
	848	+	e := newCEnv(t)
	849	+	d := e.driver(t)
	850	+	sb, _ := d.New()
	851	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	852	+	if err != nil {
	853	+	t.Fatal(err)
	854	+	}
	855	+	stub(t, conn)
	856	+	e.mounts.failUm = errors.New("target is busy")
	857	+	if err := sb.Stop(context.Background()); !errors.Is(err, sandbox.ErrLeaked) {
	858	+	t.Fatalf("got %v", err)
	859	+	}
	860	+	if _, err := os.Stat(filepath.Join(e.cfg.StateDir, "containers", sb.ID(), "scratch.ext4")); err != nil {
	861	+	t.Fatal("files below a mount that is still there were deleted")
	862	+	}
	863	+	e.mounts.failUm = nil
	864	+	if err := d.Reap(context.Background()); err != nil {
	865	+	t.Fatal(err)
	866	+	}
	867	+	if e.mounts.count() != 0 {
	868	+	t.Fatal("Reap must unmount leftover scratch disks")
	869	+	}
	870	+	if left := e.leftovers(); len(left) != 0 {
	871	+	t.Fatalf("leftovers: %v", left)
	872	+	}
	873	+	}
	874	+
	875	+	// C7: Stop aborts a boot in progress instead of waiting for the timeout.
	876	+	func TestStopAbortsABootInProgress(t *testing.T) {
	877	+	e := newCEnv(t)
	878	+	e.eng.noConnect = true
	879	+	o := e.opts()
	880	+	o.ConnectTimeout = time.Minute
	881	+	d, err := New(context.Background(), e.cfg, o)
	882	+	if err != nil {
	883	+	t.Fatal(err)
	884	+	}
	885	+	defer func() { _ = d.Close() }()
	886	+	sb, _ := d.New()
	887	+	started := make(chan error, 1)
	888	+	go func() { _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); started <- err }()
	889	+	time.Sleep(500 * time.Millisecond)
	890	+	t0 := time.Now()
	891	+	if err := sb.Stop(context.Background()); err != nil {
	892	+	t.Fatal(err)
	893	+	}
	894	+	select {
	895	+	case err := <-started:
	896	+	if !errors.Is(err, sandbox.ErrAborted) {
	897	+	t.Fatalf("Start must report the abort, got %v", err)
	898	+	}
	899	+	case <-time.After(10 * time.Second):
	900	+	t.Fatal("Start was not aborted")
	901	+	}
	902	+	if time.Since(t0) > 5*time.Second {
	903	+	t.Fatalf("Stop took %v", time.Since(t0))
	904	+	}
	905	+	if left := e.leftovers(); len(left) != 0 {
	906	+	t.Fatalf("leftovers: %v", left)
	907	+	}
	908	+	}
	909	+
	910	+	// C16: ID names the Docker objects.
	911	+	func TestSandboxIDNamesDockerObjects(t *testing.T) {
	912	+	e := newCEnv(t)
	913	+	d := e.driver(t)
	914	+	sb, _ := d.New()
	915	+	if !sandbox.ValidID(sb.ID()) {
	916	+	t.Fatal(sb.ID())
	917	+	}
	918	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	919	+	if err != nil {
	920	+	t.Fatal(err)
	921	+	}
	922	+	stub(t, conn)
	923	+	defer func() { _ = sb.Stop(context.Background()) }()
	924	+	e.eng.mu.Lock()
	925	+	defer e.eng.mu.Unlock()
	926	+	for _, s := range e.eng.containers {
	927	+	if s.Name != "ermite-"+sb.ID() {
	928	+	t.Fatalf("container name %q", s.Name)
	929	+	}
	930	+	}
	931	+	}
	932	+
	933	+	// S3: a second runner on the same host is refused whatever its state directory.
	934	+	func TestHostLockAcrossStateDirs(t *testing.T) {
	935	+	e := newCEnv(t)
	936	+	d := e.driver(t)
	937	+	_ = d
	938	+	e2 := newCEnv(t)
	939	+	e2.cfg.LockFile = e.cfg.LockFile // same host, another state directory
	940	+	_, err := New(context.Background(), e2.cfg, e2.opts())
	941	+	if err == nil \|\| !strings.Contains(err.Error(), "another ermite-runner is running on this host") {
	942	+	t.Fatalf("got %v", err)
	943	+	}
	944	+	}
	945	+
	946	+	// C10: a cancelled wait is reported as cancelled, never as a clean exit.
	947	+	func TestAwaitExit(t *testing.T) {
	948	+	res := make(chan container.WaitResponse, 1)
	949	+	errc := make(chan error, 1)
	950	+	res <- container.WaitResponse{StatusCode: 3}
	951	+	if r := <-awaitExit(context.Background(), res, errc); r.Code != 3 \|\| r.Err != nil {
	952	+	t.Fatalf("%+v", r)
	953	+	}
	954	+	errc <- errors.New("daemon gone")
	955	+	if r := <-awaitExit(context.Background(), make(chan container.WaitResponse), errc); r.Code != -1 \|\| r.Err == nil {
	956	+	t.Fatalf("%+v", r)
	957	+	}
	958	+	// Both ready and the context already cancelled: the cancellation wins.
	959	+	ctx, cancel := context.WithCancel(context.Background())
	960	+	cancel()
	961	+	for range 50 {
	962	+	res2 := make(chan container.WaitResponse, 1)
	963	+	res2 <- container.WaitResponse{StatusCode: 0}
	964	+	if r := <-awaitExit(ctx, res2, make(chan error)); r.Err == nil \|\| !errors.Is(r.Err, context.Canceled) \|\| r.Code != -1 {
	965	+	t.Fatalf("a cancelled wait must not look like a clean exit: %+v", r)
	966	+	}
	967	+	}
	968	+	}
	969	+
	970	+	func TestFreeSpaceWarningInContainerMode(t *testing.T) {
	971	+	e := newCEnv(t)
	972	+	var buf bytes.Buffer
	973	+	o := e.opts()
	974	+	o.FreeSpace = func(string) (int64, error) { return 1, nil }
	975	+	d, err := New(logger(&buf), e.cfg, o)
	976	+	if err != nil {
	977	+	t.Fatal(err)
	978	+	}
	979	+	defer func() { _ = d.Close() }()
	980	+	if !strings.Contains(buf.String(), "less free space") {
	981	+	t.Fatalf("no warning: %s", buf.String())
	982	+	}
	983	+	}
	984	+
	985	+	func TestMkfsTooOldRefusedAtStart(t *testing.T) {
	986	+	e := newCEnv(t)
	987	+	run := &cmdrun.Fake{Handle: func(c cmdrun.Call) ([]byte, error) {
	988	+	if c.Name == "mkfs.ext4" {
	989	+	return []byte("mke2fs 1.42.9 (4-Feb-2014)\n"), nil
	990	+	}
	991	+	return nil, nil
	992	+	}}
	993	+	o := e.opts()
	994	+	o.Run = run
	995	+	if _, err := New(context.Background(), e.cfg, o); err == nil \|\| !strings.Contains(err.Error(), "1.43") {
	996	+	t.Fatalf("got %v", err)
	997	+	}
	998	+	}
	999	+
	1000	+	func TestSocketPathTooLongIsExplained(t *testing.T) {
	1001	+	e := newCEnv(t)
	1002	+	e.cfg.StateDir = filepath.Join(e.cfg.StateDir, strings.Repeat("d", 100))
	1003	+	d := e.driver(t)
	1004	+	sb, _ := d.New()
	1005	+	_, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	1006	+	if err == nil \|\| !strings.Contains(err.Error(), "108-byte limit") && !strings.Contains(err.Error(), "limit of a Unix socket") {
	1007	+	t.Fatalf("got %v", err)
	1008	+	}
	1009	+	if left := e.leftovers(); len(left) != 0 {
	1010	+	t.Fatalf("leftovers: %v", left)
	1011	+	}
	1012	+	}

A internal/sandbox/container/docker_api_test.go +145 -0

@@ -0,0 +1,145 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package container
	4	+
	5	+	import (
	6	+	"context"
	7	+	"io"
	8	+	"os"
	9	+	"os/exec"
	10	+	"path/filepath"
	11	+	"strings"
	12	+	"testing"
	13	+	"time"
	14	+
	15	+	"bnema.dev/ermite-runner/internal/cmdrun"
	16	+	"bnema.dev/ermite-runner/internal/execstream"
	17	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	18	+	"bnema.dev/ermite-runner/internal/sandbox"
	19	+	"bnema.dev/ermite-runner/internal/sandbox/disk"
	20	+	"bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
	21	+	)
	22	+
	23	+	// TestDockerAPIWiring runs the real Docker Engine calls (network with a named
	24	+	// bridge, labelled container, bind mounts, wait, removal, reaping by
	25	+	// label) against whatever daemon DOCKER_HOST points at, including a rootless
	26	+	// one. It does not test isolation (a rootless daemon cannot run the nested
	27	+	// dockerd): the integration test in internal/sandbox/integration does.
	28	+	//
	29	+	// ERMITE_RUNNER_IT_DOCKER_API=1 ERMITE_RUNNER_TEST_IMAGE=<image> go test -run DockerAPIWiring ./internal/sandbox/container
	30	+	func TestDockerAPIWiring(t *testing.T) {
	31	+	if os.Getenv("ERMITE_RUNNER_IT_DOCKER_API") != "1" {
	32	+	t.Skip("set ERMITE_RUNNER_IT_DOCKER_API=1 (needs a reachable Docker daemon)")
	33	+	}
	34	+	image := os.Getenv("ERMITE_RUNNER_TEST_IMAGE")
	35	+	if image == "" {
	36	+	t.Fatal("set ERMITE_RUNNER_TEST_IMAGE to an image with /bin/sh (e.g. the runner image)")
	37	+	}
	38	+	if _, err := exec.LookPath("docker"); err != nil {
	39	+	t.Skip("docker CLI needed for the verification")
	40	+	}
	41	+	// A static test executable that plays the executor: the entrypoint of the
	42	+	// container is "<exe> init", and init runs "<exe> exec"... replaced here by a
	43	+	// tiny shell executor that speaks the stream through a Go helper binary.
	44	+	dir := t.TempDir()
	45	+	helper := filepath.Join(dir, "ermite-runner")
	46	+	build := exec.Command("go", "build", "-o", helper, "bnema.dev/ermite-runner")
	47	+	build.Env = append(os.Environ(), "CGO_ENABLED=0", "GOWORK=off")
	48	+	build.Dir = repoRoot(t)
	49	+	if out, err := build.CombinedOutput(); err != nil {
	50	+	t.Fatalf("build runner: %v\n%s", err, out)
	51	+	}
	52	+
	53	+	plumb := sandboxtest.NewPlumbing()
	54	+	cfg := sandbox.Config{
	55	+	Isolation: sandbox.Container, StateDir: filepath.Join(dir, "state"), Capacity: 1, Image: image,
	56	+	JobCPUs: 1, JobMemoryMiB: 512, JobDiskMiB: 1024, UIDBase: 900000,
	57	+	Network: mustPrefix("10.213.0.0/16"), Executable: helper, DNSPort: 0,
	58	+	LockFile: filepath.Join(dir, "host.lock"), JobPids: 4096,
	59	+	}
	60	+	ctx := context.Background()
	61	+	// Loop mounts need root: replace them by plain directories (mkfs still runs).
	62	+	run := noMountRunner{}
	63	+	empty := filepath.Join(dir, "mountinfo")
	64	+	_ = os.WriteFile(empty, nil, 0o644)
	65	+	d, err := New(ctx, cfg, Options{Net: plumb, allowRootless: true, ConnectTimeout: 60 * time.Second, Run: run,
	66	+	Mounter: &disk.Mounter{Run: run, Mountinfo: empty}})
	67	+	if err != nil {
	68	+	t.Fatal(err)
	69	+	}
	70	+	defer func() { _ = d.Close() }()
	71	+	if err := d.Reap(ctx); err != nil {
	72	+	t.Fatal(err)
	73	+	}
	74	+
	75	+	for i := range 2 {
	76	+	sb, err := d.New()
	77	+	if err != nil {
	78	+	t.Fatal(err)
	79	+	}
	80	+	conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: "api-wiring"})
	81	+	if err != nil {
	82	+	t.Fatalf("job %d: %v", i, err)
	83	+	}
	84	+	var out strings.Builder
	85	+	hello, exit, err := execstream.Drive(conn, &execpb.Run{Script: "echo wired; id -u; exit 4"}, nil, func(o *execpb.Output) { out.Write(o.Data) })
	86	+	if err != nil {
	87	+	t.Fatal(err)
	88	+	}
	89	+	if hello.Sandbox != "container" \|\| exit.Code != 4 \|\| !strings.HasPrefix(out.String(), "wired\n") {
	90	+	t.Fatalf("hello=%v exit=%v out=%q", hello, exit, out.String())
	91	+	}
	92	+	// While running, Docker knows our objects by label.
	93	+	if i == 0 {
	94	+	if ids := dockerLines(t, "ps", "-aq", "--filter", "label="+sandbox.LabelJob); len(ids) != 1 {
	95	+	t.Fatalf("expected one labelled container, got %v", ids)
	96	+	}
	97	+	if n := dockerLines(t, "network", "ls", "-q", "--filter", "label="+sandbox.LabelJob); len(n) != 1 {
	98	+	t.Fatalf("expected one labelled network, got %v", n)
	99	+	}
	100	+	}
	101	+	if err := sb.Stop(ctx); err != nil {
	102	+	t.Fatal(err)
	103	+	}
	104	+	}
	105	+	for _, args := range [][]string{
	106	+	{"ps", "-aq", "--filter", "label=" + sandbox.LabelJob},
	107	+	{"network", "ls", "-q", "--filter", "label=" + sandbox.LabelJob},
	108	+	} {
	109	+	if l := dockerLines(t, args...); len(l) != 0 {
	110	+	t.Fatalf("docker %v left %v", args, l)
	111	+	}
	112	+	}
	113	+	if plumb.Live() != 0 {
	114	+	t.Fatal("firewall attachments leaked")
	115	+	}
	116	+	}
	117	+
	118	+	// noMountRunner runs real commands except mount and umount, which succeed
	119	+	// without doing anything.
	120	+	type noMountRunner struct{}
	121	+
	122	+	func (noMountRunner) Run(ctx context.Context, in io.Reader, name string, args ...string) ([]byte, error) {
	123	+	if name == "mount" \|\| name == "umount" {
	124	+	return nil, nil
	125	+	}
	126	+	return cmdrun.Exec{}.Run(ctx, in, name, args...)
	127	+	}
	128	+
	129	+	func dockerLines(t *testing.T, args ...string) []string {
	130	+	t.Helper()
	131	+	out, err := exec.Command("docker", args...).Output()
	132	+	if err != nil {
	133	+	t.Fatalf("docker %v: %v", args, err)
	134	+	}
	135	+	return strings.Fields(string(out))
	136	+	}
	137	+
	138	+	func repoRoot(t *testing.T) string {
	139	+	t.Helper()
	140	+	out, err := exec.Command("go", "list", "-m", "-f", "{{.Dir}}").Output()
	141	+	if err != nil {
	142	+	t.Fatal(err)
	143	+	}
	144	+	return strings.TrimSpace(string(out))
	145	+	}

A internal/sandbox/container/driver.go +637 -0

@@ -0,0 +1,637 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package container is the container sandbox, for hosts without KVM: one
	4	+	// container per job with its own Docker daemon inside (the sysbox runtime when
	5	+	// the daemon lists it, else a privileged container), on its own bridge network
	6	+	// and with its own scratch disk. The scratch disk is a sparse ext4 file of
	7	+	// ERMITE_RUNNER_JOB_DISK, loop-mounted on the host (so it needs root, like the
	8	+	// rest of this mode) and bind-mounted at /scratch: the nested dockerd keeps its
	9	+	// data root there and the job workspace is a directory of it, so a job cannot
	10	+	// write more than its disk size to the host (the container's own writable layer
	11	+	// stays small: nothing big is written outside /scratch and /home/runner/work).
	12	+	// Everything Docker-side carries the label sandbox.LabelJob so that Reap finds
	13	+	// it after a crash.
	14	+	//
	15	+	// The executor stream is a Unix socket in a per-job directory of the runner's
	16	+	// state directory, bind-mounted into the container. The runner executable is
	17	+	// bind-mounted read-only. Both paths are host paths given to the Docker daemon:
	18	+	// the daemon must run on the same host (and mount namespace) as the runner.
	19	+	package container
	20	+
	21	+	import (
	22	+	"context"
	23	+	"errors"
	24	+	"fmt"
	25	+	"net"
	26	+	"net/netip"
	27	+	"os"
	28	+	"path/filepath"
	29	+	"slices"
	30	+	"strings"
	31	+	"sync"
	32	+	"time"
	33	+
	34	+	"github.com/bnema/zerowrap"
	35	+
	36	+	"bnema.dev/ermite-runner/internal/cmdrun"
	37	+	"bnema.dev/ermite-runner/internal/netpolicy"
	38	+	"bnema.dev/ermite-runner/internal/sandbox"
	39	+	"bnema.dev/ermite-runner/internal/sandbox/disk"
	40	+	"bnema.dev/ermite-runner/internal/sandbox/netsetup"
	41	+	)
	42	+
	43	+	// SysboxRuntime is the name of the sysbox runtime in `docker info`.
	44	+	const SysboxRuntime = "sysbox-runc"
	45	+
	46	+	// bridgePrefix must match the "erbr*" wildcard of the nftables policy.
	47	+	const bridgePrefix = "erbr"
	48	+
	49	+	// Options are the testable knobs of the driver.
	50	+	type Options struct {
	51	+	Run cmdrun.Runner
	52	+	// Net overrides the network plumbing (tests).
	53	+	Net netsetup.Plumbing
	54	+	// engine overrides the Docker client (tests).
	55	+	engine engine
	56	+	// allowRootless lets tests exercise the Engine API against the rootless
	57	+	// daemon of a workstation. Never set outside tests: a nested dockerd cannot
	58	+	// work in a rootless daemon.
	59	+	allowRootless bool
	60	+	// ConnectTimeout bounds container start + init + executor connection
	61	+	// (default 90 s).
	62	+	ConnectTimeout time.Duration
	63	+	// Mounter overrides the loop mounts (tests).
	64	+	Mounter *disk.Mounter
	65	+	// FreeSpace overrides statfs for the free-space warning (tests).
	66	+	FreeSpace func(string) (int64, error)
	67	+	}
	68	+
	69	+	// Driver implements sandbox.Driver for containers.
	70	+	type Driver struct {
	71	+	cfg sandbox.Config
	72	+	opt Options
	73	+	eng engine
	74	+	net netsetup.Plumbing
	75	+	mnt disk.Mounter
	76	+	lock *sandbox.HostLock
	77	+	image string
	78	+	runtime string // "" when privileged
	79	+	privileged bool
	80	+	exe string
	81	+
	82	+	mu sync.Mutex
	83	+	slots *sandbox.Slots
	84	+	}
	85	+
	86	+	// New takes the host-wide lock, connects to the Docker daemon (DOCKER_HOST
	87	+	// honoured) and checks it: not rootless, the job image present (pulled if
	88	+	// missing), the runtime decided (sysbox when listed; else privileged, unless
	89	+	// ERMITE_RUNNER_DENY_PRIVILEGED=1, which refuses). IPv4 forwarding is enabled
	90	+	// host-wide and stays on (see the README).
	91	+	func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error) {
	92	+	if opt.Run == nil {
	93	+	opt.Run = cmdrun.Exec{}
	94	+	}
	95	+	if opt.ConnectTimeout == 0 {
	96	+	opt.ConnectTimeout = 90 * time.Second
	97	+	}
	98	+	d := &Driver{cfg: cfg, opt: opt, image: cfg.OCIImage(), slots: sandbox.NewSlots(cfg.Capacity)}
	99	+	if opt.Mounter != nil {
	100	+	d.mnt = *opt.Mounter
	101	+	} else {
	102	+	d.mnt = disk.Mounter{Run: opt.Run}
	103	+	}
	104	+	exe, err := cfg.ExecutablePath()
	105	+	if err != nil {
	106	+	return nil, err
	107	+	}
	108	+	if err := sandbox.CheckStatic(exe); err != nil {
	109	+	return nil, err
	110	+	}
	111	+	d.exe = exe
	112	+
	113	+	if d.lock, err = sandbox.AcquireHostLock(cfg.LockFile); err != nil {
	114	+	return nil, err
	115	+	}
	116	+	eng := opt.engine
	117	+	if eng == nil {
	118	+	e, err := newDockerEngine()
	119	+	if err != nil {
	120	+	_ = d.lock.Release()
	121	+	return nil, err
	122	+	}
	123	+	eng = e
	124	+	}
	125	+	d.eng = eng
	126	+	fail := func(err error) (*Driver, error) {
	127	+	_ = eng.Close()
	128	+	_ = d.lock.Release()
	129	+	return nil, err
	130	+	}
	131	+
	132	+	info, err := eng.Info(ctx)
	133	+	if err != nil {
	134	+	return fail(fmt.Errorf("cannot reach the Docker daemon (DOCKER_HOST): %w", err))
	135	+	}
	136	+	if info.Rootless && !opt.allowRootless {
	137	+	return fail(errRootless)
	138	+	}
	139	+	if info.OSType != "" && info.OSType != "linux" {
	140	+	return fail(fmt.Errorf("the Docker daemon runs %s containers, need linux", info.OSType))
	141	+	}
	142	+	log := zerowrap.FromCtx(ctx)
	143	+	if slices.Contains(info.Runtimes, SysboxRuntime) {
	144	+	d.runtime = SysboxRuntime
	145	+	log.Info().Str("runtime", SysboxRuntime).Msg("container isolation: using the sysbox runtime")
	146	+	} else {
	147	+	if cfg.DenyPrivileged {
	148	+	return fail(fmt.Errorf("the sysbox runtime is not available and %s=1 forbids privileged containers (install sysbox, or unset the setting to accept host-root-equivalent jobs)", sandbox.EnvDenyPrivileged))
	149	+	}
	150	+	d.privileged = true
	151	+	log.Warn().Msg(privilegedWarning)
	152	+	}
	153	+	if err := disk.CheckMkfs(ctx, opt.Run); err != nil {
	154	+	return fail(err)
	155	+	}
	156	+	if err := os.MkdirAll(cfg.StateDir, 0o755); err != nil {
	157	+	return fail(fmt.Errorf("state directory: %w", err))
	158	+	}
	159	+	sandbox.WarnIfLowSpace(ctx, cfg.StateDir, cfg.Capacity, cfg.JobDiskMiB, opt.FreeSpace)
	160	+	if err := eng.EnsureImage(ctx, d.image); err != nil {
	161	+	return fail(err)
	162	+	}
	163	+	d.net = opt.Net
	164	+	if d.net == nil {
	165	+	d.net = netsetup.New(opt.Run, cfg.DNSPort, "")
	166	+	}
	167	+	if err := d.net.EnableForwarding(); err != nil {
	168	+	return fail(err)
	169	+	}
	170	+	return d, nil
	171	+	}
	172	+
	173	+	// maxSockPath is the longest Unix socket path Linux accepts (108 with the NUL).
	174	+	const maxSockPath = 107
	175	+
	176	+	// privilegedWarning is logged when the driver starts and again at every job
	177	+	// start: a privileged container is host-root equivalent.
	178	+	const privilegedWarning = "privileged container: host-root equivalent, network policy best effort"
	179	+
	180	+	// Mode implements sandbox.Driver.
	181	+	func (d *Driver) Mode() sandbox.Mode { return sandbox.Container }
	182	+
	183	+	// Isolation implements sandbox.Driver.
	184	+	func (d *Driver) Isolation() sandbox.Isolation {
	185	+	if d.privileged {
	186	+	return sandbox.IsolationContainerPrivileged
	187	+	}
	188	+	return sandbox.IsolationContainerSysbox
	189	+	}
	190	+
	191	+	// Privileged reports whether jobs run privileged (no sysbox).
	192	+	func (d *Driver) Privileged() bool { return d.privileged }
	193	+
	194	+	// Close implements sandbox.Driver.
	195	+	func (d *Driver) Close() error {
	196	+	if d.net != nil {
	197	+	d.net.Close()
	198	+	}
	199	+	err := d.eng.Close()
	200	+	return errors.Join(err, d.lock.Release())
	201	+	}
	202	+
	203	+	// New implements sandbox.Driver.
	204	+	func (d *Driver) New() (sandbox.Sandbox, error) {
	205	+	id, err := sandbox.NewID()
	206	+	if err != nil {
	207	+	return nil, err
	208	+	}
	209	+	d.mu.Lock()
	210	+	defer d.mu.Unlock()
	211	+	slot, err := d.slots.Acquire()
	212	+	if err != nil {
	213	+	return nil, err
	214	+	}
	215	+	return &containerSandbox{d: d, slot: slot, id: id}, nil
	216	+	}
	217	+
	218	+	func (d *Driver) release(slot int) {
	219	+	d.mu.Lock()
	220	+	d.slots.Release(slot)
	221	+	d.mu.Unlock()
	222	+	}
	223	+
	224	+	func (d *Driver) markLeaked(slot int) {
	225	+	d.mu.Lock()
	226	+	d.slots.MarkLeaked(slot)
	227	+	d.mu.Unlock()
	228	+	}
	229	+
	230	+	func (d *Driver) runDir(id string) string { return filepath.Join(d.cfg.StateDir, "containers", id) }
	231	+
	232	+	// sockDir holds only the executor socket: it is the one directory of the job
	233	+	// that is bind-mounted (as /run/ermite), so the job never sees the scratch disk
	234	+	// file, nor the other host paths of its own sandbox.
	235	+	func (d *Driver) sockDir(id string) string { return filepath.Join(d.runDir(id), "sock") }
	236	+
	237	+	// scratchDir is where the loop-mounted scratch disk of a job is mounted.
	238	+	func (d *Driver) scratchDir(id string) string { return filepath.Join(d.runDir(id), "scratch") }
	239	+
	240	+	// workDir is the job workspace, a directory of the scratch disk.
	241	+	func (d *Driver) workDir(id string) string { return filepath.Join(d.scratchDir(id), "work") }
	242	+
	243	+	// Where the job workspace and its temporary files appear in the container.
	244	+	const (
	245	+	containerWorkspace = "/home/runner/work"
	246	+	containerTmp = "/tmp"
	247	+	)
	248	+
	249	+	type containerSandbox struct {
	250	+	d *Driver
	251	+	slot int
	252	+	id string
	253	+	gate sandbox.BootGate
	254	+
	255	+	mu sync.Mutex
	256	+	started bool
	257	+	stopped bool
	258	+	leakErr error
	259	+	netID string
	260	+	container string
	261	+	mounted bool
	262	+	listener net.Listener
	263	+	conn net.Conn
	264	+	iface string
	265	+	gateway netip.Addr
	266	+	attached bool
	267	+	}
	268	+
	269	+	// ID implements sandbox.Sandbox.
	270	+	func (s *containerSandbox) ID() string { return s.id }
	271	+
	272	+	// Start implements sandbox.Sandbox.
	273	+	func (s *containerSandbox) Start(ctx context.Context, spec sandbox.JobSpec) (sandbox.Conn, error) {
	274	+	// A Stop from another goroutine aborts the boot through the gate.
	275	+	bootCtx, err := s.gate.Begin(ctx)
	276	+	if err != nil {
	277	+	return nil, fmt.Errorf("container sandbox: %w", err)
	278	+	}
	279	+	defer s.gate.End()
	280	+
	281	+	s.mu.Lock()
	282	+	defer s.mu.Unlock()
	283	+	if s.started \|\| s.stopped {
	284	+	return nil, errors.New("container sandbox: Start called twice or after Stop")
	285	+	}
	286	+	s.started = true
	287	+	d := s.d
	288	+	log := zerowrap.FromCtx(ctx)
	289	+	spec = d.cfg.Resolve(spec)
	290	+	if d.privileged {
	291	+	// At every job start, not just at driver start: the job log and the
	292	+	// operator's logs must say what kind of isolation this job got.
	293	+	log.Warn().Str("job", spec.JobID).Str("container", s.id).Msg(privilegedWarning)
	294	+	}
	295	+	conn, err := s.run(bootCtx, spec)
	296	+	if err != nil {
	297	+	var tail string
	298	+	if s.container != "" {
	299	+	tail, _ = d.eng.LogsTail(context.WithoutCancel(ctx), s.container, 40)
	300	+	}
	301	+	cerr := s.cleanupLocked(context.WithoutCancel(ctx))
	302	+	if tail != "" {
	303	+	err = fmt.Errorf("%w\ncontainer log tail:\n%s", err, tail)
	304	+	}
	305	+	return nil, errors.Join(err, cerr)
	306	+	}
	307	+	log.Info().Str("container", s.container).Int("slot", s.slot).Str("job", spec.JobID).Bool("privileged", d.privileged).Msg("container running")
	308	+	return conn, nil
	309	+	}
	310	+
	311	+	// run runs the boot stages in order; each records what it created for cleanup.
	312	+	func (s *containerSandbox) run(ctx context.Context, spec sandbox.JobSpec) (sandbox.Conn, error) {
	313	+	netw, err := netpolicy.Subnet(s.d.cfg.Network, s.slot)
	314	+	if err != nil {
	315	+	return nil, err
	316	+	}
	317	+	accepted, err := s.runSocket()
	318	+	if err != nil {
	319	+	return nil, err
	320	+	}
	321	+	if err := s.runScratch(ctx, spec); err != nil {
	322	+	return nil, err
	323	+	}
	324	+	labels := map[string]string{sandbox.LabelJob: spec.JobID}
	325	+	if err := s.runNetwork(ctx, netw, labels); err != nil {
	326	+	return nil, err
	327	+	}
	328	+	exited, err := s.runContainer(ctx, spec, netw, labels)
	329	+	if err != nil {
	330	+	return nil, err
	331	+	}
	332	+	return s.waitExecutor(ctx, accepted, exited)
	333	+	}
	334	+
	335	+	type acceptResult struct {
	336	+	c net.Conn
	337	+	err error
	338	+	}
	339	+
	340	+	// runSocket creates the per-job directory and listens on the executor socket in
	341	+	// it. The directory exists before anything on the Docker side, so a crash
	342	+	// leaves a recognisable one.
	343	+	func (s *containerSandbox) runSocket() (<-chan acceptResult, error) {
	344	+	d := s.d
	345	+	for _, dir := range []string{d.runDir(s.id), d.sockDir(s.id)} {
	346	+	if err := os.MkdirAll(dir, 0o711); err != nil {
	347	+	return nil, err
	348	+	}
	349	+	if err := os.Chmod(dir, 0o711); err != nil {
	350	+	return nil, err
	351	+	}
	352	+	}
	353	+	dir := d.sockDir(s.id)
	354	+	sock := filepath.Join(dir, sandbox.ContainerSocket)
	355	+	// sockaddr_un.sun_path holds 108 bytes: say so instead of "invalid argument".
	356	+	if len(sock) > maxSockPath {
	357	+	return nil, fmt.Errorf("executor socket path %q is %d bytes, over the %d-byte limit of a Unix socket: use a shorter %s", sock, len(sock), maxSockPath, sandbox.EnvStateDir)
	358	+	}
	359	+	ln, err := net.Listen("unix", sock)
	360	+	if err != nil {
	361	+	return nil, fmt.Errorf("listen executor socket: %w", err)
	362	+	}
	363	+	s.listener = ln
	364	+	// The container's root may be a mapped (sysbox) user: it must be able to
	365	+	// connect. The path is private to the job (random id, 0711 directory).
	366	+	if err := os.Chmod(sock, 0o666); err != nil {
	367	+	return nil, err
	368	+	}
	369	+	accepted := make(chan acceptResult, 1)
	370	+	go func() {
	371	+	c, err := ln.Accept()
	372	+	accepted <- acceptResult{c, err}
	373	+	}()
	374	+	return accepted, nil
	375	+	}
	376	+
	377	+	// runScratch creates the sparse ext4 scratch disk (ERMITE_RUNNER_JOB_DISK) and
	378	+	// loop-mounts it; the workspace directory lives on it.
	379	+	func (s *containerSandbox) runScratch(ctx context.Context, spec sandbox.JobSpec) error {
	380	+	d := s.d
	381	+	file := filepath.Join(d.runDir(s.id), "scratch.ext4")
	382	+	if err := disk.CreateScratch(ctx, d.opt.Run, file, spec.DiskMiB, -1); err != nil {
	383	+	return err
	384	+	}
	385	+	mnt := d.scratchDir(s.id)
	386	+	if err := os.MkdirAll(mnt, 0o755); err != nil {
	387	+	return err
	388	+	}
	389	+	if err := d.mnt.Mount(ctx, file, mnt); err != nil {
	390	+	return err
	391	+	}
	392	+	s.mounted = true
	393	+	// Workspace, temp and the nested dockerd's data root on the scratch disk. The
	394	+	// executor (phase G) sets the job layout and ownership inside the sandbox.
	395	+	for _, sub := range []string{"work", "tmp", "docker"} {
	396	+	if err := os.MkdirAll(filepath.Join(mnt, sub), 0o755); err != nil {
	397	+	return err
	398	+	}
	399	+	}
	400	+	return os.Chmod(filepath.Join(mnt, "tmp"), 0o1777)
	401	+	}
	402	+
	403	+	// runNetwork creates the job's bridge and installs the firewall policy.
	404	+	func (s *containerSandbox) runNetwork(ctx context.Context, netw netpolicy.Net, labels map[string]string) error {
	405	+	d := s.d
	406	+	s.iface = fmt.Sprintf("%s%d", bridgePrefix, s.slot)
	407	+	s.gateway = netw.Gateway
	408	+	netID, err := d.eng.CreateNetwork(ctx, networkSpec{Name: "ermite-" + s.id, Bridge: s.iface, Subnet: netw.Prefix, Gateway: netw.Gateway, Labels: labels})
	409	+	if err != nil {
	410	+	return err
	411	+	}
	412	+	s.netID = netID
	413	+	s.attached = true
	414	+	return d.net.Attach(ctx, netpolicy.Guest{Iface: s.iface, Gateway: netw.Gateway, Addr: netw.Guest})
	415	+	}
	416	+
	417	+	// runContainer creates and starts the container.
	418	+	func (s *containerSandbox) runContainer(ctx context.Context, spec sandbox.JobSpec, netw netpolicy.Net, labels map[string]string) (<-chan exitResult, error) {
	419	+	d := s.d
	420	+	entry := sandbox.GuestExecDir + "/" + sandbox.GuestExecName
	421	+	cid, err := d.eng.CreateContainer(ctx, containerSpec{
	422	+	Name: "ermite-" + s.id,
	423	+	Image: d.image,
	424	+	Entrypoint: []string{entry, "init"},
	425	+	Env: []string{"ERMITE_CONNECT=unix:" + sandbox.ContainerRunDir + "/" + sandbox.ContainerSocket},
	426	+	Labels: labels,
	427	+	Runtime: d.runtime,
	428	+	Privileged: d.privileged,
	429	+	Network: "ermite-" + s.id,
	430	+	IPv4: netw.Guest,
	431	+	DNS: netw.Gateway,
	432	+	Binds: []bindMount{
	433	+	{Source: d.exe, Target: entry, ReadOnly: true},
	434	+	{Source: d.sockDir(s.id), Target: sandbox.ContainerRunDir},
	435	+	{Source: d.scratchDir(s.id), Target: sandbox.GuestScratch},
	436	+	{Source: d.workDir(s.id), Target: containerWorkspace},
	437	+	{Source: filepath.Join(d.scratchDir(s.id), "tmp"), Target: containerTmp},
	438	+	},
	439	+	MemoryMiB: spec.MemoryMiB,
	440	+	CPUs: spec.CPUs,
	441	+	Pids: d.cfg.JobPids,
	442	+	})
	443	+	if err != nil {
	444	+	return nil, err
	445	+	}
	446	+	s.container = cid
	447	+	exited := d.eng.WaitExit(ctx, cid)
	448	+	if err := d.eng.StartContainer(ctx, cid); err != nil {
	449	+	return nil, fmt.Errorf("start container: %w", err)
	450	+	}
	451	+	return exited, nil
	452	+	}
	453	+
	454	+	// waitExecutor waits for the executor to connect. Only ctx (the caller's
	455	+	// context, or Stop through the boot gate), the container exiting, and the
	456	+	// timeout end the wait.
	457	+	func (s *containerSandbox) waitExecutor(ctx context.Context, accepted <-chan acceptResult, exited <-chan exitResult) (sandbox.Conn, error) {
	458	+	timer := time.NewTimer(s.d.opt.ConnectTimeout)
	459	+	defer timer.Stop()
	460	+	select {
	461	+	case r := <-accepted:
	462	+	if r.err != nil {
	463	+	return nil, fmt.Errorf("accept executor connection: %w", r.err)
	464	+	}
	465	+	s.conn = r.c
	466	+	return r.c, nil
	467	+	case ex := <-exited:
	468	+	if ex.Err != nil {
	469	+	return nil, fmt.Errorf("waiting for the container: %w", ex.Err)
	470	+	}
	471	+	return nil, fmt.Errorf("container exited (code %d) before the executor connected", ex.Code)
	472	+	case <-timer.C:
	473	+	return nil, fmt.Errorf("executor did not connect within %s", s.d.opt.ConnectTimeout)
	474	+	case <-ctx.Done():
	475	+	return nil, context.Cause(ctx)
	476	+	}
	477	+	}
	478	+
	479	+	// Stop implements sandbox.Sandbox.
	480	+	func (s *containerSandbox) Stop(ctx context.Context) error {
	481	+	// Abort a boot in progress first: Start holds s.mu for its whole duration.
	482	+	s.gate.Abort()
	483	+	s.mu.Lock()
	484	+	defer s.mu.Unlock()
	485	+	if s.stopped {
	486	+	return s.leakErr
	487	+	}
	488	+	if !s.started {
	489	+	s.releaseLocked()
	490	+	return nil
	491	+	}
	492	+	return s.cleanupLocked(context.WithoutCancel(ctx))
	493	+	}
	494	+
	495	+	func (s *containerSandbox) releaseLocked() {
	496	+	s.stopped = true
	497	+	s.d.release(s.slot)
	498	+	}
	499	+
	500	+	// cleanupLocked removes the container first; only when it is confirmed gone are
	501	+	// the firewall policy, the network, the scratch mount and the files removed. A
	502	+	// container that may still run keeps its policy and bridge: tearing the policy
	503	+	// down under it would remove the isolation it relies on. Failures mark the
	504	+	// sandbox as leaked (sandbox.ErrLeaked): its slot stays taken until Reap.
	505	+	func (s *containerSandbox) cleanupLocked(ctx context.Context) error {
	506	+	d := s.d
	507	+	var errs []error
	508	+	if s.conn != nil {
	509	+	_ = s.conn.Close()
	510	+	s.conn = nil
	511	+	}
	512	+	if s.listener != nil {
	513	+	_ = s.listener.Close()
	514	+	s.listener = nil
	515	+	}
	516	+	if s.container != "" {
	517	+	if err := d.eng.RemoveContainer(ctx, s.container); err != nil {
	518	+	errs = append(errs, fmt.Errorf("remove container: %w", err))
	519	+	} else {
	520	+	s.container = ""
	521	+	}
	522	+	}
	523	+	if s.container == "" {
	524	+	if s.attached {
	525	+	if err := d.net.Detach(ctx, s.iface, s.gateway); err != nil {
	526	+	errs = append(errs, err)
	527	+	} else {
	528	+	s.attached = false
	529	+	}
	530	+	}
	531	+	if s.netID != "" {
	532	+	if err := d.eng.RemoveNetwork(ctx, s.netID); err != nil {
	533	+	errs = append(errs, fmt.Errorf("remove network: %w", err))
	534	+	} else {
	535	+	s.netID = ""
	536	+	}
	537	+	}
	538	+	if s.mounted {
	539	+	if err := d.mnt.Unmount(ctx, d.scratchDir(s.id)); err != nil {
	540	+	errs = append(errs, err)
	541	+	} else {
	542	+	s.mounted = false
	543	+	}
	544	+	}
	545	+	// Never delete files below a mount that is still there.
	546	+	if !s.mounted {
	547	+	if err := os.RemoveAll(d.runDir(s.id)); err != nil {
	548	+	errs = append(errs, err)
	549	+	}
	550	+	}
	551	+	}
	552	+	if len(errs) == 0 {
	553	+	s.releaseLocked()
	554	+	return nil
	555	+	}
	556	+	s.stopped = true
	557	+	d.markLeaked(s.slot)
	558	+	s.leakErr = fmt.Errorf("%w: container sandbox %s: %w", sandbox.ErrLeaked, s.id, errors.Join(errs...))
	559	+	log := zerowrap.FromCtx(ctx)
	560	+	log.Error().Err(s.leakErr).Str("job", s.id).Msg("container cleanup incomplete; call Reap to finish it")
	561	+	return s.leakErr
	562	+	}
	563	+
	564	+	// Reap implements sandbox.Driver: remove every container and network carrying
	565	+	// the job label, the nftables table, the scratch mounts and the per-job
	566	+	// directories, and free the slots of leaked sandboxes. It returns ErrBusy while
	567	+	// a sandbox is running.
	568	+	func (d *Driver) Reap(ctx context.Context) error {
	569	+	log := zerowrap.FromCtx(ctx)
	570	+	d.mu.Lock()
	571	+	busy := d.slots.Busy()
	572	+	d.mu.Unlock()
	573	+	if busy > 0 {
	574	+	return fmt.Errorf("%w (%d running)", sandbox.ErrBusy, busy)
	575	+	}
	576	+	var errs []error
	577	+	ids, err := d.eng.ListContainers(ctx, sandbox.LabelJob)
	578	+	if err != nil {
	579	+	errs = append(errs, fmt.Errorf("list containers: %w", err))
	580	+	}
	581	+	for _, id := range ids {
	582	+	log.Warn().Str("container", id).Msg("removing leftover job container")
	583	+	if err := d.eng.RemoveContainer(ctx, id); err != nil {
	584	+	errs = append(errs, err)
	585	+	}
	586	+	}
	587	+	nets, err := d.eng.ListNetworks(ctx, sandbox.LabelJob)
	588	+	if err != nil {
	589	+	errs = append(errs, fmt.Errorf("list networks: %w", err))
	590	+	}
	591	+	for _, n := range nets {
	592	+	log.Warn().Str("network", n).Msg("removing leftover job network")
	593	+	if err := d.eng.RemoveNetwork(ctx, n); err != nil {
	594	+	errs = append(errs, err)
	595	+	}
	596	+	}
	597	+	if err := d.net.Reset(ctx); err != nil {
	598	+	errs = append(errs, fmt.Errorf("reset nftables table: %w", err))
	599	+	}
	600	+	// Unmount before deleting: a directory with a live mount below it must not
	601	+	// be removed (and EBUSY would be the best case).
	602	+	root := filepath.Join(d.cfg.StateDir, "containers")
	603	+	mounts, merr := d.mnt.MountPoints()
	604	+	if merr != nil {
	605	+	errs = append(errs, fmt.Errorf("read mounts: %w", merr))
	606	+	}
	607	+	stuck := map[string]bool{}
	608	+	for _, mp := range mounts {
	609	+	rel, ok := strings.CutPrefix(mp, root+"/")
	610	+	if !ok {
	611	+	continue
	612	+	}
	613	+	id, _, _ := strings.Cut(rel, "/")
	614	+	if !sandbox.ValidID(id) {
	615	+	continue
	616	+	}
	617	+	log.Warn().Str("mount", mp).Msg("unmounting leftover scratch disk")
	618	+	if err := d.mnt.Unmount(ctx, mp); err != nil {
	619	+	stuck[id] = true
	620	+	errs = append(errs, err)
	621	+	}
	622	+	}
	623	+	ents, _ := os.ReadDir(root)
	624	+	for _, e := range ents {
	625	+	if sandbox.ValidID(e.Name()) && !stuck[e.Name()] {
	626	+	if err := os.RemoveAll(filepath.Join(root, e.Name())); err != nil {
	627	+	errs = append(errs, err)
	628	+	}
	629	+	}
	630	+	}
	631	+	if len(errs) == 0 {
	632	+	d.mu.Lock()
	633	+	d.slots.ReleaseLeaked()
	634	+	d.mu.Unlock()
	635	+	}
	636	+	return errors.Join(errs...)
	637	+	}

A internal/sandbox/container/engine.go +284 -0

@@ -0,0 +1,284 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package container
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"fmt"
	9	+	"io"
	10	+	"net/netip"
	11	+
	12	+	cerrdefs "github.com/containerd/errdefs"
	13	+	"github.com/moby/moby/api/types/container"
	14	+	"github.com/moby/moby/api/types/mount"
	15	+	"github.com/moby/moby/api/types/network"
	16	+	"github.com/moby/moby/client"
	17	+	)
	18	+
	19	+	// engine is the part of the Docker Engine API the driver uses. The real
	20	+	// implementation wraps github.com/moby/moby/client (the maintained client of
	21	+	// the Moby project; github.com/docker/docker/client is its frozen
	22	+	// predecessor); tests use a fake.
	23	+	type engine interface {
	24	+	Info(ctx context.Context) (daemonInfo, error)
	25	+	// EnsureImage returns nil when ref exists locally, pulling it otherwise.
	26	+	EnsureImage(ctx context.Context, ref string) error
	27	+	CreateNetwork(ctx context.Context, spec networkSpec) (string, error)
	28	+	RemoveNetwork(ctx context.Context, id string) error
	29	+	ListNetworks(ctx context.Context, labelKey string) ([]string, error)
	30	+	CreateContainer(ctx context.Context, spec containerSpec) (string, error)
	31	+	StartContainer(ctx context.Context, id string) error
	32	+	// WaitExit returns a channel that yields the exit code once the container
	33	+	// stops running (or -1 and an error when the wait itself fails).
	34	+	WaitExit(ctx context.Context, id string) <-chan exitResult
	35	+	RemoveContainer(ctx context.Context, id string) error
	36	+	ListContainers(ctx context.Context, labelKey string) ([]string, error)
	37	+	LogsTail(ctx context.Context, id string, lines int) (string, error)
	38	+	Close() error
	39	+	}
	40	+
	41	+	type exitResult struct {
	42	+	Code int
	43	+	Err error
	44	+	}
	45	+
	46	+	type daemonInfo struct {
	47	+	Rootless bool
	48	+	Runtimes []string
	49	+	OSType string
	50	+	}
	51	+
	52	+	type networkSpec struct {
	53	+	Name string
	54	+	Bridge string // host bridge device name
	55	+	Subnet netip.Prefix
	56	+	Gateway netip.Addr
	57	+	Labels map[string]string
	58	+	}
	59	+
	60	+	type containerSpec struct {
	61	+	Name string
	62	+	Image string
	63	+	Entrypoint []string
	64	+	Env []string
	65	+	Labels map[string]string
	66	+	Runtime string // "" = daemon default
	67	+	Privileged bool
	68	+	Network string
	69	+	IPv4 netip.Addr
	70	+	DNS netip.Addr
	71	+	Binds []bindMount
	72	+	MemoryMiB int // also the swap limit: no swap
	73	+	CPUs int
	74	+	Pids int // PidsLimit
	75	+	}
	76	+
	77	+	type bindMount struct {
	78	+	Source, Target string
	79	+	ReadOnly bool
	80	+	}
	81	+
	82	+	// dockerEngine implements engine with the moby client.
	83	+	type dockerEngine struct{ cli *client.Client }
	84	+
	85	+	// newDockerEngine connects like the docker CLI: DOCKER_HOST, DOCKER_TLS_VERIFY,
	86	+	// DOCKER_CERT_PATH, DOCKER_API_VERSION, with API version negotiation.
	87	+	func newDockerEngine() (*dockerEngine, error) {
	88	+	cli, err := client.New(client.FromEnv)
	89	+	if err != nil {
	90	+	return nil, fmt.Errorf("docker client: %w", err)
	91	+	}
	92	+	return &dockerEngine{cli: cli}, nil
	93	+	}
	94	+
	95	+	func (e *dockerEngine) Close() error { return e.cli.Close() }
	96	+
	97	+	func (e *dockerEngine) Info(ctx context.Context) (daemonInfo, error) {
	98	+	r, err := e.cli.Info(ctx, client.InfoOptions{})
	99	+	if err != nil {
	100	+	return daemonInfo{}, fmt.Errorf("docker info: %w", err)
	101	+	}
	102	+	di := daemonInfo{OSType: r.Info.OSType}
	103	+	for _, o := range r.Info.SecurityOptions {
	104	+	if o == "name=rootless" {
	105	+	di.Rootless = true
	106	+	}
	107	+	}
	108	+	for name := range r.Info.Runtimes {
	109	+	di.Runtimes = append(di.Runtimes, name)
	110	+	}
	111	+	return di, nil
	112	+	}
	113	+
	114	+	func (e *dockerEngine) EnsureImage(ctx context.Context, ref string) error {
	115	+	if _, err := e.cli.ImageInspect(ctx, ref); err == nil {
	116	+	return nil
	117	+	} else if !cerrdefs.IsNotFound(err) {
	118	+	return fmt.Errorf("inspect image %s: %w", ref, err)
	119	+	}
	120	+	rc, err := e.cli.ImagePull(ctx, ref, client.ImagePullOptions{})
	121	+	if err != nil {
	122	+	return fmt.Errorf("image %s is not present and cannot be pulled (build it with scripts/build-image.sh): %w", ref, err)
	123	+	}
	124	+	defer func() { _ = rc.Close() }()
	125	+	if err := rc.Wait(ctx); err != nil {
	126	+	return fmt.Errorf("pull image %s: %w", ref, err)
	127	+	}
	128	+	return nil
	129	+	}
	130	+
	131	+	func (e *dockerEngine) CreateNetwork(ctx context.Context, s networkSpec) (string, error) {
	132	+	f := false
	133	+	r, err := e.cli.NetworkCreate(ctx, s.Name, client.NetworkCreateOptions{
	134	+	Driver: "bridge",
	135	+	EnableIPv6: &f,
	136	+	IPAM: &network.IPAM{Config: []network.IPAMConfig{{
	137	+	Subnet: s.Subnet, Gateway: s.Gateway,
	138	+	}}},
	139	+	Options: map[string]string{
	140	+	"com.docker.network.bridge.name": s.Bridge,
	141	+	// The runner's nftables table does the NAT, with its own policy.
	142	+	"com.docker.network.bridge.enable_ip_masquerade": "false",
	143	+	},
	144	+	Labels: s.Labels,
	145	+	})
	146	+	if err != nil {
	147	+	return "", fmt.Errorf("create network %s: %w", s.Name, err)
	148	+	}
	149	+	return r.ID, nil
	150	+	}
	151	+
	152	+	func (e *dockerEngine) RemoveNetwork(ctx context.Context, id string) error {
	153	+	_, err := e.cli.NetworkRemove(ctx, id, client.NetworkRemoveOptions{})
	154	+	return ignoreNotFound(err)
	155	+	}
	156	+
	157	+	func (e *dockerEngine) ListNetworks(ctx context.Context, key string) ([]string, error) {
	158	+	r, err := e.cli.NetworkList(ctx, client.NetworkListOptions{Filters: client.Filters{}.Add("label", key)})
	159	+	if err != nil {
	160	+	return nil, err
	161	+	}
	162	+	var ids []string
	163	+	for _, n := range r.Items {
	164	+	ids = append(ids, n.ID)
	165	+	}
	166	+	return ids, nil
	167	+	}
	168	+
	169	+	// hostConfigFor builds the Docker host configuration of a job container.
	170	+	func hostConfigFor(s containerSpec) *container.HostConfig {
	171	+	hc := &container.HostConfig{
	172	+	NetworkMode: container.NetworkMode(s.Network),
	173	+	Privileged: s.Privileged,
	174	+	Runtime: s.Runtime,
	175	+	CgroupnsMode: container.CgroupnsModePrivate,
	176	+	DNS: []netip.Addr{s.DNS},
	177	+	Sysctls: map[string]string{"net.ipv6.conf.all.disable_ipv6": "1", "net.ipv6.conf.default.disable_ipv6": "1"},
	178	+	}
	179	+	if s.MemoryMiB > 0 {
	180	+	hc.Memory = int64(s.MemoryMiB) << 20
	181	+	hc.MemorySwap = hc.Memory // swap = memory: the job cannot use swap
	182	+	}
	183	+	if s.CPUs > 0 {
	184	+	hc.NanoCPUs = int64(s.CPUs) * 1_000_000_000
	185	+	}
	186	+	if s.Pids > 0 {
	187	+	p := int64(s.Pids)
	188	+	hc.PidsLimit = &p
	189	+	}
	190	+	for _, b := range s.Binds {
	191	+	hc.Mounts = append(hc.Mounts, mount.Mount{Type: mount.TypeBind, Source: b.Source, Target: b.Target, ReadOnly: b.ReadOnly,
	192	+	BindOptions: &mount.BindOptions{CreateMountpoint: true}})
	193	+	}
	194	+	return hc
	195	+	}
	196	+
	197	+	func (e *dockerEngine) CreateContainer(ctx context.Context, s containerSpec) (string, error) {
	198	+	hc := hostConfigFor(s)
	199	+	cfg := &container.Config{
	200	+	Image: s.Image,
	201	+	Entrypoint: s.Entrypoint,
	202	+	Cmd: []string{},
	203	+	Env: s.Env,
	204	+	Labels: s.Labels,
	205	+	Hostname: "ermite-job",
	206	+	}
	207	+	nc := &network.NetworkingConfig{EndpointsConfig: map[string]*network.EndpointSettings{
	208	+	s.Network: {IPAMConfig: &network.EndpointIPAMConfig{IPv4Address: s.IPv4}},
	209	+	}}
	210	+	r, err := e.cli.ContainerCreate(ctx, client.ContainerCreateOptions{Name: s.Name, Config: cfg, HostConfig: hc, NetworkingConfig: nc})
	211	+	if err != nil {
	212	+	return "", fmt.Errorf("create container: %w", err)
	213	+	}
	214	+	return r.ID, nil
	215	+	}
	216	+
	217	+	func (e *dockerEngine) StartContainer(ctx context.Context, id string) error {
	218	+	_, err := e.cli.ContainerStart(ctx, id, client.ContainerStartOptions{})
	219	+	return err
	220	+	}
	221	+
	222	+	func (e *dockerEngine) WaitExit(ctx context.Context, id string) <-chan exitResult {
	223	+	w := e.cli.ContainerWait(ctx, id, client.ContainerWaitOptions{Condition: container.WaitConditionNextExit})
	224	+	return awaitExit(ctx, w.Result, w.Error)
	225	+	}
	226	+
	227	+	// awaitExit turns the two channels of a Docker wait into one result. A context
	228	+	// that ended is reported as such (code -1, ctx.Err()), whichever channel the
	229	+	// select picked: a cancelled wait must never look like a clean exit.
	230	+	func awaitExit(ctx context.Context, result <-chan container.WaitResponse, errc <-chan error) <-chan exitResult {
	231	+	out := make(chan exitResult, 1)
	232	+	go func() {
	233	+	var res exitResult
	234	+	select {
	235	+	case r := <-result:
	236	+	res = exitResult{Code: int(r.StatusCode)}
	237	+	case err := <-errc:
	238	+	res = exitResult{Code: -1, Err: err}
	239	+	case <-ctx.Done():
	240	+	res = exitResult{Code: -1, Err: ctx.Err()}
	241	+	}
	242	+	if err := ctx.Err(); err != nil {
	243	+	res = exitResult{Code: -1, Err: err}
	244	+	}
	245	+	out <- res
	246	+	}()
	247	+	return out
	248	+	}
	249	+
	250	+	func (e *dockerEngine) RemoveContainer(ctx context.Context, id string) error {
	251	+	_, err := e.cli.ContainerRemove(ctx, id, client.ContainerRemoveOptions{Force: true, RemoveVolumes: true})
	252	+	return ignoreNotFound(err)
	253	+	}
	254	+
	255	+	func (e *dockerEngine) ListContainers(ctx context.Context, key string) ([]string, error) {
	256	+	r, err := e.cli.ContainerList(ctx, client.ContainerListOptions{All: true, Filters: client.Filters{}.Add("label", key)})
	257	+	if err != nil {
	258	+	return nil, err
	259	+	}
	260	+	var ids []string
	261	+	for _, c := range r.Items {
	262	+	ids = append(ids, c.ID)
	263	+	}
	264	+	return ids, nil
	265	+	}
	266	+
	267	+	func (e *dockerEngine) LogsTail(ctx context.Context, id string, lines int) (string, error) {
	268	+	rc, err := e.cli.ContainerLogs(ctx, id, client.ContainerLogsOptions{ShowStdout: true, ShowStderr: true, Tail: fmt.Sprint(lines)})
	269	+	if err != nil {
	270	+	return "", err
	271	+	}
	272	+	defer func() { _ = rc.Close() }()
	273	+	b, err := io.ReadAll(io.LimitReader(rc, 64<<10))
	274	+	return string(b), err
	275	+	}
	276	+
	277	+	func ignoreNotFound(err error) error {
	278	+	if err == nil \|\| cerrdefs.IsNotFound(err) {
	279	+	return nil
	280	+	}
	281	+	return err
	282	+	}
	283	+
	284	+	var errRootless = errors.New("the Docker daemon is rootless: a job needs its own nested dockerd, which requires a rootful daemon (or the sysbox runtime); point DOCKER_HOST at a rootful daemon or use ERMITE_RUNNER_ISOLATION=vm")

A internal/sandbox/contract.go +30 -0

@@ -0,0 +1,30 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	// The host/guest contract shared by the drivers and the guest side
	6	+	// (internal/guest). Changing a value here changes both sides at once because
	7	+	// the host always attaches its own executable to the sandbox.
	8	+	const (
	9	+	// VsockPort is the vsock port the executor connects to on the host (CID 2).
	10	+	VsockPort = 1024
	11	+	// GuestExecDir and GuestExecName locate the runner executable inside the
	12	+	// sandbox (read-only), in both modes.
	13	+	GuestExecDir = "/opt/ermite"
	14	+	GuestExecName = "ermite-runner"
	15	+	// ContainerRunDir is the per-job directory bind-mounted into containers;
	16	+	// ContainerSocket is the executor stream socket inside it.
	17	+	ContainerRunDir = "/run/ermite"
	18	+	ContainerSocket = "exec.sock"
	19	+	// GuestScratch is where the per-job scratch disk (VM) or volume (container)
	20	+	// is mounted; dockerd keeps its data root there.
	21	+	GuestScratch = "/scratch"
	22	+
	23	+	// LabelJob marks every Docker object created for a job; reaping lists by it.
	24	+	LabelJob = "dev.bnema.ermite-runner.job"
	25	+
	26	+	// Kernel command line keys of the VM (read from /proc/cmdline by init).
	27	+	CmdlineSandbox = "ermite.sandbox" // "vm"
	28	+	CmdlineConnect = "ermite.connect" // "vsock:1024"
	29	+	CmdlineGateway = "ermite.gw" // host side of the /30, also the DNS server
	30	+	)

A internal/sandbox/core_test.go +161 -0

@@ -0,0 +1,161 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"context"
	8	+	"errors"
	9	+	"path/filepath"
	10	+	"strings"
	11	+	"testing"
	12	+	"time"
	13	+	)
	14	+
	15	+	// S3: one runner per host, whatever its state directory.
	16	+	func TestHostLockIsExclusiveAndReleasable(t *testing.T) {
	17	+	p := filepath.Join(t.TempDir(), "run", "ermite-runner.lock")
	18	+	a, err := AcquireHostLock(p)
	19	+	if err != nil {
	20	+	t.Fatal(err)
	21	+	}
	22	+	if _, err := AcquireHostLock(p); err == nil \|\| !strings.Contains(err.Error(), "another ermite-runner is running on this host") {
	23	+	t.Fatalf("a second runner must be refused with a clear message, got %v", err)
	24	+	}
	25	+	if err := a.Release(); err != nil {
	26	+	t.Fatal(err)
	27	+	}
	28	+	if err := a.Release(); err != nil {
	29	+	t.Fatalf("double release: %v", err)
	30	+	}
	31	+	b, err := AcquireHostLock(p)
	32	+	if err != nil {
	33	+	t.Fatalf("the lock must be free after Release: %v", err)
	34	+	}
	35	+	_ = b.Release()
	36	+	var nilLock *HostLock
	37	+	if err := nilLock.Release(); err != nil {
	38	+	t.Fatal(err)
	39	+	}
	40	+	}
	41	+
	42	+	func TestHostLockErrorNamesTheOverride(t *testing.T) {
	43	+	_, err := AcquireHostLock("/proc/nope/ermite.lock")
	44	+	if err == nil \|\| !strings.Contains(err.Error(), EnvLockFile) {
	45	+	t.Fatalf("got %v", err)
	46	+	}
	47	+	}
	48	+
	49	+	func TestConfigNewSettings(t *testing.T) {
	50	+	c, err := LoadConfig(env(nil))
	51	+	if err != nil {
	52	+	t.Fatal(err)
	53	+	}
	54	+	if c.JobPids != 32768 \|\| c.DenyPrivileged \|\| c.LockFile != "/run/ermite-runner.lock" \|\|
	55	+	c.DiskBandwidthMiB != 400 \|\| c.DiskIOPS != 20000 \|\| c.NetMbit != 1000 {
	56	+	t.Fatalf("%+v", c)
	57	+	}
	58	+	c, err = LoadConfig(env(map[string]string{
	59	+	EnvJobPids: "1000", EnvDenyPrivileged: "1", EnvLockFile: "/tmp/x.lock",
	60	+	EnvJobDiskBW: "0", EnvJobDiskIOPS: "5000", EnvJobNetBandwidth: "100",
	61	+	}))
	62	+	if err != nil {
	63	+	t.Fatal(err)
	64	+	}
	65	+	if c.JobPids != 1000 \|\| !c.DenyPrivileged \|\| c.LockFile != "/tmp/x.lock" \|\| c.DiskBandwidthMiB != 0 \|\| c.DiskIOPS != 5000 \|\| c.NetMbit != 100 {
	66	+	t.Fatalf("%+v", c)
	67	+	}
	68	+	for name, m := range map[string]map[string]string{
	69	+	"pids low": {EnvJobPids: "3"},
	70	+	"deny": {EnvDenyPrivileged: "maybe"},
	71	+	"lock rel": {EnvLockFile: "run/x.lock"},
	72	+	"bw negative": {EnvJobDiskBW: "-1"},
	73	+	"net not int": {EnvJobNetBandwidth: "fast"},
	74	+	} {
	75	+	if _, err := LoadConfig(env(m)); err == nil {
	76	+	t.Errorf("%s accepted", name)
	77	+	}
	78	+	}
	79	+	}
	80	+
	81	+	// C8: leaked slots stay taken until ReleaseLeaked; Busy ignores them.
	82	+	func TestSlotsLeakedLifecycle(t *testing.T) {
	83	+	s := NewSlots(3)
	84	+	a, _ := s.Acquire()
	85	+	b, _ := s.Acquire()
	86	+	s.MarkLeaked(a)
	87	+	if s.InUse() != 2 \|\| s.Busy() != 1 {
	88	+	t.Fatalf("InUse=%d Busy=%d", s.InUse(), s.Busy())
	89	+	}
	90	+	c, _ := s.Acquire()
	91	+	if c == a \|\| c == b {
	92	+	t.Fatalf("a leaked slot must not be handed out again: %d", c)
	93	+	}
	94	+	s.Release(a) // a normal release also clears the leak flag
	95	+	if s.InUse() != 2 {
	96	+	t.Fatal(s.InUse())
	97	+	}
	98	+	s.MarkLeaked(b)
	99	+	s.ReleaseLeaked()
	100	+	if s.InUse() != 1 \|\| s.Busy() != 1 {
	101	+	t.Fatalf("after ReleaseLeaked InUse=%d Busy=%d", s.InUse(), s.Busy())
	102	+	}
	103	+	}
	104	+
	105	+	func TestErrLeakedIsMatchable(t *testing.T) {
	106	+	err := errors.Join(errors.New("remove container: busy"), ErrLeaked)
	107	+	if !errors.Is(err, ErrLeaked) {
	108	+	t.Fatal("errors.Is must see ErrLeaked through a join")
	109	+	}
	110	+	}
	111	+
	112	+	// C7: Abort cancels a boot in progress and forbids a later one.
	113	+	func TestBootGate(t *testing.T) {
	114	+	var g BootGate
	115	+	ctx, err := g.Begin(context.Background())
	116	+	if err != nil {
	117	+	t.Fatal(err)
	118	+	}
	119	+	go func() { time.Sleep(20 * time.Millisecond); g.Abort() }()
	120	+	select {
	121	+	case <-ctx.Done():
	122	+	if !errors.Is(context.Cause(ctx), ErrAborted) {
	123	+	t.Fatalf("cause %v", context.Cause(ctx))
	124	+	}
	125	+	case <-time.After(2 * time.Second):
	126	+	t.Fatal("Abort did not cancel the boot context")
	127	+	}
	128	+	g.End()
	129	+	if _, err := g.Begin(context.Background()); !errors.Is(err, ErrAborted) {
	130	+	t.Fatalf("Begin after Abort: %v", err)
	131	+	}
	132	+	// Parent cancellation still works, and End releases the context.
	133	+	var g2 BootGate
	134	+	parent, cancel := context.WithCancel(context.Background())
	135	+	c2, _ := g2.Begin(parent)
	136	+	cancel()
	137	+	<-c2.Done()
	138	+	g2.End()
	139	+	g2.End()
	140	+	}
	141	+
	142	+	func TestWarnIfLowSpace(t *testing.T) {
	143	+	var buf bytes.Buffer
	144	+	ctx := logCtx(&buf)
	145	+	WarnIfLowSpace(ctx, "/x", 2, 641024, func(string) (int64, error) { return 10 1024, nil })
	146	+	if !strings.Contains(buf.String(), "less free space") \|\| !strings.Contains(buf.String(), `"worst_case_mib":131072`) {
	147	+	t.Fatalf("expected a warning, got %q", buf.String())
	148	+	}
	149	+	buf.Reset()
	150	+	WarnIfLowSpace(ctx, "/x", 2, 1024, func(string) (int64, error) { return 10 * 1024, nil })
	151	+	if buf.Len() != 0 {
	152	+	t.Fatalf("no warning expected: %q", buf.String())
	153	+	}
	154	+	WarnIfLowSpace(ctx, "/x", 1, 1, func(string) (int64, error) { return 0, errors.New("statfs failed") })
	155	+	if !strings.Contains(buf.String(), "cannot check") {
	156	+	t.Fatalf("a statfs error must be logged: %q", buf.String())
	157	+	}
	158	+	if n, err := FreeMiB(t.TempDir()); err != nil \|\| n <= 0 {
	159	+	t.Fatalf("FreeMiB: %d %v", n, err)
	160	+	}
	161	+	}

A internal/sandbox/disk/disk.go +83 -0

@@ -0,0 +1,83 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package disk creates the per-job scratch disks (sparse ext4 files) and, for
	4	+	// the container mode, loop-mounts them. Both isolation modes size a job's
	5	+	// writable space the same way: a sparse file of ERMITE_RUNNER_JOB_DISK.
	6	+	package disk
	7	+
	8	+	import (
	9	+	"context"
	10	+	"fmt"
	11	+	"os"
	12	+	"regexp"
	13	+	"strconv"
	14	+	"strings"
	15	+	"syscall"
	16	+
	17	+	"bnema.dev/ermite-runner/internal/cmdrun"
	18	+	)
	19	+
	20	+	// CreateScratch creates the sparse scratch disk of sizeMiB at path and formats
	21	+	// it. The file is created exclusively without following symlinks and handed to
	22	+	// owner (fchown on the open descriptor, never by path) when owner >= 0.
	23	+	func CreateScratch(ctx context.Context, run cmdrun.Runner, path string, sizeMiB, owner int) error {
	24	+	f, err := os.OpenFile(path, os.O_WRONLY\|os.O_CREATE\|os.O_EXCL\|syscall.O_NOFOLLOW, 0o600)
	25	+	if err != nil {
	26	+	return err
	27	+	}
	28	+	if owner >= 0 {
	29	+	if err := f.Chown(owner, owner); err != nil {
	30	+	_ = f.Close()
	31	+	return fmt.Errorf("chown scratch disk: %w", err)
	32	+	}
	33	+	}
	34	+	if err := f.Truncate(int64(sizeMiB) << 20); err != nil {
	35	+	_ = f.Close()
	36	+	return err
	37	+	}
	38	+	if err := f.Close(); err != nil {
	39	+	return err
	40	+	}
	41	+	// No journal and no reserved blocks: it is a throw-away disk. Lazy table
	42	+	// initialisation keeps creation instant and the file sparse.
	43	+	if _, err := run.Run(ctx, nil, "mkfs.ext4", "-q", "-F", "-L", "ermite-scratch", "-m", "0",
	44	+	"-O", "^has_journal,^resize_inode", "-E", "lazy_itable_init=1,nodiscard", path); err != nil {
	45	+	return fmt.Errorf("format scratch disk: %w", err)
	46	+	}
	47	+	return nil
	48	+	}
	49	+
	50	+	var mkfsVersionRe = regexp.MustCompile(`(?m)^mke2fs (\d+)\.(\d+)(?:\.(\d+))?`)
	51	+
	52	+	// MinMkfs is the oldest e2fsprogs the runner accepts: mkfs.ext4 -d (populate
	53	+	// from a directory, without root) appeared in 1.43. scripts/build-image.sh needs
	54	+	// 1.47.1 to populate from a tarball.
	55	+	const MinMkfs = "1.43"
	56	+
	57	+	// CheckMkfs fails when mkfs.ext4 is missing or older than MinMkfs.
	58	+	func CheckMkfs(ctx context.Context, run cmdrun.Runner) error {
	59	+	out, err := run.Run(ctx, nil, "mkfs.ext4", "-V")
	60	+	// mke2fs prints its version on stderr and exits 0 or 1 depending on the
	61	+	// release: judge by the output, not by the exit status.
	62	+	maj, min, ok := parseMkfsVersion(string(out))
	63	+	if !ok {
	64	+	if err != nil {
	65	+	return fmt.Errorf("mkfs.ext4 not usable (install e2fsprogs >= %s): %w", MinMkfs, err)
	66	+	}
	67	+	return fmt.Errorf("cannot read the mkfs.ext4 version from %q (need e2fsprogs >= %s)", strings.TrimSpace(string(out)), MinMkfs)
	68	+	}
	69	+	if maj < 1 \|\| (maj == 1 && min < 43) {
	70	+	return fmt.Errorf("mkfs.ext4 is e2fsprogs %d.%d, need >= %s for mkfs.ext4 -d", maj, min, MinMkfs)
	71	+	}
	72	+	return nil
	73	+	}
	74	+
	75	+	func parseMkfsVersion(s string) (maj, min int, ok bool) {
	76	+	m := mkfsVersionRe.FindStringSubmatch(s)
	77	+	if m == nil {
	78	+	return 0, 0, false
	79	+	}
	80	+	maj, _ = strconv.Atoi(m[1])
	81	+	min, _ = strconv.Atoi(m[2])
	82	+	return maj, min, true
	83	+	}

A internal/sandbox/disk/disk_test.go +126 -0

@@ -0,0 +1,126 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package disk
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"os"
	9	+	"path/filepath"
	10	+	"strings"
	11	+	"testing"
	12	+
	13	+	"bnema.dev/ermite-runner/internal/cmdrun"
	14	+	)
	15	+
	16	+	func TestCreateScratchIsSparseExclusiveAndNoFollow(t *testing.T) {
	17	+	dir := t.TempDir()
	18	+	run := &cmdrun.Fake{}
	19	+	p := filepath.Join(dir, "scratch.ext4")
	20	+	if err := CreateScratch(context.Background(), run, p, 64, os.Getuid()); err != nil {
	21	+	t.Fatal(err)
	22	+	}
	23	+	st, _ := os.Stat(p)
	24	+	if st.Size() != 64<<20 \|\| st.Mode().Perm() != 0o600 {
	25	+	t.Fatalf("%v", st)
	26	+	}
	27	+	if got := run.Lines(); len(got) != 1 \|\| !strings.Contains(got[0], "mkfs.ext4") \|\| !strings.Contains(got[0], "^has_journal") {
	28	+	t.Fatalf("mkfs call: %v", got)
	29	+	}
	30	+	if err := CreateScratch(context.Background(), run, p, 64, -1); err == nil {
	31	+	t.Fatal("an existing file must be refused")
	32	+	}
	33	+	victim := filepath.Join(dir, "victim")
	34	+	_ = os.WriteFile(victim, []byte("x"), 0o600)
	35	+	link := filepath.Join(dir, "link")
	36	+	_ = os.Symlink(victim, link)
	37	+	if err := CreateScratch(context.Background(), run, link, 1, -1); err == nil {
	38	+	t.Fatal("a symlink must be refused")
	39	+	}
	40	+	// mkfs failure is reported.
	41	+	bad := &cmdrun.Fake{Handle: func(cmdrun.Call) ([]byte, error) { return nil, errors.New("boom") }}
	42	+	if err := CreateScratch(context.Background(), bad, filepath.Join(dir, "x.ext4"), 1, -1); err == nil \|\| !strings.Contains(err.Error(), "format scratch disk") {
	43	+	t.Fatalf("%v", err)
	44	+	}
	45	+	}
	46	+
	47	+	// C14: the mkfs.ext4 version is checked.
	48	+	func TestCheckMkfs(t *testing.T) {
	49	+	ver := func(out string, err error) cmdrun.Runner {
	50	+	return &cmdrun.Fake{Handle: func(cmdrun.Call) ([]byte, error) { return []byte(out), err }}
	51	+	}
	52	+	for name, tc := range map[string]struct {
	53	+	r cmdrun.Runner
	54	+	ok bool
	55	+	}{
	56	+	"current": {ver("mke2fs 1.47.4 (6-Mar-2025)\n", nil), true},
	57	+	"exit 1 on -V": {ver("mke2fs 1.46.5 (30-Dec-2021)\nUsing EXT2FS Library version 1.46.5\n", errors.New("exit status 1")), true},
	58	+	"minimum": {ver("mke2fs 1.43 (17-May-2016)\n", nil), true},
	59	+	"too old": {ver("mke2fs 1.42.13 (17-May-2015)\n", nil), false},
	60	+	"major 0": {ver("mke2fs 0.9 (1-Jan-2000)\n", nil), false},
	61	+	"garbage": {ver("hello\n", nil), false},
	62	+	"missing": {ver("", errors.New("executable file not found")), false},
	63	+	} {
	64	+	err := CheckMkfs(context.Background(), tc.r)
	65	+	if (err == nil) != tc.ok {
	66	+	t.Errorf("%s: err=%v ok=%v", name, err, tc.ok)
	67	+	}
	68	+	}
	69	+	if err := CheckMkfs(context.Background(), ver("mke2fs 1.42.9 (4-Feb-2014)", nil)); err == nil \|\| !strings.Contains(err.Error(), "1.43") {
	70	+	t.Errorf("the error must name the minimum: %v", err)
	71	+	}
	72	+	}
	73	+
	74	+	const mountinfo = `22 28 0:21 / /sys rw,nosuid shared:7 - sysfs sysfs rw
	75	+	36 28 7:0 / /var/lib/ermite-runner/containers/er00112233/scratch rw,nosuid,nodev shared:9 - ext4 /dev/loop0 rw
	76	+	37 28 7:1 / /srv/with\040space rw - ext4 /dev/loop1 rw
	77	+	`
	78	+
	79	+	func TestMounterIsMountedAndUnmount(t *testing.T) {
	80	+	mi := filepath.Join(t.TempDir(), "mountinfo")
	81	+	_ = os.WriteFile(mi, []byte(mountinfo), 0o644)
	82	+	f := &cmdrun.Fake{}
	83	+	m := Mounter{Run: f, Mountinfo: mi}
	84	+	for dir, want := range map[string]bool{
	85	+	"/var/lib/ermite-runner/containers/er00112233/scratch": true,
	86	+	"/srv/with space": true,
	87	+	"/var/lib/ermite-runner/containers/er00112233": false,
	88	+	"/nowhere": false,
	89	+	} {
	90	+	if got, err := m.IsMounted(dir); err != nil \|\| got != want {
	91	+	t.Errorf("%s: %v %v", dir, got, err)
	92	+	}
	93	+	}
	94	+	if err := m.Unmount(context.Background(), "/nowhere"); err != nil \|\| len(f.Calls) != 0 {
	95	+	t.Fatalf("a directory that is not mounted needs no umount: %v %v", err, f.Lines())
	96	+	}
	97	+	if err := m.Unmount(context.Background(), "/var/lib/ermite-runner/containers/er00112233/scratch"); err != nil {
	98	+	t.Fatal(err)
	99	+	}
	100	+	if got := f.Lines(); len(got) != 1 \|\| got[0] != "umount /var/lib/ermite-runner/containers/er00112233/scratch" {
	101	+	t.Fatalf("%v", got)
	102	+	}
	103	+	if err := m.Mount(context.Background(), "/s/scratch.ext4", "/s/mnt"); err != nil {
	104	+	t.Fatal(err)
	105	+	}
	106	+	if last := f.Lines()[1]; !strings.Contains(last, "loop,noatime") \|\| !strings.HasSuffix(last, "/s/scratch.ext4 /s/mnt") {
	107	+	t.Fatalf("mount call: %s", last)
	108	+	}
	109	+	bad := Mounter{Run: &cmdrun.Fake{Handle: func(cmdrun.Call) ([]byte, error) { return nil, errors.New("busy") }}, Mountinfo: mi}
	110	+	if err := bad.Unmount(context.Background(), "/srv/with space"); err == nil {
	111	+	t.Fatal("umount failure must be reported")
	112	+	}
	113	+	if _, err := (Mounter{Mountinfo: "/nonexistent/mountinfo"}).IsMounted("/x"); err == nil {
	114	+	t.Fatal("an unreadable mountinfo must be an error, not 'not mounted'")
	115	+	}
	116	+	}
	117	+
	118	+	func TestUnescapeMountinfo(t *testing.T) {
	119	+	for in, want := range map[string]string{
	120	+	`/plain`: "/plain", `/a\040b`: "/a b", `/tab\011x`: "/tab\tx", `/bs\134x`: `/bs\x`, `/bad\9zz`: `/bad\9zz`, `/end\04`: `/end\04`,
	121	+	} {
	122	+	if got := unescapeMountinfo(in); got != want {
	123	+	t.Errorf("%q: got %q want %q", in, got, want)
	124	+	}
	125	+	}
	126	+	}

A internal/sandbox/disk/mount.go +127 -0

@@ -0,0 +1,127 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package disk
	4	+
	5	+	import (
	6	+	"bufio"
	7	+	"context"
	8	+	"fmt"
	9	+	"os"
	10	+	"strings"
	11	+
	12	+	"bnema.dev/ermite-runner/internal/cmdrun"
	13	+	)
	14	+
	15	+	// Mounter loop-mounts and unmounts scratch disks. It needs root (loop devices),
	16	+	// like the rest of the container mode.
	17	+	type Mounter struct {
	18	+	Run cmdrun.Runner
	19	+	// Mountinfo is /proc/self/mountinfo; tests point it at a fixture.
	20	+	Mountinfo string
	21	+	}
	22	+
	23	+	// Mount loop-mounts the ext4 file at dir (which must exist). The mount is
	24	+	// deliberately not nosuid/nodev: the nested dockerd keeps its overlay2 layers
	25	+	// here, and overlayfs opens the real files of a layer through the layer's own
	26	+	// mount, so device nodes and setuid binaries of container images would stop
	27	+	// working on a nodev/nosuid backing mount.
	28	+	func (m Mounter) Mount(ctx context.Context, file, dir string) error {
	29	+	if _, err := m.Run.Run(ctx, nil, "mount", "-o", "loop,noatime", "-t", "ext4", file, dir); err != nil {
	30	+	return fmt.Errorf("loop-mount scratch disk: %w", err)
	31	+	}
	32	+	return nil
	33	+	}
	34	+
	35	+	// Unmount unmounts dir; the loop device is released with it (mount -o loop sets
	36	+	// autoclear). A directory that is not a mount point is not an error.
	37	+	func (m Mounter) Unmount(ctx context.Context, dir string) error {
	38	+	mounted, err := m.IsMounted(dir)
	39	+	if err != nil {
	40	+	return err
	41	+	}
	42	+	if !mounted {
	43	+	return nil
	44	+	}
	45	+	if _, err := m.Run.Run(ctx, nil, "umount", dir); err != nil {
	46	+	return fmt.Errorf("unmount scratch disk: %w", err)
	47	+	}
	48	+	return nil
	49	+	}
	50	+
	51	+	// IsMounted reports whether dir is a mount point.
	52	+	func (m Mounter) IsMounted(dir string) (bool, error) {
	53	+	mp, err := m.MountPoints()
	54	+	if err != nil {
	55	+	return false, err
	56	+	}
	57	+	for _, p := range mp {
	58	+	if p == dir {
	59	+	return true, nil
	60	+	}
	61	+	}
	62	+	return false, nil
	63	+	}
	64	+
	65	+	// MountPoints lists every mount point of the mount namespace.
	66	+	func (m Mounter) MountPoints() ([]string, error) {
	67	+	path := m.Mountinfo
	68	+	if path == "" {
	69	+	path = "/proc/self/mountinfo"
	70	+	}
	71	+	f, err := os.Open(path)
	72	+	if err != nil {
	73	+	return nil, err
	74	+	}
	75	+	defer func() { _ = f.Close() }()
	76	+	var out []string
	77	+	sc := bufio.NewScanner(f)
	78	+	sc.Buffer(make([]byte, 64<<10), 1<<20)
	79	+	for sc.Scan() {
	80	+	if mp, ok := mountPointOf(sc.Text()); ok {
	81	+	out = append(out, mp)
	82	+	}
	83	+	}
	84	+	return out, sc.Err()
	85	+	}
	86	+
	87	+	// mountPointOf extracts the mount point (5th field) of a mountinfo line, with
	88	+	// the kernel's octal escapes (\040 for a space) decoded.
	89	+	func mountPointOf(line string) (string, bool) {
	90	+	f := strings.Fields(line)
	91	+	if len(f) < 5 {
	92	+	return "", false
	93	+	}
	94	+	return unescapeMountinfo(f[4]), true
	95	+	}
	96	+
	97	+	func unescapeMountinfo(s string) string {
	98	+	if !strings.Contains(s, `\`) {
	99	+	return s
	100	+	}
	101	+	var b strings.Builder
	102	+	for i := 0; i < len(s); i++ {
	103	+	if s[i] == '\\' && i+3 < len(s) {
	104	+	if v, ok := octal3(s[i+1 : i+4]); ok {
	105	+	b.WriteByte(v)
	106	+	i += 3
	107	+	continue
	108	+	}
	109	+	}
	110	+	b.WriteByte(s[i])
	111	+	}
	112	+	return b.String()
	113	+	}
	114	+
	115	+	func octal3(s string) (byte, bool) {
	116	+	if len(s) != 3 {
	117	+	return 0, false
	118	+	}
	119	+	var v int
	120	+	for _, c := range s {
	121	+	if c < '0' \|\| c > '7' {
	122	+	return 0, false
	123	+	}
	124	+	v = v*8 + int(c-'0')
	125	+	}
	126	+	return byte(v), v < 256
	127	+	}

A internal/sandbox/drivers/drivers.go +24 -0

@@ -0,0 +1,24 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package drivers wires the sandbox drivers into sandbox.Select.
	4	+	package drivers
	5	+
	6	+	import (
	7	+	"context"
	8	+
	9	+	"bnema.dev/ermite-runner/internal/sandbox"
	10	+	"bnema.dev/ermite-runner/internal/sandbox/container"
	11	+	"bnema.dev/ermite-runner/internal/sandbox/vm"
	12	+	)
	13	+
	14	+	// Factories returns the real vm and container driver factories.
	15	+	func Factories() sandbox.Factories {
	16	+	return sandbox.Factories{
	17	+	VM: func(ctx context.Context, cfg sandbox.Config) (sandbox.Driver, error) {
	18	+	return vm.New(ctx, cfg, vm.Options{})
	19	+	},
	20	+	Container: func(ctx context.Context, cfg sandbox.Config) (sandbox.Driver, error) {
	21	+	return container.New(ctx, cfg, container.Options{})
	22	+	},
	23	+	}
	24	+	}

A internal/sandbox/gate.go +55 -0

@@ -0,0 +1,55 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"sync"
	9	+	)
	10	+
	11	+	// BootGate lets Stop abort a Start in progress. Start holds the sandbox's own
	12	+	// mutex for the whole boot (up to the boot timeout), so Stop could otherwise
	13	+	// only wait for it; with a gate, Stop first calls Abort, which cancels the
	14	+	// context Start runs its boot with, and then takes the mutex.
	15	+	type BootGate struct {
	16	+	mu sync.Mutex
	17	+	cancel context.CancelFunc
	18	+	aborted bool
	19	+	}
	20	+
	21	+	// ErrAborted is the cause of a boot cancelled by Stop.
	22	+	var ErrAborted = errors.New("sandbox: start aborted by Stop")
	23	+
	24	+	// Begin returns the context to boot with: parent, cancelled by Abort. It fails
	25	+	// when Abort was already called (Stop before Start).
	26	+	func (g *BootGate) Begin(parent context.Context) (context.Context, error) {
	27	+	g.mu.Lock()
	28	+	defer g.mu.Unlock()
	29	+	if g.aborted {
	30	+	return nil, ErrAborted
	31	+	}
	32	+	ctx, cancel := context.WithCancelCause(parent)
	33	+	g.cancel = func() { cancel(ErrAborted) }
	34	+	return ctx, nil
	35	+	}
	36	+
	37	+	// End must be called when the boot finished, successfully or not.
	38	+	func (g *BootGate) End() {
	39	+	g.mu.Lock()
	40	+	defer g.mu.Unlock()
	41	+	if g.cancel != nil {
	42	+	g.cancel()
	43	+	g.cancel = nil
	44	+	}
	45	+	}
	46	+
	47	+	// Abort cancels a boot in progress and forbids a later Begin. It never blocks.
	48	+	func (g *BootGate) Abort() {
	49	+	g.mu.Lock()
	50	+	defer g.mu.Unlock()
	51	+	g.aborted = true
	52	+	if g.cancel != nil {
	53	+	g.cancel()
	54	+	}
	55	+	}

A internal/sandbox/id.go +30 -0

@@ -0,0 +1,30 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"crypto/rand"
	7	+	"encoding/hex"
	8	+	"fmt"
	9	+	"strings"
	10	+	)
	11	+
	12	+	// NewID returns "er" + 8 random hex digits: short (socket paths), unique enough
	13	+	// for concurrent sandboxes, and safe in file, interface and Docker object names.
	14	+	func NewID() (string, error) {
	15	+	var b [4]byte
	16	+	if _, err := rand.Read(b[:]); err != nil {
	17	+	return "", fmt.Errorf("random id: %w", err)
	18	+	}
	19	+	return "er" + hex.EncodeToString(b[:]), nil
	20	+	}
	21	+
	22	+	// ValidID reports whether id has the form produced by NewID. It guards paths and
	23	+	// process matches built from ids read back from disk or from /proc.
	24	+	func ValidID(id string) bool {
	25	+	if len(id) != 10 \|\| !strings.HasPrefix(id, "er") {
	26	+	return false
	27	+	}
	28	+	_, err := hex.DecodeString(id[2:])
	29	+	return err == nil
	30	+	}

A internal/sandbox/integration/container_test.go +43 -0

@@ -0,0 +1,43 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package integration
	4	+
	5	+	import (
	6	+	"os"
	7	+	"os/exec"
	8	+	"strings"
	9	+	"testing"
	10	+
	11	+	"bnema.dev/ermite-runner/internal/sandbox"
	12	+	"bnema.dev/ermite-runner/internal/sandbox/drivers"
	13	+	)
	14	+
	15	+	func TestContainerStubExecutor(t *testing.T) {
	16	+	if os.Getenv("ERMITE_RUNNER_IT_CONTAINER") != "1" {
	17	+	t.Skip("set ERMITE_RUNNER_IT_CONTAINER=1 (needs a rootful Docker daemon and the runner image; see README)")
	18	+	}
	19	+	need(t, "docker")
	20	+	needNetAdmin(t)
	21	+	ctx := ctxWithLog(t)
	22	+	cfg := loadConfig(t, sandbox.Container)
	23	+	// A rootless daemon is refused with a clear error: nothing to test there.
	24	+	if _, err := drivers.Factories().Container(ctx, cfg); err != nil {
	25	+	if strings.Contains(err.Error(), "rootless") {
	26	+	t.Skipf("the Docker daemon is rootless, as designed the driver refuses it: %v", err)
	27	+	}
	28	+	t.Fatal(err)
	29	+	}
	30	+	scenario(t, ctx, cfg, sandbox.Container, func(t *testing.T, sb sandbox.Sandbox) {
	31	+	out, err := exec.Command("docker", "ps", "-q", "--filter", "label="+sandbox.LabelJob).Output()
	32	+	if err != nil {
	33	+	t.Fatal(err)
	34	+	}
	35	+	ids := strings.Fields(string(out))
	36	+	if len(ids) == 0 {
	37	+	t.Fatal("no job container to kill")
	38	+	}
	39	+	if b, err := exec.Command("docker", append([]string{"kill"}, ids...)...).CombinedOutput(); err != nil {
	40	+	t.Fatalf("docker kill: %v: %s", err, b)
	41	+	}
	42	+	})
	43	+	}

A internal/sandbox/integration/integration_test.go +380 -0

@@ -0,0 +1,380 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package integration holds the end-to-end tests of the sandbox drivers with the
	4	+	// real stub executor: output comes back, exit codes are reported, and nothing
	5	+	// survives 10 jobs, including killed ones. They need privileges and host
	6	+	// software, so they are opt-in:
	7	+	//
	8	+	// ERMITE_RUNNER_IT_VM=1 Firecracker + KVM + kernel + base image,
	9	+	// CAP_NET_ADMIN (TAP, nftables, port 53 on the
	10	+	// TAP address)
	11	+	// ERMITE_RUNNER_IT_CONTAINER=1 a ROOTFUL Docker daemon (DOCKER_HOST) and the
	12	+	// runner image, CAP_NET_ADMIN for nftables
	13	+	//
	14	+	// See README.md ("Integration tests") for the exact commands, including how to
	15	+	// run the VM test without root inside a user and network namespace.
	16	+	package integration
	17	+
	18	+	import (
	19	+	"context"
	20	+	"fmt"
	21	+	"os"
	22	+	"os/exec"
	23	+	"path/filepath"
	24	+	"regexp"
	25	+	"strings"
	26	+	"testing"
	27	+	"time"
	28	+
	29	+	"github.com/bnema/zerowrap"
	30	+
	31	+	"bnema.dev/ermite-runner/internal/execstream"
	32	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	33	+	"bnema.dev/ermite-runner/internal/netpolicy"
	34	+	"bnema.dev/ermite-runner/internal/sandbox"
	35	+	"bnema.dev/ermite-runner/internal/sandbox/drivers"
	36	+	)
	37	+
	38	+	// stubScript is the CI-1 stub job of the plan.
	39	+	const stubScript = "uname -a; id; cat /etc/os-release; docker info"
	40	+
	41	+	func ctxWithLog(t *testing.T) context.Context {
	42	+	return zerowrap.WithCtx(context.Background(), zerowrap.New(zerowrap.Config{Level: "info", Format: "console", Output: testWriter{t}}))
	43	+	}
	44	+
	45	+	type testWriter struct{ t *testing.T }
	46	+
	47	+	func (w testWriter) Write(p []byte) (int, error) {
	48	+	w.t.Log(strings.TrimRight(string(p), "\n"))
	49	+	return len(p), nil
	50	+	}
	51	+
	52	+	// buildRunner builds the static runner executable the sandboxes attach.
	53	+	func buildRunner(t *testing.T) string {
	54	+	t.Helper()
	55	+	out, err := exec.Command("go", "list", "-m", "-f", "{{.Dir}}").Output()
	56	+	if err != nil {
	57	+	t.Fatal(err)
	58	+	}
	59	+	exe := filepath.Join(t.TempDir(), "ermite-runner")
	60	+	cmd := exec.Command("go", "build", "-o", exe, "bnema.dev/ermite-runner")
	61	+	cmd.Dir = strings.TrimSpace(string(out))
	62	+	cmd.Env = append(os.Environ(), "CGO_ENABLED=0", "GOWORK=off")
	63	+	if b, err := cmd.CombinedOutput(); err != nil {
	64	+	t.Fatalf("build runner: %v\n%s", err, b)
	65	+	}
	66	+	return exe
	67	+	}
	68	+
	69	+	// loadConfig reads the sandbox settings from the environment, like the runner
	70	+	// does, over a private state directory.
	71	+	func loadConfig(t *testing.T, mode sandbox.Mode) sandbox.Config {
	72	+	t.Helper()
	73	+	getenv := func(k string) string {
	74	+	switch k {
	75	+	case sandbox.EnvIsolation:
	76	+	return string(mode)
	77	+	case sandbox.EnvStateDir:
	78	+	return stateDir(t)
	79	+	case sandbox.EnvCapacity:
	80	+	return "2"
	81	+	case sandbox.EnvLockFile:
	82	+	// A lock of the test's own unless the caller chose one: the default
	83	+	// (/run) is not writable without root, and each test must be able to
	84	+	// take it after the previous one released it.
	85	+	if v := os.Getenv(k); v != "" {
	86	+	return v
	87	+	}
	88	+	return filepath.Join(stateDir(t), "host.lock")
	89	+	case sandbox.EnvJobDisk:
	90	+	if v := os.Getenv(k); v != "" {
	91	+	return v
	92	+	}
	93	+	return "4G"
	94	+	case sandbox.EnvJobMemory:
	95	+	if v := os.Getenv(k); v != "" {
	96	+	return v
	97	+	}
	98	+	return "1024"
	99	+	}
	100	+	return os.Getenv(k)
	101	+	}
	102	+	cfg, err := sandbox.LoadConfig(getenv)
	103	+	if err != nil {
	104	+	t.Fatal(err)
	105	+	}
	106	+	cfg.Executable = buildRunner(t)
	107	+	return cfg
	108	+	}
	109	+
	110	+	var stateDirs = map[*testing.T]string{}
	111	+
	112	+	func stateDir(t *testing.T) string {
	113	+	if d, ok := stateDirs[t]; ok {
	114	+	return d
	115	+	}
	116	+	base := os.Getenv("ERMITE_RUNNER_IT_STATE_DIR")
	117	+	if base == "" {
	118	+	base = t.TempDir()
	119	+	} else {
	120	+	base = filepath.Join(base, fmt.Sprintf("it-%d", time.Now().UnixNano()))
	121	+	_ = os.MkdirAll(base, 0o755)
	122	+	t.Cleanup(func() { _ = os.RemoveAll(base) })
	123	+	}
	124	+	stateDirs[t] = base
	125	+	return base
	126	+	}
	127	+
	128	+	type result struct {
	129	+	out, errOut string
	130	+	hello *execpb.Hello
	131	+	exit *execpb.Exit
	132	+	}
	133	+
	134	+	func runJob(ctx context.Context, conn sandbox.Conn, script string) (result, error) {
	135	+	var so, se strings.Builder
	136	+	hello, exit, err := execstream.Drive(conn, &execpb.Run{Script: script}, nil, func(o *execpb.Output) {
	137	+	if o.Stream == execpb.StreamKind_STREAM_KIND_STDERR {
	138	+	se.Write(o.Data)
	139	+	} else {
	140	+	so.Write(o.Data)
	141	+	}
	142	+	})
	143	+	return result{so.String(), se.String(), hello, exit}, err
	144	+	}
	145	+
	146	+	// need skips the test unless every command is available.
	147	+	func need(t *testing.T, cmds ...string) {
	148	+	t.Helper()
	149	+	for _, c := range cmds {
	150	+	if _, err := exec.LookPath(c); err != nil {
	151	+	t.Skipf("%s not found in PATH", c)
	152	+	}
	153	+	}
	154	+	}
	155	+
	156	+	// needNetAdmin skips unless the process may create TAP devices and nftables
	157	+	// tables (root, CAP_NET_ADMIN, or a user+network namespace).
	158	+	func needNetAdmin(t *testing.T) {
	159	+	t.Helper()
	160	+	need(t, "ip", "nft")
	161	+	const probe = "ertapprobe"
	162	+	if out, err := exec.Command("ip", "tuntap", "add", "dev", probe, "mode", "tap").CombinedOutput(); err != nil {
	163	+	t.Skipf("cannot create a TAP device (need CAP_NET_ADMIN): %v: %s", err, strings.TrimSpace(string(out)))
	164	+	}
	165	+	_ = exec.Command("ip", "link", "del", "dev", probe).Run()
	166	+	if out, err := exec.Command("nft", "list", "tables").CombinedOutput(); err != nil {
	167	+	t.Skipf("cannot use nftables (need CAP_NET_ADMIN): %v: %s", err, strings.TrimSpace(string(out)))
	168	+	}
	169	+	}
	170	+
	171	+	// audit fails the test when anything of a job survives: network devices, the
	172	+	// nftables table, state files, processes.
	173	+	func audit(t *testing.T, stateDir string) {
	174	+	t.Helper()
	175	+	deadline := time.Now().Add(5 * time.Second)
	176	+	var left []string
	177	+	for {
	178	+	left = leftovers(stateDir)
	179	+	if len(left) == 0 \|\| time.Now().After(deadline) {
	180	+	break
	181	+	}
	182	+	time.Sleep(100 * time.Millisecond)
	183	+	}
	184	+	if len(left) != 0 {
	185	+	t.Fatalf("leftovers after the jobs:\n %s", strings.Join(left, "\n "))
	186	+	}
	187	+	}
	188	+
	189	+	var tapRe = regexp.MustCompile(`(?m)^\d+: (ertap\d+\|erbr\d+)[:@]`)
	190	+
	191	+	func leftovers(stateDir string) []string {
	192	+	var left []string
	193	+	if out, err := exec.Command("ip", "-o", "link", "show").Output(); err == nil {
	194	+	for _, m := range tapRe.FindAllStringSubmatch(string(out), -1) {
	195	+	left = append(left, "network device "+m[1])
	196	+	}
	197	+	}
	198	+	if out, err := exec.Command("nft", "list", "tables").Output(); err == nil && strings.Contains(string(out), netpolicy.Table) {
	199	+	left = append(left, "nftables table "+netpolicy.Table)
	200	+	}
	201	+	for _, d := range []string{"vm", "vms", "jail", "containers"} {
	202	+	_ = filepath.WalkDir(filepath.Join(stateDir, d), func(p string, e os.DirEntry, err error) error {
	203	+	if err == nil && p != filepath.Join(stateDir, d) {
	204	+	rel, _ := filepath.Rel(stateDir, p)
	205	+	left = append(left, "file "+rel)
	206	+	}
	207	+	return nil
	208	+	})
	209	+	}
	210	+	for pid, id := range vmmProcs() {
	211	+	left = append(left, fmt.Sprintf("process %d %s", pid, id))
	212	+	}
	213	+	return left
	214	+	}
	215	+
	216	+	// vmmProcs returns the processes whose command line carries one of our VM ids
	217	+	// (and --api-sock or --exec-file), whatever the program is called: the rule the
	218	+	// driver's Reap uses.
	219	+	func vmmProcs() map[int]string {
	220	+	out := map[int]string{}
	221	+	procs, _ := filepath.Glob("/proc/[0-9]*/cmdline")
	222	+	for _, p := range procs {
	223	+	b, err := os.ReadFile(p)
	224	+	if err != nil {
	225	+	continue
	226	+	}
	227	+	args := strings.Split(strings.TrimRight(string(b), "\x00"), "\x00")
	228	+	var id string
	229	+	var marker bool
	230	+	for i, a := range args {
	231	+	switch a {
	232	+	case "--id":
	233	+	if i+1 < len(args) && sandbox.ValidID(args[i+1]) {
	234	+	id = args[i+1]
	235	+	}
	236	+	case "--api-sock", "--exec-file":
	237	+	marker = true
	238	+	}
	239	+	}
	240	+	if id == "" \|\| !marker {
	241	+	continue
	242	+	}
	243	+	var pid int
	244	+	if _, err := fmt.Sscanf(p, "/proc/%d/cmdline", &pid); err == nil {
	245	+	out[pid] = id
	246	+	}
	247	+	}
	248	+	return out
	249	+	}
	250	+
	251	+	func vmmPIDs() []int {
	252	+	var pids []int
	253	+	for pid := range vmmProcs() {
	254	+	pids = append(pids, pid)
	255	+	}
	256	+	return pids
	257	+	}
	258	+
	259	+	func checkStubOutput(t *testing.T, r result, wantSandbox string) {
	260	+	t.Helper()
	261	+	if r.hello == nil \|\| r.hello.Sandbox != wantSandbox {
	262	+	t.Errorf("hello: %v", r.hello)
	263	+	}
	264	+	for _, want := range []string{"Linux", "uid=", "Ubuntu", "Server Version"} {
	265	+	if !strings.Contains(r.out, want) {
	266	+	t.Errorf("output lacks %q:\n%s\nstderr:\n%s", want, r.out, r.errOut)
	267	+	}
	268	+	}
	269	+	if r.exit == nil \|\| r.exit.Code != 0 {
	270	+	t.Errorf("stub exit: %v\nstderr: %s", r.exit, r.errOut)
	271	+	}
	272	+	}
	273	+
	274	+	// jobs runs the common scenario on a driver: 10 jobs, 2 at a time where
	275	+	// capacity allows, including a failing job, a job killed from outside, and a
	276	+	// job abandoned by a "crashed" runner; then audits.
	277	+	func scenario(t testing.T, ctx context.Context, cfg sandbox.Config, mode sandbox.Mode, killSandbox func(t testing.T, sb sandbox.Sandbox)) {
	278	+	t.Helper()
	279	+	drv, err := sandbox.Select(ctx, cfg, drivers.Factories(), sandbox.Prober{})
	280	+	if err != nil {
	281	+	t.Fatal(err)
	282	+	}
	283	+	if drv.Mode() != mode {
	284	+	t.Fatalf("driver mode %s, want %s", drv.Mode(), mode)
	285	+	}
	286	+	// Whatever happens below (a Fatal included), leave nothing behind on the host.
	287	+	t.Cleanup(func() {
	288	+	_ = drv.Close()
	289	+	d2, err := sandbox.Select(context.Background(), cfg, drivers.Factories(), sandbox.Prober{})
	290	+	if err != nil {
	291	+	t.Logf("emergency cleanup: %v", err)
	292	+	return
	293	+	}
	294	+	_ = d2.Reap(context.Background())
	295	+	_ = d2.Close()
	296	+	})
	297	+	if err := drv.Reap(ctx); err != nil {
	298	+	t.Fatalf("initial reap: %v", err)
	299	+	}
	300	+
	301	+	start := func(id string) (sandbox.Sandbox, sandbox.Conn) {
	302	+	sb, err := drv.New()
	303	+	if err != nil {
	304	+	t.Fatal(err)
	305	+	}
	306	+	conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: id})
	307	+	if err != nil {
	308	+	_ = sb.Stop(ctx)
	309	+	t.Fatalf("job %s: %v", id, err)
	310	+	}
	311	+	return sb, conn
	312	+	}
	313	+
	314	+	// Jobs 1-7: the stub, in pairs (capacity 2) except the last.
	315	+	for i := 1; i <= 7; i++ {
	316	+	sb, conn := start(fmt.Sprintf("stub-%d", i))
	317	+	r, err := runJob(ctx, conn, stubScript)
	318	+	if err != nil {
	319	+	t.Fatalf("job %d: %v", i, err)
	320	+	}
	321	+	checkStubOutput(t, r, string(mode))
	322	+	if err := sb.Stop(ctx); err != nil {
	323	+	t.Fatalf("job %d stop: %v", i, err)
	324	+	}
	325	+	}
	326	+
	327	+	// Job 8: a failing script: the exit code comes back.
	328	+	{
	329	+	sb, conn := start("failing")
	330	+	r, err := runJob(ctx, conn, "echo before; echo oops >&2; exit 42")
	331	+	if err != nil {
	332	+	t.Fatal(err)
	333	+	}
	334	+	if r.exit.Code != 42 \|\| !strings.Contains(r.out, "before") \|\| !strings.Contains(r.errOut, "oops") {
	335	+	t.Errorf("failing job: %+v", r)
	336	+	}
	337	+	if err := sb.Stop(ctx); err != nil {
	338	+	t.Fatal(err)
	339	+	}
	340	+	}
	341	+
	342	+	// Job 9: killed from outside while running (the VMM or the container dies).
	343	+	{
	344	+	sb, conn := start("killed")
	345	+	done := make(chan error, 1)
	346	+	go func() { _, err := runJob(ctx, conn, "echo started; sleep 600"); done <- err }()
	347	+	time.Sleep(2 * time.Second)
	348	+	killSandbox(t, sb)
	349	+	select {
	350	+	case err := <-done:
	351	+	if err == nil {
	352	+	t.Error("a killed sandbox must make the session fail")
	353	+	}
	354	+	case <-time.After(30 * time.Second):
	355	+	t.Error("session did not end after the sandbox was killed")
	356	+	}
	357	+	if err := sb.Stop(ctx); err != nil {
	358	+	t.Fatalf("stop after kill: %v", err)
	359	+	}
	360	+	}
	361	+
	362	+	// Job 10: the runner itself dies mid-job (no Stop, driver closed); the next
	363	+	// start reaps everything.
	364	+	{
	365	+	_, conn := start("abandoned")
	366	+	go func() { _, _ = runJob(ctx, conn, "sleep 600") }()
	367	+	time.Sleep(2 * time.Second)
	368	+	_ = drv.Close() // what a crash does: locks and listeners vanish, the sandbox stays
	369	+	_ = conn.Close()
	370	+	drv2, err := sandbox.Select(ctx, cfg, drivers.Factories(), sandbox.Prober{})
	371	+	if err != nil {
	372	+	t.Fatal(err)
	373	+	}
	374	+	defer func() { _ = drv2.Close() }()
	375	+	if err := drv2.Reap(ctx); err != nil {
	376	+	t.Fatalf("reap after crash: %v", err)
	377	+	}
	378	+	}
	379	+	audit(t, cfg.StateDir)
	380	+	}

A internal/sandbox/integration/vm_test.go +103 -0

@@ -0,0 +1,103 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package integration
	4	+
	5	+	import (
	6	+	"context"
	7	+	"os"
	8	+	"strings"
	9	+	"syscall"
	10	+	"testing"
	11	+
	12	+	"bnema.dev/ermite-runner/internal/sandbox"
	13	+	"bnema.dev/ermite-runner/internal/sandbox/drivers"
	14	+	)
	15	+
	16	+	func TestVMStubExecutor(t *testing.T) {
	17	+	if os.Getenv("ERMITE_RUNNER_IT_VM") != "1" {
	18	+	t.Skip("set ERMITE_RUNNER_IT_VM=1 (needs KVM, firecracker, a kernel and a base image; see README)")
	19	+	}
	20	+	need(t, "mkfs.ext4")
	21	+	needNetAdmin(t)
	22	+	ctx := ctxWithLog(t)
	23	+	cfg := loadConfig(t, sandbox.VM)
	24	+	if why := sandbox.VMUnavailable(cfg, sandbox.Prober{}); len(why) > 0 {
	25	+	t.Fatalf("vm isolation is not available here: %s", strings.Join(why, "; "))
	26	+	}
	27	+
	28	+	scenario(t, ctx, cfg, sandbox.VM, func(t *testing.T, sb sandbox.Sandbox) {
	29	+	// Kill every VMM process of ours (found like Reap finds them: by id).
	30	+	pids := vmmPIDs()
	31	+	if len(pids) == 0 {
	32	+	t.Fatal("no VMM process to kill")
	33	+	}
	34	+	for _, pid := range pids {
	35	+	_ = syscall.Kill(pid, syscall.SIGKILL)
	36	+	}
	37	+	})
	38	+	}
	39	+
	40	+	// The guest must reach DNS on its gateway and nothing else on the host.
	41	+	func TestVMNetworkPolicy(t *testing.T) {
	42	+	if os.Getenv("ERMITE_RUNNER_IT_VM") != "1" {
	43	+	t.Skip("set ERMITE_RUNNER_IT_VM=1")
	44	+	}
	45	+	need(t, "mkfs.ext4")
	46	+	needNetAdmin(t)
	47	+	ctx := ctxWithLog(t)
	48	+	cfg := loadConfig(t, sandbox.VM)
	49	+	drv, err := sandbox.Select(ctx, cfg, drivers.Factories(), sandbox.Prober{})
	50	+	if err != nil {
	51	+	t.Fatal(err)
	52	+	}
	53	+	defer func() { _ = drv.Close() }()
	54	+	_ = drv.Reap(ctx)
	55	+	sb, err := drv.New()
	56	+	if err != nil {
	57	+	t.Fatal(err)
	58	+	}
	59	+	defer func() { _ = sb.Stop(context.Background()) }()
	60	+	conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: "netpolicy"})
	61	+	if err != nil {
	62	+	t.Fatal(err)
	63	+	}
	64	+	// 10.200.0.1 is the gateway of slot 0.
	65	+	script := `
	66	+	probe() { timeout 4 bash -c "echo > /dev/tcp/$1/$2" >/dev/null 2>&1; echo "$1:$2 rc=$?"; }
	67	+	probe 10.200.0.1 22 # host SSH-like port: must be DROPPED (timeout, rc=124), not refused
	68	+	probe 10.200.0.1 8080
	69	+	probe 169.254.169.254 80 # cloud metadata
	70	+	probe 100.64.0.1 80 # CGNAT / VPN overlay
	71	+	probe 192.168.0.1 80 # RFC 1918
	72	+	probe 172.16.0.1 80
	73	+	probe 10.0.0.1 80
	74	+	python3 - <<'PY'
	75	+	import socket
	76	+	q=bytes.fromhex("abcd01000001000000000000076578616d706c6503636f6d0000010001")
	77	+	s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM); s.settimeout(8); s.sendto(q,("10.200.0.1",53))
	78	+	try:
	79	+	r=s.recv(512); print("dns answered", len(r))
	80	+	except Exception as e: print("dns silent", e)
	81	+	PY
	82	+	ip -6 addr show scope global \| grep -c inet6 \|\| true
	83	+	`
	84	+	r, err := runJob(ctx, conn, script)
	85	+	if err != nil {
	86	+	t.Fatal(err)
	87	+	}
	88	+	t.Log("\n" + r.out)
	89	+	for _, line := range strings.Split(r.out, "\n") {
	90	+	if !strings.Contains(line, " rc=") {
	91	+	continue
	92	+	}
	93	+	if strings.HasSuffix(line, "rc=0") {
	94	+	t.Errorf("a guest reached a forbidden destination: %s", line)
	95	+	}
	96	+	}
	97	+	if !strings.Contains(r.out, "10.200.0.1:22 rc=124") {
	98	+	t.Errorf("host port must be dropped (timeout), got:\n%s", r.out)
	99	+	}
	100	+	if !strings.Contains(r.out, "dns answered") {
	101	+	t.Errorf("the DNS forwarder on the gateway must answer:\n%s", r.out)
	102	+	}
	103	+	}

A internal/sandbox/lock.go +49 -0

@@ -0,0 +1,49 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"fmt"
	7	+	"os"
	8	+	"path/filepath"
	9	+
	10	+	"golang.org/x/sys/unix"
	11	+	)
	12	+
	13	+	// HostLock is the host-wide single-instance lock. Reaping removes every process,
	14	+	// device, firewall rule and Docker object labelled as the runner's own, so it is
	15	+	// only safe when no other runner of the same host is alive: the lock lives at a
	16	+	// fixed path of the host (DefaultLockFile), not in a state directory, which two
	17	+	// runners could choose differently, and it is taken by both isolation modes.
	18	+	type HostLock struct{ f *os.File }
	19	+
	20	+	// AcquireHostLock takes the lock at path (the default when empty) without
	21	+	// waiting. The kernel releases it when the process dies, so a crash never leaves
	22	+	// it stuck.
	23	+	func AcquireHostLock(path string) (*HostLock, error) {
	24	+	if path == "" {
	25	+	path = DefaultLockFile
	26	+	}
	27	+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
	28	+	return nil, fmt.Errorf("host lock %s: %w (set %s for another path)", path, err, EnvLockFile)
	29	+	}
	30	+	f, err := os.OpenFile(path, os.O_CREATE\|os.O_RDWR, 0o600)
	31	+	if err != nil {
	32	+	return nil, fmt.Errorf("host lock %s: %w (set %s for another path)", path, err, EnvLockFile)
	33	+	}
	34	+	if err := unix.Flock(int(f.Fd()), unix.LOCK_EX\|unix.LOCK_NB); err != nil {
	35	+	_ = f.Close()
	36	+	return nil, fmt.Errorf("another ermite-runner is running on this host (lock %s): only one runner per host is supported, because reaping removes everything the runner owns: %w", path, err)
	37	+	}
	38	+	return &HostLock{f: f}, nil
	39	+	}
	40	+
	41	+	// Release drops the lock; releasing twice is fine.
	42	+	func (l *HostLock) Release() error {
	43	+	if l == nil \|\| l.f == nil {
	44	+	return nil
	45	+	}
	46	+	err := l.f.Close()
	47	+	l.f = nil
	48	+	return err
	49	+	}

A internal/sandbox/netsetup/netsetup.go +91 -0

@@ -0,0 +1,91 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package netsetup is the host-side network plumbing shared by both sandbox
	4	+	// drivers: the nftables policy (package netpolicy), the DNS forwarder (package
	5	+	// dnsfwd) and IPv4 forwarding. Each driver creates its own interface (a TAP or
	6	+	// a Docker bridge) and then Attaches it here.
	7	+	package netsetup
	8	+
	9	+	import (
	10	+	"context"
	11	+	"errors"
	12	+	"fmt"
	13	+	"net/netip"
	14	+	"os"
	15	+	"strings"
	16	+
	17	+	"bnema.dev/ermite-runner/internal/cmdrun"
	18	+	"bnema.dev/ermite-runner/internal/dnsfwd"
	19	+	"bnema.dev/ermite-runner/internal/netpolicy"
	20	+	)
	21	+
	22	+	// Plumbing is the host network surface a driver needs. Net implements it; tests
	23	+	// of the drivers use a fake because binding gateway addresses and loading nft
	24	+	// rules needs privileges.
	25	+	type Plumbing interface {
	26	+	EnableForwarding() error
	27	+	Attach(ctx context.Context, g netpolicy.Guest) error
	28	+	Detach(ctx context.Context, iface string, gateway netip.Addr) error
	29	+	Reset(ctx context.Context) error
	30	+	Close()
	31	+	}
	32	+
	33	+	var _ Plumbing = (*Net)(nil)
	34	+
	35	+	// Net is the host network state of one driver.
	36	+	type Net struct {
	37	+	Policy *netpolicy.Manager
	38	+	DNS *dnsfwd.Forwarder
	39	+	// ForwardFile is the ip_forward sysctl; overridden in tests.
	40	+	ForwardFile string
	41	+	}
	42	+
	43	+	// New builds a Net: nft runs through run, DNS listens on dnsPort and forwards
	44	+	// to the resolvers of resolvConf ("" = /etc/resolv.conf).
	45	+	func New(run cmdrun.Runner, dnsPort int, resolvConf string) *Net {
	46	+	r := dnsfwd.NewResolvers(resolvConf)
	47	+	return &Net{
	48	+	Policy: netpolicy.NewManager(run, "nft"),
	49	+	DNS: dnsfwd.New(r.Addrs, dnsPort),
	50	+	ForwardFile: "/proc/sys/net/ipv4/ip_forward",
	51	+	}
	52	+	}
	53	+
	54	+	// EnableForwarding turns on IPv4 forwarding (NAT needs it). It is a no-op when
	55	+	// already on, and an error that names the fix when it cannot be turned on.
	56	+	func (n *Net) EnableForwarding() error {
	57	+	b, err := os.ReadFile(n.ForwardFile)
	58	+	if err == nil && strings.TrimSpace(string(b)) == "1" {
	59	+	return nil
	60	+	}
	61	+	if err := os.WriteFile(n.ForwardFile, []byte("1\n"), 0o644); err != nil {
	62	+	return fmt.Errorf("enable IPv4 forwarding (%s): %w; set net.ipv4.ip_forward=1", n.ForwardFile, err)
	63	+	}
	64	+	return nil
	65	+	}
	66	+
	67	+	// Attach installs the firewall policy of g and starts the DNS forwarder on its
	68	+	// gateway address, which must already be configured on the host interface.
	69	+	// On error nothing stays installed.
	70	+	func (n *Net) Attach(ctx context.Context, g netpolicy.Guest) error {
	71	+	if err := n.Policy.Add(ctx, g); err != nil {
	72	+	return err
	73	+	}
	74	+	if err := n.DNS.Listen(ctx, g.Gateway); err != nil {
	75	+	return errors.Join(err, n.Policy.Remove(context.WithoutCancel(ctx), g.Iface))
	76	+	}
	77	+	return nil
	78	+	}
	79	+
	80	+	// Detach undoes Attach. It is idempotent and keeps going after an error.
	81	+	func (n *Net) Detach(ctx context.Context, iface string, gateway netip.Addr) error {
	82	+	n.DNS.Close(gateway)
	83	+	return n.Policy.Remove(ctx, iface)
	84	+	}
	85	+
	86	+	// Reset removes the runner's nftables table: used at startup to reap a previous
	87	+	// run. Only the table is touched, never other tables.
	88	+	func (n *Net) Reset(ctx context.Context) error { return n.Policy.Reset(ctx) }
	89	+
	90	+	// Close stops every DNS listener.
	91	+	func (n *Net) Close() { n.DNS.CloseAll() }

A internal/sandbox/netsetup/netsetup_test.go +89 -0

@@ -0,0 +1,89 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package netsetup
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"net/netip"
	9	+	"os"
	10	+	"path/filepath"
	11	+	"strings"
	12	+	"testing"
	13	+
	14	+	"bnema.dev/ermite-runner/internal/cmdrun"
	15	+	"bnema.dev/ermite-runner/internal/netpolicy"
	16	+	)
	17	+
	18	+	func guest() netpolicy.Guest {
	19	+	return netpolicy.Guest{Iface: "ertap0", Gateway: netip.MustParseAddr("127.0.0.1"), Addr: netip.MustParseAddr("10.200.0.2")}
	20	+	}
	21	+
	22	+	func TestAttachDetach(t *testing.T) {
	23	+	f := &cmdrun.Fake{}
	24	+	n := New(f, 0, "")
	25	+	t.Cleanup(n.Close)
	26	+	ctx := context.Background()
	27	+	if err := n.Attach(ctx, guest()); err != nil {
	28	+	t.Skip("cannot bind loopback:", err)
	29	+	}
	30	+	if _, ok := n.DNS.Addr(guest().Gateway); !ok {
	31	+	t.Fatal("DNS forwarder not listening on the gateway")
	32	+	}
	33	+	if err := n.Detach(ctx, "ertap0", guest().Gateway); err != nil {
	34	+	t.Fatal(err)
	35	+	}
	36	+	if _, ok := n.DNS.Addr(guest().Gateway); ok {
	37	+	t.Fatal("DNS forwarder still listening after Detach")
	38	+	}
	39	+	if len(f.Calls) != 2 \|\| f.Calls[1].Stdin != netpolicy.Ruleset(nil) {
	40	+	t.Fatalf("calls: %v", f.Lines())
	41	+	}
	42	+	if err := n.Detach(ctx, "ertap0", guest().Gateway); err != nil {
	43	+	t.Fatalf("Detach must be idempotent: %v", err)
	44	+	}
	45	+	}
	46	+
	47	+	func TestAttachRollsBackPolicyWhenDNSFails(t *testing.T) {
	48	+	f := &cmdrun.Fake{}
	49	+	n := New(f, 0, "")
	50	+	t.Cleanup(n.Close)
	51	+	ctx := context.Background()
	52	+	if err := n.Attach(ctx, guest()); err != nil {
	53	+	t.Skip(err)
	54	+	}
	55	+	before := len(f.Calls)
	56	+	if err := n.Attach(ctx, guest()); err == nil { // already listening
	57	+	t.Fatal("second Attach on the same gateway must fail")
	58	+	}
	59	+	if len(f.Calls) != before+2 {
	60	+	t.Fatalf("expected add then rollback remove, got %v", f.Lines())
	61	+	}
	62	+	}
	63	+
	64	+	func TestAttachPolicyError(t *testing.T) {
	65	+	boom := errors.New("nft failed")
	66	+	n := New(&cmdrun.Fake{Handle: func(cmdrun.Call) ([]byte, error) { return nil, boom }}, 0, "")
	67	+	if err := n.Attach(context.Background(), guest()); !errors.Is(err, boom) {
	68	+	t.Fatalf("got %v", err)
	69	+	}
	70	+	if _, ok := n.DNS.Addr(guest().Gateway); ok {
	71	+	t.Fatal("DNS must not start when the policy failed")
	72	+	}
	73	+	}
	74	+
	75	+	func TestEnableForwarding(t *testing.T) {
	76	+	n := New(&cmdrun.Fake{}, 0, "")
	77	+	n.ForwardFile = filepath.Join(t.TempDir(), "ip_forward")
	78	+	_ = os.WriteFile(n.ForwardFile, []byte("0\n"), 0o644)
	79	+	if err := n.EnableForwarding(); err != nil {
	80	+	t.Fatal(err)
	81	+	}
	82	+	if b, _ := os.ReadFile(n.ForwardFile); strings.TrimSpace(string(b)) != "1" {
	83	+	t.Fatalf("not enabled: %q", b)
	84	+	}
	85	+	n.ForwardFile = filepath.Join(t.TempDir(), "missing", "ip_forward")
	86	+	if err := n.EnableForwarding(); err == nil \|\| !strings.Contains(err.Error(), "ip_forward") {
	87	+	t.Fatalf("want a clear error, got %v", err)
	88	+	}
	89	+	}

A internal/sandbox/sandbox.go +176 -0

@@ -0,0 +1,176 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package sandbox defines the isolation boundary of a CI job: one Sandbox per
	4	+	// job, either a Firecracker micro-VM (package vm) or a container with its own
	5	+	// Docker daemon (package container). Both run the same executable
	6	+	// (`ermite-runner init`, then `ermite-runner exec`) and expose the same framed
	7	+	// stream (package execstream) to the host process.
	8	+	package sandbox
	9	+
	10	+	import (
	11	+	"context"
	12	+	"errors"
	13	+	"io"
	14	+	)
	15	+
	16	+	// Mode is an isolation mode.
	17	+	type Mode string
	18	+
	19	+	// Isolation modes. Auto is a setting only: a Driver always reports VM or
	20	+	// Container.
	21	+	const (
	22	+	Auto Mode = "auto"
	23	+	VM Mode = "vm"
	24	+	Container Mode = "container"
	25	+	)
	26	+
	27	+	// Isolation is the effective isolation of a driver, reported by the host at
	28	+	// registration and shown on the job page.
	29	+	type Isolation string
	30	+
	31	+	// Effective isolations.
	32	+	const (
	33	+	IsolationVM Isolation = "vm"
	34	+	IsolationContainerSysbox Isolation = "container_sysbox"
	35	+	IsolationContainerPrivileged Isolation = "container_privileged"
	36	+	)
	37	+
	38	+	// ErrCapacity is returned by Driver.New when every slot is in use.
	39	+	var ErrCapacity = errors.New("sandbox: no free slot")
	40	+
	41	+	// ErrLeaked is wrapped by the error of Start or Stop when the cleanup of a
	42	+	// sandbox is incomplete: something it created (process, network device,
	43	+	// firewall rules, files, containers) may still exist and its slot stays taken.
	44	+	// The host should stop assigning jobs to this runner slot, let running jobs
	45	+	// finish and call Driver.Reap, which retries the cleanup and frees the slot.
	46	+	var ErrLeaked = errors.New("sandbox: cleanup incomplete, resources leaked")
	47	+
	48	+	// ErrBusy is returned by Driver.Reap while sandboxes are running: reaping
	49	+	// removes everything labelled as ours and would destroy them.
	50	+	var ErrBusy = errors.New("sandbox: cannot reap while sandboxes are running")
	51	+
	52	+	// JobSpec describes the sandbox of one job. Zero resource fields take the
	53	+	// configured defaults.
	54	+	type JobSpec struct {
	55	+	// JobID identifies the job in logs, labels and leftover reports. It never
	56	+	// appears in file or interface names.
	57	+	JobID string
	58	+	// CPUs, MemoryMiB and DiskMiB size the sandbox (ERMITE_RUNNER_JOB_*).
	59	+	CPUs int
	60	+	MemoryMiB int
	61	+	DiskMiB int
	62	+	}
	63	+
	64	+	// Conn is the executor stream: bytes in both directions, framed by execstream.
	65	+	// Closing it ends the session from the host side.
	66	+	type Conn io.ReadWriteCloser
	67	+
	68	+	// Sandbox isolates one job. A Sandbox is single use: Start once, then Stop.
	69	+	type Sandbox interface {
	70	+	// ID is the sandbox's identifier ("er" + 8 hex digits), assigned by
	71	+	// Driver.New. It names the sandbox's files and Docker objects, so logs and
	72	+	// leftover reports can be matched with them.
	73	+	ID() string
	74	+	// Start creates the sandbox, boots it and waits for the executor inside to
	75	+	// connect back. It aborts when ctx ends or when Stop is called from another
	76	+	// goroutine. On error everything created so far has been removed, or the
	77	+	// error wraps ErrLeaked.
	78	+	Start(ctx context.Context, spec JobSpec) (Conn, error)
	79	+	// Stop destroys the sandbox and everything it created (processes, network
	80	+	// devices, firewall rules, files). It is idempotent, safe after a failed or
	81	+	// never-made Start, runs to completion even when ctx is already done, and
	82	+	// aborts a Start in progress. When something could not be removed the error
	83	+	// wraps ErrLeaked.
	84	+	Stop(ctx context.Context) error
	85	+	}
	86	+
	87	+	// Driver creates sandboxes of one mode and reaps what a previous run left.
	88	+	type Driver interface {
	89	+	// Mode is VM or Container.
	90	+	Mode() Mode
	91	+	// Isolation is the effective isolation (vm, container_sysbox,
	92	+	// container_privileged), for registration with ermite.
	93	+	Isolation() Isolation
	94	+	// New returns a fresh Sandbox bound to a free slot (ErrCapacity if none).
	95	+	// The slot is released by Stop, unless the cleanup leaked (ErrLeaked).
	96	+	New() (Sandbox, error)
	97	+	// Reap removes everything a previous run of the runner left behind
	98	+	// (leftover processes, devices, firewall rules, containers, files) and
	99	+	// frees the slots of leaked sandboxes. Call it once at startup, before any
	100	+	// New, and after ErrLeaked when no sandbox is running; it returns ErrBusy
	101	+	// while one is.
	102	+	Reap(ctx context.Context) error
	103	+	// Close releases the driver's own resources (DNS forwarder, API client,
	104	+	// host lock). Sandboxes still running are not stopped.
	105	+	Close() error
	106	+	}
	107	+
	108	+	// Slots hands out small integers, lowest first. Slots identify a sandbox's uid,
	109	+	// network device and /30 subnet, so two live sandboxes never collide. A slot
	110	+	// whose sandbox leaked stays taken (its device name and subnet may still be in
	111	+	// use) until ReleaseLeaked.
	112	+	type Slots struct {
	113	+	used []bool
	114	+	leaked []bool
	115	+	}
	116	+
	117	+	// NewSlots returns a pool of n slots.
	118	+	func NewSlots(n int) *Slots { return &Slots{used: make([]bool, n), leaked: make([]bool, n)} }
	119	+
	120	+	// Acquire returns the lowest free slot, or ErrCapacity. Callers serialise
	121	+	// access (the drivers hold their mutex).
	122	+	func (s *Slots) Acquire() (int, error) {
	123	+	for i, u := range s.used {
	124	+	if !u {
	125	+	s.used[i] = true
	126	+	return i, nil
	127	+	}
	128	+	}
	129	+	return 0, ErrCapacity
	130	+	}
	131	+
	132	+	// Release frees a slot; releasing a free or unknown slot is a no-op.
	133	+	func (s *Slots) Release(i int) {
	134	+	if i >= 0 && i < len(s.used) {
	135	+	s.used[i], s.leaked[i] = false, false
	136	+	}
	137	+	}
	138	+
	139	+	// MarkLeaked keeps slot i taken after a failed cleanup.
	140	+	func (s *Slots) MarkLeaked(i int) {
	141	+	if i >= 0 && i < len(s.used) {
	142	+	s.used[i], s.leaked[i] = true, true
	143	+	}
	144	+	}
	145	+
	146	+	// ReleaseLeaked frees every leaked slot: the caller has reaped what they held.
	147	+	func (s *Slots) ReleaseLeaked() {
	148	+	for i := range s.used {
	149	+	if s.leaked[i] {
	150	+	s.used[i], s.leaked[i] = false, false
	151	+	}
	152	+	}
	153	+	}
	154	+
	155	+	// InUse reports how many slots are taken, leaked ones included.
	156	+	func (s *Slots) InUse() int {
	157	+	n := 0
	158	+	for _, u := range s.used {
	159	+	if u {
	160	+	n++
	161	+	}
	162	+	}
	163	+	return n
	164	+	}
	165	+
	166	+	// Busy reports how many slots belong to a sandbox that is not leaked: running,
	167	+	// starting, or handed out by New and not yet stopped.
	168	+	func (s *Slots) Busy() int {
	169	+	n := 0
	170	+	for i, u := range s.used {
	171	+	if u && !s.leaked[i] {
	172	+	n++
	173	+	}
	174	+	}
	175	+	return n
	176	+	}

A internal/sandbox/sandbox_test.go +271 -0

@@ -0,0 +1,271 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"context"
	8	+	"errors"
	9	+	"os"
	10	+	"path/filepath"
	11	+	"strings"
	12	+	"testing"
	13	+
	14	+	"github.com/bnema/zerowrap"
	15	+	)
	16	+
	17	+	func env(m map[string]string) func(string) string { return func(k string) string { return m[k] } }
	18	+
	19	+	func TestLoadConfigDefaults(t *testing.T) {
	20	+	c, err := LoadConfig(env(nil))
	21	+	if err != nil {
	22	+	t.Fatal(err)
	23	+	}
	24	+	if c.Isolation != Auto \|\| c.Capacity != 1 \|\| c.UIDBase != 900000 \|\| c.StateDir != DefaultStateDir \|\|
	25	+	c.JobCPUs != 2 \|\| c.JobMemoryMiB != 4096 \|\| c.JobDiskMiB != 64*1024 \|\| c.Network.String() != "10.200.0.0/16" \|\| c.DNSPort != 53 {
	26	+	t.Fatalf("%+v", c)
	27	+	}
	28	+	}
	29	+
	30	+	func TestLoadConfigValues(t *testing.T) {
	31	+	c, err := LoadConfig(env(map[string]string{
	32	+	EnvIsolation: "VM", EnvStateDir: "/srv/er", EnvCapacity: "4", EnvImage: "/images/u24",
	33	+	EnvJobCPUs: "8", EnvJobMemory: "16G", EnvJobDisk: "100G", EnvUIDBase: "910000",
	34	+	EnvNetwork: "10.50.0.0/24", EnvJailerMode: "off", EnvFirecracker: "/x/fc", EnvKernel: "/k",
	35	+	}))
	36	+	if err != nil {
	37	+	t.Fatal(err)
	38	+	}
	39	+	if c.Isolation != VM \|\| c.Capacity != 4 \|\| c.JobMemoryMiB != 16384 \|\| c.JobDiskMiB != 100*1024 \|\| c.UIDBase != 910000 \|\|
	40	+	!c.JailerOff \|\| c.Network.String() != "10.50.0.0/24" \|\| c.ImageDir() != "/images/u24" \|\| c.KernelPath() != "/k" \|\|
	41	+	c.BaseImagePath() != "/images/u24/base.ext4" \|\| c.OCIImage() != DefaultOCIImage {
	42	+	t.Fatalf("%+v", c)
	43	+	}
	44	+	c, _ = LoadConfig(env(map[string]string{EnvImage: "registry.example/ermite:1"}))
	45	+	if c.OCIImage() != "registry.example/ermite:1" \|\| c.ImageDir() != filepath.Join(DefaultStateDir, "image") {
	46	+	t.Fatalf("OCI ref misread: %q %q", c.OCIImage(), c.ImageDir())
	47	+	}
	48	+	}
	49	+
	50	+	func TestLoadConfigErrors(t *testing.T) {
	51	+	for name, m := range map[string]map[string]string{
	52	+	"isolation": {EnvIsolation: "docker"},
	53	+	"relative": {EnvStateDir: "state"},
	54	+	"capacity 0": {EnvCapacity: "0"},
	55	+	"capacity abc": {EnvCapacity: "abc"},
	56	+	"cpus": {EnvJobCPUs: "0"},
	57	+	"memory": {EnvJobMemory: "10M"},
	58	+	"disk": {EnvJobDisk: "100M"},
	59	+	"uid base": {EnvUIDBase: "10"},
	60	+	"network v6": {EnvNetwork: "fd00::/64"},
	61	+	"network /31": {EnvNetwork: "10.0.0.0/31"},
	62	+	"network few": {EnvNetwork: "10.0.0.0/30", EnvCapacity: "2"},
	63	+	"jailer mode": {EnvJailerMode: "maybe"},
	64	+	} {
	65	+	if _, err := LoadConfig(env(m)); err == nil {
	66	+	t.Errorf("%s: accepted", name)
	67	+	}
	68	+	}
	69	+	}
	70	+
	71	+	func TestParseSizeMiB(t *testing.T) {
	72	+	for in, want := range map[string]int{"4096": 4096, "512M": 512, "4G": 4096, "64GiB": 65536, "1t": 1048576, " 2g ": 2048, "1024MB": 1024} {
	73	+	if got, err := ParseSizeMiB(in); err != nil \|\| got != want {
	74	+	t.Errorf("%q: %d %v", in, got, err)
	75	+	}
	76	+	}
	77	+	for _, in := range []string{"", "0", "-1", "G", "12K", "1x", "99999999999T"} {
	78	+	if _, err := ParseSizeMiB(in); err == nil {
	79	+	t.Errorf("%q accepted", in)
	80	+	}
	81	+	}
	82	+	}
	83	+
	84	+	func TestResolve(t *testing.T) {
	85	+	c, _ := LoadConfig(env(nil))
	86	+	s := c.Resolve(JobSpec{JobID: "j", CPUs: 6})
	87	+	if s.CPUs != 6 \|\| s.MemoryMiB != 4096 \|\| s.DiskMiB != 64*1024 \|\| s.JobID != "j" {
	88	+	t.Fatalf("%+v", s)
	89	+	}
	90	+	}
	91	+
	92	+	func TestSlots(t *testing.T) {
	93	+	s := NewSlots(2)
	94	+	a, _ := s.Acquire()
	95	+	b, _ := s.Acquire()
	96	+	if a != 0 \|\| b != 1 \|\| s.InUse() != 2 {
	97	+	t.Fatal(a, b)
	98	+	}
	99	+	if _, err := s.Acquire(); !errors.Is(err, ErrCapacity) {
	100	+	t.Fatal(err)
	101	+	}
	102	+	s.Release(0)
	103	+	s.Release(0)
	104	+	s.Release(99)
	105	+	if c, _ := s.Acquire(); c != 0 {
	106	+	t.Fatalf("lowest free slot first, got %d", c)
	107	+	}
	108	+	}
	109	+
	110	+	func TestIDs(t *testing.T) {
	111	+	id, err := NewID()
	112	+	if err != nil \|\| !ValidID(id) {
	113	+	t.Fatal(id, err)
	114	+	}
	115	+	for _, bad := range []string{"", "er", "er12345", "xx12345678", "er1234567g", "er../../etc", "er123456789"} {
	116	+	if ValidID(bad) {
	117	+	t.Errorf("%q valid", bad)
	118	+	}
	119	+	}
	120	+	}
	121	+
	122	+	// fakeDriver is a Driver that records its construction.
	123	+	type fakeDriver struct{ mode Mode }
	124	+
	125	+	func (f fakeDriver) Mode() Mode { return f.mode }
	126	+	func (f fakeDriver) Isolation() Isolation { return Isolation(f.mode) }
	127	+	func (fakeDriver) New() (Sandbox, error) { return nil, nil }
	128	+	func (fakeDriver) Reap(context.Context) error { return nil }
	129	+	func (fakeDriver) Close() error { return nil }
	130	+
	131	+	func facts(built *[]Mode) Factories {
	132	+	return Factories{
	133	+	VM: func(context.Context, Config) (Driver, error) { built = append(built, VM); return fakeDriver{VM}, nil },
	134	+	Container: func(context.Context, Config) (Driver, error) {
	135	+	built = append(built, Container)
	136	+	return fakeDriver{Container}, nil
	137	+	},
	138	+	}
	139	+	}
	140	+
	141	+	// goodProber reports a host where every vm prerequisite exists.
	142	+	func goodProber() Prober {
	143	+	return Prober{
	144	+	OpenKVM: func(string) error { return nil },
	145	+	LookPath: func(n string) (string, error) { return "/usr/bin/" + n, nil },
	146	+	Stat: func(string) error { return nil },
	147	+	}
	148	+	}
	149	+
	150	+	func logCtx(buf *bytes.Buffer) context.Context {
	151	+	return zerowrap.WithCtx(context.Background(), zerowrap.New(zerowrap.Config{Level: "debug", Format: "json", Output: buf}))
	152	+	}
	153	+
	154	+	func TestSelectAutoPicksVMWhenEverythingIsThere(t *testing.T) {
	155	+	var built []Mode
	156	+	var log bytes.Buffer
	157	+	cfg, _ := LoadConfig(env(nil))
	158	+	d, err := Select(logCtx(&log), cfg, facts(&built), goodProber())
	159	+	if err != nil \|\| d.Mode() != VM \|\| len(built) != 1 {
	160	+	t.Fatalf("%v %v %v", d, err, built)
	161	+	}
	162	+	if !strings.Contains(log.String(), `"mode":"vm"`) \|\| !strings.Contains(log.String(), "reason") {
	163	+	t.Fatalf("mode and reason must be logged: %s", log.String())
	164	+	}
	165	+	}
	166	+
	167	+	// AC4: auto picks container when /dev/kvm is hidden, and says why.
	168	+	func TestSelectAutoFallsBackWithoutKVM(t *testing.T) {
	169	+	var built []Mode
	170	+	var log bytes.Buffer
	171	+	cfg, _ := LoadConfig(env(nil))
	172	+	cfg.KVMPath = filepath.Join(t.TempDir(), "kvm") // hidden: does not exist
	173	+	p := goodProber()
	174	+	p.OpenKVM = Prober{}.withDefaults().OpenKVM // the real open, on the hidden path
	175	+	d, err := Select(logCtx(&log), cfg, facts(&built), p)
	176	+	if err != nil \|\| d.Mode() != Container {
	177	+	t.Fatalf("%v %v", d, err)
	178	+	}
	179	+	out := log.String()
	180	+	if !strings.Contains(out, `"mode":"container"`) \|\| !strings.Contains(out, "not usable read-write") {
	181	+	t.Fatalf("fallback reason not logged: %s", out)
	182	+	}
	183	+	}
	184	+
	185	+	func TestSelectAutoFallbackReasons(t *testing.T) {
	186	+	cfg, _ := LoadConfig(env(nil))
	187	+	cases := map[string]Prober{
	188	+	"firecracker not found": {OpenKVM: func(string) error { return nil }, LookPath: func(n string) (string, error) {
	189	+	if n == "firecracker" {
	190	+	return "", os.ErrNotExist
	191	+	}
	192	+	return "/x", nil
	193	+	}, Stat: func(string) error { return nil }},
	194	+	"jailer not found": {OpenKVM: func(string) error { return nil }, LookPath: func(n string) (string, error) {
	195	+	if n == "jailer" {
	196	+	return "", os.ErrNotExist
	197	+	}
	198	+	return "/x", nil
	199	+	}, Stat: func(string) error { return nil }},
	200	+	"guest kernel": {OpenKVM: func(string) error { return nil }, LookPath: goodProber().LookPath, Stat: func(p string) error {
	201	+	if strings.HasSuffix(p, "vmlinux") {
	202	+	return os.ErrNotExist
	203	+	}
	204	+	return nil
	205	+	}},
	206	+	"base image": {OpenKVM: func(string) error { return nil }, LookPath: goodProber().LookPath, Stat: func(p string) error {
	207	+	if strings.HasSuffix(p, "base.ext4") {
	208	+	return os.ErrNotExist
	209	+	}
	210	+	return nil
	211	+	}},
	212	+	}
	213	+	for want, p := range cases {
	214	+	var built []Mode
	215	+	d, err := Select(context.Background(), cfg, facts(&built), p)
	216	+	if err != nil \|\| d.Mode() != Container {
	217	+	t.Errorf("%s: %v %v", want, d, err)
	218	+	}
	219	+	if why := VMUnavailable(cfg, p); len(why) != 1 \|\| !strings.Contains(why[0], want) {
	220	+	t.Errorf("%s: reasons %v", want, why)
	221	+	}
	222	+	}
	223	+	}
	224	+
	225	+	func TestSelectExplicitModes(t *testing.T) {
	226	+	cfg, _ := LoadConfig(env(map[string]string{EnvIsolation: "vm"}))
	227	+	var built []Mode
	228	+	bad := Prober{OpenKVM: func(string) error { return os.ErrPermission }, LookPath: goodProber().LookPath, Stat: goodProber().Stat}
	229	+	if _, err := Select(context.Background(), cfg, facts(&built), bad); err == nil \|\| !strings.Contains(err.Error(), "kvm") \|\| len(built) != 0 {
	230	+	t.Fatalf("vm must fail loudly without KVM and build nothing: %v %v", err, built)
	231	+	}
	232	+	cfg.Isolation = Container
	233	+	d, err := Select(context.Background(), cfg, facts(&built), bad)
	234	+	if err != nil \|\| d.Mode() != Container {
	235	+	t.Fatalf("container never probes KVM: %v %v", d, err)
	236	+	}
	237	+	cfg.Isolation = Mode("x")
	238	+	if _, err := Select(context.Background(), cfg, facts(&built), bad); err == nil {
	239	+	t.Fatal("unknown mode accepted")
	240	+	}
	241	+	}
	242	+
	243	+	func TestSelectDoesNotMaskBrokenVM(t *testing.T) {
	244	+	cfg, _ := LoadConfig(env(nil))
	245	+	boom := errors.New("kernel does not boot")
	246	+	f := Factories{
	247	+	VM: func(context.Context, Config) (Driver, error) { return nil, boom },
	248	+	Container: func(context.Context, Config) (Driver, error) { t.Fatal("must not fall back silently"); return nil, nil },
	249	+	}
	250	+	if _, err := Select(context.Background(), cfg, f, goodProber()); !errors.Is(err, boom) {
	251	+	t.Fatalf("got %v", err)
	252	+	}
	253	+	if _, err := Select(context.Background(), cfg, Factories{}, goodProber()); err == nil {
	254	+	t.Fatal("missing factory accepted")
	255	+	}
	256	+	}
	257	+
	258	+	func TestCheckStatic(t *testing.T) {
	259	+	dir := t.TempDir()
	260	+	if err := CheckStatic(filepath.Join(dir, "missing")); err == nil {
	261	+	t.Fatal("missing file accepted")
	262	+	}
	263	+	txt := filepath.Join(dir, "script")
	264	+	_ = os.WriteFile(txt, []byte("#!/bin/sh\n"), 0o755)
	265	+	if err := CheckStatic(txt); err == nil \|\| !strings.Contains(err.Error(), "CGO_ENABLED=0") {
	266	+	t.Fatalf("script accepted: %v", err)
	267	+	}
	268	+	if err := CheckStatic("/bin/sh"); err == nil {
	269	+	t.Skip("/bin/sh is static here")
	270	+	}
	271	+	}

A internal/sandbox/sandboxtest/sandboxtest.go +92 -0

@@ -0,0 +1,92 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package sandboxtest has helpers shared by the tests of the sandbox drivers.
	4	+	package sandboxtest
	5	+
	6	+	import (
	7	+	"context"
	8	+	"encoding/binary"
	9	+	"net/netip"
	10	+	"os"
	11	+	"sync"
	12	+	"testing"
	13	+
	14	+	"bnema.dev/ermite-runner/internal/netpolicy"
	15	+	)
	16	+
	17	+	// WriteStaticELF writes a minimal statically linked ELF64 file (one PT_LOAD, no
	18	+	// PT_INTERP): enough for sandbox.CheckStatic, never executed.
	19	+	func WriteStaticELF(t testing.TB, path string) {
	20	+	t.Helper()
	21	+	var b [64 + 56]byte
	22	+	copy(b[:], "\x7fELF")
	23	+	b[4], b[5], b[6] = 2, 1, 1 // 64-bit, little endian, version
	24	+	binary.LittleEndian.PutUint16(b[16:], 2) // ET_EXEC
	25	+	binary.LittleEndian.PutUint16(b[18:], 62) // x86-64
	26	+	binary.LittleEndian.PutUint32(b[20:], 1)
	27	+	binary.LittleEndian.PutUint64(b[32:], 64) // e_phoff
	28	+	binary.LittleEndian.PutUint16(b[52:], 64) // e_ehsize
	29	+	binary.LittleEndian.PutUint16(b[54:], 56) // e_phentsize
	30	+	binary.LittleEndian.PutUint16(b[56:], 1) // e_phnum
	31	+	binary.LittleEndian.PutUint32(b[64:], 1) // PT_LOAD
	32	+	if err := os.WriteFile(path, b[:], 0o755); err != nil {
	33	+	t.Fatal(err)
	34	+	}
	35	+	}
	36	+
	37	+	// Plumbing is a netsetup.Plumbing that records what is attached.
	38	+	type Plumbing struct {
	39	+	mu sync.Mutex
	40	+	Attached map[string]netpolicy.Guest
	41	+	Events []string
	42	+	Resets int
	43	+	FailNext error
	44	+	}
	45	+
	46	+	// NewPlumbing returns an empty Plumbing.
	47	+	func NewPlumbing() *Plumbing { return &Plumbing{Attached: map[string]netpolicy.Guest{}} }
	48	+
	49	+	// EnableForwarding implements netsetup.Plumbing.
	50	+	func (p *Plumbing) EnableForwarding() error { return nil }
	51	+
	52	+	// Attach implements netsetup.Plumbing.
	53	+	func (p *Plumbing) Attach(_ context.Context, g netpolicy.Guest) error {
	54	+	p.mu.Lock()
	55	+	defer p.mu.Unlock()
	56	+	if p.FailNext != nil {
	57	+	err := p.FailNext
	58	+	p.FailNext = nil
	59	+	return err
	60	+	}
	61	+	p.Attached[g.Iface] = g
	62	+	p.Events = append(p.Events, "attach "+g.Iface)
	63	+	return nil
	64	+	}
	65	+
	66	+	// Detach implements netsetup.Plumbing.
	67	+	func (p *Plumbing) Detach(_ context.Context, iface string, _ netip.Addr) error {
	68	+	p.mu.Lock()
	69	+	defer p.mu.Unlock()
	70	+	delete(p.Attached, iface)
	71	+	p.Events = append(p.Events, "detach "+iface)
	72	+	return nil
	73	+	}
	74	+
	75	+	// Reset implements netsetup.Plumbing.
	76	+	func (p *Plumbing) Reset(context.Context) error {
	77	+	p.mu.Lock()
	78	+	defer p.mu.Unlock()
	79	+	p.Resets++
	80	+	p.Attached = map[string]netpolicy.Guest{}
	81	+	return nil
	82	+	}
	83	+
	84	+	// Close implements netsetup.Plumbing.
	85	+	func (p *Plumbing) Close() {}
	86	+
	87	+	// Live returns the number of attached guests.
	88	+	func (p *Plumbing) Live() int {
	89	+	p.mu.Lock()
	90	+	defer p.mu.Unlock()
	91	+	return len(p.Attached)
	92	+	}

A internal/sandbox/select.go +156 -0

@@ -0,0 +1,156 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"context"
	7	+	"debug/elf"
	8	+	"errors"
	9	+	"fmt"
	10	+	"os"
	11	+	"os/exec"
	12	+	"strings"
	13	+
	14	+	"github.com/bnema/zerowrap"
	15	+	)
	16	+
	17	+	// Factory builds the driver of one mode. It is injected so that this package
	18	+	// does not import the drivers (which import it).
	19	+	type Factory func(ctx context.Context, cfg Config) (Driver, error)
	20	+
	21	+	// Factories are the available drivers.
	22	+	type Factories struct {
	23	+	VM Factory
	24	+	Container Factory
	25	+	}
	26	+
	27	+	// Prober checks the host. The zero value probes the real host; tests replace
	28	+	// fields.
	29	+	type Prober struct {
	30	+	// OpenKVM opens the KVM device read-write and closes it.
	31	+	OpenKVM func(path string) error
	32	+	// LookPath finds a binary in PATH.
	33	+	LookPath func(name string) (string, error)
	34	+	// Stat reports whether a path exists.
	35	+	Stat func(path string) error
	36	+	}
	37	+
	38	+	func (p Prober) withDefaults() Prober {
	39	+	if p.OpenKVM == nil {
	40	+	p.OpenKVM = func(path string) error {
	41	+	f, err := os.OpenFile(path, os.O_RDWR, 0)
	42	+	if err != nil {
	43	+	return err
	44	+	}
	45	+	return f.Close()
	46	+	}
	47	+	}
	48	+	if p.LookPath == nil {
	49	+	p.LookPath = exec.LookPath
	50	+	}
	51	+	if p.Stat == nil {
	52	+	p.Stat = func(path string) error { _, err := os.Stat(path); return err }
	53	+	}
	54	+	return p
	55	+	}
	56	+
	57	+	// VMUnavailable lists why the vm mode cannot run on this host: an empty result
	58	+	// means every prerequisite (readable-writable KVM, Firecracker, jailer unless
	59	+	// disabled, guest kernel, base image) is there.
	60	+	func VMUnavailable(cfg Config, p Prober) []string {
	61	+	p = p.withDefaults()
	62	+	var why []string
	63	+	if err := p.OpenKVM(cfg.KVMPath); err != nil {
	64	+	why = append(why, fmt.Sprintf("%s not usable read-write: %v", cfg.KVMPath, err))
	65	+	}
	66	+	if cfg.Firecracker != "" {
	67	+	if err := p.Stat(cfg.Firecracker); err != nil {
	68	+	why = append(why, fmt.Sprintf("firecracker %s: %v", cfg.Firecracker, err))
	69	+	}
	70	+	} else if _, err := p.LookPath("firecracker"); err != nil {
	71	+	why = append(why, "firecracker not found in PATH")
	72	+	}
	73	+	if !cfg.JailerOff {
	74	+	if cfg.Jailer != "" {
	75	+	if err := p.Stat(cfg.Jailer); err != nil {
	76	+	why = append(why, fmt.Sprintf("jailer %s: %v", cfg.Jailer, err))
	77	+	}
	78	+	} else if _, err := p.LookPath("jailer"); err != nil {
	79	+	why = append(why, "jailer not found in PATH")
	80	+	}
	81	+	}
	82	+	if err := p.Stat(cfg.KernelPath()); err != nil {
	83	+	why = append(why, fmt.Sprintf("guest kernel %s: %v", cfg.KernelPath(), err))
	84	+	}
	85	+	if err := p.Stat(cfg.BaseImagePath()); err != nil {
	86	+	why = append(why, fmt.Sprintf("base image %s: %v", cfg.BaseImagePath(), err))
	87	+	}
	88	+	return why
	89	+	}
	90	+
	91	+	// Select picks the isolation mode and builds its driver. For auto it chooses vm
	92	+	// when VMUnavailable is empty, container otherwise. For vm it fails when a
	93	+	// prerequisite is missing; for container it never probes the VM side. The
	94	+	// chosen mode and the reasons are logged with the logger of ctx. A driver
	95	+	// factory error is returned as is (auto does not mask a broken vm setup by
	96	+	// silently running jobs with weaker isolation).
	97	+	func Select(ctx context.Context, cfg Config, f Factories, p Prober) (Driver, error) {
	98	+	log := zerowrap.FromCtx(ctx)
	99	+	mode := cfg.Isolation
	100	+	var reasons []string
	101	+	switch mode {
	102	+	case VM, Auto:
	103	+	reasons = VMUnavailable(cfg, p)
	104	+	if mode == VM && len(reasons) > 0 {
	105	+	return nil, fmt.Errorf("isolation vm is not available: %s", strings.Join(reasons, "; "))
	106	+	}
	107	+	if len(reasons) == 0 {
	108	+	mode = VM
	109	+	} else {
	110	+	mode = Container
	111	+	}
	112	+	case Container:
	113	+	default:
	114	+	return nil, fmt.Errorf("unknown isolation %q", mode)
	115	+	}
	116	+
	117	+	build := f.VM
	118	+	if mode == Container {
	119	+	build = f.Container
	120	+	}
	121	+	if build == nil {
	122	+	return nil, fmt.Errorf("isolation %s: no driver compiled in", mode)
	123	+	}
	124	+	ev := log.Info().Str("configured", string(cfg.Isolation)).Str("mode", string(mode))
	125	+	switch {
	126	+	case cfg.Isolation == Auto && mode == VM:
	127	+	ev = ev.Str("reason", "KVM, firecracker, kernel and base image are available")
	128	+	case cfg.Isolation == Auto:
	129	+	ev = ev.Strs("reasons", reasons)
	130	+	default:
	131	+	ev = ev.Str("reason", "set by "+EnvIsolation)
	132	+	}
	133	+	ev.Msg("isolation mode chosen")
	134	+	d, err := build(ctx, cfg)
	135	+	if err != nil {
	136	+	return nil, fmt.Errorf("isolation %s: %w", mode, err)
	137	+	}
	138	+	return d, nil
	139	+	}
	140	+
	141	+	// CheckStatic fails unless the executable at path is a statically linked ELF
	142	+	// binary: it runs as PID 1 on a root disk that holds nothing else, and in an
	143	+	// image of another distribution.
	144	+	func CheckStatic(path string) error {
	145	+	f, err := elf.Open(path)
	146	+	if err != nil {
	147	+	return fmt.Errorf("%s is not an ELF executable (%w); build the runner with CGO_ENABLED=0", path, err)
	148	+	}
	149	+	defer func() { _ = f.Close() }()
	150	+	for _, p := range f.Progs {
	151	+	if p.Type == elf.PT_INTERP {
	152	+	return errors.New(path + " is dynamically linked; build the runner with CGO_ENABLED=0")
	153	+	}
	154	+	}
	155	+	return nil
	156	+	}

A internal/sandbox/space.go +43 -0

@@ -0,0 +1,43 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package sandbox
	4	+
	5	+	import (
	6	+	"context"
	7	+
	8	+	"github.com/bnema/zerowrap"
	9	+	"golang.org/x/sys/unix"
	10	+	)
	11	+
	12	+	// FreeMiB returns the space available to unprivileged users on the filesystem
	13	+	// of path, in MiB.
	14	+	func FreeMiB(path string) (int64, error) {
	15	+	var st unix.Statfs_t
	16	+	if err := unix.Statfs(path, &st); err != nil {
	17	+	return 0, err
	18	+	}
	19	+	return int64(st.Bavail) * st.Bsize >> 20, nil
	20	+	}
	21	+
	22	+	// WarnIfLowSpace logs a warning when the filesystem of dir has less free space
	23	+	// than capacity sandboxes of diskMiB would need if they all filled their sparse
	24	+	// scratch disk. It is only a warning: scratch disks are sparse and jobs rarely
	25	+	// fill them, so a smaller filesystem works until it does not. free is FreeMiB in
	26	+	// production.
	27	+	func WarnIfLowSpace(ctx context.Context, dir string, capacity, diskMiB int, free func(string) (int64, error)) {
	28	+	if free == nil {
	29	+	free = FreeMiB
	30	+	}
	31	+	log := zerowrap.FromCtx(ctx)
	32	+	have, err := free(dir)
	33	+	if err != nil {
	34	+	log.Warn().Err(err).Str("dir", dir).Msg("cannot check the free space of the state directory")
	35	+	return
	36	+	}
	37	+	need := int64(capacity) * int64(diskMiB)
	38	+	if have < need {
	39	+	log.Warn().Str("dir", dir).Int64("free_mib", have).Int64("worst_case_mib", need).
	40	+	Int("capacity", capacity).Int("job_disk_mib", diskMiB).
	41	+	Msg("the state directory has less free space than capacity x job disk: jobs that fill their scratch disk will fail with 'no space left on device'")
	42	+	}
	43	+	}

A internal/sandbox/vm/api.go +208 -0

@@ -0,0 +1,208 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"context"
	8	+	"encoding/json"
	9	+	"fmt"
	10	+	"io"
	11	+	"net"
	12	+	"net/http"
	13	+	"time"
	14	+	)
	15	+
	16	+	// Firecracker is controlled through its HTTP API on a unix socket, with
	17	+	// net/http, rather than through firecracker-go-sdk: the SDK's last release
	18	+	// (v1.0.0, 2022) predates Firecracker 1.17 and drags in CNI, go-openapi and
	19	+	// netlink for what is a handful of PUT requests. The structures below are the
	20	+	// subset of the API this driver uses; Firecracker rejects unknown or invalid
	21	+	// fields with a fault message, which surfaces in the returned error.
	22	+
	23	+	type machineConfig struct {
	24	+	VcpuCount int `json:"vcpu_count"`
	25	+	MemSizeMiB int `json:"mem_size_mib"`
	26	+	SMT bool `json:"smt"`
	27	+	}
	28	+
	29	+	type bootSource struct {
	30	+	KernelImagePath string `json:"kernel_image_path"`
	31	+	BootArgs string `json:"boot_args"`
	32	+	}
	33	+
	34	+	type drive struct {
	35	+	DriveID string `json:"drive_id"`
	36	+	PathOnHost string `json:"path_on_host"`
	37	+	IsRootDevice bool `json:"is_root_device"`
	38	+	IsReadOnly bool `json:"is_read_only"`
	39	+	RateLimiter *rateLimiter `json:"rate_limiter,omitempty"`
	40	+	}
	41	+
	42	+	type networkInterface struct {
	43	+	IfaceID string `json:"iface_id"`
	44	+	HostDevName string `json:"host_dev_name"`
	45	+	GuestMAC string `json:"guest_mac"`
	46	+	RxRateLimiter *rateLimiter `json:"rx_rate_limiter,omitempty"`
	47	+	TxRateLimiter *rateLimiter `json:"tx_rate_limiter,omitempty"`
	48	+	}
	49	+
	50	+	// tokenBucket is Firecracker's rate limiter bucket: Size tokens (bytes or
	51	+	// operations) are refilled every RefillTime milliseconds.
	52	+	type tokenBucket struct {
	53	+	Size int64 `json:"size"`
	54	+	OneTimeBurst int64 `json:"one_time_burst,omitempty"`
	55	+	RefillTime int64 `json:"refill_time"`
	56	+	}
	57	+
	58	+	type rateLimiter struct {
	59	+	Bandwidth *tokenBucket `json:"bandwidth,omitempty"`
	60	+	Ops *tokenBucket `json:"ops,omitempty"`
	61	+	}
	62	+
	63	+	// diskLimiter limits a disk to bandwidthMiB MiB/s and iops operations/s (zero
	64	+	// disables a limit; both zero means no limiter). A one-second burst is allowed
	65	+	// at the start so that short bursts (a git checkout) are not slowed down.
	66	+	func diskLimiter(bandwidthMiB, iops int) *rateLimiter {
	67	+	var rl rateLimiter
	68	+	if bandwidthMiB > 0 {
	69	+	b := int64(bandwidthMiB) << 20
	70	+	rl.Bandwidth = &tokenBucket{Size: b, OneTimeBurst: b, RefillTime: 1000}
	71	+	}
	72	+	if iops > 0 {
	73	+	rl.Ops = &tokenBucket{Size: int64(iops), OneTimeBurst: int64(iops), RefillTime: 1000}
	74	+	}
	75	+	if rl.Bandwidth == nil && rl.Ops == nil {
	76	+	return nil
	77	+	}
	78	+	return &rl
	79	+	}
	80	+
	81	+	// netLimiter limits one direction of a network interface to mbit Mbit/s.
	82	+	func netLimiter(mbit int) *rateLimiter {
	83	+	if mbit <= 0 {
	84	+	return nil
	85	+	}
	86	+	b := int64(mbit) * 1_000_000 / 8
	87	+	return &rateLimiter{Bandwidth: &tokenBucket{Size: b, OneTimeBurst: b, RefillTime: 1000}}
	88	+	}
	89	+
	90	+	type vsockDevice struct {
	91	+	GuestCID uint32 `json:"guest_cid"`
	92	+	UDSPath string `json:"uds_path"`
	93	+	}
	94	+
	95	+	type action struct {
	96	+	ActionType string `json:"action_type"`
	97	+	}
	98	+
	99	+	// apiClient talks to one Firecracker instance.
	100	+	type apiClient struct {
	101	+	http *http.Client
	102	+	}
	103	+
	104	+	func newAPIClient(socket string) *apiClient {
	105	+	return &apiClient{http: &http.Client{
	106	+	Timeout: 10 * time.Second,
	107	+	Transport: &http.Transport{
	108	+	DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
	109	+	var d net.Dialer
	110	+	return d.DialContext(ctx, "unix", socket)
	111	+	},
	112	+	DisableKeepAlives: true,
	113	+	},
	114	+	}}
	115	+	}
	116	+
	117	+	// put sends a PUT with a JSON body and expects 204 (or 200).
	118	+	func (c *apiClient) put(ctx context.Context, path string, body any) error {
	119	+	b, err := json.Marshal(body)
	120	+	if err != nil {
	121	+	return err
	122	+	}
	123	+	req, err := http.NewRequestWithContext(ctx, http.MethodPut, "http://firecracker"+path, bytes.NewReader(b))
	124	+	if err != nil {
	125	+	return err
	126	+	}
	127	+	req.Header.Set("Content-Type", "application/json")
	128	+	resp, err := c.http.Do(req)
	129	+	if err != nil {
	130	+	return fmt.Errorf("firecracker PUT %s: %w", path, err)
	131	+	}
	132	+	defer func() { _ = resp.Body.Close() }()
	133	+	if resp.StatusCode/100 != 2 {
	134	+	msg, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
	135	+	return fmt.Errorf("firecracker PUT %s: %s: %s", path, resp.Status, bytes.TrimSpace(msg))
	136	+	}
	137	+	return nil
	138	+	}
	139	+
	140	+	// ready reports whether the API answers GET /.
	141	+	func (c *apiClient) ready(ctx context.Context) error {
	142	+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://firecracker/", nil)
	143	+	if err != nil {
	144	+	return err
	145	+	}
	146	+	resp, err := c.http.Do(req)
	147	+	if err != nil {
	148	+	return err
	149	+	}
	150	+	defer func() { _ = resp.Body.Close() }()
	151	+	_, _ = io.Copy(io.Discard, resp.Body)
	152	+	if resp.StatusCode/100 != 2 {
	153	+	return fmt.Errorf("firecracker GET /: %s", resp.Status)
	154	+	}
	155	+	return nil
	156	+	}
	157	+
	158	+	// vmConfig is everything configured before boot.
	159	+	type vmConfig struct {
	160	+	Machine machineConfig
	161	+	Boot bootSource
	162	+	Drives []drive
	163	+	Net networkInterface
	164	+	Vsock vsockDevice
	165	+	}
	166	+
	167	+	// configure applies the pre-boot configuration and boots the VM. The order is
	168	+	// fixed (and covered by a unit test) because Firecracker rejects some changes
	169	+	// after boot and drive order decides the guest device names vda, vdb, vdc.
	170	+	func (c *apiClient) configure(ctx context.Context, cfg vmConfig) error {
	171	+	steps := []struct {
	172	+	path string
	173	+	body any
	174	+	}{
	175	+	{"/machine-config", cfg.Machine},
	176	+	{"/boot-source", cfg.Boot},
	177	+	}
	178	+	for _, d := range cfg.Drives {
	179	+	steps = append(steps, struct {
	180	+	path string
	181	+	body any
	182	+	}{"/drives/" + d.DriveID, d})
	183	+	}
	184	+	steps = append(steps,
	185	+	struct {
	186	+	path string
	187	+	body any
	188	+	}{"/network-interfaces/" + cfg.Net.IfaceID, cfg.Net},
	189	+	struct {
	190	+	path string
	191	+	body any
	192	+	}{"/vsock", cfg.Vsock},
	193	+	struct {
	194	+	path string
	195	+	body any
	196	+	}{"/entropy", struct{}{}},
	197	+	struct {
	198	+	path string
	199	+	body any
	200	+	}{"/actions", action{ActionType: "InstanceStart"}},
	201	+	)
	202	+	for _, s := range steps {
	203	+	if err := c.put(ctx, s.path, s.body); err != nil {
	204	+	return err
	205	+	}
	206	+	}
	207	+	return nil
	208	+	}

A internal/sandbox/vm/bootargs.go +42 -0

@@ -0,0 +1,42 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"fmt"
	7	+	"net/netip"
	8	+	"strconv"
	9	+	"strings"
	10	+
	11	+	"bnema.dev/ermite-runner/internal/sandbox"
	12	+	)
	13	+
	14	+	// bootArgs builds the guest kernel command line.
	15	+	//
	16	+	// - console=ttyS0 reboot=k panic=1: serial console; a reboot (what init does
	17	+	// at the end) or a panic ends the VMM instead of hanging;
	18	+	// - ipv6.disable=1: the guest has no IPv6 (the host policy drops it too);
	19	+	// - root=/dev/vda ro: the exec disk; init=/ermite-runner -- init: the runner
	20	+	// executable is PID 1 and runs its init subcommand;
	21	+	// - ip=…: static address, gateway and DNS from the kernel, no DHCP client;
	22	+	// - ermite.*: read by init from /proc/cmdline (connect target, gateway, time).
	23	+	func bootArgs(guest, gateway netip.Addr, unixSeconds int64) string {
	24	+	args := []string{
	25	+	"console=ttyS0", "reboot=k", "panic=1", "pci=off", "acpi=off", "8250.nr_uarts=1",
	26	+	"ipv6.disable=1", "random.trust_cpu=on", "i8042.noaux", "i8042.nomux", "i8042.nopnp", "i8042.dumbkbd",
	27	+	"root=/dev/vda", "ro", "rootfstype=ext4",
	28	+	"init=/" + sandbox.GuestExecName,
	29	+	fmt.Sprintf("ip=%s::%s:255.255.255.252:ermite-job:eth0:off:%s", guest, gateway, gateway),
	30	+	sandbox.CmdlineSandbox + "=" + string(sandbox.VM),
	31	+	sandbox.CmdlineConnect + "=vsock:" + strconv.Itoa(sandbox.VsockPort),
	32	+	sandbox.CmdlineGateway + "=" + gateway.String(),
	33	+	"ermite.time=" + strconv.FormatInt(unixSeconds, 10),
	34	+	"--", "init",
	35	+	}
	36	+	return strings.Join(args, " ")
	37	+	}
	38	+
	39	+	// macFor derives a locally administered MAC from the slot.
	40	+	func macFor(slot int) string {
	41	+	return fmt.Sprintf("06:00:ac:10:%02x:%02x", (slot>>8)&0xff, slot&0xff)
	42	+	}

A internal/sandbox/vm/driver.go +663 -0

@@ -0,0 +1,663 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	// Package vm is the micro-VM sandbox: one Firecracker VM per job.
	4	+	//
	5	+	// Per VM, on the host: a jailer chroot (or, without jailer, a plain directory)
	6	+	// holding the per-job sparse scratch disk and the vsock socket; a TAP device
	7	+	// with a /30; rules in the dedicated nftables table; the DNS forwarder bound on
	8	+	// the TAP's host address. In the guest: three virtio disks (read-only runner
	9	+	// executable as the boot root, read-only shared base image, per-job scratch)
	10	+	// and a vsock device. Everything is traced in a state-directory record before it
	11	+	// is created, so that Reap can remove it after a crash.
	12	+	package vm
	13	+
	14	+	import (
	15	+	"context"
	16	+	"errors"
	17	+	"fmt"
	18	+	"net"
	19	+	"net/netip"
	20	+	"os"
	21	+	"os/exec"
	22	+	"path/filepath"
	23	+	"strconv"
	24	+	"sync"
	25	+	"syscall"
	26	+	"time"
	27	+
	28	+	"github.com/bnema/zerowrap"
	29	+
	30	+	"bnema.dev/ermite-runner/internal/cmdrun"
	31	+	"bnema.dev/ermite-runner/internal/netpolicy"
	32	+	"bnema.dev/ermite-runner/internal/sandbox"
	33	+	"bnema.dev/ermite-runner/internal/sandbox/disk"
	34	+	"bnema.dev/ermite-runner/internal/sandbox/netsetup"
	35	+	)
	36	+
	37	+	// Options are the testable knobs of the driver.
	38	+	type Options struct {
	39	+	Run cmdrun.Runner
	40	+	// ProcDir is the /proc used to find orphan VMM processes.
	41	+	ProcDir string
	42	+	// CgroupRoot is the cgroup v2 mount, where the jailer creates one cgroup per
	43	+	// VM (default /sys/fs/cgroup).
	44	+	CgroupRoot string
	45	+	// BootTimeout bounds boot + init + executor connection (default 90 s).
	46	+	BootTimeout time.Duration
	47	+	// Net overrides the network plumbing (tests).
	48	+	Net netsetup.Plumbing
	49	+	// FirecrackerArgs are extra arguments for Firecracker (tests use them to
	50	+	// select the fake).
	51	+	FirecrackerArgs []string
	52	+	// FreeSpace overrides statfs for the free-space warning (tests).
	53	+	FreeSpace func(string) (int64, error)
	54	+	// kill overrides killAndWait (tests simulate a VMM that cannot be killed).
	55	+	kill func(pid int, timeout time.Duration) bool
	56	+	}
	57	+
	58	+	// vmmOverheadMiB is what the Firecracker process itself may use on top of the
	59	+	// guest memory (device emulation, the API server, the seccomp filter, stacks):
	60	+	// it is added to memory.max so that the guest, not the VMM, is what gets limited.
	61	+	const vmmOverheadMiB = 256
	62	+
	63	+	// Driver implements sandbox.Driver for micro-VMs.
	64	+	type Driver struct {
	65	+	cfg sandbox.Config
	66	+	opt Options
	67	+	fcPath string
	68	+	jailer string
	69	+	execDisk string
	70	+	store store
	71	+	net netsetup.Plumbing
	72	+	lock *sandbox.HostLock
	73	+
	74	+	mu sync.Mutex
	75	+	slots *sandbox.Slots
	76	+	}
	77	+
	78	+	// New builds the driver: takes the host-wide lock, resolves binaries, builds the
	79	+	// exec disk and enables IPv4 forwarding (a host-wide sysctl that stays on: see
	80	+	// the README). Run Reap before New sandboxes.
	81	+	func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error) {
	82	+	if opt.Run == nil {
	83	+	opt.Run = cmdrun.Exec{}
	84	+	}
	85	+	if opt.ProcDir == "" {
	86	+	opt.ProcDir = "/proc"
	87	+	}
	88	+	if opt.CgroupRoot == "" {
	89	+	opt.CgroupRoot = "/sys/fs/cgroup"
	90	+	}
	91	+	if opt.BootTimeout == 0 {
	92	+	opt.BootTimeout = 90 * time.Second
	93	+	}
	94	+	if opt.kill == nil {
	95	+	opt.kill = killAndWait
	96	+	}
	97	+	d := &Driver{cfg: cfg, opt: opt, slots: sandbox.NewSlots(cfg.Capacity), store: store{dir: filepath.Join(cfg.StateDir, "vms")}}
	98	+
	99	+	var err error
	100	+	if d.fcPath, err = resolveBin(cfg.Firecracker, "firecracker"); err != nil {
	101	+	return nil, err
	102	+	}
	103	+	if !cfg.JailerOff {
	104	+	if d.jailer, err = resolveBin(cfg.Jailer, "jailer"); err != nil {
	105	+	return nil, err
	106	+	}
	107	+	}
	108	+	exe, err := cfg.ExecutablePath()
	109	+	if err != nil {
	110	+	return nil, err
	111	+	}
	112	+	if err := sandbox.CheckStatic(exe); err != nil {
	113	+	return nil, err
	114	+	}
	115	+	if err := os.MkdirAll(cfg.StateDir, 0o755); err != nil {
	116	+	return nil, fmt.Errorf("state directory: %w", err)
	117	+	}
	118	+	if d.lock, err = sandbox.AcquireHostLock(cfg.LockFile); err != nil {
	119	+	return nil, err
	120	+	}
	121	+	fail := func(err error) (*Driver, error) { _ = d.Close(); return nil, err }
	122	+
	123	+	if !cfg.JailerOff {
	124	+	for _, p := range []string{cfg.KernelPath(), cfg.BaseImagePath()} {
	125	+	if err := checkWorldReadable(p); err != nil {
	126	+	return fail(err)
	127	+	}
	128	+	}
	129	+	}
	130	+	sandbox.WarnIfLowSpace(ctx, cfg.StateDir, cfg.Capacity, cfg.JobDiskMiB, opt.FreeSpace)
	131	+	if err := disk.CheckMkfs(ctx, opt.Run); err != nil {
	132	+	return fail(err)
	133	+	}
	134	+	if d.execDisk, err = ensureExecDisk(ctx, opt.Run, exe, cfg.StateDir); err != nil {
	135	+	return fail(err)
	136	+	}
	137	+	d.net = opt.Net
	138	+	if d.net == nil {
	139	+	d.net = netsetup.New(opt.Run, cfg.DNSPort, "")
	140	+	}
	141	+	if err := d.net.EnableForwarding(); err != nil {
	142	+	return fail(err)
	143	+	}
	144	+	return d, nil
	145	+	}
	146	+
	147	+	func resolveBin(configured, name string) (string, error) {
	148	+	p := configured
	149	+	if p == "" {
	150	+	var err error
	151	+	if p, err = exec.LookPath(name); err != nil {
	152	+	return "", fmt.Errorf("%s not found in PATH (set ERMITE_RUNNER_%s)", name, map[string]string{"firecracker": "FIRECRACKER", "jailer": "JAILER"}[name])
	153	+	}
	154	+	}
	155	+	abs, err := filepath.Abs(p)
	156	+	if err != nil {
	157	+	return "", err
	158	+	}
	159	+	if _, err := os.Stat(abs); err != nil {
	160	+	return "", fmt.Errorf("%s: %w", name, err)
	161	+	}
	162	+	return abs, nil
	163	+	}
	164	+
	165	+	// Mode implements sandbox.Driver.
	166	+	func (d *Driver) Mode() sandbox.Mode { return sandbox.VM }
	167	+
	168	+	// Isolation implements sandbox.Driver.
	169	+	func (d *Driver) Isolation() sandbox.Isolation { return sandbox.IsolationVM }
	170	+
	171	+	// Close releases the host lock and the DNS listeners.
	172	+	func (d *Driver) Close() error {
	173	+	if d.net != nil {
	174	+	d.net.Close()
	175	+	}
	176	+	return d.lock.Release()
	177	+	}
	178	+
	179	+	// New implements sandbox.Driver.
	180	+	func (d *Driver) New() (sandbox.Sandbox, error) {
	181	+	id, err := sandbox.NewID()
	182	+	if err != nil {
	183	+	return nil, err
	184	+	}
	185	+	d.mu.Lock()
	186	+	defer d.mu.Unlock()
	187	+	slot, err := d.slots.Acquire()
	188	+	if err != nil {
	189	+	return nil, err
	190	+	}
	191	+	return &vmSandbox{d: d, slot: slot, id: id}, nil
	192	+	}
	193	+
	194	+	func (d *Driver) release(slot int) {
	195	+	d.mu.Lock()
	196	+	d.slots.Release(slot)
	197	+	d.mu.Unlock()
	198	+	}
	199	+
	200	+	func (d *Driver) markLeaked(slot int) {
	201	+	d.mu.Lock()
	202	+	d.slots.MarkLeaked(slot)
	203	+	d.mu.Unlock()
	204	+	}
	205	+
	206	+	// vmSandbox is one VM, single use.
	207	+	type vmSandbox struct {
	208	+	d *Driver
	209	+	slot int
	210	+	id string
	211	+	gate sandbox.BootGate
	212	+
	213	+	mu sync.Mutex
	214	+	started bool
	215	+	stopped bool
	216	+	// leakErr is the error of a cleanup that left something behind; later Stop
	217	+	// calls return it again.
	218	+	leakErr error
	219	+
	220	+	rec record
	221	+	lay layout
	222	+	cmd *exec.Cmd
	223	+	exited chan struct{}
	224	+	console *ring
	225	+	listener net.Listener
	226	+	conn net.Conn
	227	+	tap string
	228	+	gateway netip.Addr
	229	+	attached bool
	230	+	}
	231	+
	232	+	// ID implements sandbox.Sandbox.
	233	+	func (s *vmSandbox) ID() string { return s.id }
	234	+
	235	+	func (s *vmSandbox) uid() int { return s.d.cfg.UIDBase + s.slot }
	236	+
	237	+	// Start implements sandbox.Sandbox.
	238	+	func (s *vmSandbox) Start(ctx context.Context, spec sandbox.JobSpec) (sandbox.Conn, error) {
	239	+	// A Stop from another goroutine aborts the boot through the gate.
	240	+	bootCtx, err := s.gate.Begin(ctx)
	241	+	if err != nil {
	242	+	return nil, fmt.Errorf("vm sandbox: %w", err)
	243	+	}
	244	+	defer s.gate.End()
	245	+
	246	+	s.mu.Lock()
	247	+	defer s.mu.Unlock()
	248	+	if s.started \|\| s.stopped {
	249	+	return nil, errors.New("vm sandbox: Start called twice or after Stop")
	250	+	}
	251	+	s.started = true
	252	+	log := zerowrap.FromCtx(ctx)
	253	+	d := s.d
	254	+	spec = d.cfg.Resolve(spec)
	255	+
	256	+	jailed := !d.cfg.JailerOff
	257	+	runUID := s.uid()
	258	+	if !jailed {
	259	+	runUID = os.Getuid()
	260	+	}
	261	+	s.lay = newLayout(d.cfg.StateDir, filepath.Base(d.fcPath), s.id, jailed, runUID)
	262	+	s.rec = record{ID: s.id, Slot: s.slot, Job: spec.JobID, Dir: s.lay.dir}
	263	+	// The record exists before the first resource, so any crash from here on is
	264	+	// reapable.
	265	+	if err := d.store.save(s.rec); err != nil {
	266	+	s.releaseLocked()
	267	+	return nil, fmt.Errorf("write state record: %w", err)
	268	+	}
	269	+
	270	+	conn, err := s.boot(bootCtx, spec)
	271	+	if err != nil {
	272	+	console := ""
	273	+	if s.console != nil {
	274	+	console = s.console.String()
	275	+	}
	276	+	cerr := s.cleanupLocked(context.WithoutCancel(ctx))
	277	+	if console != "" {
	278	+	err = fmt.Errorf("%w\nVMM console tail:\n%s", err, console)
	279	+	}
	280	+	return nil, errors.Join(err, cerr)
	281	+	}
	282	+	log.Info().Str("vm", s.id).Int("slot", s.slot).Str("job", spec.JobID).Msg("micro-VM running")
	283	+	return conn, nil
	284	+	}
	285	+
	286	+	// boot runs the stages in order; each stage records what it created so that
	287	+	// cleanupLocked can undo exactly that.
	288	+	func (s *vmSandbox) boot(ctx context.Context, spec sandbox.JobSpec) (sandbox.Conn, error) {
	289	+	netw, err := netpolicy.Subnet(s.d.cfg.Network, s.slot)
	290	+	if err != nil {
	291	+	return nil, err
	292	+	}
	293	+	if err := s.bootNetwork(ctx, netw); err != nil {
	294	+	return nil, err
	295	+	}
	296	+	api, err := s.bootVMM(ctx, spec)
	297	+	if err != nil {
	298	+	return nil, err
	299	+	}
	300	+	files, err := s.bootFiles(ctx, spec)
	301	+	if err != nil {
	302	+	return nil, err
	303	+	}
	304	+	accepted, err := s.bootVsock()
	305	+	if err != nil {
	306	+	return nil, err
	307	+	}
	308	+	if err := api.configure(ctx, s.vmConfig(spec, netw, files)); err != nil {
	309	+	return nil, err
	310	+	}
	311	+	return s.waitExecutor(ctx, accepted)
	312	+	}
	313	+
	314	+	// bootNetwork creates the TAP (it must exist when Firecracker opens it) and
	315	+	// installs the firewall policy and the DNS listener.
	316	+	func (s *vmSandbox) bootNetwork(ctx context.Context, netw netpolicy.Net) error {
	317	+	d := s.d
	318	+	s.tap, s.gateway = tapName(s.slot), netw.Gateway
	319	+	s.rec.Tap, s.rec.Gate = s.tap, s.gateway.String()
	320	+	if err := d.store.save(s.rec); err != nil {
	321	+	return err
	322	+	}
	323	+	if err := createTap(ctx, d.opt.Run, s.tap, s.lay.uid, s.gateway); err != nil {
	324	+	return err
	325	+	}
	326	+	s.attached = true // from here Detach is idempotent and always safe
	327	+	return d.net.Attach(ctx, netpolicy.Guest{Iface: s.tap, Gateway: netw.Gateway, Addr: netw.Guest})
	328	+	}
	329	+
	330	+	// bootVMM starts the jailer (or Firecracker) and waits for its API.
	331	+	func (s vmSandbox) bootVMM(ctx context.Context, spec sandbox.JobSpec) (apiClient, error) {
	332	+	apiSock := s.lay.host("api.sock")
	333	+	if s.lay.jailed {
	334	+	apiSock = s.lay.host("run/firecracker.socket")
	335	+	} else if err := os.MkdirAll(s.lay.root, 0o700); err != nil {
	336	+	return nil, err
	337	+	}
	338	+	if err := s.startVMM(spec, apiSock); err != nil {
	339	+	return nil, err
	340	+	}
	341	+	api := newAPIClient(apiSock)
	342	+	if err := s.waitAPI(ctx, api, apiSock); err != nil {
	343	+	return nil, err
	344	+	}
	345	+	return api, nil
	346	+	}
	347	+
	348	+	// vmFiles are the paths Firecracker is given.
	349	+	type vmFiles struct{ execDisk, baseDisk, kernel string }
	350	+
	351	+	// bootFiles makes the shared files available and creates the scratch disk.
	352	+	func (s *vmSandbox) bootFiles(ctx context.Context, spec sandbox.JobSpec) (vmFiles, error) {
	353	+	d := s.d
	354	+	var f vmFiles
	355	+	var err error
	356	+	if f.execDisk, err = s.lay.share(d.execDisk, "exec.ext4"); err != nil {
	357	+	return f, err
	358	+	}
	359	+	if f.baseDisk, err = s.lay.share(d.cfg.BaseImagePath(), "base.ext4"); err != nil {
	360	+	return f, err
	361	+	}
	362	+	if f.kernel, err = s.lay.share(d.cfg.KernelPath(), "kernel"); err != nil {
	363	+	return f, err
	364	+	}
	365	+	owner := -1
	366	+	if s.lay.jailed {
	367	+	owner = s.lay.uid
	368	+	}
	369	+	return f, disk.CreateScratch(ctx, d.opt.Run, s.lay.host("scratch.ext4"), spec.DiskMiB, owner)
	370	+	}
	371	+
	372	+	// bootVsock listens on the host side of the vsock stream: Firecracker connects
	373	+	// to <uds_path>_<port> when the guest connects to the host CID.
	374	+	func (s *vmSandbox) bootVsock() (<-chan acceptResult, error) {
	375	+	path := s.lay.host(udsName) + fmt.Sprintf("_%d", sandbox.VsockPort)
	376	+	ln, err := net.Listen("unix", path)
	377	+	if err != nil {
	378	+	return nil, fmt.Errorf("listen vsock socket: %w", err)
	379	+	}
	380	+	s.listener = ln
	381	+	if s.lay.jailed {
	382	+	if err := os.Lchown(path, s.lay.uid, s.lay.uid); err != nil {
	383	+	return nil, err
	384	+	}
	385	+	}
	386	+	accepted := make(chan acceptResult, 1)
	387	+	go func() { c, err := ln.Accept(); accepted <- acceptResult{c, err} }()
	388	+	return accepted, nil
	389	+	}
	390	+
	391	+	const udsName = "v.sock"
	392	+
	393	+	func (s *vmSandbox) vmConfig(spec sandbox.JobSpec, netw netpolicy.Net, f vmFiles) vmConfig {
	394	+	cfg := s.d.cfg
	395	+	return vmConfig{
	396	+	Machine: machineConfig{VcpuCount: spec.CPUs, MemSizeMiB: spec.MemoryMiB},
	397	+	Boot: bootSource{KernelImagePath: f.kernel, BootArgs: bootArgs(netw.Guest, netw.Gateway, time.Now().Unix())},
	398	+	Drives: []drive{
	399	+	{DriveID: "rootfs", PathOnHost: f.execDisk, IsRootDevice: true, IsReadOnly: true},
	400	+	{DriveID: "base", PathOnHost: f.baseDisk, IsReadOnly: true},
	401	+	{DriveID: "scratch", PathOnHost: s.lay.fc("scratch.ext4"), RateLimiter: diskLimiter(cfg.DiskBandwidthMiB, cfg.DiskIOPS)},
	402	+	},
	403	+	Net: networkInterface{
	404	+	IfaceID: "eth0", HostDevName: s.tap, GuestMAC: macFor(s.slot),
	405	+	RxRateLimiter: netLimiter(cfg.NetMbit), TxRateLimiter: netLimiter(cfg.NetMbit),
	406	+	},
	407	+	Vsock: vsockDevice{GuestCID: 3, UDSPath: s.lay.fc(udsName)},
	408	+	}
	409	+	}
	410	+
	411	+	// waitExecutor waits for the guest's executor to connect. Only ctx (the caller's
	412	+	// context, or Stop through the boot gate) and the boot timeout end the wait: a
	413	+	// boot cannot be aborted any other way.
	414	+	func (s *vmSandbox) waitExecutor(ctx context.Context, accepted <-chan acceptResult) (sandbox.Conn, error) {
	415	+	timer := time.NewTimer(s.d.opt.BootTimeout)
	416	+	defer timer.Stop()
	417	+	select {
	418	+	case r := <-accepted:
	419	+	if r.err != nil {
	420	+	return nil, fmt.Errorf("accept executor connection: %w", r.err)
	421	+	}
	422	+	s.conn = r.c
	423	+	return r.c, nil
	424	+	case <-s.exited:
	425	+	return nil, errors.New("firecracker exited before the executor connected")
	426	+	case <-timer.C:
	427	+	return nil, fmt.Errorf("executor did not connect within %s", s.d.opt.BootTimeout)
	428	+	case <-ctx.Done():
	429	+	return nil, context.Cause(ctx)
	430	+	}
	431	+	}
	432	+
	433	+	type acceptResult struct {
	434	+	c net.Conn
	435	+	err error
	436	+	}
	437	+
	438	+	// jailerArgs builds the jailer command line (after the binary name).
	439	+	//
	440	+	// - cgroup v2 with memory.max (guest memory + VMM overhead) and pids.max (a
	441	+	// VMM needs a handful of threads per vCPU): a runaway or compromised VMM
	442	+	// cannot take the host's memory or fork-bomb it;
	443	+	// - a new PID namespace: the VMM cannot see or signal host processes;
	444	+	// - no-file: the descriptor budget of the VMM.
	445	+	func jailerArgs(jailer, fc, id string, uid int, chrootBase string, spec sandbox.JobSpec, extra []string) []string {
	446	+	memBytes := (int64(spec.MemoryMiB) + vmmOverheadMiB) << 20
	447	+	pids := 32 + 8*spec.CPUs
	448	+	args := []string{
	449	+	"--id", id, "--exec-file", fc,
	450	+	"--uid", strconv.Itoa(uid), "--gid", strconv.Itoa(uid),
	451	+	"--chroot-base-dir", chrootBase,
	452	+	"--cgroup-version", "2",
	453	+	"--cgroup", "memory.max=" + strconv.FormatInt(memBytes, 10),
	454	+	"--cgroup", "pids.max=" + strconv.Itoa(pids),
	455	+	"--new-pid-ns",
	456	+	"--resource-limit", "no-file=4096",
	457	+	"--",
	458	+	}
	459	+	return append(args, extra...)
	460	+	}
	461	+
	462	+	func (s *vmSandbox) startVMM(spec sandbox.JobSpec, apiSock string) error {
	463	+	d := s.d
	464	+	var cmd *exec.Cmd
	465	+	if s.lay.jailed {
	466	+	args := jailerArgs(d.jailer, d.fcPath, s.id, s.uid(), jailBase(d.cfg.StateDir), spec, d.opt.FirecrackerArgs)
	467	+	cmd = exec.Command(d.jailer, args...)
	468	+	} else {
	469	+	args := append([]string{"--id", s.id, "--api-sock", apiSock}, d.opt.FirecrackerArgs...)
	470	+	cmd = exec.Command(d.fcPath, args...)
	471	+	}
	472	+	s.console = newRing(16 << 10)
	473	+	cmd.Stdout, cmd.Stderr = s.console, s.console
	474	+	cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
	475	+	if err := cmd.Start(); err != nil {
	476	+	return fmt.Errorf("start firecracker: %w", err)
	477	+	}
	478	+	s.cmd = cmd
	479	+	s.exited = make(chan struct{})
	480	+	jailed := s.lay.jailed
	481	+	go func() {
	482	+	_ = cmd.Wait()
	483	+	// With --new-pid-ns the jailer may exit as soon as it has started the
	484	+	// VMM in its namespace: the VMM is then gone only when its own pid is.
	485	+	if jailed {
	486	+	s.awaitVMMGone()
	487	+	}
	488	+	close(s.exited)
	489	+	}()
	490	+	s.rec.PID = cmd.Process.Pid
	491	+	return s.d.store.save(s.rec)
	492	+	}
	493	+
	494	+	// awaitVMMGone blocks until no process of this VM's id is left.
	495	+	func (s *vmSandbox) awaitVMMGone() {
	496	+	for {
	497	+	alive := false
	498	+	for pid, id := range vmmProcs(s.d.opt.ProcDir) {
	499	+	if id == s.id && pidAlive(pid) {
	500	+	alive = true
	501	+	}
	502	+	}
	503	+	if !alive {
	504	+	return
	505	+	}
	506	+	time.Sleep(50 * time.Millisecond)
	507	+	}
	508	+	}
	509	+
	510	+	func (s vmSandbox) waitAPI(ctx context.Context, api apiClient, sock string) error {
	511	+	deadline := time.Now().Add(10 * time.Second)
	512	+	for {
	513	+	if err := api.ready(ctx); err == nil {
	514	+	return nil
	515	+	}
	516	+	select {
	517	+	case <-s.exited:
	518	+	return errors.New("firecracker exited before its API was ready")
	519	+	case <-ctx.Done():
	520	+	return context.Cause(ctx)
	521	+	case <-time.After(25 * time.Millisecond):
	522	+	}
	523	+	if time.Now().After(deadline) {
	524	+	return fmt.Errorf("firecracker API socket %s not ready after 10s", sock)
	525	+	}
	526	+	}
	527	+	}
	528	+
	529	+	// Stop implements sandbox.Sandbox.
	530	+	func (s *vmSandbox) Stop(ctx context.Context) error {
	531	+	// Abort a boot in progress first: Start holds s.mu for its whole duration.
	532	+	s.gate.Abort()
	533	+	s.mu.Lock()
	534	+	defer s.mu.Unlock()
	535	+	if s.stopped {
	536	+	return s.leakErr
	537	+	}
	538	+	if !s.started {
	539	+	s.releaseLocked()
	540	+	return nil
	541	+	}
	542	+	return s.cleanupLocked(context.WithoutCancel(ctx))
	543	+	}
	544	+
	545	+	func (s *vmSandbox) releaseLocked() {
	546	+	s.stopped = true
	547	+	s.d.release(s.slot)
	548	+	}
	549	+
	550	+	// vmmPIDs returns every host process of this VM: the launched command and
	551	+	// whatever the /proc scan finds with this id.
	552	+	func (s *vmSandbox) vmmPIDs() []int {
	553	+	set := map[int]bool{}
	554	+	if s.cmd != nil && s.cmd.Process != nil {
	555	+	set[s.cmd.Process.Pid] = true
	556	+	}
	557	+	for pid, id := range vmmProcs(s.d.opt.ProcDir) {
	558	+	if id == s.id {
	559	+	set[pid] = true
	560	+	}
	561	+	}
	562	+	var out []int
	563	+	for pid := range set {
	564	+	out = append(out, pid)
	565	+	}
	566	+	return out
	567	+	}
	568	+
	569	+	// cleanupLocked removes everything the VM created, in dependency order. Steps
	570	+	// that need the VMM to be dead (network device, firewall rules, files, cgroup)
	571	+	// are skipped when it could not be killed: removing the TAP or the policy under
	572	+	// a VM that may still run would break the isolation it relies on, so the
	573	+	// sandbox is reported as leaked (sandbox.ErrLeaked), its slot stays taken, and
	574	+	// Reap finishes the job.
	575	+	func (s *vmSandbox) cleanupLocked(ctx context.Context) error {
	576	+	log := zerowrap.FromCtx(ctx)
	577	+	d := s.d
	578	+	var errs []error
	579	+	if s.conn != nil {
	580	+	_ = s.conn.Close()
	581	+	s.conn = nil
	582	+	}
	583	+	if s.listener != nil {
	584	+	_ = s.listener.Close()
	585	+	s.listener = nil
	586	+	}
	587	+	dead := true
	588	+	if s.cmd != nil && s.cmd.Process != nil {
	589	+	_ = syscall.Kill(-s.cmd.Process.Pid, syscall.SIGKILL) // the whole group
	590	+	}
	591	+	for _, pid := range s.vmmPIDs() {
	592	+	if !d.opt.kill(pid, 5*time.Second) {
	593	+	dead = false
	594	+	errs = append(errs, fmt.Errorf("vmm process %d did not exit", pid))
	595	+	}
	596	+	}
	597	+	if dead && s.exited != nil {
	598	+	select {
	599	+	case <-s.exited:
	600	+	case <-time.After(5 * time.Second):
	601	+	dead = false
	602	+	errs = append(errs, errors.New("vmm exit was not observed"))
	603	+	}
	604	+	}
	605	+	if dead {
	606	+	if s.attached {
	607	+	if err := d.net.Detach(ctx, s.tap, s.gateway); err != nil {
	608	+	errs = append(errs, err)
	609	+	}
	610	+	if err := deleteTap(ctx, d.opt.Run, s.tap); err != nil {
	611	+	errs = append(errs, err)
	612	+	} else {
	613	+	s.attached = false
	614	+	}
	615	+	}
	616	+	if s.lay.dir != "" {
	617	+	if err := removeAll(s.lay.dir); err != nil {
	618	+	errs = append(errs, err)
	619	+	}
	620	+	}
	621	+	if s.lay.jailed {
	622	+	if err := removeCgroup(d.opt.CgroupRoot, filepath.Base(d.fcPath), s.id); err != nil {
	623	+	errs = append(errs, err)
	624	+	}
	625	+	}
	626	+	if len(errs) == 0 {
	627	+	if err := d.store.remove(s.id); err != nil {
	628	+	errs = append(errs, err)
	629	+	}
	630	+	}
	631	+	}
	632	+	if len(errs) == 0 {
	633	+	s.releaseLocked()
	634	+	return nil
	635	+	}
	636	+	s.stopped = true
	637	+	d.markLeaked(s.slot)
	638	+	s.leakErr = fmt.Errorf("%w: vm %s: %w", sandbox.ErrLeaked, s.id, errors.Join(errs...))
	639	+	log.Error().Err(s.leakErr).Str("vm", s.id).Msg("micro-VM cleanup incomplete; call Reap to finish it")
	640	+	return s.leakErr
	641	+	}
	642	+
	643	+	// removeAll is os.RemoveAll that also reports a directory that is still there.
	644	+	func removeAll(dir string) error {
	645	+	if err := os.RemoveAll(dir); err != nil {
	646	+	return err
	647	+	}
	648	+	if _, err := os.Lstat(dir); err == nil {
	649	+	return fmt.Errorf("%s still exists after removal", dir)
	650	+	}
	651	+	return nil
	652	+	}
	653	+
	654	+	// removeCgroup removes the cgroup the jailer created for a VM
	655	+	// (<root>/<parent>/<id>, parent = the exec file name). The jailer never removes
	656	+	// it. A missing cgroup is fine; a non-empty one fails (rmdir semantics).
	657	+	func removeCgroup(root, parent, id string) error {
	658	+	err := os.Remove(filepath.Join(root, parent, id))
	659	+	if err == nil \|\| errors.Is(err, os.ErrNotExist) {
	660	+	return nil
	661	+	}
	662	+	return fmt.Errorf("remove cgroup %s: %w", filepath.Join(root, parent, id), err)
	663	+	}

A internal/sandbox/vm/images.go +97 -0

@@ -0,0 +1,97 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"context"
	7	+	"crypto/sha256"
	8	+	"encoding/hex"
	9	+	"fmt"
	10	+	"io"
	11	+	"os"
	12	+	"path/filepath"
	13	+
	14	+	"bnema.dev/ermite-runner/internal/cmdrun"
	15	+	"bnema.dev/ermite-runner/internal/sandbox"
	16	+	)
	17	+
	18	+	// ensureExecDisk builds (or reuses) the small read-only ext4 disk that carries
	19	+	// the runner's own executable: the guest root during boot, and /opt/ermite
	20	+	// afterwards. It is named after the executable's hash, so a runner upgrade
	21	+	// builds a new one and no image rebuild is needed; older ones are removed.
	22	+	// mkfs.ext4 -d needs no root.
	23	+	func ensureExecDisk(ctx context.Context, run cmdrun.Runner, exe, stateDir string) (string, error) {
	24	+	sum, size, err := hashFile(exe)
	25	+	if err != nil {
	26	+	return "", fmt.Errorf("hash executable: %w", err)
	27	+	}
	28	+	cache := filepath.Join(stateDir, "cache")
	29	+	if err := os.MkdirAll(cache, 0o755); err != nil {
	30	+	return "", err
	31	+	}
	32	+	name := "execdisk-" + sum[:16] + ".ext4"
	33	+	dest := filepath.Join(cache, name)
	34	+	if _, err := os.Stat(dest); err == nil {
	35	+	return dest, nil
	36	+	}
	37	+	// Drop disks of other executable versions.
	38	+	if olds, _ := filepath.Glob(filepath.Join(cache, "execdisk-*")); len(olds) > 0 {
	39	+	for _, o := range olds {
	40	+	_ = os.Remove(o)
	41	+	}
	42	+	}
	43	+
	44	+	tmp, err := os.MkdirTemp(cache, "execroot-")
	45	+	if err != nil {
	46	+	return "", err
	47	+	}
	48	+	defer func() { _ = os.RemoveAll(tmp) }()
	49	+	for _, d := range []string{"proc", "sys", "dev", "mnt/base", "mnt/scratch", "mnt/newroot"} {
	50	+	if err := os.MkdirAll(filepath.Join(tmp, d), 0o755); err != nil {
	51	+	return "", err
	52	+	}
	53	+	}
	54	+	if err := copyFile(exe, filepath.Join(tmp, sandbox.GuestExecName)); err != nil {
	55	+	return "", fmt.Errorf("copy executable: %w", err)
	56	+	}
	57	+	if err := os.Chmod(filepath.Join(tmp, sandbox.GuestExecName), 0o755); err != nil {
	58	+	return "", err
	59	+	}
	60	+
	61	+	part := dest + ".part"
	62	+	_ = os.Remove(part)
	63	+	f, err := os.OpenFile(part, os.O_WRONLY\|os.O_CREATE\|os.O_EXCL, 0o644)
	64	+	if err != nil {
	65	+	return "", err
	66	+	}
	67	+	// The executable, ext4 overhead and some room; the disk is read-only.
	68	+	mib := size/(1<<20)*115/100 + 16
	69	+	if err := f.Truncate(mib << 20); err != nil {
	70	+	_ = f.Close()
	71	+	return "", err
	72	+	}
	73	+	_ = f.Close()
	74	+	if _, err := run.Run(ctx, nil, "mkfs.ext4", "-q", "-F", "-L", "ermite-exec",
	75	+	"-O", "^has_journal", "-E", "root_owner=0:0", "-d", tmp, part); err != nil {
	76	+	_ = os.Remove(part)
	77	+	return "", fmt.Errorf("build exec disk: %w", err)
	78	+	}
	79	+	if err := os.Rename(part, dest); err != nil {
	80	+	return "", err
	81	+	}
	82	+	return dest, nil
	83	+	}
	84	+
	85	+	func hashFile(path string) (sum string, size int64, err error) {
	86	+	f, err := os.Open(path)
	87	+	if err != nil {
	88	+	return "", 0, err
	89	+	}
	90	+	defer func() { _ = f.Close() }()
	91	+	h := sha256.New()
	92	+	n, err := io.Copy(h, f)
	93	+	if err != nil {
	94	+	return "", 0, err
	95	+	}
	96	+	return hex.EncodeToString(h.Sum(nil)), n, nil
	97	+	}

A internal/sandbox/vm/layout.go +114 -0

@@ -0,0 +1,114 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"errors"
	7	+	"fmt"
	8	+	"io"
	9	+	"os"
	10	+	"path/filepath"
	11	+	"syscall"
	12	+	)
	13	+
	14	+	// layout is where the files of one VM live and how Firecracker names them.
	15	+	//
	16	+	// With the jailer, Firecracker sees only its chroot: <jailBase>/firecracker/<id>/root,
	17	+	// so shared files are hard-linked into it and named "/name"; per-job files are
	18	+	// created there and handed to the jailed uid. Without the jailer (development
	19	+	// and tests), Firecracker sees the real filesystem: shared files are used in
	20	+	// place and per-job files live in <stateDir>/vm/<id>.
	21	+	type layout struct {
	22	+	jailed bool
	23	+	root string // host path of the VM's working directory
	24	+	dir string // the directory removed whole at cleanup
	25	+	uid int
	26	+	}
	27	+
	28	+	// newLayout: execName is the base name of the Firecracker binary, which the
	29	+	// jailer uses as the first directory below its chroot base.
	30	+	func newLayout(stateDir, execName, id string, jailed bool, uid int) layout {
	31	+	if jailed {
	32	+	dir := filepath.Join(stateDir, "jail", execName, id)
	33	+	return layout{jailed: true, root: filepath.Join(dir, "root"), dir: dir, uid: uid}
	34	+	}
	35	+	dir := filepath.Join(stateDir, "vm", id)
	36	+	return layout{root: dir, dir: dir, uid: uid}
	37	+	}
	38	+
	39	+	// jailBase is the --chroot-base-dir given to the jailer.
	40	+	func jailBase(stateDir string) string { return filepath.Join(stateDir, "jail") }
	41	+
	42	+	// host returns the host path of a per-job file called name.
	43	+	func (l layout) host(name string) string { return filepath.Join(l.root, name) }
	44	+
	45	+	// fc returns the path Firecracker must be given for a per-job file.
	46	+	func (l layout) fc(name string) string {
	47	+	if l.jailed {
	48	+	return "/" + name
	49	+	}
	50	+	return l.host(name)
	51	+	}
	52	+
	53	+	// share makes the read-only shared file src available to Firecracker and
	54	+	// returns its path for Firecracker. In a jail it is hard-linked, which requires
	55	+	// the same filesystem as the state directory, and src must be readable by the
	56	+	// jailed uid (the link shares the inode: it is never chowned).
	57	+	func (l layout) share(src, name string) (string, error) {
	58	+	if !l.jailed {
	59	+	return src, nil
	60	+	}
	61	+	dst := l.host(name)
	62	+	if err := os.Link(src, dst); err != nil {
	63	+	if errors.Is(err, syscall.EXDEV) {
	64	+	return "", fmt.Errorf("cannot link %s into the jail: it must be on the same filesystem as the state directory (hard link across filesystems)", src)
	65	+	}
	66	+	return "", fmt.Errorf("link %s into the jail: %w", src, err)
	67	+	}
	68	+	return "/" + name, nil
	69	+	}
	70	+
	71	+	// own hands a per-job file in the jail to the jailed uid; no-op without jailer.
	72	+	// It never follows a symlink (Lchown): the jail is writable by the jailed uid
	73	+	// once Firecracker runs, so a link planted there must not redirect the chown.
	74	+	func (l layout) own(name string) error {
	75	+	if !l.jailed {
	76	+	return nil
	77	+	}
	78	+	return os.Lchown(l.host(name), l.uid, l.uid)
	79	+	}
	80	+
	81	+	// checkReadable verifies that other users can read the shared file, which the
	82	+	// jailed uid needs. Called once per file at driver start for jailed mode.
	83	+	func checkWorldReadable(path string) error {
	84	+	st, err := os.Stat(path)
	85	+	if err != nil {
	86	+	return err
	87	+	}
	88	+	if st.Mode().Perm()&0o004 == 0 {
	89	+	return fmt.Errorf("%s is not world-readable (mode %v): the jailed uid could not open it", path, st.Mode().Perm())
	90	+	}
	91	+	return nil
	92	+	}
	93	+
	94	+	// copyFile copies src to dst (mode 0600) and fsyncs it.
	95	+	func copyFile(src, dst string) error {
	96	+	in, err := os.Open(src)
	97	+	if err != nil {
	98	+	return err
	99	+	}
	100	+	defer func() { _ = in.Close() }()
	101	+	out, err := os.OpenFile(dst, os.O_WRONLY\|os.O_CREATE\|os.O_EXCL, 0o600)
	102	+	if err != nil {
	103	+	return err
	104	+	}
	105	+	if _, err := io.Copy(out, in); err != nil {
	106	+	_ = out.Close()
	107	+	return err
	108	+	}
	109	+	if err := out.Sync(); err != nil {
	110	+	_ = out.Close()
	111	+	return err
	112	+	}
	113	+	return out.Close()
	114	+	}

A internal/sandbox/vm/proc.go +122 -0

@@ -0,0 +1,122 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"os"
	8	+	"path/filepath"
	9	+	"strconv"
	10	+	"strings"
	11	+	"sync"
	12	+	"syscall"
	13	+	"time"
	14	+
	15	+	"bnema.dev/ermite-runner/internal/sandbox"
	16	+	)
	17	+
	18	+	// vmmProcs scans a /proc tree for processes started by this runner for a VM:
	19	+	// the Firecracker process and the jailer. A process is ours when its command
	20	+	// line has "--id <id>" with an id of the form sandbox.NewID produces and also
	21	+	// "--api-sock" (Firecracker) or "--exec-file" (jailer). The program name is not
	22	+	// looked at: the binary may be renamed or versioned (firecracker-v1.17), and a
	23	+	// jailer in a PID namespace leaves a Firecracker whose argv[0] is whatever the
	24	+	// jailer chose. Processes of other users of Firecracker (other ids) are never
	25	+	// matched. It finds orphans even when the state directory was lost.
	26	+	func vmmProcs(procDir string) map[int]string {
	27	+	out := map[int]string{}
	28	+	ents, err := os.ReadDir(procDir)
	29	+	if err != nil {
	30	+	return out
	31	+	}
	32	+	for _, e := range ents {
	33	+	pid, err := strconv.Atoi(e.Name())
	34	+	if err != nil \|\| pid <= 1 {
	35	+	continue
	36	+	}
	37	+	b, err := os.ReadFile(filepath.Join(procDir, e.Name(), "cmdline"))
	38	+	if err != nil \|\| len(b) == 0 {
	39	+	continue
	40	+	}
	41	+	if id := vmIDFromCmdline(strings.Split(strings.TrimRight(string(b), "\x00"), "\x00")); id != "" {
	42	+	out[pid] = id
	43	+	}
	44	+	}
	45	+	return out
	46	+	}
	47	+
	48	+	// vmIDFromCmdline returns the VM id of a command line, or "".
	49	+	func vmIDFromCmdline(args []string) string {
	50	+	var id string
	51	+	var marker bool
	52	+	for i := 1; i < len(args); i++ {
	53	+	switch args[i] {
	54	+	case "--id":
	55	+	if i+1 < len(args) && sandbox.ValidID(args[i+1]) {
	56	+	id = args[i+1]
	57	+	}
	58	+	case "--api-sock", "--exec-file":
	59	+	marker = true
	60	+	}
	61	+	}
	62	+	if marker {
	63	+	return id
	64	+	}
	65	+	return ""
	66	+	}
	67	+
	68	+	// pidAlive reports whether pid exists and is not a zombie.
	69	+	func pidAlive(pid int) bool {
	70	+	if err := syscall.Kill(pid, 0); err != nil {
	71	+	return false
	72	+	}
	73	+	if b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat"); err == nil {
	74	+	if i := bytes.LastIndexByte(b, ')'); i >= 0 && i+2 < len(b) && b[i+2] == 'Z' {
	75	+	return false
	76	+	}
	77	+	}
	78	+	return true
	79	+	}
	80	+
	81	+	// ring keeps the last N bytes written to it: the VMM's stdout/stderr, which
	82	+	// carries the guest serial console, for diagnostics when a boot fails.
	83	+	type ring struct {
	84	+	mu sync.Mutex
	85	+	max int
	86	+	buf []byte
	87	+	}
	88	+
	89	+	func newRing(max int) *ring { return &ring{max: max} }
	90	+
	91	+	func (r *ring) Write(p []byte) (int, error) {
	92	+	r.mu.Lock()
	93	+	defer r.mu.Unlock()
	94	+	r.buf = append(r.buf, p...)
	95	+	if len(r.buf) > r.max {
	96	+	r.buf = append([]byte(nil), r.buf[len(r.buf)-r.max:]...)
	97	+	}
	98	+	return len(p), nil
	99	+	}
	100	+
	101	+	// String returns the retained tail, trimmed.
	102	+	func (r *ring) String() string {
	103	+	r.mu.Lock()
	104	+	defer r.mu.Unlock()
	105	+	return string(bytes.TrimSpace(r.buf))
	106	+	}
	107	+
	108	+	// killAndWait sends SIGKILL to pid and waits until it is gone (or a zombie that
	109	+	// its parent will reap). It reports whether the process is gone.
	110	+	func killAndWait(pid int, timeout time.Duration) bool {
	111	+	_ = syscall.Kill(pid, syscall.SIGKILL)
	112	+	deadline := time.Now().Add(timeout)
	113	+	for {
	114	+	if !pidAlive(pid) {
	115	+	return true
	116	+	}
	117	+	if time.Now().After(deadline) {
	118	+	return false
	119	+	}
	120	+	time.Sleep(20 * time.Millisecond)
	121	+	}
	122	+	}

A internal/sandbox/vm/reap.go +133 -0

@@ -0,0 +1,133 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"fmt"
	9	+	"os"
	10	+	"path/filepath"
	11	+	"strconv"
	12	+	"strings"
	13	+	"time"
	14	+
	15	+	"github.com/bnema/zerowrap"
	16	+
	17	+	"bnema.dev/ermite-runner/internal/netpolicy"
	18	+	"bnema.dev/ermite-runner/internal/sandbox"
	19	+	)
	20	+
	21	+	// Reap implements sandbox.Driver. It removes what a previous run of the runner
	22	+	// left: VMM processes (Firecracker and jailer: matched by "--id <our id>" in
	23	+	// their command line, never by program name), TAP devices named ertap<N>, the
	24	+	// nftables table, VM directories and jailer cgroups. It also finishes the
	25	+	// cleanup of sandboxes that reported ErrLeaked and frees their slots. It relies
	26	+	// on the host-wide lock taken in New: no other runner can own what it removes.
	27	+	// It returns ErrBusy while a sandbox is running.
	28	+	func (d *Driver) Reap(ctx context.Context) error {
	29	+	log := zerowrap.FromCtx(ctx)
	30	+	d.mu.Lock()
	31	+	busy := d.slots.Busy()
	32	+	d.mu.Unlock()
	33	+	if busy > 0 {
	34	+	return fmt.Errorf("%w (%d running)", sandbox.ErrBusy, busy)
	35	+	}
	36	+	var errs []error
	37	+
	38	+	recs, err := d.store.list()
	39	+	if err != nil {
	40	+	errs = append(errs, err)
	41	+	}
	42	+	procs := vmmProcs(d.opt.ProcDir)
	43	+	killed := map[int]bool{}
	44	+	killProc := func(pid int, id string) {
	45	+	if killed[pid] {
	46	+	return
	47	+	}
	48	+	killed[pid] = true
	49	+	log.Warn().Int("pid", pid).Str("vm", id).Msg("reaping leftover VMM process")
	50	+	if !d.opt.kill(pid, 5*time.Second) {
	51	+	errs = append(errs, fmt.Errorf("vmm process %d (%s) did not exit", pid, id))
	52	+	}
	53	+	}
	54	+	// 1. Processes. A recorded pid is only trusted when /proc/<pid>/cmdline still
	55	+	// carries this record's id (pids are reused); then every other process of
	56	+	// ours found in /proc, which also covers a lost state directory.
	57	+	for _, r := range recs {
	58	+	if r.PID > 0 && cmdlineID(d.opt.ProcDir, r.PID) == r.ID {
	59	+	killProc(r.PID, r.ID)
	60	+	}
	61	+	}
	62	+	for pid, id := range procs {
	63	+	killProc(pid, id)
	64	+	}
	65	+
	66	+	// 2. Network: the nft table first (it names interfaces), then every TAP.
	67	+	if err := d.net.Reset(ctx); err != nil {
	68	+	errs = append(errs, fmt.Errorf("reset nftables table %s: %w", netpolicy.Table, err))
	69	+	}
	70	+	taps, err := leftoverTaps(ctx, d.opt.Run)
	71	+	if err != nil {
	72	+	errs = append(errs, fmt.Errorf("list interfaces: %w", err))
	73	+	}
	74	+	for _, t := range taps {
	75	+	log.Warn().Str("tap", t).Msg("removing leftover TAP device")
	76	+	if err := deleteTap(ctx, d.opt.Run, t); err != nil {
	77	+	errs = append(errs, err)
	78	+	}
	79	+	}
	80	+
	81	+	// 3. Files and cgroups. The jailer's chroot base has one directory per
	82	+	// exec-file name.
	83	+	parents := []string{filepath.Join(d.cfg.StateDir, "vm")}
	84	+	jails, _ := os.ReadDir(filepath.Join(d.cfg.StateDir, "jail"))
	85	+	for _, j := range jails {
	86	+	if j.IsDir() {
	87	+	parents = append(parents, filepath.Join(d.cfg.StateDir, "jail", j.Name()))
	88	+	}
	89	+	}
	90	+	for _, parent := range parents {
	91	+	ents, _ := os.ReadDir(parent)
	92	+	for _, e := range ents {
	93	+	if sandbox.ValidID(e.Name()) {
	94	+	log.Warn().Str("dir", filepath.Join(parent, e.Name())).Msg("removing leftover VM directory")
	95	+	if err := removeAll(filepath.Join(parent, e.Name())); err != nil {
	96	+	errs = append(errs, err)
	97	+	}
	98	+	}
	99	+	}
	100	+	}
	101	+	cg := filepath.Join(d.opt.CgroupRoot, filepath.Base(d.fcPath))
	102	+	cents, _ := os.ReadDir(cg)
	103	+	for _, e := range cents {
	104	+	if e.IsDir() && sandbox.ValidID(e.Name()) {
	105	+	if err := removeCgroup(d.opt.CgroupRoot, filepath.Base(d.fcPath), e.Name()); err != nil {
	106	+	errs = append(errs, err)
	107	+	}
	108	+	}
	109	+	}
	110	+	if len(errs) == 0 {
	111	+	for _, r := range recs {
	112	+	if err := d.store.remove(r.ID); err != nil {
	113	+	errs = append(errs, err)
	114	+	}
	115	+	}
	116	+	}
	117	+	if len(errs) == 0 {
	118	+	d.mu.Lock()
	119	+	d.slots.ReleaseLeaked()
	120	+	d.mu.Unlock()
	121	+	}
	122	+	return errors.Join(errs...)
	123	+	}
	124	+
	125	+	// cmdlineID returns the VM id in the command line of pid, or "" (not running,
	126	+	// not ours, or recycled for another program).
	127	+	func cmdlineID(procDir string, pid int) string {
	128	+	b, err := os.ReadFile(filepath.Join(procDir, strconv.Itoa(pid), "cmdline"))
	129	+	if err != nil \|\| len(b) == 0 {
	130	+	return ""
	131	+	}
	132	+	return vmIDFromCmdline(strings.Split(strings.TrimRight(string(b), "\x00"), "\x00"))
	133	+	}

A internal/sandbox/vm/state.go +82 -0

@@ -0,0 +1,82 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"encoding/json"
	7	+	"errors"
	8	+	"os"
	9	+	"path/filepath"
	10	+	"strings"
	11	+	)
	12	+
	13	+	// record is the on-disk trace of one VM. It is written before the first
	14	+	// resource is created and rewritten as resources appear, so that Reap can
	15	+	// remove them after a crash of the runner. The record is deleted last.
	16	+	type record struct {
	17	+	ID string `json:"id"`
	18	+	Slot int `json:"slot"`
	19	+	Job string `json:"job,omitempty"`
	20	+	PID int `json:"pid,omitempty"`
	21	+	Tap string `json:"tap,omitempty"`
	22	+	Gate string `json:"gateway,omitempty"`
	23	+	Dir string `json:"dir,omitempty"` // the jail directory of this VM (removed whole)
	24	+	Files []string `json:"files,omitempty"`
	25	+	}
	26	+
	27	+	type store struct{ dir string }
	28	+
	29	+	func (s store) path(id string) string { return filepath.Join(s.dir, id+".json") }
	30	+
	31	+	func (s store) save(r record) error {
	32	+	if err := os.MkdirAll(s.dir, 0o700); err != nil {
	33	+	return err
	34	+	}
	35	+	b, err := json.Marshal(r)
	36	+	if err != nil {
	37	+	return err
	38	+	}
	39	+	tmp := s.path(r.ID) + ".tmp"
	40	+	if err := os.WriteFile(tmp, b, 0o600); err != nil {
	41	+	return err
	42	+	}
	43	+	return os.Rename(tmp, s.path(r.ID))
	44	+	}
	45	+
	46	+	func (s store) remove(id string) error {
	47	+	if err := os.Remove(s.path(id)); err != nil && !errors.Is(err, os.ErrNotExist) {
	48	+	return err
	49	+	}
	50	+	return nil
	51	+	}
	52	+
	53	+	// list returns every readable record; unreadable or corrupt files are returned
	54	+	// as ID-only records so that Reap still deletes them.
	55	+	func (s store) list() ([]record, error) {
	56	+	ents, err := os.ReadDir(s.dir)
	57	+	if errors.Is(err, os.ErrNotExist) {
	58	+	return nil, nil
	59	+	}
	60	+	if err != nil {
	61	+	return nil, err
	62	+	}
	63	+	var out []record
	64	+	for _, e := range ents {
	65	+	name := e.Name()
	66	+	if e.IsDir() \|\| !strings.HasSuffix(name, ".json") {
	67	+	// A *.tmp left by a crash mid-write.
	68	+	if strings.HasSuffix(name, ".tmp") {
	69	+	_ = os.Remove(filepath.Join(s.dir, name))
	70	+	}
	71	+	continue
	72	+	}
	73	+	id := strings.TrimSuffix(name, ".json")
	74	+	var r record
	75	+	if b, err := os.ReadFile(filepath.Join(s.dir, name)); err == nil && json.Unmarshal(b, &r) == nil && r.ID == id {
	76	+	out = append(out, r)
	77	+	continue
	78	+	}
	79	+	out = append(out, record{ID: id, Slot: -1})
	80	+	}
	81	+	return out, nil
	82	+	}

A internal/sandbox/vm/tap.go +76 -0

@@ -0,0 +1,76 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"context"
	7	+	"fmt"
	8	+	"net/netip"
	9	+	"strconv"
	10	+	"strings"
	11	+
	12	+	"bnema.dev/ermite-runner/internal/cmdrun"
	13	+	)
	14	+
	15	+	// tapPrefix must match the "ertap*" wildcard of the nftables policy.
	16	+	const tapPrefix = "ertap"
	17	+
	18	+	func tapName(slot int) string { return tapPrefix + strconv.Itoa(slot) }
	19	+
	20	+	// createTap creates a persistent TAP owned by uid (the user Firecracker runs
	21	+	// as, so it may open it), gives the host side its /30 address and brings it up.
	22	+	// It needs CAP_NET_ADMIN. On error the TAP is removed again.
	23	+	func createTap(ctx context.Context, run cmdrun.Runner, name string, uid int, gateway netip.Addr) error {
	24	+	u := strconv.Itoa(uid)
	25	+	if _, err := run.Run(ctx, nil, "ip", "tuntap", "add", "dev", name, "mode", "tap", "user", u, "group", u); err != nil {
	26	+	return fmt.Errorf("create tap %s: %w", name, err)
	27	+	}
	28	+	for _, args := range [][]string{
	29	+	{"addr", "add", gateway.String() + "/30", "dev", name},
	30	+	{"link", "set", "dev", name, "up"},
	31	+	} {
	32	+	if _, err := run.Run(ctx, nil, "ip", args...); err != nil {
	33	+	_ = deleteTap(context.WithoutCancel(ctx), run, name)
	34	+	return fmt.Errorf("configure tap %s: %w", name, err)
	35	+	}
	36	+	}
	37	+	return nil
	38	+	}
	39	+
	40	+	// deleteTap removes a TAP; a missing device is not an error.
	41	+	func deleteTap(ctx context.Context, run cmdrun.Runner, name string) error {
	42	+	out, err := run.Run(ctx, nil, "ip", "link", "del", "dev", name)
	43	+	if err != nil && (strings.Contains(string(out), "Cannot find device") \|\| strings.Contains(err.Error(), "Cannot find device")) {
	44	+	return nil
	45	+	}
	46	+	return err
	47	+	}
	48	+
	49	+	// leftoverTaps lists interfaces named like ours ("ertap<N>").
	50	+	func leftoverTaps(ctx context.Context, run cmdrun.Runner) ([]string, error) {
	51	+	out, err := run.Run(ctx, nil, "ip", "-o", "link", "show")
	52	+	if err != nil {
	53	+	return nil, err
	54	+	}
	55	+	return parseTapNames(string(out)), nil
	56	+	}
	57	+
	58	+	// parseTapNames extracts "ertap<digits>" interface names from `ip -o link show`
	59	+	// ("3: ertap0: <...>" or "3: ertap0@if2: <...>").
	60	+	func parseTapNames(s string) []string {
	61	+	var out []string
	62	+	for _, line := range strings.Split(s, "\n") {
	63	+	_, rest, ok := strings.Cut(line, ": ")
	64	+	if !ok {
	65	+	continue
	66	+	}
	67	+	name, _, _ := strings.Cut(rest, ":")
	68	+	name, _, _ = strings.Cut(name, "@")
	69	+	if n, ok := strings.CutPrefix(name, tapPrefix); ok && n != "" {
	70	+	if _, err := strconv.Atoi(n); err == nil {
	71	+	out = append(out, name)
	72	+	}
	73	+	}
	74	+	}
	75	+	return out
	76	+	}

A internal/sandbox/vm/vm_test.go +1024 -0

@@ -0,0 +1,1024 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package vm
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"context"
	8	+	"encoding/json"
	9	+	"errors"
	10	+	"fmt"
	11	+	"io"
	12	+	"net"
	13	+	"net/http"
	14	+	"net/netip"
	15	+	"os"
	16	+	"os/exec"
	17	+	"path/filepath"
	18	+	"strings"
	19	+	"sync"
	20	+	"syscall"
	21	+	"testing"
	22	+	"time"
	23	+
	24	+	"github.com/bnema/zerowrap"
	25	+
	26	+	"bnema.dev/ermite-runner/internal/cmdrun"
	27	+	"bnema.dev/ermite-runner/internal/execstream"
	28	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	29	+	"bnema.dev/ermite-runner/internal/sandbox"
	30	+	"bnema.dev/ermite-runner/internal/sandbox/disk"
	31	+	"bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
	32	+	)
	33	+
	34	+	// The tests run a fake Firecracker: this test binary, re-executed through a
	35	+	// shell script named "firecracker" with FC_HELPER set. It serves the HTTP API on
	36	+	// the unix socket, records every request, and on InstanceStart plays the guest:
	37	+	// it connects to the vsock socket and speaks the executor protocol.
	38	+	func TestMain(m *testing.M) {
	39	+	if os.Getenv("FC_HELPER") != "" {
	40	+	fakeFirecracker()
	41	+	return
	42	+	}
	43	+	os.Exit(m.Run())
	44	+	}
	45	+
	46	+	func fakeFirecracker() {
	47	+	var sock, id string
	48	+	args := os.Args[1:]
	49	+	for i := 0; i+1 < len(args); i++ {
	50	+	switch args[i] {
	51	+	case "--api-sock":
	52	+	sock = args[i+1]
	53	+	case "--id":
	54	+	id = args[i+1]
	55	+	}
	56	+	}
	57	+	_ = id
	58	+	mode := os.Getenv("FC_MODE")
	59	+	if mode == "die" {
	60	+	fmt.Fprintln(os.Stderr, "fake firecracker: refusing to start")
	61	+	os.Exit(3)
	62	+	}
	63	+	logf, _ := os.OpenFile(os.Getenv("FC_LOG"), os.O_APPEND\|os.O_CREATE\|os.O_WRONLY, 0o644)
	64	+	var vsock string
	65	+	mux := http.NewServeMux()
	66	+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
	67	+	body, _ := io.ReadAll(r.Body)
	68	+	if r.Method == http.MethodPut {
	69	+	_, _ = fmt.Fprintf(logf, "%s %s %s\n", r.Method, r.URL.Path, body)
	70	+	}
	71	+	if r.URL.Path == "/vsock" {
	72	+	var v vsockDevice
	73	+	_ = json.Unmarshal(body, &v)
	74	+	vsock = v.UDSPath
	75	+	}
	76	+	w.WriteHeader(http.StatusNoContent)
	77	+	if r.URL.Path == "/actions" && mode != "hang" {
	78	+	go func() {
	79	+	c, err := net.Dial("unix", fmt.Sprintf("%s_%d", vsock, sandbox.VsockPort))
	80	+	if err != nil {
	81	+	_, _ = fmt.Fprintln(logf, "vsock dial:", err)
	82	+	return
	83	+	}
	84	+	defer func() { _ = c.Close() }()
	85	+	g := execstream.NewConn(c)
	86	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Hello{Hello: &execpb.Hello{Sandbox: "vm"}}})
	87	+	if _, err := g.Recv(); err != nil {
	88	+	return
	89	+	}
	90	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Output{Output: &execpb.Output{Data: []byte("hello from fake guest\n")}}})
	91	+	_ = g.Send(&execpb.Frame{Body: &execpb.Frame_Exit{Exit: &execpb.Exit{Code: 0}}})
	92	+	// Like a real VM, stay alive until killed.
	93	+	select {}
	94	+	}()
	95	+	}
	96	+	})
	97	+	ln, err := net.Listen("unix", sock)
	98	+	if err != nil {
	99	+	fmt.Fprintln(os.Stderr, err)
	100	+	os.Exit(2)
	101	+	}
	102	+	_ = http.Serve(ln, mux)
	103	+	}
	104	+
	105	+	type env struct {
	106	+	cfg sandbox.Config
	107	+	run *cmdrun.Fake
	108	+	plumb *sandboxtest.Plumbing
	109	+	log string
	110	+	taps map[string]bool
	111	+	mu sync.Mutex
	112	+	}
	113	+
	114	+	// newEnv prepares a state dir, fake binaries, kernel and images.
	115	+	func newEnv(t testing.T, mode string) env { return newEnvNamed(t, mode, "firecracker") }
	116	+
	117	+	// newEnvNamed runs the fake VMM under another program name (argv[0]).
	118	+	func newEnvNamed(t testing.T, mode, argv0 string) env {
	119	+	t.Helper()
	120	+	bash, err := exec.LookPath("bash")
	121	+	if err != nil {
	122	+	t.Skip("bash needed for the fake firecracker")
	123	+	}
	124	+	dir := t.TempDir()
	125	+	e := &env{taps: map[string]bool{}, plumb: sandboxtest.NewPlumbing(), log: filepath.Join(dir, "fc.log")}
	126	+	self, _ := os.Executable()
	127	+	fc := filepath.Join(dir, "firecracker")
	128	+	script := fmt.Sprintf("#!%s\nexport FC_HELPER=1 FC_MODE=%q FC_LOG=%q\nexec -a %q %q \"$@\"\n", bash, mode, e.log, argv0, self)
	129	+	if err := os.WriteFile(fc, []byte(script), 0o755); err != nil {
	130	+	t.Fatal(err)
	131	+	}
	132	+	img := filepath.Join(dir, "image")
	133	+	_ = os.MkdirAll(img, 0o755)
	134	+	for _, f := range []string{"vmlinux", "base.ext4"} {
	135	+	_ = os.WriteFile(filepath.Join(img, f), []byte("x"), 0o644)
	136	+	}
	137	+	exe := filepath.Join(dir, "runner")
	138	+	sandboxtest.WriteStaticELF(t, exe)
	139	+	e.cfg = sandbox.Config{
	140	+	Isolation: sandbox.VM, StateDir: filepath.Join(dir, "state"), Capacity: 2, Image: img,
	141	+	JobCPUs: 1, JobMemoryMiB: 512, JobDiskMiB: 1024, UIDBase: 900000, KVMPath: "/dev/null",
	142	+	Network: mustPrefix("10.200.0.0/16"), Firecracker: fc, JailerOff: true, Executable: exe, DNSPort: 0,
	143	+	LockFile: filepath.Join(dir, "host.lock"), JobPids: 1000, DiskBandwidthMiB: 400, DiskIOPS: 20000, NetMbit: 1000,
	144	+	}
	145	+	e.run = &cmdrun.Fake{Handle: func(c cmdrun.Call) ([]byte, error) {
	146	+	e.mu.Lock()
	147	+	defer e.mu.Unlock()
	148	+	switch {
	149	+	case c.Name == "mkfs.ext4" && len(c.Args) == 1 && c.Args[0] == "-V":
	150	+	return []byte("mke2fs 1.47.4 (6-Mar-2025)\n"), nil
	151	+	case c.Name == "ip" && len(c.Args) > 1 && c.Args[0] == "tuntap":
	152	+	e.taps[c.Args[3]] = true
	153	+	case c.Name == "ip" && len(c.Args) > 2 && c.Args[0] == "link" && c.Args[1] == "del":
	154	+	if !e.taps[c.Args[3]] {
	155	+	return []byte("Cannot find device"), errors.New("exit status 1")
	156	+	}
	157	+	delete(e.taps, c.Args[3])
	158	+	case c.Name == "ip" && len(c.Args) > 1 && c.Args[0] == "-o":
	159	+	var b strings.Builder
	160	+	i := 1
	161	+	for t := range e.taps {
	162	+	fmt.Fprintf(&b, "%d: %s: <BROADCAST> mtu 1500\n", i, t)
	163	+	i++
	164	+	}
	165	+	return []byte(b.String()), nil
	166	+	}
	167	+	return nil, nil
	168	+	}}
	169	+	return e
	170	+	}
	171	+
	172	+	func (e *env) tapCount() int { e.mu.Lock(); defer e.mu.Unlock(); return len(e.taps) }
	173	+
	174	+	func (e env) driver(t testing.T) *Driver {
	175	+	t.Helper()
	176	+	d, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: 5 * time.Second})
	177	+	if err != nil {
	178	+	t.Fatal(err)
	179	+	}
	180	+	t.Cleanup(func() { _ = d.Close() })
	181	+	return d
	182	+	}
	183	+
	184	+	// leftovers lists everything that must not survive a job.
	185	+	func (e env) leftovers(t testing.T) []string {
	186	+	t.Helper()
	187	+	var out []string
	188	+	_ = filepath.WalkDir(e.cfg.StateDir, func(p string, d os.DirEntry, err error) error {
	189	+	rel, _ := filepath.Rel(e.cfg.StateDir, p)
	190	+	switch {
	191	+	case err != nil \|\| rel == ".":
	192	+	case strings.HasPrefix(rel, "vm/") \|\| strings.HasPrefix(rel, "vms/") \|\| strings.HasPrefix(rel, "jail/"):
	193	+	out = append(out, rel)
	194	+	}
	195	+	return nil
	196	+	})
	197	+	if n := e.tapCount(); n != 0 {
	198	+	out = append(out, fmt.Sprintf("%d TAP devices", n))
	199	+	}
	200	+	if n := e.plumb.Live(); n != 0 {
	201	+	out = append(out, fmt.Sprintf("%d attached guests", n))
	202	+	}
	203	+	return out
	204	+	}
	205	+
	206	+	func runStub(t *testing.T, conn sandbox.Conn) (string, int) {
	207	+	t.Helper()
	208	+	var out strings.Builder
	209	+	_, exit, err := execstream.Drive(conn, &execpb.Run{}, nil, func(o *execpb.Output) { out.Write(o.Data) })
	210	+	if err != nil {
	211	+	t.Fatal(err)
	212	+	}
	213	+	return out.String(), int(exit.Code)
	214	+	}
	215	+
	216	+	func TestStartRunStop(t *testing.T) {
	217	+	e := newEnv(t, "")
	218	+	d := e.driver(t)
	219	+	sb, err := d.New()
	220	+	if err != nil {
	221	+	t.Fatal(err)
	222	+	}
	223	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "job-1"})
	224	+	if err != nil {
	225	+	t.Fatal(err)
	226	+	}
	227	+	out, code := runStub(t, conn)
	228	+	if out != "hello from fake guest\n" \|\| code != 0 {
	229	+	t.Fatalf("out=%q code=%d", out, code)
	230	+	}
	231	+	// While running: record, TAP, attachment, process.
	232	+	if e.tapCount() != 1 \|\| e.plumb.Live() != 1 {
	233	+	t.Fatalf("tap=%d attached=%d", e.tapCount(), e.plumb.Live())
	234	+	}
	235	+	if recs, _ := d.store.list(); len(recs) != 1 \|\| recs[0].Job != "job-1" \|\| recs[0].PID == 0 \|\| recs[0].Tap != "ertap0" {
	236	+	t.Fatalf("records: %+v", recs)
	237	+	}
	238	+	pid := sb.(*vmSandbox).rec.PID
	239	+
	240	+	if err := sb.Stop(context.Background()); err != nil {
	241	+	t.Fatal(err)
	242	+	}
	243	+	if left := e.leftovers(t); len(left) != 0 {
	244	+	t.Fatalf("leftovers after Stop: %v", left)
	245	+	}
	246	+	if err := syscall.Kill(pid, 0); err == nil {
	247	+	t.Fatalf("firecracker pid %d still alive", pid)
	248	+	}
	249	+	if err := sb.Stop(context.Background()); err != nil {
	250	+	t.Fatalf("Stop must be idempotent: %v", err)
	251	+	}
	252	+	if d.slots.InUse() != 0 {
	253	+	t.Fatal("slot not released")
	254	+	}
	255	+	}
	256	+
	257	+	// The API calls must follow the order Firecracker needs and carry the
	258	+	// plan's disk layout: exec disk (boot root, RO), base (RO), scratch (RW).
	259	+	func TestAPICallsAndDiskLayout(t *testing.T) {
	260	+	e := newEnv(t, "")
	261	+	d := e.driver(t)
	262	+	sb, _ := d.New()
	263	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j", CPUs: 3, MemoryMiB: 768, DiskMiB: 2048})
	264	+	if err != nil {
	265	+	t.Fatal(err)
	266	+	}
	267	+	runStub(t, conn)
	268	+	defer func() { _ = sb.Stop(context.Background()) }()
	269	+
	270	+	b, _ := os.ReadFile(e.log)
	271	+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
	272	+	var paths []string
	273	+	for _, l := range lines {
	274	+	f := strings.Fields(l)
	275	+	if len(f) >= 2 && f[0] == "PUT" {
	276	+	paths = append(paths, f[1])
	277	+	}
	278	+	}
	279	+	want := []string{"/machine-config", "/boot-source", "/drives/rootfs", "/drives/base", "/drives/scratch", "/network-interfaces/eth0", "/vsock", "/entropy", "/actions"}
	280	+	if strings.Join(paths, ",") != strings.Join(want, ",") {
	281	+	t.Fatalf("call order:\n got %v\nwant %v", paths, want)
	282	+	}
	283	+	log := string(b)
	284	+	for _, frag := range []string{
	285	+	`"vcpu_count":3`, `"mem_size_mib":768`, `"smt":false`,
	286	+	`"drive_id":"rootfs"`, `"is_root_device":true,"is_read_only":true`,
	287	+	`"drive_id":"base"`, `"is_root_device":false,"is_read_only":true`,
	288	+	`"drive_id":"scratch"`, `"is_root_device":false,"is_read_only":false`,
	289	+	`"host_dev_name":"ertap0"`, `"guest_cid":3`, `"action_type":"InstanceStart"`,
	290	+	"ipv6.disable=1", "ermite.connect=vsock:1024", "ermite.gw=10.200.0.1", "ip=10.200.0.2::10.200.0.1:255.255.255.252",
	291	+	} {
	292	+	if !strings.Contains(log, frag) {
	293	+	t.Errorf("API log lacks %q", frag)
	294	+	}
	295	+	}
	296	+	// The scratch disk was created sparse at the requested size, and formatted.
	297	+	var mkfs []string
	298	+	for _, l := range e.run.Lines() {
	299	+	if strings.HasPrefix(l, "mkfs.ext4") {
	300	+	mkfs = append(mkfs, l)
	301	+	}
	302	+	}
	303	+	if len(mkfs) == 0 {
	304	+	t.Fatalf("no mkfs call: %v", e.run.Lines())
	305	+	}
	306	+	}
	307	+
	308	+	func TestJobsDoNotShareSlotsOrTaps(t *testing.T) {
	309	+	e := newEnv(t, "")
	310	+	d := e.driver(t)
	311	+	var sbs []sandbox.Sandbox
	312	+	for i := range 2 {
	313	+	sb, err := d.New()
	314	+	if err != nil {
	315	+	t.Fatal(err)
	316	+	}
	317	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: fmt.Sprint(i)})
	318	+	if err != nil {
	319	+	t.Fatal(err)
	320	+	}
	321	+	runStub(t, conn)
	322	+	sbs = append(sbs, sb)
	323	+	}
	324	+	if _, err := d.New(); !errors.Is(err, sandbox.ErrCapacity) {
	325	+	t.Fatalf("capacity not enforced: %v", err)
	326	+	}
	327	+	if e.tapCount() != 2 {
	328	+	t.Fatalf("taps: %v", e.taps)
	329	+	}
	330	+	for _, sb := range sbs {
	331	+	_ = sb.Stop(context.Background())
	332	+	}
	333	+	if left := e.leftovers(t); len(left) != 0 {
	334	+	t.Fatalf("leftovers: %v", left)
	335	+	}
	336	+	}
	337	+
	338	+	func TestStartFailureCleansEverything(t *testing.T) {
	339	+	e := newEnv(t, "die")
	340	+	d := e.driver(t)
	341	+	sb, _ := d.New()
	342	+	_, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	343	+	if err == nil \|\| !strings.Contains(err.Error(), "refusing to start") {
	344	+	t.Fatalf("want the VMM console in the error, got %v", err)
	345	+	}
	346	+	if left := e.leftovers(t); len(left) != 0 {
	347	+	t.Fatalf("leftovers after failed Start: %v", left)
	348	+	}
	349	+	if d.slots.InUse() != 0 {
	350	+	t.Fatal("slot leaked by a failed Start")
	351	+	}
	352	+	}
	353	+
	354	+	func TestBootTimeoutCleansEverything(t *testing.T) {
	355	+	e := newEnv(t, "hang")
	356	+	d, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: 300 * time.Millisecond})
	357	+	if err != nil {
	358	+	t.Fatal(err)
	359	+	}
	360	+	defer func() { _ = d.Close() }()
	361	+	sb, _ := d.New()
	362	+	_, err = sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	363	+	if err == nil \|\| !strings.Contains(err.Error(), "did not connect") {
	364	+	t.Fatalf("got %v", err)
	365	+	}
	366	+	if left := e.leftovers(t); len(left) != 0 {
	367	+	t.Fatalf("leftovers: %v", left)
	368	+	}
	369	+	}
	370	+
	371	+	func TestStartCancelledByContext(t *testing.T) {
	372	+	e := newEnv(t, "hang")
	373	+	d := e.driver(t)
	374	+	sb, _ := d.New()
	375	+	ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
	376	+	defer cancel()
	377	+	if _, err := sb.Start(ctx, sandbox.JobSpec{JobID: "j"}); !errors.Is(err, context.DeadlineExceeded) {
	378	+	t.Fatalf("got %v", err)
	379	+	}
	380	+	if left := e.leftovers(t); len(left) != 0 {
	381	+	t.Fatalf("leftovers: %v", left)
	382	+	}
	383	+	}
	384	+
	385	+	func TestNetworkAttachFailureCleans(t *testing.T) {
	386	+	e := newEnv(t, "")
	387	+	d := e.driver(t)
	388	+	e.plumb.FailNext = errors.New("nft broke")
	389	+	sb, _ := d.New()
	390	+	if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err == nil \|\| !strings.Contains(err.Error(), "nft broke") {
	391	+	t.Fatalf("got %v", err)
	392	+	}
	393	+	if left := e.leftovers(t); len(left) != 0 {
	394	+	t.Fatalf("leftovers: %v", left)
	395	+	}
	396	+	}
	397	+
	398	+	func TestStopWithoutStartReleasesSlot(t *testing.T) {
	399	+	e := newEnv(t, "")
	400	+	d := e.driver(t)
	401	+	sb, _ := d.New()
	402	+	if err := sb.Stop(context.Background()); err != nil {
	403	+	t.Fatal(err)
	404	+	}
	405	+	if d.slots.InUse() != 0 {
	406	+	t.Fatal("slot leaked")
	407	+	}
	408	+	if _, err := sb.Start(context.Background(), sandbox.JobSpec{}); err == nil {
	409	+	t.Fatal("Start after Stop accepted")
	410	+	}
	411	+	}
	412	+
	413	+	// Stop must finish the cleanup even when the caller's context is already done
	414	+	// (a cancelled job).
	415	+	func TestStopRunsWithCancelledContext(t *testing.T) {
	416	+	e := newEnv(t, "")
	417	+	d := e.driver(t)
	418	+	sb, _ := d.New()
	419	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	420	+	if err != nil {
	421	+	t.Fatal(err)
	422	+	}
	423	+	runStub(t, conn)
	424	+	ctx, cancel := context.WithCancel(context.Background())
	425	+	cancel()
	426	+	if err := sb.Stop(ctx); err != nil {
	427	+	t.Fatal(err)
	428	+	}
	429	+	if left := e.leftovers(t); len(left) != 0 {
	430	+	t.Fatalf("leftovers: %v", left)
	431	+	}
	432	+	}
	433	+
	434	+	// A killed runner leaves a running Firecracker, a TAP, nft rules and files. The
	435	+	// next start must reap all of it (AC4: nothing survives, incl. killed jobs).
	436	+	func TestReapAfterRunnerCrash(t *testing.T) {
	437	+	e := newEnv(t, "")
	438	+	d1 := e.driver(t)
	439	+	sb, _ := d1.New()
	440	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	441	+	if err != nil {
	442	+	t.Fatal(err)
	443	+	}
	444	+	runStub(t, conn)
	445	+	pid := sb.(*vmSandbox).rec.PID
	446	+	if err := syscall.Kill(pid, 0); err != nil {
	447	+	t.Fatal("fake VM not running")
	448	+	}
	449	+	// The runner dies: its lock goes away, nothing else is cleaned.
	450	+	_ = d1.lock.Release()
	451	+	_ = conn.Close()
	452	+	if e.tapCount() != 1 {
	453	+	t.Fatal("setup: no TAP")
	454	+	}
	455	+
	456	+	d2 := e.driver(t)
	457	+	if err := d2.Reap(context.Background()); err != nil {
	458	+	t.Fatal(err)
	459	+	}
	460	+	if left := e.leftovers(t); len(left) != 0 {
	461	+	t.Fatalf("leftovers after Reap: %v", left)
	462	+	}
	463	+	deadline := time.Now().Add(3 * time.Second)
	464	+	for syscall.Kill(pid, 0) == nil && time.Now().Before(deadline) {
	465	+	// The fake is a child of this test process: it is a zombie until the
	466	+	// original driver's Wait goroutine collects it.
	467	+	if b, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid)); err == nil && strings.Contains(string(b), ") Z") {
	468	+	break
	469	+	}
	470	+	time.Sleep(20 * time.Millisecond)
	471	+	}
	472	+	if b, _ := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid)); len(b) > 0 && !strings.Contains(string(b), ") Z") {
	473	+	t.Fatalf("firecracker %d survived Reap: %s", pid, b)
	474	+	}
	475	+	if e.plumb.Resets == 0 {
	476	+	t.Fatal("the nftables table was not reset")
	477	+	}
	478	+	}
	479	+
	480	+	func TestReapRemovesCorruptRecordAndTempFiles(t *testing.T) {
	481	+	e := newEnv(t, "")
	482	+	d := e.driver(t)
	483	+	_ = os.MkdirAll(d.store.dir, 0o700)
	484	+	_ = os.WriteFile(filepath.Join(d.store.dir, "er00112233.json"), []byte("{garbage"), 0o600)
	485	+	_ = os.WriteFile(filepath.Join(d.store.dir, "er44556677.json.tmp"), []byte("{"), 0o600)
	486	+	_ = os.MkdirAll(filepath.Join(e.cfg.StateDir, "vm", "er00112233"), 0o700)
	487	+	_ = os.MkdirAll(filepath.Join(e.cfg.StateDir, "vm", "not-ours"), 0o700) // must be left alone
	488	+	if err := d.Reap(context.Background()); err != nil {
	489	+	t.Fatal(err)
	490	+	}
	491	+	if _, err := os.Stat(filepath.Join(e.cfg.StateDir, "vm", "not-ours")); err != nil {
	492	+	t.Fatal("Reap removed a directory that is not ours")
	493	+	}
	494	+	_ = os.Remove(filepath.Join(e.cfg.StateDir, "vm", "not-ours"))
	495	+	if left := e.leftovers(t); len(left) != 0 {
	496	+	t.Fatalf("leftovers: %v", left)
	497	+	}
	498	+	}
	499	+
	500	+	func TestReapLeavesForeignTapsAlone(t *testing.T) {
	501	+	e := newEnv(t, "")
	502	+	e.taps["tap0"] = true
	503	+	e.taps["ertapx"] = true // not ertap<digits>
	504	+	e.taps["ertap7"] = true
	505	+	d := e.driver(t)
	506	+	if err := d.Reap(context.Background()); err != nil {
	507	+	t.Fatal(err)
	508	+	}
	509	+	if !e.taps["tap0"] \|\| !e.taps["ertapx"] \|\| e.taps["ertap7"] {
	510	+	t.Fatalf("taps after reap: %v", e.taps)
	511	+	}
	512	+	}
	513	+
	514	+	func TestSecondRunnerOnSameStateDirRefused(t *testing.T) {
	515	+	e := newEnv(t, "")
	516	+	_ = e.driver(t)
	517	+	_, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb})
	518	+	if err == nil \|\| !strings.Contains(err.Error(), "another ermite-runner is running on this host") {
	519	+	t.Fatalf("got %v", err)
	520	+	}
	521	+	}
	522	+
	523	+	func TestNewRejectsDynamicExecutable(t *testing.T) {
	524	+	e := newEnv(t, "")
	525	+	e.cfg.Executable = "/bin/sh"
	526	+	if _, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb}); err == nil {
	527	+	t.Skip("/bin/sh is static here")
	528	+	}
	529	+	}
	530	+
	531	+	func TestExecDiskIsKeyedByExecutableHash(t *testing.T) {
	532	+	e := newEnv(t, "")
	533	+	d1 := e.driver(t)
	534	+	first := d1.execDisk
	535	+	_ = d1.Close()
	536	+	// Same executable: reused, no second mkfs of an exec disk.
	537	+	n := len(e.run.Calls)
	538	+	d2 := e.driver(t)
	539	+	if d2.execDisk != first {
	540	+	t.Fatal("exec disk rebuilt for the same executable")
	541	+	}
	542	+	for _, c := range e.run.Calls[n:] {
	543	+	if c.Name == "mkfs.ext4" && (len(c.Args) != 1 \|\| c.Args[0] != "-V") {
	544	+	t.Fatal("mkfs run again for an unchanged executable")
	545	+	}
	546	+	}
	547	+	_ = d2.Close()
	548	+	// A runner upgrade: new disk, old one removed.
	549	+	if err := os.WriteFile(e.cfg.Executable, append(readFile(t, e.cfg.Executable), 0), 0o755); err != nil {
	550	+	t.Fatal(err)
	551	+	}
	552	+	d3 := e.driver(t)
	553	+	if d3.execDisk == first {
	554	+	t.Fatal("exec disk not rebuilt after the executable changed")
	555	+	}
	556	+	if _, err := os.Stat(first); err == nil {
	557	+	t.Fatal("stale exec disk kept")
	558	+	}
	559	+	}
	560	+
	561	+	func readFile(t *testing.T, p string) []byte {
	562	+	t.Helper()
	563	+	b, err := os.ReadFile(p)
	564	+	if err != nil {
	565	+	t.Fatal(err)
	566	+	}
	567	+	return b
	568	+	}
	569	+
	570	+	func TestJailedLayout(t *testing.T) {
	571	+	l := newLayout("/var/lib/er", "firecracker", "er00112233", true, 900003)
	572	+	if l.root != "/var/lib/er/jail/firecracker/er00112233/root" \|\| l.dir != "/var/lib/er/jail/firecracker/er00112233" {
	573	+	t.Fatalf("%+v", l)
	574	+	}
	575	+	if l.fc("scratch.ext4") != "/scratch.ext4" \|\| l.host("scratch.ext4") != l.root+"/scratch.ext4" {
	576	+	t.Fatal("jailed paths")
	577	+	}
	578	+	if jailBase("/var/lib/er") != "/var/lib/er/jail" {
	579	+	t.Fatal("jail base")
	580	+	}
	581	+	if l2 := newLayout("/var/lib/er", "firecracker-v1.17", "er00112233", true, 1); l2.dir != "/var/lib/er/jail/firecracker-v1.17/er00112233" {
	582	+	t.Fatalf("the jailer names its chroot after the exec file: %s", l2.dir)
	583	+	}
	584	+	u := newLayout("/var/lib/er", "firecracker", "er00112233", false, 1000)
	585	+	if u.fc("api.sock") != "/var/lib/er/vm/er00112233/api.sock" {
	586	+	t.Fatalf("unjailed path %s", u.fc("api.sock"))
	587	+	}
	588	+	if p, _ := u.share("/img/base.ext4", "base.ext4"); p != "/img/base.ext4" {
	589	+	t.Fatal("unjailed share must use the file in place")
	590	+	}
	591	+	}
	592	+
	593	+	func TestJailedShareHardLinks(t *testing.T) {
	594	+	dir := t.TempDir()
	595	+	src := filepath.Join(dir, "base.ext4")
	596	+	_ = os.WriteFile(src, []byte("data"), 0o644)
	597	+	l := newLayout(dir, "firecracker", "er00112233", true, os.Getuid())
	598	+	if err := os.MkdirAll(l.root, 0o700); err != nil {
	599	+	t.Fatal(err)
	600	+	}
	601	+	p, err := l.share(src, "base.ext4")
	602	+	if err != nil \|\| p != "/base.ext4" {
	603	+	t.Fatalf("%q %v", p, err)
	604	+	}
	605	+	a, _ := os.Stat(src)
	606	+	b, _ := os.Stat(l.host("base.ext4"))
	607	+	if !os.SameFile(a, b) {
	608	+	t.Fatal("shared file must be a hard link, never a copy")
	609	+	}
	610	+	if err := checkWorldReadable(src); err != nil {
	611	+	t.Fatal(err)
	612	+	}
	613	+	_ = os.Chmod(src, 0o600)
	614	+	if err := checkWorldReadable(src); err == nil {
	615	+	t.Fatal("unreadable-by-others file accepted for the jail")
	616	+	}
	617	+	}
	618	+
	619	+	func TestBootArgs(t *testing.T) {
	620	+	a := bootArgs(mustAddr("10.200.0.6"), mustAddr("10.200.0.5"), 1791818522)
	621	+	for _, want := range []string{
	622	+	"ipv6.disable=1", "root=/dev/vda ro", "init=/ermite-runner", "reboot=k", "panic=1",
	623	+	"ip=10.200.0.6::10.200.0.5:255.255.255.252:ermite-job:eth0:off:10.200.0.5",
	624	+	"ermite.sandbox=vm", "ermite.connect=vsock:1024", "ermite.gw=10.200.0.5", "ermite.time=1791818522",
	625	+	} {
	626	+	if !strings.Contains(a, want) {
	627	+	t.Errorf("boot args lack %q: %s", want, a)
	628	+	}
	629	+	}
	630	+	if !strings.HasSuffix(a, "-- init") {
	631	+	t.Errorf("init subcommand missing: %s", a)
	632	+	}
	633	+	if macFor(0) != "06:00:ac:10:00:00" \|\| macFor(257) != "06:00:ac:10:01:01" {
	634	+	t.Fatal(macFor(257))
	635	+	}
	636	+	}
	637	+
	638	+	func TestParseTapNames(t *testing.T) {
	639	+	in := "1: lo: <LOOPBACK> mtu 65536\n2: eth0: <UP> mtu 1500\n5: ertap0: <BROADCAST> mtu 1500\n9: ertap12@NONE: <X>\n10: ertapx: <X>\n11: tap0: <X>\n12: ertap: <X>\n"
	640	+	got := strings.Join(parseTapNames(in), ",")
	641	+	if got != "ertap0,ertap12" {
	642	+	t.Fatalf("got %q", got)
	643	+	}
	644	+	}
	645	+
	646	+	func TestFirecrackerProcsScan(t *testing.T) {
	647	+	proc := t.TempDir()
	648	+	mk := func(pid, cmdline string) {
	649	+	_ = os.MkdirAll(filepath.Join(proc, pid), 0o755)
	650	+	_ = os.WriteFile(filepath.Join(proc, pid, "cmdline"), []byte(cmdline), 0o644)
	651	+	}
	652	+	mk("100", "firecracker\x00--id\x00er00112233\x00--api-sock\x00/x\x00")
	653	+	mk("101", "/usr/bin/firecracker\x00--api-sock\x00/x\x00--id\x00er44556677\x00")
	654	+	mk("102", "firecracker\x00--id\x00other-tool-vm\x00") // someone else's VM
	655	+	mk("103", "bash\x00--id\x00er00112233\x00") // not firecracker
	656	+	mk("1", "firecracker\x00--id\x00er99999999\x00") // pid 1 is never touched
	657	+	mk("abc", "")
	658	+	got := vmmProcs(proc)
	659	+	if len(got) != 2 \|\| got[100] != "er00112233" \|\| got[101] != "er44556677" {
	660	+	t.Fatalf("%v", got)
	661	+	}
	662	+	}
	663	+
	664	+	func TestRingKeepsTail(t *testing.T) {
	665	+	r := newRing(8)
	666	+	_, _ = r.Write([]byte("0123456789"))
	667	+	_, _ = r.Write([]byte("ab"))
	668	+	if r.String() != "23456789ab"[2:] {
	669	+	t.Fatalf("%q", r.String())
	670	+	}
	671	+	}
	672	+
	673	+	func TestAPIErrorsSurfaceFirecrackerFault(t *testing.T) {
	674	+	sock := filepath.Join(t.TempDir(), "s")
	675	+	ln, err := net.Listen("unix", sock)
	676	+	if err != nil {
	677	+	t.Fatal(err)
	678	+	}
	679	+	defer func() { _ = ln.Close() }()
	680	+	go func() {
	681	+	_ = http.Serve(ln, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	682	+	w.WriteHeader(400)
	683	+	_, _ = w.Write([]byte(`{"fault_message":"Invalid request"}`))
	684	+	}))
	685	+	}()
	686	+	err = newAPIClient(sock).put(context.Background(), "/boot-source", bootSource{})
	687	+	if err == nil \|\| !strings.Contains(err.Error(), "Invalid request") \|\| !strings.Contains(err.Error(), "/boot-source") {
	688	+	t.Fatalf("got %v", err)
	689	+	}
	690	+	}
	691	+
	692	+	func mustPrefix(s string) netip.Prefix { return netip.MustParsePrefix(s) }
	693	+	func mustAddr(s string) netip.Addr { return netip.MustParseAddr(s) }
	694	+
	695	+	// C3: processes are matched by "--id <our id>" in their command line, not by the
	696	+	// program name: a renamed or versioned Firecracker binary is still found and
	697	+	// reaped, and a recycled pid is never killed on the strength of a record alone.
	698	+	func TestReapFindsRenamedVMMByID(t *testing.T) {
	699	+	e := newEnvNamed(t, "", "fc-renamed-1.17")
	700	+	d1 := e.driver(t)
	701	+	sb, _ := d1.New()
	702	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	703	+	if err != nil {
	704	+	t.Fatal(err)
	705	+	}
	706	+	runStub(t, conn)
	707	+	pid := sb.(*vmSandbox).rec.PID
	708	+	if got := cmdlineID("/proc", pid); got != sb.ID() {
	709	+	t.Fatalf("cmdlineID=%q want %q (argv[0] is %q)", got, sb.ID(), "fc-renamed-1.17")
	710	+	}
	711	+	_ = d1.lock.Release()
	712	+	_ = conn.Close()
	713	+	d2 := e.driver(t)
	714	+	if err := d2.Reap(context.Background()); err != nil {
	715	+	t.Fatal(err)
	716	+	}
	717	+	if pidAlive(pid) {
	718	+	t.Fatalf("a VMM with a renamed binary survived Reap (pid %d)", pid)
	719	+	}
	720	+	}
	721	+
	722	+	func TestVMIDFromCmdline(t *testing.T) {
	723	+	for name, tc := range map[string]struct {
	724	+	args []string
	725	+	want string
	726	+	}{
	727	+	"firecracker": {[]string{"firecracker", "--id", "er00112233", "--api-sock", "/x"}, "er00112233"},
	728	+	"jailer": {[]string{"jailer", "--id", "er00112233", "--exec-file", "/fc", "--uid", "1"}, "er00112233"},
	729	+	"renamed": {[]string{"anything", "--api-sock", "/x", "--id", "er44556677"}, "er44556677"},
	730	+	"foreign id": {[]string{"firecracker", "--id", "my-vm", "--api-sock", "/x"}, ""},
	731	+	"no marker": {[]string{"python", "--id", "er00112233"}, ""},
	732	+	"id without value": {[]string{"firecracker", "--api-sock", "/x", "--id"}, ""},
	733	+	"path-like id": {[]string{"firecracker", "--id", "er../../../x", "--api-sock", "/x"}, ""},
	734	+	} {
	735	+	if got := vmIDFromCmdline(tc.args); got != tc.want {
	736	+	t.Errorf("%s: got %q want %q", name, got, tc.want)
	737	+	}
	738	+	}
	739	+	}
	740	+
	741	+	func TestReapDoesNotTrustARecycledPid(t *testing.T) {
	742	+	e := newEnv(t, "")
	743	+	d := e.driver(t)
	744	+	// A record pointing at this very test process: its command line carries no
	745	+	// VM id, so it must not be killed.
	746	+	_ = d.store.save(record{ID: "er00112233", Slot: 0, PID: os.Getpid()})
	747	+	if err := d.Reap(context.Background()); err != nil {
	748	+	t.Fatal(err)
	749	+	}
	750	+	if !pidAlive(os.Getpid()) {
	751	+	t.Fatal("Reap killed a process on the strength of a stale record")
	752	+	}
	753	+	}
	754	+
	755	+	// C5 + C8: when the VMM cannot be killed, the TAP, the firewall policy and the
	756	+	// files are left alone (the VM may still be running), the error wraps ErrLeaked,
	757	+	// the slot stays taken, and Reap (with a working kill) finishes the cleanup and
	758	+	// frees the slot.
	759	+	func TestStopKeepsNetworkWhenTheVMMSurvives(t *testing.T) {
	760	+	e := newEnv(t, "")
	761	+	stubborn := true
	762	+	d, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: 5 * time.Second,
	763	+	kill: func(pid int, to time.Duration) bool {
	764	+	if stubborn {
	765	+	return false
	766	+	}
	767	+	return killAndWait(pid, to)
	768	+	}})
	769	+	if err != nil {
	770	+	t.Fatal(err)
	771	+	}
	772	+	defer func() { _ = d.Close() }()
	773	+	sb, _ := d.New()
	774	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	775	+	if err != nil {
	776	+	t.Fatal(err)
	777	+	}
	778	+	runStub(t, conn)
	779	+
	780	+	err = sb.Stop(context.Background())
	781	+	if !errors.Is(err, sandbox.ErrLeaked) {
	782	+	t.Fatalf("Stop must report ErrLeaked, got %v", err)
	783	+	}
	784	+	if e.tapCount() != 1 \|\| e.plumb.Live() != 1 {
	785	+	t.Fatalf("the TAP and the policy must stay while the VMM may run: taps=%d attached=%d", e.tapCount(), e.plumb.Live())
	786	+	}
	787	+	if recs, _ := d.store.list(); len(recs) != 1 {
	788	+	t.Fatalf("the state record must stay for Reap: %v", recs)
	789	+	}
	790	+	if err := sb.Stop(context.Background()); !errors.Is(err, sandbox.ErrLeaked) {
	791	+	t.Fatalf("a second Stop must keep reporting the leak, got %v", err)
	792	+	}
	793	+	if d.slots.InUse() != 1 {
	794	+	t.Fatal("the slot of a leaked sandbox stays taken")
	795	+	}
	796	+	// Other jobs keep running on the free slot; the leaked one is never reused.
	797	+	other, err := d.New()
	798	+	if err != nil {
	799	+	t.Fatal(err)
	800	+	}
	801	+	if other.(vmSandbox).slot == sb.(vmSandbox).slot {
	802	+	t.Fatal("a leaked slot was handed out again")
	803	+	}
	804	+	// Reap refuses while a sandbox is running...
	805	+	if err := d.Reap(context.Background()); !errors.Is(err, sandbox.ErrBusy) {
	806	+	t.Fatalf("Reap with a handed-out sandbox: %v", err)
	807	+	}
	808	+	_ = other.Stop(context.Background())
	809	+	// ...and finishes the job once idle.
	810	+	stubborn = false
	811	+	if err := d.Reap(context.Background()); err != nil {
	812	+	t.Fatal(err)
	813	+	}
	814	+	if left := e.leftovers(t); len(left) != 0 {
	815	+	t.Fatalf("leftovers after Reap: %v", left)
	816	+	}
	817	+	if d.slots.InUse() != 0 {
	818	+	t.Fatal("Reap must free the slot of the leaked sandbox")
	819	+	}
	820	+	}
	821	+
	822	+	// C7: Stop from another goroutine aborts a boot in progress at once, instead of
	823	+	// waiting for the boot timeout, and leaves nothing behind.
	824	+	func TestStopAbortsABootInProgress(t *testing.T) {
	825	+	e := newEnv(t, "hang")
	826	+	d, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: time.Minute})
	827	+	if err != nil {
	828	+	t.Fatal(err)
	829	+	}
	830	+	defer func() { _ = d.Close() }()
	831	+	sb, _ := d.New()
	832	+	started := make(chan error, 1)
	833	+	go func() {
	834	+	_, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	835	+	started <- err
	836	+	}()
	837	+	time.Sleep(700 * time.Millisecond) // the fake is now waiting for the executor
	838	+	t0 := time.Now()
	839	+	if err := sb.Stop(context.Background()); err != nil {
	840	+	t.Fatal(err)
	841	+	}
	842	+	select {
	843	+	case err := <-started:
	844	+	if !errors.Is(err, sandbox.ErrAborted) {
	845	+	t.Fatalf("Start must report that Stop aborted it, got %v", err)
	846	+	}
	847	+	case <-time.After(10 * time.Second):
	848	+	t.Fatal("Start was not aborted by Stop")
	849	+	}
	850	+	if time.Since(t0) > 8*time.Second {
	851	+	t.Fatalf("Stop took %v", time.Since(t0))
	852	+	}
	853	+	if left := e.leftovers(t); len(left) != 0 {
	854	+	t.Fatalf("leftovers: %v", left)
	855	+	}
	856	+	// Stop before Start: Start must refuse.
	857	+	sb2, _ := d.New()
	858	+	_ = sb2.Stop(context.Background())
	859	+	if _, err := sb2.Start(context.Background(), sandbox.JobSpec{}); err == nil {
	860	+	t.Fatal("Start after Stop accepted")
	861	+	}
	862	+	}
	863	+
	864	+	// C16: the sandbox exposes its id, which names its files.
	865	+	func TestSandboxIDNamesItsFiles(t *testing.T) {
	866	+	e := newEnv(t, "")
	867	+	d := e.driver(t)
	868	+	sb, _ := d.New()
	869	+	if !sandbox.ValidID(sb.ID()) {
	870	+	t.Fatalf("id %q", sb.ID())
	871	+	}
	872	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	873	+	if err != nil {
	874	+	t.Fatal(err)
	875	+	}
	876	+	runStub(t, conn)
	877	+	defer func() { _ = sb.Stop(context.Background()) }()
	878	+	if _, err := os.Stat(filepath.Join(e.cfg.StateDir, "vm", sb.ID())); err != nil {
	879	+	t.Fatalf("the VM directory must be named after the sandbox id: %v", err)
	880	+	}
	881	+	if d.Isolation() != sandbox.IsolationVM \|\| d.Mode() != sandbox.VM {
	882	+	t.Fatal("isolation")
	883	+	}
	884	+	}
	885	+
	886	+	// S7: cgroup v2 limits, a new PID namespace, and the jailer's own options.
	887	+	func TestJailerArgs(t *testing.T) {
	888	+	args := jailerArgs("/usr/bin/jailer", "/usr/bin/firecracker", "er00112233", 900003, "/var/lib/er/jail",
	889	+	sandbox.JobSpec{CPUs: 4, MemoryMiB: 2048}, nil)
	890	+	line := " " + strings.Join(args, " ") + " "
	891	+	for _, want := range []string{
	892	+	" --id er00112233 ", " --exec-file /usr/bin/firecracker ", " --uid 900003 ", " --gid 900003 ",
	893	+	" --chroot-base-dir /var/lib/er/jail ", " --cgroup-version 2 ", " --new-pid-ns ",
	894	+	" --cgroup memory.max=" + fmt.Sprint((2048+vmmOverheadMiB)<<20) + " ",
	895	+	" --cgroup pids.max=64 ", " --resource-limit no-file=4096 ", " -- ",
	896	+	} {
	897	+	if !strings.Contains(line, want) {
	898	+	t.Errorf("jailer args lack %q: %s", want, line)
	899	+	}
	900	+	}
	901	+	if args[len(args)-1] != "--" {
	902	+	t.Errorf("the jailer separator must come last when there are no extra args: %v", args)
	903	+	}
	904	+	more := jailerArgs("j", "f", "er00112233", 1, "/b", sandbox.JobSpec{CPUs: 1, MemoryMiB: 512}, []string{"--x"})
	905	+	if more[len(more)-2] != "--" \|\| more[len(more)-1] != "--x" {
	906	+	t.Errorf("extra args follow the separator: %v", more)
	907	+	}
	908	+	}
	909	+
	910	+	// S7: sane rate limiters on the scratch disk and both directions of eth0, off
	911	+	// when configured to zero, and only on the writable disk.
	912	+	func TestRateLimitersInTheVMConfig(t *testing.T) {
	913	+	e := newEnv(t, "")
	914	+	d := e.driver(t)
	915	+	sb, _ := d.New()
	916	+	conn, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
	917	+	if err != nil {
	918	+	t.Fatal(err)
	919	+	}
	920	+	runStub(t, conn)
	921	+	defer func() { _ = sb.Stop(context.Background()) }()
	922	+	b, _ := os.ReadFile(e.log)
	923	+	log := string(b)
	924	+	for _, want := range []string{
	925	+	`"rate_limiter":{"bandwidth":{"size":419430400,"one_time_burst":419430400,"refill_time":1000},"ops":{"size":20000,"one_time_burst":20000,"refill_time":1000}}`,
	926	+	`"rx_rate_limiter":{"bandwidth":{"size":125000000,"one_time_burst":125000000,"refill_time":1000}}`,
	927	+	`"tx_rate_limiter":{"bandwidth":{"size":125000000,"one_time_burst":125000000,"refill_time":1000}}`,
	928	+	} {
	929	+	if !strings.Contains(log, want) {
	930	+	t.Errorf("API log lacks %s\n%s", want, log)
	931	+	}
	932	+	}
	933	+	if n := strings.Count(log, `"rate_limiter"`); n != 1 {
	934	+	t.Errorf("only the scratch drive is rate limited, got %d rate_limiter fields", n)
	935	+	}
	936	+	if diskLimiter(0, 0) != nil \|\| netLimiter(0) != nil {
	937	+	t.Error("zero must disable the limiters")
	938	+	}
	939	+	if l := diskLimiter(0, 100); l.Bandwidth != nil \|\| l.Ops == nil {
	940	+	t.Errorf("%+v", l)
	941	+	}
	942	+	}
	943	+
	944	+	// S8: the free-space check warns, and only warns.
	945	+	func TestNewWarnsOnLowFreeSpace(t *testing.T) {
	946	+	e := newEnv(t, "")
	947	+	var buf bytes.Buffer
	948	+	ctx := zerowrap.WithCtx(context.Background(), zerowrap.New(zerowrap.Config{Level: "debug", Format: "json", Output: &buf}))
	949	+	d, err := New(ctx, e.cfg, Options{Run: e.run, Net: e.plumb, FreeSpace: func(string) (int64, error) { return 1, nil }})
	950	+	if err != nil {
	951	+	t.Fatalf("low space must not fail New: %v", err)
	952	+	}
	953	+	defer func() { _ = d.Close() }()
	954	+	if !strings.Contains(buf.String(), "less free space") {
	955	+	t.Fatalf("no warning: %s", buf.String())
	956	+	}
	957	+	}
	958	+
	959	+	// S11: the scratch disk is created without following symlinks and chowned
	960	+	// through its descriptor.
	961	+	func TestMakeScratchRefusesSymlinksAndExistingFiles(t *testing.T) {
	962	+	dir := t.TempDir()
	963	+	target := filepath.Join(dir, "victim")
	964	+	_ = os.WriteFile(target, []byte("precious"), 0o600)
	965	+	link := filepath.Join(dir, "scratch.ext4")
	966	+	if err := os.Symlink(target, link); err != nil {
	967	+	t.Fatal(err)
	968	+	}
	969	+	run := &cmdrun.Fake{}
	970	+	if err := disk.CreateScratch(context.Background(), run, link, 8, -1); err == nil {
	971	+	t.Fatal("a planted symlink must be refused")
	972	+	}
	973	+	if b, _ := os.ReadFile(target); string(b) != "precious" {
	974	+	t.Fatalf("the symlink target was modified: %q", b)
	975	+	}
	976	+	if len(run.Calls) != 0 {
	977	+	t.Fatalf("mkfs must not run: %v", run.Lines())
	978	+	}
	979	+	ok := filepath.Join(dir, "fresh.ext4")
	980	+	if err := disk.CreateScratch(context.Background(), run, ok, 8, os.Getuid()); err != nil {
	981	+	t.Fatal(err)
	982	+	}
	983	+	if st, err := os.Stat(ok); err != nil \|\| st.Size() != 8<<20 \|\| st.Mode().Perm() != 0o600 {
	984	+	t.Fatalf("%v %v", st, err)
	985	+	}
	986	+	if err := disk.CreateScratch(context.Background(), run, ok, 8, -1); err == nil {
	987	+	t.Fatal("an existing file must be refused (O_EXCL)")
	988	+	}
	989	+	}
	990	+
	991	+	func TestOwnUsesLchown(t *testing.T) {
	992	+	dir := t.TempDir()
	993	+	l := newLayout(dir, "firecracker", "er00112233", true, os.Getuid())
	994	+	_ = os.MkdirAll(l.root, 0o700)
	995	+	target := filepath.Join(dir, "victim")
	996	+	_ = os.WriteFile(target, nil, 0o600)
	997	+	if err := os.Symlink(target, l.host("link")); err != nil {
	998	+	t.Fatal(err)
	999	+	}
	1000	+	if err := l.own("link"); err != nil {
	1001	+	t.Fatalf("Lchown on a link must act on the link itself: %v", err)
	1002	+	}
	1003	+	if err := l.own("missing"); err == nil {
	1004	+	t.Fatal("a missing file is an error")
	1005	+	}
	1006	+	if u := newLayout(dir, "f", "er00112233", false, 1); u.own("anything") != nil {
	1007	+	t.Fatal("no-op without jailer")
	1008	+	}
	1009	+	}
	1010	+
	1011	+	func TestRemoveCgroup(t *testing.T) {
	1012	+	root := t.TempDir()
	1013	+	_ = os.MkdirAll(filepath.Join(root, "firecracker", "er00112233"), 0o755)
	1014	+	if err := removeCgroup(root, "firecracker", "er00112233"); err != nil {
	1015	+	t.Fatal(err)
	1016	+	}
	1017	+	if err := removeCgroup(root, "firecracker", "er00112233"); err != nil {
	1018	+	t.Fatalf("a missing cgroup is fine: %v", err)
	1019	+	}
	1020	+	_ = os.MkdirAll(filepath.Join(root, "firecracker", "er44556677", "child"), 0o755)
	1021	+	if err := removeCgroup(root, "firecracker", "er44556677"); err == nil {
	1022	+	t.Fatal("a non-empty cgroup cannot be removed and must be reported")
	1023	+	}
	1024	+	}

M main.go +66 -4


@@ -3,21 +3,83 @@
3	3		// Command ermite-runner runs CI jobs for an ermite server. Each job runs in
4	4		// its own Firecracker micro-VM, or in its own container when the host has no
5	5		// KVM, and executes workflows compatible with GitHub Actions.
	6	+	//
	7	+	// Subcommands: version; init and exec run inside a sandbox (guest PID 1, and
	8	+	// the executor); sandbox-check runs one stub job through the selected isolation
	9	+	// mode and prints its output (phase D of the CI-1 plan: the host loop that
	10	+	// fetches jobs from ermite comes later).
6	11		package main
7	12
8	13		import (
	14	+	"context"
9	15		"fmt"
10	16		"os"
	17	+	"os/signal"
	18	+	"syscall"
	19	+
	20	+	"github.com/bnema/zerowrap"
	21	+
	22	+	"bnema.dev/ermite-runner/internal/guest"
11	23		)
12	24
13	25		// version is set at build time with -ldflags "-X main.version=...".
14	26		var version = "dev"
15	27
16	28		func main() {
17		-	if len(os.Args) > 1 && os.Args[1] == "version" {
	29	+	os.Exit(run(os.Args[1:]))
	30	+	}
	31	+
	32	+	func run(args []string) int {
	33	+	if len(args) == 0 {
	34	+	usage()
	35	+	return 2
	36	+	}
	37	+	log := zerowrap.New(zerowrap.Config{Level: envOr("ERMITE_RUNNER_LOG_LEVEL", "info"), Format: envOr("ERMITE_RUNNER_LOG_FORMAT", "console")})
	38	+	ctx := zerowrap.WithCtx(context.Background(), log)
	39	+	ctx, stop := signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)
	40	+	defer stop()
	41	+
	42	+	switch args[0] {
	43	+	case "version":
18	44		fmt.Println("ermite-runner", version)
19		-	return
	45	+	return 0
	46	+	case "init":
	47	+	return guest.Init(ctx, guest.InitConfig{Version: version})
	48	+	case "exec":
	49	+	return execCmd(ctx)
	50	+	case "sandbox-check":
	51	+	return sandboxCheck(ctx, args[1:])
	52	+	}
	53	+	usage()
	54	+	return 2
	55	+	}
	56	+
	57	+	func usage() {
	58	+	fmt.Fprintln(os.Stderr, "usage: ermite-runner version \| sandbox-check [script]\n (inside a sandbox: ermite-runner init \| exec)")
	59	+	}
	60	+
	61	+	func envOr(key, def string) string {
	62	+	if v := os.Getenv(key); v != "" {
	63	+	return v
	64	+	}
	65	+	return def
	66	+	}
	67	+
	68	+	// execCmd is `ermite-runner exec`, the stub executor.
	69	+	func execCmd(ctx context.Context) int {
	70	+	log := zerowrap.FromCtx(ctx)
	71	+	target, err := guest.ParseTarget(os.Getenv("ERMITE_CONNECT"))
	72	+	if err != nil {
	73	+	log.Error().Err(err).Msg("exec")
	74	+	return 1
	75	+	}
	76	+	code, err := guest.Exec(ctx, guest.ExecConfig{
	77	+	Target: target,
	78	+	Sandbox: envOr("ERMITE_SANDBOX", "container"),
	79	+	Version: envOr("ERMITE_VERSION", version),
	80	+	})
	81	+	if err != nil {
	82	+	log.Error().Err(err).Msg("exec")
20	83		}
21		-	fmt.Fprintln(os.Stderr, "usage: ermite-runner version")
22		-	os.Exit(2)
	84	+	return code
23	85		}

A sandbox_check.go +81 -0

@@ -0,0 +1,81 @@
	1	+	// SPDX-License-Identifier: MIT
	2	+
	3	+	package main
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"fmt"
	9	+	"os"
	10	+	"strings"
	11	+	"time"
	12	+
	13	+	"github.com/bnema/zerowrap"
	14	+
	15	+	"bnema.dev/ermite-runner/internal/execstream"
	16	+	"bnema.dev/ermite-runner/internal/execstream/execpb"
	17	+	"bnema.dev/ermite-runner/internal/sandbox"
	18	+	"bnema.dev/ermite-runner/internal/sandbox/drivers"
	19	+	)
	20	+
	21	+	// sandboxCheck runs one stub job in the sandbox chosen by
	22	+	// ERMITE_RUNNER_ISOLATION: reap leftovers, start, stream the output to stdout
	23	+	// and stderr, stop. Its exit code is the script's. It is the manual counterpart
	24	+	// of the integration tests and the smoke test of an installation.
	25	+	func sandboxCheck(ctx context.Context, args []string) int {
	26	+	log := zerowrap.FromCtx(ctx)
	27	+	cfg, err := sandbox.LoadConfig(os.Getenv)
	28	+	if err != nil {
	29	+	fmt.Fprintln(os.Stderr, "ermite-runner:", err)
	30	+	return 2
	31	+	}
	32	+	drv, err := sandbox.Select(ctx, cfg, drivers.Factories(), sandbox.Prober{})
	33	+	if err != nil {
	34	+	fmt.Fprintln(os.Stderr, "ermite-runner:", err)
	35	+	return 1
	36	+	}
	37	+	defer func() { _ = drv.Close() }()
	38	+	if err := drv.Reap(ctx); err != nil {
	39	+	log.Warn().Err(err).Msg("reaping leftovers of a previous run")
	40	+	}
	41	+	sb, err := drv.New()
	42	+	if err != nil {
	43	+	fmt.Fprintln(os.Stderr, "ermite-runner:", err)
	44	+	return 1
	45	+	}
	46	+	// Stop runs even when the job context was cancelled.
	47	+	defer func() {
	48	+	sctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 60*time.Second)
	49	+	defer cancel()
	50	+	if err := sb.Stop(sctx); err != nil {
	51	+	fmt.Fprintln(os.Stderr, "ermite-runner: cleanup:", err)
	52	+	if errors.Is(err, sandbox.ErrLeaked) {
	53	+	fmt.Fprintln(os.Stderr, "ermite-runner: something leaked: run again (the start reaps) or check the state directory")
	54	+	}
	55	+	}
	56	+	}()
	57	+	conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: "sandbox-check"})
	58	+	if err != nil {
	59	+	fmt.Fprintln(os.Stderr, "ermite-runner:", err)
	60	+	return 1
	61	+	}
	62	+	// Ctrl-C / SIGTERM cancel the job: the executor escalates by itself.
	63	+	cancel := make(chan *execpb.Cancel, 1)
	64	+	stop := context.AfterFunc(ctx, func() { cancel <- &execpb.Cancel{Reason: "interrupted"} })
	65	+	defer stop()
	66	+	_, exit, err := execstream.Drive(conn, &execpb.Run{Script: strings.Join(args, " ")}, cancel, func(o *execpb.Output) {
	67	+	f := os.Stdout
	68	+	if o.Stream == execpb.StreamKind_STREAM_KIND_STDERR {
	69	+	f = os.Stderr
	70	+	}
	71	+	_, _ = f.Write(o.Data)
	72	+	})
	73	+	if err != nil {
	74	+	fmt.Fprintln(os.Stderr, "ermite-runner:", err)
	75	+	return 1
	76	+	}
	77	+	if exit.Error != "" {
	78	+	fmt.Fprintln(os.Stderr, "ermite-runner: executor:", exit.Error)
	79	+	}
	80	+	return int(exit.Code)
	81	+	}

A scripts/build-image.sh +66 -0

@@ -0,0 +1,66 @@
	1	+	#!/usr/bin/env bash
	2	+	# SPDX-License-Identifier: MIT
	3	+	#
	4	+	# Build the ermite-runner-ubuntu-24.04 image as an OCI image (container
	5	+	# isolation) and as a sparse ext4 base image (vm isolation). No root needed:
	6	+	# the OCI build uses docker or podman (rootless is fine), the ext4 image is
	7	+	# written by `mkfs.ext4 -d` from the exported rootfs tarball.
	8	+	#
	9	+	# scripts/build-image.sh [OUT_DIR]
	10	+	#
	11	+	# Environment:
	12	+	# ENGINE docker or podman (default: first found)
	13	+	# IMAGE_TAG OCI tag (default ermite-runner-ubuntu-24.04:latest, the
	14	+	# default of ERMITE_RUNNER_IMAGE in container mode)
	15	+	# EXT4_SLACK free space added to the base image in MiB (default 512)
	16	+	#
	17	+	# Output: OUT_DIR/base.ext4 (read-only shared base disk; its UUID and size are
	18	+	# stable per build) and the OCI image IMAGE_TAG in the engine's store.
	19	+	set -euo pipefail
	20	+
	21	+	root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
	22	+	out=${1:-$root/tmp/image}
	23	+	tag=${IMAGE_TAG:-ermite-runner-ubuntu-24.04:latest}
	24	+	slack=${EXT4_SLACK:-512}
	25	+	engine=${ENGINE:-}
	26	+
	27	+	if [ -z "$engine" ]; then
	28	+	for e in docker podman; do
	29	+	if command -v "$e" >/dev/null 2>&1; then engine=$e; break; fi
	30	+	done
	31	+	fi
	32	+	[ -n "$engine" ] \|\| { echo "need docker or podman" >&2; exit 1; }
	33	+	command -v mkfs.ext4 >/dev/null \|\| { echo "need mkfs.ext4 (e2fsprogs >= 1.47.1 for tarball input)" >&2; exit 1; }
	34	+
	35	+	# mkfs.ext4 -d with a tarball needs e2fsprogs 1.47.1; older releases only take a
	36	+	# directory. mke2fs prints its version on stderr and may exit non-zero on -V.
	37	+	ver=$(mkfs.ext4 -V 2>&1 \| sed -n 's/^mke2fs $[0-9]\.[0-9]\.[0-9]$./\1/p' \| head -n1)
	38	+	[ -n "$ver" ] \|\| ver=$(mkfs.ext4 -V 2>&1 \| sed -n 's/^mke2fs $[0-9]\.[0-9]$.*/\1.0/p' \| head -n1)
	39	+	if [ -z "$ver" ] \|\| [ "$(printf '%s\n1.47.1\n' "$ver" \| sort -V \| head -n1)" != "1.47.1" ]; then
	40	+	echo "mkfs.ext4 is e2fsprogs ${ver:-of unknown version}, need >= 1.47.1 (mkfs.ext4 -d with a tarball)" >&2
	41	+	exit 1
	42	+	fi
	43	+
	44	+	mkdir -p "$out"
	45	+	out=$(cd "$out" && pwd)
	46	+	"$engine" build -t "$tag" "$root/images/ubuntu-24.04"
	47	+
	48	+	cid=$("$engine" create "$tag")
	49	+	tmp=$(mktemp -d)
	50	+	cleanup() { "$engine" rm -f "$cid" >/dev/null 2>&1 \|\| true; rm -rf "$tmp"; }
	51	+	trap cleanup EXIT
	52	+
	53	+	"$engine" export "$cid" > "$tmp/rootfs.tar"
	54	+	bytes=$(stat -c %s "$tmp/rootfs.tar")
	55	+	# ext4 metadata overhead is about 5 %; the slack leaves room for the guest's
	56	+	# own writes into the (read-only mounted) base only when someone remounts it.
	57	+	mib=$(( bytes / 1024 / 1024 * 110 / 100 + slack ))
	58	+
	59	+	img="$out/base.ext4.part"
	60	+	rm -f "$img"
	61	+	truncate -s "${mib}M" "$img"
	62	+	# No journal: the base is mounted read-only and shared between VMs.
	63	+	mkfs.ext4 -q -F -L ermite-base -O ^has_journal -E lazy_itable_init=1,nodiscard \
	64	+	-d "$tmp/rootfs.tar" "$img"
	65	+	mv "$img" "$out/base.ext4"
	66	+	echo "base image: $out/base.ext4 (${mib} MiB sparse), OCI image: $tag"

A scripts/build-kernel.sh +79 -0

@@ -0,0 +1,79 @@
	1	+	#!/usr/bin/env bash
	2	+	# SPDX-License-Identifier: MIT
	3	+	#
	4	+	# Build the Firecracker guest kernel (Linux 6.18 LTS) from kernel.org.
	5	+	#
	6	+	# scripts/build-kernel.sh [OUT_DIR]
	7	+	#
	8	+	# Environment:
	9	+	# KERNEL_VERSION version to build (default 6.18.56)
	10	+	# ARCH_NAME x86_64 (default: uname -m) or arm64 (aarch64)
	11	+	# WORK_DIR download and build directory (default tmp/kernel, gitignored)
	12	+	# JOBS make parallelism (default: nproc)
	13	+	#
	14	+	# Output: OUT_DIR/vmlinux (x86_64 ELF, accepted by Firecracker) or
	15	+	# OUT_DIR/Image (arm64), plus OUT_DIR/kernel.config. No root needed. The tarball
	16	+	# is verified against the sha256sums.asc published next to it on kernel.org
	17	+	# (integrity over TLS; the PGP signature is not checked, see README).
	18	+	set -euo pipefail
	19	+
	20	+	root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
	21	+	version=${KERNEL_VERSION:-6.18.56}
	22	+	major=${version%%.*}
	23	+	work=${WORK_DIR:-$root/tmp/kernel}
	24	+	out=${1:-$root/tmp/image}
	25	+	jobs=${JOBS:-$(nproc)}
	26	+	arch_name=${ARCH_NAME:-$(uname -m)}
	27	+
	28	+	case "$arch_name" in
	29	+	x86_64\|amd64) karch=x86; arch_frag=x86_64; target=vmlinux; artifact=vmlinux ;;
	30	+	aarch64\|arm64) karch=arm64; arch_frag=arm64; target=Image; artifact=arch/arm64/boot/Image ;;
	31	+	*) echo "unsupported architecture: $arch_name" >&2; exit 2 ;;
	32	+	esac
	33	+
	34	+	mkdir -p "$work" "$out"
	35	+	work=$(cd "$work" && pwd)
	36	+	out=$(cd "$out" && pwd)
	37	+	tarball="linux-$version.tar.xz"
	38	+	base="https://cdn.kernel.org/pub/linux/kernel/v${major}.x"
	39	+
	40	+	if [ ! -f "$work/$tarball" ]; then
	41	+	curl -fsSL -o "$work/$tarball.part" "$base/$tarball"
	42	+	mv "$work/$tarball.part" "$work/$tarball"
	43	+	fi
	44	+	if [ ! -f "$work/sha256sums.asc" ] \|\| ! grep -qF " $tarball" "$work/sha256sums.asc"; then
	45	+	curl -fsSL -o "$work/sha256sums.asc" "$base/sha256sums.asc"
	46	+	fi
	47	+	want=$(awk -v f="$tarball" '$2 == f {print $1; exit}' "$work/sha256sums.asc")
	48	+	have=$(sha256sum "$work/$tarball" \| awk '{print $1}')
	49	+	if [ -z "$want" ] \|\| [ "$want" != "$have" ]; then
	50	+	echo "checksum mismatch for $tarball (want '$want', have '$have')" >&2
	51	+	exit 1
	52	+	fi
	53	+
	54	+	src="$work/linux-$version"
	55	+	[ -d "$src" ] \|\| tar -C "$work" -xf "$work/$tarball"
	56	+
	57	+	cd "$src"
	58	+	export ARCH=$karch
	59	+	make tinyconfig >/dev/null
	60	+	scripts/kconfig/merge_config.sh -m .config \
	61	+	"$root/images/kernel/config-6.18" "$root/images/kernel/config-6.18.$arch_frag" >/dev/null
	62	+	make olddefconfig >/dev/null
	63	+
	64	+	# Fail if an option of the fragments was dropped by a missing dependency.
	65	+	missing=0
	66	+	while IFS== read -r key val; do
	67	+	# Whole-line, fixed-string matches: option names contain no regex, but
	68	+	# CONFIG_FOO must never match CONFIG_FOO_BAR=y.
	69	+	if grep -qxF "$key=$val" .config; then continue; fi
	70	+	if [ "$val" = n ] && ! grep -q "^${key}=" .config; then continue; fi
	71	+	echo "kernel option not applied: $key=$val" >&2
	72	+	missing=1
	73	+	done < <(cat "$root/images/kernel/config-6.18" "$root/images/kernel/config-6.18.$arch_frag" \| grep -E '^CONFIG_[A-Z0-9_]+=')
	74	+	[ "$missing" = 0 ] \|\| exit 1
	75	+
	76	+	make -j"$jobs" "$target"
	77	+	cp "$artifact" "$out/$(basename "$artifact")"
	78	+	cp .config "$out/kernel.config"
	79	+	echo "kernel: $out/$(basename "$artifact")"

A scripts/it-vm-userns.sh +46 -0

@@ -0,0 +1,46 @@
	1	+	#!/usr/bin/env bash
	2	+	# SPDX-License-Identifier: MIT
	3	+	#
	4	+	# Run the VM integration tests WITHOUT root: inside a user + network namespace
	5	+	# the process is "root" with CAP_NET_ADMIN over a private network stack, which is
	6	+	# enough for TAP devices, nftables and the DNS forwarder on the TAP address.
	7	+	# Only /dev/kvm access (group kvm) and the Firecracker binary are needed. The
	8	+	# jailer needs real root, so it is switched off here (see README: this tests the
	9	+	# whole driver except the jailer's chroot, uid drop and cgroups).
	10	+	#
	11	+	# scripts/it-vm-userns.sh [go test args]
	12	+	#
	13	+	# Environment: FIRECRACKER (default: firecracker in PATH), IMAGE_DIR (directory
	14	+	# with vmlinux and base.ext4, default tmp/image). Outbound internet from the
	15	+	# private namespace is not available: the tests do not need it.
	16	+	set -euo pipefail
	17	+	root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
	18	+	cd "$root"
	19	+	fc=${FIRECRACKER:-$(command -v firecracker \|\| true)}
	20	+	[ -x "${fc:-}" ] \|\| { echo "firecracker not found: set FIRECRACKER" >&2; exit 1; }
	21	+	img=${IMAGE_DIR:-$root/tmp/image}
	22	+	# Two separate namespaces: the VM tests leave TAP devices behind when they fail,
	23	+	# and the packet-level firewall test must start from a clean network stack.
	24	+	rc=0
	25	+	unshare --user --map-root-user --net bash -c '
	26	+	ip link set lo up
	27	+	# A default route, so that forwarded packets reach the firewall instead of
	28	+	# being answered "network unreachable" by the routing table: the drops the
	29	+	# network test sees are then the nftables policy.
	30	+	ip link add dummy0 type dummy
	31	+	ip addr add 192.0.2.1/24 dev dummy0
	32	+	ip link set dummy0 up
	33	+	ip route add default via 192.0.2.254 dev dummy0
	34	+	export GOWORK=off ERMITE_RUNNER_IT_VM=1 \
	35	+	ERMITE_RUNNER_ISOLATION=vm ERMITE_RUNNER_IMAGE="$1" \
	36	+	ERMITE_RUNNER_FIRECRACKER="$2" ERMITE_RUNNER_JAILER_MODE=off \
	37	+	ERMITE_RUNNER_LOCK_FILE="${TMPDIR:-/tmp}/ermite-runner-it-$$.lock"
	38	+	shift 2
	39	+	exec go test -count=1 -v -timeout 10m ./internal/sandbox/integration "$@"
	40	+	' _ "$img" "$fc" "$@" \|\| rc=$?
	41	+	# The packet-level firewall test needs only a namespace, not Firecracker.
	42	+	unshare --user --map-root-user --net bash -c '
	43	+	ip link set lo up
	44	+	GOWORK=off ERMITE_RUNNER_IT_NET=1 exec go test -count=1 -v -timeout 10m -run TwoGuests ./internal/netpolicy
	45	+	' \|\| rc=$?
	46	+	exit $rc