1# Development
2
3## Build and check
4
5```sh
6CGO_ENABLED=0 go build -trimpath -o ermite-runner .
7GOWORK=off go vet ./... && GOWORK=off go test -race ./...
8```
9
10- The binary must be static: the sandbox has no libc of ours.
11- Unit tests need nothing special. The drivers are tested with a fake Firecracker and a fake Docker engine.
12- Protobuf: `go generate ./internal/execstream`. `buf` and `protoc-gen-go` are pinned as `tool` directives in `go.mod`. CI lints the proto and fails when the committed generated code differs.
13- `ermite-runner sandbox-check [script]` runs one stub job in the selected mode. It is the smoke test of an installation.
14
15## Integration tests
16
17They are opt-in. They run the real stub executor. Output and exit codes come back, and nothing survives 10 jobs, including a killed one and a runner crash.
18
19| Variable | Enables |
20| --- | --- |
21| `ERMITE_RUNNER_IT_VM=1` | VM tests. Need `/dev/kvm`, Firecracker, the kernel and the base image |
22| `ERMITE_RUNNER_IT_CONTAINER=1` | Container tests. Need a rootful Docker daemon, root and the OCI image |
23| `ERMITE_RUNNER_IT_E2E=1` | End-to-end job tests in `internal/e2e` |
24| `ERMITE_RUNNER_IT_NET=1` | Packet-level firewall test |
25| `ERMITE_RUNNER_IT_DOCKER_API=1` | Docker Engine calls against any daemon, including rootless |
26| `ERMITE_RUNNER_IT_REGISTRY=1` | Live Docker Hub pulls through the registry proxy with `curl` (`TestDockerHubCurl`). Needs internet access |
27| `ERMITE_RUNNER_IT_NODE_DIR` | Directory with `node-v24.21.0-linux-x64.tar.gz`, for offline hosts |
28| `ERMITE_RUNNER_TEST_IMAGE` | Image used by the container and end-to-end tests |
29
30Build the kernel and images first: see [images](images.md).
31
32### VM
33
34```sh
35# Unprivileged, in a user and network namespace. The jailer is off.
36FIRECRACKER=/path/to/firecracker IMAGE_DIR=$PWD/tmp/image scripts/it-vm-userns.sh
37
38# Privileged, with the jailer (the production configuration)
39sudo env GOWORK=off ERMITE_RUNNER_IT_VM=1 ERMITE_RUNNER_ISOLATION=vm \
40  ERMITE_RUNNER_IMAGE=$PWD/tmp/image ERMITE_RUNNER_STATE_DIR=/var/lib/ermite-runner-it \
41  "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run VM
42```
43
44### Container
45
46```sh
47sudo env GOWORK=off ERMITE_RUNNER_IT_CONTAINER=1 ERMITE_RUNNER_IMAGE=ermite-runner-ubuntu-26.04:latest \
48  DOCKER_HOST=unix:///var/run/docker.sock "$(command -v go)" test -count=1 -v ./internal/sandbox/integration -run Container
49```
50
51### End to end
52
53The real executor runs in the real image, started by the container driver. It also works with a rootless Docker daemon, where `mount` and `umount` are replaced by stand-ins.
54
55```sh
56IMAGE_TAG=ermite-runner-ubuntu-26.04:dev scripts/build-image.sh tmp/image
57ERMITE_RUNNER_IT_E2E=1 ERMITE_RUNNER_TEST_IMAGE=ermite-runner-ubuntu-26.04:dev GOWORK=off go test -race -count=1 ./internal/e2e
58```
59
60- `TestChainCorpus` runs jobs of the ermite-conformance corpus through a fake server and compares the result with recordings made on github.com. It reads `ERMITE_CONFORMANCE_DIR` (default: a sibling checkout of ermite-conformance) and never writes to it.
61
62## Notes
63
64- **Docker API.** Production `container` mode supports Docker Engine and sysbox. Podman's compatibility API is not a drop-in: it cannot disable DNS on the network that the runner creates.
65- **Jailer.** The jailer needs `--chroot-base-dir` to exist. The driver creates `<state dir>/jail`. It leaves `<state dir>/jail/<exec name>` and `/sys/fs/cgroup/firecracker` behind.
66- **Host firewalls.** A firewall that rejects `53` on `ertap*` or `erbr*` breaks guest DNS. The symptom is `dns silent` in `TestVMNetworkPolicy`.
67
68## Measuring start-up on a small host
69
70Build and register the [runner container](container.md) as usual, then constrain
71its CPU budget to reproduce a small host without changing the boot sequence:
72
73```sh
74docker run -d --name runner-timing --cpus=1 \
75  --device /dev/kvm --device /dev/net/tun --cap-add NET_ADMIN \
76  -v runner-timing-state:/var/lib/ermite-runner \
77  --env-file runner.env -e ERMITE_RUNNER_LOG_FORMAT=json ermite-runner:<tag>
78```
79
80Use a registered state volume or supply the registration settings in
81`runner.env`; never put tokens in shared shell history. Podman accepts the same
82CPU limit (device access may need the CDI setup described in the container
83guide). Submit representative jobs, then extract completed-job measurements:
84
85```sh
86docker logs runner-timing 2>&1 | jq 'select(.message=="job finished")'
87```
88
89Compare `sandbox_ms`, the `sb_*` stages and `guest_*` boot stages across jobs.
90`run_ms` includes action downloads and step execution; archive misses have their
91own counters. See [Job timing](service.md#job-timing) for the full field table.