service.md Raw History Folder

Service

contrib/ermite-runner.service is a systemd unit. For the commands to install it, see the README.

Privileges

As a host service, the runner runs as root. It needs TAP devices, nftables, loop mounts, the jailer and Docker.
To run without root on the host, use the unprivileged container instead.
Jobs never run as root on the host. They run as user runner in a microVM or a container.

Registration

Register in the same environment as the service, so register and run agree on the URL, name, labels, isolation and state directory.
systemd-run with the same EnvironmentFile does this.
register picks the isolation mode like run will. It sends the name, labels, version, isolation and architecture.

Logs

The runner writes to standard error, which journald collects.
Set ERMITE_RUNNER_LOG_FORMAT=json under journald. Every line about a job has job_id, run_id and attempt.
warn marks what is retried. error marks what is refused or leaked.
ERMITE_RUNNER_LOG_LEVEL=debug adds detail.
Tokens are never logged.

Stopping

systemctl stop sends SIGTERM. The runner:

Cancels running jobs with runner_shutdown.
Reports them to the server.
Removes their sandboxes.

Budget with the defaults:

Step	Time
Executor stops its steps and runs post-steps	30 s
Destroy the sandbox	30 s
Flush the log and report	30 s
Total	90 s

Jobs stop in parallel. The total does not grow with ERMITE_RUNNER_CAPACITY.
The unit sets TimeoutStopSec=120.
A job whose first heartbeat is pending at SIGTERM is not started. The server gives it to another runner after 30 s.
A crash or power cut leaves sandboxes behind. The next start reaps them.

Unit hardening

The unit sets only options that keep it in the host's mount namespace:

NoNewPrivileges
ProtectClock
RestrictSUIDSGID
LockPersonality
RestrictRealtime
SystemCallArchitectures=native
RestrictAddressFamilies (UNIX, INET, INET6, NETLINK, VSOCK)

Do not add:

Options that give the unit its own mount namespace: PrivateTmp, ProtectHome, ProtectSystem, ProtectKernelModules, ProtectKernelLogs, ProtectKernelTunables, ProtectControlGroups, PrivateDevices, ReadOnlyPaths, ReadWritePaths, InaccessiblePaths, TemporaryFileSystem, BindPaths. In container mode the runner loop-mounts the job disk and the Docker daemon, in the host namespace, must see it. A test fails if the unit sets one of them.
PrivateNetwork, RestrictNamespaces, CapabilityBoundingSet, SystemCallFilter: the runner creates network devices, mounts filesystems and hands capabilities to the jailer and Docker.
MemoryDenyWriteExecute: Firecracker maps guest memory writable and executable.
Delegate and UMask: not needed.

The security boundary of a job is its microVM or container, not the unit.

Job timing

The job finished line carries integer milliseconds (all *_ms fields). Phases that did not run are omitted, and the driver (sb_*) timings appear only when the sandbox started. resolve_ms includes archive_miss_ms, and both are part of run_ms.

Fields	Meaning
`ack_ms`	Assignment received to successful first heartbeat acknowledgement
`sandbox_ms`	Host `startSandbox` call
`run_ms`	Executor session (`drive`)
`stop_ms`	Sandbox `Stop` call
`complete_ms`	Completion call, including retries
`total_ms`	Assignment received to this log line, including log flushing
`resolve_calls`, `resolve_ms`	Number and summed duration of action resolution requests
`archive_misses`, `archive_miss_ms`, `archive_miss_bytes`	Archive misses and their summed fetch duration and bytes
`sb_tap_ms`, `sb_net_ms` (VM)	TAP creation and the rest of network setup
`sb_vmm_ms`, `sb_files_ms`, `sb_scratch_ms` (VM)	VMM launch/API readiness, shared files excluding scratch, scratch creation
`sb_vsock_ms`, `sb_configure_ms`, `sb_connect_ms` (VM)	Executor/cache sockets, VM configuration/start, executor connection wait
`sb_socket_ms`, `sb_cache_ms`, `sb_scratch_ms` (container)	Executor socket, cache socket, scratch setup
`sb_net_ms`, `sb_container_ms`, `sb_connect_ms` (container)	Network setup, container creation/start, executor connection wait
`guest_kernel_ms`, `guest_setup_ms` (VM only)	Kernel uptime at init entry, VM filesystem/system setup
`guest_containerd_ms`, `guest_dockerd_ms`	Containerd start/readiness, dockerd spawn through readiness wait

Guest boot overlaps the driver's executor connection wait: do not sum those fields with sandbox_ms. Driver container running and micro-VM running lines also carry unprefixed phase fields and total_ms for their boot sequence. The action archive cache line carries verify_ms for integrity verification; a miss also carries fetch_ms and bytes. Archive verification is never skipped.

source

1# Service
2
3`contrib/ermite-runner.service` is a systemd unit. For the commands to install it, see the [README](../README.md#register-and-run).
4
5## Privileges
6
7- As a host service, the runner runs as root. It needs TAP devices, nftables, loop mounts, the jailer and Docker.
8- To run without root on the host, use the [unprivileged container](container.md) instead.
9- Jobs never run as root on the host. They run as user `runner` in a microVM or a container.
10
11## Registration
12
13- Register in the same environment as the service, so `register` and `run` agree on the URL, name, labels, isolation and state directory.
14- `systemd-run` with the same `EnvironmentFile` does this.
15- `register` picks the isolation mode like `run` will. It sends the name, labels, version, isolation and architecture.
16
17## Logs
18
19- The runner writes to standard error, which journald collects.
20- Set `ERMITE_RUNNER_LOG_FORMAT=json` under journald. Every line about a job has `job_id`, `run_id` and `attempt`.
21- `warn` marks what is retried. `error` marks what is refused or leaked.
22- `ERMITE_RUNNER_LOG_LEVEL=debug` adds detail.
23- Tokens are never logged.
24
25## Stopping
26
27`systemctl stop` sends SIGTERM. The runner:
28
291. Cancels running jobs with `runner_shutdown`.
302. Reports them to the server.
313. Removes their sandboxes.
32
33Budget with the defaults:
34
35| Step | Time |
36| --- | --- |
37| Executor stops its steps and runs post-steps | 30 s |
38| Destroy the sandbox | 30 s |
39| Flush the log and report | 30 s |
40| **Total** | **90 s** |
41
42- Jobs stop in parallel. The total does not grow with `ERMITE_RUNNER_CAPACITY`.
43- The unit sets `TimeoutStopSec=120`.
44- A job whose first heartbeat is pending at SIGTERM is not started. The server gives it to another runner after 30 s.
45- A crash or power cut leaves sandboxes behind. The next start reaps them.
46
47## Unit hardening
48
49The unit sets only options that keep it in the host's mount namespace:
50
51- `NoNewPrivileges`
52- `ProtectClock`
53- `RestrictSUIDSGID`
54- `LockPersonality`
55- `RestrictRealtime`
56- `SystemCallArchitectures=native`
57- `RestrictAddressFamilies` (UNIX, INET, INET6, NETLINK, VSOCK)
58
59Do not add:
60
61- Options that give the unit its own mount namespace: `PrivateTmp`, `ProtectHome`, `ProtectSystem`, `ProtectKernelModules`, `ProtectKernelLogs`, `ProtectKernelTunables`, `ProtectControlGroups`, `PrivateDevices`, `ReadOnlyPaths`, `ReadWritePaths`, `InaccessiblePaths`, `TemporaryFileSystem`, `BindPaths`. In `container` mode the runner loop-mounts the job disk and the Docker daemon, in the host namespace, must see it. A test fails if the unit sets one of them.
62- `PrivateNetwork`, `RestrictNamespaces`, `CapabilityBoundingSet`, `SystemCallFilter`: the runner creates network devices, mounts filesystems and hands capabilities to the jailer and Docker.
63- `MemoryDenyWriteExecute`: Firecracker maps guest memory writable and executable.
64- `Delegate` and `UMask`: not needed.
65
66The security boundary of a job is its microVM or container, not the unit.
67
68## Job timing
69
70The `job finished` line carries integer milliseconds (all `*_ms` fields).
71Phases that did not run are omitted, and the driver (`sb_*`) timings appear only
72when the sandbox started. `resolve_ms` includes `archive_miss_ms`, and both are
73part of `run_ms`.
74
75| Fields | Meaning |
76| --- | --- |
77| `ack_ms` | Assignment received to successful first heartbeat acknowledgement |
78| `sandbox_ms` | Host `startSandbox` call |
79| `run_ms` | Executor session (`drive`) |
80| `stop_ms` | Sandbox `Stop` call |
81| `complete_ms` | Completion call, including retries |
82| `total_ms` | Assignment received to this log line, including log flushing |
83| `resolve_calls`, `resolve_ms` | Number and summed duration of action resolution requests |
84| `archive_misses`, `archive_miss_ms`, `archive_miss_bytes` | Archive misses and their summed fetch duration and bytes |
85| `sb_tap_ms`, `sb_net_ms` (VM) | TAP creation and the rest of network setup |
86| `sb_vmm_ms`, `sb_files_ms`, `sb_scratch_ms` (VM) | VMM launch/API readiness, shared files excluding scratch, scratch creation |
87| `sb_vsock_ms`, `sb_configure_ms`, `sb_connect_ms` (VM) | Executor/cache sockets, VM configuration/start, executor connection wait |
88| `sb_socket_ms`, `sb_cache_ms`, `sb_scratch_ms` (container) | Executor socket, cache socket, scratch setup |
89| `sb_net_ms`, `sb_container_ms`, `sb_connect_ms` (container) | Network setup, container creation/start, executor connection wait |
90| `guest_kernel_ms`, `guest_setup_ms` (VM only) | Kernel uptime at init entry, VM filesystem/system setup |
91| `guest_containerd_ms`, `guest_dockerd_ms` | Containerd start/readiness, dockerd spawn through readiness wait |
92
93Guest boot overlaps the driver's executor connection wait: do not sum those
94fields with `sandbox_ms`. Driver `container running` and `micro-VM running`
95lines also carry unprefixed phase fields and `total_ms` for their boot sequence.
96The `action archive cache` line carries `verify_ms` for integrity verification;
97a miss also carries `fetch_ms` and `bytes`. Archive verification is never skipped.

service.md RawHistoryFolder