detached 93b85fe

Merge pull request 'feat(netpolicy): opt-in guest access to the forge for staging hosts' (#9) from feat/guest-forge-access into main

commit
93b85fe868fd84c3df3583755585e7cf8d47cac0
author
bnema · · 2026-10-13 07:35 UTC
merge of
cbb8ad6 + a236b1d · diff against the first parent
signature
SSH UnverifiedSHA256:VOCA5lYARHcbKDZ7GhTV45VmcjWnJ6ZQIBazj17LivA The key is unknown
20 files · +1241 -15
Unified
UnifiedSplit
M README.md +17 -2
@@ -37,10 +37,23 @@ Reaping removes every process, device, firewall rule and Docker object the runne
3737
3838 ### Network policy (both modes)
3939
40-One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address, where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
40+One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
4141
4242 **Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
4343
44+### Guest forge access (opt-in, staging only)
45+
46+Off by default. `ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the forge **the way the runner does**, for a staging host where ermite is only reachable through an overlay (NetBird) and the runner itself uses a local route, e.g. `/etc/hosts` `ermite.bnema.dev -> 127.0.0.1` and a reverse proxy (Gordon) on `*:8088` with `ERMITE_RUNNER_URL=http://ermite.bnema.dev:8088`. Jobs otherwise cannot `git clone ${{ github.server_url }}/...`: the public URL (`https://ermite.bnema.dev`) is TLS-terminated off-host, and the private ranges are denied.
47+
48+The host name and port come from `ERMITE_RUNNER_URL` (port defaulted from the scheme: 443 or 80). The runner **refuses to start** when the URL host is an IP literal (sandboxes are pointed at the host by DNS), when the URL has a path, when the name does not resolve on the host, or when the URL is plain `http://` without `ERMITE_RUNNER_INSECURE_HTTP=1` (same rule as for the runner itself: tokens then travel in clear). Then, for every sandbox in both modes:
49+
50+- the DNS forwarder answers `A` queries for exactly that host name (case-insensitive, trailing dot ignored) with the sandbox's **gateway address**, and `AAAA` with an empty `NOERROR` answer; every other name is forwarded as before;
51+- the nftables input chain accepts TCP from the sandbox's own address to its own gateway address on that one port, before the sandbox's catch-all drop (one extra line per sandbox; see `internal/netpolicy/testdata/guest_forge.nft`). Nothing else on the host opens; the forward chain (private-range denies, source check, no sandbox-to-sandbox) is unchanged;
52+- the `github.server_url`, `github.api_url` and `github.graphql_url` of the job's contexts, and so `GITHUB_SERVER_URL`, `GITHUB_API_URL` and `GITHUB_GRAPHQL_URL`, are rewritten to the origin of `ERMITE_RUNNER_URL` when their origin is the server's public one (`github.server_url`); paths are kept (`/-/github/api`). Other values, and the URLs inside `github.event`, stay as the server recorded them;
53+- `guest forge access: sandboxes may reach <origin> on the host` is logged at start and at the start of every job, like the privileged-container warning.
54+
55+**The runner does not proxy.** The traffic reaches the host's own listener on that port, so a service must listen on it on an address the gateway reaches (`*`, as Gordon does; a listener bound to `127.0.0.1` receives nothing). A shared reverse proxy on that port routes by `Host` header, so **a job can reach every app that proxy serves on that port**, not only the forge, by sending another `Host` header: use this on staging hosts only, with a proxy whose other apps you accept to expose to job code. That traffic comes from a private address (`10.200.x.x`), so anything behind the proxy that trusts private sources (admin pages, metrics, rate-limit exemptions) trusts job code too. Jobs also see the forge over plain HTTP in that setup.
56+
4457 ### Cancellation
4558
4659 The host sends a `Cancel{reason}` frame (`execstream.Drive` takes a cancel channel, not a context). The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later, with the timers stopped as soon as the shell exits and nothing signalled afterwards (the group id could be reused). Background children that keep the output pipes open are killed when the shell exits, after a short drain. Init forwards SIGTERM and SIGINT to the executor.
@@ -96,7 +109,8 @@ the host process; the sandbox receives the workflow and its contexts only.
96109 | `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |
97110 | `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` (default) or `self-hosted`: the `runner.environment` value the jobs see. ermite's runners stand in for GitHub-hosted ones, so workflows written for github.com (`runner.environment == 'github-hosted'`) behave the same; `self-hosted` also makes "Set up job" print the runner and machine name, as GitHub does for self-hosted runners |
98111 | `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` (see above) |
99-| `ERMITE_RUNNER_INSECURE_HTTP` | `1` allows a non-loopback `http://` URL (tests only) |
112+| `ERMITE_RUNNER_INSECURE_HTTP` | `1` allows a non-loopback `http://` URL (tests only); also required for `ERMITE_RUNNER_GUEST_FORGE` with an `http://` URL |
113+| `ERMITE_RUNNER_GUEST_FORGE` | `1` lets sandboxes reach the forge as the runner does (staging only; see "Guest forge access") |
100114 | `ERMITE_RUNNER_LOG_LEVEL` | `debug`, `info` (default), `warn`, `error` |
101115 | `ERMITE_RUNNER_LOG_FORMAT` | `console` (default, colours) or `json` |
102116
@@ -246,6 +260,7 @@ the moment it starts.
246260 | `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T`. A warning is logged at start when the state directory has less free space than capacity x disk (disks are sparse: it is only a warning) |
247261 | `ERMITE_RUNNER_JOB_PIDS` | `32768` | container process limit |
248262 | `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses container mode without sysbox |
263+| `ERMITE_RUNNER_GUEST_FORGE` | `0` | `1` lets sandboxes reach the host and port of `ERMITE_RUNNER_URL` on the host (staging only, see "Guest forge access") |
249264 | `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | host-wide single-instance lock (tests only) |
250265 | `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits in MiB/s, IOPS and Mbit/s; 0 disables |
251266 | `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |
@@ -37,10 +37,23 @@ Reaping removes every process, device, firewall rule and Docker object the runne
37 37
38 ### Network policy (both modes)38 ### Network policy (both modes)
39 39
40-One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address, where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.40+One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
41 41
42 **Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).42 **Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
43 43
44+### Guest forge access (opt-in, staging only)
45+
46+Off by default. `ERMITE_RUNNER_GUEST_FORGE=1` lets sandboxes reach the forge **the way the runner does**, for a staging host where ermite is only reachable through an overlay (NetBird) and the runner itself uses a local route, e.g. `/etc/hosts` `ermite.bnema.dev -> 127.0.0.1` and a reverse proxy (Gordon) on `*:8088` with `ERMITE_RUNNER_URL=http://ermite.bnema.dev:8088`. Jobs otherwise cannot `git clone ${{ github.server_url }}/...`: the public URL (`https://ermite.bnema.dev`) is TLS-terminated off-host, and the private ranges are denied.
47+
48+The host name and port come from `ERMITE_RUNNER_URL` (port defaulted from the scheme: 443 or 80). The runner **refuses to start** when the URL host is an IP literal (sandboxes are pointed at the host by DNS), when the URL has a path, when the name does not resolve on the host, or when the URL is plain `http://` without `ERMITE_RUNNER_INSECURE_HTTP=1` (same rule as for the runner itself: tokens then travel in clear). Then, for every sandbox in both modes:
49+
50+- the DNS forwarder answers `A` queries for exactly that host name (case-insensitive, trailing dot ignored) with the sandbox's **gateway address**, and `AAAA` with an empty `NOERROR` answer; every other name is forwarded as before;
51+- the nftables input chain accepts TCP from the sandbox's own address to its own gateway address on that one port, before the sandbox's catch-all drop (one extra line per sandbox; see `internal/netpolicy/testdata/guest_forge.nft`). Nothing else on the host opens; the forward chain (private-range denies, source check, no sandbox-to-sandbox) is unchanged;
52+- the `github.server_url`, `github.api_url` and `github.graphql_url` of the job's contexts, and so `GITHUB_SERVER_URL`, `GITHUB_API_URL` and `GITHUB_GRAPHQL_URL`, are rewritten to the origin of `ERMITE_RUNNER_URL` when their origin is the server's public one (`github.server_url`); paths are kept (`/-/github/api`). Other values, and the URLs inside `github.event`, stay as the server recorded them;
53+- `guest forge access: sandboxes may reach <origin> on the host` is logged at start and at the start of every job, like the privileged-container warning.
54+
55+**The runner does not proxy.** The traffic reaches the host's own listener on that port, so a service must listen on it on an address the gateway reaches (`*`, as Gordon does; a listener bound to `127.0.0.1` receives nothing). A shared reverse proxy on that port routes by `Host` header, so **a job can reach every app that proxy serves on that port**, not only the forge, by sending another `Host` header: use this on staging hosts only, with a proxy whose other apps you accept to expose to job code. That traffic comes from a private address (`10.200.x.x`), so anything behind the proxy that trusts private sources (admin pages, metrics, rate-limit exemptions) trusts job code too. Jobs also see the forge over plain HTTP in that setup.
56+
44 ### Cancellation57 ### Cancellation
45 58
46 The host sends a `Cancel{reason}` frame (`execstream.Drive` takes a cancel channel, not a context). The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later, with the timers stopped as soon as the shell exits and nothing signalled afterwards (the group id could be reused). Background children that keep the output pipes open are killed when the shell exits, after a short drain. Init forwards SIGTERM and SIGINT to the executor.59 The host sends a `Cancel{reason}` frame (`execstream.Drive` takes a cancel channel, not a context). The executor signals the script's process group like the official runner: SIGINT at once, SIGTERM after 7.5 s, SIGKILL 2.5 s later, with the timers stopped as soon as the shell exits and nothing signalled afterwards (the group id could be reused). Background children that keep the output pipes open are killed when the shell exits, after a short drain. Init forwards SIGTERM and SIGINT to the executor.
@@ -96,7 +109,8 @@ the host process; the sandbox receives the workflow and its contexts only.
96 | `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |109 | `ERMITE_RUNNER_LABELS` | extra labels, comma separated (`ubuntu-latest,ubuntu-24.04`) |
97 | `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` (default) or `self-hosted`: the `runner.environment` value the jobs see. ermite's runners stand in for GitHub-hosted ones, so workflows written for github.com (`runner.environment == 'github-hosted'`) behave the same; `self-hosted` also makes "Set up job" print the runner and machine name, as GitHub does for self-hosted runners |110 | `ERMITE_RUNNER_ENVIRONMENT` | `github-hosted` (default) or `self-hosted`: the `runner.environment` value the jobs see. ermite's runners stand in for GitHub-hosted ones, so workflows written for github.com (`runner.environment == 'github-hosted'`) behave the same; `self-hosted` also makes "Set up job" print the runner and machine name, as GitHub does for self-hosted runners |
98 | `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` (see above) |111 | `ERMITE_RUNNER_REGISTER_TOKEN` | token for `register` (see above) |
99-| `ERMITE_RUNNER_INSECURE_HTTP` | `1` allows a non-loopback `http://` URL (tests only) |112+| `ERMITE_RUNNER_INSECURE_HTTP` | `1` allows a non-loopback `http://` URL (tests only); also required for `ERMITE_RUNNER_GUEST_FORGE` with an `http://` URL |
113+| `ERMITE_RUNNER_GUEST_FORGE` | `1` lets sandboxes reach the forge as the runner does (staging only; see "Guest forge access") |
100 | `ERMITE_RUNNER_LOG_LEVEL` | `debug`, `info` (default), `warn`, `error` |114 | `ERMITE_RUNNER_LOG_LEVEL` | `debug`, `info` (default), `warn`, `error` |
101 | `ERMITE_RUNNER_LOG_FORMAT` | `console` (default, colours) or `json` |115 | `ERMITE_RUNNER_LOG_FORMAT` | `console` (default, colours) or `json` |
102 116
@@ -246,6 +260,7 @@ the moment it starts.
246 | `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T`. A warning is logged at start when the state directory has less free space than capacity x disk (disks are sparse: it is only a warning) |260 | `ERMITE_RUNNER_JOB_CPUS` / `_JOB_MEMORY` / `_JOB_DISK` | `2` / `4G` / `64G` | per job; sizes accept `M`, `G`, `T`. A warning is logged at start when the state directory has less free space than capacity x disk (disks are sparse: it is only a warning) |
247 | `ERMITE_RUNNER_JOB_PIDS` | `32768` | container process limit |261 | `ERMITE_RUNNER_JOB_PIDS` | `32768` | container process limit |
248 | `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses container mode without sysbox |262 | `ERMITE_RUNNER_DENY_PRIVILEGED` | `0` | `1` refuses container mode without sysbox |
263+| `ERMITE_RUNNER_GUEST_FORGE` | `0` | `1` lets sandboxes reach the host and port of `ERMITE_RUNNER_URL` on the host (staging only, see "Guest forge access") |
249 | `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | host-wide single-instance lock (tests only) |264 | `ERMITE_RUNNER_LOCK_FILE` | `/run/ermite-runner.lock` | host-wide single-instance lock (tests only) |
250 | `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits in MiB/s, IOPS and Mbit/s; 0 disables |265 | `ERMITE_RUNNER_JOB_DISK_BANDWIDTH`, `_JOB_DISK_IOPS`, `_JOB_NET_BANDWIDTH` | `400`, `20000`, `1000` | VM rate limits in MiB/s, IOPS and Mbit/s; 0 disables |
251 | `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |266 | `ERMITE_RUNNER_UID_BASE` | `900000` | jailer uid pool start |
M host_cmd.go +12 -0
@@ -7,6 +7,7 @@ import (
77 "errors"
88 "flag"
99 "fmt"
10+ "net"
1011 "os"
1112
1213 "github.com/bnema/zerowrap"
@@ -104,6 +105,17 @@ func runCmd(ctx context.Context) int {
104105 fmt.Fprintf(os.Stderr, "ermite-runner: %s is not set\n", host.EnvURL)
105106 return 2
106107 }
108+ if scfg.GuestForgeEnabled {
109+ // The sandboxes reach the forge the way the runner does: by the host name
110+ // and port of the URL the runner itself uses.
111+ gf, err := sandbox.ResolveGuestForge(ctx, url, hcfg.InsecureHTTP, net.DefaultResolver.LookupHost)
112+ if err != nil {
113+ fmt.Fprintln(os.Stderr, "ermite-runner:", err)
114+ return 2
115+ }
116+ scfg.GuestForge = gf
117+ hcfg.GuestForgeOrigin = gf.Origin
118+ }
107119 drv, err := buildDriver(ctx, scfg)
108120 if err != nil {
109121 fmt.Fprintln(os.Stderr, "ermite-runner:", err)
@@ -7,6 +7,7 @@ import (
7 "errors"7 "errors"
8 "flag"8 "flag"
9 "fmt"9 "fmt"
10+ "net"
10 "os"11 "os"
11 12
12 "github.com/bnema/zerowrap"13 "github.com/bnema/zerowrap"
@@ -104,6 +105,17 @@ func runCmd(ctx context.Context) int {
104 fmt.Fprintf(os.Stderr, "ermite-runner: %s is not set\n", host.EnvURL)105 fmt.Fprintf(os.Stderr, "ermite-runner: %s is not set\n", host.EnvURL)
105 return 2106 return 2
106 }107 }
108+ if scfg.GuestForgeEnabled {
109+ // The sandboxes reach the forge the way the runner does: by the host name
110+ // and port of the URL the runner itself uses.
111+ gf, err := sandbox.ResolveGuestForge(ctx, url, hcfg.InsecureHTTP, net.DefaultResolver.LookupHost)
112+ if err != nil {
113+ fmt.Fprintln(os.Stderr, "ermite-runner:", err)
114+ return 2
115+ }
116+ scfg.GuestForge = gf
117+ hcfg.GuestForgeOrigin = gf.Origin
118+ }
107 drv, err := buildDriver(ctx, scfg)119 drv, err := buildDriver(ctx, scfg)
108 if err != nil {120 if err != nil {
109 fmt.Fprintln(os.Stderr, "ermite-runner:", err)121 fmt.Fprintln(os.Stderr, "ermite-runner:", err)
M internal/dnsfwd/dnsfwd.go +19 -5
@@ -12,6 +12,10 @@
1212 // connections, an absolute lifetime of TCPLifetime per TCP connection (a client
1313 // cannot hold a slot by trickling bytes), a deadline on every upstream
1414 // exchange, and a cap on queries per TCP connection.
15+//
16+// One name can be overridden (Forwarder.Override, the opt-in guest forge
17+// access): queries for it are answered locally with the address of the
18+// listener, see override.go. Every other name is forwarded.
1519 package dnsfwd
1620
1721 import (
@@ -62,12 +66,18 @@ type Forwarder struct {
6266 // exists when the network is created and a missing one is an error to
6367 // report at once, not a listener on an address nobody owns. Set before Listen.
6468 FreeBind bool
69+ // Override, when set, is a host name that every listener answers itself:
70+ // A with the address the listener is bound to, and NOERROR with no data for
71+ // every other type, whatever the host's resolvers say. Case-insensitive; a
72+ // trailing dot is ignored. Set before Listen.
73+ Override string
6574
6675 mu sync.Mutex
6776 listeners map[netip.Addr]*listener
6877 }
6978
7079 type listener struct {
80+ addr netip.Addr
7181 udp net.PacketConn
7282 tcp net.Listener
7383 cancel context.CancelFunc
@@ -126,7 +136,7 @@ func (f *Forwarder) Listen(ctx context.Context, addr netip.Addr) error {
126136 return fmt.Errorf("dnsfwd: listen tcp %s: %w", hp, err)
127137 }
128138 lctx, cancel := context.WithCancel(context.WithoutCancel(ctx))
129- l := &listener{udp: pc, tcp: tl, cancel: cancel, inflight: make(chan struct{}, MaxInflight), conns: make(chan struct{}, MaxTCPConns)}
139+ l := &listener{addr: addr, udp: pc, tcp: tl, cancel: cancel, inflight: make(chan struct{}, MaxInflight), conns: make(chan struct{}, MaxTCPConns)}
130140 f.listeners[addr] = l
131141 log := zerowrap.FromCtx(ctx)
132142 l.done.Add(2)
@@ -211,7 +221,7 @@ func (f *Forwarder) serveUDP(ctx context.Context, l *listener, log zerowrap.Logg
211221 q := append([]byte(nil), buf[:n]...)
212222 go func() {
213223 defer func() { <-l.inflight }()
214- resp := f.exchange(ctx, q, false)
224+ resp := f.exchange(ctx, l, q, false)
215225 if len(resp) > 0 {
216226 _, _ = pc.WriteTo(resp, from)
217227 }
@@ -277,7 +287,7 @@ func (f *Forwarder) serveConn(ctx context.Context, l *listener, c net.Conn, dead
277287 default:
278288 return // saturated: close, the client retries
279289 }
280- resp := f.exchange(ctx, q, true)
290+ resp := f.exchange(ctx, l, q, true)
281291 <-l.inflight
282292 if len(resp) == 0 || len(resp) > 0xffff {
283293 return
@@ -293,8 +303,12 @@ func (f *Forwarder) serveConn(ctx context.Context, l *listener, c net.Conn, dead
293303
294304 // exchange relays q to the upstreams in order and returns the first answer, or
295305 // a SERVFAIL when none answers. The transport towards the upstream matches the
296-// client's: a client that fell back to TCP gets a complete answer.
297-func (f *Forwarder) exchange(ctx context.Context, q []byte, overTCP bool) []byte {
306+// client's: a client that fell back to TCP gets a complete answer. The
307+// overridden name, if any, is answered without asking the upstreams.
308+func (f *Forwarder) exchange(ctx context.Context, l *listener, q []byte, overTCP bool) []byte {
309+ if resp := f.overrideAnswer(l.addr, q); resp != nil {
310+ return resp
311+ }
298312 for _, up := range f.upstreams() {
299313 var resp []byte
300314 var err error
@@ -12,6 +12,10 @@
12 // connections, an absolute lifetime of TCPLifetime per TCP connection (a client12 // connections, an absolute lifetime of TCPLifetime per TCP connection (a client
13 // cannot hold a slot by trickling bytes), a deadline on every upstream13 // cannot hold a slot by trickling bytes), a deadline on every upstream
14 // exchange, and a cap on queries per TCP connection.14 // exchange, and a cap on queries per TCP connection.
15+//
16+// One name can be overridden (Forwarder.Override, the opt-in guest forge
17+// access): queries for it are answered locally with the address of the
18+// listener, see override.go. Every other name is forwarded.
15 package dnsfwd19 package dnsfwd
16 20
17 import (21 import (
@@ -62,12 +66,18 @@ type Forwarder struct {
62 // exists when the network is created and a missing one is an error to66 // exists when the network is created and a missing one is an error to
63 // report at once, not a listener on an address nobody owns. Set before Listen.67 // report at once, not a listener on an address nobody owns. Set before Listen.
64 FreeBind bool68 FreeBind bool
69+ // Override, when set, is a host name that every listener answers itself:
70+ // A with the address the listener is bound to, and NOERROR with no data for
71+ // every other type, whatever the host's resolvers say. Case-insensitive; a
72+ // trailing dot is ignored. Set before Listen.
73+ Override string
65 74
66 mu sync.Mutex75 mu sync.Mutex
67 listeners map[netip.Addr]*listener76 listeners map[netip.Addr]*listener
68 }77 }
69 78
70 type listener struct {79 type listener struct {
80+ addr netip.Addr
71 udp net.PacketConn81 udp net.PacketConn
72 tcp net.Listener82 tcp net.Listener
73 cancel context.CancelFunc83 cancel context.CancelFunc
@@ -126,7 +136,7 @@ func (f *Forwarder) Listen(ctx context.Context, addr netip.Addr) error {
126 return fmt.Errorf("dnsfwd: listen tcp %s: %w", hp, err)136 return fmt.Errorf("dnsfwd: listen tcp %s: %w", hp, err)
127 }137 }
128 lctx, cancel := context.WithCancel(context.WithoutCancel(ctx))138 lctx, cancel := context.WithCancel(context.WithoutCancel(ctx))
129- l := &listener{udp: pc, tcp: tl, cancel: cancel, inflight: make(chan struct{}, MaxInflight), conns: make(chan struct{}, MaxTCPConns)}139+ l := &listener{addr: addr, udp: pc, tcp: tl, cancel: cancel, inflight: make(chan struct{}, MaxInflight), conns: make(chan struct{}, MaxTCPConns)}
130 f.listeners[addr] = l140 f.listeners[addr] = l
131 log := zerowrap.FromCtx(ctx)141 log := zerowrap.FromCtx(ctx)
132 l.done.Add(2)142 l.done.Add(2)
@@ -211,7 +221,7 @@ func (f *Forwarder) serveUDP(ctx context.Context, l *listener, log zerowrap.Logg
211 q := append([]byte(nil), buf[:n]...)221 q := append([]byte(nil), buf[:n]...)
212 go func() {222 go func() {
213 defer func() { <-l.inflight }()223 defer func() { <-l.inflight }()
214- resp := f.exchange(ctx, q, false)224+ resp := f.exchange(ctx, l, q, false)
215 if len(resp) > 0 {225 if len(resp) > 0 {
216 _, _ = pc.WriteTo(resp, from)226 _, _ = pc.WriteTo(resp, from)
217 }227 }
@@ -277,7 +287,7 @@ func (f *Forwarder) serveConn(ctx context.Context, l *listener, c net.Conn, dead
277 default:287 default:
278 return // saturated: close, the client retries288 return // saturated: close, the client retries
279 }289 }
280- resp := f.exchange(ctx, q, true)290+ resp := f.exchange(ctx, l, q, true)
281 <-l.inflight291 <-l.inflight
282 if len(resp) == 0 || len(resp) > 0xffff {292 if len(resp) == 0 || len(resp) > 0xffff {
283 return293 return
@@ -293,8 +303,12 @@ func (f *Forwarder) serveConn(ctx context.Context, l *listener, c net.Conn, dead
293 303
294 // exchange relays q to the upstreams in order and returns the first answer, or304 // exchange relays q to the upstreams in order and returns the first answer, or
295 // a SERVFAIL when none answers. The transport towards the upstream matches the305 // a SERVFAIL when none answers. The transport towards the upstream matches the
296-// client's: a client that fell back to TCP gets a complete answer.306+// client's: a client that fell back to TCP gets a complete answer. The
297-func (f *Forwarder) exchange(ctx context.Context, q []byte, overTCP bool) []byte {307+// overridden name, if any, is answered without asking the upstreams.
308+func (f *Forwarder) exchange(ctx context.Context, l *listener, q []byte, overTCP bool) []byte {
309+ if resp := f.overrideAnswer(l.addr, q); resp != nil {
310+ return resp
311+ }
298 for _, up := range f.upstreams() {312 for _, up := range f.upstreams() {
299 var resp []byte313 var resp []byte
300 var err error314 var err error
M internal/dnsfwd/dnsfwd_test.go +146 -0
@@ -466,3 +466,149 @@ func TestFreeBindListensOnAnAddressTheHostDoesNotOwnYet(t *testing.T) {
466466 t.Fatal("no listener")
467467 }
468468 }
469+
470+// queryType is query with a chosen type (and optionally EDNS0).
471+func queryType(id uint16, name string, qtype uint16, edns bool) []byte {
472+ q := query(id, name)
473+ binary.BigEndian.PutUint16(q[len(q)-4:], qtype)
474+ if edns {
475+ q[11] = 1
476+ q = append(q, 0, 0, 41, 0x10, 0, 0, 0, 0, 0, 0, 0)
477+ }
478+ return q
479+}
480+
481+func udpExchange(t *testing.T, addr string, q []byte) []byte {
482+ t.Helper()
483+ c, err := net.Dial("udp", addr)
484+ if err != nil {
485+ t.Fatal(err)
486+ }
487+ defer func() { _ = c.Close() }()
488+ _ = c.SetDeadline(time.Now().Add(5 * time.Second))
489+ _, _ = c.Write(q)
490+ buf := make([]byte, 1024)
491+ n, err := c.Read(buf)
492+ if err != nil {
493+ t.Fatal(err)
494+ }
495+ return buf[:n]
496+}
497+
498+func startOverride(t *testing.T, name string, ups ...string) (*Forwarder, string) {
499+ t.Helper()
500+ f := New(func() []string { return ups }, 0)
501+ f.Override = name
502+ ip := netip.MustParseAddr("127.0.0.1")
503+ if err := f.Listen(context.Background(), ip); err != nil {
504+ t.Fatal(err)
505+ }
506+ t.Cleanup(f.CloseAll)
507+ a, _ := f.Addr(ip)
508+ return f, a.String()
509+}
510+
511+func TestOverrideAnswersAWithTheListenerAddress(t *testing.T) {
512+ var asked atomic.Int64
513+ up := fakeUpstream(t, func(q []byte) []byte { asked.Add(1); return answer(q) })
514+ // A trailing dot in the setting is ignored; on the wire a name has none.
515+ _, addr := startOverride(t, "Ermite.Example.org.", up)
516+ for _, name := range []string{"ermite.example.org", "ERMITE.example.ORG", "Ermite.Example.Org"} {
517+ r := udpExchange(t, addr, queryType(0x4242, name, 1, false))
518+ if binary.BigEndian.Uint16(r) != 0x4242 || r[2]&0x80 == 0 || r[2]&0x01 == 0 || r[3]&0x0F != 0 {
519+ t.Fatalf("%s: header %x", name, r[:hdrLen])
520+ }
521+ if an := binary.BigEndian.Uint16(r[6:]); an != 1 {
522+ t.Fatalf("%s: %d answers", name, an)
523+ }
524+ if !strings.HasSuffix(string(r), "\x00\x04\x7f\x00\x00\x01") {
525+ t.Fatalf("%s: rdata is not 127.0.0.1: %x", name, r)
526+ }
527+ }
528+ if asked.Load() != 0 {
529+ t.Fatalf("the override went upstream %d times", asked.Load())
530+ }
531+}
532+
533+func TestOverrideAAAAIsEmptyNoError(t *testing.T) {
534+ up := fakeUpstream(t, answer)
535+ _, addr := startOverride(t, "ermite.example.org", up)
536+ for _, edns := range []bool{false, true} {
537+ r := udpExchange(t, addr, queryType(7, "ermite.EXAMPLE.org", 28, edns))
538+ if r[3]&0x0F != 0 || binary.BigEndian.Uint16(r[6:]) != 0 || r[2]&0x80 == 0 {
539+ t.Fatalf("AAAA (edns %v): %x", edns, r)
540+ }
541+ if edns != (binary.BigEndian.Uint16(r[10:]) == 1) {
542+ t.Fatalf("AAAA (edns %v): arcount %d", edns, binary.BigEndian.Uint16(r[10:]))
543+ }
544+ }
545+}
546+
547+func TestOverrideForwardsEveryOtherName(t *testing.T) {
548+ var asked atomic.Int64
549+ up := fakeUpstream(t, func(q []byte) []byte { asked.Add(1); return answer(q) })
550+ _, addr := startOverride(t, "ermite.example.org", up)
551+ for _, name := range []string{"example.org", "other.example.org", "x.ermite.example.org", "ermite.example.org.evil.test", "ermite.example.or"} {
552+ r := udpExchange(t, addr, query(9, name))
553+ if string(r[len(r)-6:]) != "ANSWER" {
554+ t.Fatalf("%s was not forwarded: %x", name, r)
555+ }
556+ }
557+ if asked.Load() != 5 {
558+ t.Fatalf("%d upstream queries, want 5", asked.Load())
559+ }
560+}
561+
562+func TestOverrideOverTCPAndDisabledByDefault(t *testing.T) {
563+ up := fakeUpstream(t, answer)
564+ _, addr := startOverride(t, "ermite.example.org", up)
565+ tc, err := net.Dial("tcp", addr)
566+ if err != nil {
567+ t.Fatal(err)
568+ }
569+ defer func() { _ = tc.Close() }()
570+ _ = tc.SetDeadline(time.Now().Add(5 * time.Second))
571+ q := queryType(3, "ermite.example.org", 1, false)
572+ _, _ = tc.Write(append([]byte{byte(len(q) >> 8), byte(len(q))}, q...))
573+ var l [2]byte
574+ if _, err := io.ReadFull(tc, l[:]); err != nil {
575+ t.Fatal(err)
576+ }
577+ r := make([]byte, binary.BigEndian.Uint16(l[:]))
578+ if _, err := io.ReadFull(tc, r); err != nil {
579+ t.Fatal(err)
580+ }
581+ if binary.BigEndian.Uint16(r[6:]) != 1 || !strings.HasSuffix(string(r), "\x7f\x00\x00\x01") {
582+ t.Fatalf("tcp: %x", r)
583+ }
584+
585+ // Off by default: the name is forwarded like any other.
586+ _, addr2 := start(t, up)
587+ r2 := udpExchange(t, addr2, query(4, "ermite.example.org"))
588+ if string(r2[len(r2)-6:]) != "ANSWER" {
589+ t.Fatalf("not forwarded without Override: %x", r2)
590+ }
591+}
592+
593+func TestOverrideIgnoresMalformedAndNonQueries(t *testing.T) {
594+ f := New(func() []string { return nil }, 0)
595+ f.Override = "ermite.example.org"
596+ gw := netip.MustParseAddr("10.200.0.1")
597+ good := queryType(1, "ermite.example.org", 1, false)
598+ if f.overrideAnswer(gw, good) == nil {
599+ t.Fatal("control query not answered")
600+ }
601+ cut := good[:len(good)-3]
602+ chaos := queryType(1, "ermite.example.org", 1, false)
603+ binary.BigEndian.PutUint16(chaos[len(chaos)-2:], 3) // class CH
604+ resp := append([]byte(nil), good...)
605+ resp[2] |= 0x80 // QR set: not a query
606+ notify := append([]byte(nil), good...)
607+ notify[2] |= 4 << 3 // opcode 4
608+ ptr := append(append([]byte(nil), good[:hdrLen]...), 0xC0, 0x0C, 0, 1, 0, 1)
609+ for name, q := range map[string][]byte{"truncated": cut, "class CH": chaos, "response": resp, "opcode": notify, "pointer": ptr, "short": good[:5]} {
610+ if f.overrideAnswer(gw, q) != nil {
611+ t.Errorf("%s was answered locally", name)
612+ }
613+ }
614+}
@@ -466,3 +466,149 @@ func TestFreeBindListensOnAnAddressTheHostDoesNotOwnYet(t *testing.T) {
466 t.Fatal("no listener")466 t.Fatal("no listener")
467 }467 }
468 }468 }
469+
470+// queryType is query with a chosen type (and optionally EDNS0).
471+func queryType(id uint16, name string, qtype uint16, edns bool) []byte {
472+ q := query(id, name)
473+ binary.BigEndian.PutUint16(q[len(q)-4:], qtype)
474+ if edns {
475+ q[11] = 1
476+ q = append(q, 0, 0, 41, 0x10, 0, 0, 0, 0, 0, 0, 0)
477+ }
478+ return q
479+}
480+
481+func udpExchange(t *testing.T, addr string, q []byte) []byte {
482+ t.Helper()
483+ c, err := net.Dial("udp", addr)
484+ if err != nil {
485+ t.Fatal(err)
486+ }
487+ defer func() { _ = c.Close() }()
488+ _ = c.SetDeadline(time.Now().Add(5 * time.Second))
489+ _, _ = c.Write(q)
490+ buf := make([]byte, 1024)
491+ n, err := c.Read(buf)
492+ if err != nil {
493+ t.Fatal(err)
494+ }
495+ return buf[:n]
496+}
497+
498+func startOverride(t *testing.T, name string, ups ...string) (*Forwarder, string) {
499+ t.Helper()
500+ f := New(func() []string { return ups }, 0)
501+ f.Override = name
502+ ip := netip.MustParseAddr("127.0.0.1")
503+ if err := f.Listen(context.Background(), ip); err != nil {
504+ t.Fatal(err)
505+ }
506+ t.Cleanup(f.CloseAll)
507+ a, _ := f.Addr(ip)
508+ return f, a.String()
509+}
510+
511+func TestOverrideAnswersAWithTheListenerAddress(t *testing.T) {
512+ var asked atomic.Int64
513+ up := fakeUpstream(t, func(q []byte) []byte { asked.Add(1); return answer(q) })
514+ // A trailing dot in the setting is ignored; on the wire a name has none.
515+ _, addr := startOverride(t, "Ermite.Example.org.", up)
516+ for _, name := range []string{"ermite.example.org", "ERMITE.example.ORG", "Ermite.Example.Org"} {
517+ r := udpExchange(t, addr, queryType(0x4242, name, 1, false))
518+ if binary.BigEndian.Uint16(r) != 0x4242 || r[2]&0x80 == 0 || r[2]&0x01 == 0 || r[3]&0x0F != 0 {
519+ t.Fatalf("%s: header %x", name, r[:hdrLen])
520+ }
521+ if an := binary.BigEndian.Uint16(r[6:]); an != 1 {
522+ t.Fatalf("%s: %d answers", name, an)
523+ }
524+ if !strings.HasSuffix(string(r), "\x00\x04\x7f\x00\x00\x01") {
525+ t.Fatalf("%s: rdata is not 127.0.0.1: %x", name, r)
526+ }
527+ }
528+ if asked.Load() != 0 {
529+ t.Fatalf("the override went upstream %d times", asked.Load())
530+ }
531+}
532+
533+func TestOverrideAAAAIsEmptyNoError(t *testing.T) {
534+ up := fakeUpstream(t, answer)
535+ _, addr := startOverride(t, "ermite.example.org", up)
536+ for _, edns := range []bool{false, true} {
537+ r := udpExchange(t, addr, queryType(7, "ermite.EXAMPLE.org", 28, edns))
538+ if r[3]&0x0F != 0 || binary.BigEndian.Uint16(r[6:]) != 0 || r[2]&0x80 == 0 {
539+ t.Fatalf("AAAA (edns %v): %x", edns, r)
540+ }
541+ if edns != (binary.BigEndian.Uint16(r[10:]) == 1) {
542+ t.Fatalf("AAAA (edns %v): arcount %d", edns, binary.BigEndian.Uint16(r[10:]))
543+ }
544+ }
545+}
546+
547+func TestOverrideForwardsEveryOtherName(t *testing.T) {
548+ var asked atomic.Int64
549+ up := fakeUpstream(t, func(q []byte) []byte { asked.Add(1); return answer(q) })
550+ _, addr := startOverride(t, "ermite.example.org", up)
551+ for _, name := range []string{"example.org", "other.example.org", "x.ermite.example.org", "ermite.example.org.evil.test", "ermite.example.or"} {
552+ r := udpExchange(t, addr, query(9, name))
553+ if string(r[len(r)-6:]) != "ANSWER" {
554+ t.Fatalf("%s was not forwarded: %x", name, r)
555+ }
556+ }
557+ if asked.Load() != 5 {
558+ t.Fatalf("%d upstream queries, want 5", asked.Load())
559+ }
560+}
561+
562+func TestOverrideOverTCPAndDisabledByDefault(t *testing.T) {
563+ up := fakeUpstream(t, answer)
564+ _, addr := startOverride(t, "ermite.example.org", up)
565+ tc, err := net.Dial("tcp", addr)
566+ if err != nil {
567+ t.Fatal(err)
568+ }
569+ defer func() { _ = tc.Close() }()
570+ _ = tc.SetDeadline(time.Now().Add(5 * time.Second))
571+ q := queryType(3, "ermite.example.org", 1, false)
572+ _, _ = tc.Write(append([]byte{byte(len(q) >> 8), byte(len(q))}, q...))
573+ var l [2]byte
574+ if _, err := io.ReadFull(tc, l[:]); err != nil {
575+ t.Fatal(err)
576+ }
577+ r := make([]byte, binary.BigEndian.Uint16(l[:]))
578+ if _, err := io.ReadFull(tc, r); err != nil {
579+ t.Fatal(err)
580+ }
581+ if binary.BigEndian.Uint16(r[6:]) != 1 || !strings.HasSuffix(string(r), "\x7f\x00\x00\x01") {
582+ t.Fatalf("tcp: %x", r)
583+ }
584+
585+ // Off by default: the name is forwarded like any other.
586+ _, addr2 := start(t, up)
587+ r2 := udpExchange(t, addr2, query(4, "ermite.example.org"))
588+ if string(r2[len(r2)-6:]) != "ANSWER" {
589+ t.Fatalf("not forwarded without Override: %x", r2)
590+ }
591+}
592+
593+func TestOverrideIgnoresMalformedAndNonQueries(t *testing.T) {
594+ f := New(func() []string { return nil }, 0)
595+ f.Override = "ermite.example.org"
596+ gw := netip.MustParseAddr("10.200.0.1")
597+ good := queryType(1, "ermite.example.org", 1, false)
598+ if f.overrideAnswer(gw, good) == nil {
599+ t.Fatal("control query not answered")
600+ }
601+ cut := good[:len(good)-3]
602+ chaos := queryType(1, "ermite.example.org", 1, false)
603+ binary.BigEndian.PutUint16(chaos[len(chaos)-2:], 3) // class CH
604+ resp := append([]byte(nil), good...)
605+ resp[2] |= 0x80 // QR set: not a query
606+ notify := append([]byte(nil), good...)
607+ notify[2] |= 4 << 3 // opcode 4
608+ ptr := append(append([]byte(nil), good[:hdrLen]...), 0xC0, 0x0C, 0, 1, 0, 1)
609+ for name, q := range map[string][]byte{"truncated": cut, "class CH": chaos, "response": resp, "opcode": notify, "pointer": ptr, "short": good[:5]} {
610+ if f.overrideAnswer(gw, q) != nil {
611+ t.Errorf("%s was answered locally", name)
612+ }
613+ }
614+}
A internal/dnsfwd/override.go +109 -0
@@ -0,0 +1,109 @@
1+// SPDX-License-Identifier: MIT
2+
3+package dnsfwd
4+
5+import (
6+ "encoding/binary"
7+ "net/netip"
8+ "strings"
9+)
10+
11+const (
12+ typeA = 1
13+ classIN = 1
14+ // overrideTTL is short: the sandbox lives for one job.
15+ overrideTTL = 30
16+)
17+
18+// overrideName returns Override normalised for comparison: lower case, without
19+// a trailing dot.
20+func (f *Forwarder) overrideName() string {
21+ return strings.TrimSuffix(strings.ToLower(f.Override), ".")
22+}
23+
24+// questionName reads the name of the question that starts at q[hdrLen]. It
25+// returns the lower-cased name without trailing dot and the offset just after
26+// the name. Compression pointers and over-long labels are refused (a question
27+// never uses them): ok is false and the query is forwarded as usual.
28+func questionName(q []byte) (name string, end int, ok bool) {
29+ var sb strings.Builder
30+ i := hdrLen
31+ for {
32+ if i >= len(q) {
33+ return "", 0, false
34+ }
35+ n := int(q[i])
36+ i++
37+ if n == 0 {
38+ return sb.String(), i, true
39+ }
40+ if n&0xC0 != 0 || i+n > len(q) {
41+ return "", 0, false
42+ }
43+ if sb.Len() > 0 {
44+ sb.WriteByte('.')
45+ }
46+ sb.WriteString(strings.ToLower(string(q[i : i+n])))
47+ i += n
48+ if sb.Len() > 253 {
49+ return "", 0, false
50+ }
51+ }
52+}
53+
54+// overrideAnswer builds the local answer to q when q asks for the overridden
55+// name in class IN, else nil. A gets addr; every other type gets NOERROR with
56+// no data (so AAAA is an empty answer: sandboxes have no IPv6, and an upstream
57+// NXDOMAIN or CNAME for the name must not leak in).
58+func (f *Forwarder) overrideAnswer(addr netip.Addr, q []byte) []byte {
59+ want := f.overrideName()
60+ if want == "" || len(q) < hdrLen {
61+ return nil
62+ }
63+ // A plain query: QR=0, opcode 0, one question, nothing else but additional
64+ // records (EDNS).
65+ if q[2]&0xF8 != 0 || binary.BigEndian.Uint16(q[4:]) != 1 ||
66+ binary.BigEndian.Uint16(q[6:]) != 0 || binary.BigEndian.Uint16(q[8:]) != 0 {
67+ return nil
68+ }
69+ name, end, ok := questionName(q)
70+ if !ok || name != want || end+4 > len(q) {
71+ return nil
72+ }
73+ qtype := binary.BigEndian.Uint16(q[end:])
74+ if binary.BigEndian.Uint16(q[end+2:]) != classIN {
75+ return nil
76+ }
77+ end += 4
78+ withA := qtype == typeA && addr.Is4()
79+
80+ r := make([]byte, 0, end+16+11)
81+ r = append(r, q[0], q[1])
82+ r = append(r, 0x80|(q[2]&0x01), 0x80) // QR, RD copied; RA; NOERROR
83+ r = append(r, 0, 1) // QDCOUNT
84+ if withA {
85+ r = append(r, 0, 1)
86+ } else {
87+ r = append(r, 0, 0)
88+ }
89+ r = append(r, 0, 0) // NSCOUNT
90+ // An EDNS query gets an OPT record back (RFC 6891), without options.
91+ edns := binary.BigEndian.Uint16(q[10:]) >= 1 && len(q) >= end+11 &&
92+ q[end] == 0 && binary.BigEndian.Uint16(q[end+1:]) == 41
93+ if edns {
94+ r = append(r, 0, 1)
95+ } else {
96+ r = append(r, 0, 0)
97+ }
98+ r = append(r, q[hdrLen:end]...)
99+ if withA {
100+ r = append(r, 0xC0, hdrLen, 0, typeA, 0, classIN)
101+ r = binary.BigEndian.AppendUint32(r, overrideTTL)
102+ a := addr.As4()
103+ r = append(r, 0, 4, a[0], a[1], a[2], a[3])
104+ }
105+ if edns {
106+ r = append(r, 0, 0, 41, 0x10, 0, 0, 0, 0, 0, 0, 0) // root, OPT, 4096 bytes, no options
107+ }
108+ return r
109+}
@@ -0,0 +1,109 @@
1+// SPDX-License-Identifier: MIT
2+
3+package dnsfwd
4+
5+import (
6+ "encoding/binary"
7+ "net/netip"
8+ "strings"
9+)
10+
11+const (
12+ typeA = 1
13+ classIN = 1
14+ // overrideTTL is short: the sandbox lives for one job.
15+ overrideTTL = 30
16+)
17+
18+// overrideName returns Override normalised for comparison: lower case, without
19+// a trailing dot.
20+func (f *Forwarder) overrideName() string {
21+ return strings.TrimSuffix(strings.ToLower(f.Override), ".")
22+}
23+
24+// questionName reads the name of the question that starts at q[hdrLen]. It
25+// returns the lower-cased name without trailing dot and the offset just after
26+// the name. Compression pointers and over-long labels are refused (a question
27+// never uses them): ok is false and the query is forwarded as usual.
28+func questionName(q []byte) (name string, end int, ok bool) {
29+ var sb strings.Builder
30+ i := hdrLen
31+ for {
32+ if i >= len(q) {
33+ return "", 0, false
34+ }
35+ n := int(q[i])
36+ i++
37+ if n == 0 {
38+ return sb.String(), i, true
39+ }
40+ if n&0xC0 != 0 || i+n > len(q) {
41+ return "", 0, false
42+ }
43+ if sb.Len() > 0 {
44+ sb.WriteByte('.')
45+ }
46+ sb.WriteString(strings.ToLower(string(q[i : i+n])))
47+ i += n
48+ if sb.Len() > 253 {
49+ return "", 0, false
50+ }
51+ }
52+}
53+
54+// overrideAnswer builds the local answer to q when q asks for the overridden
55+// name in class IN, else nil. A gets addr; every other type gets NOERROR with
56+// no data (so AAAA is an empty answer: sandboxes have no IPv6, and an upstream
57+// NXDOMAIN or CNAME for the name must not leak in).
58+func (f *Forwarder) overrideAnswer(addr netip.Addr, q []byte) []byte {
59+ want := f.overrideName()
60+ if want == "" || len(q) < hdrLen {
61+ return nil
62+ }
63+ // A plain query: QR=0, opcode 0, one question, nothing else but additional
64+ // records (EDNS).
65+ if q[2]&0xF8 != 0 || binary.BigEndian.Uint16(q[4:]) != 1 ||
66+ binary.BigEndian.Uint16(q[6:]) != 0 || binary.BigEndian.Uint16(q[8:]) != 0 {
67+ return nil
68+ }
69+ name, end, ok := questionName(q)
70+ if !ok || name != want || end+4 > len(q) {
71+ return nil
72+ }
73+ qtype := binary.BigEndian.Uint16(q[end:])
74+ if binary.BigEndian.Uint16(q[end+2:]) != classIN {
75+ return nil
76+ }
77+ end += 4
78+ withA := qtype == typeA && addr.Is4()
79+
80+ r := make([]byte, 0, end+16+11)
81+ r = append(r, q[0], q[1])
82+ r = append(r, 0x80|(q[2]&0x01), 0x80) // QR, RD copied; RA; NOERROR
83+ r = append(r, 0, 1) // QDCOUNT
84+ if withA {
85+ r = append(r, 0, 1)
86+ } else {
87+ r = append(r, 0, 0)
88+ }
89+ r = append(r, 0, 0) // NSCOUNT
90+ // An EDNS query gets an OPT record back (RFC 6891), without options.
91+ edns := binary.BigEndian.Uint16(q[10:]) >= 1 && len(q) >= end+11 &&
92+ q[end] == 0 && binary.BigEndian.Uint16(q[end+1:]) == 41
93+ if edns {
94+ r = append(r, 0, 1)
95+ } else {
96+ r = append(r, 0, 0)
97+ }
98+ r = append(r, q[hdrLen:end]...)
99+ if withA {
100+ r = append(r, 0xC0, hdrLen, 0, typeA, 0, classIN)
101+ r = binary.BigEndian.AppendUint32(r, overrideTTL)
102+ a := addr.As4()
103+ r = append(r, 0, 4, a[0], a[1], a[2], a[3])
104+ }
105+ if edns {
106+ r = append(r, 0, 0, 41, 0x10, 0, 0, 0, 0, 0, 0, 0) // root, OPT, 4096 bytes, no options
107+ }
108+ return r
109+}
M internal/host/config.go +5 -0
@@ -62,6 +62,11 @@ type Config struct {
6262 Environment string
6363 // InsecureHTTP allows a non-loopback http URL (ERMITE_RUNNER_INSECURE_HTTP=1).
6464 InsecureHTTP bool
65+ // GuestForgeOrigin is the origin (scheme://host[:port]) of ERMITE_RUNNER_URL
66+ // when ERMITE_RUNNER_GUEST_FORGE=1, "" otherwise. The server URLs of every
67+ // job are rewritten to it, see RewriteServerURLs. Set by the caller after
68+ // sandbox.ResolveGuestForge.
69+ GuestForgeOrigin string
6570 }
6671
6772 // Client returns the client for a token, honouring InsecureHTTP.
@@ -62,6 +62,11 @@ type Config struct {
62 Environment string62 Environment string
63 // InsecureHTTP allows a non-loopback http URL (ERMITE_RUNNER_INSECURE_HTTP=1).63 // InsecureHTTP allows a non-loopback http URL (ERMITE_RUNNER_INSECURE_HTTP=1).
64 InsecureHTTP bool64 InsecureHTTP bool
65+ // GuestForgeOrigin is the origin (scheme://host[:port]) of ERMITE_RUNNER_URL
66+ // when ERMITE_RUNNER_GUEST_FORGE=1, "" otherwise. The server URLs of every
67+ // job are rewritten to it, see RewriteServerURLs. Set by the caller after
68+ // sandbox.ResolveGuestForge.
69+ GuestForgeOrigin string
65 }70 }
66 71
67 // Client returns the client for a token, honouring InsecureHTTP.72 // Client returns the client for a token, honouring InsecureHTTP.
A internal/host/guestforge.go +189 -0
@@ -0,0 +1,189 @@
1+// SPDX-License-Identifier: MIT
2+
3+package host
4+
5+import (
6+ "bytes"
7+ "encoding/json"
8+ "errors"
9+ "fmt"
10+ "io"
11+ "net/url"
12+ "strings"
13+)
14+
15+// serverURLKeys are the keys of the github context that actions use to reach
16+// the forge: the server's web URL, its REST API and its GraphQL API. The
17+// environment variables GITHUB_SERVER_URL, GITHUB_API_URL and GITHUB_GRAPHQL_URL
18+// are derived from them by the executor (State.RuntimeEnv), so rewriting the
19+// context rewrites the environment too. Everything else that carries the
20+// origin (the URLs inside github.event, recorded as the server built them;
21+// repositoryUrl, which is a git:// URL) is left alone.
22+var serverURLKeys = []string{"server_url", "api_url", "graphql_url"}
23+
24+type jsonMember struct {
25+ key string
26+ val json.RawMessage
27+}
28+
29+// RewriteServerURLs returns the contexts JSON of a job with the origin of
30+// github.server_url, github.api_url and github.graphql_url replaced by target
31+// (scheme://host[:port], the ERMITE_RUNNER_URL origin) when, and only when, the
32+// value has the same origin as github.server_url, the server's public origin.
33+// Paths, queries and fragments are kept ("/-/github/api"); values with another
34+// origin, with credentials, or that are not strings are untouched, as is the
35+// rest of the document (order of keys included). The input is returned as it
36+// is when nothing changes.
37+func RewriteServerURLs(contexts, target string) (string, error) {
38+ if target == "" {
39+ return contexts, nil
40+ }
41+ top, err := parseObject([]byte(contexts))
42+ if err != nil {
43+ return "", fmt.Errorf("job contexts: %w", err)
44+ }
45+ gi := -1
46+ for i, m := range top {
47+ if m.key == "github" {
48+ gi = i
49+ }
50+ }
51+ if gi < 0 {
52+ return contexts, nil
53+ }
54+ gh, err := parseObject(top[gi].val)
55+ if err != nil {
56+ return "", fmt.Errorf("job contexts: github: %w", err)
57+ }
58+ str := func(raw json.RawMessage) (string, bool) {
59+ var s string
60+ return s, json.Unmarshal(raw, &s) == nil
61+ }
62+ var public string
63+ for _, m := range gh {
64+ if m.key == "server_url" {
65+ if s, ok := str(m.val); ok {
66+ public = originOf(s)
67+ }
68+ }
69+ }
70+ if public == "" || public == originOf(target) {
71+ return contexts, nil
72+ }
73+ changed := false
74+ for i, m := range gh {
75+ isURL := false
76+ for _, k := range serverURLKeys {
77+ isURL = isURL || m.key == k
78+ }
79+ if !isURL {
80+ continue
81+ }
82+ s, ok := str(m.val)
83+ if !ok || originOf(s) != public {
84+ continue
85+ }
86+ rest := s[strings.Index(s, "://")+3:]
87+ if k := strings.IndexAny(rest, "/?#"); k >= 0 {
88+ rest = rest[k:]
89+ } else {
90+ rest = ""
91+ }
92+ nv, err := marshalNoEscape(target + rest)
93+ if err != nil {
94+ return "", err
95+ }
96+ gh[i].val = nv
97+ changed = true
98+ }
99+ if !changed {
100+ return contexts, nil
101+ }
102+ if top[gi].val, err = encodeObject(gh); err != nil {
103+ return "", err
104+ }
105+ out, err := encodeObject(top)
106+ if err != nil {
107+ return "", err
108+ }
109+ return string(out), nil
110+}
111+
112+// originOf returns "scheme://host[:port]" of an http(s) URL, lower-cased and
113+// without the default port, or "" for anything else (including URLs with
114+// credentials, which are never rewritten).
115+func originOf(raw string) string {
116+ u, err := url.Parse(raw)
117+ if err != nil || u.Host == "" || u.User != nil {
118+ return ""
119+ }
120+ scheme := strings.ToLower(u.Scheme)
121+ if scheme != "http" && scheme != "https" {
122+ return ""
123+ }
124+ host, port := strings.ToLower(u.Hostname()), u.Port()
125+ if port == "" || (scheme == "http" && port == "80") || (scheme == "https" && port == "443") {
126+ return scheme + "://" + host
127+ }
128+ return scheme + "://" + host + ":" + port
129+}
130+
131+func marshalNoEscape(v any) (json.RawMessage, error) {
132+ var b bytes.Buffer
133+ enc := json.NewEncoder(&b)
134+ enc.SetEscapeHTML(false)
135+ if err := enc.Encode(v); err != nil {
136+ return nil, err
137+ }
138+ return bytes.TrimRight(b.Bytes(), "\n"), nil
139+}
140+
141+// parseObject reads a JSON object into its members, in order.
142+func parseObject(b []byte) ([]jsonMember, error) {
143+ dec := json.NewDecoder(bytes.NewReader(b))
144+ if t, err := dec.Token(); err != nil || t != json.Delim('{') {
145+ return nil, errors.New("not a JSON object")
146+ }
147+ var out []jsonMember
148+ for dec.More() {
149+ kt, err := dec.Token()
150+ if err != nil {
151+ return nil, err
152+ }
153+ k, ok := kt.(string)
154+ if !ok {
155+ return nil, errors.New("not a JSON object")
156+ }
157+ var v json.RawMessage
158+ if err := dec.Decode(&v); err != nil {
159+ return nil, err
160+ }
161+ out = append(out, jsonMember{k, v})
162+ }
163+ if _, err := dec.Token(); err != nil {
164+ return nil, err
165+ }
166+ if _, err := dec.Token(); err != io.EOF {
167+ return nil, errors.New("data after the JSON object")
168+ }
169+ return out, nil
170+}
171+
172+func encodeObject(ms []jsonMember) (json.RawMessage, error) {
173+ var b bytes.Buffer
174+ b.WriteByte('{')
175+ for i, m := range ms {
176+ if i > 0 {
177+ b.WriteByte(',')
178+ }
179+ k, err := marshalNoEscape(m.key)
180+ if err != nil {
181+ return nil, err
182+ }
183+ b.Write(k)
184+ b.WriteByte(':')
185+ b.Write(m.val)
186+ }
187+ b.WriteByte('}')
188+ return b.Bytes(), nil
189+}
@@ -0,0 +1,189 @@
1+// SPDX-License-Identifier: MIT
2+
3+package host
4+
5+import (
6+ "bytes"
7+ "encoding/json"
8+ "errors"
9+ "fmt"
10+ "io"
11+ "net/url"
12+ "strings"
13+)
14+
15+// serverURLKeys are the keys of the github context that actions use to reach
16+// the forge: the server's web URL, its REST API and its GraphQL API. The
17+// environment variables GITHUB_SERVER_URL, GITHUB_API_URL and GITHUB_GRAPHQL_URL
18+// are derived from them by the executor (State.RuntimeEnv), so rewriting the
19+// context rewrites the environment too. Everything else that carries the
20+// origin (the URLs inside github.event, recorded as the server built them;
21+// repositoryUrl, which is a git:// URL) is left alone.
22+var serverURLKeys = []string{"server_url", "api_url", "graphql_url"}
23+
24+type jsonMember struct {
25+ key string
26+ val json.RawMessage
27+}
28+
29+// RewriteServerURLs returns the contexts JSON of a job with the origin of
30+// github.server_url, github.api_url and github.graphql_url replaced by target
31+// (scheme://host[:port], the ERMITE_RUNNER_URL origin) when, and only when, the
32+// value has the same origin as github.server_url, the server's public origin.
33+// Paths, queries and fragments are kept ("/-/github/api"); values with another
34+// origin, with credentials, or that are not strings are untouched, as is the
35+// rest of the document (order of keys included). The input is returned as it
36+// is when nothing changes.
37+func RewriteServerURLs(contexts, target string) (string, error) {
38+ if target == "" {
39+ return contexts, nil
40+ }
41+ top, err := parseObject([]byte(contexts))
42+ if err != nil {
43+ return "", fmt.Errorf("job contexts: %w", err)
44+ }
45+ gi := -1
46+ for i, m := range top {
47+ if m.key == "github" {
48+ gi = i
49+ }
50+ }
51+ if gi < 0 {
52+ return contexts, nil
53+ }
54+ gh, err := parseObject(top[gi].val)
55+ if err != nil {
56+ return "", fmt.Errorf("job contexts: github: %w", err)
57+ }
58+ str := func(raw json.RawMessage) (string, bool) {
59+ var s string
60+ return s, json.Unmarshal(raw, &s) == nil
61+ }
62+ var public string
63+ for _, m := range gh {
64+ if m.key == "server_url" {
65+ if s, ok := str(m.val); ok {
66+ public = originOf(s)
67+ }
68+ }
69+ }
70+ if public == "" || public == originOf(target) {
71+ return contexts, nil
72+ }
73+ changed := false
74+ for i, m := range gh {
75+ isURL := false
76+ for _, k := range serverURLKeys {
77+ isURL = isURL || m.key == k
78+ }
79+ if !isURL {
80+ continue
81+ }
82+ s, ok := str(m.val)
83+ if !ok || originOf(s) != public {
84+ continue
85+ }
86+ rest := s[strings.Index(s, "://")+3:]
87+ if k := strings.IndexAny(rest, "/?#"); k >= 0 {
88+ rest = rest[k:]
89+ } else {
90+ rest = ""
91+ }
92+ nv, err := marshalNoEscape(target + rest)
93+ if err != nil {
94+ return "", err
95+ }
96+ gh[i].val = nv
97+ changed = true
98+ }
99+ if !changed {
100+ return contexts, nil
101+ }
102+ if top[gi].val, err = encodeObject(gh); err != nil {
103+ return "", err
104+ }
105+ out, err := encodeObject(top)
106+ if err != nil {
107+ return "", err
108+ }
109+ return string(out), nil
110+}
111+
112+// originOf returns "scheme://host[:port]" of an http(s) URL, lower-cased and
113+// without the default port, or "" for anything else (including URLs with
114+// credentials, which are never rewritten).
115+func originOf(raw string) string {
116+ u, err := url.Parse(raw)
117+ if err != nil || u.Host == "" || u.User != nil {
118+ return ""
119+ }
120+ scheme := strings.ToLower(u.Scheme)
121+ if scheme != "http" && scheme != "https" {
122+ return ""
123+ }
124+ host, port := strings.ToLower(u.Hostname()), u.Port()
125+ if port == "" || (scheme == "http" && port == "80") || (scheme == "https" && port == "443") {
126+ return scheme + "://" + host
127+ }
128+ return scheme + "://" + host + ":" + port
129+}
130+
131+func marshalNoEscape(v any) (json.RawMessage, error) {
132+ var b bytes.Buffer
133+ enc := json.NewEncoder(&b)
134+ enc.SetEscapeHTML(false)
135+ if err := enc.Encode(v); err != nil {
136+ return nil, err
137+ }
138+ return bytes.TrimRight(b.Bytes(), "\n"), nil
139+}
140+
141+// parseObject reads a JSON object into its members, in order.
142+func parseObject(b []byte) ([]jsonMember, error) {
143+ dec := json.NewDecoder(bytes.NewReader(b))
144+ if t, err := dec.Token(); err != nil || t != json.Delim('{') {
145+ return nil, errors.New("not a JSON object")
146+ }
147+ var out []jsonMember
148+ for dec.More() {
149+ kt, err := dec.Token()
150+ if err != nil {
151+ return nil, err
152+ }
153+ k, ok := kt.(string)
154+ if !ok {
155+ return nil, errors.New("not a JSON object")
156+ }
157+ var v json.RawMessage
158+ if err := dec.Decode(&v); err != nil {
159+ return nil, err
160+ }
161+ out = append(out, jsonMember{k, v})
162+ }
163+ if _, err := dec.Token(); err != nil {
164+ return nil, err
165+ }
166+ if _, err := dec.Token(); err != io.EOF {
167+ return nil, errors.New("data after the JSON object")
168+ }
169+ return out, nil
170+}
171+
172+func encodeObject(ms []jsonMember) (json.RawMessage, error) {
173+ var b bytes.Buffer
174+ b.WriteByte('{')
175+ for i, m := range ms {
176+ if i > 0 {
177+ b.WriteByte(',')
178+ }
179+ k, err := marshalNoEscape(m.key)
180+ if err != nil {
181+ return nil, err
182+ }
183+ b.Write(k)
184+ b.WriteByte(':')
185+ b.Write(m.val)
186+ }
187+ b.WriteByte('}')
188+ return b.Bytes(), nil
189+}
A internal/host/guestforge_test.go +148 -0
@@ -0,0 +1,148 @@
1+// SPDX-License-Identifier: MIT
2+
3+package host
4+
5+import (
6+ "encoding/json"
7+ "strings"
8+ "testing"
9+
10+ ermite "bnema.dev/ermite-api/gen/ermite"
11+
12+ "bnema.dev/ermite-runner/internal/execstream"
13+ "bnema.dev/ermite-runner/internal/execstream/execpb"
14+)
15+
16+const publicContexts = `{"github":{"job":"","ref":"refs/heads/main","repository":"octo/hello","repositoryUrl":"git://ermite.bnema.dev/octo/hello.git",` +
17+ `"server_url":"https://ermite.bnema.dev","api_url":"https://ermite.bnema.dev/-/github/api","graphql_url":"https://ermite.bnema.dev/-/github/api/graphql",` +
18+ `"event":{"compare":"https://ermite.bnema.dev/octo/hello/compare/a...b","repository":{"html_url":"https://ermite.bnema.dev/octo/hello"}},` +
19+ `"note":"a & <b>"},"needs":{},"vars":{"U":"https://ermite.bnema.dev/x"}}`
20+
21+func TestRewriteServerURLs(t *testing.T) {
22+ got, err := RewriteServerURLs(publicContexts, "http://ermite.bnema.dev:8088")
23+ if err != nil {
24+ t.Fatal(err)
25+ }
26+ want := strings.NewReplacer(
27+ `"server_url":"https://ermite.bnema.dev"`, `"server_url":"http://ermite.bnema.dev:8088"`,
28+ `"api_url":"https://ermite.bnema.dev/-/github/api"`, `"api_url":"http://ermite.bnema.dev:8088/-/github/api"`,
29+ `"graphql_url":"https://ermite.bnema.dev/-/github/api/graphql"`, `"graphql_url":"http://ermite.bnema.dev:8088/-/github/api/graphql"`,
30+ ).Replace(publicContexts)
31+ // Only the three fields change: same bytes everywhere else (key order, the
32+ // event payload as recorded, other URLs, escaping).
33+ if got != want {
34+ t.Fatalf("got\n%s\nwant\n%s", got, want)
35+ }
36+ if !json.Valid([]byte(got)) {
37+ t.Fatal("invalid JSON")
38+ }
39+}
40+
41+func TestRewriteServerURLsMatchesOriginsOnly(t *testing.T) {
42+ const target = "http://forge.internal:8088"
43+ cases := map[string]struct {
44+ in string
45+ same bool
46+ }{
47+ "other origin is untouched": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":"https://api.example.org/x","graphql_url":"https://ermite.bnema.dev:8443/g"}}`, false},
48+ "host prefix is not a match": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":"https://ermite.bnema.dev.evil.test/x","graphql_url":"https://xermite.bnema.dev/g"}}`, false},
49+ "no server_url": {`{"github":{"api_url":"https://ermite.bnema.dev/-/github/api"}}`, true},
50+ "no github": {`{"vars":{"a":"b"}}`, true},
51+ "empty server_url": {`{"github":{"server_url":"","api_url":"https://ermite.bnema.dev/a"}}`, true},
52+ "non-string values": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":null,"graphql_url":3}}`, false},
53+ "already the target": {`{"github":{"server_url":"http://forge.internal:8088","api_url":"http://forge.internal:8088/a"}}`, true},
54+ "credentials are not origins": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":"https://u:p@ermite.bnema.dev/a"}}`, false},
55+ }
56+ for name, c := range cases {
57+ got, err := RewriteServerURLs(c.in, target)
58+ if err != nil {
59+ t.Fatalf("%s: %v", name, err)
60+ }
61+ if c.same && got != c.in {
62+ t.Errorf("%s: changed:\n%s", name, got)
63+ }
64+ var v struct {
65+ GitHub map[string]any `json:"github"`
66+ }
67+ if err := json.Unmarshal([]byte(got), &v); err != nil {
68+ t.Fatalf("%s: %v", name, err)
69+ }
70+ switch name {
71+ case "other origin is untouched", "host prefix is not a match", "credentials are not origins":
72+ if v.GitHub["server_url"] != target {
73+ t.Errorf("%s: server_url %v", name, v.GitHub["server_url"])
74+ }
75+ if u, _ := v.GitHub["api_url"].(string); strings.HasPrefix(u, target) {
76+ t.Errorf("%s: api_url rewritten: %s", name, u)
77+ }
78+ if u, _ := v.GitHub["graphql_url"].(string); strings.HasPrefix(u, target) {
79+ t.Errorf("%s: graphql_url rewritten: %s", name, u)
80+ }
81+ case "non-string values":
82+ if v.GitHub["server_url"] != target || v.GitHub["api_url"] != nil || v.GitHub["graphql_url"] != float64(3) {
83+ t.Errorf("%s: %v", name, v.GitHub)
84+ }
85+ }
86+ }
87+}
88+
89+func TestRewriteServerURLsSpellings(t *testing.T) {
90+ // Case of scheme and host and a default port do not change the origin.
91+ in := `{"github":{"server_url":"HTTPS://Ermite.BNEMA.dev:443/","api_url":"https://ermite.bnema.dev/-/github/api?x=1#f"}}`
92+ got, err := RewriteServerURLs(in, "https://forge.test")
93+ if err != nil {
94+ t.Fatal(err)
95+ }
96+ if got != `{"github":{"server_url":"https://forge.test/","api_url":"https://forge.test/-/github/api?x=1#f"}}` {
97+ t.Fatal(got)
98+ }
99+ // Off: the contexts are returned as received, valid or not.
100+ if got, err := RewriteServerURLs("not json", ""); err != nil || got != "not json" {
101+ t.Fatalf("%q %v", got, err)
102+ }
103+ for _, bad := range []string{"not json", `[1]`, `{"github":[1]}`, `{"github":{}} x`} {
104+ if _, err := RewriteServerURLs(bad, "http://a.test"); err == nil {
105+ t.Errorf("%q accepted", bad)
106+ }
107+ }
108+}
109+
110+func TestOriginOf(t *testing.T) {
111+ for in, want := range map[string]string{
112+ "https://Ermite.example/x": "https://ermite.example", "http://a.test:80": "http://a.test",
113+ "http://a.test:8088/p": "http://a.test:8088", "https://a.test:443": "https://a.test",
114+ "ftp://a.test": "", "a.test": "", "https://u@a.test": "", "": "",
115+ } {
116+ if got := originOf(in); got != want {
117+ t.Errorf("originOf(%q) = %q, want %q", in, got, want)
118+ }
119+ }
120+}
121+
122+// The executor receives the rewritten contexts when the access is on, and the
123+// recorded ones when it is off.
124+func TestJobContextsAreRewrittenOnlyWithGuestForge(t *testing.T) {
125+ for _, on := range []bool{true, false} {
126+ h := newHarness(t, simpleScript("x"))
127+ job := testJob("j1")
128+ job.Contexts = publicContexts
129+ h.srv.jobs = []*ermite.Job{job}
130+ if on {
131+ h.runner.Cfg.GuestForgeOrigin = "http://ermite.bnema.dev:8088"
132+ }
133+ var seen string
134+ inner := h.driver.script
135+ h.driver.script = func(t *testing.T, c *execstream.Conn, run *execpb.Run, cn <-chan *execpb.Cancel) {
136+ seen = run.Job.Contexts
137+ inner(t, c, run, cn)
138+ }
139+ h.run(t)
140+ eventually(t, "CompleteJob", func() bool { return h.srv.completeCount() == 1 })
141+ if got := strings.Contains(seen, `"server_url":"http://ermite.bnema.dev:8088"`); got != on {
142+ t.Errorf("guest forge %v: contexts %s", on, seen)
143+ }
144+ if !on && seen != publicContexts {
145+ t.Errorf("contexts changed while off: %s", seen)
146+ }
147+ }
148+}
@@ -0,0 +1,148 @@
1+// SPDX-License-Identifier: MIT
2+
3+package host
4+
5+import (
6+ "encoding/json"
7+ "strings"
8+ "testing"
9+
10+ ermite "bnema.dev/ermite-api/gen/ermite"
11+
12+ "bnema.dev/ermite-runner/internal/execstream"
13+ "bnema.dev/ermite-runner/internal/execstream/execpb"
14+)
15+
16+const publicContexts = `{"github":{"job":"","ref":"refs/heads/main","repository":"octo/hello","repositoryUrl":"git://ermite.bnema.dev/octo/hello.git",` +
17+ `"server_url":"https://ermite.bnema.dev","api_url":"https://ermite.bnema.dev/-/github/api","graphql_url":"https://ermite.bnema.dev/-/github/api/graphql",` +
18+ `"event":{"compare":"https://ermite.bnema.dev/octo/hello/compare/a...b","repository":{"html_url":"https://ermite.bnema.dev/octo/hello"}},` +
19+ `"note":"a & <b>"},"needs":{},"vars":{"U":"https://ermite.bnema.dev/x"}}`
20+
21+func TestRewriteServerURLs(t *testing.T) {
22+ got, err := RewriteServerURLs(publicContexts, "http://ermite.bnema.dev:8088")
23+ if err != nil {
24+ t.Fatal(err)
25+ }
26+ want := strings.NewReplacer(
27+ `"server_url":"https://ermite.bnema.dev"`, `"server_url":"http://ermite.bnema.dev:8088"`,
28+ `"api_url":"https://ermite.bnema.dev/-/github/api"`, `"api_url":"http://ermite.bnema.dev:8088/-/github/api"`,
29+ `"graphql_url":"https://ermite.bnema.dev/-/github/api/graphql"`, `"graphql_url":"http://ermite.bnema.dev:8088/-/github/api/graphql"`,
30+ ).Replace(publicContexts)
31+ // Only the three fields change: same bytes everywhere else (key order, the
32+ // event payload as recorded, other URLs, escaping).
33+ if got != want {
34+ t.Fatalf("got\n%s\nwant\n%s", got, want)
35+ }
36+ if !json.Valid([]byte(got)) {
37+ t.Fatal("invalid JSON")
38+ }
39+}
40+
41+func TestRewriteServerURLsMatchesOriginsOnly(t *testing.T) {
42+ const target = "http://forge.internal:8088"
43+ cases := map[string]struct {
44+ in string
45+ same bool
46+ }{
47+ "other origin is untouched": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":"https://api.example.org/x","graphql_url":"https://ermite.bnema.dev:8443/g"}}`, false},
48+ "host prefix is not a match": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":"https://ermite.bnema.dev.evil.test/x","graphql_url":"https://xermite.bnema.dev/g"}}`, false},
49+ "no server_url": {`{"github":{"api_url":"https://ermite.bnema.dev/-/github/api"}}`, true},
50+ "no github": {`{"vars":{"a":"b"}}`, true},
51+ "empty server_url": {`{"github":{"server_url":"","api_url":"https://ermite.bnema.dev/a"}}`, true},
52+ "non-string values": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":null,"graphql_url":3}}`, false},
53+ "already the target": {`{"github":{"server_url":"http://forge.internal:8088","api_url":"http://forge.internal:8088/a"}}`, true},
54+ "credentials are not origins": {`{"github":{"server_url":"https://ermite.bnema.dev","api_url":"https://u:p@ermite.bnema.dev/a"}}`, false},
55+ }
56+ for name, c := range cases {
57+ got, err := RewriteServerURLs(c.in, target)
58+ if err != nil {
59+ t.Fatalf("%s: %v", name, err)
60+ }
61+ if c.same && got != c.in {
62+ t.Errorf("%s: changed:\n%s", name, got)
63+ }
64+ var v struct {
65+ GitHub map[string]any `json:"github"`
66+ }
67+ if err := json.Unmarshal([]byte(got), &v); err != nil {
68+ t.Fatalf("%s: %v", name, err)
69+ }
70+ switch name {
71+ case "other origin is untouched", "host prefix is not a match", "credentials are not origins":
72+ if v.GitHub["server_url"] != target {
73+ t.Errorf("%s: server_url %v", name, v.GitHub["server_url"])
74+ }
75+ if u, _ := v.GitHub["api_url"].(string); strings.HasPrefix(u, target) {
76+ t.Errorf("%s: api_url rewritten: %s", name, u)
77+ }
78+ if u, _ := v.GitHub["graphql_url"].(string); strings.HasPrefix(u, target) {
79+ t.Errorf("%s: graphql_url rewritten: %s", name, u)
80+ }
81+ case "non-string values":
82+ if v.GitHub["server_url"] != target || v.GitHub["api_url"] != nil || v.GitHub["graphql_url"] != float64(3) {
83+ t.Errorf("%s: %v", name, v.GitHub)
84+ }
85+ }
86+ }
87+}
88+
89+func TestRewriteServerURLsSpellings(t *testing.T) {
90+ // Case of scheme and host and a default port do not change the origin.
91+ in := `{"github":{"server_url":"HTTPS://Ermite.BNEMA.dev:443/","api_url":"https://ermite.bnema.dev/-/github/api?x=1#f"}}`
92+ got, err := RewriteServerURLs(in, "https://forge.test")
93+ if err != nil {
94+ t.Fatal(err)
95+ }
96+ if got != `{"github":{"server_url":"https://forge.test/","api_url":"https://forge.test/-/github/api?x=1#f"}}` {
97+ t.Fatal(got)
98+ }
99+ // Off: the contexts are returned as received, valid or not.
100+ if got, err := RewriteServerURLs("not json", ""); err != nil || got != "not json" {
101+ t.Fatalf("%q %v", got, err)
102+ }
103+ for _, bad := range []string{"not json", `[1]`, `{"github":[1]}`, `{"github":{}} x`} {
104+ if _, err := RewriteServerURLs(bad, "http://a.test"); err == nil {
105+ t.Errorf("%q accepted", bad)
106+ }
107+ }
108+}
109+
110+func TestOriginOf(t *testing.T) {
111+ for in, want := range map[string]string{
112+ "https://Ermite.example/x": "https://ermite.example", "http://a.test:80": "http://a.test",
113+ "http://a.test:8088/p": "http://a.test:8088", "https://a.test:443": "https://a.test",
114+ "ftp://a.test": "", "a.test": "", "https://u@a.test": "", "": "",
115+ } {
116+ if got := originOf(in); got != want {
117+ t.Errorf("originOf(%q) = %q, want %q", in, got, want)
118+ }
119+ }
120+}
121+
122+// The executor receives the rewritten contexts when the access is on, and the
123+// recorded ones when it is off.
124+func TestJobContextsAreRewrittenOnlyWithGuestForge(t *testing.T) {
125+ for _, on := range []bool{true, false} {
126+ h := newHarness(t, simpleScript("x"))
127+ job := testJob("j1")
128+ job.Contexts = publicContexts
129+ h.srv.jobs = []*ermite.Job{job}
130+ if on {
131+ h.runner.Cfg.GuestForgeOrigin = "http://ermite.bnema.dev:8088"
132+ }
133+ var seen string
134+ inner := h.driver.script
135+ h.driver.script = func(t *testing.T, c *execstream.Conn, run *execpb.Run, cn <-chan *execpb.Cancel) {
136+ seen = run.Job.Contexts
137+ inner(t, c, run, cn)
138+ }
139+ h.run(t)
140+ eventually(t, "CompleteJob", func() bool { return h.srv.completeCount() == 1 })
141+ if got := strings.Contains(seen, `"server_url":"http://ermite.bnema.dev:8088"`); got != on {
142+ t.Errorf("guest forge %v: contexts %s", on, seen)
143+ }
144+ if !on && seen != publicContexts {
145+ t.Errorf("contexts changed while off: %s", seen)
146+ }
147+ }
148+}
M internal/host/job.go +9 -1
@@ -67,6 +67,9 @@ type jobRun struct {
6767 maxStarted int32
6868 lastLogStep int32
6969 violation string
70+ // contexts is Job.Contexts as handed to the executor: with the server URLs
71+ // pointed at the runner's URL when guest forge access is on.
72+ contexts string
7073 // hard stops the sandbox at once; set before the session starts.
7174 hard func()
7275 result *execpb.JobResult
@@ -562,6 +565,11 @@ func failure() outcome { return outcome{conclusion: ermite.CiConclusion_CI_CONCL
562565
563566 // execute starts the sandbox, runs the job in it and returns how it ended.
564567 func (j *jobRun) execute(ctx context.Context, rt *Runner, stopHard <-chan struct{}) outcome {
568+ var err error
569+ if j.contexts, err = RewriteServerURLs(j.job.Contexts, rt.Cfg.GuestForgeOrigin); err != nil {
570+ j.failEarly(fmt.Sprintf("The job's contexts cannot be prepared: %v", err))
571+ return failure()
572+ }
565573 sb, err := rt.Driver.New()
566574 if err != nil {
567575 j.failEarly(fmt.Sprintf("The runner has no free sandbox: %v", err))
@@ -632,7 +640,7 @@ func (j *jobRun) startSandbox(ctx context.Context, rt *Runner, sb sandbox.Sandbo
632640 func (j *jobRun) drive(conn sandbox.Conn, rt *Runner) (*execpb.Exit, error) {
633641 run := &execpb.Run{Job: &execpb.JobSpec{
634642 WorkflowPath: j.job.WorkflowPath, WorkflowYaml: j.job.WorkflowYaml, JobKey: j.job.JobKey,
635- Matrix: j.job.Matrix, Contexts: j.job.Contexts, Repo: j.job.Repo, TimeoutMinutes: int32(j.timeoutMinutes()),
643+ Matrix: j.job.Matrix, Contexts: j.contexts, Repo: j.job.Repo, TimeoutMinutes: int32(j.timeoutMinutes()),
636644 DeadlineMs: max(1, time.Until(j.deadline()).Milliseconds()),
637645 RunnerName: rt.Cfg.Name, RunnerVersion: rt.Cfg.Version, RunnerEnvironment: rt.Cfg.environment(),
638646 }}
@@ -67,6 +67,9 @@ type jobRun struct {
67 maxStarted int3267 maxStarted int32
68 lastLogStep int3268 lastLogStep int32
69 violation string69 violation string
70+ // contexts is Job.Contexts as handed to the executor: with the server URLs
71+ // pointed at the runner's URL when guest forge access is on.
72+ contexts string
70 // hard stops the sandbox at once; set before the session starts.73 // hard stops the sandbox at once; set before the session starts.
71 hard func()74 hard func()
72 result *execpb.JobResult75 result *execpb.JobResult
@@ -562,6 +565,11 @@ func failure() outcome { return outcome{conclusion: ermite.CiConclusion_CI_CONCL
562 565
563 // execute starts the sandbox, runs the job in it and returns how it ended.566 // execute starts the sandbox, runs the job in it and returns how it ended.
564 func (j *jobRun) execute(ctx context.Context, rt *Runner, stopHard <-chan struct{}) outcome {567 func (j *jobRun) execute(ctx context.Context, rt *Runner, stopHard <-chan struct{}) outcome {
568+ var err error
569+ if j.contexts, err = RewriteServerURLs(j.job.Contexts, rt.Cfg.GuestForgeOrigin); err != nil {
570+ j.failEarly(fmt.Sprintf("The job's contexts cannot be prepared: %v", err))
571+ return failure()
572+ }
565 sb, err := rt.Driver.New()573 sb, err := rt.Driver.New()
566 if err != nil {574 if err != nil {
567 j.failEarly(fmt.Sprintf("The runner has no free sandbox: %v", err))575 j.failEarly(fmt.Sprintf("The runner has no free sandbox: %v", err))
@@ -632,7 +640,7 @@ func (j *jobRun) startSandbox(ctx context.Context, rt *Runner, sb sandbox.Sandbo
632 func (j *jobRun) drive(conn sandbox.Conn, rt *Runner) (*execpb.Exit, error) {640 func (j *jobRun) drive(conn sandbox.Conn, rt *Runner) (*execpb.Exit, error) {
633 run := &execpb.Run{Job: &execpb.JobSpec{641 run := &execpb.Run{Job: &execpb.JobSpec{
634 WorkflowPath: j.job.WorkflowPath, WorkflowYaml: j.job.WorkflowYaml, JobKey: j.job.JobKey,642 WorkflowPath: j.job.WorkflowPath, WorkflowYaml: j.job.WorkflowYaml, JobKey: j.job.JobKey,
635- Matrix: j.job.Matrix, Contexts: j.job.Contexts, Repo: j.job.Repo, TimeoutMinutes: int32(j.timeoutMinutes()),643+ Matrix: j.job.Matrix, Contexts: j.contexts, Repo: j.job.Repo, TimeoutMinutes: int32(j.timeoutMinutes()),
636 DeadlineMs: max(1, time.Until(j.deadline()).Milliseconds()),644 DeadlineMs: max(1, time.Until(j.deadline()).Milliseconds()),
637 RunnerName: rt.Cfg.Name, RunnerVersion: rt.Cfg.Version, RunnerEnvironment: rt.Cfg.environment(),645 RunnerName: rt.Cfg.Name, RunnerVersion: rt.Cfg.Version, RunnerEnvironment: rt.Cfg.environment(),
638 }}646 }}
M internal/netpolicy/netns_test.go +49 -0
@@ -72,6 +72,7 @@ func TestRulesetBetweenTwoGuestsAndAForgedSource(t *testing.T) {
7272
7373 gA := guest(t, "ertap0", 0) // 10.200.0.1 / .2
7474 gB := guest(t, "ertap1", 1) // 10.200.0.5 / .6
75+ gA.ForgePort = 8088 // guest A has the guest forge exception, guest B has not
7576 for _, g := range []Guest{gA, gB} {
7677 sh("ip", "addr", "add", g.Gateway.String()+"/30", "dev", g.Iface)
7778 sh("ip", "link", "set", g.Iface, "up")
@@ -188,6 +189,54 @@ except Exception: pass
188189 t.Errorf("TEST-NET-2 reached: %s", got)
189190 }
190191
192+ // Guest forge access: the host listens on 8088 and 8089 on every address
193+ // (like a reverse proxy on *:8088). Guest A, which has the exception for
194+ // 8088, reaches its own gateway there and nowhere else; guest B, which has
195+ // none, reaches nothing.
196+ const tcpServer = `
197+import socket,threading
198+def serve(port):
199+ s=socket.socket(); s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1); s.bind(("0.0.0.0",port)); s.listen(8)
200+ while True:
201+ c,_=s.accept(); c.sendall(b"hello"); c.close()
202+for p in (8088,8089): threading.Thread(target=serve,args=(p,),daemon=True).start()
203+threading.Event().wait()
204+`
205+ tsrv := exec.Command("python3", "-c", tcpServer)
206+ if err := tsrv.Start(); err != nil {
207+ t.Fatal(err)
208+ }
209+ t.Cleanup(func() { _ = tsrv.Process.Kill(); _ = tsrv.Wait() })
210+ time.Sleep(500 * time.Millisecond)
211+ const tcpClient = `
212+import socket,sys
213+s=socket.socket(); s.settimeout(2)
214+try:
215+ s.connect((sys.argv[1],int(sys.argv[2]))); print(s.recv(10).decode())
216+except Exception: print("closed")
217+`
218+ tcp := func(n, dst string, port int) string {
219+ return in(n, "python3", "-c", tcpClient, dst, fmt.Sprint(port))
220+ }
221+ if got := tcp("a", gA.Gateway.String(), 8088); got != "hello" {
222+ t.Errorf("guest A cannot reach its gateway on the forge port: %s", got)
223+ }
224+ if got := tcp("a", gA.Gateway.String(), 8089); got != "closed" {
225+ t.Errorf("guest A reached its gateway on another port: %s", got)
226+ }
227+ if got := tcp("a", gB.Gateway.String(), 8088); got != "closed" {
228+ t.Errorf("guest A reached guest B's gateway on the forge port: %s", got)
229+ }
230+ if got := tcp("a", "203.0.114.1", 8088); got != "closed" {
231+ t.Errorf("guest A reached the host's LAN address on the forge port: %s", got)
232+ }
233+ if got := tcp("b", gB.Gateway.String(), 8088); got != "closed" {
234+ t.Errorf("guest B, without the exception, reached its gateway: %s", got)
235+ }
236+ if got := tcp("b", gA.Gateway.String(), 8088); got != "closed" {
237+ t.Errorf("guest B reached guest A's gateway on the forge port: %s", got)
238+ }
239+
191240 out := seen()
192241 for _, bad := range []string{"forged-src", "a-to-b", "ext-to-a", "doc-range"} {
193242 if strings.Contains(out, bad) {
@@ -72,6 +72,7 @@ func TestRulesetBetweenTwoGuestsAndAForgedSource(t *testing.T) {
72 72
73 gA := guest(t, "ertap0", 0) // 10.200.0.1 / .273 gA := guest(t, "ertap0", 0) // 10.200.0.1 / .2
74 gB := guest(t, "ertap1", 1) // 10.200.0.5 / .674 gB := guest(t, "ertap1", 1) // 10.200.0.5 / .6
75+ gA.ForgePort = 8088 // guest A has the guest forge exception, guest B has not
75 for _, g := range []Guest{gA, gB} {76 for _, g := range []Guest{gA, gB} {
76 sh("ip", "addr", "add", g.Gateway.String()+"/30", "dev", g.Iface)77 sh("ip", "addr", "add", g.Gateway.String()+"/30", "dev", g.Iface)
77 sh("ip", "link", "set", g.Iface, "up")78 sh("ip", "link", "set", g.Iface, "up")
@@ -188,6 +189,54 @@ except Exception: pass
188 t.Errorf("TEST-NET-2 reached: %s", got)189 t.Errorf("TEST-NET-2 reached: %s", got)
189 }190 }
190 191
192+ // Guest forge access: the host listens on 8088 and 8089 on every address
193+ // (like a reverse proxy on *:8088). Guest A, which has the exception for
194+ // 8088, reaches its own gateway there and nowhere else; guest B, which has
195+ // none, reaches nothing.
196+ const tcpServer = `
197+import socket,threading
198+def serve(port):
199+ s=socket.socket(); s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1); s.bind(("0.0.0.0",port)); s.listen(8)
200+ while True:
201+ c,_=s.accept(); c.sendall(b"hello"); c.close()
202+for p in (8088,8089): threading.Thread(target=serve,args=(p,),daemon=True).start()
203+threading.Event().wait()
204+`
205+ tsrv := exec.Command("python3", "-c", tcpServer)
206+ if err := tsrv.Start(); err != nil {
207+ t.Fatal(err)
208+ }
209+ t.Cleanup(func() { _ = tsrv.Process.Kill(); _ = tsrv.Wait() })
210+ time.Sleep(500 * time.Millisecond)
211+ const tcpClient = `
212+import socket,sys
213+s=socket.socket(); s.settimeout(2)
214+try:
215+ s.connect((sys.argv[1],int(sys.argv[2]))); print(s.recv(10).decode())
216+except Exception: print("closed")
217+`
218+ tcp := func(n, dst string, port int) string {
219+ return in(n, "python3", "-c", tcpClient, dst, fmt.Sprint(port))
220+ }
221+ if got := tcp("a", gA.Gateway.String(), 8088); got != "hello" {
222+ t.Errorf("guest A cannot reach its gateway on the forge port: %s", got)
223+ }
224+ if got := tcp("a", gA.Gateway.String(), 8089); got != "closed" {
225+ t.Errorf("guest A reached its gateway on another port: %s", got)
226+ }
227+ if got := tcp("a", gB.Gateway.String(), 8088); got != "closed" {
228+ t.Errorf("guest A reached guest B's gateway on the forge port: %s", got)
229+ }
230+ if got := tcp("a", "203.0.114.1", 8088); got != "closed" {
231+ t.Errorf("guest A reached the host's LAN address on the forge port: %s", got)
232+ }
233+ if got := tcp("b", gB.Gateway.String(), 8088); got != "closed" {
234+ t.Errorf("guest B, without the exception, reached its gateway: %s", got)
235+ }
236+ if got := tcp("b", gA.Gateway.String(), 8088); got != "closed" {
237+ t.Errorf("guest B reached guest A's gateway on the forge port: %s", got)
238+ }
239+
191 out := seen()240 out := seen()
192 for _, bad := range []string{"forged-src", "a-to-b", "ext-to-a", "doc-range"} {241 for _, bad := range []string{"forged-src", "a-to-b", "ext-to-a", "doc-range"} {
193 if strings.Contains(out, bad) {242 if strings.Contains(out, bad) {
M internal/netpolicy/netpolicy.go +14 -0
@@ -12,6 +12,10 @@
1212 // address of that guest, where the runner's DNS forwarder listens; every
1313 // other packet is dropped, and port 53 of a gateway is closed to every
1414 // other interface;
15+// - optionally (Guest.ForgePort, the opt-in guest forge access, staging
16+// only): TCP from the guest's source address to its own gateway address on
17+// that one port. The host's own listener on that port receives the traffic;
18+// nothing is proxied, and every other host port stays closed;
1519 // - IPv6 from the guest is dropped (the guest also disables IPv6);
1620 // - forwarded traffic must carry the guest's own source address;
1721 // - forwarded traffic to RFC 1918, CGNAT 100.64.0.0/10 (VPN overlays),
@@ -73,6 +77,11 @@ type Guest struct {
7377 Iface string // host-side interface: TAP or bridge name
7478 Gateway netip.Addr // host address on Iface; the DNS forwarder listens here
7579 Addr netip.Addr // the guest's only allowed source address
80+ // ForgePort, when not 0, opens TCP from Addr to Gateway on that one port
81+ // (guest forge access). The accept is in the input chain, scoped to this
82+ // guest's interface, source and gateway, so it neither reaches other
83+ // addresses of the host nor other guests.
84+ ForgePort uint16
7685 }
7786
7887 // Validate checks the fields used in generated rules.
@@ -155,6 +164,11 @@ func Ruleset(guests []Guest) string {
155164 for _, proto := range []string{"udp", "tcp"} {
156165 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s %s dport 53 accept\n", g.Iface, g.Addr, g.Gateway, proto)
157166 }
167+ if g.ForgePort != 0 {
168+ // New connections only: the replies leave through the output hook,
169+ // which this table does not touch.
170+ fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s tcp dport %d accept\n", g.Iface, g.Addr, g.Gateway, g.ForgePort)
171+ }
158172 fmt.Fprintf(&b, "\t\tiifname \"%s\" drop\n", g.Iface)
159173 }
160174 // An interface with our prefix that no live guest owns (a leftover) is
@@ -12,6 +12,10 @@
12 // address of that guest, where the runner's DNS forwarder listens; every12 // address of that guest, where the runner's DNS forwarder listens; every
13 // other packet is dropped, and port 53 of a gateway is closed to every13 // other packet is dropped, and port 53 of a gateway is closed to every
14 // other interface;14 // other interface;
15+// - optionally (Guest.ForgePort, the opt-in guest forge access, staging
16+// only): TCP from the guest's source address to its own gateway address on
17+// that one port. The host's own listener on that port receives the traffic;
18+// nothing is proxied, and every other host port stays closed;
15 // - IPv6 from the guest is dropped (the guest also disables IPv6);19 // - IPv6 from the guest is dropped (the guest also disables IPv6);
16 // - forwarded traffic must carry the guest's own source address;20 // - forwarded traffic must carry the guest's own source address;
17 // - forwarded traffic to RFC 1918, CGNAT 100.64.0.0/10 (VPN overlays),21 // - forwarded traffic to RFC 1918, CGNAT 100.64.0.0/10 (VPN overlays),
@@ -73,6 +77,11 @@ type Guest struct {
73 Iface string // host-side interface: TAP or bridge name77 Iface string // host-side interface: TAP or bridge name
74 Gateway netip.Addr // host address on Iface; the DNS forwarder listens here78 Gateway netip.Addr // host address on Iface; the DNS forwarder listens here
75 Addr netip.Addr // the guest's only allowed source address79 Addr netip.Addr // the guest's only allowed source address
80+ // ForgePort, when not 0, opens TCP from Addr to Gateway on that one port
81+ // (guest forge access). The accept is in the input chain, scoped to this
82+ // guest's interface, source and gateway, so it neither reaches other
83+ // addresses of the host nor other guests.
84+ ForgePort uint16
76 }85 }
77 86
78 // Validate checks the fields used in generated rules.87 // Validate checks the fields used in generated rules.
@@ -155,6 +164,11 @@ func Ruleset(guests []Guest) string {
155 for _, proto := range []string{"udp", "tcp"} {164 for _, proto := range []string{"udp", "tcp"} {
156 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s %s dport 53 accept\n", g.Iface, g.Addr, g.Gateway, proto)165 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s %s dport 53 accept\n", g.Iface, g.Addr, g.Gateway, proto)
157 }166 }
167+ if g.ForgePort != 0 {
168+ // New connections only: the replies leave through the output hook,
169+ // which this table does not touch.
170+ fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s tcp dport %d accept\n", g.Iface, g.Addr, g.Gateway, g.ForgePort)
171+ }
158 fmt.Fprintf(&b, "\t\tiifname \"%s\" drop\n", g.Iface)172 fmt.Fprintf(&b, "\t\tiifname \"%s\" drop\n", g.Iface)
159 }173 }
160 // An interface with our prefix that no live guest owns (a leftover) is174 // An interface with our prefix that no live guest owns (a leftover) is
M internal/netpolicy/netpolicy_test.go +60 -0
@@ -232,3 +232,63 @@ func TestManagerRollsBackOnError(t *testing.T) {
232232 }
233233
234234 var _ io.Reader = strings.NewReader("")
235+
236+func forgeGuest(t *testing.T, iface string, slot int, port uint16) Guest {
237+ t.Helper()
238+ g := guest(t, iface, slot)
239+ g.ForgePort = port
240+ return g
241+}
242+
243+func TestRulesetGuestForgeGolden(t *testing.T) {
244+ golden(t, "guest_forge.nft", Ruleset([]Guest{forgeGuest(t, "ertap0", 0, 8088)}))
245+}
246+
247+// Without ForgePort the ruleset has no TCP rule beyond DNS: the exception is
248+// opt-in (and one.nft, the golden file of that case, is unchanged).
249+func TestRulesetWithoutForgePortHasNoForgeRule(t *testing.T) {
250+ rs := Ruleset([]Guest{guest(t, "ertap0", 0)})
251+ if n, dns := strings.Count(rs, "tcp dport"), strings.Count(rs, "tcp dport 53 "); n != dns {
252+ t.Fatalf("%d tcp port rules, %d of them for DNS:\n%s", n, dns, rs)
253+ }
254+}
255+
256+// The exception is one TCP port towards the guest's own gateway, from its own
257+// address and interface, placed before the guest's catch-all drop; a second
258+// guest gets its own, never the other's gateway; nothing else opens, and the
259+// forward chain (private ranges, source checks, sandbox-to-sandbox) is the
260+// same with and without it.
261+func TestGuestForgeRuleScopeAndOrder(t *testing.T) {
262+ gs := []Guest{forgeGuest(t, "ertap0", 0, 8088), forgeGuest(t, "erbr1", 1, 8088), guest(t, "ertap2", 2)}
263+ rs := Ruleset(gs)
264+ in := rs[strings.Index(rs, "chain input"):strings.Index(rs, "chain forward")]
265+ r0 := `iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8088 accept`
266+ r1 := `iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 8088 accept`
267+ for _, want := range []string{r0, r1} {
268+ if strings.Count(in, want) != 1 {
269+ t.Errorf("want exactly one %q in the input chain:\n%s", want, in)
270+ }
271+ }
272+ if strings.Count(rs, "8088") != 2 {
273+ t.Errorf("the port appears %d times, want 2 (one per guest with the exception):\n%s", strings.Count(rs, "8088"), rs)
274+ }
275+ if strings.Index(in, r0) > strings.Index(in, `iifname "ertap0" drop`) || strings.Index(in, r1) > strings.Index(in, `iifname "erbr1" drop`) {
276+ t.Error("the forge accept must precede the guest's catch-all drop")
277+ }
278+ // The guest without the exception has none.
279+ if strings.Contains(in, `iifname "ertap2" ip saddr 10.200.0.10 ip daddr 10.200.0.9 tcp dport 8088`) {
280+ t.Error("ertap2 got a forge rule")
281+ }
282+ // The forward and postrouting chains do not depend on the exception.
283+ plain := Ruleset([]Guest{guest(t, "ertap0", 0), guest(t, "erbr1", 1), guest(t, "ertap2", 2)})
284+ tail := func(s string) string { return s[strings.Index(s, "chain forward"):] }
285+ if tail(rs) != tail(plain) {
286+ t.Error("the forward/postrouting chains changed with the exception")
287+ }
288+ // Every accept of the input chain that is not DNS is one of the two.
289+ for _, l := range strings.Split(in, "\n") {
290+ if strings.HasSuffix(l, "accept") && !strings.Contains(l, "dport 53 accept") && !strings.Contains(l, "tcp dport 8088 accept") {
291+ t.Errorf("unexpected accept: %s", l)
292+ }
293+ }
294+}
@@ -232,3 +232,63 @@ func TestManagerRollsBackOnError(t *testing.T) {
232 }232 }
233 233
234 var _ io.Reader = strings.NewReader("")234 var _ io.Reader = strings.NewReader("")
235+
236+func forgeGuest(t *testing.T, iface string, slot int, port uint16) Guest {
237+ t.Helper()
238+ g := guest(t, iface, slot)
239+ g.ForgePort = port
240+ return g
241+}
242+
243+func TestRulesetGuestForgeGolden(t *testing.T) {
244+ golden(t, "guest_forge.nft", Ruleset([]Guest{forgeGuest(t, "ertap0", 0, 8088)}))
245+}
246+
247+// Without ForgePort the ruleset has no TCP rule beyond DNS: the exception is
248+// opt-in (and one.nft, the golden file of that case, is unchanged).
249+func TestRulesetWithoutForgePortHasNoForgeRule(t *testing.T) {
250+ rs := Ruleset([]Guest{guest(t, "ertap0", 0)})
251+ if n, dns := strings.Count(rs, "tcp dport"), strings.Count(rs, "tcp dport 53 "); n != dns {
252+ t.Fatalf("%d tcp port rules, %d of them for DNS:\n%s", n, dns, rs)
253+ }
254+}
255+
256+// The exception is one TCP port towards the guest's own gateway, from its own
257+// address and interface, placed before the guest's catch-all drop; a second
258+// guest gets its own, never the other's gateway; nothing else opens, and the
259+// forward chain (private ranges, source checks, sandbox-to-sandbox) is the
260+// same with and without it.
261+func TestGuestForgeRuleScopeAndOrder(t *testing.T) {
262+ gs := []Guest{forgeGuest(t, "ertap0", 0, 8088), forgeGuest(t, "erbr1", 1, 8088), guest(t, "ertap2", 2)}
263+ rs := Ruleset(gs)
264+ in := rs[strings.Index(rs, "chain input"):strings.Index(rs, "chain forward")]
265+ r0 := `iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8088 accept`
266+ r1 := `iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 8088 accept`
267+ for _, want := range []string{r0, r1} {
268+ if strings.Count(in, want) != 1 {
269+ t.Errorf("want exactly one %q in the input chain:\n%s", want, in)
270+ }
271+ }
272+ if strings.Count(rs, "8088") != 2 {
273+ t.Errorf("the port appears %d times, want 2 (one per guest with the exception):\n%s", strings.Count(rs, "8088"), rs)
274+ }
275+ if strings.Index(in, r0) > strings.Index(in, `iifname "ertap0" drop`) || strings.Index(in, r1) > strings.Index(in, `iifname "erbr1" drop`) {
276+ t.Error("the forge accept must precede the guest's catch-all drop")
277+ }
278+ // The guest without the exception has none.
279+ if strings.Contains(in, `iifname "ertap2" ip saddr 10.200.0.10 ip daddr 10.200.0.9 tcp dport 8088`) {
280+ t.Error("ertap2 got a forge rule")
281+ }
282+ // The forward and postrouting chains do not depend on the exception.
283+ plain := Ruleset([]Guest{guest(t, "ertap0", 0), guest(t, "erbr1", 1), guest(t, "ertap2", 2)})
284+ tail := func(s string) string { return s[strings.Index(s, "chain forward"):] }
285+ if tail(rs) != tail(plain) {
286+ t.Error("the forward/postrouting chains changed with the exception")
287+ }
288+ // Every accept of the input chain that is not DNS is one of the two.
289+ for _, l := range strings.Split(in, "\n") {
290+ if strings.HasSuffix(l, "accept") && !strings.Contains(l, "dport 53 accept") && !strings.Contains(l, "tcp dport 8088 accept") {
291+ t.Errorf("unexpected accept: %s", l)
292+ }
293+ }
294+}
A internal/netpolicy/testdata/guest_forge.nft +39 -0
@@ -0,0 +1,39 @@
1+table inet ermite_runner
2+delete table inet ermite_runner
3+table inet ermite_runner {
4+ set blocked_v4 {
5+ type ipv4_addr
6+ flags interval
7+ elements = { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4 }
8+ }
9+ chain input {
10+ type filter hook input priority filter; policy accept;
11+ iifname "ertap*" meta nfproto ipv6 drop
12+ iifname "erbr*" meta nfproto ipv6 drop
13+ ip daddr 10.200.0.1 udp dport 53 iifname != "ertap0" drop
14+ ip daddr 10.200.0.1 tcp dport 53 iifname != "ertap0" drop
15+ iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 udp dport 53 accept
16+ iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 53 accept
17+ iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8088 accept
18+ iifname "ertap0" drop
19+ iifname "ertap*" drop
20+ iifname "erbr*" drop
21+ }
22+ chain forward {
23+ type filter hook forward priority filter; policy accept;
24+ iifname "ertap*" meta nfproto ipv6 drop
25+ iifname "erbr*" meta nfproto ipv6 drop
26+ iifname "ertap0" ip saddr != 10.200.0.2 drop
27+ iifname "ertap0" ip daddr @blocked_v4 drop
28+ iifname != { "ertap*", "erbr*" } oifname { "ertap*", "erbr*" } ct state established,related accept
29+ oifname "ertap*" drop
30+ oifname "erbr*" drop
31+ iifname "ertap0" accept
32+ iifname "ertap*" drop
33+ iifname "erbr*" drop
34+ }
35+ chain postrouting {
36+ type nat hook postrouting priority srcnat; policy accept;
37+ ip saddr 10.200.0.2 oifname != "ertap0" masquerade
38+ }
39+}
@@ -0,0 +1,39 @@
1+table inet ermite_runner
2+delete table inet ermite_runner
3+table inet ermite_runner {
4+ set blocked_v4 {
5+ type ipv4_addr
6+ flags interval
7+ elements = { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4 }
8+ }
9+ chain input {
10+ type filter hook input priority filter; policy accept;
11+ iifname "ertap*" meta nfproto ipv6 drop
12+ iifname "erbr*" meta nfproto ipv6 drop
13+ ip daddr 10.200.0.1 udp dport 53 iifname != "ertap0" drop
14+ ip daddr 10.200.0.1 tcp dport 53 iifname != "ertap0" drop
15+ iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 udp dport 53 accept
16+ iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 53 accept
17+ iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8088 accept
18+ iifname "ertap0" drop
19+ iifname "ertap*" drop
20+ iifname "erbr*" drop
21+ }
22+ chain forward {
23+ type filter hook forward priority filter; policy accept;
24+ iifname "ertap*" meta nfproto ipv6 drop
25+ iifname "erbr*" meta nfproto ipv6 drop
26+ iifname "ertap0" ip saddr != 10.200.0.2 drop
27+ iifname "ertap0" ip daddr @blocked_v4 drop
28+ iifname != { "ertap*", "erbr*" } oifname { "ertap*", "erbr*" } ct state established,related accept
29+ oifname "ertap*" drop
30+ oifname "erbr*" drop
31+ iifname "ertap0" accept
32+ iifname "ertap*" drop
33+ iifname "erbr*" drop
34+ }
35+ chain postrouting {
36+ type nat hook postrouting priority srcnat; policy accept;
37+ ip saddr 10.200.0.2 oifname != "ertap0" masquerade
38+ }
39+}
M internal/sandbox/config.go +14 -0
@@ -108,6 +108,13 @@ type Config struct {
108108 DenyPrivileged bool
109109 // LockFile is the host-wide single-instance lock.
110110 LockFile string
111+ // GuestForgeEnabled is ERMITE_RUNNER_GUEST_FORGE=1. The endpoint itself
112+ // comes from the server URL, which is not a sandbox setting: the caller
113+ // sets GuestForge (ResolveGuestForge) before building the drivers.
114+ GuestForgeEnabled bool
115+ // GuestForge, when not nil, is the one host endpoint that every sandbox may
116+ // reach, and the name the DNS forwarder answers with the sandbox's gateway.
117+ GuestForge *GuestForge
111118 // Rate limits of micro-VMs; zero disables.
112119 DiskBandwidthMiB, DiskIOPS, NetMbit int
113120 }
@@ -189,6 +196,13 @@ func LoadConfig(getenv func(string) string) (Config, error) {
189196 default:
190197 return c, fmt.Errorf("%s: %q is not 0 or 1", EnvDenyPrivileged, getenv(EnvDenyPrivileged))
191198 }
199+ switch strings.ToLower(strings.TrimSpace(getenv(EnvGuestForge))) {
200+ case "", "0", "false", "no":
201+ case "1", "true", "yes":
202+ c.GuestForgeEnabled = true
203+ default:
204+ return c, fmt.Errorf("%s: %q is not 0 or 1", EnvGuestForge, getenv(EnvGuestForge))
205+ }
192206 if v := getenv(EnvLockFile); v != "" {
193207 if !filepath.IsAbs(v) {
194208 return c, fmt.Errorf("%s: %q must be an absolute path", EnvLockFile, v)
@@ -108,6 +108,13 @@ type Config struct {
108 DenyPrivileged bool108 DenyPrivileged bool
109 // LockFile is the host-wide single-instance lock.109 // LockFile is the host-wide single-instance lock.
110 LockFile string110 LockFile string
111+ // GuestForgeEnabled is ERMITE_RUNNER_GUEST_FORGE=1. The endpoint itself
112+ // comes from the server URL, which is not a sandbox setting: the caller
113+ // sets GuestForge (ResolveGuestForge) before building the drivers.
114+ GuestForgeEnabled bool
115+ // GuestForge, when not nil, is the one host endpoint that every sandbox may
116+ // reach, and the name the DNS forwarder answers with the sandbox's gateway.
117+ GuestForge *GuestForge
111 // Rate limits of micro-VMs; zero disables.118 // Rate limits of micro-VMs; zero disables.
112 DiskBandwidthMiB, DiskIOPS, NetMbit int119 DiskBandwidthMiB, DiskIOPS, NetMbit int
113 }120 }
@@ -189,6 +196,13 @@ func LoadConfig(getenv func(string) string) (Config, error) {
189 default:196 default:
190 return c, fmt.Errorf("%s: %q is not 0 or 1", EnvDenyPrivileged, getenv(EnvDenyPrivileged))197 return c, fmt.Errorf("%s: %q is not 0 or 1", EnvDenyPrivileged, getenv(EnvDenyPrivileged))
191 }198 }
199+ switch strings.ToLower(strings.TrimSpace(getenv(EnvGuestForge))) {
200+ case "", "0", "false", "no":
201+ case "1", "true", "yes":
202+ c.GuestForgeEnabled = true
203+ default:
204+ return c, fmt.Errorf("%s: %q is not 0 or 1", EnvGuestForge, getenv(EnvGuestForge))
205+ }
192 if v := getenv(EnvLockFile); v != "" {206 if v := getenv(EnvLockFile); v != "" {
193 if !filepath.IsAbs(v) {207 if !filepath.IsAbs(v) {
194 return c, fmt.Errorf("%s: %q must be an absolute path", EnvLockFile, v)208 return c, fmt.Errorf("%s: %q must be an absolute path", EnvLockFile, v)
M internal/sandbox/container/container_test.go +65 -2
@@ -1014,10 +1014,73 @@ func TestSocketPathTooLongIsExplained(t *testing.T) {
10141014 // GO-002: only a podman daemon gets a DNS listener that may precede the gateway
10151015 // address; Docker keeps a normal bind.
10161016 func TestOnlyPodmanGetsAFreeBindDNSListener(t *testing.T) {
1017- if newNet(nil, 0, false).DNS.FreeBind {
1017+ if newNet(nil, 0, false, "").DNS.FreeBind {
10181018 t.Fatal("Docker must keep a normal bind")
10191019 }
1020- if !newNet(nil, 0, true).DNS.FreeBind {
1020+ if !newNet(nil, 0, true, "").DNS.FreeBind {
10211021 t.Fatal("podman needs the listener before the bridge address")
10221022 }
10231023 }
1024+
1025+// Guest forge access: the exception reaches the policy of every guest, and the
1026+// warning naming the origin is logged at the driver start and at every job.
1027+func TestGuestForgeIsAttachedAndWarnedAtEveryJob(t *testing.T) {
1028+ e := newCEnv(t)
1029+ var err error
1030+ if e.cfg.GuestForge, err = sandbox.ParseGuestForge("http://ermite.bnema.dev:8088", true); err != nil {
1031+ t.Fatal(err)
1032+ }
1033+ var buf bytes.Buffer
1034+ ctx := logger(&buf)
1035+ d, err := New(ctx, e.cfg, e.opts())
1036+ if err != nil {
1037+ t.Fatal(err)
1038+ }
1039+ defer func() { _ = d.Close() }()
1040+ msg := "guest forge access: sandboxes may reach http://ermite.bnema.dev:8088 on the host"
1041+ if n := strings.Count(buf.String(), msg); n != 1 {
1042+ t.Fatalf("one warning at driver start, got %d", n)
1043+ }
1044+ for i := range 2 {
1045+ sb, _ := d.New()
1046+ conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: fmt.Sprint("job", i)})
1047+ if err != nil {
1048+ t.Fatal(err)
1049+ }
1050+ for _, g := range e.plumb.Attached {
1051+ if g.ForgePort != 8088 {
1052+ t.Errorf("guest attached with ForgePort %d", g.ForgePort)
1053+ }
1054+ }
1055+ stub(t, conn)
1056+ _ = sb.Stop(ctx)
1057+ }
1058+ if n := strings.Count(buf.String(), msg); n != 3 {
1059+ t.Fatalf("the warning must be logged at every job start (1 + 2), got %d", n)
1060+ }
1061+}
1062+
1063+func TestNoGuestForgeByDefault(t *testing.T) {
1064+ e := newCEnv(t)
1065+ var buf bytes.Buffer
1066+ ctx := logger(&buf)
1067+ d := e.driver(t)
1068+ sb, _ := d.New()
1069+ conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: "j"})
1070+ if err != nil {
1071+ t.Fatal(err)
1072+ }
1073+ for _, g := range e.plumb.Attached {
1074+ if g.ForgePort != 0 {
1075+ t.Errorf("ForgePort %d without the setting", g.ForgePort)
1076+ }
1077+ }
1078+ stub(t, conn)
1079+ _ = sb.Stop(ctx)
1080+ if strings.Contains(buf.String(), "guest forge") {
1081+ t.Fatal("a guest forge warning without the setting")
1082+ }
1083+ if n := newNet(nil, 0, false, "ermite.example.org").DNS.Override; n != "ermite.example.org" {
1084+ t.Fatalf("override %q", n)
1085+ }
1086+}
@@ -1014,10 +1014,73 @@ func TestSocketPathTooLongIsExplained(t *testing.T) {
1014 // GO-002: only a podman daemon gets a DNS listener that may precede the gateway1014 // GO-002: only a podman daemon gets a DNS listener that may precede the gateway
1015 // address; Docker keeps a normal bind.1015 // address; Docker keeps a normal bind.
1016 func TestOnlyPodmanGetsAFreeBindDNSListener(t *testing.T) {1016 func TestOnlyPodmanGetsAFreeBindDNSListener(t *testing.T) {
1017- if newNet(nil, 0, false).DNS.FreeBind {1017+ if newNet(nil, 0, false, "").DNS.FreeBind {
1018 t.Fatal("Docker must keep a normal bind")1018 t.Fatal("Docker must keep a normal bind")
1019 }1019 }
1020- if !newNet(nil, 0, true).DNS.FreeBind {1020+ if !newNet(nil, 0, true, "").DNS.FreeBind {
1021 t.Fatal("podman needs the listener before the bridge address")1021 t.Fatal("podman needs the listener before the bridge address")
1022 }1022 }
1023 }1023 }
1024+
1025+// Guest forge access: the exception reaches the policy of every guest, and the
1026+// warning naming the origin is logged at the driver start and at every job.
1027+func TestGuestForgeIsAttachedAndWarnedAtEveryJob(t *testing.T) {
1028+ e := newCEnv(t)
1029+ var err error
1030+ if e.cfg.GuestForge, err = sandbox.ParseGuestForge("http://ermite.bnema.dev:8088", true); err != nil {
1031+ t.Fatal(err)
1032+ }
1033+ var buf bytes.Buffer
1034+ ctx := logger(&buf)
1035+ d, err := New(ctx, e.cfg, e.opts())
1036+ if err != nil {
1037+ t.Fatal(err)
1038+ }
1039+ defer func() { _ = d.Close() }()
1040+ msg := "guest forge access: sandboxes may reach http://ermite.bnema.dev:8088 on the host"
1041+ if n := strings.Count(buf.String(), msg); n != 1 {
1042+ t.Fatalf("one warning at driver start, got %d", n)
1043+ }
1044+ for i := range 2 {
1045+ sb, _ := d.New()
1046+ conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: fmt.Sprint("job", i)})
1047+ if err != nil {
1048+ t.Fatal(err)
1049+ }
1050+ for _, g := range e.plumb.Attached {
1051+ if g.ForgePort != 8088 {
1052+ t.Errorf("guest attached with ForgePort %d", g.ForgePort)
1053+ }
1054+ }
1055+ stub(t, conn)
1056+ _ = sb.Stop(ctx)
1057+ }
1058+ if n := strings.Count(buf.String(), msg); n != 3 {
1059+ t.Fatalf("the warning must be logged at every job start (1 + 2), got %d", n)
1060+ }
1061+}
1062+
1063+func TestNoGuestForgeByDefault(t *testing.T) {
1064+ e := newCEnv(t)
1065+ var buf bytes.Buffer
1066+ ctx := logger(&buf)
1067+ d := e.driver(t)
1068+ sb, _ := d.New()
1069+ conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: "j"})
1070+ if err != nil {
1071+ t.Fatal(err)
1072+ }
1073+ for _, g := range e.plumb.Attached {
1074+ if g.ForgePort != 0 {
1075+ t.Errorf("ForgePort %d without the setting", g.ForgePort)
1076+ }
1077+ }
1078+ stub(t, conn)
1079+ _ = sb.Stop(ctx)
1080+ if strings.Contains(buf.String(), "guest forge") {
1081+ t.Fatal("a guest forge warning without the setting")
1082+ }
1083+ if n := newNet(nil, 0, false, "ermite.example.org").DNS.Override; n != "ermite.example.org" {
1084+ t.Fatalf("override %q", n)
1085+ }
1086+}
M internal/sandbox/container/driver.go +10 -3
@@ -163,20 +163,24 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
163163 }
164164 d.net = opt.Net
165165 if d.net == nil {
166- d.net = newNet(opt.Run, cfg.DNSPort, info.Podman)
166+ d.net = newNet(opt.Run, cfg.DNSPort, info.Podman, cfg.GuestForge.DNSName())
167167 }
168168 if err := d.net.EnableForwarding(); err != nil {
169169 return fail(err)
170170 }
171+ if cfg.GuestForge != nil {
172+ log.Warn().Msg(cfg.GuestForge.Warning())
173+ }
171174 return d, nil
172175 }
173176
174177 // newNet builds the host network plumbing. Only podman creates the bridge
175178 // address late (see daemonInfo.Podman), so only there the DNS listener may come
176179 // before the address; on Docker a missing gateway address stays a startup error.
177-func newNet(run cmdrun.Runner, dnsPort int, podman bool) *netsetup.Net {
180+func newNet(run cmdrun.Runner, dnsPort int, podman bool, forgeHost string) *netsetup.Net {
178181 n := netsetup.New(run, dnsPort, "")
179182 n.DNS.FreeBind = podman
183+ n.DNS.Override = forgeHost
180184 return n
181185 }
182186
@@ -302,6 +306,9 @@ func (s *containerSandbox) Start(ctx context.Context, spec sandbox.JobSpec) (san
302306 // operator's logs must say what kind of isolation this job got.
303307 log.Warn().Str("job", spec.JobID).Str("container", s.id).Msg(privilegedWarning)
304308 }
309+ if d.cfg.GuestForge != nil {
310+ log.Warn().Str("job", spec.JobID).Str("container", s.id).Msg(d.cfg.GuestForge.Warning())
311+ }
305312 conn, err := s.run(bootCtx, spec)
306313 if err != nil {
307314 var tail string
@@ -421,7 +428,7 @@ func (s *containerSandbox) runNetwork(ctx context.Context, netw netpolicy.Net, l
421428 }
422429 s.netID = netID
423430 s.attached = true
424- return d.net.Attach(ctx, netpolicy.Guest{Iface: s.iface, Gateway: netw.Gateway, Addr: netw.Guest})
431+ return d.net.Attach(ctx, netpolicy.Guest{Iface: s.iface, Gateway: netw.Gateway, Addr: netw.Guest, ForgePort: d.cfg.GuestForge.TCPPort()})
425432 }
426433
427434 // runContainer creates and starts the container.
@@ -163,20 +163,24 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
163 }163 }
164 d.net = opt.Net164 d.net = opt.Net
165 if d.net == nil {165 if d.net == nil {
166- d.net = newNet(opt.Run, cfg.DNSPort, info.Podman)166+ d.net = newNet(opt.Run, cfg.DNSPort, info.Podman, cfg.GuestForge.DNSName())
167 }167 }
168 if err := d.net.EnableForwarding(); err != nil {168 if err := d.net.EnableForwarding(); err != nil {
169 return fail(err)169 return fail(err)
170 }170 }
171+ if cfg.GuestForge != nil {
172+ log.Warn().Msg(cfg.GuestForge.Warning())
173+ }
171 return d, nil174 return d, nil
172 }175 }
173 176
174 // newNet builds the host network plumbing. Only podman creates the bridge177 // newNet builds the host network plumbing. Only podman creates the bridge
175 // address late (see daemonInfo.Podman), so only there the DNS listener may come178 // address late (see daemonInfo.Podman), so only there the DNS listener may come
176 // before the address; on Docker a missing gateway address stays a startup error.179 // before the address; on Docker a missing gateway address stays a startup error.
177-func newNet(run cmdrun.Runner, dnsPort int, podman bool) *netsetup.Net {180+func newNet(run cmdrun.Runner, dnsPort int, podman bool, forgeHost string) *netsetup.Net {
178 n := netsetup.New(run, dnsPort, "")181 n := netsetup.New(run, dnsPort, "")
179 n.DNS.FreeBind = podman182 n.DNS.FreeBind = podman
183+ n.DNS.Override = forgeHost
180 return n184 return n
181 }185 }
182 186
@@ -302,6 +306,9 @@ func (s *containerSandbox) Start(ctx context.Context, spec sandbox.JobSpec) (san
302 // operator's logs must say what kind of isolation this job got.306 // operator's logs must say what kind of isolation this job got.
303 log.Warn().Str("job", spec.JobID).Str("container", s.id).Msg(privilegedWarning)307 log.Warn().Str("job", spec.JobID).Str("container", s.id).Msg(privilegedWarning)
304 }308 }
309+ if d.cfg.GuestForge != nil {
310+ log.Warn().Str("job", spec.JobID).Str("container", s.id).Msg(d.cfg.GuestForge.Warning())
311+ }
305 conn, err := s.run(bootCtx, spec)312 conn, err := s.run(bootCtx, spec)
306 if err != nil {313 if err != nil {
307 var tail string314 var tail string
@@ -421,7 +428,7 @@ func (s *containerSandbox) runNetwork(ctx context.Context, netw netpolicy.Net, l
421 }428 }
422 s.netID = netID429 s.netID = netID
423 s.attached = true430 s.attached = true
424- return d.net.Attach(ctx, netpolicy.Guest{Iface: s.iface, Gateway: netw.Gateway, Addr: netw.Guest})431+ return d.net.Attach(ctx, netpolicy.Guest{Iface: s.iface, Gateway: netw.Gateway, Addr: netw.Guest, ForgePort: d.cfg.GuestForge.TCPPort()})
425 }432 }
426 433
427 // runContainer creates and starts the container.434 // runContainer creates and starts the container.
A internal/sandbox/guestforge.go +156 -0
@@ -0,0 +1,156 @@
1+// SPDX-License-Identifier: MIT
2+
3+package sandbox
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "net/netip"
10+ "net/url"
11+ "strconv"
12+ "strings"
13+ "time"
14+)
15+
16+// EnvGuestForge is ERMITE_RUNNER_GUEST_FORGE=1: sandboxes reach the forge the
17+// way the runner does, that is at the host and port of ERMITE_RUNNER_URL, on
18+// the host. For staging hosts where the forge is not reachable from the
19+// internet side of the NAT (see the README). Off by default.
20+const EnvGuestForge = "ERMITE_RUNNER_GUEST_FORGE"
21+
22+// GuestForge is the one forge endpoint that sandboxes may reach on the host.
23+// It is derived from the server URL by ParseGuestForge.
24+type GuestForge struct {
25+ // Host is the lower-case DNS name of the forge, as in the URL. The DNS
26+ // forwarder answers it with the gateway address of each sandbox.
27+ Host string
28+ // Port is the TCP port: the only host port a sandbox may reach.
29+ Port uint16
30+ // Origin is scheme://host[:port] of the URL: the jobs' server URLs are
31+ // rewritten to it.
32+ Origin string
33+}
34+
35+// Warning is the line logged at start and for each job.
36+func (g *GuestForge) Warning() string {
37+ return "guest forge access: sandboxes may reach " + g.Origin + " on the host"
38+}
39+
40+// DNSName is the overridden name, "" when the access is off. Safe on nil.
41+func (g *GuestForge) DNSName() string {
42+ if g == nil {
43+ return ""
44+ }
45+ return g.Host
46+}
47+
48+// TCPPort is the allowed port, 0 when the access is off. Safe on nil.
49+func (g *GuestForge) TCPPort() uint16 {
50+ if g == nil {
51+ return 0
52+ }
53+ return g.Port
54+}
55+
56+// ParseGuestForge derives the endpoint from the server URL. The URL must have
57+// a DNS host name (an IP literal cannot be redirected by the DNS forwarder),
58+// no path, query or credentials, and the port defaults from the scheme (443,
59+// 80). A plain http URL is accepted only with insecureHTTP (ERMITE_RUNNER_
60+// INSECURE_HTTP=1): the job tokens then cross the host in clear, and that must
61+// be asked for explicitly even for localhost.
62+func ParseGuestForge(raw string, insecureHTTP bool) (*GuestForge, error) {
63+ u, err := url.Parse(raw)
64+ if err != nil || u.Host == "" {
65+ return nil, fmt.Errorf("%s: the server URL %q is not a valid URL", EnvGuestForge, raw)
66+ }
67+ if u.User != nil || u.RawQuery != "" || u.Fragment != "" || (u.Path != "" && u.Path != "/") {
68+ return nil, fmt.Errorf("%s: the server URL must be a bare origin (scheme://host[:port]): the jobs' URLs are rewritten to the origin alone", EnvGuestForge)
69+ }
70+ var def uint16
71+ switch u.Scheme {
72+ case "https":
73+ def = 443
74+ case "http":
75+ def = 80
76+ if !insecureHTTP {
77+ return nil, fmt.Errorf("%s: the server URL is plain http: set ERMITE_RUNNER_INSECURE_HTTP=1 to accept that sandboxes then send their tokens in clear", EnvGuestForge)
78+ }
79+ default:
80+ return nil, fmt.Errorf("%s: the server URL must be http or https", EnvGuestForge)
81+ }
82+ host := strings.ToLower(strings.TrimSuffix(u.Hostname(), "."))
83+ if err := CheckForgeHostname(host); err != nil {
84+ return nil, fmt.Errorf("%s: %w", EnvGuestForge, err)
85+ }
86+ port := def
87+ if p := u.Port(); p != "" {
88+ n, err := strconv.Atoi(p)
89+ if err != nil || n < 1 || n > 65535 {
90+ return nil, fmt.Errorf("%s: port %q is not in 1-65535", EnvGuestForge, p)
91+ }
92+ port = uint16(n)
93+ }
94+ origin := u.Scheme + "://" + host
95+ if u.Port() != "" {
96+ origin += ":" + u.Port()
97+ }
98+ return &GuestForge{Host: host, Port: port, Origin: origin}, nil
99+}
100+
101+// CheckForgeHostname accepts one DNS host name: dot separated labels of ASCII
102+// letters, digits and hyphens (not at either end), at most 63 bytes each and
103+// 253 in all. IP literals, wildcards, underscores, IDNs and empty labels are
104+// refused.
105+func CheckForgeHostname(h string) error {
106+ if h == "" || len(h) > 253 {
107+ return errors.New("the host name is empty or longer than 253 bytes")
108+ }
109+ if _, err := netip.ParseAddr(h); err == nil {
110+ return fmt.Errorf("%q is an IP address: a host name is required (sandboxes are pointed at the forge by DNS)", h)
111+ }
112+ labels := strings.Split(h, ".")
113+ allNumeric := true
114+ for _, l := range labels {
115+ if l == "" || len(l) > 63 || l[0] == '-' || l[len(l)-1] == '-' {
116+ return fmt.Errorf("%q is not a valid host name", h)
117+ }
118+ numeric := true
119+ for _, r := range l {
120+ ok := r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-'
121+ if !ok {
122+ return fmt.Errorf("%q is not a valid host name (wildcards, underscores and non-ASCII are refused)", h)
123+ }
124+ if r < '0' || r > '9' {
125+ numeric = false
126+ }
127+ }
128+ allNumeric = allNumeric && numeric
129+ }
130+ if allNumeric {
131+ // "10.1" or "2130706433": legacy spellings of an address.
132+ return fmt.Errorf("%q looks like an IP address: a host name is required", h)
133+ }
134+ return nil
135+}
136+
137+// ResolveGuestForge is ParseGuestForge plus a check that the host name
138+// resolves on the host (lookup is net.DefaultResolver.LookupHost in
139+// production): the runner reaches the forge by that name, so a name that
140+// resolves to nothing is a configuration error to report at start.
141+func ResolveGuestForge(ctx context.Context, raw string, insecureHTTP bool, lookup func(ctx context.Context, host string) ([]string, error)) (*GuestForge, error) {
142+ g, err := ParseGuestForge(raw, insecureHTTP)
143+ if err != nil {
144+ return nil, err
145+ }
146+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
147+ defer cancel()
148+ addrs, err := lookup(ctx, g.Host)
149+ if err != nil {
150+ return nil, fmt.Errorf("%s: %s does not resolve on this host: %w", EnvGuestForge, g.Host, err)
151+ }
152+ if len(addrs) == 0 {
153+ return nil, fmt.Errorf("%s: %s resolves to no address on this host", EnvGuestForge, g.Host)
154+ }
155+ return g, nil
156+}
@@ -0,0 +1,156 @@
1+// SPDX-License-Identifier: MIT
2+
3+package sandbox
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "net/netip"
10+ "net/url"
11+ "strconv"
12+ "strings"
13+ "time"
14+)
15+
16+// EnvGuestForge is ERMITE_RUNNER_GUEST_FORGE=1: sandboxes reach the forge the
17+// way the runner does, that is at the host and port of ERMITE_RUNNER_URL, on
18+// the host. For staging hosts where the forge is not reachable from the
19+// internet side of the NAT (see the README). Off by default.
20+const EnvGuestForge = "ERMITE_RUNNER_GUEST_FORGE"
21+
22+// GuestForge is the one forge endpoint that sandboxes may reach on the host.
23+// It is derived from the server URL by ParseGuestForge.
24+type GuestForge struct {
25+ // Host is the lower-case DNS name of the forge, as in the URL. The DNS
26+ // forwarder answers it with the gateway address of each sandbox.
27+ Host string
28+ // Port is the TCP port: the only host port a sandbox may reach.
29+ Port uint16
30+ // Origin is scheme://host[:port] of the URL: the jobs' server URLs are
31+ // rewritten to it.
32+ Origin string
33+}
34+
35+// Warning is the line logged at start and for each job.
36+func (g *GuestForge) Warning() string {
37+ return "guest forge access: sandboxes may reach " + g.Origin + " on the host"
38+}
39+
40+// DNSName is the overridden name, "" when the access is off. Safe on nil.
41+func (g *GuestForge) DNSName() string {
42+ if g == nil {
43+ return ""
44+ }
45+ return g.Host
46+}
47+
48+// TCPPort is the allowed port, 0 when the access is off. Safe on nil.
49+func (g *GuestForge) TCPPort() uint16 {
50+ if g == nil {
51+ return 0
52+ }
53+ return g.Port
54+}
55+
56+// ParseGuestForge derives the endpoint from the server URL. The URL must have
57+// a DNS host name (an IP literal cannot be redirected by the DNS forwarder),
58+// no path, query or credentials, and the port defaults from the scheme (443,
59+// 80). A plain http URL is accepted only with insecureHTTP (ERMITE_RUNNER_
60+// INSECURE_HTTP=1): the job tokens then cross the host in clear, and that must
61+// be asked for explicitly even for localhost.
62+func ParseGuestForge(raw string, insecureHTTP bool) (*GuestForge, error) {
63+ u, err := url.Parse(raw)
64+ if err != nil || u.Host == "" {
65+ return nil, fmt.Errorf("%s: the server URL %q is not a valid URL", EnvGuestForge, raw)
66+ }
67+ if u.User != nil || u.RawQuery != "" || u.Fragment != "" || (u.Path != "" && u.Path != "/") {
68+ return nil, fmt.Errorf("%s: the server URL must be a bare origin (scheme://host[:port]): the jobs' URLs are rewritten to the origin alone", EnvGuestForge)
69+ }
70+ var def uint16
71+ switch u.Scheme {
72+ case "https":
73+ def = 443
74+ case "http":
75+ def = 80
76+ if !insecureHTTP {
77+ return nil, fmt.Errorf("%s: the server URL is plain http: set ERMITE_RUNNER_INSECURE_HTTP=1 to accept that sandboxes then send their tokens in clear", EnvGuestForge)
78+ }
79+ default:
80+ return nil, fmt.Errorf("%s: the server URL must be http or https", EnvGuestForge)
81+ }
82+ host := strings.ToLower(strings.TrimSuffix(u.Hostname(), "."))
83+ if err := CheckForgeHostname(host); err != nil {
84+ return nil, fmt.Errorf("%s: %w", EnvGuestForge, err)
85+ }
86+ port := def
87+ if p := u.Port(); p != "" {
88+ n, err := strconv.Atoi(p)
89+ if err != nil || n < 1 || n > 65535 {
90+ return nil, fmt.Errorf("%s: port %q is not in 1-65535", EnvGuestForge, p)
91+ }
92+ port = uint16(n)
93+ }
94+ origin := u.Scheme + "://" + host
95+ if u.Port() != "" {
96+ origin += ":" + u.Port()
97+ }
98+ return &GuestForge{Host: host, Port: port, Origin: origin}, nil
99+}
100+
101+// CheckForgeHostname accepts one DNS host name: dot separated labels of ASCII
102+// letters, digits and hyphens (not at either end), at most 63 bytes each and
103+// 253 in all. IP literals, wildcards, underscores, IDNs and empty labels are
104+// refused.
105+func CheckForgeHostname(h string) error {
106+ if h == "" || len(h) > 253 {
107+ return errors.New("the host name is empty or longer than 253 bytes")
108+ }
109+ if _, err := netip.ParseAddr(h); err == nil {
110+ return fmt.Errorf("%q is an IP address: a host name is required (sandboxes are pointed at the forge by DNS)", h)
111+ }
112+ labels := strings.Split(h, ".")
113+ allNumeric := true
114+ for _, l := range labels {
115+ if l == "" || len(l) > 63 || l[0] == '-' || l[len(l)-1] == '-' {
116+ return fmt.Errorf("%q is not a valid host name", h)
117+ }
118+ numeric := true
119+ for _, r := range l {
120+ ok := r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-'
121+ if !ok {
122+ return fmt.Errorf("%q is not a valid host name (wildcards, underscores and non-ASCII are refused)", h)
123+ }
124+ if r < '0' || r > '9' {
125+ numeric = false
126+ }
127+ }
128+ allNumeric = allNumeric && numeric
129+ }
130+ if allNumeric {
131+ // "10.1" or "2130706433": legacy spellings of an address.
132+ return fmt.Errorf("%q looks like an IP address: a host name is required", h)
133+ }
134+ return nil
135+}
136+
137+// ResolveGuestForge is ParseGuestForge plus a check that the host name
138+// resolves on the host (lookup is net.DefaultResolver.LookupHost in
139+// production): the runner reaches the forge by that name, so a name that
140+// resolves to nothing is a configuration error to report at start.
141+func ResolveGuestForge(ctx context.Context, raw string, insecureHTTP bool, lookup func(ctx context.Context, host string) ([]string, error)) (*GuestForge, error) {
142+ g, err := ParseGuestForge(raw, insecureHTTP)
143+ if err != nil {
144+ return nil, err
145+ }
146+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
147+ defer cancel()
148+ addrs, err := lookup(ctx, g.Host)
149+ if err != nil {
150+ return nil, fmt.Errorf("%s: %s does not resolve on this host: %w", EnvGuestForge, g.Host, err)
151+ }
152+ if len(addrs) == 0 {
153+ return nil, fmt.Errorf("%s: %s resolves to no address on this host", EnvGuestForge, g.Host)
154+ }
155+ return g, nil
156+}
A internal/sandbox/guestforge_test.go +127 -0
@@ -0,0 +1,127 @@
1+// SPDX-License-Identifier: MIT
2+
3+package sandbox
4+
5+import (
6+ "context"
7+ "errors"
8+ "strings"
9+ "testing"
10+)
11+
12+func TestParseGuestForge(t *testing.T) {
13+ ok := map[string]struct {
14+ insecure bool
15+ host, origin string
16+ port uint16
17+ }{
18+ "https://ermite.example.org": {false, "ermite.example.org", "https://ermite.example.org", 443},
19+ "https://Ermite.Example.org:8443": {false, "ermite.example.org", "https://ermite.example.org:8443", 8443},
20+ "https://ermite.example.org/": {false, "ermite.example.org", "https://ermite.example.org", 443},
21+ "https://ermite.example.org.": {false, "ermite.example.org", "https://ermite.example.org", 443},
22+ "http://ermite.bnema.dev:8088": {true, "ermite.bnema.dev", "http://ermite.bnema.dev:8088", 8088},
23+ "http://forge": {true, "forge", "http://forge", 80},
24+ "https://a-b.c1.test:65535": {false, "a-b.c1.test", "https://a-b.c1.test:65535", 65535},
25+ }
26+ for raw, w := range ok {
27+ g, err := ParseGuestForge(raw, w.insecure)
28+ if err != nil {
29+ t.Errorf("%s: %v", raw, err)
30+ continue
31+ }
32+ if g.Host != w.host || g.Port != w.port || g.Origin != w.origin {
33+ t.Errorf("%s: %+v", raw, g)
34+ }
35+ }
36+ bad := map[string]string{
37+ "http://ermite.bnema.dev:8088": "INSECURE_HTTP", // plain http needs the opt-in
38+ "https://10.0.0.1": "IP address",
39+ "https://10.0.0.1:8443": "IP address",
40+ "https://[::1]": "IP address",
41+ "https://[2001:db8::1]:8443": "IP address",
42+ "https://2130706433": "IP address",
43+ "https://127.1": "IP address",
44+ "https://*.example.org": "valid host name",
45+ "https://exa_mple.org": "valid host name",
46+ "https://-a.example.org": "valid host name",
47+ "https://a..example.org": "valid host name",
48+ "https://bücher.example": "valid host name",
49+ "https://" + strings.Repeat("a", 64) + ".test": "valid host name",
50+ "https://ermite.example.org:0": "1-65535",
51+ "https://ermite.example.org:65536": "",
52+ "https://ermite.example.org:99999": "",
53+ "https://ermite.example.org/sub": "bare origin",
54+ "https://ermite.example.org/?q=1": "bare origin",
55+ "https://u:p@ermite.example.org": "bare origin",
56+ "ftp://ermite.example.org": "http or https",
57+ "ermite.example.org": "valid URL",
58+ "": "valid URL",
59+ }
60+ for raw, frag := range bad {
61+ if _, err := ParseGuestForge(raw, false); err == nil || !strings.Contains(err.Error(), frag) || !strings.Contains(err.Error(), EnvGuestForge) {
62+ t.Errorf("%q: %v (want an error with %q)", raw, err, frag)
63+ }
64+ }
65+ // With the opt-in for plain http, the other refusals still apply.
66+ for _, raw := range []string{"http://10.0.0.1:8088", "http://localhost:99999", "http://[::1]:8088"} {
67+ if _, err := ParseGuestForge(raw, true); err == nil {
68+ t.Errorf("%q accepted", raw)
69+ }
70+ }
71+}
72+
73+func TestResolveGuestForge(t *testing.T) {
74+ ctx := context.Background()
75+ var asked string
76+ lookup := func(_ context.Context, h string) ([]string, error) { asked = h; return []string{"127.0.0.1"}, nil }
77+ g, err := ResolveGuestForge(ctx, "http://ermite.bnema.dev:8088", true, lookup)
78+ if err != nil || g.Port != 8088 || asked != "ermite.bnema.dev" {
79+ t.Fatalf("%+v %v %q", g, err, asked)
80+ }
81+ _, err = ResolveGuestForge(ctx, "https://nx.example.org", false, func(context.Context, string) ([]string, error) {
82+ return nil, errors.New("no such host")
83+ })
84+ if err == nil || !strings.Contains(err.Error(), "does not resolve") {
85+ t.Fatalf("%v", err)
86+ }
87+ _, err = ResolveGuestForge(ctx, "https://empty.example.org", false, func(context.Context, string) ([]string, error) { return nil, nil })
88+ if err == nil || !strings.Contains(err.Error(), "no address") {
89+ t.Fatalf("%v", err)
90+ }
91+ called := false
92+ _, err = ResolveGuestForge(ctx, "https://10.0.0.1", false, func(context.Context, string) ([]string, error) { called = true; return nil, nil })
93+ if err == nil || called {
94+ t.Fatalf("an IP literal was looked up or accepted: %v", err)
95+ }
96+}
97+
98+func TestGuestForgeNilIsOff(t *testing.T) {
99+ var g *GuestForge
100+ if g.DNSName() != "" || g.TCPPort() != 0 {
101+ t.Fatal("nil must mean off")
102+ }
103+ g, _ = ParseGuestForge("https://ermite.example.org:8443", false)
104+ if g.Warning() != "guest forge access: sandboxes may reach https://ermite.example.org:8443 on the host" {
105+ t.Fatal(g.Warning())
106+ }
107+}
108+
109+func TestLoadConfigGuestForge(t *testing.T) {
110+ get := func(v string) func(string) string {
111+ return func(k string) string {
112+ if k == EnvGuestForge {
113+ return v
114+ }
115+ return ""
116+ }
117+ }
118+ for v, want := range map[string]bool{"": false, "0": false, "no": false, "1": true, " true ": true, "YES": true} {
119+ c, err := LoadConfig(get(v))
120+ if err != nil || c.GuestForgeEnabled != want {
121+ t.Errorf("%q: %v %v", v, c.GuestForgeEnabled, err)
122+ }
123+ }
124+ if _, err := LoadConfig(get("on")); err == nil || !strings.Contains(err.Error(), EnvGuestForge) {
125+ t.Errorf("a bad value was accepted: %v", err)
126+ }
127+}
@@ -0,0 +1,127 @@
1+// SPDX-License-Identifier: MIT
2+
3+package sandbox
4+
5+import (
6+ "context"
7+ "errors"
8+ "strings"
9+ "testing"
10+)
11+
12+func TestParseGuestForge(t *testing.T) {
13+ ok := map[string]struct {
14+ insecure bool
15+ host, origin string
16+ port uint16
17+ }{
18+ "https://ermite.example.org": {false, "ermite.example.org", "https://ermite.example.org", 443},
19+ "https://Ermite.Example.org:8443": {false, "ermite.example.org", "https://ermite.example.org:8443", 8443},
20+ "https://ermite.example.org/": {false, "ermite.example.org", "https://ermite.example.org", 443},
21+ "https://ermite.example.org.": {false, "ermite.example.org", "https://ermite.example.org", 443},
22+ "http://ermite.bnema.dev:8088": {true, "ermite.bnema.dev", "http://ermite.bnema.dev:8088", 8088},
23+ "http://forge": {true, "forge", "http://forge", 80},
24+ "https://a-b.c1.test:65535": {false, "a-b.c1.test", "https://a-b.c1.test:65535", 65535},
25+ }
26+ for raw, w := range ok {
27+ g, err := ParseGuestForge(raw, w.insecure)
28+ if err != nil {
29+ t.Errorf("%s: %v", raw, err)
30+ continue
31+ }
32+ if g.Host != w.host || g.Port != w.port || g.Origin != w.origin {
33+ t.Errorf("%s: %+v", raw, g)
34+ }
35+ }
36+ bad := map[string]string{
37+ "http://ermite.bnema.dev:8088": "INSECURE_HTTP", // plain http needs the opt-in
38+ "https://10.0.0.1": "IP address",
39+ "https://10.0.0.1:8443": "IP address",
40+ "https://[::1]": "IP address",
41+ "https://[2001:db8::1]:8443": "IP address",
42+ "https://2130706433": "IP address",
43+ "https://127.1": "IP address",
44+ "https://*.example.org": "valid host name",
45+ "https://exa_mple.org": "valid host name",
46+ "https://-a.example.org": "valid host name",
47+ "https://a..example.org": "valid host name",
48+ "https://bücher.example": "valid host name",
49+ "https://" + strings.Repeat("a", 64) + ".test": "valid host name",
50+ "https://ermite.example.org:0": "1-65535",
51+ "https://ermite.example.org:65536": "",
52+ "https://ermite.example.org:99999": "",
53+ "https://ermite.example.org/sub": "bare origin",
54+ "https://ermite.example.org/?q=1": "bare origin",
55+ "https://u:p@ermite.example.org": "bare origin",
56+ "ftp://ermite.example.org": "http or https",
57+ "ermite.example.org": "valid URL",
58+ "": "valid URL",
59+ }
60+ for raw, frag := range bad {
61+ if _, err := ParseGuestForge(raw, false); err == nil || !strings.Contains(err.Error(), frag) || !strings.Contains(err.Error(), EnvGuestForge) {
62+ t.Errorf("%q: %v (want an error with %q)", raw, err, frag)
63+ }
64+ }
65+ // With the opt-in for plain http, the other refusals still apply.
66+ for _, raw := range []string{"http://10.0.0.1:8088", "http://localhost:99999", "http://[::1]:8088"} {
67+ if _, err := ParseGuestForge(raw, true); err == nil {
68+ t.Errorf("%q accepted", raw)
69+ }
70+ }
71+}
72+
73+func TestResolveGuestForge(t *testing.T) {
74+ ctx := context.Background()
75+ var asked string
76+ lookup := func(_ context.Context, h string) ([]string, error) { asked = h; return []string{"127.0.0.1"}, nil }
77+ g, err := ResolveGuestForge(ctx, "http://ermite.bnema.dev:8088", true, lookup)
78+ if err != nil || g.Port != 8088 || asked != "ermite.bnema.dev" {
79+ t.Fatalf("%+v %v %q", g, err, asked)
80+ }
81+ _, err = ResolveGuestForge(ctx, "https://nx.example.org", false, func(context.Context, string) ([]string, error) {
82+ return nil, errors.New("no such host")
83+ })
84+ if err == nil || !strings.Contains(err.Error(), "does not resolve") {
85+ t.Fatalf("%v", err)
86+ }
87+ _, err = ResolveGuestForge(ctx, "https://empty.example.org", false, func(context.Context, string) ([]string, error) { return nil, nil })
88+ if err == nil || !strings.Contains(err.Error(), "no address") {
89+ t.Fatalf("%v", err)
90+ }
91+ called := false
92+ _, err = ResolveGuestForge(ctx, "https://10.0.0.1", false, func(context.Context, string) ([]string, error) { called = true; return nil, nil })
93+ if err == nil || called {
94+ t.Fatalf("an IP literal was looked up or accepted: %v", err)
95+ }
96+}
97+
98+func TestGuestForgeNilIsOff(t *testing.T) {
99+ var g *GuestForge
100+ if g.DNSName() != "" || g.TCPPort() != 0 {
101+ t.Fatal("nil must mean off")
102+ }
103+ g, _ = ParseGuestForge("https://ermite.example.org:8443", false)
104+ if g.Warning() != "guest forge access: sandboxes may reach https://ermite.example.org:8443 on the host" {
105+ t.Fatal(g.Warning())
106+ }
107+}
108+
109+func TestLoadConfigGuestForge(t *testing.T) {
110+ get := func(v string) func(string) string {
111+ return func(k string) string {
112+ if k == EnvGuestForge {
113+ return v
114+ }
115+ return ""
116+ }
117+ }
118+ for v, want := range map[string]bool{"": false, "0": false, "no": false, "1": true, " true ": true, "YES": true} {
119+ c, err := LoadConfig(get(v))
120+ if err != nil || c.GuestForgeEnabled != want {
121+ t.Errorf("%q: %v %v", v, c.GuestForgeEnabled, err)
122+ }
123+ }
124+ if _, err := LoadConfig(get("on")); err == nil || !strings.Contains(err.Error(), EnvGuestForge) {
125+ t.Errorf("a bad value was accepted: %v", err)
126+ }
127+}
M internal/sandbox/vm/driver.go +12 -2
@@ -147,7 +147,13 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
147147 }
148148 d.net = opt.Net
149149 if d.net == nil {
150- d.net = netsetup.New(opt.Run, cfg.DNSPort, "")
150+ n := netsetup.New(opt.Run, cfg.DNSPort, "")
151+ n.DNS.Override = cfg.GuestForge.DNSName()
152+ d.net = n
153+ }
154+ if cfg.GuestForge != nil {
155+ l := zerowrap.FromCtx(ctx)
156+ l.Warn().Msg(cfg.GuestForge.Warning())
151157 }
152158 if err := d.net.EnableForwarding(); err != nil {
153159 return fail(err)
@@ -266,6 +272,10 @@ func (s *vmSandbox) Start(ctx context.Context, spec sandbox.JobSpec) (sandbox.Co
266272 log := zerowrap.FromCtx(ctx)
267273 d := s.d
268274 spec = d.cfg.Resolve(spec)
275+ if d.cfg.GuestForge != nil {
276+ // At every job start, as for privileged containers.
277+ log.Warn().Str("job", spec.JobID).Str("vm", s.id).Msg(d.cfg.GuestForge.Warning())
278+ }
269279
270280 jailed := !d.cfg.JailerOff
271281 runUID := s.uid()
@@ -338,7 +348,7 @@ func (s *vmSandbox) bootNetwork(ctx context.Context, netw netpolicy.Net) error {
338348 return err
339349 }
340350 s.attached = true // from here Detach is idempotent and always safe
341- return d.net.Attach(ctx, netpolicy.Guest{Iface: s.tap, Gateway: netw.Gateway, Addr: netw.Guest})
351+ return d.net.Attach(ctx, netpolicy.Guest{Iface: s.tap, Gateway: netw.Gateway, Addr: netw.Guest, ForgePort: d.cfg.GuestForge.TCPPort()})
342352 }
343353
344354 // bootVMM starts the jailer (or Firecracker) and waits for its API.
@@ -147,7 +147,13 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
147 }147 }
148 d.net = opt.Net148 d.net = opt.Net
149 if d.net == nil {149 if d.net == nil {
150- d.net = netsetup.New(opt.Run, cfg.DNSPort, "")150+ n := netsetup.New(opt.Run, cfg.DNSPort, "")
151+ n.DNS.Override = cfg.GuestForge.DNSName()
152+ d.net = n
153+ }
154+ if cfg.GuestForge != nil {
155+ l := zerowrap.FromCtx(ctx)
156+ l.Warn().Msg(cfg.GuestForge.Warning())
151 }157 }
152 if err := d.net.EnableForwarding(); err != nil {158 if err := d.net.EnableForwarding(); err != nil {
153 return fail(err)159 return fail(err)
@@ -266,6 +272,10 @@ func (s *vmSandbox) Start(ctx context.Context, spec sandbox.JobSpec) (sandbox.Co
266 log := zerowrap.FromCtx(ctx)272 log := zerowrap.FromCtx(ctx)
267 d := s.d273 d := s.d
268 spec = d.cfg.Resolve(spec)274 spec = d.cfg.Resolve(spec)
275+ if d.cfg.GuestForge != nil {
276+ // At every job start, as for privileged containers.
277+ log.Warn().Str("job", spec.JobID).Str("vm", s.id).Msg(d.cfg.GuestForge.Warning())
278+ }
269 279
270 jailed := !d.cfg.JailerOff280 jailed := !d.cfg.JailerOff
271 runUID := s.uid()281 runUID := s.uid()
@@ -338,7 +348,7 @@ func (s *vmSandbox) bootNetwork(ctx context.Context, netw netpolicy.Net) error {
338 return err348 return err
339 }349 }
340 s.attached = true // from here Detach is idempotent and always safe350 s.attached = true // from here Detach is idempotent and always safe
341- return d.net.Attach(ctx, netpolicy.Guest{Iface: s.tap, Gateway: netw.Gateway, Addr: netw.Guest})351+ return d.net.Attach(ctx, netpolicy.Guest{Iface: s.tap, Gateway: netw.Gateway, Addr: netw.Guest, ForgePort: d.cfg.GuestForge.TCPPort()})
342 }352 }
343 353
344 // bootVMM starts the jailer (or Firecracker) and waits for its API.354 // bootVMM starts the jailer (or Firecracker) and waits for its API.
M internal/sandbox/vm/vm_test.go +41 -0
@@ -1307,3 +1307,44 @@ func TestKillAndWaitReportsAProcessThatStays(t *testing.T) {
13071307 t.Fatal("an unkillable process was reported gone")
13081308 }
13091309 }
1310+
1311+// Guest forge access: the exception reaches the policy of the guest, and the
1312+// warning naming the origin is logged at the driver start and at every job.
1313+func TestGuestForgeIsAttachedAndWarnedAtEveryJob(t *testing.T) {
1314+ e := newEnv(t, "")
1315+ var err error
1316+ if e.cfg.GuestForge, err = sandbox.ParseGuestForge("https://ermite.example.org:8443", false); err != nil {
1317+ t.Fatal(err)
1318+ }
1319+ var buf bytes.Buffer
1320+ ctx := zerowrap.WithCtx(context.Background(), zerowrap.New(zerowrap.Config{Level: "debug", Format: "json", Output: &buf}))
1321+ d, err := New(ctx, e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: 5 * time.Second})
1322+ if err != nil {
1323+ t.Fatal(err)
1324+ }
1325+ defer func() { _ = d.Close() }()
1326+ msg := "guest forge access: sandboxes may reach https://ermite.example.org:8443 on the host"
1327+ if n := strings.Count(buf.String(), msg); n != 1 {
1328+ t.Fatalf("one warning at driver start, got %d", n)
1329+ }
1330+ for i := range 2 {
1331+ sb, _ := d.New()
1332+ conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: fmt.Sprint("job", i)})
1333+ if err != nil {
1334+ t.Fatal(err)
1335+ }
1336+ if len(e.plumb.Attached) != 1 {
1337+ t.Fatalf("%d attached", len(e.plumb.Attached))
1338+ }
1339+ for _, g := range e.plumb.Attached {
1340+ if g.ForgePort != 8443 {
1341+ t.Errorf("guest attached with ForgePort %d", g.ForgePort)
1342+ }
1343+ }
1344+ runStub(t, conn)
1345+ _ = sb.Stop(ctx)
1346+ }
1347+ if n := strings.Count(buf.String(), msg); n != 3 {
1348+ t.Fatalf("the warning must be logged at every job start (1 + 2), got %d", n)
1349+ }
1350+}
@@ -1307,3 +1307,44 @@ func TestKillAndWaitReportsAProcessThatStays(t *testing.T) {
1307 t.Fatal("an unkillable process was reported gone")1307 t.Fatal("an unkillable process was reported gone")
1308 }1308 }
1309 }1309 }
1310+
1311+// Guest forge access: the exception reaches the policy of the guest, and the
1312+// warning naming the origin is logged at the driver start and at every job.
1313+func TestGuestForgeIsAttachedAndWarnedAtEveryJob(t *testing.T) {
1314+ e := newEnv(t, "")
1315+ var err error
1316+ if e.cfg.GuestForge, err = sandbox.ParseGuestForge("https://ermite.example.org:8443", false); err != nil {
1317+ t.Fatal(err)
1318+ }
1319+ var buf bytes.Buffer
1320+ ctx := zerowrap.WithCtx(context.Background(), zerowrap.New(zerowrap.Config{Level: "debug", Format: "json", Output: &buf}))
1321+ d, err := New(ctx, e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: 5 * time.Second})
1322+ if err != nil {
1323+ t.Fatal(err)
1324+ }
1325+ defer func() { _ = d.Close() }()
1326+ msg := "guest forge access: sandboxes may reach https://ermite.example.org:8443 on the host"
1327+ if n := strings.Count(buf.String(), msg); n != 1 {
1328+ t.Fatalf("one warning at driver start, got %d", n)
1329+ }
1330+ for i := range 2 {
1331+ sb, _ := d.New()
1332+ conn, err := sb.Start(ctx, sandbox.JobSpec{JobID: fmt.Sprint("job", i)})
1333+ if err != nil {
1334+ t.Fatal(err)
1335+ }
1336+ if len(e.plumb.Attached) != 1 {
1337+ t.Fatalf("%d attached", len(e.plumb.Attached))
1338+ }
1339+ for _, g := range e.plumb.Attached {
1340+ if g.ForgePort != 8443 {
1341+ t.Errorf("guest attached with ForgePort %d", g.ForgePort)
1342+ }
1343+ }
1344+ runStub(t, conn)
1345+ _ = sb.Stop(ctx)
1346+ }
1347+ if n := strings.Count(buf.String(), msg); n != 3 {
1348+ t.Fatalf("the warning must be logged at every job start (1 + 2), got %d", n)
1349+ }
1350+}

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 4

Tags 11

↑↓ moveenter openesc close