detached 5df99e1

feat(cache): serve the cache on the guest loopback over the host channel (#27)

commit
5df99e1eb3235a8a71eb24d177bda5844ddeed51
author
brice · · 2026-10-15 13:55 UTC
committer
bnema · · 2026-10-15 13:55 UTC
parent
9c50523
signature
SSH UnverifiedSHA256:VOCA5lYARHcbKDZ7GhTV45VmcjWnJ6ZQIBazj17LivA The key is unknown
36 files · +2420 -369
Unified
UnifiedSplit
M README.md +8 -4
@@ -58,7 +58,7 @@ Reaping removes every process, device, firewall rule and Docker object the runne
5858
5959 ### Network policy (both modes)
6060
61-One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below, and the cache service port, see "Cache"), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
61+One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below; the cache has no rule, see "Cache"), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
6262
6363 **Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
6464
@@ -79,15 +79,19 @@ The host name and port come from `ERMITE_RUNNER_URL` (port defaulted from the sc
7979
8080 The runner serves the cache of `actions/cache`, `setup-go`/`setup-node` with `cache:` and everything else built on `@actions/cache`. The toolkit uses **cache API v1** for any `GITHUB_SERVER_URL` that is not `github.com`, `*.ghe.com` or `*.localhost` (`toolkit/packages/cache/src/internal/config.ts:1-20`), so that is what is served: `GET _apis/artifactcache/cache?keys=&version=`, `POST caches` (reserve), `PATCH caches/{id}` (chunks with `Content-Range`, in any order), `POST caches/{id}` (commit), and the archive by a plain `GET` of the `archiveLocation`. `ACTIONS_CACHE_SERVICE_V2` is never set.
8181
82-- **Where.** One service per runner, listening on the gateway address of each sandbox on TCP port 8089, in both modes (the container driver and the VM driver share `netsetup`). The nftables input chain accepts that port from the sandbox's own address and interface only, and drops it from every other interface, like port 53 (`internal/netpolicy/testdata/cache.nft`). Node and docker actions get `ACTIONS_CACHE_URL=http://<gateway>:8089/` (the toolkit appends `_apis/…`), `ACTIONS_RUNTIME_URL` (same value) and `ACTIONS_RUNTIME_TOKEN`; `run:` steps get none, as on the official runner. The token is masked. Job containers and docker actions run inside the sandbox and reach the same gateway address through the sandbox's own NAT (the source stays the sandbox's address); this path is covered by the policy tests but has not been exercised in a live sandbox yet.
82+- **Where.** One service per runner, reachable **only on the loopback of the sandbox**, `127.0.0.1:9089`; it listens on no network and on no host port, and the host firewall has no rule for it (nothing to open on the host). A small relay in the sandbox (`internal/guest/relay.go`, started with the job and ended with it) listens on that address and pipes each TCP connection, in both directions with half-close, to a new connection on the host channel the executor already uses, and the runner serves HTTP on the other end:
83+ - **VM:** the vsock port 9089 (the relay connects to the host, CID 2). Firecracker hands the connection to the VM's own Unix socket `<vm dir>/v.sock_9089`, next to the one for the executor (`_1024`); with the jailer both are in the VM's chroot.
84+ - **Container:** `cache.sock` next to `exec.sock` in the per-job directory that is bind-mounted as `/run/ermite`. That directory exists for that container only.
85+
86+ Each sandbox has its own listener and handler on the runner, and the runner accepts on it only the token made for that sandbox's job, so one sandbox can neither reach another's channel nor use a token on it that was not made for its job. The connection caps apply to the channel (16 per sandbox, 64 in all; the relay itself allows 32 open connections). Node and docker actions get `ACTIONS_CACHE_URL=http://127.0.0.1:9089/` (the toolkit appends `_apis/…`), `ACTIONS_RUNTIME_URL` (same value) and `ACTIONS_RUNTIME_TOKEN`; `run:` steps get none, as on the official runner. The token is masked. A docker action or a job container with its own network namespace does not share the sandbox's loopback and does not reach the cache (host networking does); this has not been exercised in a live sandbox. A relay that cannot start (port taken) leaves that job without a cache; the job runs.
8387 - **Storage.** `<state dir>/cache/data/<id>` plus `index.json` (mode 0700/0600), written atomically (temporary file, fsync, rename, directory fsync). Total size is capped by `ERMITE_RUNNER_CACHE_MAX` (default 10G; `500M`, `20G`…): the least recently used entries are evicted first, both on commit and when a reservation needs room. A reservation is dropped after an hour without activity, and in any case at the expiry of the token that made it or an hour after it was made, whichever comes first (one with a chunk in flight is reclaimed when that chunk ends; every transfer's deadline is capped at the token's expiry). A restart keeps the entries and drops unfinished uploads.
8488 - **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref only and reads its read refs, in priority order; keys are tried in the order the toolkit sends them in each ref (the primary key exactly, restore keys as prefixes, newest first). Another repository's entries are never visible. A second reservation of the same (repository, ref, key, version) while one exists or is in flight is a `409`.
85-- **Tokens.** `ACTIONS_RUNTIME_TOKEN` is `base64url(claims).base64url(HMAC-SHA256)` over the job id, repository, write ref, read refs and expiry (job timeout plus 15 min), with a random key made at start and kept in memory only (a restart ends every job anyway). The comparison is constant time; everything wrong (no token, bad MAC, expired, ids of another job or ref) is a bare `403`. The token is not an ermite credential and ermite refuses it. The archive URL carries its own HMAC (entry, repository, expiry) because the toolkit downloads without the bearer token.
89+- **Tokens.** `ACTIONS_RUNTIME_TOKEN` is `base64url(claims).base64url(HMAC-SHA256)` over the job id, repository, write ref, read refs and expiry (job timeout plus 15 min), with a random key made at start and kept in memory only (a restart ends every job anyway). The comparison is constant time; everything wrong (no token, bad MAC, expired, ids of another job or ref) is a bare `403`. The token is not an ermite credential and ermite refuses it. The archive URL carries its own HMAC (entry, repository, issuing job, expiry) because the toolkit downloads without the bearer token; a sandbox's channel serves only URLs issued to its own job, so a URL taken out of one sandbox is refused on every other's.
8690 - **Off.** `ERMITE_RUNNER_CACHE=0`.
8791
8892 **Scopes from the server.** The server sends `Job.cache {repository, write_ref, read_refs}` (feature `ci.cache`). The runner checks it (`internal/host/cache.go`): the repository must be the job's own, and the scope always carries the job's canonical `owner/name`; refs are full `refs/…` names, empty ones are dropped, at most 8 read refs; an empty `write_ref` makes the job read-only (reserve answers `403 cache write denied:`). A scope that fails the checks disables the cache for that job with a log line; the job runs. **Old server fallback:** with no `Job.cache` the scope is `github.repository` and `Job.ref`, for reading and writing, with no base or default branch fallback; a job without either gets no cache.
8993
90-**Limits.** Every entry and reservation is charged at least 4 KiB, so tiny entries cannot pile up; uploads that give no `cacheSize` are charged for the bytes they write, before the bytes are accepted. At most 256 uploads in progress (32 per job); an entry being downloaded is never evicted (its bytes stay charged), and a reservation that cannot be given room answers `503`. At most 16 connections per sandbox and 64 in all; an upload or download that makes no progress for 60 s is cut.
94+**Limits.** Every entry and reservation is charged at least 4 KiB, so tiny entries cannot pile up; uploads that give no `cacheSize` are charged for the bytes they write, before the bytes are accepted. At most 256 uploads in progress (32 per job); an entry being downloaded is never evicted (its bytes stay charged), and a reservation that cannot be given room answers `503`. At most 16 connections per sandbox and 64 in all; an upload or download that makes no progress for 60 s is cut. **Bandwidth:** a VM's cache traffic does not cross `eth0`, so the runner limits it itself: one limit per sandbox, shared by all its connections and both directions (request bodies and replies or archives), at the sandbox's `ERMITE_RUNNER_JOB_NET_BANDWIDTH` (Mbit/s, same default; `0` is unlimited). A transfer over the limit is slowed, not cut. Containers have no per-job network rate limit, so their cache channel is unlimited.
9195
9296 ### Cancellation
9397
@@ -58,7 +58,7 @@ Reaping removes every process, device, firewall rule and Docker object the runne
58 58
59 ### Network policy (both modes)59 ### Network policy (both modes)
60 60
61-One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below, and the cache service port, see "Cache"), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.61+One nftables table, `inet ermite_runner`, replaced atomically (`nft -f -`) and deleted with the last sandbox; nothing else is touched. From a sandbox: to the host only DNS to its own gateway address (plus one TCP port when the opt-in guest forge access is on, below; the cache has no rule, see "Cache"), where the runner's DNS forwarder listens (UDP and TCP, forwarding to the resolvers of `/etc/resolv.conf`; port 53 of a gateway is closed to every other interface); no traffic to RFC 1918, `100.64.0.0/10`, link-local (including `169.254.169.254`), loopback, the documentation, benchmarking and IETF-assigned ranges, multicast or reserved ranges; no IPv6; no traffic between sandboxes; the rest is NATed. A sandbox's own checks (source address, destination ranges) run before any "established" accept, so a guest cannot forge another's flow. The golden rulesets are in `internal/netpolicy/testdata`. The DNS forwarder limits each listener to 32 in-flight queries (UDP beyond that is dropped), 16 TCP connections, and 30 s per TCP connection.
62 62
63 **Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).63 **Side effect on the host:** the runner turns `net.ipv4.ip_forward` on (NAT needs it) and leaves it on when it exits. Forwarding is host-wide, not scoped to the runner's interfaces: if the machine must not route for other networks, restrict it with its own firewall rules (the runner's table only governs traffic of its sandboxes).
64 64
@@ -79,15 +79,19 @@ The host name and port come from `ERMITE_RUNNER_URL` (port defaulted from the sc
79 79
80 The runner serves the cache of `actions/cache`, `setup-go`/`setup-node` with `cache:` and everything else built on `@actions/cache`. The toolkit uses **cache API v1** for any `GITHUB_SERVER_URL` that is not `github.com`, `*.ghe.com` or `*.localhost` (`toolkit/packages/cache/src/internal/config.ts:1-20`), so that is what is served: `GET _apis/artifactcache/cache?keys=&version=`, `POST caches` (reserve), `PATCH caches/{id}` (chunks with `Content-Range`, in any order), `POST caches/{id}` (commit), and the archive by a plain `GET` of the `archiveLocation`. `ACTIONS_CACHE_SERVICE_V2` is never set.80 The runner serves the cache of `actions/cache`, `setup-go`/`setup-node` with `cache:` and everything else built on `@actions/cache`. The toolkit uses **cache API v1** for any `GITHUB_SERVER_URL` that is not `github.com`, `*.ghe.com` or `*.localhost` (`toolkit/packages/cache/src/internal/config.ts:1-20`), so that is what is served: `GET _apis/artifactcache/cache?keys=&version=`, `POST caches` (reserve), `PATCH caches/{id}` (chunks with `Content-Range`, in any order), `POST caches/{id}` (commit), and the archive by a plain `GET` of the `archiveLocation`. `ACTIONS_CACHE_SERVICE_V2` is never set.
81 81
82-- **Where.** One service per runner, listening on the gateway address of each sandbox on TCP port 8089, in both modes (the container driver and the VM driver share `netsetup`). The nftables input chain accepts that port from the sandbox's own address and interface only, and drops it from every other interface, like port 53 (`internal/netpolicy/testdata/cache.nft`). Node and docker actions get `ACTIONS_CACHE_URL=http://<gateway>:8089/` (the toolkit appends `_apis/…`), `ACTIONS_RUNTIME_URL` (same value) and `ACTIONS_RUNTIME_TOKEN`; `run:` steps get none, as on the official runner. The token is masked. Job containers and docker actions run inside the sandbox and reach the same gateway address through the sandbox's own NAT (the source stays the sandbox's address); this path is covered by the policy tests but has not been exercised in a live sandbox yet.82+- **Where.** One service per runner, reachable **only on the loopback of the sandbox**, `127.0.0.1:9089`; it listens on no network and on no host port, and the host firewall has no rule for it (nothing to open on the host). A small relay in the sandbox (`internal/guest/relay.go`, started with the job and ended with it) listens on that address and pipes each TCP connection, in both directions with half-close, to a new connection on the host channel the executor already uses, and the runner serves HTTP on the other end:
83+ - **VM:** the vsock port 9089 (the relay connects to the host, CID 2). Firecracker hands the connection to the VM's own Unix socket `<vm dir>/v.sock_9089`, next to the one for the executor (`_1024`); with the jailer both are in the VM's chroot.
84+ - **Container:** `cache.sock` next to `exec.sock` in the per-job directory that is bind-mounted as `/run/ermite`. That directory exists for that container only.
85+
86+ Each sandbox has its own listener and handler on the runner, and the runner accepts on it only the token made for that sandbox's job, so one sandbox can neither reach another's channel nor use a token on it that was not made for its job. The connection caps apply to the channel (16 per sandbox, 64 in all; the relay itself allows 32 open connections). Node and docker actions get `ACTIONS_CACHE_URL=http://127.0.0.1:9089/` (the toolkit appends `_apis/…`), `ACTIONS_RUNTIME_URL` (same value) and `ACTIONS_RUNTIME_TOKEN`; `run:` steps get none, as on the official runner. The token is masked. A docker action or a job container with its own network namespace does not share the sandbox's loopback and does not reach the cache (host networking does); this has not been exercised in a live sandbox. A relay that cannot start (port taken) leaves that job without a cache; the job runs.
83 - **Storage.** `<state dir>/cache/data/<id>` plus `index.json` (mode 0700/0600), written atomically (temporary file, fsync, rename, directory fsync). Total size is capped by `ERMITE_RUNNER_CACHE_MAX` (default 10G; `500M`, `20G`…): the least recently used entries are evicted first, both on commit and when a reservation needs room. A reservation is dropped after an hour without activity, and in any case at the expiry of the token that made it or an hour after it was made, whichever comes first (one with a chunk in flight is reclaimed when that chunk ends; every transfer's deadline is capped at the token's expiry). A restart keeps the entries and drops unfinished uploads.87 - **Storage.** `<state dir>/cache/data/<id>` plus `index.json` (mode 0700/0600), written atomically (temporary file, fsync, rename, directory fsync). Total size is capped by `ERMITE_RUNNER_CACHE_MAX` (default 10G; `500M`, `20G`…): the least recently used entries are evicted first, both on commit and when a reservation needs room. A reservation is dropped after an hour without activity, and in any case at the expiry of the token that made it or an hour after it was made, whichever comes first (one with a chunk in flight is reclaimed when that chunk ends; every transfer's deadline is capped at the token's expiry). A restart keeps the entries and drops unfinished uploads.
84 - **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref only and reads its read refs, in priority order; keys are tried in the order the toolkit sends them in each ref (the primary key exactly, restore keys as prefixes, newest first). Another repository's entries are never visible. A second reservation of the same (repository, ref, key, version) while one exists or is in flight is a `409`.88 - **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref only and reads its read refs, in priority order; keys are tried in the order the toolkit sends them in each ref (the primary key exactly, restore keys as prefixes, newest first). Another repository's entries are never visible. A second reservation of the same (repository, ref, key, version) while one exists or is in flight is a `409`.
85-- **Tokens.** `ACTIONS_RUNTIME_TOKEN` is `base64url(claims).base64url(HMAC-SHA256)` over the job id, repository, write ref, read refs and expiry (job timeout plus 15 min), with a random key made at start and kept in memory only (a restart ends every job anyway). The comparison is constant time; everything wrong (no token, bad MAC, expired, ids of another job or ref) is a bare `403`. The token is not an ermite credential and ermite refuses it. The archive URL carries its own HMAC (entry, repository, expiry) because the toolkit downloads without the bearer token.89+- **Tokens.** `ACTIONS_RUNTIME_TOKEN` is `base64url(claims).base64url(HMAC-SHA256)` over the job id, repository, write ref, read refs and expiry (job timeout plus 15 min), with a random key made at start and kept in memory only (a restart ends every job anyway). The comparison is constant time; everything wrong (no token, bad MAC, expired, ids of another job or ref) is a bare `403`. The token is not an ermite credential and ermite refuses it. The archive URL carries its own HMAC (entry, repository, issuing job, expiry) because the toolkit downloads without the bearer token; a sandbox's channel serves only URLs issued to its own job, so a URL taken out of one sandbox is refused on every other's.
86 - **Off.** `ERMITE_RUNNER_CACHE=0`.90 - **Off.** `ERMITE_RUNNER_CACHE=0`.
87 91
88 **Scopes from the server.** The server sends `Job.cache {repository, write_ref, read_refs}` (feature `ci.cache`). The runner checks it (`internal/host/cache.go`): the repository must be the job's own, and the scope always carries the job's canonical `owner/name`; refs are full `refs/…` names, empty ones are dropped, at most 8 read refs; an empty `write_ref` makes the job read-only (reserve answers `403 cache write denied:`). A scope that fails the checks disables the cache for that job with a log line; the job runs. **Old server fallback:** with no `Job.cache` the scope is `github.repository` and `Job.ref`, for reading and writing, with no base or default branch fallback; a job without either gets no cache.92 **Scopes from the server.** The server sends `Job.cache {repository, write_ref, read_refs}` (feature `ci.cache`). The runner checks it (`internal/host/cache.go`): the repository must be the job's own, and the scope always carries the job's canonical `owner/name`; refs are full `refs/…` names, empty ones are dropped, at most 8 read refs; an empty `write_ref` makes the job read-only (reserve answers `403 cache write denied:`). A scope that fails the checks disables the cache for that job with a log line; the job runs. **Old server fallback:** with no `Job.cache` the scope is `github.repository` and `Job.ref`, for reading and writing, with no base or default branch fallback; a job without either gets no cache.
89 93
90-**Limits.** Every entry and reservation is charged at least 4 KiB, so tiny entries cannot pile up; uploads that give no `cacheSize` are charged for the bytes they write, before the bytes are accepted. At most 256 uploads in progress (32 per job); an entry being downloaded is never evicted (its bytes stay charged), and a reservation that cannot be given room answers `503`. At most 16 connections per sandbox and 64 in all; an upload or download that makes no progress for 60 s is cut.94+**Limits.** Every entry and reservation is charged at least 4 KiB, so tiny entries cannot pile up; uploads that give no `cacheSize` are charged for the bytes they write, before the bytes are accepted. At most 256 uploads in progress (32 per job); an entry being downloaded is never evicted (its bytes stay charged), and a reservation that cannot be given room answers `503`. At most 16 connections per sandbox and 64 in all; an upload or download that makes no progress for 60 s is cut. **Bandwidth:** a VM's cache traffic does not cross `eth0`, so the runner limits it itself: one limit per sandbox, shared by all its connections and both directions (request bodies and replies or archives), at the sandbox's `ERMITE_RUNNER_JOB_NET_BANDWIDTH` (Mbit/s, same default; `0` is unlimited). A transfer over the limit is slowed, not cut. Containers have no per-job network rate limit, so their cache channel is unlimited.
91 95
92 ### Cancellation96 ### Cancellation
93 97
M host_cmd.go +3 -3
@@ -57,8 +57,8 @@ func setupCache(ctx context.Context, scfg *sandbox.Config) (*cachesvc.Service, e
5757 if err != nil {
5858 return nil, err
5959 }
60- scfg.Cache, scfg.CachePort = svc, cachesvc.DefaultPort
61- log.Info().Int64("max_bytes", max).Uint16("port", scfg.CachePort).Msg("cache service on")
60+ scfg.Cache = svc
61+ log.Info().Int64("max_bytes", max).Msg("cache service on (guest loopback only, no host port)")
6262 return svc, nil
6363 }
6464
@@ -200,7 +200,7 @@ func runCmd(ctx context.Context) int {
200200 return hcfg.Client(url, tok)
201201 },
202202 Features: hcfg.ServerFeaturesFunc(url),
203- Cache: cache, CachePort: scfg.CachePort,
203+ Cache: cache,
204204 }
205205 r.Cfg.URL = url
206206 if err := r.Run(ctx); err != nil {
@@ -57,8 +57,8 @@ func setupCache(ctx context.Context, scfg *sandbox.Config) (*cachesvc.Service, e
57 if err != nil {57 if err != nil {
58 return nil, err58 return nil, err
59 }59 }
60- scfg.Cache, scfg.CachePort = svc, cachesvc.DefaultPort60+ scfg.Cache = svc
61- log.Info().Int64("max_bytes", max).Uint16("port", scfg.CachePort).Msg("cache service on")61+ log.Info().Int64("max_bytes", max).Msg("cache service on (guest loopback only, no host port)")
62 return svc, nil62 return svc, nil
63 }63 }
64 64
@@ -200,7 +200,7 @@ func runCmd(ctx context.Context) int {
200 return hcfg.Client(url, tok)200 return hcfg.Client(url, tok)
201 },201 },
202 Features: hcfg.ServerFeaturesFunc(url),202 Features: hcfg.ServerFeaturesFunc(url),
203- Cache: cache, CachePort: scfg.CachePort,203+ Cache: cache,
204 }204 }
205 r.Cfg.URL = url205 r.Cfg.URL = url
206 if err := r.Run(ctx); err != nil {206 if err := r.Run(ctx); err != nil {
A internal/cachesvc/bandwidth.go +169 -0
@@ -0,0 +1,169 @@
1+// SPDX-License-Identifier: MIT
2+
3+package cachesvc
4+
5+import (
6+ "net"
7+ "sync"
8+ "time"
9+)
10+
11+// bandwidth is the aggregate rate limit of one channel: a token bucket over
12+// time, shared by every connection of the sandbox and by both directions, and
13+// applied to the bytes read from the guest (request bodies) and written to it
14+// (replies, archives).
15+// The cache stream of a micro-VM does not cross its network interface, so the
16+// interface's rate limit does not see it; this one stands in for it.
17+//
18+// A transfer is slowed, never cut: each Read or Write moves at most chunk bytes
19+// and then sleeps for what those bytes cost, so a throttled transfer keeps
20+// making progress in small steps (the progress deadlines of the handlers and the
21+// relay's idle cut of the guest see activity). Sleeping ends when the connection
22+// is closed.
23+type bandwidth struct {
24+ rate float64 // bytes per second
25+ chunk int // largest Read or Write step, in bytes
26+
27+ mu sync.Mutex
28+ free time.Time // when the bucket is empty: the cost of the bytes taken so far
29+}
30+
31+const maxChunk = 32 << 10
32+
33+// newBandwidth returns a limiter of bytesPerSec, or nil (no limit) when it is
34+// not positive.
35+func newBandwidth(bytesPerSec int64) *bandwidth {
36+ if bytesPerSec <= 0 {
37+ return nil
38+ }
39+ chunk := int(min(max(bytesPerSec/10, 512), maxChunk))
40+ // No burst: idle time earns no credit, so a steady transfer stays at the
41+ // rate and an idle channel does not bank one.
42+ return &bandwidth{rate: float64(bytesPerSec), chunk: chunk}
43+}
44+
45+// mbitToBytes converts the Mbit/s of the sandbox configuration (same unit and
46+// meaning as the VM's eth0 limit) to bytes per second; 0 or less is no limit.
47+func mbitToBytes(mbit int) int64 {
48+ if mbit <= 0 {
49+ return 0
50+ }
51+ return int64(mbit) * 1_000_000 / 8
52+}
53+
54+// reserve takes n bytes of the bucket and returns how long to wait before
55+// moving them.
56+func (b *bandwidth) reserve(n int) time.Duration {
57+ now := time.Now()
58+ b.mu.Lock()
59+ defer b.mu.Unlock()
60+ if b.free.Before(now) {
61+ b.free = now
62+ }
63+ b.free = b.free.Add(time.Duration(float64(n) / b.rate * float64(time.Second)))
64+ return b.free.Sub(now)
65+}
66+
67+// wait pays for n bytes: it returns when they are due, or with net.ErrClosed
68+// as soon as done is closed.
69+func (b *bandwidth) wait(done <-chan struct{}, n int) error {
70+ d := b.reserve(n)
71+ if d <= 0 {
72+ return nil
73+ }
74+ t := time.NewTimer(d)
75+ defer t.Stop()
76+ select {
77+ case <-t.C:
78+ return nil
79+ case <-done:
80+ return net.ErrClosed
81+ }
82+}
83+
84+// throttleListener hands out connections that pay into b.
85+type throttleListener struct {
86+ net.Listener
87+ b *bandwidth
88+}
89+
90+func (l throttleListener) Accept() (net.Conn, error) {
91+ c, err := l.Listener.Accept()
92+ if err != nil {
93+ return nil, err
94+ }
95+ return &throttledConn{Conn: c, b: l.b, done: make(chan struct{})}, nil
96+}
97+
98+// throttledConn moves its bytes in steps of at most b.chunk and pays for each
99+// step once it has moved, after the Read or Write itself: the deadlines of the
100+// handlers, set before a call, then cover the transfer and not the wait, and a
101+// step costs at most chunk/rate (about 0.1 s) per connection ahead in line. It
102+// does not implement ReadFrom/WriteTo, so nothing bypasses the steps (no
103+// sendfile).
104+type throttledConn struct {
105+ net.Conn
106+ b *bandwidth
107+ once sync.Once
108+ done chan struct{}
109+}
110+
111+func (c *throttledConn) Read(p []byte) (int, error) {
112+ if len(p) > c.b.chunk {
113+ p = p[:c.b.chunk]
114+ }
115+ n, err := c.Conn.Read(p)
116+ if n > 0 {
117+ if werr := c.b.wait(c.done, n); werr != nil {
118+ return 0, werr
119+ }
120+ }
121+ return n, err
122+}
123+
124+func (c *throttledConn) Write(p []byte) (int, error) {
125+ var sent int
126+ for len(p) > 0 {
127+ step := p
128+ if len(step) > c.b.chunk {
129+ step = step[:c.b.chunk]
130+ }
131+ n, err := c.Conn.Write(step)
132+ sent += n
133+ if err != nil {
134+ return sent, err
135+ }
136+ if err := c.b.wait(c.done, n); err != nil {
137+ return sent, err
138+ }
139+ p = p[len(step):]
140+ }
141+ return sent, nil
142+}
143+
144+func (c *throttledConn) Close() error {
145+ c.once.Do(func() { close(c.done) })
146+ return c.Conn.Close()
147+}
148+
149+// ServeOption configures a channel (Serve).
150+type ServeOption func(*serveConfig)
151+
152+type serveConfig struct {
153+ bytesPerSec int64
154+}
155+
156+// WithBandwidth limits the channel to mbit Mbit/s, summed over all its
157+// connections and both directions. Zero or less means no limit, as for the VM
158+// interface.
159+//
160+// The cache is the one stream of a micro-VM that does not cross its network
161+// interface, so this is its share of the sandbox's NetMbit.
162+func WithBandwidth(mbit int) ServeOption {
163+ return func(c *serveConfig) { c.bytesPerSec = mbitToBytes(mbit) }
164+}
165+
166+// withRate is WithBandwidth in bytes per second (tests use small rates).
167+func withRate(bytesPerSec int64) ServeOption {
168+ return func(c *serveConfig) { c.bytesPerSec = bytesPerSec }
169+}
@@ -0,0 +1,169 @@
1+// SPDX-License-Identifier: MIT
2+
3+package cachesvc
4+
5+import (
6+ "net"
7+ "sync"
8+ "time"
9+)
10+
11+// bandwidth is the aggregate rate limit of one channel: a token bucket over
12+// time, shared by every connection of the sandbox and by both directions, and
13+// applied to the bytes read from the guest (request bodies) and written to it
14+// (replies, archives).
15+// The cache stream of a micro-VM does not cross its network interface, so the
16+// interface's rate limit does not see it; this one stands in for it.
17+//
18+// A transfer is slowed, never cut: each Read or Write moves at most chunk bytes
19+// and then sleeps for what those bytes cost, so a throttled transfer keeps
20+// making progress in small steps (the progress deadlines of the handlers and the
21+// relay's idle cut of the guest see activity). Sleeping ends when the connection
22+// is closed.
23+type bandwidth struct {
24+ rate float64 // bytes per second
25+ chunk int // largest Read or Write step, in bytes
26+
27+ mu sync.Mutex
28+ free time.Time // when the bucket is empty: the cost of the bytes taken so far
29+}
30+
31+const maxChunk = 32 << 10
32+
33+// newBandwidth returns a limiter of bytesPerSec, or nil (no limit) when it is
34+// not positive.
35+func newBandwidth(bytesPerSec int64) *bandwidth {
36+ if bytesPerSec <= 0 {
37+ return nil
38+ }
39+ chunk := int(min(max(bytesPerSec/10, 512), maxChunk))
40+ // No burst: idle time earns no credit, so a steady transfer stays at the
41+ // rate and an idle channel does not bank one.
42+ return &bandwidth{rate: float64(bytesPerSec), chunk: chunk}
43+}
44+
45+// mbitToBytes converts the Mbit/s of the sandbox configuration (same unit and
46+// meaning as the VM's eth0 limit) to bytes per second; 0 or less is no limit.
47+func mbitToBytes(mbit int) int64 {
48+ if mbit <= 0 {
49+ return 0
50+ }
51+ return int64(mbit) * 1_000_000 / 8
52+}
53+
54+// reserve takes n bytes of the bucket and returns how long to wait before
55+// moving them.
56+func (b *bandwidth) reserve(n int) time.Duration {
57+ now := time.Now()
58+ b.mu.Lock()
59+ defer b.mu.Unlock()
60+ if b.free.Before(now) {
61+ b.free = now
62+ }
63+ b.free = b.free.Add(time.Duration(float64(n) / b.rate * float64(time.Second)))
64+ return b.free.Sub(now)
65+}
66+
67+// wait pays for n bytes: it returns when they are due, or with net.ErrClosed
68+// as soon as done is closed.
69+func (b *bandwidth) wait(done <-chan struct{}, n int) error {
70+ d := b.reserve(n)
71+ if d <= 0 {
72+ return nil
73+ }
74+ t := time.NewTimer(d)
75+ defer t.Stop()
76+ select {
77+ case <-t.C:
78+ return nil
79+ case <-done:
80+ return net.ErrClosed
81+ }
82+}
83+
84+// throttleListener hands out connections that pay into b.
85+type throttleListener struct {
86+ net.Listener
87+ b *bandwidth
88+}
89+
90+func (l throttleListener) Accept() (net.Conn, error) {
91+ c, err := l.Listener.Accept()
92+ if err != nil {
93+ return nil, err
94+ }
95+ return &throttledConn{Conn: c, b: l.b, done: make(chan struct{})}, nil
96+}
97+
98+// throttledConn moves its bytes in steps of at most b.chunk and pays for each
99+// step once it has moved, after the Read or Write itself: the deadlines of the
100+// handlers, set before a call, then cover the transfer and not the wait, and a
101+// step costs at most chunk/rate (about 0.1 s) per connection ahead in line. It
102+// does not implement ReadFrom/WriteTo, so nothing bypasses the steps (no
103+// sendfile).
104+type throttledConn struct {
105+ net.Conn
106+ b *bandwidth
107+ once sync.Once
108+ done chan struct{}
109+}
110+
111+func (c *throttledConn) Read(p []byte) (int, error) {
112+ if len(p) > c.b.chunk {
113+ p = p[:c.b.chunk]
114+ }
115+ n, err := c.Conn.Read(p)
116+ if n > 0 {
117+ if werr := c.b.wait(c.done, n); werr != nil {
118+ return 0, werr
119+ }
120+ }
121+ return n, err
122+}
123+
124+func (c *throttledConn) Write(p []byte) (int, error) {
125+ var sent int
126+ for len(p) > 0 {
127+ step := p
128+ if len(step) > c.b.chunk {
129+ step = step[:c.b.chunk]
130+ }
131+ n, err := c.Conn.Write(step)
132+ sent += n
133+ if err != nil {
134+ return sent, err
135+ }
136+ if err := c.b.wait(c.done, n); err != nil {
137+ return sent, err
138+ }
139+ p = p[len(step):]
140+ }
141+ return sent, nil
142+}
143+
144+func (c *throttledConn) Close() error {
145+ c.once.Do(func() { close(c.done) })
146+ return c.Conn.Close()
147+}
148+
149+// ServeOption configures a channel (Serve).
150+type ServeOption func(*serveConfig)
151+
152+type serveConfig struct {
153+ bytesPerSec int64
154+}
155+
156+// WithBandwidth limits the channel to mbit Mbit/s, summed over all its
157+// connections and both directions. Zero or less means no limit, as for the VM
158+// interface.
159+//
160+// The cache is the one stream of a micro-VM that does not cross its network
161+// interface, so this is its share of the sandbox's NetMbit.
162+func WithBandwidth(mbit int) ServeOption {
163+ return func(c *serveConfig) { c.bytesPerSec = mbitToBytes(mbit) }
164+}
165+
166+// withRate is WithBandwidth in bytes per second (tests use small rates).
167+func withRate(bytesPerSec int64) ServeOption {
168+ return func(c *serveConfig) { c.bytesPerSec = bytesPerSec }
169+}
A internal/cachesvc/bandwidth_test.go +234 -0
@@ -0,0 +1,234 @@
1+// SPDX-License-Identifier: MIT
2+
3+package cachesvc
4+
5+import (
6+ "bytes"
7+ "encoding/json"
8+ "fmt"
9+ "io"
10+ "net/http"
11+ "net/url"
12+ "sync"
13+ "testing"
14+ "time"
15+)
16+
17+const (
18+ bwRate = 200_000 // bytes per second of a limited channel
19+ bwSize = 100_000 // bytes per transfer: 0.5 s at bwRate
20+)
21+
22+// bwLocation looks up key k for the job's token on channel c and returns the
23+// archive location.
24+func bwLocation(t *testing.T, c *http.Client, token string) string {
25+ t.Helper()
26+ req, _ := http.NewRequest("GET", "http://cache"+APIPath+"cache?keys=k&version="+v1, nil)
27+ req.Header.Set("Authorization", "Bearer "+token)
28+ resp, err := c.Do(req)
29+ if err != nil {
30+ t.Fatal(err)
31+ }
32+ defer func() { _ = resp.Body.Close() }()
33+ var e struct{ ArchiveLocation string }
34+ if err := json.NewDecoder(resp.Body).Decode(&e); err != nil || resp.StatusCode != 200 {
35+ t.Fatalf("lookup: %d %v", resp.StatusCode, err)
36+ }
37+ return e.ArchiveLocation
38+}
39+
40+// hangGuard is the generous upper bound of a transfer: it only catches a hang,
41+// never a slow machine.
42+const hangGuard = 8 * time.Second
43+
44+// limiterOf returns the limiter installed on the channel id (nil: unlimited).
45+func limiterOf(t *testing.T, svc *Service, id string) *bandwidth {
46+ t.Helper()
47+ svc.mu.Lock()
48+ defer svc.mu.Unlock()
49+ l := svc.channels[id]
50+ if l == nil {
51+ t.Fatalf("no channel %s", id)
52+ }
53+ return l.bw
54+}
55+
56+// bucketUsed reports whether any byte was ever taken from b.
57+func bucketUsed(b *bandwidth) bool {
58+ b.mu.Lock()
59+ defer b.mu.Unlock()
60+ return !b.free.IsZero()
61+}
62+
63+// bwDownload fetches the archive and returns how long the whole transfer took.
64+func bwDownload(t *testing.T, c *http.Client, loc string) time.Duration {
65+ t.Helper()
66+ start := time.Now()
67+ code, body := archiveStatus(t, c, "GET", loc)
68+ if code != 200 || len(body) != bwSize {
69+ t.Errorf("download: %d, %d bytes", code, len(body))
70+ }
71+ return time.Since(start)
72+}
73+
74+// bwRig has one stored archive of bwSize bytes (job-a, key k).
75+func bwRig(t *testing.T) (*rig, string) {
76+ t.Helper()
77+ r := newRig(t, 1<<22)
78+ j := r.job("job-a", repoA, main_, main_)
79+ // One chunk (save would send 4-byte ones).
80+ code, id := j.reserve("k", v1, bwSize)
81+ if code != http.StatusCreated {
82+ t.Fatalf("reserve: %d", code)
83+ }
84+ if code := j.chunk(id, bytes.Repeat([]byte("x"), bwSize), 0); code != http.StatusNoContent {
85+ t.Fatalf("chunk: %d", code)
86+ }
87+ if code := j.commit(id, bwSize); code != http.StatusNoContent {
88+ t.Fatalf("commit: %d", code)
89+ }
90+ return r, j.token
91+}
92+
93+// A channel with a rate serves a download at about that rate: neither much
94+// faster (the limit holds) nor stalled (the transfer keeps going to the end).
95+func TestBandwidthLimitsADownload(t *testing.T) {
96+ r, tok := bwRig(t)
97+ c := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
98+ loc := bwLocation(t, c, tok)
99+ got := bwDownload(t, c, loc)
100+ want := time.Second * bwSize / bwRate
101+ if got < want*8/10 || got > want+hangGuard {
102+ t.Fatalf("download took %v at %d B/s, want at least %v", got, bwRate, want*8/10)
103+ }
104+}
105+
106+// The request bodies are limited too.
107+func TestBandwidthLimitsAnUpload(t *testing.T) {
108+ r := newRig(t, 1<<22)
109+ c := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
110+ j := r.job("job-a", repoA, main_, main_)
111+ code, id := j.reserve("k", v1, bwSize)
112+ if code != http.StatusCreated {
113+ t.Fatalf("reserve: %d", code)
114+ }
115+ req, _ := http.NewRequest("PATCH", "http://cache"+APIPath+fmt.Sprintf("caches/%d", id), bytes.NewReader(bytes.Repeat([]byte("x"), bwSize)))
116+ req.Header.Set("Authorization", "Bearer "+j.token)
117+ req.Header.Set("Content-Range", fmt.Sprintf("bytes 0-%d/*", bwSize-1))
118+ start := time.Now()
119+ resp, err := c.Do(req)
120+ if err != nil {
121+ t.Fatal(err)
122+ }
123+ _ = resp.Body.Close()
124+ got := time.Since(start)
125+ if resp.StatusCode != http.StatusNoContent {
126+ t.Fatalf("upload: %d", resp.StatusCode)
127+ }
128+ want := time.Second * bwSize / bwRate
129+ if got < want*8/10 || got > want+hangGuard {
130+ t.Fatalf("upload took %v at %d B/s, want at least %v", got, bwRate, want*8/10)
131+ }
132+ if code := j.commit(id, bwSize); code != http.StatusNoContent {
133+ t.Fatalf("commit: %d", code)
134+ }
135+}
136+
137+// The limit is the sandbox's, not the connection's: two simultaneous downloads
138+// on one channel take as long as the two in a row at that rate would, and a
139+// channel of another sandbox, with no limit, is not slowed by them.
140+func TestBandwidthIsSharedByTheConnectionsOfASandboxOnly(t *testing.T) {
141+ r, tok := bwRig(t)
142+ a := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
143+ b := channel(t, r.svc, "er0000000b", "job-a") // same job id, so the same URL; no limit
144+ loc := bwLocation(t, b, tok)
145+
146+ if limiterOf(t, r.svc, "er0000000a") == nil || limiterOf(t, r.svc, "er0000000b") != nil {
147+ t.Fatal("A must be limited and B must have no limiter")
148+ }
149+
150+ start := time.Now()
151+ var wg sync.WaitGroup
152+ for range 2 {
153+ wg.Add(1)
154+ go func() { defer wg.Done(); bwDownload(t, a, loc) }()
155+ }
156+ bwDownload(t, b, loc) // B has no limiter, so it never waits on A's bucket
157+ wg.Wait()
158+ total := time.Since(start)
159+
160+ want := time.Second * 2 * bwSize / bwRate // 1 s for the two; 0.5 s if each had the rate
161+ if total < want*8/10 || total > want+hangGuard {
162+ t.Fatalf("two downloads took %v in all, want at least %v", total, want*8/10)
163+ }
164+}
165+
166+// Two channels with a rate each do not draw on one bucket: the limit is per
167+// Serve, so two sandboxes at once take the time of one.
168+func TestBandwidthIsPerChannel(t *testing.T) {
169+ r, tok := bwRig(t)
170+ a := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
171+ b := channel(t, r.svc, "er0000000b", "job-a", withRate(bwRate))
172+ loc := bwLocation(t, a, tok)
173+ la, lb := limiterOf(t, r.svc, "er0000000a"), limiterOf(t, r.svc, "er0000000b")
174+ if la == nil || lb == nil || la == lb {
175+ t.Fatal("each channel must have its own limiter")
176+ }
177+ // Consuming from A leaves B's bucket untouched.
178+ bwDownload(t, a, loc)
179+ if !bucketUsed(la) || bucketUsed(lb) {
180+ t.Fatalf("after a download on A only: A used=%v, B used=%v", bucketUsed(la), bucketUsed(lb))
181+ }
182+ bwDownload(t, b, loc)
183+ if !bucketUsed(lb) {
184+ t.Fatal("B's own download did not draw on B's bucket")
185+ }
186+}
187+
188+// Without a rate nothing is limited or wrapped, and a number of Mbit/s is
189+// bytes per second the way the VM interface counts it.
190+func TestBandwidthUnsetIsUnlimited(t *testing.T) {
191+ if newBandwidth(mbitToBytes(0)) != nil || newBandwidth(mbitToBytes(-1)) != nil {
192+ t.Fatal("0 must mean unlimited")
193+ }
194+ if got := mbitToBytes(1000); got != 125_000_000 {
195+ t.Fatalf("1000 Mbit/s = %d B/s", got)
196+ }
197+ r, tok := bwRig(t)
198+ c := channel(t, r.svc, "er0000000a", "job-a", WithBandwidth(0))
199+ loc := bwLocation(t, c, tok)
200+ if limiterOf(t, r.svc, "er0000000a") != nil {
201+ t.Fatal("WithBandwidth(0) installed a limiter")
202+ }
203+ if got := bwDownload(t, c, loc); got > hangGuard {
204+ t.Fatalf("unlimited download took %v", got)
205+ }
206+}
207+
208+// A throttled transfer is cut when its connection is, not left sleeping.
209+func TestBandwidthSleepEndsWithTheConnection(t *testing.T) {
210+ r, tok := bwRig(t)
211+ c := channel(t, r.svc, "er0000000a", "job-a", withRate(5_000)) // 20 s for the archive
212+ loc := bwLocation(t, c, tok)
213+ u, err := url.Parse(loc)
214+ if err != nil {
215+ t.Fatal(err)
216+ }
217+ req, _ := http.NewRequest("GET", "http://cache"+u.RequestURI(), nil)
218+ resp, err := c.Do(req)
219+ if err != nil {
220+ t.Fatal(err)
221+ }
222+ buf := make([]byte, 1000)
223+ if _, err := io.ReadFull(resp.Body, buf); err != nil {
224+ t.Fatalf("the throttled transfer does not make progress: %v", err)
225+ }
226+ _ = resp.Body.Close()
227+ done := make(chan struct{})
228+ go func() { r.svc.Stop("er0000000a"); close(done) }()
229+ select {
230+ case <-done:
231+ case <-time.After(5 * time.Second):
232+ t.Fatal("Stop waits for a throttled transfer to finish")
233+ }
234+}
@@ -0,0 +1,234 @@
1+// SPDX-License-Identifier: MIT
2+
3+package cachesvc
4+
5+import (
6+ "bytes"
7+ "encoding/json"
8+ "fmt"
9+ "io"
10+ "net/http"
11+ "net/url"
12+ "sync"
13+ "testing"
14+ "time"
15+)
16+
17+const (
18+ bwRate = 200_000 // bytes per second of a limited channel
19+ bwSize = 100_000 // bytes per transfer: 0.5 s at bwRate
20+)
21+
22+// bwLocation looks up key k for the job's token on channel c and returns the
23+// archive location.
24+func bwLocation(t *testing.T, c *http.Client, token string) string {
25+ t.Helper()
26+ req, _ := http.NewRequest("GET", "http://cache"+APIPath+"cache?keys=k&version="+v1, nil)
27+ req.Header.Set("Authorization", "Bearer "+token)
28+ resp, err := c.Do(req)
29+ if err != nil {
30+ t.Fatal(err)
31+ }
32+ defer func() { _ = resp.Body.Close() }()
33+ var e struct{ ArchiveLocation string }
34+ if err := json.NewDecoder(resp.Body).Decode(&e); err != nil || resp.StatusCode != 200 {
35+ t.Fatalf("lookup: %d %v", resp.StatusCode, err)
36+ }
37+ return e.ArchiveLocation
38+}
39+
40+// hangGuard is the generous upper bound of a transfer: it only catches a hang,
41+// never a slow machine.
42+const hangGuard = 8 * time.Second
43+
44+// limiterOf returns the limiter installed on the channel id (nil: unlimited).
45+func limiterOf(t *testing.T, svc *Service, id string) *bandwidth {
46+ t.Helper()
47+ svc.mu.Lock()
48+ defer svc.mu.Unlock()
49+ l := svc.channels[id]
50+ if l == nil {
51+ t.Fatalf("no channel %s", id)
52+ }
53+ return l.bw
54+}
55+
56+// bucketUsed reports whether any byte was ever taken from b.
57+func bucketUsed(b *bandwidth) bool {
58+ b.mu.Lock()
59+ defer b.mu.Unlock()
60+ return !b.free.IsZero()
61+}
62+
63+// bwDownload fetches the archive and returns how long the whole transfer took.
64+func bwDownload(t *testing.T, c *http.Client, loc string) time.Duration {
65+ t.Helper()
66+ start := time.Now()
67+ code, body := archiveStatus(t, c, "GET", loc)
68+ if code != 200 || len(body) != bwSize {
69+ t.Errorf("download: %d, %d bytes", code, len(body))
70+ }
71+ return time.Since(start)
72+}
73+
74+// bwRig has one stored archive of bwSize bytes (job-a, key k).
75+func bwRig(t *testing.T) (*rig, string) {
76+ t.Helper()
77+ r := newRig(t, 1<<22)
78+ j := r.job("job-a", repoA, main_, main_)
79+ // One chunk (save would send 4-byte ones).
80+ code, id := j.reserve("k", v1, bwSize)
81+ if code != http.StatusCreated {
82+ t.Fatalf("reserve: %d", code)
83+ }
84+ if code := j.chunk(id, bytes.Repeat([]byte("x"), bwSize), 0); code != http.StatusNoContent {
85+ t.Fatalf("chunk: %d", code)
86+ }
87+ if code := j.commit(id, bwSize); code != http.StatusNoContent {
88+ t.Fatalf("commit: %d", code)
89+ }
90+ return r, j.token
91+}
92+
93+// A channel with a rate serves a download at about that rate: neither much
94+// faster (the limit holds) nor stalled (the transfer keeps going to the end).
95+func TestBandwidthLimitsADownload(t *testing.T) {
96+ r, tok := bwRig(t)
97+ c := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
98+ loc := bwLocation(t, c, tok)
99+ got := bwDownload(t, c, loc)
100+ want := time.Second * bwSize / bwRate
101+ if got < want*8/10 || got > want+hangGuard {
102+ t.Fatalf("download took %v at %d B/s, want at least %v", got, bwRate, want*8/10)
103+ }
104+}
105+
106+// The request bodies are limited too.
107+func TestBandwidthLimitsAnUpload(t *testing.T) {
108+ r := newRig(t, 1<<22)
109+ c := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
110+ j := r.job("job-a", repoA, main_, main_)
111+ code, id := j.reserve("k", v1, bwSize)
112+ if code != http.StatusCreated {
113+ t.Fatalf("reserve: %d", code)
114+ }
115+ req, _ := http.NewRequest("PATCH", "http://cache"+APIPath+fmt.Sprintf("caches/%d", id), bytes.NewReader(bytes.Repeat([]byte("x"), bwSize)))
116+ req.Header.Set("Authorization", "Bearer "+j.token)
117+ req.Header.Set("Content-Range", fmt.Sprintf("bytes 0-%d/*", bwSize-1))
118+ start := time.Now()
119+ resp, err := c.Do(req)
120+ if err != nil {
121+ t.Fatal(err)
122+ }
123+ _ = resp.Body.Close()
124+ got := time.Since(start)
125+ if resp.StatusCode != http.StatusNoContent {
126+ t.Fatalf("upload: %d", resp.StatusCode)
127+ }
128+ want := time.Second * bwSize / bwRate
129+ if got < want*8/10 || got > want+hangGuard {
130+ t.Fatalf("upload took %v at %d B/s, want at least %v", got, bwRate, want*8/10)
131+ }
132+ if code := j.commit(id, bwSize); code != http.StatusNoContent {
133+ t.Fatalf("commit: %d", code)
134+ }
135+}
136+
137+// The limit is the sandbox's, not the connection's: two simultaneous downloads
138+// on one channel take as long as the two in a row at that rate would, and a
139+// channel of another sandbox, with no limit, is not slowed by them.
140+func TestBandwidthIsSharedByTheConnectionsOfASandboxOnly(t *testing.T) {
141+ r, tok := bwRig(t)
142+ a := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
143+ b := channel(t, r.svc, "er0000000b", "job-a") // same job id, so the same URL; no limit
144+ loc := bwLocation(t, b, tok)
145+
146+ if limiterOf(t, r.svc, "er0000000a") == nil || limiterOf(t, r.svc, "er0000000b") != nil {
147+ t.Fatal("A must be limited and B must have no limiter")
148+ }
149+
150+ start := time.Now()
151+ var wg sync.WaitGroup
152+ for range 2 {
153+ wg.Add(1)
154+ go func() { defer wg.Done(); bwDownload(t, a, loc) }()
155+ }
156+ bwDownload(t, b, loc) // B has no limiter, so it never waits on A's bucket
157+ wg.Wait()
158+ total := time.Since(start)
159+
160+ want := time.Second * 2 * bwSize / bwRate // 1 s for the two; 0.5 s if each had the rate
161+ if total < want*8/10 || total > want+hangGuard {
162+ t.Fatalf("two downloads took %v in all, want at least %v", total, want*8/10)
163+ }
164+}
165+
166+// Two channels with a rate each do not draw on one bucket: the limit is per
167+// Serve, so two sandboxes at once take the time of one.
168+func TestBandwidthIsPerChannel(t *testing.T) {
169+ r, tok := bwRig(t)
170+ a := channel(t, r.svc, "er0000000a", "job-a", withRate(bwRate))
171+ b := channel(t, r.svc, "er0000000b", "job-a", withRate(bwRate))
172+ loc := bwLocation(t, a, tok)
173+ la, lb := limiterOf(t, r.svc, "er0000000a"), limiterOf(t, r.svc, "er0000000b")
174+ if la == nil || lb == nil || la == lb {
175+ t.Fatal("each channel must have its own limiter")
176+ }
177+ // Consuming from A leaves B's bucket untouched.
178+ bwDownload(t, a, loc)
179+ if !bucketUsed(la) || bucketUsed(lb) {
180+ t.Fatalf("after a download on A only: A used=%v, B used=%v", bucketUsed(la), bucketUsed(lb))
181+ }
182+ bwDownload(t, b, loc)
183+ if !bucketUsed(lb) {
184+ t.Fatal("B's own download did not draw on B's bucket")
185+ }
186+}
187+
188+// Without a rate nothing is limited or wrapped, and a number of Mbit/s is
189+// bytes per second the way the VM interface counts it.
190+func TestBandwidthUnsetIsUnlimited(t *testing.T) {
191+ if newBandwidth(mbitToBytes(0)) != nil || newBandwidth(mbitToBytes(-1)) != nil {
192+ t.Fatal("0 must mean unlimited")
193+ }
194+ if got := mbitToBytes(1000); got != 125_000_000 {
195+ t.Fatalf("1000 Mbit/s = %d B/s", got)
196+ }
197+ r, tok := bwRig(t)
198+ c := channel(t, r.svc, "er0000000a", "job-a", WithBandwidth(0))
199+ loc := bwLocation(t, c, tok)
200+ if limiterOf(t, r.svc, "er0000000a") != nil {
201+ t.Fatal("WithBandwidth(0) installed a limiter")
202+ }
203+ if got := bwDownload(t, c, loc); got > hangGuard {
204+ t.Fatalf("unlimited download took %v", got)
205+ }
206+}
207+
208+// A throttled transfer is cut when its connection is, not left sleeping.
209+func TestBandwidthSleepEndsWithTheConnection(t *testing.T) {
210+ r, tok := bwRig(t)
211+ c := channel(t, r.svc, "er0000000a", "job-a", withRate(5_000)) // 20 s for the archive
212+ loc := bwLocation(t, c, tok)
213+ u, err := url.Parse(loc)
214+ if err != nil {
215+ t.Fatal(err)
216+ }
217+ req, _ := http.NewRequest("GET", "http://cache"+u.RequestURI(), nil)
218+ resp, err := c.Do(req)
219+ if err != nil {
220+ t.Fatal(err)
221+ }
222+ buf := make([]byte, 1000)
223+ if _, err := io.ReadFull(resp.Body, buf); err != nil {
224+ t.Fatalf("the throttled transfer does not make progress: %v", err)
225+ }
226+ _ = resp.Body.Close()
227+ done := make(chan struct{})
228+ go func() { r.svc.Stop("er0000000a"); close(done) }()
229+ select {
230+ case <-done:
231+ case <-time.After(5 * time.Second):
232+ t.Fatal("Stop waits for a throttled transfer to finish")
233+ }
234+}
A internal/cachesvc/channel_test.go +299 -0
@@ -0,0 +1,299 @@
1+// SPDX-License-Identifier: MIT
2+
3+package cachesvc
4+
5+import (
6+ "context"
7+ "encoding/json"
8+ "io"
9+ "net"
10+ "net/http"
11+ "net/url"
12+ "path/filepath"
13+ "strings"
14+ "testing"
15+ "time"
16+)
17+
18+// channel serves svc on a Unix socket as the channel of sandbox id and returns
19+// an HTTP client that can reach that socket only.
20+func channel(t *testing.T, svc *Service, id, job string, opts ...ServeOption) *http.Client {
21+ t.Helper()
22+ sock := filepath.Join(t.TempDir(), "c")
23+ ln, err := net.Listen("unix", sock)
24+ if err != nil {
25+ t.Fatal(err)
26+ }
27+ if err := svc.Serve(id, ln, job, opts...); err != nil {
28+ t.Fatal(err)
29+ }
30+ t.Cleanup(func() { svc.Stop(id) })
31+ return &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{
32+ DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
33+ var d net.Dialer
34+ return d.DialContext(ctx, "unix", sock)
35+ },
36+ DisableKeepAlives: true,
37+ }}
38+}
39+
40+func lookupStatus(t *testing.T, c *http.Client, token string) int {
41+ t.Helper()
42+ req, err := http.NewRequest("GET", "http://cache"+APIPath+"cache?keys=k&version="+v1, nil)
43+ if err != nil {
44+ t.Fatal(err)
45+ }
46+ if token != "" {
47+ req.Header.Set("Authorization", "Bearer "+token)
48+ }
49+ resp, err := c.Do(req)
50+ if err != nil {
51+ t.Fatal(err)
52+ }
53+ _ = resp.Body.Close()
54+ return resp.StatusCode
55+}
56+
57+// A channel is the listener of one sandbox. The token rules are the service's
58+// (a missing or foreign token is a 403, a valid one is served), and on top of
59+// them a channel serves the job of its sandbox only: the token of another job,
60+// even a valid one, gets nothing, so a token that leaves its sandbox is useless
61+// on every other sandbox's channel.
62+func TestChannelServesItsOwnJobOnly(t *testing.T) {
63+ r := newRig(t, 1<<20)
64+ a := channel(t, r.svc, "er0000000a", "job-a")
65+ b := channel(t, r.svc, "er0000000b", "job-b")
66+ tokA := r.job("job-a", repoA, main_, main_).token
67+ tokB := r.job("job-b", repoA, main_, main_).token
68+
69+ if got := lookupStatus(t, a, tokA); got != http.StatusNoContent {
70+ t.Fatalf("A's token on A's channel: %d", got)
71+ }
72+ if got := lookupStatus(t, b, tokB); got != http.StatusNoContent {
73+ t.Fatalf("B's token on B's channel: %d", got)
74+ }
75+ if got := lookupStatus(t, b, tokA); got != http.StatusForbidden {
76+ t.Fatalf("A's token on B's channel: %d", got)
77+ }
78+ if got := lookupStatus(t, a, tokB); got != http.StatusForbidden {
79+ t.Fatalf("B's token on A's channel: %d", got)
80+ }
81+ if got := lookupStatus(t, a, ""); got != http.StatusForbidden {
82+ t.Fatalf("no token: %d", got)
83+ }
84+ if got := lookupStatus(t, a, "forged.token"); got != http.StatusForbidden {
85+ t.Fatalf("forged token: %d", got)
86+ }
87+}
88+
89+// archiveStatus requests the archive at loc (a lookup's archiveLocation) on the
90+// channel c, with the given method, and returns the status and the body.
91+func archiveStatus(t *testing.T, c *http.Client, method, loc string) (int, string) {
92+ t.Helper()
93+ u, err := url.Parse(loc)
94+ if err != nil {
95+ t.Fatal(err)
96+ }
97+ req, err := http.NewRequest(method, "http://cache"+u.RequestURI(), nil)
98+ if err != nil {
99+ t.Fatal(err)
100+ }
101+ resp, err := c.Do(req)
102+ if err != nil {
103+ t.Fatal(err)
104+ }
105+ defer func() { _ = resp.Body.Close() }()
106+ b, _ := io.ReadAll(resp.Body)
107+ return resp.StatusCode, string(b)
108+}
109+
110+// The signed URL of an archive is bound to the job it was issued to: it works on
111+// the channel of that job and is refused, for GET and HEAD, on the channel of
112+// any other, even for a job of the same repository and scope.
113+func TestChannelArchiveURLBoundToJob(t *testing.T) {
114+ r := newRig(t, 1<<20)
115+ a := channel(t, r.svc, "er0000000a", "job-a")
116+ b := channel(t, r.svc, "er0000000b", "job-b")
117+ r.job("job-a", repoA, main_, main_).save("k", v1, []byte("secret-a"))
118+
119+ req, _ := http.NewRequest("GET", "http://cache"+APIPath+"cache?keys=k&version="+v1, nil)
120+ req.Header.Set("Authorization", "Bearer "+r.job("job-a", repoA, main_, main_).token)
121+ resp, err := a.Do(req)
122+ if err != nil {
123+ t.Fatal(err)
124+ }
125+ var e struct {
126+ ArchiveLocation string `json:"archiveLocation"`
127+ }
128+ err = json.NewDecoder(resp.Body).Decode(&e)
129+ _ = resp.Body.Close()
130+ if err != nil || resp.StatusCode != http.StatusOK {
131+ t.Fatalf("lookup: %d %v", resp.StatusCode, err)
132+ }
133+
134+ if code, body := archiveStatus(t, a, "GET", e.ArchiveLocation); code != 200 || body != "secret-a" {
135+ t.Fatalf("GET on the issuing channel: %d %q", code, body)
136+ }
137+ if code, _ := archiveStatus(t, a, "HEAD", e.ArchiveLocation); code != 200 {
138+ t.Fatalf("HEAD on the issuing channel: %d", code)
139+ }
140+ if code, body := archiveStatus(t, b, "GET", e.ArchiveLocation); code != http.StatusForbidden || strings.Contains(body, "secret") {
141+ t.Fatalf("GET on another job's channel: %d %q", code, body)
142+ }
143+ if code, _ := archiveStatus(t, b, "HEAD", e.ArchiveLocation); code != http.StatusForbidden {
144+ t.Fatalf("HEAD on another job's channel: %d", code)
145+ }
146+ // Rewriting the job in the URL does not help: the signature covers it.
147+ u, _ := url.Parse(e.ArchiveLocation)
148+ q := u.Query()
149+ q.Set("job", "job-b")
150+ u.RawQuery = q.Encode()
151+ if code, _ := archiveStatus(t, b, "GET", u.String()); code != http.StatusForbidden {
152+ t.Fatalf("URL with the job rewritten: %d", code)
153+ }
154+}
155+
156+// The scope rules do not depend on the channel: a read-only job cannot reserve,
157+// a job writes to its own ref and never reads another repository's entries.
158+func TestChannelKeepsScopeRules(t *testing.T) {
159+ r := newRig(t, 1<<20)
160+ w := channel(t, r.svc, "er0000000a", "job-w")
161+ ro := channel(t, r.svc, "er0000000b", "job-ro")
162+ other := channel(t, r.svc, "er0000000c", "job-o")
163+ r.job("job-w", repoA, main_, main_).save("k", v1, []byte("data"))
164+
165+ check := func(c *http.Client, jobID, repo, write string, wantHit bool) {
166+ t.Helper()
167+ tok := r.svc.Token(jobID, Scope{Repository: repo, WriteRef: write, ReadRefs: []string{main_}}, time.Hour)
168+ got := lookupStatus(t, c, tok)
169+ if wantHit != (got == http.StatusOK) || (!wantHit && got != http.StatusNoContent) {
170+ t.Fatalf("%s: lookup %d, want hit=%v", jobID, got, wantHit)
171+ }
172+ }
173+ check(w, "job-w", repoA, main_, true)
174+ check(ro, "job-ro", repoA, "", true)
175+ check(other, "job-o", "bnema/other", main_, false)
176+
177+ roTok := r.svc.Token("job-ro", Scope{Repository: repoA, ReadRefs: []string{main_}}, time.Hour)
178+ req, _ := http.NewRequest("POST", "http://cache"+APIPath+"caches", nil)
179+ req.Header.Set("Authorization", "Bearer "+roTok)
180+ resp, err := ro.Do(req)
181+ if err != nil {
182+ t.Fatal(err)
183+ }
184+ _ = resp.Body.Close()
185+ if resp.StatusCode != http.StatusForbidden {
186+ t.Fatalf("a read-only job reserved: %d", resp.StatusCode)
187+ }
188+}
189+
190+// The connection caps apply to a channel as to any listener, and Stop closes
191+// the listener and the open connections of that sandbox only.
192+func TestChannelCapsAndStop(t *testing.T) {
193+ r := newRigWith(t, Config{Max: 1 << 20, MinCharge: 1, ConnsPerListener: 1})
194+ sockA, sockB := filepath.Join(t.TempDir(), "a"), filepath.Join(t.TempDir(), "b")
195+ for id, sock := range map[string]string{"er0000000a": sockA, "er0000000b": sockB} {
196+ ln, err := net.Listen("unix", sock)
197+ if err != nil {
198+ t.Fatal(err)
199+ }
200+ if err := r.svc.Serve(id, ln, "j-"+id); err != nil {
201+ t.Fatal(err)
202+ }
203+ }
204+ held, err := net.Dial("unix", sockA)
205+ if err != nil {
206+ t.Fatal(err)
207+ }
208+ defer func() { _ = held.Close() }()
209+ // The one slot of A is taken once the server has accepted `held`: a second
210+ // connection is then closed at once. B has its own share.
211+ waitUntil(t, func() bool {
212+ c, err := net.Dial("unix", sockA)
213+ if err != nil {
214+ return false
215+ }
216+ defer func() { _ = c.Close() }()
217+ _ = c.SetReadDeadline(time.Now().Add(time.Second))
218+ _, err = c.Read(make([]byte, 1))
219+ return err != nil && !isTimeout(err)
220+ })
221+ other, err := net.Dial("unix", sockB)
222+ if err != nil {
223+ t.Fatal(err)
224+ }
225+ defer func() { _ = other.Close() }()
226+
227+ r.svc.Stop("er0000000a")
228+ _ = held.SetReadDeadline(time.Now().Add(5 * time.Second))
229+ if _, err := held.Read(make([]byte, 1)); err == nil || isTimeout(err) {
230+ t.Fatalf("the open connection of a stopped channel: %v", err)
231+ }
232+ if c, err := net.Dial("unix", sockA); err == nil {
233+ _ = c.Close()
234+ t.Fatal("a stopped channel still accepts")
235+ }
236+ // B is not affected.
237+ _ = other.SetReadDeadline(time.Now().Add(200 * time.Millisecond))
238+ if _, err := other.Read(make([]byte, 1)); err == nil || !isTimeout(err) {
239+ t.Fatalf("B's connection after A's stop: %v", err)
240+ }
241+ r.svc.Stop("er0000000a") // idempotent, unknown ids ignored
242+ r.svc.Stop("unknown")
243+}
244+
245+// One channel per sandbox id, none once the service is closed, and CloseAll
246+// closes the listeners it was given.
247+func TestChannelServeErrorsAndCloseAll(t *testing.T) {
248+ r := newRig(t, 1<<20)
249+ sock := filepath.Join(t.TempDir(), "s")
250+ ln, err := net.Listen("unix", sock)
251+ if err != nil {
252+ t.Fatal(err)
253+ }
254+ if err := r.svc.Serve("er0000000a", ln, "j"); err != nil {
255+ t.Fatal(err)
256+ }
257+ dup, err := net.Listen("unix", filepath.Join(t.TempDir(), "d"))
258+ if err != nil {
259+ t.Fatal(err)
260+ }
261+ if err := r.svc.Serve("er0000000a", dup, "j"); err == nil {
262+ t.Fatal("two channels for one sandbox")
263+ }
264+ if _, err := dup.Accept(); err == nil {
265+ t.Fatal("the refused listener was left open")
266+ }
267+ if err := r.svc.CloseAll(); err != nil {
268+ t.Fatal(err)
269+ }
270+ if c, err := net.Dial("unix", sock); err == nil {
271+ _ = c.Close()
272+ t.Fatal("CloseAll left a channel open")
273+ }
274+ late, err := net.Listen("unix", filepath.Join(t.TempDir(), "l"))
275+ if err != nil {
276+ t.Fatal(err)
277+ }
278+ if err := r.svc.Serve("er0000000b", late, "j"); err == nil {
279+ t.Fatal("Serve after CloseAll")
280+ }
281+ if _, err := late.Accept(); err == nil {
282+ t.Fatal("the listener of a refused Serve was left open")
283+ }
284+}
285+
286+func isTimeout(err error) bool {
287+ ne, ok := err.(net.Error)
288+ return ok && ne.Timeout()
289+}
290+
291+func waitUntil(t *testing.T, ok func() bool) {
292+ t.Helper()
293+ for end := time.Now().Add(10 * time.Second); time.Now().Before(end); time.Sleep(20 * time.Millisecond) {
294+ if ok() {
295+ return
296+ }
297+ }
298+ t.Fatal("condition not reached in 10s")
299+}
@@ -0,0 +1,299 @@
1+// SPDX-License-Identifier: MIT
2+
3+package cachesvc
4+
5+import (
6+ "context"
7+ "encoding/json"
8+ "io"
9+ "net"
10+ "net/http"
11+ "net/url"
12+ "path/filepath"
13+ "strings"
14+ "testing"
15+ "time"
16+)
17+
18+// channel serves svc on a Unix socket as the channel of sandbox id and returns
19+// an HTTP client that can reach that socket only.
20+func channel(t *testing.T, svc *Service, id, job string, opts ...ServeOption) *http.Client {
21+ t.Helper()
22+ sock := filepath.Join(t.TempDir(), "c")
23+ ln, err := net.Listen("unix", sock)
24+ if err != nil {
25+ t.Fatal(err)
26+ }
27+ if err := svc.Serve(id, ln, job, opts...); err != nil {
28+ t.Fatal(err)
29+ }
30+ t.Cleanup(func() { svc.Stop(id) })
31+ return &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{
32+ DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
33+ var d net.Dialer
34+ return d.DialContext(ctx, "unix", sock)
35+ },
36+ DisableKeepAlives: true,
37+ }}
38+}
39+
40+func lookupStatus(t *testing.T, c *http.Client, token string) int {
41+ t.Helper()
42+ req, err := http.NewRequest("GET", "http://cache"+APIPath+"cache?keys=k&version="+v1, nil)
43+ if err != nil {
44+ t.Fatal(err)
45+ }
46+ if token != "" {
47+ req.Header.Set("Authorization", "Bearer "+token)
48+ }
49+ resp, err := c.Do(req)
50+ if err != nil {
51+ t.Fatal(err)
52+ }
53+ _ = resp.Body.Close()
54+ return resp.StatusCode
55+}
56+
57+// A channel is the listener of one sandbox. The token rules are the service's
58+// (a missing or foreign token is a 403, a valid one is served), and on top of
59+// them a channel serves the job of its sandbox only: the token of another job,
60+// even a valid one, gets nothing, so a token that leaves its sandbox is useless
61+// on every other sandbox's channel.
62+func TestChannelServesItsOwnJobOnly(t *testing.T) {
63+ r := newRig(t, 1<<20)
64+ a := channel(t, r.svc, "er0000000a", "job-a")
65+ b := channel(t, r.svc, "er0000000b", "job-b")
66+ tokA := r.job("job-a", repoA, main_, main_).token
67+ tokB := r.job("job-b", repoA, main_, main_).token
68+
69+ if got := lookupStatus(t, a, tokA); got != http.StatusNoContent {
70+ t.Fatalf("A's token on A's channel: %d", got)
71+ }
72+ if got := lookupStatus(t, b, tokB); got != http.StatusNoContent {
73+ t.Fatalf("B's token on B's channel: %d", got)
74+ }
75+ if got := lookupStatus(t, b, tokA); got != http.StatusForbidden {
76+ t.Fatalf("A's token on B's channel: %d", got)
77+ }
78+ if got := lookupStatus(t, a, tokB); got != http.StatusForbidden {
79+ t.Fatalf("B's token on A's channel: %d", got)
80+ }
81+ if got := lookupStatus(t, a, ""); got != http.StatusForbidden {
82+ t.Fatalf("no token: %d", got)
83+ }
84+ if got := lookupStatus(t, a, "forged.token"); got != http.StatusForbidden {
85+ t.Fatalf("forged token: %d", got)
86+ }
87+}
88+
89+// archiveStatus requests the archive at loc (a lookup's archiveLocation) on the
90+// channel c, with the given method, and returns the status and the body.
91+func archiveStatus(t *testing.T, c *http.Client, method, loc string) (int, string) {
92+ t.Helper()
93+ u, err := url.Parse(loc)
94+ if err != nil {
95+ t.Fatal(err)
96+ }
97+ req, err := http.NewRequest(method, "http://cache"+u.RequestURI(), nil)
98+ if err != nil {
99+ t.Fatal(err)
100+ }
101+ resp, err := c.Do(req)
102+ if err != nil {
103+ t.Fatal(err)
104+ }
105+ defer func() { _ = resp.Body.Close() }()
106+ b, _ := io.ReadAll(resp.Body)
107+ return resp.StatusCode, string(b)
108+}
109+
110+// The signed URL of an archive is bound to the job it was issued to: it works on
111+// the channel of that job and is refused, for GET and HEAD, on the channel of
112+// any other, even for a job of the same repository and scope.
113+func TestChannelArchiveURLBoundToJob(t *testing.T) {
114+ r := newRig(t, 1<<20)
115+ a := channel(t, r.svc, "er0000000a", "job-a")
116+ b := channel(t, r.svc, "er0000000b", "job-b")
117+ r.job("job-a", repoA, main_, main_).save("k", v1, []byte("secret-a"))
118+
119+ req, _ := http.NewRequest("GET", "http://cache"+APIPath+"cache?keys=k&version="+v1, nil)
120+ req.Header.Set("Authorization", "Bearer "+r.job("job-a", repoA, main_, main_).token)
121+ resp, err := a.Do(req)
122+ if err != nil {
123+ t.Fatal(err)
124+ }
125+ var e struct {
126+ ArchiveLocation string `json:"archiveLocation"`
127+ }
128+ err = json.NewDecoder(resp.Body).Decode(&e)
129+ _ = resp.Body.Close()
130+ if err != nil || resp.StatusCode != http.StatusOK {
131+ t.Fatalf("lookup: %d %v", resp.StatusCode, err)
132+ }
133+
134+ if code, body := archiveStatus(t, a, "GET", e.ArchiveLocation); code != 200 || body != "secret-a" {
135+ t.Fatalf("GET on the issuing channel: %d %q", code, body)
136+ }
137+ if code, _ := archiveStatus(t, a, "HEAD", e.ArchiveLocation); code != 200 {
138+ t.Fatalf("HEAD on the issuing channel: %d", code)
139+ }
140+ if code, body := archiveStatus(t, b, "GET", e.ArchiveLocation); code != http.StatusForbidden || strings.Contains(body, "secret") {
141+ t.Fatalf("GET on another job's channel: %d %q", code, body)
142+ }
143+ if code, _ := archiveStatus(t, b, "HEAD", e.ArchiveLocation); code != http.StatusForbidden {
144+ t.Fatalf("HEAD on another job's channel: %d", code)
145+ }
146+ // Rewriting the job in the URL does not help: the signature covers it.
147+ u, _ := url.Parse(e.ArchiveLocation)
148+ q := u.Query()
149+ q.Set("job", "job-b")
150+ u.RawQuery = q.Encode()
151+ if code, _ := archiveStatus(t, b, "GET", u.String()); code != http.StatusForbidden {
152+ t.Fatalf("URL with the job rewritten: %d", code)
153+ }
154+}
155+
156+// The scope rules do not depend on the channel: a read-only job cannot reserve,
157+// a job writes to its own ref and never reads another repository's entries.
158+func TestChannelKeepsScopeRules(t *testing.T) {
159+ r := newRig(t, 1<<20)
160+ w := channel(t, r.svc, "er0000000a", "job-w")
161+ ro := channel(t, r.svc, "er0000000b", "job-ro")
162+ other := channel(t, r.svc, "er0000000c", "job-o")
163+ r.job("job-w", repoA, main_, main_).save("k", v1, []byte("data"))
164+
165+ check := func(c *http.Client, jobID, repo, write string, wantHit bool) {
166+ t.Helper()
167+ tok := r.svc.Token(jobID, Scope{Repository: repo, WriteRef: write, ReadRefs: []string{main_}}, time.Hour)
168+ got := lookupStatus(t, c, tok)
169+ if wantHit != (got == http.StatusOK) || (!wantHit && got != http.StatusNoContent) {
170+ t.Fatalf("%s: lookup %d, want hit=%v", jobID, got, wantHit)
171+ }
172+ }
173+ check(w, "job-w", repoA, main_, true)
174+ check(ro, "job-ro", repoA, "", true)
175+ check(other, "job-o", "bnema/other", main_, false)
176+
177+ roTok := r.svc.Token("job-ro", Scope{Repository: repoA, ReadRefs: []string{main_}}, time.Hour)
178+ req, _ := http.NewRequest("POST", "http://cache"+APIPath+"caches", nil)
179+ req.Header.Set("Authorization", "Bearer "+roTok)
180+ resp, err := ro.Do(req)
181+ if err != nil {
182+ t.Fatal(err)
183+ }
184+ _ = resp.Body.Close()
185+ if resp.StatusCode != http.StatusForbidden {
186+ t.Fatalf("a read-only job reserved: %d", resp.StatusCode)
187+ }
188+}
189+
190+// The connection caps apply to a channel as to any listener, and Stop closes
191+// the listener and the open connections of that sandbox only.
192+func TestChannelCapsAndStop(t *testing.T) {
193+ r := newRigWith(t, Config{Max: 1 << 20, MinCharge: 1, ConnsPerListener: 1})
194+ sockA, sockB := filepath.Join(t.TempDir(), "a"), filepath.Join(t.TempDir(), "b")
195+ for id, sock := range map[string]string{"er0000000a": sockA, "er0000000b": sockB} {
196+ ln, err := net.Listen("unix", sock)
197+ if err != nil {
198+ t.Fatal(err)
199+ }
200+ if err := r.svc.Serve(id, ln, "j-"+id); err != nil {
201+ t.Fatal(err)
202+ }
203+ }
204+ held, err := net.Dial("unix", sockA)
205+ if err != nil {
206+ t.Fatal(err)
207+ }
208+ defer func() { _ = held.Close() }()
209+ // The one slot of A is taken once the server has accepted `held`: a second
210+ // connection is then closed at once. B has its own share.
211+ waitUntil(t, func() bool {
212+ c, err := net.Dial("unix", sockA)
213+ if err != nil {
214+ return false
215+ }
216+ defer func() { _ = c.Close() }()
217+ _ = c.SetReadDeadline(time.Now().Add(time.Second))
218+ _, err = c.Read(make([]byte, 1))
219+ return err != nil && !isTimeout(err)
220+ })
221+ other, err := net.Dial("unix", sockB)
222+ if err != nil {
223+ t.Fatal(err)
224+ }
225+ defer func() { _ = other.Close() }()
226+
227+ r.svc.Stop("er0000000a")
228+ _ = held.SetReadDeadline(time.Now().Add(5 * time.Second))
229+ if _, err := held.Read(make([]byte, 1)); err == nil || isTimeout(err) {
230+ t.Fatalf("the open connection of a stopped channel: %v", err)
231+ }
232+ if c, err := net.Dial("unix", sockA); err == nil {
233+ _ = c.Close()
234+ t.Fatal("a stopped channel still accepts")
235+ }
236+ // B is not affected.
237+ _ = other.SetReadDeadline(time.Now().Add(200 * time.Millisecond))
238+ if _, err := other.Read(make([]byte, 1)); err == nil || !isTimeout(err) {
239+ t.Fatalf("B's connection after A's stop: %v", err)
240+ }
241+ r.svc.Stop("er0000000a") // idempotent, unknown ids ignored
242+ r.svc.Stop("unknown")
243+}
244+
245+// One channel per sandbox id, none once the service is closed, and CloseAll
246+// closes the listeners it was given.
247+func TestChannelServeErrorsAndCloseAll(t *testing.T) {
248+ r := newRig(t, 1<<20)
249+ sock := filepath.Join(t.TempDir(), "s")
250+ ln, err := net.Listen("unix", sock)
251+ if err != nil {
252+ t.Fatal(err)
253+ }
254+ if err := r.svc.Serve("er0000000a", ln, "j"); err != nil {
255+ t.Fatal(err)
256+ }
257+ dup, err := net.Listen("unix", filepath.Join(t.TempDir(), "d"))
258+ if err != nil {
259+ t.Fatal(err)
260+ }
261+ if err := r.svc.Serve("er0000000a", dup, "j"); err == nil {
262+ t.Fatal("two channels for one sandbox")
263+ }
264+ if _, err := dup.Accept(); err == nil {
265+ t.Fatal("the refused listener was left open")
266+ }
267+ if err := r.svc.CloseAll(); err != nil {
268+ t.Fatal(err)
269+ }
270+ if c, err := net.Dial("unix", sock); err == nil {
271+ _ = c.Close()
272+ t.Fatal("CloseAll left a channel open")
273+ }
274+ late, err := net.Listen("unix", filepath.Join(t.TempDir(), "l"))
275+ if err != nil {
276+ t.Fatal(err)
277+ }
278+ if err := r.svc.Serve("er0000000b", late, "j"); err == nil {
279+ t.Fatal("Serve after CloseAll")
280+ }
281+ if _, err := late.Accept(); err == nil {
282+ t.Fatal("the listener of a refused Serve was left open")
283+ }
284+}
285+
286+func isTimeout(err error) bool {
287+ ne, ok := err.(net.Error)
288+ return ok && ne.Timeout()
289+}
290+
291+func waitUntil(t *testing.T, ok func() bool) {
292+ t.Helper()
293+ for end := time.Now().Add(10 * time.Second); time.Now().Before(end); time.Sleep(20 * time.Millisecond) {
294+ if ok() {
295+ return
296+ }
297+ }
298+ t.Fatal("condition not reached in 10s")
299+}
M internal/cachesvc/handler.go +35 -19
@@ -37,8 +37,13 @@ const (
3737 readDenied = "cache read denied: "
3838 )
3939
40-// Handler returns the HTTP handler of the cache service (API v1).
41-func (s *Service) Handler() http.Handler {
40+// Handler returns the HTTP handler of the cache service (API v1), for any job.
41+func (s *Service) Handler() http.Handler { return s.handler("") }
42+
43+// handler is Handler for the channel of one job: with a job id, a token made
44+// for another job is refused, so a token that left its sandbox is useless on
45+// any other sandbox's channel.
46+func (s *Service) handler(job string) http.Handler {
4247 mux := http.NewServeMux()
4348 // The routes are those of the toolkit's v1 client (cacheHttpClient.ts):
4449 // GET cache?keys=k1,k2&version=v :89 lookup
@@ -47,13 +52,13 @@ func (s *Service) Handler() http.Handler {
4752 // PATCH caches/{id} Content-Range :230 chunk upload
4853 // POST caches/{id} {size} :330 commit
4954 // GET <archiveLocation> downloadUtils.ts:168 (no bearer token)
50- mux.HandleFunc("GET "+APIPath+"cache", s.auth(s.lookup))
51- mux.HandleFunc("GET "+APIPath+"caches", s.auth(s.listCaches))
52- mux.HandleFunc("POST "+APIPath+"caches", s.auth(s.reserve))
53- mux.HandleFunc("PATCH "+APIPath+"caches/{id}", s.auth(s.upload))
54- mux.HandleFunc("POST "+APIPath+"caches/{id}", s.auth(s.commit))
55- mux.HandleFunc("GET "+APIPath+"artifacts/{id}", s.download)
56- mux.HandleFunc("HEAD "+APIPath+"artifacts/{id}", s.download)
55+ mux.HandleFunc("GET "+APIPath+"cache", s.auth(job, s.lookup))
56+ mux.HandleFunc("GET "+APIPath+"caches", s.auth(job, s.listCaches))
57+ mux.HandleFunc("POST "+APIPath+"caches", s.auth(job, s.reserve))
58+ mux.HandleFunc("PATCH "+APIPath+"caches/{id}", s.auth(job, s.upload))
59+ mux.HandleFunc("POST "+APIPath+"caches/{id}", s.auth(job, s.commit))
60+ mux.HandleFunc("GET "+APIPath+"artifacts/{id}", s.download(job))
61+ mux.HandleFunc("HEAD "+APIPath+"artifacts/{id}", s.download(job))
5762 // net/http resets the write deadline per request only when the server has a
5863 // WriteTimeout, which would cut long uploads and downloads: set a fresh
5964 // progress deadline for each request here, so that a deadline left by the
@@ -69,9 +74,10 @@ func (s *Service) Handler() http.Handler {
6974
7075 type handlerFunc func(w http.ResponseWriter, r *http.Request, c Claims)
7176
72-// auth checks the bearer token (ACTIONS_RUNTIME_TOKEN). Every refusal is a bare
73-// 403 whatever the reason.
74-func (s *Service) auth(h handlerFunc) http.HandlerFunc {
77+// auth checks the bearer token (ACTIONS_RUNTIME_TOKEN), and when job is not
78+// empty that it was made for that job. Every refusal is a bare 403 whatever the
79+// reason.
80+func (s *Service) auth(job string, h handlerFunc) http.HandlerFunc {
7581 return func(w http.ResponseWriter, r *http.Request) {
7682 const p = "Bearer "
7783 a := r.Header.Get("Authorization")
@@ -80,7 +86,7 @@ func (s *Service) auth(h handlerFunc) http.HandlerFunc {
8086 return
8187 }
8288 c, err := s.sign.verify(strings.TrimSpace(a[len(p):]), s.now())
83- if err != nil {
89+ if err != nil || (job != "" && c.JobID != job) {
8490 reply(w, http.StatusForbidden, "forbidden")
8591 return
8692 }
@@ -123,7 +129,7 @@ type artifactCacheEntry struct {
123129
124130 func (s *Service) entryJSON(r *http.Request, c Claims, e Entry) artifactCacheEntry {
125131 exp := c.Expiry
126- q := url.Values{"exp": {strconv.FormatInt(exp, 10)}, "sig": {s.sign.archiveSig(e.ID, e.Repository, exp)}}
132+ q := url.Values{"exp": {strconv.FormatInt(exp, 10)}, "job": {c.JobID}, "sig": {s.sign.archiveSig(e.ID, e.Repository, c.JobID, exp)}}
127133 host := r.Host
128134 return artifactCacheEntry{
129135 CacheKey: e.Key, Scope: e.Ref, CacheVersion: e.Version,
@@ -398,16 +404,26 @@ func (s *Service) storeError(w http.ResponseWriter, err error) {
398404
399405 // download serves an archive at the location given by a lookup. The client
400406 // fetches it without the bearer token (downloadUtils.ts:168), so the URL
401-// carries the proof: an HMAC over the entry, its repository and an expiry.
402-func (s *Service) download(w http.ResponseWriter, r *http.Request) {
407+// carries the proof: an HMAC over the entry, its repository, the job the URL
408+// was issued to and an expiry. On the channel of a job (job not empty) a URL
409+// issued to another job is refused, as a token of another job is.
410+func (s *Service) download(job string) http.HandlerFunc {
411+ return func(w http.ResponseWriter, r *http.Request) {
412+ s.serveArchive(w, r, job)
413+ }
414+}
415+
416+func (s *Service) serveArchive(w http.ResponseWriter, r *http.Request, channelJob string) {
403417 id, ok := parseID(r)
404- exp, err := strconv.ParseInt(r.URL.Query().Get("exp"), 10, 64)
405- if !ok || err != nil {
418+ q := r.URL.Query()
419+ exp, err := strconv.ParseInt(q.Get("exp"), 10, 64)
420+ issued := q.Get("job")
421+ if !ok || err != nil || issued == "" || (channelJob != "" && issued != channelJob) {
406422 reply(w, http.StatusForbidden, "forbidden")
407423 return
408424 }
409425 e, found := s.store.get(id)
410- if !found || !s.sign.archiveOK(id, e.Repository, exp, r.URL.Query().Get("sig"), s.now()) {
426+ if !found || !s.sign.archiveOK(id, e.Repository, issued, exp, q.Get("sig"), s.now()) {
411427 reply(w, http.StatusForbidden, "forbidden")
412428 return
413429 }
@@ -37,8 +37,13 @@ const (
37 readDenied = "cache read denied: "37 readDenied = "cache read denied: "
38 )38 )
39 39
40-// Handler returns the HTTP handler of the cache service (API v1).40+// Handler returns the HTTP handler of the cache service (API v1), for any job.
41-func (s *Service) Handler() http.Handler {41+func (s *Service) Handler() http.Handler { return s.handler("") }
42+
43+// handler is Handler for the channel of one job: with a job id, a token made
44+// for another job is refused, so a token that left its sandbox is useless on
45+// any other sandbox's channel.
46+func (s *Service) handler(job string) http.Handler {
42 mux := http.NewServeMux()47 mux := http.NewServeMux()
43 // The routes are those of the toolkit's v1 client (cacheHttpClient.ts):48 // The routes are those of the toolkit's v1 client (cacheHttpClient.ts):
44 // GET cache?keys=k1,k2&version=v :89 lookup49 // GET cache?keys=k1,k2&version=v :89 lookup
@@ -47,13 +52,13 @@ func (s *Service) Handler() http.Handler {
47 // PATCH caches/{id} Content-Range :230 chunk upload52 // PATCH caches/{id} Content-Range :230 chunk upload
48 // POST caches/{id} {size} :330 commit53 // POST caches/{id} {size} :330 commit
49 // GET <archiveLocation> downloadUtils.ts:168 (no bearer token)54 // GET <archiveLocation> downloadUtils.ts:168 (no bearer token)
50- mux.HandleFunc("GET "+APIPath+"cache", s.auth(s.lookup))55+ mux.HandleFunc("GET "+APIPath+"cache", s.auth(job, s.lookup))
51- mux.HandleFunc("GET "+APIPath+"caches", s.auth(s.listCaches))56+ mux.HandleFunc("GET "+APIPath+"caches", s.auth(job, s.listCaches))
52- mux.HandleFunc("POST "+APIPath+"caches", s.auth(s.reserve))57+ mux.HandleFunc("POST "+APIPath+"caches", s.auth(job, s.reserve))
53- mux.HandleFunc("PATCH "+APIPath+"caches/{id}", s.auth(s.upload))58+ mux.HandleFunc("PATCH "+APIPath+"caches/{id}", s.auth(job, s.upload))
54- mux.HandleFunc("POST "+APIPath+"caches/{id}", s.auth(s.commit))59+ mux.HandleFunc("POST "+APIPath+"caches/{id}", s.auth(job, s.commit))
55- mux.HandleFunc("GET "+APIPath+"artifacts/{id}", s.download)60+ mux.HandleFunc("GET "+APIPath+"artifacts/{id}", s.download(job))
56- mux.HandleFunc("HEAD "+APIPath+"artifacts/{id}", s.download)61+ mux.HandleFunc("HEAD "+APIPath+"artifacts/{id}", s.download(job))
57 // net/http resets the write deadline per request only when the server has a62 // net/http resets the write deadline per request only when the server has a
58 // WriteTimeout, which would cut long uploads and downloads: set a fresh63 // WriteTimeout, which would cut long uploads and downloads: set a fresh
59 // progress deadline for each request here, so that a deadline left by the64 // progress deadline for each request here, so that a deadline left by the
@@ -69,9 +74,10 @@ func (s *Service) Handler() http.Handler {
69 74
70 type handlerFunc func(w http.ResponseWriter, r *http.Request, c Claims)75 type handlerFunc func(w http.ResponseWriter, r *http.Request, c Claims)
71 76
72-// auth checks the bearer token (ACTIONS_RUNTIME_TOKEN). Every refusal is a bare77+// auth checks the bearer token (ACTIONS_RUNTIME_TOKEN), and when job is not
73-// 403 whatever the reason.78+// empty that it was made for that job. Every refusal is a bare 403 whatever the
74-func (s *Service) auth(h handlerFunc) http.HandlerFunc {79+// reason.
80+func (s *Service) auth(job string, h handlerFunc) http.HandlerFunc {
75 return func(w http.ResponseWriter, r *http.Request) {81 return func(w http.ResponseWriter, r *http.Request) {
76 const p = "Bearer "82 const p = "Bearer "
77 a := r.Header.Get("Authorization")83 a := r.Header.Get("Authorization")
@@ -80,7 +86,7 @@ func (s *Service) auth(h handlerFunc) http.HandlerFunc {
80 return86 return
81 }87 }
82 c, err := s.sign.verify(strings.TrimSpace(a[len(p):]), s.now())88 c, err := s.sign.verify(strings.TrimSpace(a[len(p):]), s.now())
83- if err != nil {89+ if err != nil || (job != "" && c.JobID != job) {
84 reply(w, http.StatusForbidden, "forbidden")90 reply(w, http.StatusForbidden, "forbidden")
85 return91 return
86 }92 }
@@ -123,7 +129,7 @@ type artifactCacheEntry struct {
123 129
124 func (s *Service) entryJSON(r *http.Request, c Claims, e Entry) artifactCacheEntry {130 func (s *Service) entryJSON(r *http.Request, c Claims, e Entry) artifactCacheEntry {
125 exp := c.Expiry131 exp := c.Expiry
126- q := url.Values{"exp": {strconv.FormatInt(exp, 10)}, "sig": {s.sign.archiveSig(e.ID, e.Repository, exp)}}132+ q := url.Values{"exp": {strconv.FormatInt(exp, 10)}, "job": {c.JobID}, "sig": {s.sign.archiveSig(e.ID, e.Repository, c.JobID, exp)}}
127 host := r.Host133 host := r.Host
128 return artifactCacheEntry{134 return artifactCacheEntry{
129 CacheKey: e.Key, Scope: e.Ref, CacheVersion: e.Version,135 CacheKey: e.Key, Scope: e.Ref, CacheVersion: e.Version,
@@ -398,16 +404,26 @@ func (s *Service) storeError(w http.ResponseWriter, err error) {
398 404
399 // download serves an archive at the location given by a lookup. The client405 // download serves an archive at the location given by a lookup. The client
400 // fetches it without the bearer token (downloadUtils.ts:168), so the URL406 // fetches it without the bearer token (downloadUtils.ts:168), so the URL
401-// carries the proof: an HMAC over the entry, its repository and an expiry.407+// carries the proof: an HMAC over the entry, its repository, the job the URL
402-func (s *Service) download(w http.ResponseWriter, r *http.Request) {408+// was issued to and an expiry. On the channel of a job (job not empty) a URL
409+// issued to another job is refused, as a token of another job is.
410+func (s *Service) download(job string) http.HandlerFunc {
411+ return func(w http.ResponseWriter, r *http.Request) {
412+ s.serveArchive(w, r, job)
413+ }
414+}
415+
416+func (s *Service) serveArchive(w http.ResponseWriter, r *http.Request, channelJob string) {
403 id, ok := parseID(r)417 id, ok := parseID(r)
404- exp, err := strconv.ParseInt(r.URL.Query().Get("exp"), 10, 64)418+ q := r.URL.Query()
405- if !ok || err != nil {419+ exp, err := strconv.ParseInt(q.Get("exp"), 10, 64)
420+ issued := q.Get("job")
421+ if !ok || err != nil || issued == "" || (channelJob != "" && issued != channelJob) {
406 reply(w, http.StatusForbidden, "forbidden")422 reply(w, http.StatusForbidden, "forbidden")
407 return423 return
408 }424 }
409 e, found := s.store.get(id)425 e, found := s.store.get(id)
410- if !found || !s.sign.archiveOK(id, e.Repository, exp, r.URL.Query().Get("sig"), s.now()) {426+ if !found || !s.sign.archiveOK(id, e.Repository, issued, exp, q.Get("sig"), s.now()) {
411 reply(w, http.StatusForbidden, "forbidden")427 reply(w, http.StatusForbidden, "forbidden")
412 return428 return
413 }429 }
M internal/cachesvc/service.go +70 -56
@@ -11,32 +11,35 @@
1111 // (repository, ref, key, version). A job reaches the service with a bearer token
1212 // (ACTIONS_RUNTIME_TOKEN) that is an HMAC, made with a key that exists only in
1313 // the memory of the runner, over its job id, repository, write ref, read refs
14-// and expiry. The service listens on the gateway address of each sandbox, on one
15-// port that netpolicy opens to that sandbox only.
14+// and expiry.
15+//
16+// The service never listens on a network. Each sandbox has a channel of its own
17+// (Serve): a Unix socket, bind-mounted into a container or reached through
18+// Firecracker's vsock proxy for a VM, on which the relay in the guest delivers
19+// the connections of the actions, which reach it on the guest's loopback
20+// (DefaultPort). The host opens no port and its firewall has no rule for it.
1621 package cachesvc
1722
1823 import (
19- "context"
2024 "errors"
2125 "fmt"
2226 "net"
2327 "net/http"
24- "net/netip"
2528 "strconv"
2629 "sync"
27- "syscall"
2830 "time"
2931
3032 "github.com/bnema/zerowrap"
31- "golang.org/x/sys/unix"
3233 )
3334
3435 // Defaults.
3536 const (
3637 // DefaultMax is the size cap, GitHub's limit per repository.
3738 DefaultMax = 10 << 30
38- // DefaultPort is the TCP port on the gateway addresses.
39- DefaultPort = 8089
39+ // DefaultPort is the TCP port of the cache on the loopback of the guest
40+ // (ACTIONS_CACHE_URL=http://127.0.0.1:9089/), and the vsock port of the
41+ // cache streams of a VM. It is not a port of the host.
42+ DefaultPort = 9089
4043 // EnvMax is ERMITE_RUNNER_CACHE_MAX, the size cap (1G, 500M…).
4144 EnvMax = "ERMITE_RUNNER_CACHE_MAX"
4245 // DefaultReservationTTL is how long an upload may stay uncommitted.
@@ -84,11 +87,11 @@ type Service struct {
8487 now func() time.Time
8588 log zerowrap.Logger
8689
87- mu sync.Mutex
88- listeners map[netip.Addr]*listener
89- stop chan struct{}
90- wg sync.WaitGroup
91- closed bool
90+ mu sync.Mutex
91+ channels map[string]*listener
92+ stop chan struct{}
93+ wg sync.WaitGroup
94+ closed bool
9295
9396 hookChunk func(id int64)
9497 progress time.Duration
@@ -99,6 +102,7 @@ type Service struct {
99102 type listener struct {
100103 srv *http.Server
101104 done chan struct{}
105+ bw *bandwidth // the channel's limiter; nil when unlimited (read by tests)
102106 }
103107
104108 // New opens the store and starts the housekeeping goroutine; Close stops it.
@@ -130,7 +134,7 @@ func New(cfg Config) (*Service, error) {
130134 if cfg.Conns <= 0 {
131135 cfg.Conns = DefaultConns
132136 }
133- s := &Service{store: st, sign: sg, now: cfg.Now, log: cfg.Log, listeners: map[netip.Addr]*listener{}, stop: make(chan struct{}),
137+ s := &Service{store: st, sign: sg, now: cfg.Now, log: cfg.Log, channels: map[string]*listener{}, stop: make(chan struct{}),
134138 hookChunk: cfg.hookChunk, progress: cfg.ProgressTimeout, perListener: cfg.ConnsPerListener, connsGlobal: make(chan struct{}, cfg.Conns)}
135139 s.wg.Add(1)
136140 go s.housekeeping()
@@ -161,62 +165,72 @@ func (s *Service) Token(jobID string, sc Scope, ttl time.Duration) string {
161165 return s.sign.token(Claims{JobID: jobID, Scope: sc, Expiry: s.now().Add(ttl).Unix()})
162166 }
163167
164-// Listen serves the cache on addr:port, an address the caller has configured
165-// (the gateway of a sandbox). freeBind lets the bind precede the address
166-// (podman configures the bridge address late, like dnsfwd.Forwarder.FreeBind).
167-func (s *Service) Listen(ctx context.Context, addr netip.Addr, port uint16, freeBind bool) error {
168- s.mu.Lock()
169- defer s.mu.Unlock()
170- if s.closed {
171- return errors.New("cache: service closed")
172- }
173- if _, ok := s.listeners[addr]; ok {
174- return fmt.Errorf("cache: already listening on %s", addr)
168+// start serves HTTP on ln, subject to the connection caps, until the listener
169+// is closed. job, when not empty, restricts the tokens accepted to that job's.
170+func (s *Service) start(name string, ln net.Listener, job string, bw *bandwidth) *listener {
171+ if bw != nil {
172+ ln = throttleListener{ln, bw}
175173 }
176- var lc net.ListenConfig
177- if freeBind {
178- lc.Control = func(_, _ string, c syscall.RawConn) error {
179- var serr error
180- if err := c.Control(func(fd uintptr) { serr = unix.SetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_FREEBIND, 1) }); err != nil {
181- return err
182- }
183- return serr
184- }
185- }
186- ln, err := lc.Listen(ctx, "tcp", net.JoinHostPort(addr.String(), strconv.Itoa(int(port))))
187- if err != nil {
188- return fmt.Errorf("cache: listen %s: %w", addr, err)
189- }
190- return s.serveLocked(addr, ln)
191-}
192-
193-func (s *Service) serveLocked(addr netip.Addr, ln net.Listener) error {
194174 ln = newLimitListener(ln, s.perListener, s.connsGlobal)
195175 l := &listener{
196176 srv: &http.Server{
197- Handler: s.Handler(),
177+ Handler: s.handler(job),
198178 ReadHeaderTimeout: 30 * time.Second,
199179 // Bodies are chunks of up to tens of MiB and archives of GiB: only
200180 // idle connections time out.
201181 IdleTimeout: 2 * time.Minute,
202182 },
203183 done: make(chan struct{}),
184+ bw: bw,
204185 }
205- s.listeners[addr] = l
206186 go func() {
207187 defer close(l.done)
208188 if err := l.srv.Serve(ln); err != nil && !errors.Is(err, http.ErrServerClosed) {
209- s.log.Error().Err(err).Str("addr", addr.String()).Msg("cache listener")
189+ s.log.Error().Err(err).Str("listener", name).Msg("cache listener")
210190 }
211191 }()
192+ return l
193+}
194+
195+// Serve serves the cache on ln, the host end of the channel of one sandbox
196+// (id): a listener that is not on the network, such as the Unix socket on which
197+// the relay of that sandbox's guest delivers its connections. Each sandbox has
198+// its own listener and so its own handler, and nothing but that sandbox's guest
199+// can connect to it. job is the id of the job the sandbox runs: only tokens made
200+// for it are accepted on this channel, besides the checks of every token. The
201+// connection caps apply as on any listener, and so does WithBandwidth when given:
202+// one limiter per call, shared by all the connections of the channel, so that
203+// the sandbox's cache traffic as a whole stays under its rate. Serve takes
204+// ownership of ln: Stop (or CloseAll) closes it, and so does a failed Serve.
205+func (s *Service) Serve(id string, ln net.Listener, job string, opts ...ServeOption) error {
206+ var sc serveConfig
207+ for _, o := range opts {
208+ o(&sc)
209+ }
210+ if job == "" {
211+ _ = ln.Close()
212+ return errors.New("cache: a channel needs the job it serves")
213+ }
214+ s.mu.Lock()
215+ defer s.mu.Unlock()
216+ if s.closed {
217+ _ = ln.Close()
218+ return errors.New("cache: service closed")
219+ }
220+ if _, ok := s.channels[id]; ok {
221+ _ = ln.Close()
222+ return fmt.Errorf("cache: already serving %s", id)
223+ }
224+ s.channels[id] = s.start(id, ln, job, newBandwidth(sc.bytesPerSec))
212225 return nil
213226 }
214227
215-// Close stops listening on addr. Unknown addresses are ignored.
216-func (s *Service) Close(addr netip.Addr) {
228+// Stop ends the service of the channel of sandbox id, closing its listener and
229+// every connection. Unknown ids are ignored.
230+func (s *Service) Stop(id string) {
217231 s.mu.Lock()
218- l := s.listeners[addr]
219- delete(s.listeners, addr)
232+ l := s.channels[id]
233+ delete(s.channels, id)
220234 s.mu.Unlock()
221235 if l != nil {
222236 _ = l.srv.Close()
@@ -224,18 +238,18 @@ func (s *Service) Close(addr netip.Addr) {
224238 }
225239 }
226240
227-// CloseAll stops every listener and the housekeeping, and writes the index.
241+// CloseAll stops every channel and the housekeeping, and writes the index.
228242 func (s *Service) CloseAll() error {
229243 s.mu.Lock()
230- var addrs []netip.Addr
231- for a := range s.listeners {
232- addrs = append(addrs, a)
244+ var ids []string
245+ for id := range s.channels {
246+ ids = append(ids, id)
233247 }
234248 already := s.closed
235249 s.closed = true
236250 s.mu.Unlock()
237- for _, a := range addrs {
238- s.Close(a)
251+ for _, id := range ids {
252+ s.Stop(id)
239253 }
240254 if already {
241255 return nil
@@ -11,32 +11,35 @@
11 // (repository, ref, key, version). A job reaches the service with a bearer token11 // (repository, ref, key, version). A job reaches the service with a bearer token
12 // (ACTIONS_RUNTIME_TOKEN) that is an HMAC, made with a key that exists only in12 // (ACTIONS_RUNTIME_TOKEN) that is an HMAC, made with a key that exists only in
13 // the memory of the runner, over its job id, repository, write ref, read refs13 // the memory of the runner, over its job id, repository, write ref, read refs
14-// and expiry. The service listens on the gateway address of each sandbox, on one14+// and expiry.
15-// port that netpolicy opens to that sandbox only.15+//
16+// The service never listens on a network. Each sandbox has a channel of its own
17+// (Serve): a Unix socket, bind-mounted into a container or reached through
18+// Firecracker's vsock proxy for a VM, on which the relay in the guest delivers
19+// the connections of the actions, which reach it on the guest's loopback
20+// (DefaultPort). The host opens no port and its firewall has no rule for it.
16 package cachesvc21 package cachesvc
17 22
18 import (23 import (
19- "context"
20 "errors"24 "errors"
21 "fmt"25 "fmt"
22 "net"26 "net"
23 "net/http"27 "net/http"
24- "net/netip"
25 "strconv"28 "strconv"
26 "sync"29 "sync"
27- "syscall"
28 "time"30 "time"
29 31
30 "github.com/bnema/zerowrap"32 "github.com/bnema/zerowrap"
31- "golang.org/x/sys/unix"
32 )33 )
33 34
34 // Defaults.35 // Defaults.
35 const (36 const (
36 // DefaultMax is the size cap, GitHub's limit per repository.37 // DefaultMax is the size cap, GitHub's limit per repository.
37 DefaultMax = 10 << 3038 DefaultMax = 10 << 30
38- // DefaultPort is the TCP port on the gateway addresses.39+ // DefaultPort is the TCP port of the cache on the loopback of the guest
39- DefaultPort = 808940+ // (ACTIONS_CACHE_URL=http://127.0.0.1:9089/), and the vsock port of the
41+ // cache streams of a VM. It is not a port of the host.
42+ DefaultPort = 9089
40 // EnvMax is ERMITE_RUNNER_CACHE_MAX, the size cap (1G, 500M…).43 // EnvMax is ERMITE_RUNNER_CACHE_MAX, the size cap (1G, 500M…).
41 EnvMax = "ERMITE_RUNNER_CACHE_MAX"44 EnvMax = "ERMITE_RUNNER_CACHE_MAX"
42 // DefaultReservationTTL is how long an upload may stay uncommitted.45 // DefaultReservationTTL is how long an upload may stay uncommitted.
@@ -84,11 +87,11 @@ type Service struct {
84 now func() time.Time87 now func() time.Time
85 log zerowrap.Logger88 log zerowrap.Logger
86 89
87- mu sync.Mutex90+ mu sync.Mutex
88- listeners map[netip.Addr]*listener91+ channels map[string]*listener
89- stop chan struct{}92+ stop chan struct{}
90- wg sync.WaitGroup93+ wg sync.WaitGroup
91- closed bool94+ closed bool
92 95
93 hookChunk func(id int64)96 hookChunk func(id int64)
94 progress time.Duration97 progress time.Duration
@@ -99,6 +102,7 @@ type Service struct {
99 type listener struct {102 type listener struct {
100 srv *http.Server103 srv *http.Server
101 done chan struct{}104 done chan struct{}
105+ bw *bandwidth // the channel's limiter; nil when unlimited (read by tests)
102 }106 }
103 107
104 // New opens the store and starts the housekeeping goroutine; Close stops it.108 // New opens the store and starts the housekeeping goroutine; Close stops it.
@@ -130,7 +134,7 @@ func New(cfg Config) (*Service, error) {
130 if cfg.Conns <= 0 {134 if cfg.Conns <= 0 {
131 cfg.Conns = DefaultConns135 cfg.Conns = DefaultConns
132 }136 }
133- s := &Service{store: st, sign: sg, now: cfg.Now, log: cfg.Log, listeners: map[netip.Addr]*listener{}, stop: make(chan struct{}),137+ s := &Service{store: st, sign: sg, now: cfg.Now, log: cfg.Log, channels: map[string]*listener{}, stop: make(chan struct{}),
134 hookChunk: cfg.hookChunk, progress: cfg.ProgressTimeout, perListener: cfg.ConnsPerListener, connsGlobal: make(chan struct{}, cfg.Conns)}138 hookChunk: cfg.hookChunk, progress: cfg.ProgressTimeout, perListener: cfg.ConnsPerListener, connsGlobal: make(chan struct{}, cfg.Conns)}
135 s.wg.Add(1)139 s.wg.Add(1)
136 go s.housekeeping()140 go s.housekeeping()
@@ -161,62 +165,72 @@ func (s *Service) Token(jobID string, sc Scope, ttl time.Duration) string {
161 return s.sign.token(Claims{JobID: jobID, Scope: sc, Expiry: s.now().Add(ttl).Unix()})165 return s.sign.token(Claims{JobID: jobID, Scope: sc, Expiry: s.now().Add(ttl).Unix()})
162 }166 }
163 167
164-// Listen serves the cache on addr:port, an address the caller has configured168+// start serves HTTP on ln, subject to the connection caps, until the listener
165-// (the gateway of a sandbox). freeBind lets the bind precede the address169+// is closed. job, when not empty, restricts the tokens accepted to that job's.
166-// (podman configures the bridge address late, like dnsfwd.Forwarder.FreeBind).170+func (s *Service) start(name string, ln net.Listener, job string, bw *bandwidth) *listener {
167-func (s *Service) Listen(ctx context.Context, addr netip.Addr, port uint16, freeBind bool) error {171+ if bw != nil {
168- s.mu.Lock()172+ ln = throttleListener{ln, bw}
169- defer s.mu.Unlock()
170- if s.closed {
171- return errors.New("cache: service closed")
172- }
173- if _, ok := s.listeners[addr]; ok {
174- return fmt.Errorf("cache: already listening on %s", addr)
175 }173 }
176- var lc net.ListenConfig
177- if freeBind {
178- lc.Control = func(_, _ string, c syscall.RawConn) error {
179- var serr error
180- if err := c.Control(func(fd uintptr) { serr = unix.SetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_FREEBIND, 1) }); err != nil {
181- return err
182- }
183- return serr
184- }
185- }
186- ln, err := lc.Listen(ctx, "tcp", net.JoinHostPort(addr.String(), strconv.Itoa(int(port))))
187- if err != nil {
188- return fmt.Errorf("cache: listen %s: %w", addr, err)
189- }
190- return s.serveLocked(addr, ln)
191-}
192-
193-func (s *Service) serveLocked(addr netip.Addr, ln net.Listener) error {
194 ln = newLimitListener(ln, s.perListener, s.connsGlobal)174 ln = newLimitListener(ln, s.perListener, s.connsGlobal)
195 l := &listener{175 l := &listener{
196 srv: &http.Server{176 srv: &http.Server{
197- Handler: s.Handler(),177+ Handler: s.handler(job),
198 ReadHeaderTimeout: 30 * time.Second,178 ReadHeaderTimeout: 30 * time.Second,
199 // Bodies are chunks of up to tens of MiB and archives of GiB: only179 // Bodies are chunks of up to tens of MiB and archives of GiB: only
200 // idle connections time out.180 // idle connections time out.
201 IdleTimeout: 2 * time.Minute,181 IdleTimeout: 2 * time.Minute,
202 },182 },
203 done: make(chan struct{}),183 done: make(chan struct{}),
184+ bw: bw,
204 }185 }
205- s.listeners[addr] = l
206 go func() {186 go func() {
207 defer close(l.done)187 defer close(l.done)
208 if err := l.srv.Serve(ln); err != nil && !errors.Is(err, http.ErrServerClosed) {188 if err := l.srv.Serve(ln); err != nil && !errors.Is(err, http.ErrServerClosed) {
209- s.log.Error().Err(err).Str("addr", addr.String()).Msg("cache listener")189+ s.log.Error().Err(err).Str("listener", name).Msg("cache listener")
210 }190 }
211 }()191 }()
192+ return l
193+}
194+
195+// Serve serves the cache on ln, the host end of the channel of one sandbox
196+// (id): a listener that is not on the network, such as the Unix socket on which
197+// the relay of that sandbox's guest delivers its connections. Each sandbox has
198+// its own listener and so its own handler, and nothing but that sandbox's guest
199+// can connect to it. job is the id of the job the sandbox runs: only tokens made
200+// for it are accepted on this channel, besides the checks of every token. The
201+// connection caps apply as on any listener, and so does WithBandwidth when given:
202+// one limiter per call, shared by all the connections of the channel, so that
203+// the sandbox's cache traffic as a whole stays under its rate. Serve takes
204+// ownership of ln: Stop (or CloseAll) closes it, and so does a failed Serve.
205+func (s *Service) Serve(id string, ln net.Listener, job string, opts ...ServeOption) error {
206+ var sc serveConfig
207+ for _, o := range opts {
208+ o(&sc)
209+ }
210+ if job == "" {
211+ _ = ln.Close()
212+ return errors.New("cache: a channel needs the job it serves")
213+ }
214+ s.mu.Lock()
215+ defer s.mu.Unlock()
216+ if s.closed {
217+ _ = ln.Close()
218+ return errors.New("cache: service closed")
219+ }
220+ if _, ok := s.channels[id]; ok {
221+ _ = ln.Close()
222+ return fmt.Errorf("cache: already serving %s", id)
223+ }
224+ s.channels[id] = s.start(id, ln, job, newBandwidth(sc.bytesPerSec))
212 return nil225 return nil
213 }226 }
214 227
215-// Close stops listening on addr. Unknown addresses are ignored.228+// Stop ends the service of the channel of sandbox id, closing its listener and
216-func (s *Service) Close(addr netip.Addr) {229+// every connection. Unknown ids are ignored.
230+func (s *Service) Stop(id string) {
217 s.mu.Lock()231 s.mu.Lock()
218- l := s.listeners[addr]232+ l := s.channels[id]
219- delete(s.listeners, addr)233+ delete(s.channels, id)
220 s.mu.Unlock()234 s.mu.Unlock()
221 if l != nil {235 if l != nil {
222 _ = l.srv.Close()236 _ = l.srv.Close()
@@ -224,18 +238,18 @@ func (s *Service) Close(addr netip.Addr) {
224 }238 }
225 }239 }
226 240
227-// CloseAll stops every listener and the housekeeping, and writes the index.241+// CloseAll stops every channel and the housekeeping, and writes the index.
228 func (s *Service) CloseAll() error {242 func (s *Service) CloseAll() error {
229 s.mu.Lock()243 s.mu.Lock()
230- var addrs []netip.Addr244+ var ids []string
231- for a := range s.listeners {245+ for id := range s.channels {
232- addrs = append(addrs, a)246+ ids = append(ids, id)
233 }247 }
234 already := s.closed248 already := s.closed
235 s.closed = true249 s.closed = true
236 s.mu.Unlock()250 s.mu.Unlock()
237- for _, a := range addrs {251+ for _, id := range ids {
238- s.Close(a)252+ s.Stop(id)
239 }253 }
240 if already {254 if already {
241 return nil255 return nil
M internal/cachesvc/service_test.go +2 -2
@@ -359,7 +359,7 @@ func TestOtherRepositoryNeverRestored(t *testing.T) {
359359 if code, data := r.fetch(e.Location); code != 200 || string(data) != "secret-a" {
360360 t.Fatalf("control: %d %q", code, data)
361361 }
362- forged := url.Values{"exp": {u.Query().Get("exp")}, "sig": {r.svc.sign.archiveSig(id, repoB, exp)}}
362+ forged := url.Values{"exp": {u.Query().Get("exp")}, "job": {"j3"}, "sig": {r.svc.sign.archiveSig(id, repoB, "j3", exp)}}
363363 if code, _ := r.fetch(r.srv.URL + APIPath + "artifacts/" + idStr + "?" + forged.Encode()); code != 403 {
364364 t.Fatalf("signature for the other repository: %d", code)
365365 }
@@ -399,7 +399,7 @@ func TestBadTokens(t *testing.T) {
399399 t.Fatalf("no scheme: %d", resp.StatusCode)
400400 }
401401 // A token is not an archive signature, nor the other way round.
402- if r.svc.sign.archiveOK(1, repoA, 1<<40, good.token, r.clk.now()) {
402+ if r.svc.sign.archiveOK(1, repoA, "j1", 1<<40, good.token, r.clk.now()) {
403403 t.Fatal("token accepted as archive signature")
404404 }
405405 if code, _ := r.fetch(r.srv.URL + APIPath + "artifacts/1?exp=99999999999&sig=x"); code != 403 {
@@ -359,7 +359,7 @@ func TestOtherRepositoryNeverRestored(t *testing.T) {
359 if code, data := r.fetch(e.Location); code != 200 || string(data) != "secret-a" {359 if code, data := r.fetch(e.Location); code != 200 || string(data) != "secret-a" {
360 t.Fatalf("control: %d %q", code, data)360 t.Fatalf("control: %d %q", code, data)
361 }361 }
362- forged := url.Values{"exp": {u.Query().Get("exp")}, "sig": {r.svc.sign.archiveSig(id, repoB, exp)}}362+ forged := url.Values{"exp": {u.Query().Get("exp")}, "job": {"j3"}, "sig": {r.svc.sign.archiveSig(id, repoB, "j3", exp)}}
363 if code, _ := r.fetch(r.srv.URL + APIPath + "artifacts/" + idStr + "?" + forged.Encode()); code != 403 {363 if code, _ := r.fetch(r.srv.URL + APIPath + "artifacts/" + idStr + "?" + forged.Encode()); code != 403 {
364 t.Fatalf("signature for the other repository: %d", code)364 t.Fatalf("signature for the other repository: %d", code)
365 }365 }
@@ -399,7 +399,7 @@ func TestBadTokens(t *testing.T) {
399 t.Fatalf("no scheme: %d", resp.StatusCode)399 t.Fatalf("no scheme: %d", resp.StatusCode)
400 }400 }
401 // A token is not an archive signature, nor the other way round.401 // A token is not an archive signature, nor the other way round.
402- if r.svc.sign.archiveOK(1, repoA, 1<<40, good.token, r.clk.now()) {402+ if r.svc.sign.archiveOK(1, repoA, "j1", 1<<40, good.token, r.clk.now()) {
403 t.Fatal("token accepted as archive signature")403 t.Fatal("token accepted as archive signature")
404 }404 }
405 if code, _ := r.fetch(r.srv.URL + APIPath + "artifacts/1?exp=99999999999&sig=x"); code != 403 {405 if code, _ := r.fetch(r.srv.URL + APIPath + "artifacts/1?exp=99999999999&sig=x"); code != 403 {
M internal/cachesvc/token.go +7 -6
@@ -101,17 +101,18 @@ func (s *signer) verify(tok string, now time.Time) (Claims, error) {
101101
102102 // archiveSig signs the download URL of one entry. The archive is fetched by a
103103 // client without the bearer token (downloadUtils.ts:168 uses a bare HttpClient),
104-// so the URL is the credential; it is bound to the entry, its repository and an
105-// expiry.
106-func (s *signer) archiveSig(id int64, repo string, exp int64) string {
107- return b64.EncodeToString(s.mac("archive", []byte(strconv.FormatInt(id, 10)), []byte(repo), []byte(strconv.FormatInt(exp, 10))))
104+// so the URL is the credential; it is bound to the entry, its repository, the
105+// job it was issued to and an expiry. The job is what keeps a URL that left its
106+// sandbox useless on the channel of any other.
107+func (s *signer) archiveSig(id int64, repo, job string, exp int64) string {
108+ return b64.EncodeToString(s.mac("archive", []byte(strconv.FormatInt(id, 10)), []byte(repo), []byte(job), []byte(strconv.FormatInt(exp, 10))))
108109 }
109110
110-func (s *signer) archiveOK(id int64, repo string, exp int64, sig string, now time.Time) bool {
111+func (s *signer) archiveOK(id int64, repo, job string, exp int64, sig string, now time.Time) bool {
111112 got, err := b64.DecodeString(sig)
112113 if err != nil || now.Unix() >= exp {
113114 return false
114115 }
115- want, _ := b64.DecodeString(s.archiveSig(id, repo, exp))
116+ want, _ := b64.DecodeString(s.archiveSig(id, repo, job, exp))
116117 return hmac.Equal(got, want)
117118 }
@@ -101,17 +101,18 @@ func (s *signer) verify(tok string, now time.Time) (Claims, error) {
101 101
102 // archiveSig signs the download URL of one entry. The archive is fetched by a102 // archiveSig signs the download URL of one entry. The archive is fetched by a
103 // client without the bearer token (downloadUtils.ts:168 uses a bare HttpClient),103 // client without the bearer token (downloadUtils.ts:168 uses a bare HttpClient),
104-// so the URL is the credential; it is bound to the entry, its repository and an104+// so the URL is the credential; it is bound to the entry, its repository, the
105-// expiry.105+// job it was issued to and an expiry. The job is what keeps a URL that left its
106-func (s *signer) archiveSig(id int64, repo string, exp int64) string {106+// sandbox useless on the channel of any other.
107- return b64.EncodeToString(s.mac("archive", []byte(strconv.FormatInt(id, 10)), []byte(repo), []byte(strconv.FormatInt(exp, 10))))107+func (s *signer) archiveSig(id int64, repo, job string, exp int64) string {
108+ return b64.EncodeToString(s.mac("archive", []byte(strconv.FormatInt(id, 10)), []byte(repo), []byte(job), []byte(strconv.FormatInt(exp, 10))))
108 }109 }
109 110
110-func (s *signer) archiveOK(id int64, repo string, exp int64, sig string, now time.Time) bool {111+func (s *signer) archiveOK(id int64, repo, job string, exp int64, sig string, now time.Time) bool {
111 got, err := b64.DecodeString(sig)112 got, err := b64.DecodeString(sig)
112 if err != nil || now.Unix() >= exp {113 if err != nil || now.Unix() >= exp {
113 return false114 return false
114 }115 }
115- want, _ := b64.DecodeString(s.archiveSig(id, repo, exp))116+ want, _ := b64.DecodeString(s.archiveSig(id, repo, job, exp))
116 return hmac.Equal(got, want)117 return hmac.Equal(got, want)
117 }118 }
A internal/e2e/cache_test.go +190 -0
@@ -0,0 +1,190 @@
1+// SPDX-License-Identifier: MIT
2+
3+package e2e
4+
5+import (
6+ "context"
7+ "fmt"
8+ "os"
9+ "path/filepath"
10+ "strings"
11+ "sync"
12+ "testing"
13+ "time"
14+
15+ "bnema.dev/ermite-runner/internal/cachesvc"
16+ "bnema.dev/ermite-runner/internal/execstream/execpb"
17+ "bnema.dev/ermite-runner/internal/sandbox"
18+ "bnema.dev/ermite-runner/internal/sandbox/container"
19+ "bnema.dev/ermite-runner/internal/sandbox/disk"
20+ "bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
21+)
22+
23+// newCacheDriver is newDriver with the cache service of the runner.
24+func newCacheDriver(t *testing.T, image string, svc *cachesvc.Service) sandbox.Driver {
25+ t.Helper()
26+ dir := t.TempDir()
27+ cfg := sandbox.Config{
28+ Isolation: sandbox.Container, StateDir: filepath.Join(dir, "state"), Capacity: 2, Image: image,
29+ JobCPUs: 2, JobMemoryMiB: 1024, JobDiskMiB: 2048, UIDBase: 900000,
30+ Network: netPrefix("10.215.0.0/16"), Executable: buildRunner(t, dir), DNSPort: 0,
31+ LockFile: filepath.Join(dir, "host.lock"), JobPids: 4096, Cache: svc,
32+ }
33+ empty := filepath.Join(dir, "mountinfo")
34+ _ = os.WriteFile(empty, nil, 0o644)
35+ run := rootlessMountRunner{image: image, mountinfo: empty, mu: &sync.Mutex{}}
36+ d, err := container.New(context.Background(), cfg, container.Options{
37+ Net: sandboxtest.NewPlumbing(), UnsafeAllowRootlessForTests: true, ConnectTimeout: 90 * time.Second, Run: run,
38+ Mounter: &disk.Mounter{Run: run, Mountinfo: empty},
39+ })
40+ if err != nil {
41+ t.Fatal(err)
42+ }
43+ t.Cleanup(func() { _ = d.Close() })
44+ if err := d.Reap(context.Background()); err != nil {
45+ t.Fatal(err)
46+ }
47+ return d
48+}
49+
50+// cacheClient is what @actions/cache does against the v1 API, in python (the
51+// image has it): reserve, upload a chunk, commit, look up, download the archive.
52+// It talks to 127.0.0.1:9089, the loopback of the container, where the relay of
53+// the guest delivers every connection to the runner over the cache socket.
54+const cacheClient = `
55+import json, sys, urllib.request, urllib.error
56+base = "http://127.0.0.1:%d/_apis/artifactcache/"
57+tok = "%s"
58+def call(method, path, body=None, headers=None):
59+ h = {"Authorization": "Bearer " + tok, "Accept": "application/json;api-version=6.0-preview.1"}
60+ h.update(headers or {})
61+ req = urllib.request.Request(base + path, data=body, method=method, headers=h)
62+ try:
63+ with urllib.request.urlopen(req, timeout=20) as r:
64+ return r.status, r.read()
65+ except urllib.error.HTTPError as e:
66+ return e.code, e.read()
67+data = bytes(range(256)) * 4096 # 1 MiB
68+code, body = call("GET", "cache?keys=e2e-key&version=v1")
69+print("first lookup", code)
70+code, body = call("POST", "caches", json.dumps({"key": "e2e-key", "version": "v1", "cacheSize": len(data)}).encode(), {"Content-Type": "application/json"})
71+print("reserve", code)
72+cid = json.loads(body)["cacheId"]
73+half = len(data) // 2
74+for lo, hi in ((half, len(data)), (0, half)): # out of order, like the toolkit may
75+ code, _ = call("PATCH", "caches/%%d" %% cid, data[lo:hi], {"Content-Type": "application/octet-stream", "Content-Range": "bytes %%d-%%d/*" %% (lo, hi - 1)})
76+ print("chunk", lo, code)
77+code, _ = call("POST", "caches/%%d" %% cid, json.dumps({"size": len(data)}).encode(), {"Content-Type": "application/json"})
78+print("commit", code)
79+code, body = call("GET", "cache?keys=e2e-key&version=v1")
80+print("second lookup", code)
81+loc = json.loads(body)["archiveLocation"]
82+print("archive host", loc.split("/")[2])
83+with urllib.request.urlopen(loc, timeout=20) as r:
84+ got = r.read()
85+print("download", "same" if got == data else "DIFFERENT", len(got))
86+code, _ = call("GET", "cache?keys=e2e-key&version=v1", None, {"Authorization": "Bearer nope"})
87+print("bad token", code)
88+`
89+
90+// The cache works from inside a container: the service listens on no network,
91+// the guest relay turns its loopback port into the cache socket of the
92+// container, and a client in the job does a full save and restore through it.
93+// A token made for another job is refused on this container's socket.
94+func TestContainerCacheOnTheGuestLoopback(t *testing.T) {
95+ image := needE2E(t)
96+ svc := sandboxtest.NewCache(t)
97+ d := newCacheDriver(t, image, svc)
98+
99+ tok := sandboxtest.CacheToken(svc, "e2e")
100+ spec := jobSpec(fmt.Sprintf(`on: push
101+jobs:
102+ j:
103+ runs-on: x
104+ steps:
105+ - name: cache client
106+ shell: python3 {0}
107+ run: |
108+%s
109+ - name: nothing listens on the gateway
110+ shell: python3 {0}
111+ run: |
112+ import socket, struct
113+ gw = ""
114+ for line in open("/proc/net/route").read().splitlines()[1:]:
115+ f = line.split()
116+ if f[1] == "00000000":
117+ gw = socket.inet_ntoa(struct.pack("<L", int(f[2], 16)))
118+ print("default gateway", gw)
119+ for port in (8089, 9089):
120+ s = socket.socket(); s.settimeout(2)
121+ try:
122+ s.connect((gw, port)); print("gateway", gw, port, "OPEN")
123+ except Exception as e:
124+ print("gateway", port, "closed")
125+ s.close()
126+`, indent(fmt.Sprintf(cacheClient, cachesvc.DefaultPort, tok), 10)))
127+ spec.CachePort, spec.CacheToken = cachesvc.DefaultPort, tok
128+
129+ s := runJob(t, d, spec, nil)
130+ all := strings.Join(s.Logs, "\n")
131+ if s.Result == nil || s.Result.Conclusion != execpb.StepConclusion_STEP_CONCLUSION_SUCCESS || s.Exit.Code != 0 {
132+ t.Fatalf("result %v exit %v\n%s", s.Result, s.Exit, all)
133+ }
134+ for _, want := range []string{
135+ "first lookup 204", "reserve 201", "chunk 524288 204", "chunk 0 204", "commit 204", "second lookup 200",
136+ "archive host 127.0.0.1:9089", "download same 1048576", "bad token 403",
137+ "default gateway 10.215.", "gateway 8089 closed", "gateway 9089 closed",
138+ } {
139+ if !strings.Contains(all, want) {
140+ t.Errorf("log lacks %q\n%s", want, all)
141+ }
142+ }
143+ // (The log also echoes the script, which contains the word: match the output.)
144+ for _, port := range []string{"8089", "9089"} {
145+ if strings.Contains(all, "10.215.0.1 "+port+" OPEN") {
146+ t.Errorf("the cache port %s answers on the gateway:\n%s", port, all)
147+ }
148+ }
149+}
150+
151+// A job whose token was made for another job gets 403 on its own socket.
152+func TestContainerCacheRefusesAnotherJobsToken(t *testing.T) {
153+ image := needE2E(t)
154+ svc := sandboxtest.NewCache(t)
155+ d := newCacheDriver(t, image, svc)
156+ tok := sandboxtest.CacheToken(svc, "someone-else")
157+ spec := jobSpec(fmt.Sprintf(`on: push
158+jobs:
159+ j:
160+ runs-on: x
161+ steps:
162+ - shell: python3 {0}
163+ run: |
164+%s
165+`, indent(fmt.Sprintf(`import urllib.request, urllib.error
166+req = urllib.request.Request("http://127.0.0.1:%d/_apis/artifactcache/cache?keys=k&version=v", headers={"Authorization": "Bearer %s"})
167+try:
168+ urllib.request.urlopen(req, timeout=20); print("status 200")
169+except urllib.error.HTTPError as e:
170+ print("status", e.code)
171+`, cachesvc.DefaultPort, tok), 10)))
172+ spec.CachePort, spec.CacheToken = cachesvc.DefaultPort, tok
173+ s := runJob(t, d, spec, nil)
174+ all := strings.Join(s.Logs, "\n")
175+ if s.Result == nil || s.Result.Conclusion != execpb.StepConclusion_STEP_CONCLUSION_SUCCESS || !strings.Contains(all, "status 403") {
176+ t.Fatalf("result %v\n%s", s.Result, all)
177+ }
178+}
179+
180+// indent prefixes every non-empty line of s with n spaces.
181+func indent(s string, n int) string {
182+ pad := strings.Repeat(" ", n)
183+ lines := strings.Split(strings.TrimSpace(s), "\n")
184+ for i, l := range lines {
185+ if l != "" {
186+ lines[i] = pad + l
187+ }
188+ }
189+ return strings.Join(lines, "\n")
190+}
@@ -0,0 +1,190 @@
1+// SPDX-License-Identifier: MIT
2+
3+package e2e
4+
5+import (
6+ "context"
7+ "fmt"
8+ "os"
9+ "path/filepath"
10+ "strings"
11+ "sync"
12+ "testing"
13+ "time"
14+
15+ "bnema.dev/ermite-runner/internal/cachesvc"
16+ "bnema.dev/ermite-runner/internal/execstream/execpb"
17+ "bnema.dev/ermite-runner/internal/sandbox"
18+ "bnema.dev/ermite-runner/internal/sandbox/container"
19+ "bnema.dev/ermite-runner/internal/sandbox/disk"
20+ "bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
21+)
22+
23+// newCacheDriver is newDriver with the cache service of the runner.
24+func newCacheDriver(t *testing.T, image string, svc *cachesvc.Service) sandbox.Driver {
25+ t.Helper()
26+ dir := t.TempDir()
27+ cfg := sandbox.Config{
28+ Isolation: sandbox.Container, StateDir: filepath.Join(dir, "state"), Capacity: 2, Image: image,
29+ JobCPUs: 2, JobMemoryMiB: 1024, JobDiskMiB: 2048, UIDBase: 900000,
30+ Network: netPrefix("10.215.0.0/16"), Executable: buildRunner(t, dir), DNSPort: 0,
31+ LockFile: filepath.Join(dir, "host.lock"), JobPids: 4096, Cache: svc,
32+ }
33+ empty := filepath.Join(dir, "mountinfo")
34+ _ = os.WriteFile(empty, nil, 0o644)
35+ run := rootlessMountRunner{image: image, mountinfo: empty, mu: &sync.Mutex{}}
36+ d, err := container.New(context.Background(), cfg, container.Options{
37+ Net: sandboxtest.NewPlumbing(), UnsafeAllowRootlessForTests: true, ConnectTimeout: 90 * time.Second, Run: run,
38+ Mounter: &disk.Mounter{Run: run, Mountinfo: empty},
39+ })
40+ if err != nil {
41+ t.Fatal(err)
42+ }
43+ t.Cleanup(func() { _ = d.Close() })
44+ if err := d.Reap(context.Background()); err != nil {
45+ t.Fatal(err)
46+ }
47+ return d
48+}
49+
50+// cacheClient is what @actions/cache does against the v1 API, in python (the
51+// image has it): reserve, upload a chunk, commit, look up, download the archive.
52+// It talks to 127.0.0.1:9089, the loopback of the container, where the relay of
53+// the guest delivers every connection to the runner over the cache socket.
54+const cacheClient = `
55+import json, sys, urllib.request, urllib.error
56+base = "http://127.0.0.1:%d/_apis/artifactcache/"
57+tok = "%s"
58+def call(method, path, body=None, headers=None):
59+ h = {"Authorization": "Bearer " + tok, "Accept": "application/json;api-version=6.0-preview.1"}
60+ h.update(headers or {})
61+ req = urllib.request.Request(base + path, data=body, method=method, headers=h)
62+ try:
63+ with urllib.request.urlopen(req, timeout=20) as r:
64+ return r.status, r.read()
65+ except urllib.error.HTTPError as e:
66+ return e.code, e.read()
67+data = bytes(range(256)) * 4096 # 1 MiB
68+code, body = call("GET", "cache?keys=e2e-key&version=v1")
69+print("first lookup", code)
70+code, body = call("POST", "caches", json.dumps({"key": "e2e-key", "version": "v1", "cacheSize": len(data)}).encode(), {"Content-Type": "application/json"})
71+print("reserve", code)
72+cid = json.loads(body)["cacheId"]
73+half = len(data) // 2
74+for lo, hi in ((half, len(data)), (0, half)): # out of order, like the toolkit may
75+ code, _ = call("PATCH", "caches/%%d" %% cid, data[lo:hi], {"Content-Type": "application/octet-stream", "Content-Range": "bytes %%d-%%d/*" %% (lo, hi - 1)})
76+ print("chunk", lo, code)
77+code, _ = call("POST", "caches/%%d" %% cid, json.dumps({"size": len(data)}).encode(), {"Content-Type": "application/json"})
78+print("commit", code)
79+code, body = call("GET", "cache?keys=e2e-key&version=v1")
80+print("second lookup", code)
81+loc = json.loads(body)["archiveLocation"]
82+print("archive host", loc.split("/")[2])
83+with urllib.request.urlopen(loc, timeout=20) as r:
84+ got = r.read()
85+print("download", "same" if got == data else "DIFFERENT", len(got))
86+code, _ = call("GET", "cache?keys=e2e-key&version=v1", None, {"Authorization": "Bearer nope"})
87+print("bad token", code)
88+`
89+
90+// The cache works from inside a container: the service listens on no network,
91+// the guest relay turns its loopback port into the cache socket of the
92+// container, and a client in the job does a full save and restore through it.
93+// A token made for another job is refused on this container's socket.
94+func TestContainerCacheOnTheGuestLoopback(t *testing.T) {
95+ image := needE2E(t)
96+ svc := sandboxtest.NewCache(t)
97+ d := newCacheDriver(t, image, svc)
98+
99+ tok := sandboxtest.CacheToken(svc, "e2e")
100+ spec := jobSpec(fmt.Sprintf(`on: push
101+jobs:
102+ j:
103+ runs-on: x
104+ steps:
105+ - name: cache client
106+ shell: python3 {0}
107+ run: |
108+%s
109+ - name: nothing listens on the gateway
110+ shell: python3 {0}
111+ run: |
112+ import socket, struct
113+ gw = ""
114+ for line in open("/proc/net/route").read().splitlines()[1:]:
115+ f = line.split()
116+ if f[1] == "00000000":
117+ gw = socket.inet_ntoa(struct.pack("<L", int(f[2], 16)))
118+ print("default gateway", gw)
119+ for port in (8089, 9089):
120+ s = socket.socket(); s.settimeout(2)
121+ try:
122+ s.connect((gw, port)); print("gateway", gw, port, "OPEN")
123+ except Exception as e:
124+ print("gateway", port, "closed")
125+ s.close()
126+`, indent(fmt.Sprintf(cacheClient, cachesvc.DefaultPort, tok), 10)))
127+ spec.CachePort, spec.CacheToken = cachesvc.DefaultPort, tok
128+
129+ s := runJob(t, d, spec, nil)
130+ all := strings.Join(s.Logs, "\n")
131+ if s.Result == nil || s.Result.Conclusion != execpb.StepConclusion_STEP_CONCLUSION_SUCCESS || s.Exit.Code != 0 {
132+ t.Fatalf("result %v exit %v\n%s", s.Result, s.Exit, all)
133+ }
134+ for _, want := range []string{
135+ "first lookup 204", "reserve 201", "chunk 524288 204", "chunk 0 204", "commit 204", "second lookup 200",
136+ "archive host 127.0.0.1:9089", "download same 1048576", "bad token 403",
137+ "default gateway 10.215.", "gateway 8089 closed", "gateway 9089 closed",
138+ } {
139+ if !strings.Contains(all, want) {
140+ t.Errorf("log lacks %q\n%s", want, all)
141+ }
142+ }
143+ // (The log also echoes the script, which contains the word: match the output.)
144+ for _, port := range []string{"8089", "9089"} {
145+ if strings.Contains(all, "10.215.0.1 "+port+" OPEN") {
146+ t.Errorf("the cache port %s answers on the gateway:\n%s", port, all)
147+ }
148+ }
149+}
150+
151+// A job whose token was made for another job gets 403 on its own socket.
152+func TestContainerCacheRefusesAnotherJobsToken(t *testing.T) {
153+ image := needE2E(t)
154+ svc := sandboxtest.NewCache(t)
155+ d := newCacheDriver(t, image, svc)
156+ tok := sandboxtest.CacheToken(svc, "someone-else")
157+ spec := jobSpec(fmt.Sprintf(`on: push
158+jobs:
159+ j:
160+ runs-on: x
161+ steps:
162+ - shell: python3 {0}
163+ run: |
164+%s
165+`, indent(fmt.Sprintf(`import urllib.request, urllib.error
166+req = urllib.request.Request("http://127.0.0.1:%d/_apis/artifactcache/cache?keys=k&version=v", headers={"Authorization": "Bearer %s"})
167+try:
168+ urllib.request.urlopen(req, timeout=20); print("status 200")
169+except urllib.error.HTTPError as e:
170+ print("status", e.code)
171+`, cachesvc.DefaultPort, tok), 10)))
172+ spec.CachePort, spec.CacheToken = cachesvc.DefaultPort, tok
173+ s := runJob(t, d, spec, nil)
174+ all := strings.Join(s.Logs, "\n")
175+ if s.Result == nil || s.Result.Conclusion != execpb.StepConclusion_STEP_CONCLUSION_SUCCESS || !strings.Contains(all, "status 403") {
176+ t.Fatalf("result %v\n%s", s.Result, all)
177+ }
178+}
179+
180+// indent prefixes every non-empty line of s with n spaces.
181+func indent(s string, n int) string {
182+ pad := strings.Repeat(" ", n)
183+ lines := strings.Split(strings.TrimSpace(s), "\n")
184+ for i, l := range lines {
185+ if l != "" {
186+ lines[i] = pad + l
187+ }
188+ }
189+ return strings.Join(lines, "\n")
190+}
M internal/execstream/exec.proto +5 -3
@@ -88,9 +88,11 @@ message JobSpec {
8888 bool send_annotations = 13;
8989 bool send_summaries = 14;
9090 // cache_port and cache_token, when set, are the runner's cache service for
91- // this job: it listens on the sandbox's gateway address (the executor knows
92- // it: ERMITE_GATEWAY) on that TCP port. The executor gives
93- // ACTIONS_CACHE_URL=http://<gateway>:<port>/ and the token as
91+ // this job. The service is not on the network: the executor starts a relay
92+ // on the sandbox's own loopback, 127.0.0.1:<cache_port>, that carries each
93+ // connection over the host channel (a vsock port, or a Unix socket next to
94+ // the executor's) to the service. The executor gives
95+ // ACTIONS_CACHE_URL=http://127.0.0.1:<port>/ and the token as
9496 // ACTIONS_RUNTIME_TOKEN to node and docker actions only, as the official
9597 // runner does, and masks the token. The token is not an ermite credential: it
9698 // is an HMAC that only the runner's cache service accepts, scoped to this
@@ -88,9 +88,11 @@ message JobSpec {
88 bool send_annotations = 13;88 bool send_annotations = 13;
89 bool send_summaries = 14;89 bool send_summaries = 14;
90 // cache_port and cache_token, when set, are the runner's cache service for90 // cache_port and cache_token, when set, are the runner's cache service for
91- // this job: it listens on the sandbox's gateway address (the executor knows91+ // this job. The service is not on the network: the executor starts a relay
92- // it: ERMITE_GATEWAY) on that TCP port. The executor gives92+ // on the sandbox's own loopback, 127.0.0.1:<cache_port>, that carries each
93- // ACTIONS_CACHE_URL=http://<gateway>:<port>/ and the token as93+ // connection over the host channel (a vsock port, or a Unix socket next to
94+ // the executor's) to the service. The executor gives
95+ // ACTIONS_CACHE_URL=http://127.0.0.1:<port>/ and the token as
94 // ACTIONS_RUNTIME_TOKEN to node and docker actions only, as the official96 // ACTIONS_RUNTIME_TOKEN to node and docker actions only, as the official
95 // runner does, and masks the token. The token is not an ermite credential: it97 // runner does, and masks the token. The token is not an ermite credential: it
96 // is an HMAC that only the runner's cache service accepts, scoped to this98 // is an HMAC that only the runner's cache service accepts, scoped to this
M internal/execstream/execpb/exec.pb.go +5 -3 generated file: collapsed
load diff
M internal/executor/actions_cache_test.go +1 -1
@@ -8,7 +8,7 @@ import (
88 )
99
1010 const (
11- cacheTestURL = "http://10.200.0.1:8089/"
11+ cacheTestURL = "http://127.0.0.1:9089/"
1212 cacheTestToken = "eyJqIjoiMSJ9.c2lnbmF0dXJl"
1313 )
1414
@@ -8,7 +8,7 @@ import (
8 )8 )
9 9
10 const (10 const (
11- cacheTestURL = "http://10.200.0.1:8089/"11+ cacheTestURL = "http://127.0.0.1:9089/"
12 cacheTestToken = "eyJqIjoiMSJ9.c2lnbmF0dXJl"12 cacheTestToken = "eyJqIjoiMSJ9.c2lnbmF0dXJl"
13 )13 )
14 14
M internal/guest/cacheurl_test.go +90 -8
@@ -2,22 +2,104 @@
22
33 package guest
44
5-import "testing"
5+import (
6+ "context"
7+ "io"
8+ "net"
9+ "net/http"
10+ "path/filepath"
11+ "testing"
12+ "time"
13+
14+ "bnema.dev/ermite-runner/internal/execstream/execpb"
15+)
616
717 func TestCacheURL(t *testing.T) {
818 for _, tc := range []struct {
9- gw string
1019 port uint32
1120 want string
1221 }{
1322 // The toolkit appends "_apis/artifactcache/…" to this: the slash is needed.
14- {"10.200.0.1", 8089, "http://10.200.0.1:8089/"},
15- {"", 8089, ""},
16- {"10.200.0.1", 0, ""},
17- {"10.200.0.1", 70000, ""},
23+ {9089, "http://127.0.0.1:9089/"},
24+ {0, ""},
25+ {70000, ""},
1826 } {
19- if got := cacheURL(tc.gw, tc.port); got != tc.want {
20- t.Errorf("cacheURL(%q, %d) = %q, want %q", tc.gw, tc.port, got, tc.want)
27+ if got := cacheURL(tc.port); got != tc.want {
28+ t.Errorf("cacheURL(%d) = %q, want %q", tc.port, got, tc.want)
29+ }
30+ }
31+}
32+
33+// freePort returns a loopback port nobody listens on.
34+func freePort(t *testing.T) uint32 {
35+ t.Helper()
36+ ln, err := net.Listen("tcp", "127.0.0.1:0")
37+ if err != nil {
38+ t.Skipf("loopback sockets are not allowed here: %v", err)
39+ }
40+ defer func() { _ = ln.Close() }()
41+ return uint32(ln.Addr().(*net.TCPAddr).Port)
42+}
43+
44+// A job with a cache service gets the relay on its loopback port, and the URL
45+// and token for the actions; the relay reaches the cache channel of the host,
46+// the sibling of the executor target, and ends with the job.
47+func TestStartCacheRelay(t *testing.T) {
48+ dir := t.TempDir()
49+ execSock := filepath.Join(dir, "exec.sock")
50+ ln, err := net.Listen("unix", filepath.Join(dir, "cache.sock"))
51+ if err != nil {
52+ t.Fatal(err)
53+ }
54+ srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
55+ _, _ = io.WriteString(w, "cache says "+r.Header.Get("Authorization"))
56+ })}
57+ go func() { _ = srv.Serve(ln) }()
58+ t.Cleanup(func() { _ = srv.Close() })
59+
60+ port := freePort(t)
61+ spec := &execpb.JobSpec{CachePort: port, CacheToken: "tok"}
62+ got := startCacheRelay(context.Background(), Target{Network: "unix", Addr: execSock}, spec)
63+ if got.relay == nil || got.token != "tok" || got.url != cacheURL(port) {
64+ t.Fatalf("relay %+v", got)
65+ }
66+
67+ req, _ := http.NewRequest("GET", got.url+"_apis/artifactcache/cache", nil)
68+ req.Header.Set("Authorization", "Bearer tok")
69+ cl := &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{DisableKeepAlives: true}}
70+ resp, err := cl.Do(req)
71+ if err != nil {
72+ t.Fatal(err)
73+ }
74+ body, _ := io.ReadAll(resp.Body)
75+ _ = resp.Body.Close()
76+ if string(body) != "cache says Bearer tok" {
77+ t.Fatalf("got %q", body)
78+ }
79+
80+ got.close()
81+ if c, err := net.DialTimeout("tcp", got.relay.Addr().String(), time.Second); err == nil {
82+ _ = c.Close()
83+ t.Fatal("the relay outlives the job")
84+ }
85+}
86+
87+// No cache service, no relay and no variables; a port that is taken leaves the
88+// job without a cache instead of failing it.
89+func TestStartCacheRelayWithoutServiceOrPort(t *testing.T) {
90+ target := Target{Network: "unix", Addr: filepath.Join(t.TempDir(), "exec.sock")}
91+ for _, spec := range []*execpb.JobSpec{{}, {CachePort: 9089}, {CacheToken: "tok"}} {
92+ if got := startCacheRelay(context.Background(), target, spec); got != (cacheRelay{}) {
93+ t.Errorf("%+v: relay %+v", spec, got)
2194 }
2295 }
96+ busy, err := net.Listen("tcp", "127.0.0.1:0")
97+ if err != nil {
98+ t.Skipf("loopback sockets are not allowed here: %v", err)
99+ }
100+ defer func() { _ = busy.Close() }()
101+ spec := &execpb.JobSpec{CachePort: uint32(busy.Addr().(*net.TCPAddr).Port), CacheToken: "tok"}
102+ if got := startCacheRelay(context.Background(), target, spec); got != (cacheRelay{}) {
103+ t.Errorf("taken port: relay %+v", got)
104+ }
23105 }
@@ -2,22 +2,104 @@
2 2
3 package guest3 package guest
4 4
5-import "testing"5+import (
6+ "context"
7+ "io"
8+ "net"
9+ "net/http"
10+ "path/filepath"
11+ "testing"
12+ "time"
13+
14+ "bnema.dev/ermite-runner/internal/execstream/execpb"
15+)
6 16
7 func TestCacheURL(t *testing.T) {17 func TestCacheURL(t *testing.T) {
8 for _, tc := range []struct {18 for _, tc := range []struct {
9- gw string
10 port uint3219 port uint32
11 want string20 want string
12 }{21 }{
13 // The toolkit appends "_apis/artifactcache/…" to this: the slash is needed.22 // The toolkit appends "_apis/artifactcache/…" to this: the slash is needed.
14- {"10.200.0.1", 8089, "http://10.200.0.1:8089/"},23+ {9089, "http://127.0.0.1:9089/"},
15- {"", 8089, ""},24+ {0, ""},
16- {"10.200.0.1", 0, ""},25+ {70000, ""},
17- {"10.200.0.1", 70000, ""},
18 } {26 } {
19- if got := cacheURL(tc.gw, tc.port); got != tc.want {27+ if got := cacheURL(tc.port); got != tc.want {
20- t.Errorf("cacheURL(%q, %d) = %q, want %q", tc.gw, tc.port, got, tc.want)28+ t.Errorf("cacheURL(%d) = %q, want %q", tc.port, got, tc.want)
29+ }
30+ }
31+}
32+
33+// freePort returns a loopback port nobody listens on.
34+func freePort(t *testing.T) uint32 {
35+ t.Helper()
36+ ln, err := net.Listen("tcp", "127.0.0.1:0")
37+ if err != nil {
38+ t.Skipf("loopback sockets are not allowed here: %v", err)
39+ }
40+ defer func() { _ = ln.Close() }()
41+ return uint32(ln.Addr().(*net.TCPAddr).Port)
42+}
43+
44+// A job with a cache service gets the relay on its loopback port, and the URL
45+// and token for the actions; the relay reaches the cache channel of the host,
46+// the sibling of the executor target, and ends with the job.
47+func TestStartCacheRelay(t *testing.T) {
48+ dir := t.TempDir()
49+ execSock := filepath.Join(dir, "exec.sock")
50+ ln, err := net.Listen("unix", filepath.Join(dir, "cache.sock"))
51+ if err != nil {
52+ t.Fatal(err)
53+ }
54+ srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
55+ _, _ = io.WriteString(w, "cache says "+r.Header.Get("Authorization"))
56+ })}
57+ go func() { _ = srv.Serve(ln) }()
58+ t.Cleanup(func() { _ = srv.Close() })
59+
60+ port := freePort(t)
61+ spec := &execpb.JobSpec{CachePort: port, CacheToken: "tok"}
62+ got := startCacheRelay(context.Background(), Target{Network: "unix", Addr: execSock}, spec)
63+ if got.relay == nil || got.token != "tok" || got.url != cacheURL(port) {
64+ t.Fatalf("relay %+v", got)
65+ }
66+
67+ req, _ := http.NewRequest("GET", got.url+"_apis/artifactcache/cache", nil)
68+ req.Header.Set("Authorization", "Bearer tok")
69+ cl := &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{DisableKeepAlives: true}}
70+ resp, err := cl.Do(req)
71+ if err != nil {
72+ t.Fatal(err)
73+ }
74+ body, _ := io.ReadAll(resp.Body)
75+ _ = resp.Body.Close()
76+ if string(body) != "cache says Bearer tok" {
77+ t.Fatalf("got %q", body)
78+ }
79+
80+ got.close()
81+ if c, err := net.DialTimeout("tcp", got.relay.Addr().String(), time.Second); err == nil {
82+ _ = c.Close()
83+ t.Fatal("the relay outlives the job")
84+ }
85+}
86+
87+// No cache service, no relay and no variables; a port that is taken leaves the
88+// job without a cache instead of failing it.
89+func TestStartCacheRelayWithoutServiceOrPort(t *testing.T) {
90+ target := Target{Network: "unix", Addr: filepath.Join(t.TempDir(), "exec.sock")}
91+ for _, spec := range []*execpb.JobSpec{{}, {CachePort: 9089}, {CacheToken: "tok"}} {
92+ if got := startCacheRelay(context.Background(), target, spec); got != (cacheRelay{}) {
93+ t.Errorf("%+v: relay %+v", spec, got)
21 }94 }
22 }95 }
96+ busy, err := net.Listen("tcp", "127.0.0.1:0")
97+ if err != nil {
98+ t.Skipf("loopback sockets are not allowed here: %v", err)
99+ }
100+ defer func() { _ = busy.Close() }()
101+ spec := &execpb.JobSpec{CachePort: uint32(busy.Addr().(*net.TCPAddr).Port), CacheToken: "tok"}
102+ if got := startCacheRelay(context.Background(), target, spec); got != (cacheRelay{}) {
103+ t.Errorf("taken port: relay %+v", got)
104+ }
23 }105 }
M internal/guest/connect.go +34 -2
@@ -7,11 +7,14 @@ import (
77 "fmt"
88 "io"
99 "net"
10+ "path/filepath"
1011 "strconv"
1112 "strings"
1213 "time"
1314
1415 "golang.org/x/sys/unix"
16+
17+ "bnema.dev/ermite-runner/internal/sandbox"
1518 )
1619
1720 // Target is where the executor connects back to: "unix:/path" (container) or
@@ -41,6 +44,32 @@ func ParseTarget(s string) (Target, error) {
4144
4245 func (t Target) String() string { return t.Network + ":" + t.Addr }
4346
47+// CacheTarget is where the cache streams of the sandbox connect: the host
48+// channel of the executor (t) has a sibling for the cache, on the same
49+// sandbox-private path. A VM uses its own vsock port (sandbox.CacheVsockPort);
50+// a container uses sandbox.CacheSocket next to the executor socket.
51+func (t Target) CacheTarget() Target {
52+ if t.Network == "vsock" {
53+ return Target{Network: "vsock", Addr: strconv.Itoa(sandbox.CacheVsockPort)}
54+ }
55+ return Target{Network: "unix", Addr: filepath.Join(filepath.Dir(t.Addr), sandbox.CacheSocket)}
56+}
57+
58+// DialStream opens one new connection to the target, without retrying: the
59+// cache relay opens one per accepted TCP connection and a failure closes that
60+// connection (the client retries).
61+func (t Target) DialStream(ctx context.Context) (io.ReadWriteCloser, error) {
62+ if t.Network == "unix" {
63+ var d net.Dialer
64+ return d.DialContext(ctx, "unix", t.Addr)
65+ }
66+ c, err := dialVsock(t.Addr)
67+ if err != nil {
68+ return nil, err
69+ }
70+ return c, nil
71+}
72+
4473 // Dial connects to the host, retrying until ctx ends or timeout elapses: the
4574 // host listens before the sandbox starts, but a vsock device may take a moment.
4675 func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCloser, error) {
@@ -54,7 +83,10 @@ func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCl
5483 var d net.Dialer
5584 c, err = d.DialContext(ctx, "unix", t.Addr)
5685 } else {
57- c, err = dialVsock(t.Addr)
86+ var vc *fdConn
87+ if vc, err = dialVsock(t.Addr); err == nil {
88+ c = vc
89+ }
5890 }
5991 if err == nil {
6092 return c, nil
@@ -71,7 +103,7 @@ func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCl
71103 // vsockHostCID is the well-known CID of the host.
72104 const vsockHostCID = 2
73105
74-func dialVsock(port string) (io.ReadWriteCloser, error) {
106+func dialVsock(port string) (*fdConn, error) {
75107 p, _ := strconv.ParseUint(port, 10, 32)
76108 fd, err := unix.Socket(unix.AF_VSOCK, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)
77109 if err != nil {
@@ -7,11 +7,14 @@ import (
7 "fmt"7 "fmt"
8 "io"8 "io"
9 "net"9 "net"
10+ "path/filepath"
10 "strconv"11 "strconv"
11 "strings"12 "strings"
12 "time"13 "time"
13 14
14 "golang.org/x/sys/unix"15 "golang.org/x/sys/unix"
16+
17+ "bnema.dev/ermite-runner/internal/sandbox"
15 )18 )
16 19
17 // Target is where the executor connects back to: "unix:/path" (container) or20 // Target is where the executor connects back to: "unix:/path" (container) or
@@ -41,6 +44,32 @@ func ParseTarget(s string) (Target, error) {
41 44
42 func (t Target) String() string { return t.Network + ":" + t.Addr }45 func (t Target) String() string { return t.Network + ":" + t.Addr }
43 46
47+// CacheTarget is where the cache streams of the sandbox connect: the host
48+// channel of the executor (t) has a sibling for the cache, on the same
49+// sandbox-private path. A VM uses its own vsock port (sandbox.CacheVsockPort);
50+// a container uses sandbox.CacheSocket next to the executor socket.
51+func (t Target) CacheTarget() Target {
52+ if t.Network == "vsock" {
53+ return Target{Network: "vsock", Addr: strconv.Itoa(sandbox.CacheVsockPort)}
54+ }
55+ return Target{Network: "unix", Addr: filepath.Join(filepath.Dir(t.Addr), sandbox.CacheSocket)}
56+}
57+
58+// DialStream opens one new connection to the target, without retrying: the
59+// cache relay opens one per accepted TCP connection and a failure closes that
60+// connection (the client retries).
61+func (t Target) DialStream(ctx context.Context) (io.ReadWriteCloser, error) {
62+ if t.Network == "unix" {
63+ var d net.Dialer
64+ return d.DialContext(ctx, "unix", t.Addr)
65+ }
66+ c, err := dialVsock(t.Addr)
67+ if err != nil {
68+ return nil, err
69+ }
70+ return c, nil
71+}
72+
44 // Dial connects to the host, retrying until ctx ends or timeout elapses: the73 // Dial connects to the host, retrying until ctx ends or timeout elapses: the
45 // host listens before the sandbox starts, but a vsock device may take a moment.74 // host listens before the sandbox starts, but a vsock device may take a moment.
46 func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCloser, error) {75 func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCloser, error) {
@@ -54,7 +83,10 @@ func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCl
54 var d net.Dialer83 var d net.Dialer
55 c, err = d.DialContext(ctx, "unix", t.Addr)84 c, err = d.DialContext(ctx, "unix", t.Addr)
56 } else {85 } else {
57- c, err = dialVsock(t.Addr)86+ var vc *fdConn
87+ if vc, err = dialVsock(t.Addr); err == nil {
88+ c = vc
89+ }
58 }90 }
59 if err == nil {91 if err == nil {
60 return c, nil92 return c, nil
@@ -71,7 +103,7 @@ func (t Target) Dial(ctx context.Context, timeout time.Duration) (io.ReadWriteCl
71 // vsockHostCID is the well-known CID of the host.103 // vsockHostCID is the well-known CID of the host.
72 const vsockHostCID = 2104 const vsockHostCID = 2
73 105
74-func dialVsock(port string) (io.ReadWriteCloser, error) {106+func dialVsock(port string) (*fdConn, error) {
75 p, _ := strconv.ParseUint(port, 10, 32)107 p, _ := strconv.ParseUint(port, 10, 32)
76 fd, err := unix.Socket(unix.AF_VSOCK, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)108 fd, err := unix.Socket(unix.AF_VSOCK, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)
77 if err != nil {109 if err != nil {
M internal/guest/fdconn.go +20 -5
@@ -3,18 +3,33 @@
33 package guest
44
55 import (
6- "io"
76 "os"
87
98 "golang.org/x/sys/unix"
109 )
1110
12-// newFDConn wraps a connected stream socket (net.FileConn does not know
13-// AF_VSOCK) as a ReadWriteCloser registered with the runtime poller.
14-func newFDConn(fd int) (io.ReadWriteCloser, error) {
11+// fdConn is a connected stream socket as a ReadWriteCloser registered with the
12+// runtime poller (net.FileConn does not know AF_VSOCK). It can shut down its
13+// write side, which the cache relay passes on from the client.
14+type fdConn struct{ *os.File }
15+
16+func newFDConn(fd int) (*fdConn, error) {
1517 if err := unix.SetNonblock(fd, true); err != nil {
1618 _ = unix.Close(fd)
1719 return nil, err
1820 }
19- return os.NewFile(uintptr(fd), "vsock"), nil
21+ return &fdConn{os.NewFile(uintptr(fd), "vsock")}, nil
22+}
23+
24+// CloseWrite sends the end of the stream (shutdown(SHUT_WR)) and keeps reading.
25+func (c *fdConn) CloseWrite() error {
26+ rc, err := c.SyscallConn()
27+ if err != nil {
28+ return err
29+ }
30+ var serr error
31+ if err := rc.Control(func(fd uintptr) { serr = unix.Shutdown(int(fd), unix.SHUT_WR) }); err != nil {
32+ return err
33+ }
34+ return serr
2035 }
@@ -3,18 +3,33 @@
3 package guest3 package guest
4 4
5 import (5 import (
6- "io"
7 "os"6 "os"
8 7
9 "golang.org/x/sys/unix"8 "golang.org/x/sys/unix"
10 )9 )
11 10
12-// newFDConn wraps a connected stream socket (net.FileConn does not know11+// fdConn is a connected stream socket as a ReadWriteCloser registered with the
13-// AF_VSOCK) as a ReadWriteCloser registered with the runtime poller.12+// runtime poller (net.FileConn does not know AF_VSOCK). It can shut down its
14-func newFDConn(fd int) (io.ReadWriteCloser, error) {13+// write side, which the cache relay passes on from the client.
14+type fdConn struct{ *os.File }
15+
16+func newFDConn(fd int) (*fdConn, error) {
15 if err := unix.SetNonblock(fd, true); err != nil {17 if err := unix.SetNonblock(fd, true); err != nil {
16 _ = unix.Close(fd)18 _ = unix.Close(fd)
17 return nil, err19 return nil, err
18 }20 }
19- return os.NewFile(uintptr(fd), "vsock"), nil21+ return &fdConn{os.NewFile(uintptr(fd), "vsock")}, nil
22+}
23+
24+// CloseWrite sends the end of the stream (shutdown(SHUT_WR)) and keeps reading.
25+func (c *fdConn) CloseWrite() error {
26+ rc, err := c.SyscallConn()
27+ if err != nil {
28+ return err
29+ }
30+ var serr error
31+ if err := rc.Control(func(fd uintptr) { serr = unix.Shutdown(int(fd), unix.SHUT_WR) }); err != nil {
32+ return err
33+ }
34+ return serr
20 }35 }
M internal/guest/init.go +0 -5
@@ -182,11 +182,6 @@ func runPayload(ctx context.Context, cfg InitConfig, mode string, args map[strin
182182 }
183183
184184 execEnv := append(os.Environ(), "ERMITE_CONNECT="+target, "ERMITE_SANDBOX="+mode, "ERMITE_VERSION="+cfg.Version)
185- if gw := args[sandbox.CmdlineGateway]; gw != "" {
186- // A VM learns its gateway from the kernel command line; a container is
187- // given it in its environment (the container driver).
188- execEnv = append(execEnv, "ERMITE_GATEWAY="+gw)
189- }
190185 execPid, err := spawn(cfg.ExecArgs, execEnv)
191186 if err != nil {
192187 log.Error().Err(err).Msg("init: start executor")
@@ -182,11 +182,6 @@ func runPayload(ctx context.Context, cfg InitConfig, mode string, args map[strin
182 }182 }
183 183
184 execEnv := append(os.Environ(), "ERMITE_CONNECT="+target, "ERMITE_SANDBOX="+mode, "ERMITE_VERSION="+cfg.Version)184 execEnv := append(os.Environ(), "ERMITE_CONNECT="+target, "ERMITE_SANDBOX="+mode, "ERMITE_VERSION="+cfg.Version)
185- if gw := args[sandbox.CmdlineGateway]; gw != "" {
186- // A VM learns its gateway from the kernel command line; a container is
187- // given it in its environment (the container driver).
188- execEnv = append(execEnv, "ERMITE_GATEWAY="+gw)
189- }
190 execPid, err := spawn(cfg.ExecArgs, execEnv)185 execPid, err := spawn(cfg.ExecArgs, execEnv)
191 if err != nil {186 if err != nil {
192 log.Error().Err(err).Msg("init: start executor")187 log.Error().Err(err).Msg("init: start executor")
M internal/guest/jobexec.go +44 -7
@@ -214,6 +214,8 @@ func runJob(ctx context.Context, conn *execstream.Conn, cfg ExecConfig, spec *ex
214214 canceller := executor.NewCanceller()
215215 rep := &frameReporter{conn: conn, failed: make(chan struct{})}
216216 res := newResolver(conn)
217+ cacheEnv := startCacheRelay(ctx, cfg.Target, spec)
218+ defer cacheEnv.close()
217219
218220 // Frames from the host while the job runs: a Cancel stops the job; a lost
219221 // host does too, like a runner shutdown. The reader ends with the stream: the
@@ -257,7 +259,7 @@ func runJob(ctx context.Context, conn *execstream.Conn, cfg ExecConfig, spec *ex
257259 Runner: executor.RunnerInfo{Name: spec.GetRunnerName(), Environment: env},
258260 RunnerVersion: spec.GetRunnerVersion(),
259261 Home: su.Home, ToolCache: os.Getenv("RUNNER_TOOL_CACHE"), BaseEnv: stepEnv(os.Environ(), su), Credential: su.Credential,
260- CacheURL: cacheURL(os.Getenv("ERMITE_GATEWAY"), spec.GetCachePort()), CacheToken: spec.GetCacheToken(),
262+ CacheURL: cacheEnv.url, CacheToken: cacheEnv.token,
261263 Debug: spec.GetDebug(), Reporter: rep, Actions: executor.DefaultActionsConfig(res.resolve), Log: zerowrap.FromCtx(ctx),
262264 Secrets: vals.secrets, Vars: vals.vars, SecretsWithheld: spec.GetSecretsWithheld(), TokenPermissions: tokenPermissions(spec),
263265 SendAnnotations: spec.GetSendAnnotations(), SendSummaries: spec.GetSendSummaries(),
@@ -269,14 +271,49 @@ func runJob(ctx context.Context, conn *execstream.Conn, cfg ExecConfig, spec *ex
269271 return 1
270272 }
271273
272-// cacheURL is ACTIONS_CACHE_URL: the cache service of the runner on the gateway
273-// of the sandbox. The trailing slash matters: the toolkit appends "_apis/…"
274-// to it (cacheHttpClient.ts:47). Empty when the job has no cache service.
275-func cacheURL(gateway string, port uint32) string {
276- if gateway == "" || port == 0 || port > 65535 {
274+// cacheURL is ACTIONS_CACHE_URL: the cache service of the runner on the
275+// loopback of the sandbox, where the relay listens. The trailing slash matters:
276+// the toolkit appends "_apis/…" to it (cacheHttpClient.ts:47). Empty when the
277+// job has no cache service.
278+func cacheURL(port uint32) string {
279+ if port == 0 || port > 65535 {
277280 return ""
278281 }
279- return "http://" + net.JoinHostPort(gateway, strconv.FormatUint(uint64(port), 10)) + "/"
282+ return "http://" + net.JoinHostPort(CacheListenHost, strconv.FormatUint(uint64(port), 10)) + "/"
283+}
284+
285+// cacheRelay is the cache of a job in the sandbox: the relay, the URL of the
286+// loopback it listens on and the token of the job. All are empty without a cache
287+// service.
288+type cacheRelay struct {
289+ relay *Relay
290+ url, token string
291+}
292+
293+func (c cacheRelay) close() {
294+ if c.relay != nil {
295+ _ = c.relay.Close()
296+ }
297+}
298+
299+// startCacheRelay starts the relay of the cache for a job that has a cache
300+// service: it listens on the loopback port the host says and pipes every
301+// connection to a new stream on the cache channel of the host (a sibling of
302+// the executor's target). A relay that cannot start leaves the job without a
303+// cache, like a scope the host refuses: the job runs.
304+func startCacheRelay(ctx context.Context, target Target, spec *execpb.JobSpec) cacheRelay {
305+ url := cacheURL(spec.GetCachePort())
306+ if url == "" || spec.GetCacheToken() == "" {
307+ return cacheRelay{}
308+ }
309+ addr := net.JoinHostPort(CacheListenHost, strconv.FormatUint(uint64(spec.GetCachePort()), 10))
310+ r, err := StartRelay(ctx, addr, target.CacheTarget().DialStream, DefaultRelayMax)
311+ if err != nil {
312+ log := zerowrap.FromCtx(ctx)
313+ log.Warn().Err(err).Msg("job runs without the cache")
314+ return cacheRelay{}
315+ }
316+ return cacheRelay{relay: r, url: url, token: spec.GetCacheToken()}
280317 }
281318
282319 // failSetup ends a job that cannot start, like a failed "Set up job": step 1
@@ -214,6 +214,8 @@ func runJob(ctx context.Context, conn *execstream.Conn, cfg ExecConfig, spec *ex
214 canceller := executor.NewCanceller()214 canceller := executor.NewCanceller()
215 rep := &frameReporter{conn: conn, failed: make(chan struct{})}215 rep := &frameReporter{conn: conn, failed: make(chan struct{})}
216 res := newResolver(conn)216 res := newResolver(conn)
217+ cacheEnv := startCacheRelay(ctx, cfg.Target, spec)
218+ defer cacheEnv.close()
217 219
218 // Frames from the host while the job runs: a Cancel stops the job; a lost220 // Frames from the host while the job runs: a Cancel stops the job; a lost
219 // host does too, like a runner shutdown. The reader ends with the stream: the221 // host does too, like a runner shutdown. The reader ends with the stream: the
@@ -257,7 +259,7 @@ func runJob(ctx context.Context, conn *execstream.Conn, cfg ExecConfig, spec *ex
257 Runner: executor.RunnerInfo{Name: spec.GetRunnerName(), Environment: env},259 Runner: executor.RunnerInfo{Name: spec.GetRunnerName(), Environment: env},
258 RunnerVersion: spec.GetRunnerVersion(),260 RunnerVersion: spec.GetRunnerVersion(),
259 Home: su.Home, ToolCache: os.Getenv("RUNNER_TOOL_CACHE"), BaseEnv: stepEnv(os.Environ(), su), Credential: su.Credential,261 Home: su.Home, ToolCache: os.Getenv("RUNNER_TOOL_CACHE"), BaseEnv: stepEnv(os.Environ(), su), Credential: su.Credential,
260- CacheURL: cacheURL(os.Getenv("ERMITE_GATEWAY"), spec.GetCachePort()), CacheToken: spec.GetCacheToken(),262+ CacheURL: cacheEnv.url, CacheToken: cacheEnv.token,
261 Debug: spec.GetDebug(), Reporter: rep, Actions: executor.DefaultActionsConfig(res.resolve), Log: zerowrap.FromCtx(ctx),263 Debug: spec.GetDebug(), Reporter: rep, Actions: executor.DefaultActionsConfig(res.resolve), Log: zerowrap.FromCtx(ctx),
262 Secrets: vals.secrets, Vars: vals.vars, SecretsWithheld: spec.GetSecretsWithheld(), TokenPermissions: tokenPermissions(spec),264 Secrets: vals.secrets, Vars: vals.vars, SecretsWithheld: spec.GetSecretsWithheld(), TokenPermissions: tokenPermissions(spec),
263 SendAnnotations: spec.GetSendAnnotations(), SendSummaries: spec.GetSendSummaries(),265 SendAnnotations: spec.GetSendAnnotations(), SendSummaries: spec.GetSendSummaries(),
@@ -269,14 +271,49 @@ func runJob(ctx context.Context, conn *execstream.Conn, cfg ExecConfig, spec *ex
269 return 1271 return 1
270 }272 }
271 273
272-// cacheURL is ACTIONS_CACHE_URL: the cache service of the runner on the gateway274+// cacheURL is ACTIONS_CACHE_URL: the cache service of the runner on the
273-// of the sandbox. The trailing slash matters: the toolkit appends "_apis/…"275+// loopback of the sandbox, where the relay listens. The trailing slash matters:
274-// to it (cacheHttpClient.ts:47). Empty when the job has no cache service.276+// the toolkit appends "_apis/…" to it (cacheHttpClient.ts:47). Empty when the
275-func cacheURL(gateway string, port uint32) string {277+// job has no cache service.
276- if gateway == "" || port == 0 || port > 65535 {278+func cacheURL(port uint32) string {
279+ if port == 0 || port > 65535 {
277 return ""280 return ""
278 }281 }
279- return "http://" + net.JoinHostPort(gateway, strconv.FormatUint(uint64(port), 10)) + "/"282+ return "http://" + net.JoinHostPort(CacheListenHost, strconv.FormatUint(uint64(port), 10)) + "/"
283+}
284+
285+// cacheRelay is the cache of a job in the sandbox: the relay, the URL of the
286+// loopback it listens on and the token of the job. All are empty without a cache
287+// service.
288+type cacheRelay struct {
289+ relay *Relay
290+ url, token string
291+}
292+
293+func (c cacheRelay) close() {
294+ if c.relay != nil {
295+ _ = c.relay.Close()
296+ }
297+}
298+
299+// startCacheRelay starts the relay of the cache for a job that has a cache
300+// service: it listens on the loopback port the host says and pipes every
301+// connection to a new stream on the cache channel of the host (a sibling of
302+// the executor's target). A relay that cannot start leaves the job without a
303+// cache, like a scope the host refuses: the job runs.
304+func startCacheRelay(ctx context.Context, target Target, spec *execpb.JobSpec) cacheRelay {
305+ url := cacheURL(spec.GetCachePort())
306+ if url == "" || spec.GetCacheToken() == "" {
307+ return cacheRelay{}
308+ }
309+ addr := net.JoinHostPort(CacheListenHost, strconv.FormatUint(uint64(spec.GetCachePort()), 10))
310+ r, err := StartRelay(ctx, addr, target.CacheTarget().DialStream, DefaultRelayMax)
311+ if err != nil {
312+ log := zerowrap.FromCtx(ctx)
313+ log.Warn().Err(err).Msg("job runs without the cache")
314+ return cacheRelay{}
315+ }
316+ return cacheRelay{relay: r, url: url, token: spec.GetCacheToken()}
280 }317 }
281 318
282 // failSetup ends a job that cannot start, like a failed "Set up job": step 1319 // failSetup ends a job that cannot start, like a failed "Set up job": step 1
A internal/guest/relay.go +233 -0
@@ -0,0 +1,233 @@
1+// SPDX-License-Identifier: MIT
2+
3+package guest
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "io"
10+ "net"
11+ "sync"
12+ "time"
13+)
14+
15+// The cache service of the runner never listens on a network. In the sandbox it
16+// is reached on the guest's own loopback (CacheListenHost, on the port the
17+// host gives in JobSpec.cache_port), where a Relay accepts the TCP connections
18+// of the actions and pipes each of them to a new stream on the host↔guest
19+// channel (a vsock port for a VM, a Unix socket for a container); the host
20+// serves HTTP on the other end of that stream.
21+const (
22+ // CacheListenHost is the loopback address of the relay.
23+ CacheListenHost = "127.0.0.1"
24+ // DefaultRelayMax bounds the connections a Relay pipes at once. The host
25+ // bounds them too (cachesvc.DefaultConnsPerListener): a connection over
26+ // either limit is closed at once and the client retries.
27+ DefaultRelayMax = 32
28+ // relayDialTimeout bounds one connection to the host: a stream either opens
29+ // at once or fails.
30+ relayDialTimeout = 5 * time.Second
31+ // DefaultHalfCloseGrace is how long the second direction of a connection may
32+ // stay idle once the first has ended. A half-close is passed on, but whether
33+ // every channel carries it is not something the relay can rely on (it is
34+ // untested over Firecracker's vsock proxy): without a limit, a client that
35+ // closed its connection could hold a slot until the host's idle timeout. A
36+ // transfer that makes progress is never idle, so only a stalled half is cut.
37+ DefaultHalfCloseGrace = 30 * time.Second
38+)
39+
40+// DialFunc opens one new stream to the host.
41+type DialFunc func(ctx context.Context) (io.ReadWriteCloser, error)
42+
43+// Relay accepts TCP connections on a loopback address and pipes each one to a
44+// new stream from its DialFunc, in both directions, until Close.
45+type Relay struct {
46+ ln net.Listener
47+ dial DialFunc
48+ slot chan struct{}
49+ grace time.Duration
50+
51+ ctx context.Context
52+ cancel context.CancelFunc
53+ wg sync.WaitGroup
54+
55+ mu sync.Mutex
56+ active map[io.Closer]struct{}
57+}
58+
59+// StartRelay listens on addr ("127.0.0.1:9089") and relays every connection to
60+// a stream made by dial; at most max connections are open at once (<= 0 means
61+// DefaultRelayMax). The relay ends with Close or when ctx ends.
62+func StartRelay(ctx context.Context, addr string, dial DialFunc, max int) (*Relay, error) {
63+ return newRelay(ctx, addr, dial, max, DefaultHalfCloseGrace)
64+}
65+
66+func newRelay(ctx context.Context, addr string, dial DialFunc, max int, grace time.Duration) (*Relay, error) {
67+ if max <= 0 {
68+ max = DefaultRelayMax
69+ }
70+ ln, err := net.Listen("tcp", addr)
71+ if err != nil {
72+ return nil, fmt.Errorf("cache relay: %w", err)
73+ }
74+ rctx, cancel := context.WithCancel(ctx)
75+ r := &Relay{ln: ln, dial: dial, slot: make(chan struct{}, max), grace: grace, ctx: rctx, cancel: cancel, active: map[io.Closer]struct{}{}}
76+ r.wg.Add(1)
77+ go r.accept()
78+ context.AfterFunc(rctx, func() { _ = r.Close() })
79+ return r, nil
80+}
81+
82+// Addr is the address the relay listens on.
83+func (r *Relay) Addr() net.Addr { return r.ln.Addr() }
84+
85+func (r *Relay) accept() {
86+ defer r.wg.Done()
87+ for {
88+ c, err := r.ln.Accept()
89+ if err != nil {
90+ if r.ctx.Err() != nil || errors.Is(err, net.ErrClosed) {
91+ return
92+ }
93+ // A transient accept error (descriptors, aborted connection): go on.
94+ select {
95+ case <-time.After(10 * time.Millisecond):
96+ continue
97+ case <-r.ctx.Done():
98+ return
99+ }
100+ }
101+ select {
102+ case r.slot <- struct{}{}:
103+ default:
104+ _ = c.Close() // over the cap: never queued
105+ continue
106+ }
107+ r.wg.Add(1)
108+ go func() {
109+ defer r.wg.Done()
110+ defer func() { <-r.slot }()
111+ r.pipe(c)
112+ }()
113+ }
114+}
115+
116+// track registers a connection to be closed by Close; it reports false (and
117+// closes c) when the relay is already closing.
118+func (r *Relay) track(c io.Closer) bool {
119+ r.mu.Lock()
120+ defer r.mu.Unlock()
121+ if r.ctx.Err() != nil {
122+ _ = c.Close()
123+ return false
124+ }
125+ r.active[c] = struct{}{}
126+ return true
127+}
128+
129+func (r *Relay) untrack(c io.Closer) {
130+ r.mu.Lock()
131+ delete(r.active, c)
132+ r.mu.Unlock()
133+ _ = c.Close()
134+}
135+
136+func (r *Relay) pipe(client net.Conn) {
137+ if !r.track(client) {
138+ return
139+ }
140+ defer r.untrack(client)
141+ dctx, cancel := context.WithTimeout(r.ctx, relayDialTimeout)
142+ host, err := r.dial(dctx)
143+ cancel()
144+ if err != nil {
145+ return
146+ }
147+ if !r.track(host) {
148+ return
149+ }
150+ defer r.untrack(host)
151+ Pipe(client, host, r.grace)
152+}
153+
154+// closeWriter is a stream that can shut down its write side only.
155+type closeWriter interface{ CloseWrite() error }
156+
157+// Pipe copies a to b and b to a until both directions end. The end of one
158+// direction is passed on as a half-close of the other side's write end, so a
159+// request sent in full and answered afterwards works; a stream that cannot
160+// half-close is closed once its peer is done. Once one direction has ended, the
161+// other may stay idle for grace (0: no limit) before both are closed. Pipe
162+// closes a and b only in that case; the caller closes them otherwise.
163+func Pipe(a, b io.ReadWriteCloser, grace time.Duration) {
164+ var (
165+ wg sync.WaitGroup
166+ mu sync.Mutex
167+ timer *time.Timer
168+ done int
169+ )
170+ touch := func() {
171+ mu.Lock()
172+ if timer != nil {
173+ timer.Reset(grace)
174+ }
175+ mu.Unlock()
176+ }
177+ ended := func() {
178+ mu.Lock()
179+ defer mu.Unlock()
180+ done++
181+ switch {
182+ case done == 1 && grace > 0:
183+ timer = time.AfterFunc(grace, func() { _ = a.Close(); _ = b.Close() })
184+ case done == 2 && timer != nil:
185+ timer.Stop()
186+ }
187+ }
188+ cp := func(dst, src io.ReadWriteCloser) {
189+ defer wg.Done()
190+ _, _ = io.Copy(activityWriter{dst, touch}, src)
191+ if cw, ok := dst.(closeWriter); ok {
192+ _ = cw.CloseWrite()
193+ } else {
194+ _ = dst.Close()
195+ }
196+ ended()
197+ }
198+ wg.Add(2)
199+ go cp(a, b)
200+ go cp(b, a)
201+ wg.Wait()
202+}
203+
204+// activityWriter calls touch after every write that goes through.
205+type activityWriter struct {
206+ io.Writer
207+ touch func()
208+}
209+
210+func (w activityWriter) Write(p []byte) (int, error) {
211+ n, err := w.Writer.Write(p)
212+ if n > 0 {
213+ w.touch()
214+ }
215+ return n, err
216+}
217+
218+// Close stops listening, closes every open connection and waits for the pipes
219+// to end. It is idempotent.
220+func (r *Relay) Close() error {
221+ r.cancel()
222+ err := r.ln.Close()
223+ r.mu.Lock()
224+ for c := range r.active {
225+ _ = c.Close()
226+ }
227+ r.mu.Unlock()
228+ r.wg.Wait()
229+ if errors.Is(err, net.ErrClosed) {
230+ return nil
231+ }
232+ return err
233+}
@@ -0,0 +1,233 @@
1+// SPDX-License-Identifier: MIT
2+
3+package guest
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "io"
10+ "net"
11+ "sync"
12+ "time"
13+)
14+
15+// The cache service of the runner never listens on a network. In the sandbox it
16+// is reached on the guest's own loopback (CacheListenHost, on the port the
17+// host gives in JobSpec.cache_port), where a Relay accepts the TCP connections
18+// of the actions and pipes each of them to a new stream on the host↔guest
19+// channel (a vsock port for a VM, a Unix socket for a container); the host
20+// serves HTTP on the other end of that stream.
21+const (
22+ // CacheListenHost is the loopback address of the relay.
23+ CacheListenHost = "127.0.0.1"
24+ // DefaultRelayMax bounds the connections a Relay pipes at once. The host
25+ // bounds them too (cachesvc.DefaultConnsPerListener): a connection over
26+ // either limit is closed at once and the client retries.
27+ DefaultRelayMax = 32
28+ // relayDialTimeout bounds one connection to the host: a stream either opens
29+ // at once or fails.
30+ relayDialTimeout = 5 * time.Second
31+ // DefaultHalfCloseGrace is how long the second direction of a connection may
32+ // stay idle once the first has ended. A half-close is passed on, but whether
33+ // every channel carries it is not something the relay can rely on (it is
34+ // untested over Firecracker's vsock proxy): without a limit, a client that
35+ // closed its connection could hold a slot until the host's idle timeout. A
36+ // transfer that makes progress is never idle, so only a stalled half is cut.
37+ DefaultHalfCloseGrace = 30 * time.Second
38+)
39+
40+// DialFunc opens one new stream to the host.
41+type DialFunc func(ctx context.Context) (io.ReadWriteCloser, error)
42+
43+// Relay accepts TCP connections on a loopback address and pipes each one to a
44+// new stream from its DialFunc, in both directions, until Close.
45+type Relay struct {
46+ ln net.Listener
47+ dial DialFunc
48+ slot chan struct{}
49+ grace time.Duration
50+
51+ ctx context.Context
52+ cancel context.CancelFunc
53+ wg sync.WaitGroup
54+
55+ mu sync.Mutex
56+ active map[io.Closer]struct{}
57+}
58+
59+// StartRelay listens on addr ("127.0.0.1:9089") and relays every connection to
60+// a stream made by dial; at most max connections are open at once (<= 0 means
61+// DefaultRelayMax). The relay ends with Close or when ctx ends.
62+func StartRelay(ctx context.Context, addr string, dial DialFunc, max int) (*Relay, error) {
63+ return newRelay(ctx, addr, dial, max, DefaultHalfCloseGrace)
64+}
65+
66+func newRelay(ctx context.Context, addr string, dial DialFunc, max int, grace time.Duration) (*Relay, error) {
67+ if max <= 0 {
68+ max = DefaultRelayMax
69+ }
70+ ln, err := net.Listen("tcp", addr)
71+ if err != nil {
72+ return nil, fmt.Errorf("cache relay: %w", err)
73+ }
74+ rctx, cancel := context.WithCancel(ctx)
75+ r := &Relay{ln: ln, dial: dial, slot: make(chan struct{}, max), grace: grace, ctx: rctx, cancel: cancel, active: map[io.Closer]struct{}{}}
76+ r.wg.Add(1)
77+ go r.accept()
78+ context.AfterFunc(rctx, func() { _ = r.Close() })
79+ return r, nil
80+}
81+
82+// Addr is the address the relay listens on.
83+func (r *Relay) Addr() net.Addr { return r.ln.Addr() }
84+
85+func (r *Relay) accept() {
86+ defer r.wg.Done()
87+ for {
88+ c, err := r.ln.Accept()
89+ if err != nil {
90+ if r.ctx.Err() != nil || errors.Is(err, net.ErrClosed) {
91+ return
92+ }
93+ // A transient accept error (descriptors, aborted connection): go on.
94+ select {
95+ case <-time.After(10 * time.Millisecond):
96+ continue
97+ case <-r.ctx.Done():
98+ return
99+ }
100+ }
101+ select {
102+ case r.slot <- struct{}{}:
103+ default:
104+ _ = c.Close() // over the cap: never queued
105+ continue
106+ }
107+ r.wg.Add(1)
108+ go func() {
109+ defer r.wg.Done()
110+ defer func() { <-r.slot }()
111+ r.pipe(c)
112+ }()
113+ }
114+}
115+
116+// track registers a connection to be closed by Close; it reports false (and
117+// closes c) when the relay is already closing.
118+func (r *Relay) track(c io.Closer) bool {
119+ r.mu.Lock()
120+ defer r.mu.Unlock()
121+ if r.ctx.Err() != nil {
122+ _ = c.Close()
123+ return false
124+ }
125+ r.active[c] = struct{}{}
126+ return true
127+}
128+
129+func (r *Relay) untrack(c io.Closer) {
130+ r.mu.Lock()
131+ delete(r.active, c)
132+ r.mu.Unlock()
133+ _ = c.Close()
134+}
135+
136+func (r *Relay) pipe(client net.Conn) {
137+ if !r.track(client) {
138+ return
139+ }
140+ defer r.untrack(client)
141+ dctx, cancel := context.WithTimeout(r.ctx, relayDialTimeout)
142+ host, err := r.dial(dctx)
143+ cancel()
144+ if err != nil {
145+ return
146+ }
147+ if !r.track(host) {
148+ return
149+ }
150+ defer r.untrack(host)
151+ Pipe(client, host, r.grace)
152+}
153+
154+// closeWriter is a stream that can shut down its write side only.
155+type closeWriter interface{ CloseWrite() error }
156+
157+// Pipe copies a to b and b to a until both directions end. The end of one
158+// direction is passed on as a half-close of the other side's write end, so a
159+// request sent in full and answered afterwards works; a stream that cannot
160+// half-close is closed once its peer is done. Once one direction has ended, the
161+// other may stay idle for grace (0: no limit) before both are closed. Pipe
162+// closes a and b only in that case; the caller closes them otherwise.
163+func Pipe(a, b io.ReadWriteCloser, grace time.Duration) {
164+ var (
165+ wg sync.WaitGroup
166+ mu sync.Mutex
167+ timer *time.Timer
168+ done int
169+ )
170+ touch := func() {
171+ mu.Lock()
172+ if timer != nil {
173+ timer.Reset(grace)
174+ }
175+ mu.Unlock()
176+ }
177+ ended := func() {
178+ mu.Lock()
179+ defer mu.Unlock()
180+ done++
181+ switch {
182+ case done == 1 && grace > 0:
183+ timer = time.AfterFunc(grace, func() { _ = a.Close(); _ = b.Close() })
184+ case done == 2 && timer != nil:
185+ timer.Stop()
186+ }
187+ }
188+ cp := func(dst, src io.ReadWriteCloser) {
189+ defer wg.Done()
190+ _, _ = io.Copy(activityWriter{dst, touch}, src)
191+ if cw, ok := dst.(closeWriter); ok {
192+ _ = cw.CloseWrite()
193+ } else {
194+ _ = dst.Close()
195+ }
196+ ended()
197+ }
198+ wg.Add(2)
199+ go cp(a, b)
200+ go cp(b, a)
201+ wg.Wait()
202+}
203+
204+// activityWriter calls touch after every write that goes through.
205+type activityWriter struct {
206+ io.Writer
207+ touch func()
208+}
209+
210+func (w activityWriter) Write(p []byte) (int, error) {
211+ n, err := w.Writer.Write(p)
212+ if n > 0 {
213+ w.touch()
214+ }
215+ return n, err
216+}
217+
218+// Close stops listening, closes every open connection and waits for the pipes
219+// to end. It is idempotent.
220+func (r *Relay) Close() error {
221+ r.cancel()
222+ err := r.ln.Close()
223+ r.mu.Lock()
224+ for c := range r.active {
225+ _ = c.Close()
226+ }
227+ r.mu.Unlock()
228+ r.wg.Wait()
229+ if errors.Is(err, net.ErrClosed) {
230+ return nil
231+ }
232+ return err
233+}
A internal/guest/relay_test.go +393 -0
@@ -0,0 +1,393 @@
1+// SPDX-License-Identifier: MIT
2+
3+package guest
4+
5+import (
6+ "context"
7+ "errors"
8+ "io"
9+ "net"
10+ "path/filepath"
11+ "sync/atomic"
12+ "testing"
13+ "time"
14+)
15+
16+// hostSide is a Unix socket server standing for the host end of the channel.
17+type hostSide struct {
18+ ln net.Listener
19+ accepts atomic.Int32
20+}
21+
22+func newHostSide(t *testing.T, serve func(net.Conn)) *hostSide {
23+ t.Helper()
24+ ln, err := net.Listen("unix", filepath.Join(t.TempDir(), "h"))
25+ if err != nil {
26+ t.Fatal(err)
27+ }
28+ h := &hostSide{ln: ln}
29+ t.Cleanup(func() { _ = ln.Close() })
30+ go func() {
31+ for {
32+ c, err := ln.Accept()
33+ if err != nil {
34+ return
35+ }
36+ h.accepts.Add(1)
37+ go serve(c)
38+ }
39+ }()
40+ return h
41+}
42+
43+func (h *hostSide) dial(ctx context.Context) (io.ReadWriteCloser, error) {
44+ var d net.Dialer
45+ return d.DialContext(ctx, "unix", h.ln.Addr().String())
46+}
47+
48+func startRelay(t *testing.T, dial DialFunc, max int) *Relay {
49+ t.Helper()
50+ r, err := StartRelay(context.Background(), "127.0.0.1:0", dial, max)
51+ if err != nil {
52+ t.Skipf("loopback sockets are not allowed here: %v", err)
53+ }
54+ t.Cleanup(func() { _ = r.Close() })
55+ return r
56+}
57+
58+func dialRelay(t *testing.T, r *Relay) *net.TCPConn {
59+ t.Helper()
60+ c, err := net.Dial("tcp", r.Addr().String())
61+ if err != nil {
62+ t.Fatal(err)
63+ }
64+ t.Cleanup(func() { _ = c.Close() })
65+ return c.(*net.TCPConn)
66+}
67+
68+func readAllTimeout(t *testing.T, c net.Conn) []byte {
69+ t.Helper()
70+ _ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
71+ b, err := io.ReadAll(c)
72+ if err != nil {
73+ t.Fatalf("read: %v", err)
74+ }
75+ return b
76+}
77+
78+// Bytes go both ways, and the end of the client's stream reaches the host as an
79+// EOF while the host can still answer (half-close).
80+func TestRelayCopiesBothWaysAndHalfCloses(t *testing.T) {
81+ h := newHostSide(t, func(c net.Conn) {
82+ defer func() { _ = c.Close() }()
83+ req, _ := io.ReadAll(c) // returns only at the client's half-close
84+ _, _ = c.Write(append([]byte("echo:"), req...))
85+ })
86+ r := startRelay(t, h.dial, 4)
87+ c := dialRelay(t, r)
88+ if _, err := c.Write([]byte("hello")); err != nil {
89+ t.Fatal(err)
90+ }
91+ if err := c.CloseWrite(); err != nil {
92+ t.Fatal(err)
93+ }
94+ if got := string(readAllTimeout(t, c)); got != "echo:hello" {
95+ t.Fatalf("got %q", got)
96+ }
97+}
98+
99+// The host closing its side reaches the client as an EOF, and bulk data in both
100+// directions is not lost.
101+func TestRelayBulkAndHostClose(t *testing.T) {
102+ const n = 3 << 20
103+ h := newHostSide(t, func(c net.Conn) {
104+ defer func() { _ = c.Close() }()
105+ got, _ := io.CopyN(io.Discard, c, n)
106+ _, _ = io.CopyN(c, zeroReader{}, got)
107+ })
108+ r := startRelay(t, h.dial, 4)
109+ c := dialRelay(t, r)
110+ go func() { _, _ = io.CopyN(c, zeroReader{}, n) }()
111+ if got := len(readAllTimeout(t, c)); got != n {
112+ t.Fatalf("read %d bytes, want %d", got, n)
113+ }
114+}
115+
116+type zeroReader struct{}
117+
118+func (zeroReader) Read(p []byte) (int, error) { clear(p); return len(p), nil }
119+
120+// Every accepted connection is a new stream to the host.
121+func TestRelayOneStreamPerConnection(t *testing.T) {
122+ h := newHostSide(t, func(c net.Conn) {
123+ defer func() { _ = c.Close() }()
124+ _, _ = io.Copy(c, c)
125+ })
126+ r := startRelay(t, h.dial, 8)
127+ for i := 0; i < 3; i++ {
128+ c := dialRelay(t, r)
129+ _, _ = c.Write([]byte("x"))
130+ _ = c.CloseWrite()
131+ if got := string(readAllTimeout(t, c)); got != "x" {
132+ t.Fatalf("got %q", got)
133+ }
134+ }
135+ if n := h.accepts.Load(); n != 3 {
136+ t.Fatalf("%d host streams for 3 connections", n)
137+ }
138+}
139+
140+// The cap is on open connections: one over it is closed at once, never queued,
141+// and a slot freed by a closing connection is usable again.
142+func TestRelayConcurrencyCap(t *testing.T) {
143+ h := newHostSide(t, func(c net.Conn) {
144+ defer func() { _ = c.Close() }()
145+ _, _ = io.Copy(c, c)
146+ })
147+ r := startRelay(t, h.dial, 2)
148+ a, b := dialRelay(t, r), dialRelay(t, r)
149+ roundTrip := func(c net.Conn) {
150+ t.Helper()
151+ _ = c.SetDeadline(time.Now().Add(5 * time.Second))
152+ if _, err := c.Write([]byte("p")); err != nil {
153+ t.Fatal(err)
154+ }
155+ buf := make([]byte, 1)
156+ if _, err := io.ReadFull(c, buf); err != nil {
157+ t.Fatal(err)
158+ }
159+ }
160+ roundTrip(a)
161+ roundTrip(b)
162+ over := dialRelay(t, r)
163+ _ = over.SetReadDeadline(time.Now().Add(5 * time.Second))
164+ if _, err := over.Read(make([]byte, 1)); !errors.Is(err, io.EOF) && !errors.Is(err, net.ErrClosed) {
165+ // A reset is as good as an EOF: the connection was refused service.
166+ var ne net.Error
167+ if !errors.As(err, &ne) || ne.Timeout() {
168+ t.Fatalf("a connection over the cap must be closed, got %v", err)
169+ }
170+ }
171+ if n := h.accepts.Load(); n != 2 {
172+ t.Fatalf("%d host streams, want 2", n)
173+ }
174+ _ = a.Close()
175+ deadline := time.Now().Add(5 * time.Second)
176+ for {
177+ c := dialRelay(t, r)
178+ _ = c.SetDeadline(time.Now().Add(time.Second))
179+ _, _ = c.Write([]byte("p"))
180+ if _, err := io.ReadFull(c, make([]byte, 1)); err == nil {
181+ break
182+ }
183+ if time.Now().After(deadline) {
184+ t.Fatal("the slot of a closed connection is never freed")
185+ }
186+ time.Sleep(20 * time.Millisecond)
187+ }
188+}
189+
190+// A channel that does not carry the half-close (Firecracker's vsock proxy may
191+// not) must not hold a slot for ever once the client has gone: the idle half is
192+// cut after the grace, while a transfer that makes progress is not.
193+func TestRelayCutsAnIdleHalfAfterTheGrace(t *testing.T) {
194+ release := make(chan struct{})
195+ t.Cleanup(func() { close(release) })
196+ h := newHostSide(t, func(c net.Conn) {
197+ defer func() { _ = c.Close() }()
198+ <-release // never answers and never closes its side
199+ })
200+ const grace = 150 * time.Millisecond
201+ r, err := newRelay(context.Background(), "127.0.0.1:0", h.dial, 1, grace)
202+ if err != nil {
203+ t.Skipf("loopback sockets are not allowed here: %v", err)
204+ }
205+ t.Cleanup(func() { _ = r.Close() })
206+ c := dialRelay(t, r)
207+ _, _ = c.Write([]byte("x"))
208+ _ = c.CloseWrite() // the client is done: the host half stays idle
209+ start := time.Now()
210+ // The host never ends its half, so an EOF here is the relay cutting it.
211+ if b := readAllTimeout(t, c); len(b) != 0 {
212+ t.Fatalf("read %q", b)
213+ }
214+ if took := time.Since(start); took < grace {
215+ t.Fatalf("cut after %v, before the grace of %v", took, grace)
216+ }
217+ // The slot is back (the cap is 1): a new connection reaches the host.
218+ waitFor(t, func() bool {
219+ d := dialRelay(t, r)
220+ _, _ = d.Write([]byte("y"))
221+ return h.accepts.Load() >= 2
222+ })
223+}
224+
225+// A slow but live transfer after the first direction ended is not cut: the
226+// grace counts idle time, not total time.
227+func TestRelayGraceIsIdleTimeNotTotalTime(t *testing.T) {
228+ h := newHostSide(t, func(c net.Conn) {
229+ defer func() { _ = c.Close() }()
230+ _, _ = io.Copy(io.Discard, c) // the request, until the client's half-close
231+ for i := 0; i < 8; i++ { // a response slower than the grace in total
232+ _, _ = c.Write([]byte("0123456789"))
233+ time.Sleep(60 * time.Millisecond)
234+ }
235+ })
236+ r, err := newRelay(context.Background(), "127.0.0.1:0", h.dial, 2, 200*time.Millisecond)
237+ if err != nil {
238+ t.Skipf("loopback sockets are not allowed here: %v", err)
239+ }
240+ t.Cleanup(func() { _ = r.Close() })
241+ c := dialRelay(t, r)
242+ _, _ = c.Write([]byte("req"))
243+ _ = c.CloseWrite()
244+ if got := len(readAllTimeout(t, c)); got != 80 {
245+ t.Fatalf("read %d bytes, want 80", got)
246+ }
247+}
248+
249+// A host that cannot be reached closes the client's connection.
250+func TestRelayDialFailureClosesClient(t *testing.T) {
251+ var calls atomic.Int32
252+ r := startRelay(t, func(context.Context) (io.ReadWriteCloser, error) {
253+ calls.Add(1)
254+ return nil, errors.New("no host")
255+ }, 2)
256+ c := dialRelay(t, r)
257+ _ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
258+ if _, err := c.Read(make([]byte, 1)); err == nil || calls.Load() != 1 {
259+ t.Fatalf("read err %v after %d dials", err, calls.Load())
260+ }
261+ // The failed connection holds no slot.
262+ for i := 0; i < 4; i++ {
263+ d := dialRelay(t, r)
264+ _ = d.SetReadDeadline(time.Now().Add(5 * time.Second))
265+ if _, err := d.Read(make([]byte, 1)); err == nil {
266+ t.Fatal("expected a close")
267+ }
268+ }
269+ if calls.Load() != 5 {
270+ t.Fatalf("%d dials, a leaked slot would have refused some", calls.Load())
271+ }
272+}
273+
274+// Close stops the listener, cuts the open connections on both sides and returns
275+// once every goroutine has ended; it can be called twice.
276+func TestRelayCloseCutsEverything(t *testing.T) {
277+ hostGone := make(chan struct{}, 1)
278+ h := newHostSide(t, func(c net.Conn) {
279+ defer func() { _ = c.Close() }()
280+ _, _ = io.Copy(io.Discard, c) // ends when the relay closes its stream
281+ hostGone <- struct{}{}
282+ })
283+ r, err := StartRelay(context.Background(), "127.0.0.1:0", h.dial, 4)
284+ if err != nil {
285+ t.Skipf("loopback sockets are not allowed here: %v", err)
286+ }
287+ c := dialRelay(t, r)
288+ _, _ = c.Write([]byte("x"))
289+ waitFor(t, func() bool { return h.accepts.Load() == 1 })
290+
291+ done := make(chan error, 1)
292+ go func() { done <- r.Close() }()
293+ select {
294+ case err := <-done:
295+ if err != nil {
296+ t.Fatal(err)
297+ }
298+ case <-time.After(5 * time.Second):
299+ t.Fatal("Close does not return")
300+ }
301+ select {
302+ case <-hostGone:
303+ case <-time.After(5 * time.Second):
304+ t.Fatal("the host stream was not closed")
305+ }
306+ _ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
307+ if _, err := c.Read(make([]byte, 1)); err == nil {
308+ t.Fatal("the client connection is still open")
309+ }
310+ if c2, err := net.DialTimeout("tcp", r.Addr().String(), time.Second); err == nil {
311+ _ = c2.Close()
312+ t.Fatal("the relay still listens")
313+ }
314+ if err := r.Close(); err != nil {
315+ t.Fatalf("second Close: %v", err)
316+ }
317+}
318+
319+// The relay ends with the context of the job.
320+func TestRelayEndsWithItsContext(t *testing.T) {
321+ h := newHostSide(t, func(c net.Conn) { _ = c.Close() })
322+ ctx, cancel := context.WithCancel(context.Background())
323+ r, err := StartRelay(ctx, "127.0.0.1:0", h.dial, 2)
324+ if err != nil {
325+ t.Skipf("loopback sockets are not allowed here: %v", err)
326+ }
327+ addr := r.Addr().String()
328+ cancel()
329+ waitFor(t, func() bool {
330+ c, err := net.DialTimeout("tcp", addr, 200*time.Millisecond)
331+ if err == nil {
332+ _ = c.Close()
333+ }
334+ return err != nil
335+ })
336+}
337+
338+// A port that is taken is an error, not a silent no-cache.
339+func TestRelayListenError(t *testing.T) {
340+ busy, err := net.Listen("tcp", "127.0.0.1:0")
341+ if err != nil {
342+ t.Skipf("loopback sockets are not allowed here: %v", err)
343+ }
344+ defer func() { _ = busy.Close() }()
345+ if _, err := StartRelay(context.Background(), busy.Addr().String(), nil, 1); err == nil {
346+ t.Fatal("expected an error")
347+ }
348+}
349+
350+// The cache channel is a sibling of the executor's: its own vsock port in a VM,
351+// a second socket in the same private directory in a container.
352+func TestCacheTarget(t *testing.T) {
353+ for in, want := range map[string]string{
354+ "vsock:1024": "vsock:9089",
355+ "unix:/run/ermite/exec.sock": "unix:/run/ermite/cache.sock",
356+ } {
357+ tg, err := ParseTarget(in)
358+ if err != nil {
359+ t.Fatal(err)
360+ }
361+ if got := tg.CacheTarget().String(); got != want {
362+ t.Errorf("%s: cache target %s, want %s", in, got, want)
363+ }
364+ }
365+}
366+
367+// DialStream opens a new connection at every call and does not retry.
368+func TestDialStreamUnix(t *testing.T) {
369+ h := newHostSide(t, func(c net.Conn) { _ = c.Close() })
370+ tg := Target{Network: "unix", Addr: h.ln.Addr().String()}
371+ for i := 0; i < 2; i++ {
372+ c, err := tg.DialStream(context.Background())
373+ if err != nil {
374+ t.Fatal(err)
375+ }
376+ _ = c.Close()
377+ }
378+ waitFor(t, func() bool { return h.accepts.Load() == 2 })
379+ start := time.Now()
380+ if _, err := (Target{Network: "unix", Addr: filepath.Join(t.TempDir(), "none")}).DialStream(context.Background()); err == nil || time.Since(start) > time.Second {
381+ t.Fatalf("want an immediate error, got %v after %s", err, time.Since(start))
382+ }
383+}
384+
385+func waitFor(t *testing.T, ok func() bool) {
386+ t.Helper()
387+ for end := time.Now().Add(5 * time.Second); time.Now().Before(end); time.Sleep(10 * time.Millisecond) {
388+ if ok() {
389+ return
390+ }
391+ }
392+ t.Fatal("condition not reached in 5s")
393+}
@@ -0,0 +1,393 @@
1+// SPDX-License-Identifier: MIT
2+
3+package guest
4+
5+import (
6+ "context"
7+ "errors"
8+ "io"
9+ "net"
10+ "path/filepath"
11+ "sync/atomic"
12+ "testing"
13+ "time"
14+)
15+
16+// hostSide is a Unix socket server standing for the host end of the channel.
17+type hostSide struct {
18+ ln net.Listener
19+ accepts atomic.Int32
20+}
21+
22+func newHostSide(t *testing.T, serve func(net.Conn)) *hostSide {
23+ t.Helper()
24+ ln, err := net.Listen("unix", filepath.Join(t.TempDir(), "h"))
25+ if err != nil {
26+ t.Fatal(err)
27+ }
28+ h := &hostSide{ln: ln}
29+ t.Cleanup(func() { _ = ln.Close() })
30+ go func() {
31+ for {
32+ c, err := ln.Accept()
33+ if err != nil {
34+ return
35+ }
36+ h.accepts.Add(1)
37+ go serve(c)
38+ }
39+ }()
40+ return h
41+}
42+
43+func (h *hostSide) dial(ctx context.Context) (io.ReadWriteCloser, error) {
44+ var d net.Dialer
45+ return d.DialContext(ctx, "unix", h.ln.Addr().String())
46+}
47+
48+func startRelay(t *testing.T, dial DialFunc, max int) *Relay {
49+ t.Helper()
50+ r, err := StartRelay(context.Background(), "127.0.0.1:0", dial, max)
51+ if err != nil {
52+ t.Skipf("loopback sockets are not allowed here: %v", err)
53+ }
54+ t.Cleanup(func() { _ = r.Close() })
55+ return r
56+}
57+
58+func dialRelay(t *testing.T, r *Relay) *net.TCPConn {
59+ t.Helper()
60+ c, err := net.Dial("tcp", r.Addr().String())
61+ if err != nil {
62+ t.Fatal(err)
63+ }
64+ t.Cleanup(func() { _ = c.Close() })
65+ return c.(*net.TCPConn)
66+}
67+
68+func readAllTimeout(t *testing.T, c net.Conn) []byte {
69+ t.Helper()
70+ _ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
71+ b, err := io.ReadAll(c)
72+ if err != nil {
73+ t.Fatalf("read: %v", err)
74+ }
75+ return b
76+}
77+
78+// Bytes go both ways, and the end of the client's stream reaches the host as an
79+// EOF while the host can still answer (half-close).
80+func TestRelayCopiesBothWaysAndHalfCloses(t *testing.T) {
81+ h := newHostSide(t, func(c net.Conn) {
82+ defer func() { _ = c.Close() }()
83+ req, _ := io.ReadAll(c) // returns only at the client's half-close
84+ _, _ = c.Write(append([]byte("echo:"), req...))
85+ })
86+ r := startRelay(t, h.dial, 4)
87+ c := dialRelay(t, r)
88+ if _, err := c.Write([]byte("hello")); err != nil {
89+ t.Fatal(err)
90+ }
91+ if err := c.CloseWrite(); err != nil {
92+ t.Fatal(err)
93+ }
94+ if got := string(readAllTimeout(t, c)); got != "echo:hello" {
95+ t.Fatalf("got %q", got)
96+ }
97+}
98+
99+// The host closing its side reaches the client as an EOF, and bulk data in both
100+// directions is not lost.
101+func TestRelayBulkAndHostClose(t *testing.T) {
102+ const n = 3 << 20
103+ h := newHostSide(t, func(c net.Conn) {
104+ defer func() { _ = c.Close() }()
105+ got, _ := io.CopyN(io.Discard, c, n)
106+ _, _ = io.CopyN(c, zeroReader{}, got)
107+ })
108+ r := startRelay(t, h.dial, 4)
109+ c := dialRelay(t, r)
110+ go func() { _, _ = io.CopyN(c, zeroReader{}, n) }()
111+ if got := len(readAllTimeout(t, c)); got != n {
112+ t.Fatalf("read %d bytes, want %d", got, n)
113+ }
114+}
115+
116+type zeroReader struct{}
117+
118+func (zeroReader) Read(p []byte) (int, error) { clear(p); return len(p), nil }
119+
120+// Every accepted connection is a new stream to the host.
121+func TestRelayOneStreamPerConnection(t *testing.T) {
122+ h := newHostSide(t, func(c net.Conn) {
123+ defer func() { _ = c.Close() }()
124+ _, _ = io.Copy(c, c)
125+ })
126+ r := startRelay(t, h.dial, 8)
127+ for i := 0; i < 3; i++ {
128+ c := dialRelay(t, r)
129+ _, _ = c.Write([]byte("x"))
130+ _ = c.CloseWrite()
131+ if got := string(readAllTimeout(t, c)); got != "x" {
132+ t.Fatalf("got %q", got)
133+ }
134+ }
135+ if n := h.accepts.Load(); n != 3 {
136+ t.Fatalf("%d host streams for 3 connections", n)
137+ }
138+}
139+
140+// The cap is on open connections: one over it is closed at once, never queued,
141+// and a slot freed by a closing connection is usable again.
142+func TestRelayConcurrencyCap(t *testing.T) {
143+ h := newHostSide(t, func(c net.Conn) {
144+ defer func() { _ = c.Close() }()
145+ _, _ = io.Copy(c, c)
146+ })
147+ r := startRelay(t, h.dial, 2)
148+ a, b := dialRelay(t, r), dialRelay(t, r)
149+ roundTrip := func(c net.Conn) {
150+ t.Helper()
151+ _ = c.SetDeadline(time.Now().Add(5 * time.Second))
152+ if _, err := c.Write([]byte("p")); err != nil {
153+ t.Fatal(err)
154+ }
155+ buf := make([]byte, 1)
156+ if _, err := io.ReadFull(c, buf); err != nil {
157+ t.Fatal(err)
158+ }
159+ }
160+ roundTrip(a)
161+ roundTrip(b)
162+ over := dialRelay(t, r)
163+ _ = over.SetReadDeadline(time.Now().Add(5 * time.Second))
164+ if _, err := over.Read(make([]byte, 1)); !errors.Is(err, io.EOF) && !errors.Is(err, net.ErrClosed) {
165+ // A reset is as good as an EOF: the connection was refused service.
166+ var ne net.Error
167+ if !errors.As(err, &ne) || ne.Timeout() {
168+ t.Fatalf("a connection over the cap must be closed, got %v", err)
169+ }
170+ }
171+ if n := h.accepts.Load(); n != 2 {
172+ t.Fatalf("%d host streams, want 2", n)
173+ }
174+ _ = a.Close()
175+ deadline := time.Now().Add(5 * time.Second)
176+ for {
177+ c := dialRelay(t, r)
178+ _ = c.SetDeadline(time.Now().Add(time.Second))
179+ _, _ = c.Write([]byte("p"))
180+ if _, err := io.ReadFull(c, make([]byte, 1)); err == nil {
181+ break
182+ }
183+ if time.Now().After(deadline) {
184+ t.Fatal("the slot of a closed connection is never freed")
185+ }
186+ time.Sleep(20 * time.Millisecond)
187+ }
188+}
189+
190+// A channel that does not carry the half-close (Firecracker's vsock proxy may
191+// not) must not hold a slot for ever once the client has gone: the idle half is
192+// cut after the grace, while a transfer that makes progress is not.
193+func TestRelayCutsAnIdleHalfAfterTheGrace(t *testing.T) {
194+ release := make(chan struct{})
195+ t.Cleanup(func() { close(release) })
196+ h := newHostSide(t, func(c net.Conn) {
197+ defer func() { _ = c.Close() }()
198+ <-release // never answers and never closes its side
199+ })
200+ const grace = 150 * time.Millisecond
201+ r, err := newRelay(context.Background(), "127.0.0.1:0", h.dial, 1, grace)
202+ if err != nil {
203+ t.Skipf("loopback sockets are not allowed here: %v", err)
204+ }
205+ t.Cleanup(func() { _ = r.Close() })
206+ c := dialRelay(t, r)
207+ _, _ = c.Write([]byte("x"))
208+ _ = c.CloseWrite() // the client is done: the host half stays idle
209+ start := time.Now()
210+ // The host never ends its half, so an EOF here is the relay cutting it.
211+ if b := readAllTimeout(t, c); len(b) != 0 {
212+ t.Fatalf("read %q", b)
213+ }
214+ if took := time.Since(start); took < grace {
215+ t.Fatalf("cut after %v, before the grace of %v", took, grace)
216+ }
217+ // The slot is back (the cap is 1): a new connection reaches the host.
218+ waitFor(t, func() bool {
219+ d := dialRelay(t, r)
220+ _, _ = d.Write([]byte("y"))
221+ return h.accepts.Load() >= 2
222+ })
223+}
224+
225+// A slow but live transfer after the first direction ended is not cut: the
226+// grace counts idle time, not total time.
227+func TestRelayGraceIsIdleTimeNotTotalTime(t *testing.T) {
228+ h := newHostSide(t, func(c net.Conn) {
229+ defer func() { _ = c.Close() }()
230+ _, _ = io.Copy(io.Discard, c) // the request, until the client's half-close
231+ for i := 0; i < 8; i++ { // a response slower than the grace in total
232+ _, _ = c.Write([]byte("0123456789"))
233+ time.Sleep(60 * time.Millisecond)
234+ }
235+ })
236+ r, err := newRelay(context.Background(), "127.0.0.1:0", h.dial, 2, 200*time.Millisecond)
237+ if err != nil {
238+ t.Skipf("loopback sockets are not allowed here: %v", err)
239+ }
240+ t.Cleanup(func() { _ = r.Close() })
241+ c := dialRelay(t, r)
242+ _, _ = c.Write([]byte("req"))
243+ _ = c.CloseWrite()
244+ if got := len(readAllTimeout(t, c)); got != 80 {
245+ t.Fatalf("read %d bytes, want 80", got)
246+ }
247+}
248+
249+// A host that cannot be reached closes the client's connection.
250+func TestRelayDialFailureClosesClient(t *testing.T) {
251+ var calls atomic.Int32
252+ r := startRelay(t, func(context.Context) (io.ReadWriteCloser, error) {
253+ calls.Add(1)
254+ return nil, errors.New("no host")
255+ }, 2)
256+ c := dialRelay(t, r)
257+ _ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
258+ if _, err := c.Read(make([]byte, 1)); err == nil || calls.Load() != 1 {
259+ t.Fatalf("read err %v after %d dials", err, calls.Load())
260+ }
261+ // The failed connection holds no slot.
262+ for i := 0; i < 4; i++ {
263+ d := dialRelay(t, r)
264+ _ = d.SetReadDeadline(time.Now().Add(5 * time.Second))
265+ if _, err := d.Read(make([]byte, 1)); err == nil {
266+ t.Fatal("expected a close")
267+ }
268+ }
269+ if calls.Load() != 5 {
270+ t.Fatalf("%d dials, a leaked slot would have refused some", calls.Load())
271+ }
272+}
273+
274+// Close stops the listener, cuts the open connections on both sides and returns
275+// once every goroutine has ended; it can be called twice.
276+func TestRelayCloseCutsEverything(t *testing.T) {
277+ hostGone := make(chan struct{}, 1)
278+ h := newHostSide(t, func(c net.Conn) {
279+ defer func() { _ = c.Close() }()
280+ _, _ = io.Copy(io.Discard, c) // ends when the relay closes its stream
281+ hostGone <- struct{}{}
282+ })
283+ r, err := StartRelay(context.Background(), "127.0.0.1:0", h.dial, 4)
284+ if err != nil {
285+ t.Skipf("loopback sockets are not allowed here: %v", err)
286+ }
287+ c := dialRelay(t, r)
288+ _, _ = c.Write([]byte("x"))
289+ waitFor(t, func() bool { return h.accepts.Load() == 1 })
290+
291+ done := make(chan error, 1)
292+ go func() { done <- r.Close() }()
293+ select {
294+ case err := <-done:
295+ if err != nil {
296+ t.Fatal(err)
297+ }
298+ case <-time.After(5 * time.Second):
299+ t.Fatal("Close does not return")
300+ }
301+ select {
302+ case <-hostGone:
303+ case <-time.After(5 * time.Second):
304+ t.Fatal("the host stream was not closed")
305+ }
306+ _ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
307+ if _, err := c.Read(make([]byte, 1)); err == nil {
308+ t.Fatal("the client connection is still open")
309+ }
310+ if c2, err := net.DialTimeout("tcp", r.Addr().String(), time.Second); err == nil {
311+ _ = c2.Close()
312+ t.Fatal("the relay still listens")
313+ }
314+ if err := r.Close(); err != nil {
315+ t.Fatalf("second Close: %v", err)
316+ }
317+}
318+
319+// The relay ends with the context of the job.
320+func TestRelayEndsWithItsContext(t *testing.T) {
321+ h := newHostSide(t, func(c net.Conn) { _ = c.Close() })
322+ ctx, cancel := context.WithCancel(context.Background())
323+ r, err := StartRelay(ctx, "127.0.0.1:0", h.dial, 2)
324+ if err != nil {
325+ t.Skipf("loopback sockets are not allowed here: %v", err)
326+ }
327+ addr := r.Addr().String()
328+ cancel()
329+ waitFor(t, func() bool {
330+ c, err := net.DialTimeout("tcp", addr, 200*time.Millisecond)
331+ if err == nil {
332+ _ = c.Close()
333+ }
334+ return err != nil
335+ })
336+}
337+
338+// A port that is taken is an error, not a silent no-cache.
339+func TestRelayListenError(t *testing.T) {
340+ busy, err := net.Listen("tcp", "127.0.0.1:0")
341+ if err != nil {
342+ t.Skipf("loopback sockets are not allowed here: %v", err)
343+ }
344+ defer func() { _ = busy.Close() }()
345+ if _, err := StartRelay(context.Background(), busy.Addr().String(), nil, 1); err == nil {
346+ t.Fatal("expected an error")
347+ }
348+}
349+
350+// The cache channel is a sibling of the executor's: its own vsock port in a VM,
351+// a second socket in the same private directory in a container.
352+func TestCacheTarget(t *testing.T) {
353+ for in, want := range map[string]string{
354+ "vsock:1024": "vsock:9089",
355+ "unix:/run/ermite/exec.sock": "unix:/run/ermite/cache.sock",
356+ } {
357+ tg, err := ParseTarget(in)
358+ if err != nil {
359+ t.Fatal(err)
360+ }
361+ if got := tg.CacheTarget().String(); got != want {
362+ t.Errorf("%s: cache target %s, want %s", in, got, want)
363+ }
364+ }
365+}
366+
367+// DialStream opens a new connection at every call and does not retry.
368+func TestDialStreamUnix(t *testing.T) {
369+ h := newHostSide(t, func(c net.Conn) { _ = c.Close() })
370+ tg := Target{Network: "unix", Addr: h.ln.Addr().String()}
371+ for i := 0; i < 2; i++ {
372+ c, err := tg.DialStream(context.Background())
373+ if err != nil {
374+ t.Fatal(err)
375+ }
376+ _ = c.Close()
377+ }
378+ waitFor(t, func() bool { return h.accepts.Load() == 2 })
379+ start := time.Now()
380+ if _, err := (Target{Network: "unix", Addr: filepath.Join(t.TempDir(), "none")}).DialStream(context.Background()); err == nil || time.Since(start) > time.Second {
381+ t.Fatalf("want an immediate error, got %v after %s", err, time.Since(start))
382+ }
383+}
384+
385+func waitFor(t *testing.T, ok func() bool) {
386+ t.Helper()
387+ for end := time.Now().Add(5 * time.Second); time.Now().Before(end); time.Sleep(10 * time.Millisecond) {
388+ if ok() {
389+ return
390+ }
391+ }
392+ t.Fatal("condition not reached in 5s")
393+}
M internal/host/cache.go +4 -4
@@ -106,10 +106,10 @@ func githubRepository(contexts string) string {
106106 return c.Github.Repository
107107 }
108108
109-// cacheFor returns the cache port and the token of a job, or 0 and "" when the
110-// runner serves no cache or the job has no scope.
109+// cacheFor returns the loopback port the cache has in the guest and the token
110+// of a job, or 0 and "" when the runner serves no cache or the job has no scope.
111111 func (j *jobRun) cacheFor(rt *Runner) (port uint32, token string) {
112- if rt.Cache == nil || rt.CachePort == 0 {
112+ if rt.Cache == nil {
113113 return 0, ""
114114 }
115115 sc, ok, err := cacheScope(j.job, j.contexts)
@@ -123,5 +123,5 @@ func (j *jobRun) cacheFor(rt *Runner) (port uint32, token string) {
123123 return 0, ""
124124 }
125125 ttl := time.Duration(j.timeoutMinutes())*time.Minute + cacheTokenMargin
126- return uint32(rt.CachePort), rt.Cache.Token(j.job.JobId, sc, ttl)
126+ return cachesvc.DefaultPort, rt.Cache.Token(j.job.JobId, sc, ttl)
127127 }
@@ -106,10 +106,10 @@ func githubRepository(contexts string) string {
106 return c.Github.Repository106 return c.Github.Repository
107 }107 }
108 108
109-// cacheFor returns the cache port and the token of a job, or 0 and "" when the109+// cacheFor returns the loopback port the cache has in the guest and the token
110-// runner serves no cache or the job has no scope.110+// of a job, or 0 and "" when the runner serves no cache or the job has no scope.
111 func (j *jobRun) cacheFor(rt *Runner) (port uint32, token string) {111 func (j *jobRun) cacheFor(rt *Runner) (port uint32, token string) {
112- if rt.Cache == nil || rt.CachePort == 0 {112+ if rt.Cache == nil {
113 return 0, ""113 return 0, ""
114 }114 }
115 sc, ok, err := cacheScope(j.job, j.contexts)115 sc, ok, err := cacheScope(j.job, j.contexts)
@@ -123,5 +123,5 @@ func (j *jobRun) cacheFor(rt *Runner) (port uint32, token string) {
123 return 0, ""123 return 0, ""
124 }124 }
125 ttl := time.Duration(j.timeoutMinutes())*time.Minute + cacheTokenMargin125 ttl := time.Duration(j.timeoutMinutes())*time.Minute + cacheTokenMargin
126- return uint32(rt.CachePort), rt.Cache.Token(j.job.JobId, sc, ttl)126+ return cachesvc.DefaultPort, rt.Cache.Token(j.job.JobId, sc, ttl)
127 }127 }
M internal/host/cache_test.go +4 -3
@@ -134,7 +134,7 @@ func TestCacheForMismatchDisablesCache(t *testing.T) {
134134 job: &ermite.Job{JobId: "j1", Ref: "refs/heads/main", Cache: &ermite.JobCache{Repository: "o/other", WriteRef: "refs/heads/main"}},
135135 contexts: scopeCtx, log: zerowrap.New(zerowrap.Config{Output: &logs, Format: "json", Level: "debug"}),
136136 }
137- if port, tok := j.cacheFor(&Runner{Cache: svc, CachePort: 8089}); port != 0 || tok != "" {
137+ if port, tok := j.cacheFor(&Runner{Cache: svc}); port != 0 || tok != "" {
138138 t.Fatalf("cache for a mismatched scope: %d %q", port, tok)
139139 }
140140 if !strings.Contains(logs.String(), "cache disabled for the job") || !strings.Contains(logs.String(), "o/other") {
@@ -159,10 +159,11 @@ func TestCacheForToken(t *testing.T) {
159159 t.Cleanup(func() { _ = svc.CloseAll() })
160160 srv := httptest.NewServer(svc.Handler())
161161 defer srv.Close()
162- rt := &Runner{Cache: svc, CachePort: 8089}
162+ rt := &Runner{Cache: svc}
163163 j := &jobRun{job: &ermite.Job{JobId: "j1", Ref: "refs/heads/main", TimeoutMinutes: 5}, contexts: `{"github":{"repository":"o/r"}}`}
164164 port, tok := j.cacheFor(rt)
165- if port != 8089 || tok == "" {
165+ // The port is the cache's loopback port in the guest, not a host port.
166+ if port != 9089 || tok == "" {
166167 t.Fatalf("%d %q", port, tok)
167168 }
168169 do := func(token string) int {
@@ -134,7 +134,7 @@ func TestCacheForMismatchDisablesCache(t *testing.T) {
134 job: &ermite.Job{JobId: "j1", Ref: "refs/heads/main", Cache: &ermite.JobCache{Repository: "o/other", WriteRef: "refs/heads/main"}},134 job: &ermite.Job{JobId: "j1", Ref: "refs/heads/main", Cache: &ermite.JobCache{Repository: "o/other", WriteRef: "refs/heads/main"}},
135 contexts: scopeCtx, log: zerowrap.New(zerowrap.Config{Output: &logs, Format: "json", Level: "debug"}),135 contexts: scopeCtx, log: zerowrap.New(zerowrap.Config{Output: &logs, Format: "json", Level: "debug"}),
136 }136 }
137- if port, tok := j.cacheFor(&Runner{Cache: svc, CachePort: 8089}); port != 0 || tok != "" {137+ if port, tok := j.cacheFor(&Runner{Cache: svc}); port != 0 || tok != "" {
138 t.Fatalf("cache for a mismatched scope: %d %q", port, tok)138 t.Fatalf("cache for a mismatched scope: %d %q", port, tok)
139 }139 }
140 if !strings.Contains(logs.String(), "cache disabled for the job") || !strings.Contains(logs.String(), "o/other") {140 if !strings.Contains(logs.String(), "cache disabled for the job") || !strings.Contains(logs.String(), "o/other") {
@@ -159,10 +159,11 @@ func TestCacheForToken(t *testing.T) {
159 t.Cleanup(func() { _ = svc.CloseAll() })159 t.Cleanup(func() { _ = svc.CloseAll() })
160 srv := httptest.NewServer(svc.Handler())160 srv := httptest.NewServer(svc.Handler())
161 defer srv.Close()161 defer srv.Close()
162- rt := &Runner{Cache: svc, CachePort: 8089}162+ rt := &Runner{Cache: svc}
163 j := &jobRun{job: &ermite.Job{JobId: "j1", Ref: "refs/heads/main", TimeoutMinutes: 5}, contexts: `{"github":{"repository":"o/r"}}`}163 j := &jobRun{job: &ermite.Job{JobId: "j1", Ref: "refs/heads/main", TimeoutMinutes: 5}, contexts: `{"github":{"repository":"o/r"}}`}
164 port, tok := j.cacheFor(rt)164 port, tok := j.cacheFor(rt)
165- if port != 8089 || tok == "" {165+ // The port is the cache's loopback port in the guest, not a host port.
166+ if port != 9089 || tok == "" {
166 t.Fatalf("%d %q", port, tok)167 t.Fatalf("%d %q", port, tok)
167 }168 }
168 do := func(token string) int {169 do := func(token string) int {
M internal/host/runner.go +4 -5
@@ -38,11 +38,10 @@ type Runner struct {
3838 // "ci.summaries". Nil means neither is sent.
3939 Features ServerFeatures
4040 // Cache, when not nil, is the cache service of the runner; the sandbox
41- // drivers serve it on CachePort of each sandbox's gateway (see
42- // sandbox.Config.Cache), and each job gets a token scoped to its repository
43- // and refs.
44- Cache *cachesvc.Service
45- CachePort uint16
41+ // drivers serve it on a channel of each sandbox (see sandbox.Config.Cache),
42+ // which the guest reaches on its loopback (cachesvc.DefaultPort), and each
43+ // job gets a token scoped to its repository and refs.
44+ Cache *cachesvc.Service
4645
4746 // now returns the current time; tests replace it.
4847 now func() time.Time
@@ -38,11 +38,10 @@ type Runner struct {
38 // "ci.summaries". Nil means neither is sent.38 // "ci.summaries". Nil means neither is sent.
39 Features ServerFeatures39 Features ServerFeatures
40 // Cache, when not nil, is the cache service of the runner; the sandbox40 // Cache, when not nil, is the cache service of the runner; the sandbox
41- // drivers serve it on CachePort of each sandbox's gateway (see41+ // drivers serve it on a channel of each sandbox (see sandbox.Config.Cache),
42- // sandbox.Config.Cache), and each job gets a token scoped to its repository42+ // which the guest reaches on its loopback (cachesvc.DefaultPort), and each
43- // and refs.43+ // job gets a token scoped to its repository and refs.
44- Cache *cachesvc.Service44+ Cache *cachesvc.Service
45- CachePort uint16
46 45
47 // now returns the current time; tests replace it.46 // now returns the current time; tests replace it.
48 now func() time.Time47 now func() time.Time
M internal/netpolicy/netpolicy.go +2 -16
@@ -16,9 +16,8 @@
1616 // only): TCP from the guest's source address to its own gateway address on
1717 // that one port. The host's own listener on that port receives the traffic;
1818 // nothing is proxied, and every other host port stays closed;
19-// - optionally (Guest.CachePort, when the runner serves the GitHub Actions
20-// cache): the same, for the cache service port; that port of a gateway is
21-// also closed to every other interface;
19+// the GitHub Actions cache has no rule here: it is not on the network (the
20+// guest reaches it on its own loopback, through the host channel);
2221 // - IPv6 from the guest is dropped (the guest also disables IPv6);
2322 // - forwarded traffic must carry the guest's own source address;
2423 // - forwarded traffic to RFC 1918, CGNAT 100.64.0.0/10 (VPN overlays),
@@ -85,11 +84,6 @@ type Guest struct {
8584 // guest's interface, source and gateway, so it neither reaches other
8685 // addresses of the host nor other guests.
8786 ForgePort uint16
88- // CachePort, when not 0, is the port of the runner's cache service on
89- // Gateway (package cachesvc). Same scope as ForgePort, and in addition the
90- // port is dropped from every other interface, as port 53 is: the service
91- // answers the guest of that gateway only.
92- CachePort uint16
9387 }
9488
9589 // Validate checks the fields used in generated rules.
@@ -168,11 +162,6 @@ func Ruleset(guests []Guest) string {
168162 fmt.Fprintf(&b, "\t\tip daddr %s %s dport 53 iifname != \"%s\" drop\n", g.Gateway, proto, g.Iface)
169163 }
170164 }
171- for _, g := range gs {
172- if g.CachePort != 0 {
173- fmt.Fprintf(&b, "\t\tip daddr %s tcp dport %d iifname != \"%s\" drop\n", g.Gateway, g.CachePort, g.Iface)
174- }
175- }
176165 for _, g := range gs {
177166 for _, proto := range []string{"udp", "tcp"} {
178167 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s %s dport 53 accept\n", g.Iface, g.Addr, g.Gateway, proto)
@@ -182,9 +171,6 @@ func Ruleset(guests []Guest) string {
182171 // which this table does not touch.
183172 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s tcp dport %d accept\n", g.Iface, g.Addr, g.Gateway, g.ForgePort)
184173 }
185- if g.CachePort != 0 {
186- fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s tcp dport %d accept\n", g.Iface, g.Addr, g.Gateway, g.CachePort)
187- }
188174 fmt.Fprintf(&b, "\t\tiifname \"%s\" drop\n", g.Iface)
189175 }
190176 // An interface with our prefix that no live guest owns (a leftover) is
@@ -16,9 +16,8 @@
16 // only): TCP from the guest's source address to its own gateway address on16 // only): TCP from the guest's source address to its own gateway address on
17 // that one port. The host's own listener on that port receives the traffic;17 // that one port. The host's own listener on that port receives the traffic;
18 // nothing is proxied, and every other host port stays closed;18 // nothing is proxied, and every other host port stays closed;
19-// - optionally (Guest.CachePort, when the runner serves the GitHub Actions19+// the GitHub Actions cache has no rule here: it is not on the network (the
20-// cache): the same, for the cache service port; that port of a gateway is20+// guest reaches it on its own loopback, through the host channel);
21-// also closed to every other interface;
22 // - IPv6 from the guest is dropped (the guest also disables IPv6);21 // - IPv6 from the guest is dropped (the guest also disables IPv6);
23 // - forwarded traffic must carry the guest's own source address;22 // - forwarded traffic must carry the guest's own source address;
24 // - forwarded traffic to RFC 1918, CGNAT 100.64.0.0/10 (VPN overlays),23 // - forwarded traffic to RFC 1918, CGNAT 100.64.0.0/10 (VPN overlays),
@@ -85,11 +84,6 @@ type Guest struct {
85 // guest's interface, source and gateway, so it neither reaches other84 // guest's interface, source and gateway, so it neither reaches other
86 // addresses of the host nor other guests.85 // addresses of the host nor other guests.
87 ForgePort uint1686 ForgePort uint16
88- // CachePort, when not 0, is the port of the runner's cache service on
89- // Gateway (package cachesvc). Same scope as ForgePort, and in addition the
90- // port is dropped from every other interface, as port 53 is: the service
91- // answers the guest of that gateway only.
92- CachePort uint16
93 }87 }
94 88
95 // Validate checks the fields used in generated rules.89 // Validate checks the fields used in generated rules.
@@ -168,11 +162,6 @@ func Ruleset(guests []Guest) string {
168 fmt.Fprintf(&b, "\t\tip daddr %s %s dport 53 iifname != \"%s\" drop\n", g.Gateway, proto, g.Iface)162 fmt.Fprintf(&b, "\t\tip daddr %s %s dport 53 iifname != \"%s\" drop\n", g.Gateway, proto, g.Iface)
169 }163 }
170 }164 }
171- for _, g := range gs {
172- if g.CachePort != 0 {
173- fmt.Fprintf(&b, "\t\tip daddr %s tcp dport %d iifname != \"%s\" drop\n", g.Gateway, g.CachePort, g.Iface)
174- }
175- }
176 for _, g := range gs {165 for _, g := range gs {
177 for _, proto := range []string{"udp", "tcp"} {166 for _, proto := range []string{"udp", "tcp"} {
178 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s %s dport 53 accept\n", g.Iface, g.Addr, g.Gateway, proto)167 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s %s dport 53 accept\n", g.Iface, g.Addr, g.Gateway, proto)
@@ -182,9 +171,6 @@ func Ruleset(guests []Guest) string {
182 // which this table does not touch.171 // which this table does not touch.
183 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s tcp dport %d accept\n", g.Iface, g.Addr, g.Gateway, g.ForgePort)172 fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s tcp dport %d accept\n", g.Iface, g.Addr, g.Gateway, g.ForgePort)
184 }173 }
185- if g.CachePort != 0 {
186- fmt.Fprintf(&b, "\t\tiifname \"%s\" ip saddr %s ip daddr %s tcp dport %d accept\n", g.Iface, g.Addr, g.Gateway, g.CachePort)
187- }
188 fmt.Fprintf(&b, "\t\tiifname \"%s\" drop\n", g.Iface)174 fmt.Fprintf(&b, "\t\tiifname \"%s\" drop\n", g.Iface)
189 }175 }
190 // An interface with our prefix that no live guest owns (a leftover) is176 // An interface with our prefix that no live guest owns (a leftover) is
M internal/netpolicy/netpolicy_test.go +20 -40
@@ -293,47 +293,27 @@ func TestGuestForgeRuleScopeAndOrder(t *testing.T) {
293293 }
294294 }
295295
296-func cacheGuest(t *testing.T, iface string, slot int, forge, cache uint16) Guest {
297- t.Helper()
298- g := guest(t, iface, slot)
299- g.ForgePort, g.CachePort = forge, cache
300- return g
301-}
302-
303-func TestRulesetCacheGolden(t *testing.T) {
304- golden(t, "cache.nft", Ruleset([]Guest{cacheGuest(t, "ertap0", 0, 8088, 8089), cacheGuest(t, "erbr1", 1, 0, 8089)}))
305-}
306-
307-// The cache port is open to the guest's own source and gateway only, closed to
308-// every other interface, and a runner without the cache has no such rule
309-// (one.nft and guest_forge.nft are unchanged).
310-func TestCachePortScope(t *testing.T) {
311- rs := Ruleset([]Guest{cacheGuest(t, "ertap0", 0, 0, 8089), cacheGuest(t, "erbr1", 1, 0, 8089), guest(t, "ertap2", 2)})
312- in := rs[strings.Index(rs, "chain input"):strings.Index(rs, "chain forward")]
313- for _, want := range []string{
314- `iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8089 accept`,
315- `iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 8089 accept`,
316- `ip daddr 10.200.0.1 tcp dport 8089 iifname != "ertap0" drop`,
317- `ip daddr 10.200.0.5 tcp dport 8089 iifname != "erbr1" drop`,
296+// The cache is not on the network: the policy opens nothing for it, with or
297+// without the guest forge exception. A guest only ever gets DNS (and the forge
298+// port when asked for); the cache port of the guest's loopback (9089) and the
299+// old gateway port (8089) appear nowhere, and the only accepts of the input
300+// chain are DNS and the forge port.
301+func TestNoCachePortInThePolicy(t *testing.T) {
302+ for name, gs := range map[string][]Guest{
303+ "plain": {guest(t, "ertap0", 0), guest(t, "erbr1", 1)},
304+ "forge": {forgeGuest(t, "ertap0", 0, 8088), guest(t, "erbr1", 1)},
318305 } {
319- if strings.Count(in, want) != 1 {
320- t.Errorf("want exactly one %q in the input chain:\n%s", want, in)
306+ rs := Ruleset(gs)
307+ for _, port := range []string{"8089", "9089"} {
308+ if strings.Contains(rs, port) {
309+ t.Errorf("%s: the ruleset mentions port %s:\n%s", name, port, rs)
310+ }
311+ }
312+ in := rs[strings.Index(rs, "chain input"):strings.Index(rs, "chain forward")]
313+ for _, l := range strings.Split(in, "\n") {
314+ if strings.HasSuffix(l, "accept") && !strings.Contains(l, "dport 53 accept") && !strings.Contains(l, "tcp dport 8088 accept") {
315+ t.Errorf("%s: unexpected accept: %s", name, l)
316+ }
321317 }
322- }
323- if strings.Count(rs, "8089") != 4 {
324- t.Errorf("port 8089 appears %d times, want 4:\n%s", strings.Count(rs, "8089"), rs)
325- }
326- // Drops of other interfaces come before every accept, accepts before the
327- // guest's catch-all drop.
328- if strings.LastIndex(in, `iifname != "erbr1" drop`) > strings.Index(in, `dport 53 accept`) {
329- t.Error("the cross-interface drops must precede the accepts")
330- }
331- if strings.Index(in, `tcp dport 8089 accept`) > strings.Index(in, `iifname "ertap0" drop`) {
332- t.Error("the cache accept must precede the guest's catch-all drop")
333- }
334- plain := Ruleset([]Guest{guest(t, "ertap0", 0), guest(t, "erbr1", 1), guest(t, "ertap2", 2)})
335- tail := func(s string) string { return s[strings.Index(s, "chain forward"):] }
336- if tail(rs) != tail(plain) {
337- t.Error("the forward/postrouting chains changed with the cache port")
338318 }
339319 }
@@ -293,47 +293,27 @@ func TestGuestForgeRuleScopeAndOrder(t *testing.T) {
293 }293 }
294 }294 }
295 295
296-func cacheGuest(t *testing.T, iface string, slot int, forge, cache uint16) Guest {296+// The cache is not on the network: the policy opens nothing for it, with or
297- t.Helper()297+// without the guest forge exception. A guest only ever gets DNS (and the forge
298- g := guest(t, iface, slot)298+// port when asked for); the cache port of the guest's loopback (9089) and the
299- g.ForgePort, g.CachePort = forge, cache299+// old gateway port (8089) appear nowhere, and the only accepts of the input
300- return g300+// chain are DNS and the forge port.
301-}301+func TestNoCachePortInThePolicy(t *testing.T) {
302-302+ for name, gs := range map[string][]Guest{
303-func TestRulesetCacheGolden(t *testing.T) {303+ "plain": {guest(t, "ertap0", 0), guest(t, "erbr1", 1)},
304- golden(t, "cache.nft", Ruleset([]Guest{cacheGuest(t, "ertap0", 0, 8088, 8089), cacheGuest(t, "erbr1", 1, 0, 8089)}))304+ "forge": {forgeGuest(t, "ertap0", 0, 8088), guest(t, "erbr1", 1)},
305-}
306-
307-// The cache port is open to the guest's own source and gateway only, closed to
308-// every other interface, and a runner without the cache has no such rule
309-// (one.nft and guest_forge.nft are unchanged).
310-func TestCachePortScope(t *testing.T) {
311- rs := Ruleset([]Guest{cacheGuest(t, "ertap0", 0, 0, 8089), cacheGuest(t, "erbr1", 1, 0, 8089), guest(t, "ertap2", 2)})
312- in := rs[strings.Index(rs, "chain input"):strings.Index(rs, "chain forward")]
313- for _, want := range []string{
314- `iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8089 accept`,
315- `iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 8089 accept`,
316- `ip daddr 10.200.0.1 tcp dport 8089 iifname != "ertap0" drop`,
317- `ip daddr 10.200.0.5 tcp dport 8089 iifname != "erbr1" drop`,
318 } {305 } {
319- if strings.Count(in, want) != 1 {306+ rs := Ruleset(gs)
320- t.Errorf("want exactly one %q in the input chain:\n%s", want, in)307+ for _, port := range []string{"8089", "9089"} {
308+ if strings.Contains(rs, port) {
309+ t.Errorf("%s: the ruleset mentions port %s:\n%s", name, port, rs)
310+ }
311+ }
312+ in := rs[strings.Index(rs, "chain input"):strings.Index(rs, "chain forward")]
313+ for _, l := range strings.Split(in, "\n") {
314+ if strings.HasSuffix(l, "accept") && !strings.Contains(l, "dport 53 accept") && !strings.Contains(l, "tcp dport 8088 accept") {
315+ t.Errorf("%s: unexpected accept: %s", name, l)
316+ }
321 }317 }
322- }
323- if strings.Count(rs, "8089") != 4 {
324- t.Errorf("port 8089 appears %d times, want 4:\n%s", strings.Count(rs, "8089"), rs)
325- }
326- // Drops of other interfaces come before every accept, accepts before the
327- // guest's catch-all drop.
328- if strings.LastIndex(in, `iifname != "erbr1" drop`) > strings.Index(in, `dport 53 accept`) {
329- t.Error("the cross-interface drops must precede the accepts")
330- }
331- if strings.Index(in, `tcp dport 8089 accept`) > strings.Index(in, `iifname "ertap0" drop`) {
332- t.Error("the cache accept must precede the guest's catch-all drop")
333- }
334- plain := Ruleset([]Guest{guest(t, "ertap0", 0), guest(t, "erbr1", 1), guest(t, "ertap2", 2)})
335- tail := func(s string) string { return s[strings.Index(s, "chain forward"):] }
336- if tail(rs) != tail(plain) {
337- t.Error("the forward/postrouting chains changed with the cache port")
338 }318 }
339 }319 }
D internal/netpolicy/testdata/cache.nft +0 -52
@@ -1,52 +0,0 @@
1-table inet ermite_runner
2-delete table inet ermite_runner
3-table inet ermite_runner {
4- set blocked_v4 {
5- type ipv4_addr
6- flags interval
7- elements = { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4 }
8- }
9- chain input {
10- type filter hook input priority filter; policy accept;
11- iifname "ertap*" meta nfproto ipv6 drop
12- iifname "erbr*" meta nfproto ipv6 drop
13- ip daddr 10.200.0.5 udp dport 53 iifname != "erbr1" drop
14- ip daddr 10.200.0.5 tcp dport 53 iifname != "erbr1" drop
15- ip daddr 10.200.0.1 udp dport 53 iifname != "ertap0" drop
16- ip daddr 10.200.0.1 tcp dport 53 iifname != "ertap0" drop
17- ip daddr 10.200.0.5 tcp dport 8089 iifname != "erbr1" drop
18- ip daddr 10.200.0.1 tcp dport 8089 iifname != "ertap0" drop
19- iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 udp dport 53 accept
20- iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 53 accept
21- iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 8089 accept
22- iifname "erbr1" drop
23- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 udp dport 53 accept
24- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 53 accept
25- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8088 accept
26- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8089 accept
27- iifname "ertap0" drop
28- iifname "ertap*" drop
29- iifname "erbr*" drop
30- }
31- chain forward {
32- type filter hook forward priority filter; policy accept;
33- iifname "ertap*" meta nfproto ipv6 drop
34- iifname "erbr*" meta nfproto ipv6 drop
35- iifname "erbr1" ip saddr != 10.200.0.6 drop
36- iifname "erbr1" ip daddr @blocked_v4 drop
37- iifname "ertap0" ip saddr != 10.200.0.2 drop
38- iifname "ertap0" ip daddr @blocked_v4 drop
39- iifname != { "ertap*", "erbr*" } oifname { "ertap*", "erbr*" } ct state established,related accept
40- oifname "ertap*" drop
41- oifname "erbr*" drop
42- iifname "erbr1" accept
43- iifname "ertap0" accept
44- iifname "ertap*" drop
45- iifname "erbr*" drop
46- }
47- chain postrouting {
48- type nat hook postrouting priority srcnat; policy accept;
49- ip saddr 10.200.0.6 oifname != "erbr1" masquerade
50- ip saddr 10.200.0.2 oifname != "ertap0" masquerade
51- }
52-}
@@ -1,52 +0,0 @@
1-table inet ermite_runner
2-delete table inet ermite_runner
3-table inet ermite_runner {
4- set blocked_v4 {
5- type ipv4_addr
6- flags interval
7- elements = { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.168.0.0/16, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4 }
8- }
9- chain input {
10- type filter hook input priority filter; policy accept;
11- iifname "ertap*" meta nfproto ipv6 drop
12- iifname "erbr*" meta nfproto ipv6 drop
13- ip daddr 10.200.0.5 udp dport 53 iifname != "erbr1" drop
14- ip daddr 10.200.0.5 tcp dport 53 iifname != "erbr1" drop
15- ip daddr 10.200.0.1 udp dport 53 iifname != "ertap0" drop
16- ip daddr 10.200.0.1 tcp dport 53 iifname != "ertap0" drop
17- ip daddr 10.200.0.5 tcp dport 8089 iifname != "erbr1" drop
18- ip daddr 10.200.0.1 tcp dport 8089 iifname != "ertap0" drop
19- iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 udp dport 53 accept
20- iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 53 accept
21- iifname "erbr1" ip saddr 10.200.0.6 ip daddr 10.200.0.5 tcp dport 8089 accept
22- iifname "erbr1" drop
23- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 udp dport 53 accept
24- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 53 accept
25- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8088 accept
26- iifname "ertap0" ip saddr 10.200.0.2 ip daddr 10.200.0.1 tcp dport 8089 accept
27- iifname "ertap0" drop
28- iifname "ertap*" drop
29- iifname "erbr*" drop
30- }
31- chain forward {
32- type filter hook forward priority filter; policy accept;
33- iifname "ertap*" meta nfproto ipv6 drop
34- iifname "erbr*" meta nfproto ipv6 drop
35- iifname "erbr1" ip saddr != 10.200.0.6 drop
36- iifname "erbr1" ip daddr @blocked_v4 drop
37- iifname "ertap0" ip saddr != 10.200.0.2 drop
38- iifname "ertap0" ip daddr @blocked_v4 drop
39- iifname != { "ertap*", "erbr*" } oifname { "ertap*", "erbr*" } ct state established,related accept
40- oifname "ertap*" drop
41- oifname "erbr*" drop
42- iifname "erbr1" accept
43- iifname "ertap0" accept
44- iifname "ertap*" drop
45- iifname "erbr*" drop
46- }
47- chain postrouting {
48- type nat hook postrouting priority srcnat; policy accept;
49- ip saddr 10.200.0.6 oifname != "erbr1" masquerade
50- ip saddr 10.200.0.2 oifname != "ertap0" masquerade
51- }
52-}
A internal/sandbox/cache.go +46 -0
@@ -0,0 +1,46 @@
1+// SPDX-License-Identifier: MIT
2+
3+package sandbox
4+
5+import (
6+ "net"
7+
8+ "bnema.dev/ermite-runner/internal/cachesvc"
9+)
10+
11+// ServeCache serves the runner's cache service (Config.Cache, when set) on a
12+// Unix socket at path, the host end of the cache channel of sandbox id. The
13+// guest relay opens one connection to it per TCP connection it accepts on its
14+// loopback: through the sandbox's own bind mount (container) or through
15+// Firecracker's vsock proxy (VM, which connects to <uds>_<port> when the guest
16+// connects to the host on that port). Nothing but that sandbox can reach the
17+// path, and the service accepts on it only tokens made for job (see
18+// cachesvc.Service.Serve). own, when not nil, sets the owner and mode of the
19+// socket before it is served. mbit caps the channel's traffic, all its
20+// connections and both directions together (0: unlimited); a VM passes its
21+// NetMbit, as the cache stream does not cross eth0, which carries that limit.
22+// With no cache service this does nothing.
23+func (c Config) ServeCache(id, path, job string, mbit int, own func(path string) error) error {
24+ if c.Cache == nil {
25+ return nil
26+ }
27+ ln, err := net.Listen("unix", path)
28+ if err != nil {
29+ return err
30+ }
31+ if own != nil {
32+ if err := own(path); err != nil {
33+ _ = ln.Close()
34+ return err
35+ }
36+ }
37+ return c.Cache.Serve(id, ln, job, cachesvc.WithBandwidth(mbit))
38+}
39+
40+// StopCache ends the cache channel of sandbox id: it closes the listener (which
41+// removes the socket) and the open connections. It is idempotent.
42+func (c Config) StopCache(id string) {
43+ if c.Cache != nil {
44+ c.Cache.Stop(id)
45+ }
46+}
@@ -0,0 +1,46 @@
1+// SPDX-License-Identifier: MIT
2+
3+package sandbox
4+
5+import (
6+ "net"
7+
8+ "bnema.dev/ermite-runner/internal/cachesvc"
9+)
10+
11+// ServeCache serves the runner's cache service (Config.Cache, when set) on a
12+// Unix socket at path, the host end of the cache channel of sandbox id. The
13+// guest relay opens one connection to it per TCP connection it accepts on its
14+// loopback: through the sandbox's own bind mount (container) or through
15+// Firecracker's vsock proxy (VM, which connects to <uds>_<port> when the guest
16+// connects to the host on that port). Nothing but that sandbox can reach the
17+// path, and the service accepts on it only tokens made for job (see
18+// cachesvc.Service.Serve). own, when not nil, sets the owner and mode of the
19+// socket before it is served. mbit caps the channel's traffic, all its
20+// connections and both directions together (0: unlimited); a VM passes its
21+// NetMbit, as the cache stream does not cross eth0, which carries that limit.
22+// With no cache service this does nothing.
23+func (c Config) ServeCache(id, path, job string, mbit int, own func(path string) error) error {
24+ if c.Cache == nil {
25+ return nil
26+ }
27+ ln, err := net.Listen("unix", path)
28+ if err != nil {
29+ return err
30+ }
31+ if own != nil {
32+ if err := own(path); err != nil {
33+ _ = ln.Close()
34+ return err
35+ }
36+ }
37+ return c.Cache.Serve(id, ln, job, cachesvc.WithBandwidth(mbit))
38+}
39+
40+// StopCache ends the cache channel of sandbox id: it closes the listener (which
41+// removes the socket) and the open connections. It is idempotent.
42+func (c Config) StopCache(id string) {
43+ if c.Cache != nil {
44+ c.Cache.Stop(id)
45+ }
46+}
M internal/sandbox/config.go +3 -4
@@ -121,10 +121,9 @@ type Config struct {
121121 // reach, and the name the DNS forwarder answers with the sandbox's gateway.
122122 GuestForge *GuestForge
123123 // Cache, when not nil, is the runner's cache service (package cachesvc),
124- // served on the gateway of every sandbox on CachePort. The caller makes it
125- // before building the drivers, like GuestForge.
126- Cache *cachesvc.Service
127- CachePort uint16
124+ // served to every sandbox on a channel of its own (ServeCache), never on a
125+ // network. The caller makes it before building the drivers, like GuestForge.
126+ Cache *cachesvc.Service
128127 // Rate limits of micro-VMs; zero disables.
129128 DiskBandwidthMiB, DiskIOPS, NetMbit int
130129
@@ -121,10 +121,9 @@ type Config struct {
121 // reach, and the name the DNS forwarder answers with the sandbox's gateway.121 // reach, and the name the DNS forwarder answers with the sandbox's gateway.
122 GuestForge *GuestForge122 GuestForge *GuestForge
123 // Cache, when not nil, is the runner's cache service (package cachesvc),123 // Cache, when not nil, is the runner's cache service (package cachesvc),
124- // served on the gateway of every sandbox on CachePort. The caller makes it124+ // served to every sandbox on a channel of its own (ServeCache), never on a
125- // before building the drivers, like GuestForge.125+ // network. The caller makes it before building the drivers, like GuestForge.
126- Cache *cachesvc.Service126+ Cache *cachesvc.Service
127- CachePort uint16
128 // Rate limits of micro-VMs; zero disables.127 // Rate limits of micro-VMs; zero disables.
129 DiskBandwidthMiB, DiskIOPS, NetMbit int128 DiskBandwidthMiB, DiskIOPS, NetMbit int
130 129
A internal/sandbox/container/cache_test.go +200 -0
@@ -0,0 +1,200 @@
1+// SPDX-License-Identifier: MIT
2+
3+package container
4+
5+import (
6+ "context"
7+ "net"
8+ "net/http"
9+ "os"
10+ "path/filepath"
11+ "strings"
12+ "testing"
13+
14+ "bnema.dev/ermite-runner/internal/sandbox"
15+ "bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
16+)
17+
18+func cacheSock(d *Driver, sb sandbox.Sandbox) string {
19+ return filepath.Join(d.sockDir(sb.ID()), sandbox.CacheSocket)
20+}
21+
22+// Each container gets its own cache socket in its private run directory (the
23+// one directory bind-mounted into it), served for its own job only: a token of
24+// the other container's job gets a 403 on it, and nothing listens on a network.
25+// Stopping a container removes its channel and leaves the other one alone.
26+func TestCacheSocketPerContainer(t *testing.T) {
27+ e := newCEnv(t)
28+ svc := sandboxtest.NewCache(t)
29+ e.cfg.Cache = svc
30+ d := e.driver(t)
31+ start := func(job string) sandbox.Sandbox {
32+ t.Helper()
33+ sb, err := d.New()
34+ if err != nil {
35+ t.Fatal(err)
36+ }
37+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: job}); err != nil {
38+ t.Fatal(err)
39+ }
40+ return sb
41+ }
42+ a, b := start("job-a"), start("job-b")
43+ tokA, tokB := sandboxtest.CacheToken(svc, "job-a"), sandboxtest.CacheToken(svc, "job-b")
44+
45+ if cacheSock(d, a) == cacheSock(d, b) {
46+ t.Fatal("two containers share a cache socket")
47+ }
48+ // The socket is in the directory that is mounted as /run/ermite, next to
49+ // the executor socket, and in no other container's.
50+ e.eng.mu.Lock()
51+ mounts := map[string]bool{}
52+ for _, c := range e.eng.containers {
53+ for _, bm := range c.Binds {
54+ if bm.Target == sandbox.ContainerRunDir {
55+ mounts[bm.Source] = true
56+ }
57+ }
58+ }
59+ e.eng.mu.Unlock()
60+ for _, sb := range []sandbox.Sandbox{a, b} {
61+ if !mounts[filepath.Dir(cacheSock(d, sb))] {
62+ t.Fatalf("%s: the cache socket is not in a directory mounted in a container", sb.ID())
63+ }
64+ if st, err := os.Stat(cacheSock(d, sb)); err != nil || st.Mode().Perm() != 0o666 {
65+ t.Fatalf("%s: cache socket %v %v", sb.ID(), st, err)
66+ }
67+ }
68+
69+ for _, tc := range []struct {
70+ sb sandbox.Sandbox
71+ tok string
72+ want int
73+ }{
74+ {a, tokA, http.StatusNoContent},
75+ {b, tokB, http.StatusNoContent},
76+ {a, tokB, http.StatusForbidden},
77+ {b, tokA, http.StatusForbidden},
78+ } {
79+ got, err := sandboxtest.CacheLookup(cacheSock(d, tc.sb), tc.tok)
80+ if err != nil || got != tc.want {
81+ t.Errorf("%s: status %d err %v, want %d", tc.sb.ID(), got, err, tc.want)
82+ }
83+ }
84+
85+ if err := a.Stop(context.Background()); err != nil {
86+ t.Fatal(err)
87+ }
88+ if c, err := net.Dial("unix", cacheSock(d, a)); err == nil {
89+ _ = c.Close()
90+ t.Fatal("A's cache socket answers after Stop")
91+ }
92+ if got, err := sandboxtest.CacheLookup(cacheSock(d, b), tokB); err != nil || got != http.StatusNoContent {
93+ t.Fatalf("B after A's Stop: %d %v", got, err)
94+ }
95+ if err := b.Stop(context.Background()); err != nil {
96+ t.Fatal(err)
97+ }
98+ if left := e.leftovers(); len(left) != 0 {
99+ t.Fatalf("leftovers: %v", left)
100+ }
101+}
102+
103+// A failed start does not leave the channel behind.
104+func TestCacheSocketClosedWhenStartFails(t *testing.T) {
105+ e := newCEnv(t)
106+ svc := sandboxtest.NewCache(t)
107+ e.cfg.Cache = svc
108+ e.eng.failStart = os.ErrInvalid
109+ d := e.driver(t)
110+ sb, _ := d.New()
111+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err == nil {
112+ t.Fatal("expected a start error")
113+ }
114+ if err := sb.Stop(context.Background()); err != nil {
115+ t.Fatal(err)
116+ }
117+ // The same sandbox id could be served again: the service forgot it.
118+ ln, err := net.Listen("unix", filepath.Join(t.TempDir(), "s"))
119+ if err != nil {
120+ t.Fatal(err)
121+ }
122+ if err := svc.Serve(sb.ID(), ln, "j"); err != nil {
123+ t.Fatalf("the channel of the failed start is still registered: %v", err)
124+ }
125+}
126+
127+// The sun_path limit covers the cache socket, which is one byte longer than the
128+// executor socket: a directory that fits the executor socket exactly is refused
129+// when the cache is on, and still accepted when it is not.
130+func TestSockPathLimitCoversTheCacheSocket(t *testing.T) {
131+ pad := func(n int) string { return "/" + strings.Repeat("d", n-1) }
132+ // dir + "/exec.sock" is exactly maxSockPath bytes.
133+ exact := pad(maxSockPath - len("/"+sandbox.ContainerSocket))
134+ if len(filepath.Join(exact, sandbox.ContainerSocket)) != maxSockPath {
135+ t.Fatalf("setup: %d", len(filepath.Join(exact, sandbox.ContainerSocket)))
136+ }
137+ if err := checkSockPaths(exact, false); err != nil {
138+ t.Fatalf("%d-byte executor socket without cache: %v", maxSockPath, err)
139+ }
140+ err := checkSockPaths(exact, true)
141+ if err == nil || !strings.Contains(err.Error(), "cache socket path") || !strings.Contains(err.Error(), sandbox.EnvStateDir) {
142+ t.Fatalf("%d-byte executor socket with cache: %v", maxSockPath, err)
143+ }
144+ // One byte shorter: both fit.
145+ if err := checkSockPaths(exact[:len(exact)-1], true); err != nil {
146+ t.Fatalf("both sockets fit: %v", err)
147+ }
148+ // One byte longer: the executor socket is refused first.
149+ if err := checkSockPaths(exact+"d", true); err == nil || !strings.Contains(err.Error(), "executor socket path") {
150+ t.Fatalf("executor socket over the limit: %v", err)
151+ }
152+}
153+
154+// At Start, a state directory that fits the executor socket but not the cache
155+// socket fails with the actionable error and leaves nothing behind.
156+func TestStartRefusesACacheSocketPathOverTheLimit(t *testing.T) {
157+ e := newCEnv(t)
158+ e.cfg.Cache = sandboxtest.NewCache(t)
159+ pid, err := sandbox.NewID()
160+ if err != nil {
161+ t.Fatal(err)
162+ }
163+ // Grow the state directory so that exec.sock is exactly at the limit (ids
164+ // have a fixed length).
165+ probe := &Driver{cfg: e.cfg}
166+ grow := maxSockPath - len(filepath.Join(probe.sockDir(pid), sandbox.ContainerSocket))
167+ if grow < 2 {
168+ t.Skipf("state directory already %d bytes over", -grow)
169+ }
170+ e.cfg.StateDir = e.cfg.StateDir + "/" + strings.Repeat("d", grow-1)
171+ d := e.driver(t)
172+ sb, err := d.New()
173+ if err != nil {
174+ t.Fatal(err)
175+ }
176+ if got := len(filepath.Join(d.sockDir(sb.ID()), sandbox.ContainerSocket)); got != maxSockPath {
177+ t.Fatalf("setup: executor socket path is %d bytes, want %d", got, maxSockPath)
178+ }
179+ _, err = sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
180+ if err == nil || !strings.Contains(err.Error(), "cache socket path") || !strings.Contains(err.Error(), "limit of a Unix socket") {
181+ t.Fatalf("got %v", err)
182+ }
183+ if left := e.leftovers(); len(left) != 0 {
184+ t.Fatalf("leftovers: %v", left)
185+ }
186+}
187+
188+// Without a cache service there is no cache socket.
189+func TestNoCacheSocketWithoutService(t *testing.T) {
190+ e := newCEnv(t)
191+ d := e.driver(t)
192+ sb, _ := d.New()
193+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err != nil {
194+ t.Fatal(err)
195+ }
196+ if _, err := os.Stat(cacheSock(d, sb)); err == nil {
197+ t.Fatal("a cache socket exists with no cache service")
198+ }
199+ _ = sb.Stop(context.Background())
200+}
@@ -0,0 +1,200 @@
1+// SPDX-License-Identifier: MIT
2+
3+package container
4+
5+import (
6+ "context"
7+ "net"
8+ "net/http"
9+ "os"
10+ "path/filepath"
11+ "strings"
12+ "testing"
13+
14+ "bnema.dev/ermite-runner/internal/sandbox"
15+ "bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
16+)
17+
18+func cacheSock(d *Driver, sb sandbox.Sandbox) string {
19+ return filepath.Join(d.sockDir(sb.ID()), sandbox.CacheSocket)
20+}
21+
22+// Each container gets its own cache socket in its private run directory (the
23+// one directory bind-mounted into it), served for its own job only: a token of
24+// the other container's job gets a 403 on it, and nothing listens on a network.
25+// Stopping a container removes its channel and leaves the other one alone.
26+func TestCacheSocketPerContainer(t *testing.T) {
27+ e := newCEnv(t)
28+ svc := sandboxtest.NewCache(t)
29+ e.cfg.Cache = svc
30+ d := e.driver(t)
31+ start := func(job string) sandbox.Sandbox {
32+ t.Helper()
33+ sb, err := d.New()
34+ if err != nil {
35+ t.Fatal(err)
36+ }
37+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: job}); err != nil {
38+ t.Fatal(err)
39+ }
40+ return sb
41+ }
42+ a, b := start("job-a"), start("job-b")
43+ tokA, tokB := sandboxtest.CacheToken(svc, "job-a"), sandboxtest.CacheToken(svc, "job-b")
44+
45+ if cacheSock(d, a) == cacheSock(d, b) {
46+ t.Fatal("two containers share a cache socket")
47+ }
48+ // The socket is in the directory that is mounted as /run/ermite, next to
49+ // the executor socket, and in no other container's.
50+ e.eng.mu.Lock()
51+ mounts := map[string]bool{}
52+ for _, c := range e.eng.containers {
53+ for _, bm := range c.Binds {
54+ if bm.Target == sandbox.ContainerRunDir {
55+ mounts[bm.Source] = true
56+ }
57+ }
58+ }
59+ e.eng.mu.Unlock()
60+ for _, sb := range []sandbox.Sandbox{a, b} {
61+ if !mounts[filepath.Dir(cacheSock(d, sb))] {
62+ t.Fatalf("%s: the cache socket is not in a directory mounted in a container", sb.ID())
63+ }
64+ if st, err := os.Stat(cacheSock(d, sb)); err != nil || st.Mode().Perm() != 0o666 {
65+ t.Fatalf("%s: cache socket %v %v", sb.ID(), st, err)
66+ }
67+ }
68+
69+ for _, tc := range []struct {
70+ sb sandbox.Sandbox
71+ tok string
72+ want int
73+ }{
74+ {a, tokA, http.StatusNoContent},
75+ {b, tokB, http.StatusNoContent},
76+ {a, tokB, http.StatusForbidden},
77+ {b, tokA, http.StatusForbidden},
78+ } {
79+ got, err := sandboxtest.CacheLookup(cacheSock(d, tc.sb), tc.tok)
80+ if err != nil || got != tc.want {
81+ t.Errorf("%s: status %d err %v, want %d", tc.sb.ID(), got, err, tc.want)
82+ }
83+ }
84+
85+ if err := a.Stop(context.Background()); err != nil {
86+ t.Fatal(err)
87+ }
88+ if c, err := net.Dial("unix", cacheSock(d, a)); err == nil {
89+ _ = c.Close()
90+ t.Fatal("A's cache socket answers after Stop")
91+ }
92+ if got, err := sandboxtest.CacheLookup(cacheSock(d, b), tokB); err != nil || got != http.StatusNoContent {
93+ t.Fatalf("B after A's Stop: %d %v", got, err)
94+ }
95+ if err := b.Stop(context.Background()); err != nil {
96+ t.Fatal(err)
97+ }
98+ if left := e.leftovers(); len(left) != 0 {
99+ t.Fatalf("leftovers: %v", left)
100+ }
101+}
102+
103+// A failed start does not leave the channel behind.
104+func TestCacheSocketClosedWhenStartFails(t *testing.T) {
105+ e := newCEnv(t)
106+ svc := sandboxtest.NewCache(t)
107+ e.cfg.Cache = svc
108+ e.eng.failStart = os.ErrInvalid
109+ d := e.driver(t)
110+ sb, _ := d.New()
111+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err == nil {
112+ t.Fatal("expected a start error")
113+ }
114+ if err := sb.Stop(context.Background()); err != nil {
115+ t.Fatal(err)
116+ }
117+ // The same sandbox id could be served again: the service forgot it.
118+ ln, err := net.Listen("unix", filepath.Join(t.TempDir(), "s"))
119+ if err != nil {
120+ t.Fatal(err)
121+ }
122+ if err := svc.Serve(sb.ID(), ln, "j"); err != nil {
123+ t.Fatalf("the channel of the failed start is still registered: %v", err)
124+ }
125+}
126+
127+// The sun_path limit covers the cache socket, which is one byte longer than the
128+// executor socket: a directory that fits the executor socket exactly is refused
129+// when the cache is on, and still accepted when it is not.
130+func TestSockPathLimitCoversTheCacheSocket(t *testing.T) {
131+ pad := func(n int) string { return "/" + strings.Repeat("d", n-1) }
132+ // dir + "/exec.sock" is exactly maxSockPath bytes.
133+ exact := pad(maxSockPath - len("/"+sandbox.ContainerSocket))
134+ if len(filepath.Join(exact, sandbox.ContainerSocket)) != maxSockPath {
135+ t.Fatalf("setup: %d", len(filepath.Join(exact, sandbox.ContainerSocket)))
136+ }
137+ if err := checkSockPaths(exact, false); err != nil {
138+ t.Fatalf("%d-byte executor socket without cache: %v", maxSockPath, err)
139+ }
140+ err := checkSockPaths(exact, true)
141+ if err == nil || !strings.Contains(err.Error(), "cache socket path") || !strings.Contains(err.Error(), sandbox.EnvStateDir) {
142+ t.Fatalf("%d-byte executor socket with cache: %v", maxSockPath, err)
143+ }
144+ // One byte shorter: both fit.
145+ if err := checkSockPaths(exact[:len(exact)-1], true); err != nil {
146+ t.Fatalf("both sockets fit: %v", err)
147+ }
148+ // One byte longer: the executor socket is refused first.
149+ if err := checkSockPaths(exact+"d", true); err == nil || !strings.Contains(err.Error(), "executor socket path") {
150+ t.Fatalf("executor socket over the limit: %v", err)
151+ }
152+}
153+
154+// At Start, a state directory that fits the executor socket but not the cache
155+// socket fails with the actionable error and leaves nothing behind.
156+func TestStartRefusesACacheSocketPathOverTheLimit(t *testing.T) {
157+ e := newCEnv(t)
158+ e.cfg.Cache = sandboxtest.NewCache(t)
159+ pid, err := sandbox.NewID()
160+ if err != nil {
161+ t.Fatal(err)
162+ }
163+ // Grow the state directory so that exec.sock is exactly at the limit (ids
164+ // have a fixed length).
165+ probe := &Driver{cfg: e.cfg}
166+ grow := maxSockPath - len(filepath.Join(probe.sockDir(pid), sandbox.ContainerSocket))
167+ if grow < 2 {
168+ t.Skipf("state directory already %d bytes over", -grow)
169+ }
170+ e.cfg.StateDir = e.cfg.StateDir + "/" + strings.Repeat("d", grow-1)
171+ d := e.driver(t)
172+ sb, err := d.New()
173+ if err != nil {
174+ t.Fatal(err)
175+ }
176+ if got := len(filepath.Join(d.sockDir(sb.ID()), sandbox.ContainerSocket)); got != maxSockPath {
177+ t.Fatalf("setup: executor socket path is %d bytes, want %d", got, maxSockPath)
178+ }
179+ _, err = sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"})
180+ if err == nil || !strings.Contains(err.Error(), "cache socket path") || !strings.Contains(err.Error(), "limit of a Unix socket") {
181+ t.Fatalf("got %v", err)
182+ }
183+ if left := e.leftovers(); len(left) != 0 {
184+ t.Fatalf("leftovers: %v", left)
185+ }
186+}
187+
188+// Without a cache service there is no cache socket.
189+func TestNoCacheSocketWithoutService(t *testing.T) {
190+ e := newCEnv(t)
191+ d := e.driver(t)
192+ sb, _ := d.New()
193+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err != nil {
194+ t.Fatal(err)
195+ }
196+ if _, err := os.Stat(cacheSock(d, sb)); err == nil {
197+ t.Fatal("a cache socket exists with no cache service")
198+ }
199+ _ = sb.Stop(context.Background())
200+}
M internal/sandbox/container/driver.go +50 -5
@@ -164,7 +164,6 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
164164 d.net = opt.Net
165165 if d.net == nil {
166166 n := newNet(opt.Run, cfg.DNSPort, info.Podman, cfg.GuestForge.DNSName())
167- n.Cache, n.CachePort, n.CacheFreeBind = cfg.Cache, cfg.CachePort, info.Podman
168167 d.net = n
169168 }
170169 if err := d.net.EnableForwarding(); err != nil {
@@ -189,6 +188,24 @@ func newNet(run cmdrun.Runner, dnsPort int, podman bool, forgeHost string) *nets
189188 // maxSockPath is the longest Unix socket path Linux accepts (108 with the NUL).
190189 const maxSockPath = 107
191190
191+// checkSockPaths verifies that the sockets of a job's socket directory fit in
192+// sockaddr_un.sun_path (108 bytes): the executor socket and, when the cache is
193+// enabled, the cache socket, which is one byte longer. It says so instead of the
194+// "invalid argument" of bind.
195+func checkSockPaths(dir string, cache bool) error {
196+ socks := []struct{ what, name string }{{"executor", sandbox.ContainerSocket}}
197+ if cache {
198+ socks = append(socks, struct{ what, name string }{"cache", sandbox.CacheSocket})
199+ }
200+ for _, s := range socks {
201+ p := filepath.Join(dir, s.name)
202+ if len(p) > maxSockPath {
203+ return fmt.Errorf("%s socket path %q is %d bytes, over the %d-byte limit of a Unix socket: use a shorter %s", s.what, p, len(p), maxSockPath, sandbox.EnvStateDir)
204+ }
205+ }
206+ return nil
207+}
208+
192209 // privilegedWarning is logged when the driver starts and again at every job
193210 // start: a privileged container is host-root equivalent.
194211 const privilegedWarning = "privileged container: host-root equivalent, network policy best effort"
@@ -276,6 +293,7 @@ type containerSandbox struct {
276293 container string
277294 mounted bool
278295 listener net.Listener
296+ cacheOn bool
279297 conn net.Conn
280298 iface string
281299 gateway netip.Addr
@@ -337,6 +355,9 @@ func (s *containerSandbox) run(ctx context.Context, spec sandbox.JobSpec) (sandb
337355 if err != nil {
338356 return nil, err
339357 }
358+ if err := s.runCache(spec); err != nil {
359+ return nil, err
360+ }
340361 if err := s.runScratch(ctx, spec); err != nil {
341362 return nil, err
342363 }
@@ -371,9 +392,8 @@ func (s *containerSandbox) runSocket() (<-chan acceptResult, error) {
371392 }
372393 dir := d.sockDir(s.id)
373394 sock := filepath.Join(dir, sandbox.ContainerSocket)
374- // sockaddr_un.sun_path holds 108 bytes: say so instead of "invalid argument".
375- if len(sock) > maxSockPath {
376- return nil, fmt.Errorf("executor socket path %q is %d bytes, over the %d-byte limit of a Unix socket: use a shorter %s", sock, len(sock), maxSockPath, sandbox.EnvStateDir)
395+ if err := checkSockPaths(dir, d.cfg.Cache != nil); err != nil {
396+ return nil, err
377397 }
378398 ln, err := net.Listen("unix", sock)
379399 if err != nil {
@@ -393,6 +413,27 @@ func (s *containerSandbox) runSocket() (<-chan acceptResult, error) {
393413 return accepted, nil
394414 }
395415
416+// runCache serves the cache on a second socket of the job's private
417+// directory, next to the executor socket: the guest relay of the container
418+// connects to /run/ermite/cache.sock. Only this container has that directory
419+// mounted, and the service accepts on it the tokens of this job only.
420+func (s *containerSandbox) runCache(spec sandbox.JobSpec) error {
421+ d := s.d
422+ if d.cfg.Cache == nil {
423+ return nil
424+ }
425+ sock := filepath.Join(d.sockDir(s.id), sandbox.CacheSocket)
426+ // The container's root may be a mapped (sysbox) user: it must be able to
427+ // connect, as for the executor socket.
428+ // No bandwidth limit (0): a container has no per-job network rate limit
429+ // (NetMbit is a micro-VM setting), and its cache traffic is a local socket.
430+ if err := d.cfg.ServeCache(s.id, sock, spec.JobID, 0, func(p string) error { return os.Chmod(p, 0o666) }); err != nil {
431+ return fmt.Errorf("serve the cache socket: %w", err)
432+ }
433+ s.cacheOn = true
434+ return nil
435+}
436+
396437 // runScratch creates the sparse ext4 scratch disk (ERMITE_RUNNER_JOB_DISK) and
397438 // loop-mounts it; the workspace directory lives on it.
398439 func (s *containerSandbox) runScratch(ctx context.Context, spec sandbox.JobSpec) error {
@@ -449,7 +490,7 @@ func (s *containerSandbox) runContainer(ctx context.Context, spec sandbox.JobSpe
449490 Name: "ermite-" + s.id,
450491 Image: d.image,
451492 Entrypoint: []string{entry, "init"},
452- Env: []string{"ERMITE_CONNECT=unix:" + sandbox.ContainerRunDir + "/" + sandbox.ContainerSocket, "ERMITE_GATEWAY=" + netw.Gateway.String()},
493+ Env: []string{"ERMITE_CONNECT=unix:" + sandbox.ContainerRunDir + "/" + sandbox.ContainerSocket},
453494 Labels: labels,
454495 Runtime: d.runtime,
455496 Privileged: d.privileged,
@@ -545,6 +586,10 @@ func (s *containerSandbox) cleanupLocked(ctx context.Context) error {
545586 _ = s.listener.Close()
546587 s.listener = nil
547588 }
589+ if s.cacheOn {
590+ d.cfg.StopCache(s.id)
591+ s.cacheOn = false
592+ }
548593 if s.container != "" {
549594 if err := d.eng.RemoveContainer(ctx, s.container); err != nil {
550595 errs = append(errs, fmt.Errorf("remove container: %w", err))
@@ -164,7 +164,6 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
164 d.net = opt.Net164 d.net = opt.Net
165 if d.net == nil {165 if d.net == nil {
166 n := newNet(opt.Run, cfg.DNSPort, info.Podman, cfg.GuestForge.DNSName())166 n := newNet(opt.Run, cfg.DNSPort, info.Podman, cfg.GuestForge.DNSName())
167- n.Cache, n.CachePort, n.CacheFreeBind = cfg.Cache, cfg.CachePort, info.Podman
168 d.net = n167 d.net = n
169 }168 }
170 if err := d.net.EnableForwarding(); err != nil {169 if err := d.net.EnableForwarding(); err != nil {
@@ -189,6 +188,24 @@ func newNet(run cmdrun.Runner, dnsPort int, podman bool, forgeHost string) *nets
189 // maxSockPath is the longest Unix socket path Linux accepts (108 with the NUL).188 // maxSockPath is the longest Unix socket path Linux accepts (108 with the NUL).
190 const maxSockPath = 107189 const maxSockPath = 107
191 190
191+// checkSockPaths verifies that the sockets of a job's socket directory fit in
192+// sockaddr_un.sun_path (108 bytes): the executor socket and, when the cache is
193+// enabled, the cache socket, which is one byte longer. It says so instead of the
194+// "invalid argument" of bind.
195+func checkSockPaths(dir string, cache bool) error {
196+ socks := []struct{ what, name string }{{"executor", sandbox.ContainerSocket}}
197+ if cache {
198+ socks = append(socks, struct{ what, name string }{"cache", sandbox.CacheSocket})
199+ }
200+ for _, s := range socks {
201+ p := filepath.Join(dir, s.name)
202+ if len(p) > maxSockPath {
203+ return fmt.Errorf("%s socket path %q is %d bytes, over the %d-byte limit of a Unix socket: use a shorter %s", s.what, p, len(p), maxSockPath, sandbox.EnvStateDir)
204+ }
205+ }
206+ return nil
207+}
208+
192 // privilegedWarning is logged when the driver starts and again at every job209 // privilegedWarning is logged when the driver starts and again at every job
193 // start: a privileged container is host-root equivalent.210 // start: a privileged container is host-root equivalent.
194 const privilegedWarning = "privileged container: host-root equivalent, network policy best effort"211 const privilegedWarning = "privileged container: host-root equivalent, network policy best effort"
@@ -276,6 +293,7 @@ type containerSandbox struct {
276 container string293 container string
277 mounted bool294 mounted bool
278 listener net.Listener295 listener net.Listener
296+ cacheOn bool
279 conn net.Conn297 conn net.Conn
280 iface string298 iface string
281 gateway netip.Addr299 gateway netip.Addr
@@ -337,6 +355,9 @@ func (s *containerSandbox) run(ctx context.Context, spec sandbox.JobSpec) (sandb
337 if err != nil {355 if err != nil {
338 return nil, err356 return nil, err
339 }357 }
358+ if err := s.runCache(spec); err != nil {
359+ return nil, err
360+ }
340 if err := s.runScratch(ctx, spec); err != nil {361 if err := s.runScratch(ctx, spec); err != nil {
341 return nil, err362 return nil, err
342 }363 }
@@ -371,9 +392,8 @@ func (s *containerSandbox) runSocket() (<-chan acceptResult, error) {
371 }392 }
372 dir := d.sockDir(s.id)393 dir := d.sockDir(s.id)
373 sock := filepath.Join(dir, sandbox.ContainerSocket)394 sock := filepath.Join(dir, sandbox.ContainerSocket)
374- // sockaddr_un.sun_path holds 108 bytes: say so instead of "invalid argument".395+ if err := checkSockPaths(dir, d.cfg.Cache != nil); err != nil {
375- if len(sock) > maxSockPath {396+ return nil, err
376- return nil, fmt.Errorf("executor socket path %q is %d bytes, over the %d-byte limit of a Unix socket: use a shorter %s", sock, len(sock), maxSockPath, sandbox.EnvStateDir)
377 }397 }
378 ln, err := net.Listen("unix", sock)398 ln, err := net.Listen("unix", sock)
379 if err != nil {399 if err != nil {
@@ -393,6 +413,27 @@ func (s *containerSandbox) runSocket() (<-chan acceptResult, error) {
393 return accepted, nil413 return accepted, nil
394 }414 }
395 415
416+// runCache serves the cache on a second socket of the job's private
417+// directory, next to the executor socket: the guest relay of the container
418+// connects to /run/ermite/cache.sock. Only this container has that directory
419+// mounted, and the service accepts on it the tokens of this job only.
420+func (s *containerSandbox) runCache(spec sandbox.JobSpec) error {
421+ d := s.d
422+ if d.cfg.Cache == nil {
423+ return nil
424+ }
425+ sock := filepath.Join(d.sockDir(s.id), sandbox.CacheSocket)
426+ // The container's root may be a mapped (sysbox) user: it must be able to
427+ // connect, as for the executor socket.
428+ // No bandwidth limit (0): a container has no per-job network rate limit
429+ // (NetMbit is a micro-VM setting), and its cache traffic is a local socket.
430+ if err := d.cfg.ServeCache(s.id, sock, spec.JobID, 0, func(p string) error { return os.Chmod(p, 0o666) }); err != nil {
431+ return fmt.Errorf("serve the cache socket: %w", err)
432+ }
433+ s.cacheOn = true
434+ return nil
435+}
436+
396 // runScratch creates the sparse ext4 scratch disk (ERMITE_RUNNER_JOB_DISK) and437 // runScratch creates the sparse ext4 scratch disk (ERMITE_RUNNER_JOB_DISK) and
397 // loop-mounts it; the workspace directory lives on it.438 // loop-mounts it; the workspace directory lives on it.
398 func (s *containerSandbox) runScratch(ctx context.Context, spec sandbox.JobSpec) error {439 func (s *containerSandbox) runScratch(ctx context.Context, spec sandbox.JobSpec) error {
@@ -449,7 +490,7 @@ func (s *containerSandbox) runContainer(ctx context.Context, spec sandbox.JobSpe
449 Name: "ermite-" + s.id,490 Name: "ermite-" + s.id,
450 Image: d.image,491 Image: d.image,
451 Entrypoint: []string{entry, "init"},492 Entrypoint: []string{entry, "init"},
452- Env: []string{"ERMITE_CONNECT=unix:" + sandbox.ContainerRunDir + "/" + sandbox.ContainerSocket, "ERMITE_GATEWAY=" + netw.Gateway.String()},493+ Env: []string{"ERMITE_CONNECT=unix:" + sandbox.ContainerRunDir + "/" + sandbox.ContainerSocket},
453 Labels: labels,494 Labels: labels,
454 Runtime: d.runtime,495 Runtime: d.runtime,
455 Privileged: d.privileged,496 Privileged: d.privileged,
@@ -545,6 +586,10 @@ func (s *containerSandbox) cleanupLocked(ctx context.Context) error {
545 _ = s.listener.Close()586 _ = s.listener.Close()
546 s.listener = nil587 s.listener = nil
547 }588 }
589+ if s.cacheOn {
590+ d.cfg.StopCache(s.id)
591+ s.cacheOn = false
592+ }
548 if s.container != "" {593 if s.container != "" {
549 if err := d.eng.RemoveContainer(ctx, s.container); err != nil {594 if err := d.eng.RemoveContainer(ctx, s.container); err != nil {
550 errs = append(errs, fmt.Errorf("remove container: %w", err))595 errs = append(errs, fmt.Errorf("remove container: %w", err))
M internal/sandbox/contract.go +14 -1
@@ -2,7 +2,11 @@
22
33 package sandbox
44
5-import "os"
5+import (
6+ "os"
7+
8+ "bnema.dev/ermite-runner/internal/cachesvc"
9+)
610
711 // The host/guest contract shared by the drivers and the guest side
812 // (internal/guest). Changing a value here changes both sides at once because
@@ -10,6 +14,15 @@ import "os"
1014 const (
1115 // VsockPort is the vsock port the executor connects to on the host (CID 2).
1216 VsockPort = 1024
17+ // CacheVsockPort is the vsock port of the cache streams of a VM: each TCP
18+ // connection the guest relay accepts on its loopback opens one connection to
19+ // the host (CID 2) on this port. It is also the loopback port of the cache in
20+ // the guest.
21+ CacheVsockPort = cachesvc.DefaultPort
22+ // CacheSocket is the cache stream socket of a container, next to the
23+ // executor socket in ContainerRunDir; one connection per relayed TCP
24+ // connection.
25+ CacheSocket = "cache.sock"
1326 // GuestExecDir and GuestExecName locate the runner executable inside the
1427 // sandbox (read-only), in both modes.
1528 GuestExecDir = "/opt/ermite"
@@ -2,7 +2,11 @@
2 2
3 package sandbox3 package sandbox
4 4
5-import "os"5+import (
6+ "os"
7+
8+ "bnema.dev/ermite-runner/internal/cachesvc"
9+)
6 10
7 // The host/guest contract shared by the drivers and the guest side11 // The host/guest contract shared by the drivers and the guest side
8 // (internal/guest). Changing a value here changes both sides at once because12 // (internal/guest). Changing a value here changes both sides at once because
@@ -10,6 +14,15 @@ import "os"
10 const (14 const (
11 // VsockPort is the vsock port the executor connects to on the host (CID 2).15 // VsockPort is the vsock port the executor connects to on the host (CID 2).
12 VsockPort = 102416 VsockPort = 1024
17+ // CacheVsockPort is the vsock port of the cache streams of a VM: each TCP
18+ // connection the guest relay accepts on its loopback opens one connection to
19+ // the host (CID 2) on this port. It is also the loopback port of the cache in
20+ // the guest.
21+ CacheVsockPort = cachesvc.DefaultPort
22+ // CacheSocket is the cache stream socket of a container, next to the
23+ // executor socket in ContainerRunDir; one connection per relayed TCP
24+ // connection.
25+ CacheSocket = "cache.sock"
13 // GuestExecDir and GuestExecName locate the runner executable inside the26 // GuestExecDir and GuestExecName locate the runner executable inside the
14 // sandbox (read-only), in both modes.27 // sandbox (read-only), in both modes.
15 GuestExecDir = "/opt/ermite"28 GuestExecDir = "/opt/ermite"
M internal/sandbox/netsetup/netsetup.go +1 -23
@@ -14,7 +14,6 @@ import (
1414 "os"
1515 "strings"
1616
17- "bnema.dev/ermite-runner/internal/cachesvc"
1817 "bnema.dev/ermite-runner/internal/cmdrun"
1918 "bnema.dev/ermite-runner/internal/dnsfwd"
2019 "bnema.dev/ermite-runner/internal/netpolicy"
@@ -37,14 +36,6 @@ var _ Plumbing = (*Net)(nil)
3736 type Net struct {
3837 Policy *netpolicy.Manager
3938 DNS *dnsfwd.Forwarder
40- // Cache, when not nil, is the runner's cache service: it listens on the
41- // gateway of every guest, on CachePort, and netpolicy opens that port to
42- // that guest only.
43- Cache *cachesvc.Service
44- CachePort uint16
45- // CacheFreeBind lets the cache listener bind before the gateway address
46- // exists (podman, like dnsfwd.Forwarder.FreeBind).
47- CacheFreeBind bool
4839 // ForwardFile is the ip_forward sysctl; overridden in tests.
4940 ForwardFile string
5041 }
@@ -77,30 +68,18 @@ func (n *Net) EnableForwarding() error {
7768 // gateway address, which must already be configured on the host interface.
7869 // On error nothing stays installed.
7970 func (n *Net) Attach(ctx context.Context, g netpolicy.Guest) error {
80- if n.Cache != nil {
81- g.CachePort = n.CachePort
82- }
8371 if err := n.Policy.Add(ctx, g); err != nil {
8472 return err
8573 }
8674 if err := n.DNS.Listen(ctx, g.Gateway); err != nil {
8775 return errors.Join(err, n.Policy.Remove(context.WithoutCancel(ctx), g.Iface))
8876 }
89- if n.Cache != nil {
90- if err := n.Cache.Listen(ctx, g.Gateway, n.CachePort, n.CacheFreeBind); err != nil {
91- n.DNS.Close(g.Gateway)
92- return errors.Join(err, n.Policy.Remove(context.WithoutCancel(ctx), g.Iface))
93- }
94- }
9577 return nil
9678 }
9779
9880 // Detach undoes Attach. It is idempotent and keeps going after an error.
9981 func (n *Net) Detach(ctx context.Context, iface string, gateway netip.Addr) error {
10082 n.DNS.Close(gateway)
101- if n.Cache != nil {
102- n.Cache.Close(gateway)
103- }
10483 return n.Policy.Remove(ctx, iface)
10584 }
10685
@@ -108,8 +87,7 @@ func (n *Net) Detach(ctx context.Context, iface string, gateway netip.Addr) erro
10887 // run. Only the table is touched, never other tables.
10988 func (n *Net) Reset(ctx context.Context) error { return n.Policy.Reset(ctx) }
11089
111-// Close stops every DNS listener. Cache listeners end with Detach; the cache
112-// service itself belongs to whoever made it.
90+// Close stops every DNS listener.
11391 func (n *Net) Close() {
11492 n.DNS.CloseAll()
11593 }
@@ -14,7 +14,6 @@ import (
14 "os"14 "os"
15 "strings"15 "strings"
16 16
17- "bnema.dev/ermite-runner/internal/cachesvc"
18 "bnema.dev/ermite-runner/internal/cmdrun"17 "bnema.dev/ermite-runner/internal/cmdrun"
19 "bnema.dev/ermite-runner/internal/dnsfwd"18 "bnema.dev/ermite-runner/internal/dnsfwd"
20 "bnema.dev/ermite-runner/internal/netpolicy"19 "bnema.dev/ermite-runner/internal/netpolicy"
@@ -37,14 +36,6 @@ var _ Plumbing = (*Net)(nil)
37 type Net struct {36 type Net struct {
38 Policy *netpolicy.Manager37 Policy *netpolicy.Manager
39 DNS *dnsfwd.Forwarder38 DNS *dnsfwd.Forwarder
40- // Cache, when not nil, is the runner's cache service: it listens on the
41- // gateway of every guest, on CachePort, and netpolicy opens that port to
42- // that guest only.
43- Cache *cachesvc.Service
44- CachePort uint16
45- // CacheFreeBind lets the cache listener bind before the gateway address
46- // exists (podman, like dnsfwd.Forwarder.FreeBind).
47- CacheFreeBind bool
48 // ForwardFile is the ip_forward sysctl; overridden in tests.39 // ForwardFile is the ip_forward sysctl; overridden in tests.
49 ForwardFile string40 ForwardFile string
50 }41 }
@@ -77,30 +68,18 @@ func (n *Net) EnableForwarding() error {
77 // gateway address, which must already be configured on the host interface.68 // gateway address, which must already be configured on the host interface.
78 // On error nothing stays installed.69 // On error nothing stays installed.
79 func (n *Net) Attach(ctx context.Context, g netpolicy.Guest) error {70 func (n *Net) Attach(ctx context.Context, g netpolicy.Guest) error {
80- if n.Cache != nil {
81- g.CachePort = n.CachePort
82- }
83 if err := n.Policy.Add(ctx, g); err != nil {71 if err := n.Policy.Add(ctx, g); err != nil {
84 return err72 return err
85 }73 }
86 if err := n.DNS.Listen(ctx, g.Gateway); err != nil {74 if err := n.DNS.Listen(ctx, g.Gateway); err != nil {
87 return errors.Join(err, n.Policy.Remove(context.WithoutCancel(ctx), g.Iface))75 return errors.Join(err, n.Policy.Remove(context.WithoutCancel(ctx), g.Iface))
88 }76 }
89- if n.Cache != nil {
90- if err := n.Cache.Listen(ctx, g.Gateway, n.CachePort, n.CacheFreeBind); err != nil {
91- n.DNS.Close(g.Gateway)
92- return errors.Join(err, n.Policy.Remove(context.WithoutCancel(ctx), g.Iface))
93- }
94- }
95 return nil77 return nil
96 }78 }
97 79
98 // Detach undoes Attach. It is idempotent and keeps going after an error.80 // Detach undoes Attach. It is idempotent and keeps going after an error.
99 func (n *Net) Detach(ctx context.Context, iface string, gateway netip.Addr) error {81 func (n *Net) Detach(ctx context.Context, iface string, gateway netip.Addr) error {
100 n.DNS.Close(gateway)82 n.DNS.Close(gateway)
101- if n.Cache != nil {
102- n.Cache.Close(gateway)
103- }
104 return n.Policy.Remove(ctx, iface)83 return n.Policy.Remove(ctx, iface)
105 }84 }
106 85
@@ -108,8 +87,7 @@ func (n *Net) Detach(ctx context.Context, iface string, gateway netip.Addr) erro
108 // run. Only the table is touched, never other tables.87 // run. Only the table is touched, never other tables.
109 func (n *Net) Reset(ctx context.Context) error { return n.Policy.Reset(ctx) }88 func (n *Net) Reset(ctx context.Context) error { return n.Policy.Reset(ctx) }
110 89
111-// Close stops every DNS listener. Cache listeners end with Detach; the cache90+// Close stops every DNS listener.
112-// service itself belongs to whoever made it.
113 func (n *Net) Close() {91 func (n *Net) Close() {
114 n.DNS.CloseAll()92 n.DNS.CloseAll()
115 }93 }
M internal/sandbox/netsetup/netsetup_test.go +20 -90
@@ -5,18 +5,12 @@ package netsetup
55 import (
66 "context"
77 "errors"
8- "net"
9- "net/http"
108 "net/netip"
119 "os"
1210 "path/filepath"
13- "strconv"
1411 "strings"
15- "syscall"
1612 "testing"
17- "time"
1813
19- "bnema.dev/ermite-runner/internal/cachesvc"
2014 "bnema.dev/ermite-runner/internal/cmdrun"
2115 "bnema.dev/ermite-runner/internal/netpolicy"
2216 )
@@ -50,6 +44,26 @@ func TestAttachDetach(t *testing.T) {
5044 }
5145 }
5246
47+// The host network surface of a guest is DNS and the firewall policy, nothing
48+// else: Attach opens no cache port, neither in the policy nor as a listener.
49+func TestAttachOpensNoCachePort(t *testing.T) {
50+ f := &cmdrun.Fake{}
51+ n := New(f, 0, "")
52+ t.Cleanup(n.Close)
53+ if err := n.Attach(context.Background(), guest()); err != nil {
54+ t.Skip("cannot bind loopback:", err)
55+ }
56+ rs := f.Calls[0].Stdin
57+ for _, port := range []string{"8089", "9089"} {
58+ if strings.Contains(rs, port) {
59+ t.Fatalf("the policy mentions the cache port %s:\n%s", port, rs)
60+ }
61+ }
62+ if !strings.Contains(rs, "dport 53 accept") {
63+ t.Fatalf("the policy lost DNS:\n%s", rs)
64+ }
65+}
66+
5367 func TestAttachRollsBackPolicyWhenDNSFails(t *testing.T) {
5468 f := &cmdrun.Fake{}
5569 n := New(f, 0, "")
@@ -93,87 +107,3 @@ func TestEnableForwarding(t *testing.T) {
93107 t.Fatalf("want a clear error, got %v", err)
94108 }
95109 }
96-
97-// With a cache service, Attach opens its port in the policy of that guest and
98-// serves it on the gateway; Detach closes both.
99-func TestAttachDetachCache(t *testing.T) {
100- probe := listenLoopback(t)
101- port := uint16(probe.Addr().(*net.TCPAddr).Port)
102- _ = probe.Close()
103-
104- svc, err := cachesvc.New(cachesvc.Config{Dir: filepath.Join(t.TempDir(), "cache")})
105- if err != nil {
106- t.Fatal(err)
107- }
108- t.Cleanup(func() { _ = svc.CloseAll() })
109- f := &cmdrun.Fake{}
110- n := New(f, 0, "")
111- n.Cache, n.CachePort = svc, port
112- t.Cleanup(n.Close)
113- ctx := context.Background()
114- if err := n.Attach(ctx, guest()); err != nil {
115- t.Fatalf("Attach: %v", err) // the port was free a moment ago: not an environmental skip
116- }
117- if rs := f.Calls[0].Stdin; !strings.Contains(rs, "ip daddr 127.0.0.1 tcp dport "+itoa(port)+" iifname != \"ertap0\" drop") ||
118- !strings.Contains(rs, "iifname \"ertap0\" ip saddr 10.200.0.2 ip daddr 127.0.0.1 tcp dport "+itoa(port)+" accept") {
119- t.Fatalf("the policy lacks the cache port:\n%s", rs)
120- }
121- cli := &http.Client{Timeout: 5 * time.Second}
122- url := "http://127.0.0.1:" + itoa(port) + cachesvc.APIPath + "cache?keys=k&version=v"
123- resp, err := cli.Get(url)
124- if err != nil {
125- t.Fatalf("the cache does not answer on the gateway: %v", err)
126- }
127- _ = resp.Body.Close()
128- if resp.StatusCode != http.StatusForbidden { // no token
129- t.Fatalf("status %d", resp.StatusCode)
130- }
131- if err := n.Detach(ctx, "ertap0", guest().Gateway); err != nil {
132- t.Fatal(err)
133- }
134- if resp, err := cli.Get(url); err == nil {
135- _ = resp.Body.Close()
136- t.Fatal("the cache still answers after Detach")
137- }
138-}
139-
140-// A failure to serve the cache rolls back the policy and the DNS listener.
141-func TestAttachRollsBackWhenCacheFails(t *testing.T) {
142- busy := listenLoopback(t)
143- defer func() { _ = busy.Close() }()
144- svc, err := cachesvc.New(cachesvc.Config{Dir: filepath.Join(t.TempDir(), "cache")})
145- if err != nil {
146- t.Fatal(err)
147- }
148- t.Cleanup(func() { _ = svc.CloseAll() })
149- f := &cmdrun.Fake{}
150- n := New(f, 0, "")
151- n.Cache, n.CachePort = svc, uint16(busy.Addr().(*net.TCPAddr).Port)
152- t.Cleanup(n.Close)
153- if err := n.Attach(context.Background(), guest()); err == nil {
154- t.Fatal("Attach must fail when the cache port is taken")
155- }
156- if _, ok := n.DNS.Addr(guest().Gateway); ok {
157- t.Fatal("DNS still listening after the rollback")
158- }
159- if len(f.Calls) != 2 || f.Calls[1].Stdin != netpolicy.Ruleset(nil) {
160- t.Fatalf("calls: %v", f.Lines())
161- }
162-}
163-
164-// listenLoopback binds a free loopback port. Only a sandbox that forbids
165-// sockets altogether (EPERM, EACCES, EAFNOSUPPORT) skips the test; any other
166-// error fails it.
167-func listenLoopback(t *testing.T) net.Listener {
168- t.Helper()
169- ln, err := net.Listen("tcp", "127.0.0.1:0")
170- if err != nil {
171- if errors.Is(err, syscall.EPERM) || errors.Is(err, syscall.EACCES) || errors.Is(err, syscall.EAFNOSUPPORT) {
172- t.Skip("loopback sockets are not allowed here:", err)
173- }
174- t.Fatal(err)
175- }
176- return ln
177-}
178-
179-func itoa(p uint16) string { return strconv.Itoa(int(p)) }
@@ -5,18 +5,12 @@ package netsetup
5 import (5 import (
6 "context"6 "context"
7 "errors"7 "errors"
8- "net"
9- "net/http"
10 "net/netip"8 "net/netip"
11 "os"9 "os"
12 "path/filepath"10 "path/filepath"
13- "strconv"
14 "strings"11 "strings"
15- "syscall"
16 "testing"12 "testing"
17- "time"
18 13
19- "bnema.dev/ermite-runner/internal/cachesvc"
20 "bnema.dev/ermite-runner/internal/cmdrun"14 "bnema.dev/ermite-runner/internal/cmdrun"
21 "bnema.dev/ermite-runner/internal/netpolicy"15 "bnema.dev/ermite-runner/internal/netpolicy"
22 )16 )
@@ -50,6 +44,26 @@ func TestAttachDetach(t *testing.T) {
50 }44 }
51 }45 }
52 46
47+// The host network surface of a guest is DNS and the firewall policy, nothing
48+// else: Attach opens no cache port, neither in the policy nor as a listener.
49+func TestAttachOpensNoCachePort(t *testing.T) {
50+ f := &cmdrun.Fake{}
51+ n := New(f, 0, "")
52+ t.Cleanup(n.Close)
53+ if err := n.Attach(context.Background(), guest()); err != nil {
54+ t.Skip("cannot bind loopback:", err)
55+ }
56+ rs := f.Calls[0].Stdin
57+ for _, port := range []string{"8089", "9089"} {
58+ if strings.Contains(rs, port) {
59+ t.Fatalf("the policy mentions the cache port %s:\n%s", port, rs)
60+ }
61+ }
62+ if !strings.Contains(rs, "dport 53 accept") {
63+ t.Fatalf("the policy lost DNS:\n%s", rs)
64+ }
65+}
66+
53 func TestAttachRollsBackPolicyWhenDNSFails(t *testing.T) {67 func TestAttachRollsBackPolicyWhenDNSFails(t *testing.T) {
54 f := &cmdrun.Fake{}68 f := &cmdrun.Fake{}
55 n := New(f, 0, "")69 n := New(f, 0, "")
@@ -93,87 +107,3 @@ func TestEnableForwarding(t *testing.T) {
93 t.Fatalf("want a clear error, got %v", err)107 t.Fatalf("want a clear error, got %v", err)
94 }108 }
95 }109 }
96-
97-// With a cache service, Attach opens its port in the policy of that guest and
98-// serves it on the gateway; Detach closes both.
99-func TestAttachDetachCache(t *testing.T) {
100- probe := listenLoopback(t)
101- port := uint16(probe.Addr().(*net.TCPAddr).Port)
102- _ = probe.Close()
103-
104- svc, err := cachesvc.New(cachesvc.Config{Dir: filepath.Join(t.TempDir(), "cache")})
105- if err != nil {
106- t.Fatal(err)
107- }
108- t.Cleanup(func() { _ = svc.CloseAll() })
109- f := &cmdrun.Fake{}
110- n := New(f, 0, "")
111- n.Cache, n.CachePort = svc, port
112- t.Cleanup(n.Close)
113- ctx := context.Background()
114- if err := n.Attach(ctx, guest()); err != nil {
115- t.Fatalf("Attach: %v", err) // the port was free a moment ago: not an environmental skip
116- }
117- if rs := f.Calls[0].Stdin; !strings.Contains(rs, "ip daddr 127.0.0.1 tcp dport "+itoa(port)+" iifname != \"ertap0\" drop") ||
118- !strings.Contains(rs, "iifname \"ertap0\" ip saddr 10.200.0.2 ip daddr 127.0.0.1 tcp dport "+itoa(port)+" accept") {
119- t.Fatalf("the policy lacks the cache port:\n%s", rs)
120- }
121- cli := &http.Client{Timeout: 5 * time.Second}
122- url := "http://127.0.0.1:" + itoa(port) + cachesvc.APIPath + "cache?keys=k&version=v"
123- resp, err := cli.Get(url)
124- if err != nil {
125- t.Fatalf("the cache does not answer on the gateway: %v", err)
126- }
127- _ = resp.Body.Close()
128- if resp.StatusCode != http.StatusForbidden { // no token
129- t.Fatalf("status %d", resp.StatusCode)
130- }
131- if err := n.Detach(ctx, "ertap0", guest().Gateway); err != nil {
132- t.Fatal(err)
133- }
134- if resp, err := cli.Get(url); err == nil {
135- _ = resp.Body.Close()
136- t.Fatal("the cache still answers after Detach")
137- }
138-}
139-
140-// A failure to serve the cache rolls back the policy and the DNS listener.
141-func TestAttachRollsBackWhenCacheFails(t *testing.T) {
142- busy := listenLoopback(t)
143- defer func() { _ = busy.Close() }()
144- svc, err := cachesvc.New(cachesvc.Config{Dir: filepath.Join(t.TempDir(), "cache")})
145- if err != nil {
146- t.Fatal(err)
147- }
148- t.Cleanup(func() { _ = svc.CloseAll() })
149- f := &cmdrun.Fake{}
150- n := New(f, 0, "")
151- n.Cache, n.CachePort = svc, uint16(busy.Addr().(*net.TCPAddr).Port)
152- t.Cleanup(n.Close)
153- if err := n.Attach(context.Background(), guest()); err == nil {
154- t.Fatal("Attach must fail when the cache port is taken")
155- }
156- if _, ok := n.DNS.Addr(guest().Gateway); ok {
157- t.Fatal("DNS still listening after the rollback")
158- }
159- if len(f.Calls) != 2 || f.Calls[1].Stdin != netpolicy.Ruleset(nil) {
160- t.Fatalf("calls: %v", f.Lines())
161- }
162-}
163-
164-// listenLoopback binds a free loopback port. Only a sandbox that forbids
165-// sockets altogether (EPERM, EACCES, EAFNOSUPPORT) skips the test; any other
166-// error fails it.
167-func listenLoopback(t *testing.T) net.Listener {
168- t.Helper()
169- ln, err := net.Listen("tcp", "127.0.0.1:0")
170- if err != nil {
171- if errors.Is(err, syscall.EPERM) || errors.Is(err, syscall.EACCES) || errors.Is(err, syscall.EAFNOSUPPORT) {
172- t.Skip("loopback sockets are not allowed here:", err)
173- }
174- t.Fatal(err)
175- }
176- return ln
177-}
178-
179-func itoa(p uint16) string { return strconv.Itoa(int(p)) }
M internal/sandbox/sandboxtest/sandboxtest.go +46 -0
@@ -7,12 +7,16 @@ import (
77 "context"
88 "encoding/binary"
99 "io"
10+ "net"
11+ "net/http"
1012 "net/netip"
1113 "os"
1214 "path/filepath"
1315 "sync"
1416 "testing"
17+ "time"
1518
19+ "bnema.dev/ermite-runner/internal/cachesvc"
1620 "bnema.dev/ermite-runner/internal/cmdrun"
1721 "bnema.dev/ermite-runner/internal/netpolicy"
1822 "bnema.dev/ermite-runner/internal/sandbox"
@@ -124,6 +128,48 @@ func (e *Externals) Externals() []sandbox.External {
124128 return append([]sandbox.External(nil), e.set...)
125129 }
126130
131+// NewCache starts a cache service in a temporary directory and closes it with
132+// the test.
133+func NewCache(t testing.TB) *cachesvc.Service {
134+ t.Helper()
135+ svc, err := cachesvc.New(cachesvc.Config{Dir: filepath.Join(t.TempDir(), "cache")})
136+ if err != nil {
137+ t.Fatal(err)
138+ }
139+ t.Cleanup(func() { _ = svc.CloseAll() })
140+ return svc
141+}
142+
143+// CacheLookup sends a cache lookup with token over the Unix socket sock, as the
144+// guest relay delivers it, and returns the HTTP status; the error is that of
145+// the connection (the socket is closed or gone).
146+func CacheLookup(sock, token string) (int, error) {
147+ cl := &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{
148+ DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
149+ var d net.Dialer
150+ return d.DialContext(ctx, "unix", sock)
151+ },
152+ DisableKeepAlives: true,
153+ }}
154+ req, err := http.NewRequest("GET", "http://cache"+cachesvc.APIPath+"cache?keys=k&version=v", nil)
155+ if err != nil {
156+ return 0, err
157+ }
158+ req.Header.Set("Authorization", "Bearer "+token)
159+ resp, err := cl.Do(req)
160+ if err != nil {
161+ return 0, err
162+ }
163+ _ = resp.Body.Close()
164+ return resp.StatusCode, nil
165+}
166+
167+// CacheToken is a token of the service for a job that may read and write main.
168+func CacheToken(svc *cachesvc.Service, job string) string {
169+ ref := "refs/heads/main"
170+ return svc.Token(job, cachesvc.Scope{Repository: "bnema/app", WriteRef: ref, ReadRefs: []string{ref}}, time.Hour)
171+}
172+
127173 // MakeExternal creates a host directory holding bin/node with content and
128174 // returns it as an external called name.
129175 func MakeExternal(t testing.TB, name, id, content string) sandbox.External {
@@ -7,12 +7,16 @@ import (
7 "context"7 "context"
8 "encoding/binary"8 "encoding/binary"
9 "io"9 "io"
10+ "net"
11+ "net/http"
10 "net/netip"12 "net/netip"
11 "os"13 "os"
12 "path/filepath"14 "path/filepath"
13 "sync"15 "sync"
14 "testing"16 "testing"
17+ "time"
15 18
19+ "bnema.dev/ermite-runner/internal/cachesvc"
16 "bnema.dev/ermite-runner/internal/cmdrun"20 "bnema.dev/ermite-runner/internal/cmdrun"
17 "bnema.dev/ermite-runner/internal/netpolicy"21 "bnema.dev/ermite-runner/internal/netpolicy"
18 "bnema.dev/ermite-runner/internal/sandbox"22 "bnema.dev/ermite-runner/internal/sandbox"
@@ -124,6 +128,48 @@ func (e *Externals) Externals() []sandbox.External {
124 return append([]sandbox.External(nil), e.set...)128 return append([]sandbox.External(nil), e.set...)
125 }129 }
126 130
131+// NewCache starts a cache service in a temporary directory and closes it with
132+// the test.
133+func NewCache(t testing.TB) *cachesvc.Service {
134+ t.Helper()
135+ svc, err := cachesvc.New(cachesvc.Config{Dir: filepath.Join(t.TempDir(), "cache")})
136+ if err != nil {
137+ t.Fatal(err)
138+ }
139+ t.Cleanup(func() { _ = svc.CloseAll() })
140+ return svc
141+}
142+
143+// CacheLookup sends a cache lookup with token over the Unix socket sock, as the
144+// guest relay delivers it, and returns the HTTP status; the error is that of
145+// the connection (the socket is closed or gone).
146+func CacheLookup(sock, token string) (int, error) {
147+ cl := &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{
148+ DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
149+ var d net.Dialer
150+ return d.DialContext(ctx, "unix", sock)
151+ },
152+ DisableKeepAlives: true,
153+ }}
154+ req, err := http.NewRequest("GET", "http://cache"+cachesvc.APIPath+"cache?keys=k&version=v", nil)
155+ if err != nil {
156+ return 0, err
157+ }
158+ req.Header.Set("Authorization", "Bearer "+token)
159+ resp, err := cl.Do(req)
160+ if err != nil {
161+ return 0, err
162+ }
163+ _ = resp.Body.Close()
164+ return resp.StatusCode, nil
165+}
166+
167+// CacheToken is a token of the service for a job that may read and write main.
168+func CacheToken(svc *cachesvc.Service, job string) string {
169+ ref := "refs/heads/main"
170+ return svc.Token(job, cachesvc.Scope{Repository: "bnema/app", WriteRef: ref, ReadRefs: []string{ref}}, time.Hour)
171+}
172+
127 // MakeExternal creates a host directory holding bin/node with content and173 // MakeExternal creates a host directory holding bin/node with content and
128 // returns it as an external called name.174 // returns it as an external called name.
129 func MakeExternal(t testing.TB, name, id, content string) sandbox.External {175 func MakeExternal(t testing.TB, name, id, content string) sandbox.External {
A internal/sandbox/vm/cache_test.go +128 -0
@@ -0,0 +1,128 @@
1+// SPDX-License-Identifier: MIT
2+
3+package vm
4+
5+import (
6+ "context"
7+ "net"
8+ "net/http"
9+ "os"
10+ "strings"
11+ "testing"
12+ "time"
13+
14+ "bnema.dev/ermite-runner/internal/sandbox"
15+ "bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
16+)
17+
18+func startVM(t *testing.T, d *Driver, job string) *vmSandbox {
19+ t.Helper()
20+ sb, err := d.New()
21+ if err != nil {
22+ t.Fatal(err)
23+ }
24+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: job}); err != nil {
25+ t.Fatal(err)
26+ }
27+ return sb.(*vmSandbox)
28+}
29+
30+// Each VM serves the cache on its own host-side vsock socket, <uds>_<cache
31+// port> in the VM's own directory (the path Firecracker connects to when the
32+// guest connects to the host on the cache port), for its own job only. Stopping
33+// one VM closes its channel and leaves the other alone.
34+func TestCacheVsockSocketPerVM(t *testing.T) {
35+ e := newEnv(t, "")
36+ svc := sandboxtest.NewCache(t)
37+ e.cfg.Cache = svc
38+ d := e.driver(t)
39+ a, b := startVM(t, d, "job-a"), startVM(t, d, "job-b")
40+ tokA, tokB := sandboxtest.CacheToken(svc, "job-a"), sandboxtest.CacheToken(svc, "job-b")
41+
42+ if a.cacheSocket() == b.cacheSocket() {
43+ t.Fatal("two VMs share a cache socket")
44+ }
45+ for _, sb := range []*vmSandbox{a, b} {
46+ if !strings.HasSuffix(sb.cacheSocket(), "_9089") || !strings.HasPrefix(sb.cacheSocket(), sb.lay.root+"/") {
47+ t.Fatalf("cache socket %q is not <vm dir>/v.sock_9089", sb.cacheSocket())
48+ }
49+ }
50+ for _, tc := range []struct {
51+ sb *vmSandbox
52+ tok string
53+ want int
54+ }{
55+ {a, tokA, http.StatusNoContent},
56+ {b, tokB, http.StatusNoContent},
57+ {a, tokB, http.StatusForbidden},
58+ {b, tokA, http.StatusForbidden},
59+ } {
60+ got, err := sandboxtest.CacheLookup(tc.sb.cacheSocket(), tc.tok)
61+ if err != nil || got != tc.want {
62+ t.Errorf("%s: status %d err %v, want %d", tc.sb.ID(), got, err, tc.want)
63+ }
64+ }
65+
66+ if err := a.Stop(context.Background()); err != nil {
67+ t.Fatal(err)
68+ }
69+ if c, err := net.Dial("unix", a.cacheSocket()); err == nil {
70+ _ = c.Close()
71+ t.Fatal("A's cache socket answers after Stop")
72+ }
73+ if got, err := sandboxtest.CacheLookup(b.cacheSocket(), tokB); err != nil || got != http.StatusNoContent {
74+ t.Fatalf("B after A's Stop: %d %v", got, err)
75+ }
76+ if err := b.Stop(context.Background()); err != nil {
77+ t.Fatal(err)
78+ }
79+ if left := e.leftovers(t); len(left) != 0 {
80+ t.Fatalf("leftovers: %v", left)
81+ }
82+}
83+
84+// A boot that fails after the channel was opened closes it.
85+func TestCacheChannelClosedWhenBootFails(t *testing.T) {
86+ e := newEnv(t, "hang") // the VMM never lets the executor connect
87+ svc := sandboxtest.NewCache(t)
88+ e.cfg.Cache = svc
89+ d, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: 300 * time.Millisecond})
90+ if err != nil {
91+ t.Fatal(err)
92+ }
93+ t.Cleanup(func() { _ = d.Close() })
94+ sb, _ := d.New()
95+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err == nil || !strings.Contains(err.Error(), "did not connect") {
96+ t.Fatalf("expected a boot timeout, got %v", err)
97+ }
98+ _ = sb.Stop(context.Background())
99+ ln, err := net.Listen("unix", t.TempDir()+"/s")
100+ if err != nil {
101+ t.Fatal(err)
102+ }
103+ if err := svc.Serve(sb.ID(), ln, "j"); err != nil {
104+ t.Fatalf("the channel of the failed boot is still registered: %v", err)
105+ }
106+}
107+
108+// With no cache service the VM has no cache socket.
109+func TestNoCacheSocketWithoutService(t *testing.T) {
110+ e := newEnv(t, "")
111+ d := e.driver(t)
112+ sb := startVM(t, d, "j")
113+ if _, err := os.Stat(sb.cacheSocket()); err == nil {
114+ t.Fatal("a cache socket exists with no cache service")
115+ }
116+ _ = sb.Stop(context.Background())
117+}
118+
119+// The socket path limit covers the cache socket too.
120+func TestSocketPathsIncludeTheCacheSocket(t *testing.T) {
121+ var found bool
122+ for _, p := range socketPaths("/var/lib/ermite", "firecracker", true) {
123+ found = found || strings.HasSuffix(p, "_9089")
124+ }
125+ if !found {
126+ t.Fatal("the cache socket is not checked against the sun_path limit")
127+ }
128+}
@@ -0,0 +1,128 @@
1+// SPDX-License-Identifier: MIT
2+
3+package vm
4+
5+import (
6+ "context"
7+ "net"
8+ "net/http"
9+ "os"
10+ "strings"
11+ "testing"
12+ "time"
13+
14+ "bnema.dev/ermite-runner/internal/sandbox"
15+ "bnema.dev/ermite-runner/internal/sandbox/sandboxtest"
16+)
17+
18+func startVM(t *testing.T, d *Driver, job string) *vmSandbox {
19+ t.Helper()
20+ sb, err := d.New()
21+ if err != nil {
22+ t.Fatal(err)
23+ }
24+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: job}); err != nil {
25+ t.Fatal(err)
26+ }
27+ return sb.(*vmSandbox)
28+}
29+
30+// Each VM serves the cache on its own host-side vsock socket, <uds>_<cache
31+// port> in the VM's own directory (the path Firecracker connects to when the
32+// guest connects to the host on the cache port), for its own job only. Stopping
33+// one VM closes its channel and leaves the other alone.
34+func TestCacheVsockSocketPerVM(t *testing.T) {
35+ e := newEnv(t, "")
36+ svc := sandboxtest.NewCache(t)
37+ e.cfg.Cache = svc
38+ d := e.driver(t)
39+ a, b := startVM(t, d, "job-a"), startVM(t, d, "job-b")
40+ tokA, tokB := sandboxtest.CacheToken(svc, "job-a"), sandboxtest.CacheToken(svc, "job-b")
41+
42+ if a.cacheSocket() == b.cacheSocket() {
43+ t.Fatal("two VMs share a cache socket")
44+ }
45+ for _, sb := range []*vmSandbox{a, b} {
46+ if !strings.HasSuffix(sb.cacheSocket(), "_9089") || !strings.HasPrefix(sb.cacheSocket(), sb.lay.root+"/") {
47+ t.Fatalf("cache socket %q is not <vm dir>/v.sock_9089", sb.cacheSocket())
48+ }
49+ }
50+ for _, tc := range []struct {
51+ sb *vmSandbox
52+ tok string
53+ want int
54+ }{
55+ {a, tokA, http.StatusNoContent},
56+ {b, tokB, http.StatusNoContent},
57+ {a, tokB, http.StatusForbidden},
58+ {b, tokA, http.StatusForbidden},
59+ } {
60+ got, err := sandboxtest.CacheLookup(tc.sb.cacheSocket(), tc.tok)
61+ if err != nil || got != tc.want {
62+ t.Errorf("%s: status %d err %v, want %d", tc.sb.ID(), got, err, tc.want)
63+ }
64+ }
65+
66+ if err := a.Stop(context.Background()); err != nil {
67+ t.Fatal(err)
68+ }
69+ if c, err := net.Dial("unix", a.cacheSocket()); err == nil {
70+ _ = c.Close()
71+ t.Fatal("A's cache socket answers after Stop")
72+ }
73+ if got, err := sandboxtest.CacheLookup(b.cacheSocket(), tokB); err != nil || got != http.StatusNoContent {
74+ t.Fatalf("B after A's Stop: %d %v", got, err)
75+ }
76+ if err := b.Stop(context.Background()); err != nil {
77+ t.Fatal(err)
78+ }
79+ if left := e.leftovers(t); len(left) != 0 {
80+ t.Fatalf("leftovers: %v", left)
81+ }
82+}
83+
84+// A boot that fails after the channel was opened closes it.
85+func TestCacheChannelClosedWhenBootFails(t *testing.T) {
86+ e := newEnv(t, "hang") // the VMM never lets the executor connect
87+ svc := sandboxtest.NewCache(t)
88+ e.cfg.Cache = svc
89+ d, err := New(context.Background(), e.cfg, Options{Run: e.run, Net: e.plumb, BootTimeout: 300 * time.Millisecond})
90+ if err != nil {
91+ t.Fatal(err)
92+ }
93+ t.Cleanup(func() { _ = d.Close() })
94+ sb, _ := d.New()
95+ if _, err := sb.Start(context.Background(), sandbox.JobSpec{JobID: "j"}); err == nil || !strings.Contains(err.Error(), "did not connect") {
96+ t.Fatalf("expected a boot timeout, got %v", err)
97+ }
98+ _ = sb.Stop(context.Background())
99+ ln, err := net.Listen("unix", t.TempDir()+"/s")
100+ if err != nil {
101+ t.Fatal(err)
102+ }
103+ if err := svc.Serve(sb.ID(), ln, "j"); err != nil {
104+ t.Fatalf("the channel of the failed boot is still registered: %v", err)
105+ }
106+}
107+
108+// With no cache service the VM has no cache socket.
109+func TestNoCacheSocketWithoutService(t *testing.T) {
110+ e := newEnv(t, "")
111+ d := e.driver(t)
112+ sb := startVM(t, d, "j")
113+ if _, err := os.Stat(sb.cacheSocket()); err == nil {
114+ t.Fatal("a cache socket exists with no cache service")
115+ }
116+ _ = sb.Stop(context.Background())
117+}
118+
119+// The socket path limit covers the cache socket too.
120+func TestSocketPathsIncludeTheCacheSocket(t *testing.T) {
121+ var found bool
122+ for _, p := range socketPaths("/var/lib/ermite", "firecracker", true) {
123+ found = found || strings.HasSuffix(p, "_9089")
124+ }
125+ if !found {
126+ t.Fatal("the cache socket is not checked against the sun_path limit")
127+ }
128+}
M internal/sandbox/vm/driver.go +36 -2
@@ -152,7 +152,6 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
152152 if d.net == nil {
153153 n := netsetup.New(opt.Run, cfg.DNSPort, "")
154154 n.DNS.Override = cfg.GuestForge.DNSName()
155- n.Cache, n.CachePort = cfg.Cache, cfg.CachePort
156155 d.net = n
157156 }
158157 if cfg.GuestForge != nil {
@@ -247,6 +246,7 @@ type vmSandbox struct {
247246 exited chan struct{}
248247 console *ring
249248 listener net.Listener
249+ cacheOn bool
250250 conn net.Conn
251251 tap string
252252 gateway netip.Addr
@@ -333,6 +333,9 @@ func (s *vmSandbox) boot(ctx context.Context, spec sandbox.JobSpec) (sandbox.Con
333333 if err != nil {
334334 return nil, err
335335 }
336+ if err := s.bootCache(spec); err != nil {
337+ return nil, err
338+ }
336339 if err := api.configure(ctx, s.vmConfig(spec, netw, files)); err != nil {
337340 return nil, err
338341 }
@@ -440,6 +443,33 @@ func (s *vmSandbox) bootVsock() (<-chan acceptResult, error) {
440443 return accepted, nil
441444 }
442445
446+// cacheSocket is the host side of the cache streams of the VM: Firecracker
447+// connects to <uds_path>_<CacheVsockPort> each time the guest relay connects to
448+// the host on that vsock port. The path is inside this VM's own directory (its
449+// chroot with the jailer), so no other VM reaches it.
450+func (s *vmSandbox) cacheSocket() string {
451+ return s.lay.host(udsName) + fmt.Sprintf("_%d", sandbox.CacheVsockPort)
452+}
453+
454+// bootCache serves the cache on the VM's cache socket, owned by the jailed uid
455+// as the executor's.
456+func (s *vmSandbox) bootCache(spec sandbox.JobSpec) error {
457+ if s.d.cfg.Cache == nil {
458+ return nil
459+ }
460+ err := s.d.cfg.ServeCache(s.id, s.cacheSocket(), spec.JobID, s.d.cfg.NetMbit, func(p string) error {
461+ if s.lay.jailed {
462+ return os.Lchown(p, s.lay.uid, s.lay.uid)
463+ }
464+ return nil
465+ })
466+ if err != nil {
467+ return fmt.Errorf("serve the cache socket: %w", err)
468+ }
469+ s.cacheOn = true
470+ return nil
471+}
472+
443473 const udsName = "v.sock"
444474
445475 // maxSockPath is the longest Unix socket path Linux accepts (108 with the NUL).
@@ -454,7 +484,7 @@ func socketPaths(stateDir, execName string, jailed bool) []string {
454484 if jailed {
455485 api = lay.host("run/firecracker.socket")
456486 }
457- return []string{api, lay.host(udsName) + fmt.Sprintf("_%d", sandbox.VsockPort)}
487+ return []string{api, lay.host(udsName) + fmt.Sprintf("_%d", sandbox.VsockPort), lay.host(udsName) + fmt.Sprintf("_%d", sandbox.CacheVsockPort)}
458488 }
459489
460490 // checkSocketPaths refuses a state directory that makes a VM socket path longer
@@ -707,6 +737,10 @@ func (s *vmSandbox) cleanupLocked(ctx context.Context) error {
707737 _ = s.listener.Close()
708738 s.listener = nil
709739 }
740+ if s.cacheOn {
741+ d.cfg.StopCache(s.id)
742+ s.cacheOn = false
743+ }
710744 dead := true
711745 // No kill(-pgid) of the launched command: once it has been waited for, its
712746 // pid (and so its group id) may belong to anyone. Every process of this VM is
@@ -152,7 +152,6 @@ func New(ctx context.Context, cfg sandbox.Config, opt Options) (*Driver, error)
152 if d.net == nil {152 if d.net == nil {
153 n := netsetup.New(opt.Run, cfg.DNSPort, "")153 n := netsetup.New(opt.Run, cfg.DNSPort, "")
154 n.DNS.Override = cfg.GuestForge.DNSName()154 n.DNS.Override = cfg.GuestForge.DNSName()
155- n.Cache, n.CachePort = cfg.Cache, cfg.CachePort
156 d.net = n155 d.net = n
157 }156 }
158 if cfg.GuestForge != nil {157 if cfg.GuestForge != nil {
@@ -247,6 +246,7 @@ type vmSandbox struct {
247 exited chan struct{}246 exited chan struct{}
248 console *ring247 console *ring
249 listener net.Listener248 listener net.Listener
249+ cacheOn bool
250 conn net.Conn250 conn net.Conn
251 tap string251 tap string
252 gateway netip.Addr252 gateway netip.Addr
@@ -333,6 +333,9 @@ func (s *vmSandbox) boot(ctx context.Context, spec sandbox.JobSpec) (sandbox.Con
333 if err != nil {333 if err != nil {
334 return nil, err334 return nil, err
335 }335 }
336+ if err := s.bootCache(spec); err != nil {
337+ return nil, err
338+ }
336 if err := api.configure(ctx, s.vmConfig(spec, netw, files)); err != nil {339 if err := api.configure(ctx, s.vmConfig(spec, netw, files)); err != nil {
337 return nil, err340 return nil, err
338 }341 }
@@ -440,6 +443,33 @@ func (s *vmSandbox) bootVsock() (<-chan acceptResult, error) {
440 return accepted, nil443 return accepted, nil
441 }444 }
442 445
446+// cacheSocket is the host side of the cache streams of the VM: Firecracker
447+// connects to <uds_path>_<CacheVsockPort> each time the guest relay connects to
448+// the host on that vsock port. The path is inside this VM's own directory (its
449+// chroot with the jailer), so no other VM reaches it.
450+func (s *vmSandbox) cacheSocket() string {
451+ return s.lay.host(udsName) + fmt.Sprintf("_%d", sandbox.CacheVsockPort)
452+}
453+
454+// bootCache serves the cache on the VM's cache socket, owned by the jailed uid
455+// as the executor's.
456+func (s *vmSandbox) bootCache(spec sandbox.JobSpec) error {
457+ if s.d.cfg.Cache == nil {
458+ return nil
459+ }
460+ err := s.d.cfg.ServeCache(s.id, s.cacheSocket(), spec.JobID, s.d.cfg.NetMbit, func(p string) error {
461+ if s.lay.jailed {
462+ return os.Lchown(p, s.lay.uid, s.lay.uid)
463+ }
464+ return nil
465+ })
466+ if err != nil {
467+ return fmt.Errorf("serve the cache socket: %w", err)
468+ }
469+ s.cacheOn = true
470+ return nil
471+}
472+
443 const udsName = "v.sock"473 const udsName = "v.sock"
444 474
445 // maxSockPath is the longest Unix socket path Linux accepts (108 with the NUL).475 // maxSockPath is the longest Unix socket path Linux accepts (108 with the NUL).
@@ -454,7 +484,7 @@ func socketPaths(stateDir, execName string, jailed bool) []string {
454 if jailed {484 if jailed {
455 api = lay.host("run/firecracker.socket")485 api = lay.host("run/firecracker.socket")
456 }486 }
457- return []string{api, lay.host(udsName) + fmt.Sprintf("_%d", sandbox.VsockPort)}487+ return []string{api, lay.host(udsName) + fmt.Sprintf("_%d", sandbox.VsockPort), lay.host(udsName) + fmt.Sprintf("_%d", sandbox.CacheVsockPort)}
458 }488 }
459 489
460 // checkSocketPaths refuses a state directory that makes a VM socket path longer490 // checkSocketPaths refuses a state directory that makes a VM socket path longer
@@ -707,6 +737,10 @@ func (s *vmSandbox) cleanupLocked(ctx context.Context) error {
707 _ = s.listener.Close()737 _ = s.listener.Close()
708 s.listener = nil738 s.listener = nil
709 }739 }
740+ if s.cacheOn {
741+ d.cfg.StopCache(s.id)
742+ s.cacheOn = false
743+ }
710 dead := true744 dead := true
711 // No kill(-pgid) of the launched command: once it has been waited for, its745 // No kill(-pgid) of the launched command: once it has been waited for, its
712 // pid (and so its group id) may belong to anyone. Every process of this VM is746 // pid (and so its group id) may belong to anyone. Every process of this VM is

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 4

Tags 11

↑↓ moveenter openesc close