1# Cache and artifacts
2
3The runner serves the GitHub Actions cache, artifacts v4 and a Docker Hub mirror to jobs. All three use one channel per sandbox.
4
5## Channel
6
7- The services are reachable only from inside the sandbox.
8- They listen on the sandbox loopback (`127.0.0.1:9089`) and on the address of its own Docker bridge (`docker0`, typically `172.17.0.1:9089`).
9- Nothing listens on a host network or port. The host firewall needs no rule.
10- A relay in the sandbox pipes each connection to the host over a channel that belongs to that sandbox:
11  - `vm`: vsock port 9089.
12  - `container`: `cache.sock` in the per-job directory.
13- A sandbox cannot reach another sandbox's channel.
14- Limits: 32 connections per sandbox, 128 in all. A transfer with no progress for 60 s is cut.
15
16### Variables
17
18- Node actions get `ACTIONS_CACHE_URL`, `ACTIONS_RUNTIME_URL`, `ACTIONS_RUNTIME_TOKEN`, `ACTIONS_CACHE_MODE` and `ACTIONS_RESULTS_URL`. `run:` steps get none, as on the official runner.
19- In job containers and docker actions the same variables point to the Docker bridge address.
20- The token is masked.
21- If the relay cannot start, the job runs without a cache.
22
23## Cache (API v1)
24
25Works with `actions/cache`, `setup-go` and `setup-node` with `cache:`, and anything built on `@actions/cache`. The toolkit uses cache API v1 for any server that is not `github.com`, `*.ghe.com` or `*.localhost`. `ACTIONS_CACHE_SERVICE_V2` is never set.
26
27| Setting | Value |
28| --- | --- |
29| Storage | `<state dir>/cache`, mode 0700/0600, written atomically |
30| Size cap | `ERMITE_RUNNER_CACHE_MAX`, default `10G`. Least recently used entries are evicted |
31| Off | `ERMITE_RUNNER_CACHE=0` |
32| Restart | Entries are kept. Unfinished uploads are dropped |
33
34- **Scopes.** Entries belong to (repository, ref, key, version). A job writes to its write ref and reads its read refs in priority order. Another repository's entries are never visible.
35- **Server scope.** The server sends the repository, write ref and read refs (feature `ci.cache`). The runner checks them: the repository must be the job's own, refs are full `refs/...` names, at most 8 read refs. An empty write ref makes the job read-only. A scope that fails the checks disables the cache for that job.
36- **Old servers.** Without a scope from the server, the scope is `github.repository` and the job ref, for reading and writing.
37- **Duplicates.** A second reservation of the same entry while one exists is a `409`.
38- **Reservations.** Dropped after an hour without activity, or at the expiry of the token that made them.
39- **Limits.** Each entry is charged at least 4 KiB. At most 256 uploads in progress (32 per job). An entry being downloaded is never evicted. A reservation that cannot get room answers `503`.
40- **Bandwidth.** A VM's cache traffic does not cross `eth0`. The runner limits it per sandbox at `ERMITE_RUNNER_JOB_NET_BANDWIDTH`. Containers have no limit.
41
42### Tokens
43
44- `ACTIONS_RUNTIME_TOKEN` is a JWT (HS256) bound to the job, repository, refs and an expiry (job timeout plus 15 min).
45- The key is random, made at start and kept in memory only.
46- Any invalid token gets a bare `403`.
47- The token is not an ermite credential. The server refuses it.
48- Archive URLs carry their own HMAC and are served only to the job they were issued to.
49
50### Action archives
51
52- The host downloads an action archive once into `<state dir>/actions-cache/`, keyed by forge, repository and commit sha.
53- The sandbox gets a signed URL on the cache channel and extracts the archive.
54- The SHA-256 is checked on every read. A corrupt entry is dropped and fetched again.
55- Size cap: `ERMITE_RUNNER_ACTIONS_CACHE_MAX`, default `2G`. `0` turns it off.
56- The sandbox never writes to this cache. Anything that cannot be cached is downloaded by the sandbox.
57
58## Artifacts (v4)
59
60`actions/upload-artifact@v4` and later, `actions/download-artifact@v4` and later, and anything bundling `@actions/artifact` v2 work when the server has the feature `ci.artifacts`.
61
62- **ermite stores the artifacts.** The runner keeps nothing on disk.
63- The runner serves GitHub's results service (Twirp JSON plus Azure block blob requests on signed URLs) and translates it to the server with the job token.
64- The job token never enters the sandbox.
65- A job reaches only the artifacts of its own run.
66- Quota and size limits answer `412` (`413` on a block) and are not retried. Server unavailable answers `503`, which the toolkit retries.
67- Off: `ERMITE_RUNNER_ARTIFACTS=0` or a server without `ci.artifacts`. Artifacts need the cache service, so `ERMITE_RUNNER_CACHE=0` turns them off too.
68
69### `GITHUB_SERVER_URL` in artifact steps
70
71- `@actions/artifact` refuses to run on GitHub Enterprise Server, which it detects only from `GITHUB_SERVER_URL`.
72- In node steps of `actions/upload-artifact` and `actions/download-artifact` (any subpath), and of actions that contain the toolkit's GHES message, `GITHUB_SERVER_URL` is `https://ermite.ghe.localhost`.
73- Nothing resolves or connects to that host.
74- The `github` context, `${{ github.server_url }}` and all other steps keep the real URL.
75
76### Not covered
77
78- `upload-artifact` below v4.3.1 and `download-artifact` below v4.1.2.
79- Artifacts v3.
80- The `artifact-url` output. It points at the fake host. The run page in ermite lists the artifacts.
81- Downloads from another run (`run-id` with `github-token`).
82
83## Docker Hub images
84
85The runner provides an anonymous, pull-only Docker Hub mirror on the same private
86channel. Each job's dockerd uses `http://127.0.0.1:9089` as a registry mirror.
87Other registries are not cached. There is no HTTPS interception, and the mirror
88needs no network policy rule. Guest credentials are never forwarded upstream. Private pulls
89fall back to Docker Hub directly, where dockerd can use the job's credentials.
90
91Verified sha256 blobs and manifests live under `<state dir>/registry/`. Tags are
92revalidated by an upstream HEAD on each request, with the client's Accept headers
93reduced to the known manifest media types. If Docker Hub is unreachable, times
94out, returns a server error or rate-limits (429), a previously cached tag can be
95served stale while its manifest remains stored. Unknown tags return an error so
96dockerd can fall back.
97
98A cold blob is downloaded once, whoever asks for it, and is not tied to any one
99client: every client waits for the digest check and then reads the stored file.
100A corrupt download returns 502 and leaves no file. A client that disconnects
101does not stop the download while another client waits for it; once no client
102has waited for 5 s, the download is cancelled and its temporary file removed.
103Each job may have started at most 3 cold blob downloads at once (dockerd's
104default concurrency), and at most 16 run at once across all jobs. A download
105counts against the job that started it until it ends; a job that asks for a
106layer another job is already downloading waits for it and uses none of its
107quota. A request over either limit waits up to 30 s for a slot, its connection
108kept alive, and then gets 503. A refused layer makes dockerd retry for about
10950 s before it pulls that layer from Docker Hub directly. Each sandbox may also
110have at most 8 registry requests in progress; one more gets 503 at once.
111
112Downloads have no total time limit, but they fail after 30 s without progress.
113Auth, manifest and HEAD requests are limited to 10 s. Auth and manifest requests
114never follow redirects; blob requests, HEAD included, follow the CDN redirect
115without the bearer token. Stored blobs support range requests. Cold blob HEAD
116requests fetch only upstream metadata, not the layer body.
117
118`ERMITE_RUNNER_REGISTRY_CACHE_MAX` defaults to `10G`. Blobs, manifests and tag
119files share one least-recently-used eviction; each is charged at least 4 KiB, so
120the number of files stays below the cap divided by 4 KiB. An entry that is being served is
121never evicted. This cap is independent of
122`ERMITE_RUNNER_CACHE_MAX`. A zero size in any unit (`0`, `0M`, `0G`) disables only
123the proxy. `ERMITE_RUNNER_CACHE=0` disables the whole channel and therefore also
124the proxy (logged at startup). Dockerd retains its mirror setting and falls back
125to Docker Hub if the mirror is unavailable. To clear the cache, stop the runner
126and remove `<state dir>/registry/`.
127
128Jobs with neither cache permissions nor artifacts get the loopback channel for
129dockerd when the proxy is on, but no `ACTIONS_CACHE_URL`, `ACTIONS_RESULTS_URL`, or
130`ACTIONS_RUNTIME_TOKEN` is exposed to their steps. Registry routes need no token:
131only public content is served over a channel exclusive to that sandbox.
132
133The host logs one `registry manifest` line per request with `job_id`, `repo`,
134`ref`, `hit` and `ms`. Blob misses log `registry blob` at debug level with
135`job_id`, `digest`, `bytes` and `ms`; `job_id` is the job that started the
136download.