fix(externals): build stubs once via rename, record alpine checksums, retry on a timer

- unavailable-tool directories are built once, complete, and renamed into place;
  files are never rewritten in place (readers never see a half-written stub)
- pruning keeps stub directories handed out in the last hour
- bin/node stub reads the reason from UNAVAILABLE instead of embedding it
- downloads that failed for a reason that may pass are retried by a timer
- pin the actions/alpine_nodejs checksums; their archives have no top directory,
  so stripping the first path component is now a per-artifact setting

commit: ea3c7aa2a5962b9bd024d07676cbe162cac74bdc
author: brice · 9d ago · 2026-10-13 10:22 UTC
parent: ad47140
signature: GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email

5 files · +598 -137

M README.md +3 -2


@@ -46,8 +46,9 @@ The host provides Node.js to the jobs outside the image, like its own execut
46	46		- Extraction keeps the top directory out and refuses absolute names, `..`, devices and fifos, symbolic links that are absolute or climb out of the tree, entries below a link, hard links to anything but an unpacked file, and archives over 200,000 entries or 2 GiB; writes go through `os.Root`. The tree is built in a temporary directory and renamed into place, so a directory under its final name is always complete.
47	47		- Only pinned versions are kept: at every start everything else in `externals/` (other versions, other architectures, interrupted downloads) is removed.
48	48		- Layout in the sandbox (GitHub's): `/home/runner/externals/node20/bin/node`, `/node24/bin/node`, and the alpine ones for job containers, read-only. In `container` mode they are read-only bind mounts of the host directories; in `vm` mode one ext4 disk (`mkfs.ext4 -d`) named after the content of the set (`cache/extdisk-<key>.ext4`), so a new Node.js builds a new disk and leaves the executable's disk and the base image alone. Superseded disks are removed an hour after the last build. The set is read at every job start: a download that finishes later is used by the next job.
49		-	- When a build cannot be provided (offline, HTTP error, wrong checksum, unsafe archive, no build for the architecture) the runner still runs: it logs why, retries network failures every 5 minutes in the background, and attaches in its place a directory with the file `UNAVAILABLE` and a `bin/node` script that prints the reason on stderr and exits 1, so a job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: download …: SHA-256 mismatch …` rather than `No such file or directory`.
50		-	- The alpine builds are not available yet. They are hosted on github.com (`actions/alpine_nodejs`) and their SHA-256 is not recorded in `table.go` yet: nothing is downloaded for them, and a job container that needs `node20_alpine` or `node24_alpine` fails with `… is not available yet on this runner: the checksum of the actions/alpine_nodejs build has not been recorded in this runner yet`.
	49	+	- When a build cannot be provided (offline, HTTP error, wrong checksum, unsafe archive, no build for the architecture) the runner still runs: it logs why and attaches in its place a directory with the file `UNAVAILABLE` (the reason) and a `bin/node` script, `cat "$(dirname "$0")/../UNAVAILABLE" >&2; exit 1`, so a job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: download …: SHA-256 mismatch …` rather than `No such file or directory`. These directories (`externals/.unavailable/<name>-<reason hash>`) are built once, complete, and renamed into place; a file is never rewritten in place, so a reader (the build of a VM's externals disk, a job running the stub) never sees a half-written one. A directory handed out to a sandbox in the last hour is kept by the pruning, since a running container may have it bind-mounted.
	50	+	- Retries. A download that failed for a reason that may pass (network error, HTTP status) is retried every 5 minutes by a timer, whether jobs come or not, until the runner stops. A verdict (wrong checksum, unsafe archive, no build for the architecture) is not retried before the next start.
	51	+	- Archive layouts. The nodejs.org archives have one top directory (`node-vX-linux-x64/`), which is dropped; the `actions/alpine_nodejs` archives start at `./bin/node` and are unpacked as they are. The alpine checksums are the digests of the GitHub release assets.
51	52		- `ERMITE_RUNNER_NODE=0` turns all of this off: nothing is downloaded or attached.
52	53
53	54		### One runner per host

 - **Extraction** keeps the top directory out and refuses absolute names, `..`, devices and fifos, symbolic links that are absolute or climb out of the tree, entries below a link, hard links to anything but an unpacked file, and archives over 200,000 entries or 2 GiB; writes go through `os.Root`. The tree is built in a temporary directory and renamed into place, so a directory under its final name is always complete.
 - **Only pinned versions are kept**: at every start everything else in `externals/` (other versions, other architectures, interrupted downloads) is removed.
 - **Layout in the sandbox** (GitHub's): `/home/runner/externals/node20/bin/node`, `/node24/bin/node`, and the alpine ones for job containers, read-only. In `container` mode they are read-only bind mounts of the host directories; in `vm` mode one ext4 disk (`mkfs.ext4 -d`) named after the content of the set (`cache/extdisk-<key>.ext4`), so a new Node.js builds a new disk and leaves the executable's disk and the base image alone. Superseded disks are removed an hour after the last build. The set is read at every job start: a download that finishes later is used by the next job.
-- **When a build cannot be provided** (offline, HTTP error, wrong checksum, unsafe archive, no build for the architecture) the runner **still runs**: it logs why, retries network failures every 5 minutes in the background, and attaches in its place a directory with the file `UNAVAILABLE` and a `bin/node` script that prints the reason on stderr and exits 1, so a job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: download …: SHA-256 mismatch …` rather than `No such file or directory`.
+- **When a build cannot be provided** (offline, HTTP error, wrong checksum, unsafe archive, no build for the architecture) the runner **still runs**: it logs why and attaches in its place a directory with the file `UNAVAILABLE` (the reason) and a `bin/node` script, `cat "$(dirname "$0")/../UNAVAILABLE" >&2; exit 1`, so a job that needs the tool fails with a message such as `node24 (Node.js 24.21.0) is not available on this runner: download …: SHA-256 mismatch …` rather than `No such file or directory`. These directories (`externals/.unavailable/<name>-<reason hash>`) are built once, complete, and renamed into place; a file is never rewritten in place, so a reader (the build of a VM's externals disk, a job running the stub) never sees a half-written one. A directory handed out to a sandbox in the last hour is kept by the pruning, since a running container may have it bind-mounted.
-- **The alpine builds are not available yet.** They are hosted on github.com (`actions/alpine_nodejs`) and their SHA-256 is not recorded in `table.go` yet: nothing is downloaded for them, and a job container that needs `node20_alpine` or `node24_alpine` fails with `… is not available yet on this runner: the checksum of the actions/alpine_nodejs build has not been recorded in this runner yet`.
+- **Retries.** A download that failed for a reason that may pass (network error, HTTP status) is retried every 5 minutes by a timer, whether jobs come or not, until the runner stops. A verdict (wrong checksum, unsafe archive, no build for the architecture) is not retried before the next start.
+- **Archive layouts.** The nodejs.org archives have one top directory (`node-vX-linux-x64/`), which is dropped; the `actions/alpine_nodejs` archives start at `./bin/node` and are unpacked as they are. The alpine checksums are the digests of the GitHub release assets.
 - `ERMITE_RUNNER_NODE=0` turns all of this off: nothing is downloaded or attached.
 ### One runner per host

M internal/externals/externals_test.go +361 -44


@@ -9,18 +9,23 @@ import (
9	9		"context"
10	10		"crypto/sha256"
11	11		"encoding/hex"
	12	+	"fmt"
12	13		"net/http"
13	14		"net/http/httptest"
14	15		"net/url"
15	16		"os"
	17	+	"os/exec"
16	18		"path/filepath"
17	19		"regexp"
18	20		"strings"
	21	+	"sync"
19	22		"sync/atomic"
20	23		"syscall"
21	24		"testing"
22	25		"time"
23	26
	27	+	"github.com/bnema/zerowrap"
	28	+
24	29		"bnema.dev/ermite-runner/internal/sandbox"
25	30		)
26	31

@@ -38,16 +43,12 @@ func TestPinnedTable(t *testing.T) {
38	43		if !strings.HasPrefix(a.URL, "https://") \|\| !strings.HasSuffix(a.URL, ".tar.gz") \|\| !strings.Contains(a.URL, "v"+a.Version+"/") {
39	44		t.Errorf("%s: URL %q", key, a.URL)
40	45		}
41		-	if a.SHA256 == "" {
42		-	if a.Pending == "" {
43		-	t.Errorf("%s has neither a checksum nor a reason", key)
44		-	}
45		-	if !strings.HasPrefix(a.URL, alpineDist) {
46		-	t.Errorf("%s: only the alpine builds may lack a checksum", key)
47		-	}
48		-	} else if !hex64.MatchString(a.SHA256) {
	46	+	if !hex64.MatchString(a.SHA256) {
49	47		t.Errorf("%s: SHA-256 %q", key, a.SHA256)
50	48		}
	49	+	if a.StripTop != strings.HasPrefix(a.URL, nodeDist) {
	50	+	t.Errorf("%s: StripTop %v for %s", key, a.StripTop, a.URL)
	51	+	}
51	52		}
52	53		// actions/runner's externals.sh: node20 and node24 for both architectures,
53	54		// the alpine builds for x64.

@@ -61,6 +62,11 @@ func TestPinnedTable(t *testing.T) {
61	62		t.Errorf("%s pins %s", a.Name, a.Version)
62	63		}
63	64		}
	65	+	for _, a := range Pinned {
	66	+	if strings.HasSuffix(a.Name, "_alpine") && (a.Arch != "x64" \|\| !strings.HasPrefix(a.URL, alpineDist+"v"+a.Version+"/node-v"+a.Version+"-alpine-x64.tar.gz")) {
	67	+	t.Errorf("%s: %+v", a.Name, a)
	68	+	}
	69	+	}
64	70		if got := len(ForArch(Pinned, "arm64")); got != 2 {
65	71		t.Errorf("arm64 has %d artifacts", got)
66	72		}

@@ -138,7 +144,7 @@ func extractBytes(t *testing.T, data []byte) (string, error) {
138	144		if err := os.Mkdir(dst, 0o755); err != nil {
139	145		t.Fatal(err)
140	146		}
141		-	return dst, extractTarGz(arch, dst)
	147	+	return dst, extractTarGz(arch, dst, true)
142	148		}
143	149
144	150		func TestExtractLayoutAndModes(t *testing.T) {

@@ -260,14 +266,14 @@ func TestLinkTargetRule(t *testing.T) {
260	266		name, target string
261	267		ok bool
262	268		}{
263		-	{"t/bin/npm", "../lib/x", true},
264		-	{"t/bin/npm", "../../x", false}, // one above the tree
265		-	{"t/lib/a/b/l", "../../../x", true},
266		-	{"t/lib/a/b/l", "../../../../x", false},
267		-	{"t/l", "bin/node", true},
268		-	{"t/l", "../x", false},
269		-	{"t/l", "/abs", false},
270		-	{"t/a/l", "b/../c", false},
	269	+	{"bin/npm", "../lib/x", true},
	270	+	{"bin/npm", "../../x", false}, // one above the tree
	271	+	{"lib/a/b/l", "../../../x", true},
	272	+	{"lib/a/b/l", "../../../../x", false},
	273	+	{"l", "bin/node", true},
	274	+	{"l", "../x", false},
	275	+	{"l", "/abs", false},
	276	+	{"a/l", "b/../c", false},
271	277		} {
272	278		if err := checkLinkTarget(c.name, c.target); (err == nil) != c.ok {
273	279		t.Errorf("%s -> %s: %v", c.name, c.target, err)

@@ -278,13 +284,15 @@ func TestLinkTargetRule(t *testing.T) {
278	284		// ---- the manager
279	285
280	286		type fixture struct {
281		-	srv *httptest.Server
282		-	hits atomic.Int64
283		-	archive []byte
284		-	sum string
285		-	table []Artifact
286		-	dir string
287		-	m *Manager
	287	+	srv *httptest.Server
	288	+	hits atomic.Int64
	289	+	archive []byte
	290	+	sum string
	291	+	archiveAlpine []byte // no top directory, like actions/alpine_nodejs
	292	+	sumAlpine string
	293	+	table []Artifact
	294	+	dir string
	295	+	m *Manager
288	296		}
289	297
290	298		func newFixture(t testing.T) fixture {

@@ -293,9 +301,17 @@ func newFixture(t testing.T) fixture {
293	301		f.archive = targz(t, nodeTree()...)
294	302		s := sha256.Sum256(f.archive)
295	303		f.sum = hex.EncodeToString(s[:])
	304	+	f.archiveAlpine = targz(t,
	305	+	ent{name: "./bin/", typ: tar.TypeDir, mode: 0o755},
	306	+	ent{name: "./bin/node", mode: 0o755, body: "#!/bin/sh\necho alpine\n"},
	307	+	ent{name: "./LICENSE", mode: 0o644, body: "x"})
	308	+	sa := sha256.Sum256(f.archiveAlpine)
	309	+	f.sumAlpine = hex.EncodeToString(sa[:])
296	310		f.srv = httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
297	311		f.hits.Add(1)
298	312		switch {
	313	+	case strings.HasSuffix(r.URL.Path, "/alpine.tar.gz"):
	314	+	_, _ = w.Write(f.archiveAlpine)
299	315		case strings.HasSuffix(r.URL.Path, "-linux-x64.tar.gz"):
300	316		_, _ = w.Write(f.archive)
301	317		case strings.HasSuffix(r.URL.Path, "tampered.tar.gz"):

@@ -307,9 +323,9 @@ func newFixture(t testing.T) fixture {
307	323		t.Cleanup(f.srv.Close)
308	324		f.dir = filepath.Join(t.TempDir(), "externals")
309	325		f.table = []Artifact{
310		-	{Name: "node20", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/node-v1.0.0-linux-x64.tar.gz", SHA256: f.sum},
311		-	{Name: "node24", Version: "2.0.0", Arch: "x64", URL: f.srv.URL + "/v2.0.0/node-v2.0.0-linux-x64.tar.gz", SHA256: f.sum},
312		-	{Name: "node20_alpine", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/alpine.tar.gz", Pending: "no checksum yet"},
	326	+	{Name: "node20", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/node-v1.0.0-linux-x64.tar.gz", SHA256: f.sum, StripTop: true},
	327	+	{Name: "node24", Version: "2.0.0", Arch: "x64", URL: f.srv.URL + "/v2.0.0/node-v2.0.0-linux-x64.tar.gz", SHA256: f.sum, StripTop: true},
	328	+	{Name: "node20_alpine", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/alpine.tar.gz", SHA256: f.sumAlpine},
313	329		}
314	330		f.m = &Manager{Dir: f.dir, Arch: "x64", Table: f.table, Client: f.srv.Client()}
315	331		return f

@@ -339,20 +355,15 @@ func TestSyncDownloadsVerifiesAndReuses(t *testing.T) {
339	355		if b, err := os.ReadFile(filepath.Join(n24.Dir, "bin", "node")); err != nil \|\| !strings.Contains(string(b), "echo v1") {
340	356		t.Fatalf("node24: %q %v", b, err)
341	357		}
342		-	// The alpine build has no checksum: not downloaded, with the reason.
343		-	if !al.Unavailable {
	358	+	// The alpine archive has no top directory: bin/node is at its root.
	359	+	if al.Unavailable {
344	360		t.Fatalf("alpine: %+v", al)
345	361		}
346		-	reason, err := os.ReadFile(filepath.Join(al.Dir, sandbox.ExternalUnavailableFile))
347		-	if err != nil \|\| !strings.Contains(string(reason), "not available yet") \|\| !strings.Contains(string(reason), "no checksum yet") {
348		-	t.Fatalf("reason: %q %v", reason, err)
349		-	}
350		-	stub, err := os.ReadFile(filepath.Join(al.Dir, "bin", "node"))
351		-	if err != nil \|\| !strings.Contains(string(stub), "not available yet") {
352		-	t.Fatalf("stub: %q %v", stub, err)
	362	+	if b, err := os.ReadFile(filepath.Join(al.Dir, "bin", "node")); err != nil \|\| !strings.Contains(string(b), "echo alpine") {
	363	+	t.Fatalf("alpine node: %q %v", b, err)
353	364		}
354	365		hits := f.hits.Load()
355		-	if hits != 2 {
	366	+	if hits != 3 {
356	367		t.Fatalf("downloads: %d", hits)
357	368		}
358	369

@@ -369,6 +380,29 @@ func TestSyncDownloadsVerifiesAndReuses(t *testing.T) {
369	380		}
370	381		}
371	382
	383	+	// runStub runs the bin/node of an unavailable tool: it must fail with the reason
	384	+	// on stderr, from wherever the tool is mounted.
	385	+	func runStub(t *testing.T, e sandbox.External, want string) {
	386	+	t.Helper()
	387	+	if _, err := exec.LookPath("sh"); err != nil {
	388	+	t.Skip("sh needed")
	389	+	}
	390	+	// Run it through a symlinked path too, as a guest sees it at another place.
	391	+	cmd := exec.Command(filepath.Join(e.Dir, "bin", "node"), "-e", "x")
	392	+	var stderr, stdout bytes.Buffer
	393	+	cmd.Stderr, cmd.Stdout = &stderr, &stdout
	394	+	err := cmd.Run()
	395	+	if ee, ok := err.(*exec.ExitError); !ok \|\| ee.ExitCode() != 1 {
	396	+	t.Fatalf("the stub must exit 1, got %v", err)
	397	+	}
	398	+	if !strings.Contains(stderr.String(), want) \|\| stdout.Len() != 0 {
	399	+	t.Fatalf("stub stderr %q stdout %q, want %q", stderr.String(), stdout.String(), want)
	400	+	}
	401	+	if b, _ := os.ReadFile(filepath.Join(e.Dir, "bin", "node")); strings.Contains(string(b), want) {
	402	+	t.Fatalf("the stub holds the reason: %q", b)
	403	+	}
	404	+	}
	405	+
372	406		func TestSyncRefusesAChecksumMismatchBeforeExtracting(t *testing.T) {
373	407		f := newFixture(t)
374	408		f.table[0].SHA256 = strings.Repeat("0", 64)

@@ -396,6 +430,7 @@ func TestSyncRefusesAChecksumMismatchBeforeExtracting(t *testing.T) {
396	430		if f.m.state["node20"].retry {
397	431		t.Fatal("retry scheduled for a mismatch")
398	432		}
	433	+	runStub(t, n20, "SHA-256 mismatch")
399	434		ents, _ := os.ReadDir(f.dir)
400	435		for _, e := range ents {
401	436		if strings.HasPrefix(e.Name(), ".tmp-") {

@@ -432,12 +467,9 @@ func TestSyncOfflineKeepsTheRunnerGoing(t *testing.T) {
432	467		if !e.Unavailable {
433	468		t.Fatalf("%s available offline", n)
434	469		}
435		-	stub, _ := os.ReadFile(filepath.Join(e.Dir, "bin", "node"))
436		-	if !strings.Contains(string(stub), "is not available on this runner") \|\| !strings.HasPrefix(string(stub), "#!/bin/sh") {
437		-	t.Fatalf("stub: %q", stub)
438		-	}
	470	+	runStub(t, e, "is not available on this runner")
439	471		}
440		-	if !f.m.state["node20"].retry {
	472	+	if !f.m.wantsRetry() {
441	473		t.Fatal("a network failure must be retried")
442	474		}
443	475		}

@@ -504,7 +536,7 @@ func TestSyncKeepsOnlyPinnedVersions(t *testing.T) {
504	536		for _, e := range ents {
505	537		got = append(got, e.Name())
506	538		}
507		-	want := ".unavailable node20-1.0.0-x64 node24-2.0.0-x64"
	539	+	want := "node20-1.0.0-x64 node20_alpine-1.0.0-x64 node24-2.0.0-x64"
508	540		if strings.Join(got, " ") != want {
509	541		t.Fatalf("kept %v, want %s", got, want)
510	542		}

@@ -587,7 +619,7 @@ func TestFailedDownloadIsRetriedInTheBackground(t *testing.T) {
587	619		}))
588	620		defer srv.Close()
589	621		m := &Manager{Dir: f.dir, Arch: "x64", Client: srv.Client(), RetryEvery: time.Millisecond}
590		-	m.Table = []Artifact{{Name: "node24", Version: "2.0.0", Arch: "x64", URL: srv.URL + "/a-linux-x64.tar.gz", SHA256: f.sum}}
	622	+	m.Table = []Artifact{{Name: "node24", Version: "2.0.0", Arch: "x64", URL: srv.URL + "/a-linux-x64.tar.gz", SHA256: f.sum, StripTop: true}}
591	623		ctx, cancel := context.WithCancel(context.Background())
592	624		defer cancel()
593	625		m.Start(ctx, time.Minute)

@@ -630,3 +662,288 @@ func TestStartDoesNotWaitLongerThanAsked(t *testing.T) {
630	662		}
631	663		t.Fatal("the background sync did not stop")
632	664		}
	665	+
	666	+	// ---- the directories of unavailable tools
	667	+
	668	+	func TestStubIsBuiltOnceAndNeverRewrittenInPlace(t *testing.T) {
	669	+	m := &Manager{Dir: filepath.Join(t.TempDir(), "e"), Arch: "x64"}
	670	+	e := m.unavailable("node24", "why", false)
	671	+	node := filepath.Join(e.ext.Dir, "bin", "node")
	672	+	file := filepath.Join(e.ext.Dir, sandbox.ExternalUnavailableFile)
	673	+	ino := func(p string) uint64 {
	674	+	st, err := os.Stat(p)
	675	+	if err != nil {
	676	+	t.Fatal(err)
	677	+	}
	678	+	return st.Sys().(*syscall.Stat_t).Ino
	679	+	}
	680	+	n0, f0 := ino(node), ino(file)
	681	+
	682	+	// Again, from this process and from a fresh one (a restart): same files.
	683	+	m.unavailable("node24", "why", false)
	684	+	(&Manager{Dir: m.Dir, Arch: "x64"}).unavailable("node24", "why", false)
	685	+	if ino(node) != n0 \|\| ino(file) != f0 {
	686	+	t.Fatal("an existing, correct stub was rewritten")
	687	+	}
	688	+
	689	+	// A damaged or missing file is replaced by a rename (a new inode, so a reader
	690	+	// of the old one never sees it change), with the right mode.
	691	+	if err := os.WriteFile(node, nil, 0o755); err != nil {
	692	+	t.Fatal(err)
	693	+	}
	694	+	if err := os.Remove(file); err != nil {
	695	+	t.Fatal(err)
	696	+	}
	697	+	(&Manager{Dir: m.Dir, Arch: "x64"}).unavailable("node24", "why", false)
	698	+	if b, _ := os.ReadFile(node); string(b) != unavailableStub {
	699	+	t.Fatalf("stub not repaired: %q", b)
	700	+	}
	701	+	if b, _ := os.ReadFile(file); string(b) != "why\n" {
	702	+	t.Fatalf("reason not repaired: %q", b)
	703	+	}
	704	+	if st, _ := os.Stat(node); st.Mode().Perm() != 0o755 \|\| ino(node) == n0 {
	705	+	t.Fatalf("repair in place: %v", st.Mode())
	706	+	}
	707	+	runStub(t, e.ext, "why")
	708	+	left, _ := filepath.Glob(filepath.Join(e.ext.Dir, "", ".tmp-"))
	709	+	left2, _ := filepath.Glob(filepath.Join(e.ext.Dir, ".tmp-*"))
	710	+	if len(left)+len(left2) != 0 {
	711	+	t.Fatalf("temporary files left: %v %v", left, left2)
	712	+	}
	713	+	}
	714	+
	715	+	// While Externals() and Sync() run in several goroutines, a reader that walks
	716	+	// the directories of the unavailable tools (as the build of a VM's externals
	717	+	// disk does) never sees a missing, empty or partial file.
	718	+	func TestConcurrentExternalsAndSyncNeverShowAHalfWrittenStub(t *testing.T) {
	719	+	f := newFixture(t)
	720	+	f.srv.Close() // offline: every tool is unavailable
	721	+	ctx := context.Background()
	722	+	stop := make(chan struct{})
	723	+	var wg sync.WaitGroup
	724	+	var bad atomic.Value
	725	+	check := func(dir string) {
	726	+	for _, c := range []struct{ name, want string }{{"bin/node", unavailableStub}} {
	727	+	b, err := os.ReadFile(filepath.Join(dir, c.name))
	728	+	if err != nil {
	729	+	bad.Store(fmt.Sprintf("%s/%s: %v", dir, c.name, err))
	730	+	} else if string(b) != c.want {
	731	+	bad.Store(fmt.Sprintf("%s/%s: %q", dir, c.name, b))
	732	+	}
	733	+	}
	734	+	b, err := os.ReadFile(filepath.Join(dir, sandbox.ExternalUnavailableFile))
	735	+	if err != nil \|\| len(b) == 0 \|\| !strings.HasSuffix(string(b), "\n") {
	736	+	bad.Store(fmt.Sprintf("%s/UNAVAILABLE: %q %v", dir, b, err))
	737	+	}
	738	+	}
	739	+	for i := 0; i < 4; i++ {
	740	+	wg.Add(1)
	741	+	go func() {
	742	+	defer wg.Done()
	743	+	for {
	744	+	select {
	745	+	case <-stop:
	746	+	return
	747	+	default:
	748	+	}
	749	+	for _, e := range f.m.Externals() {
	750	+	check(e.Dir) // what a driver would hand to the disk build
	751	+	}
	752	+	}
	753	+	}()
	754	+	}
	755	+	wg.Add(1)
	756	+	go func() {
	757	+	defer wg.Done()
	758	+	for {
	759	+	select {
	760	+	case <-stop:
	761	+	return
	762	+	default:
	763	+	}
	764	+	dirs, _ := filepath.Glob(filepath.Join(f.dir, unavailableDir, "*"))
	765	+	for _, d := range dirs {
	766	+	if !strings.Contains(filepath.Base(d), ".tmp-") {
	767	+	check(d)
	768	+	}
	769	+	}
	770	+	}
	771	+	}()
	772	+	for i := 0; i < 5; i++ {
	773	+	_ = f.m.Sync(ctx)
	774	+	}
	775	+	close(stop)
	776	+	wg.Wait()
	777	+	if v := bad.Load(); v != nil {
	778	+	t.Fatalf("a reader saw an incomplete stub: %v", v)
	779	+	}
	780	+	}
	781	+
	782	+	func TestPruneKeepsStubDirectoriesHandedOutInTheLastHour(t *testing.T) {
	783	+	f := newFixture(t)
	784	+	f.srv.Close()
	785	+	_ = f.m.Sync(context.Background()) // offline: stubs for the three tools
	786	+	root := filepath.Join(f.dir, unavailableDir)
	787	+	mk := func(name string, age time.Duration) string {
	788	+	p := filepath.Join(root, name)
	789	+	if err := os.MkdirAll(filepath.Join(p, "bin"), 0o755); err != nil {
	790	+	t.Fatal(err)
	791	+	}
	792	+	old := time.Now().Add(-age)
	793	+	if err := os.Chtimes(p, old, old); err != nil {
	794	+	t.Fatal(err)
	795	+	}
	796	+	return p
	797	+	}
	798	+	fresh := mk("node24-unavailable-aaaaaaaaaaaaaaaa", 5*time.Minute) // handed to a running job
	799	+	stale := mk("node24-unavailable-bbbbbbbbbbbbbbbb", 2*time.Hour)
	800	+
	801	+	// A reason that is not current any more but was handed out again just now
	802	+	// (Externals touches what it returns) survives; one that was not, does not.
	803	+	cur := f.m.Externals()[0]
	804	+	old := time.Now().Add(-3 * time.Hour)
	805	+	if err := os.Chtimes(cur.Dir, old, old); err != nil {
	806	+	t.Fatal(err)
	807	+	}
	808	+	f.m.Externals() // handed out: touched
	809	+	if st, _ := os.Stat(cur.Dir); time.Since(st.ModTime()) > time.Minute {
	810	+	t.Fatalf("Externals did not touch %s", cur.Dir)
	811	+	}
	812	+	// The set changes (the reasons are not the same any more), the touched
	813	+	// directory is no longer current.
	814	+	f.m.mu.Lock()
	815	+	f.m.state = map[string]entry{}
	816	+	f.m.mu.Unlock()
	817	+	f.m.prune(zerowrap.Logger{}, map[string]bool{})
	818	+	for _, p := range []string{fresh, cur.Dir} {
	819	+	if _, err := os.Stat(p); err != nil {
	820	+	t.Errorf("%s was pruned although handed out in the last hour", p)
	821	+	}
	822	+	}
	823	+	if _, err := os.Stat(stale); err == nil {
	824	+	t.Error("a stale stub directory was kept")
	825	+	}
	826	+	}
	827	+
	828	+	// ---- retries
	829	+
	830	+	func TestRetriesRunOnATickerWithoutAnyJob(t *testing.T) {
	831	+	f := newFixture(t)
	832	+	var up atomic.Bool
	833	+	var calls atomic.Int64
	834	+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	835	+	calls.Add(1)
	836	+	if !up.Load() {
	837	+	http.Error(w, "down", http.StatusServiceUnavailable)
	838	+	return
	839	+	}
	840	+	_, _ = w.Write(f.archive)
	841	+	}))
	842	+	defer srv.Close()
	843	+	m := &Manager{Dir: f.dir, Arch: "x64", Client: srv.Client(), RetryEvery: 20 * time.Millisecond}
	844	+	m.Table = []Artifact{{Name: "node24", Version: "2.0.0", Arch: "x64", URL: srv.URL + "/a-linux-x64.tar.gz", SHA256: f.sum, StripTop: true}}
	845	+	ctx, cancel := context.WithCancel(context.Background())
	846	+	m.Start(ctx, time.Minute)
	847	+
	848	+	// Nobody calls Externals(): the ticker alone retries.
	849	+	deadline := time.Now().Add(10 * time.Second)
	850	+	for calls.Load() < 3 && time.Now().Before(deadline) {
	851	+	time.Sleep(10 * time.Millisecond)
	852	+	}
	853	+	if calls.Load() < 3 {
	854	+	t.Fatalf("%d attempts without any job", calls.Load())
	855	+	}
	856	+	up.Store(true)
	857	+	for time.Now().Before(deadline) {
	858	+	m.mu.Lock()
	859	+	ok := !m.state["node24"].ext.Unavailable
	860	+	m.mu.Unlock()
	861	+	if ok {
	862	+	break
	863	+	}
	864	+	time.Sleep(10 * time.Millisecond)
	865	+	}
	866	+	if m.Externals()[0].Unavailable {
	867	+	t.Fatal("never recovered")
	868	+	}
	869	+
	870	+	// Once everything is there the ticker stops downloading, and with the context
	871	+	// it stops altogether.
	872	+	before := calls.Load()
	873	+	time.Sleep(150 * time.Millisecond)
	874	+	if calls.Load() != before {
	875	+	t.Fatal("the ticker keeps requesting what it has")
	876	+	}
	877	+	cancel()
	878	+	m.mu.Lock()
	879	+	m.state["node24"] = entry{ext: sandbox.External{Name: "node24", Unavailable: true}, retry: true}
	880	+	m.mu.Unlock()
	881	+	time.Sleep(150 * time.Millisecond)
	882	+	if calls.Load() != before {
	883	+	t.Fatal("a retry ran after the context ended")
	884	+	}
	885	+	}
	886	+
	887	+	// A verdict (wrong checksum) is not retried by the ticker.
	888	+	func TestNoRetryForAVerdict(t *testing.T) {
	889	+	f := newFixture(t)
	890	+	f.table[0].SHA256 = strings.Repeat("0", 64)
	891	+	m := &Manager{Dir: f.dir, Arch: "x64", Table: f.table[:1], Client: f.srv.Client(), RetryEvery: 10 * time.Millisecond}
	892	+	ctx, cancel := context.WithCancel(context.Background())
	893	+	defer cancel()
	894	+	m.Start(ctx, time.Minute)
	895	+	hits := f.hits.Load()
	896	+	time.Sleep(150 * time.Millisecond)
	897	+	if f.hits.Load() != hits {
	898	+	t.Fatal("a checksum mismatch was retried")
	899	+	}
	900	+	}
	901	+
	902	+	// ---- the real archives, when they are at hand
	903	+
	904	+	// The actions/alpine_nodejs archives have no top directory (./bin/node). Set
	905	+	// ERMITE_RUNNER_TEST_ALPINE_DIR, or leave the files in /tmp/alpine-node.
	906	+	func TestRealAlpineArchives(t *testing.T) {
	907	+	dir := os.Getenv("ERMITE_RUNNER_TEST_ALPINE_DIR")
	908	+	if dir == "" {
	909	+	dir = "/tmp/alpine-node"
	910	+	}
	911	+	ran := false
	912	+	for _, a := range Pinned {
	913	+	if !strings.HasSuffix(a.Name, "_alpine") {
	914	+	continue
	915	+	}
	916	+	archive := filepath.Join(dir, filepath.Base(a.URL))
	917	+	b, err := os.ReadFile(archive)
	918	+	if err != nil {
	919	+	t.Logf("%s: %v", a.Name, err)
	920	+	continue
	921	+	}
	922	+	ran = true
	923	+	t.Run(a.Name, func(t *testing.T) {
	924	+	sum := sha256.Sum256(b)
	925	+	if err := VerifySum(hex.EncodeToString(sum[:]), a.SHA256); err != nil {
	926	+	t.Fatalf("the pinned checksum is not the file's: %v", err)
	927	+	}
	928	+	dst := t.TempDir()
	929	+	if err := extractTarGz(archive, dst, a.StripTop); err != nil {
	930	+	t.Fatal(err)
	931	+	}
	932	+	if !isReady(dst) {
	933	+	t.Fatal("no executable bin/node")
	934	+	}
	935	+	if st, err := os.Stat(filepath.Join(dst, "LICENSE")); err != nil \|\| st.Mode().Perm() != 0o644 {
	936	+	t.Fatalf("LICENSE: %v %v", st, err)
	937	+	}
	938	+	// Taking the first component off would lose bin/node.
	939	+	wrong := t.TempDir()
	940	+	_ = extractTarGz(archive, wrong, true)
	941	+	if isReady(wrong) {
	942	+	t.Fatal("the alpine archive must not be unpacked with the top directory stripped")
	943	+	}
	944	+	})
	945	+	}
	946	+	if !ran {
	947	+	t.Skipf("no alpine archive in %s", dir)
	948	+	}
	949	+	}


@@ -9,18 +9,23 @@ import (
9		"context"	9		"context"
10		"crypto/sha256"	10		"crypto/sha256"
11		"encoding/hex"	11		"encoding/hex"
			12	+	"fmt"
12		"net/http"	13		"net/http"
13		"net/http/httptest"	14		"net/http/httptest"
14		"net/url"	15		"net/url"
15		"os"	16		"os"
			17	+	"os/exec"
16		"path/filepath"	18		"path/filepath"
17		"regexp"	19		"regexp"
18		"strings"	20		"strings"
			21	+	"sync"
19		"sync/atomic"	22		"sync/atomic"
20		"syscall"	23		"syscall"
21		"testing"	24		"testing"
22		"time"	25		"time"
23			26
			27	+	"github.com/bnema/zerowrap"
			28	+
24		"bnema.dev/ermite-runner/internal/sandbox"	29		"bnema.dev/ermite-runner/internal/sandbox"
25		)	30		)
26			31

@@ -38,16 +43,12 @@ func TestPinnedTable(t *testing.T) {
38		if !strings.HasPrefix(a.URL, "https://") \|\| !strings.HasSuffix(a.URL, ".tar.gz") \|\| !strings.Contains(a.URL, "v"+a.Version+"/") {	43		if !strings.HasPrefix(a.URL, "https://") \|\| !strings.HasSuffix(a.URL, ".tar.gz") \|\| !strings.Contains(a.URL, "v"+a.Version+"/") {
39		t.Errorf("%s: URL %q", key, a.URL)	44		t.Errorf("%s: URL %q", key, a.URL)
40		}	45		}
41	-	if a.SHA256 == "" {	46	+	if !hex64.MatchString(a.SHA256) {
42	-	if a.Pending == "" {
43	-	t.Errorf("%s has neither a checksum nor a reason", key)
44	-	}
45	-	if !strings.HasPrefix(a.URL, alpineDist) {
46	-	t.Errorf("%s: only the alpine builds may lack a checksum", key)
47	-	}
48	-	} else if !hex64.MatchString(a.SHA256) {
49		t.Errorf("%s: SHA-256 %q", key, a.SHA256)	47		t.Errorf("%s: SHA-256 %q", key, a.SHA256)
50		}	48		}
			49	+	if a.StripTop != strings.HasPrefix(a.URL, nodeDist) {
			50	+	t.Errorf("%s: StripTop %v for %s", key, a.StripTop, a.URL)
			51	+	}
51		}	52		}
52		// actions/runner's externals.sh: node20 and node24 for both architectures,	53		// actions/runner's externals.sh: node20 and node24 for both architectures,
53		// the alpine builds for x64.	54		// the alpine builds for x64.

@@ -61,6 +62,11 @@ func TestPinnedTable(t *testing.T) {
61		t.Errorf("%s pins %s", a.Name, a.Version)	62		t.Errorf("%s pins %s", a.Name, a.Version)
62		}	63		}
63		}	64		}
			65	+	for _, a := range Pinned {
			66	+	if strings.HasSuffix(a.Name, "_alpine") && (a.Arch != "x64" \|\| !strings.HasPrefix(a.URL, alpineDist+"v"+a.Version+"/node-v"+a.Version+"-alpine-x64.tar.gz")) {
			67	+	t.Errorf("%s: %+v", a.Name, a)
			68	+	}
			69	+	}
64		if got := len(ForArch(Pinned, "arm64")); got != 2 {	70		if got := len(ForArch(Pinned, "arm64")); got != 2 {
65		t.Errorf("arm64 has %d artifacts", got)	71		t.Errorf("arm64 has %d artifacts", got)
66		}	72		}

@@ -138,7 +144,7 @@ func extractBytes(t *testing.T, data []byte) (string, error) {
138		if err := os.Mkdir(dst, 0o755); err != nil {	144		if err := os.Mkdir(dst, 0o755); err != nil {
139		t.Fatal(err)	145		t.Fatal(err)
140		}	146		}
141	-	return dst, extractTarGz(arch, dst)	147	+	return dst, extractTarGz(arch, dst, true)
142		}	148		}
143			149
144		func TestExtractLayoutAndModes(t *testing.T) {	150		func TestExtractLayoutAndModes(t *testing.T) {

@@ -260,14 +266,14 @@ func TestLinkTargetRule(t *testing.T) {
260		name, target string	266		name, target string
261		ok bool	267		ok bool
262		}{	268		}{
263	-	{"t/bin/npm", "../lib/x", true},	269	+	{"bin/npm", "../lib/x", true},
264	-	{"t/bin/npm", "../../x", false}, // one above the tree	270	+	{"bin/npm", "../../x", false}, // one above the tree
265	-	{"t/lib/a/b/l", "../../../x", true},	271	+	{"lib/a/b/l", "../../../x", true},
266	-	{"t/lib/a/b/l", "../../../../x", false},	272	+	{"lib/a/b/l", "../../../../x", false},
267	-	{"t/l", "bin/node", true},	273	+	{"l", "bin/node", true},
268	-	{"t/l", "../x", false},	274	+	{"l", "../x", false},
269	-	{"t/l", "/abs", false},	275	+	{"l", "/abs", false},
270	-	{"t/a/l", "b/../c", false},	276	+	{"a/l", "b/../c", false},
271		} {	277		} {
272		if err := checkLinkTarget(c.name, c.target); (err == nil) != c.ok {	278		if err := checkLinkTarget(c.name, c.target); (err == nil) != c.ok {
273		t.Errorf("%s -> %s: %v", c.name, c.target, err)	279		t.Errorf("%s -> %s: %v", c.name, c.target, err)

@@ -278,13 +284,15 @@ func TestLinkTargetRule(t *testing.T) {
278		// ---- the manager	284		// ---- the manager
279			285
280		type fixture struct {	286		type fixture struct {
281	-	srv *httptest.Server	287	+	srv *httptest.Server
282	-	hits atomic.Int64	288	+	hits atomic.Int64
283	-	archive []byte	289	+	archive []byte
284	-	sum string	290	+	sum string
285	-	table []Artifact	291	+	archiveAlpine []byte // no top directory, like actions/alpine_nodejs
286	-	dir string	292	+	sumAlpine string
287	-	m *Manager	293	+	table []Artifact
			294	+	dir string
			295	+	m *Manager
288		}	296		}
289			297
290		func newFixture(t testing.T) fixture {	298		func newFixture(t testing.T) fixture {

@@ -293,9 +301,17 @@ func newFixture(t testing.T) fixture {
293		f.archive = targz(t, nodeTree()...)	301		f.archive = targz(t, nodeTree()...)
294		s := sha256.Sum256(f.archive)	302		s := sha256.Sum256(f.archive)
295		f.sum = hex.EncodeToString(s[:])	303		f.sum = hex.EncodeToString(s[:])
			304	+	f.archiveAlpine = targz(t,
			305	+	ent{name: "./bin/", typ: tar.TypeDir, mode: 0o755},
			306	+	ent{name: "./bin/node", mode: 0o755, body: "#!/bin/sh\necho alpine\n"},
			307	+	ent{name: "./LICENSE", mode: 0o644, body: "x"})
			308	+	sa := sha256.Sum256(f.archiveAlpine)
			309	+	f.sumAlpine = hex.EncodeToString(sa[:])
296		f.srv = httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {	310		f.srv = httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
297		f.hits.Add(1)	311		f.hits.Add(1)
298		switch {	312		switch {
			313	+	case strings.HasSuffix(r.URL.Path, "/alpine.tar.gz"):
			314	+	_, _ = w.Write(f.archiveAlpine)
299		case strings.HasSuffix(r.URL.Path, "-linux-x64.tar.gz"):	315		case strings.HasSuffix(r.URL.Path, "-linux-x64.tar.gz"):
300		_, _ = w.Write(f.archive)	316		_, _ = w.Write(f.archive)
301		case strings.HasSuffix(r.URL.Path, "tampered.tar.gz"):	317		case strings.HasSuffix(r.URL.Path, "tampered.tar.gz"):

@@ -307,9 +323,9 @@ func newFixture(t testing.T) fixture {
307		t.Cleanup(f.srv.Close)	323		t.Cleanup(f.srv.Close)
308		f.dir = filepath.Join(t.TempDir(), "externals")	324		f.dir = filepath.Join(t.TempDir(), "externals")
309		f.table = []Artifact{	325		f.table = []Artifact{
310	-	{Name: "node20", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/node-v1.0.0-linux-x64.tar.gz", SHA256: f.sum},	326	+	{Name: "node20", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/node-v1.0.0-linux-x64.tar.gz", SHA256: f.sum, StripTop: true},
311	-	{Name: "node24", Version: "2.0.0", Arch: "x64", URL: f.srv.URL + "/v2.0.0/node-v2.0.0-linux-x64.tar.gz", SHA256: f.sum},	327	+	{Name: "node24", Version: "2.0.0", Arch: "x64", URL: f.srv.URL + "/v2.0.0/node-v2.0.0-linux-x64.tar.gz", SHA256: f.sum, StripTop: true},
312	-	{Name: "node20_alpine", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/alpine.tar.gz", Pending: "no checksum yet"},	328	+	{Name: "node20_alpine", Version: "1.0.0", Arch: "x64", URL: f.srv.URL + "/v1.0.0/alpine.tar.gz", SHA256: f.sumAlpine},
313		}	329		}
314		f.m = &Manager{Dir: f.dir, Arch: "x64", Table: f.table, Client: f.srv.Client()}	330		f.m = &Manager{Dir: f.dir, Arch: "x64", Table: f.table, Client: f.srv.Client()}
315		return f	331		return f

@@ -339,20 +355,15 @@ func TestSyncDownloadsVerifiesAndReuses(t *testing.T) {
339		if b, err := os.ReadFile(filepath.Join(n24.Dir, "bin", "node")); err != nil \|\| !strings.Contains(string(b), "echo v1") {	355		if b, err := os.ReadFile(filepath.Join(n24.Dir, "bin", "node")); err != nil \|\| !strings.Contains(string(b), "echo v1") {
340		t.Fatalf("node24: %q %v", b, err)	356		t.Fatalf("node24: %q %v", b, err)
341		}	357		}
342	-	// The alpine build has no checksum: not downloaded, with the reason.	358	+	// The alpine archive has no top directory: bin/node is at its root.
343	-	if !al.Unavailable {	359	+	if al.Unavailable {
344		t.Fatalf("alpine: %+v", al)	360		t.Fatalf("alpine: %+v", al)
345		}	361		}
346	-	reason, err := os.ReadFile(filepath.Join(al.Dir, sandbox.ExternalUnavailableFile))	362	+	if b, err := os.ReadFile(filepath.Join(al.Dir, "bin", "node")); err != nil \|\| !strings.Contains(string(b), "echo alpine") {
347	-	if err != nil \|\| !strings.Contains(string(reason), "not available yet") \|\| !strings.Contains(string(reason), "no checksum yet") {	363	+	t.Fatalf("alpine node: %q %v", b, err)
348	-	t.Fatalf("reason: %q %v", reason, err)
349	-	}
350	-	stub, err := os.ReadFile(filepath.Join(al.Dir, "bin", "node"))
351	-	if err != nil \|\| !strings.Contains(string(stub), "not available yet") {
352	-	t.Fatalf("stub: %q %v", stub, err)
353		}	364		}
354		hits := f.hits.Load()	365		hits := f.hits.Load()
355	-	if hits != 2 {	366	+	if hits != 3 {
356		t.Fatalf("downloads: %d", hits)	367		t.Fatalf("downloads: %d", hits)
357		}	368		}
358			369

@@ -369,6 +380,29 @@ func TestSyncDownloadsVerifiesAndReuses(t *testing.T) {
369		}	380		}
370		}	381		}
371			382
			383	+	// runStub runs the bin/node of an unavailable tool: it must fail with the reason
			384	+	// on stderr, from wherever the tool is mounted.
			385	+	func runStub(t *testing.T, e sandbox.External, want string) {
			386	+	t.Helper()
			387	+	if _, err := exec.LookPath("sh"); err != nil {
			388	+	t.Skip("sh needed")
			389	+	}
			390	+	// Run it through a symlinked path too, as a guest sees it at another place.
			391	+	cmd := exec.Command(filepath.Join(e.Dir, "bin", "node"), "-e", "x")
			392	+	var stderr, stdout bytes.Buffer
			393	+	cmd.Stderr, cmd.Stdout = &stderr, &stdout
			394	+	err := cmd.Run()
			395	+	if ee, ok := err.(*exec.ExitError); !ok \|\| ee.ExitCode() != 1 {
			396	+	t.Fatalf("the stub must exit 1, got %v", err)
			397	+	}
			398	+	if !strings.Contains(stderr.String(), want) \|\| stdout.Len() != 0 {
			399	+	t.Fatalf("stub stderr %q stdout %q, want %q", stderr.String(), stdout.String(), want)
			400	+	}
			401	+	if b, _ := os.ReadFile(filepath.Join(e.Dir, "bin", "node")); strings.Contains(string(b), want) {
			402	+	t.Fatalf("the stub holds the reason: %q", b)
			403	+	}
			404	+	}
			405	+
372		func TestSyncRefusesAChecksumMismatchBeforeExtracting(t *testing.T) {	406		func TestSyncRefusesAChecksumMismatchBeforeExtracting(t *testing.T) {
373		f := newFixture(t)	407		f := newFixture(t)
374		f.table[0].SHA256 = strings.Repeat("0", 64)	408		f.table[0].SHA256 = strings.Repeat("0", 64)

@@ -396,6 +430,7 @@ func TestSyncRefusesAChecksumMismatchBeforeExtracting(t *testing.T) {
396		if f.m.state["node20"].retry {	430		if f.m.state["node20"].retry {
397		t.Fatal("retry scheduled for a mismatch")	431		t.Fatal("retry scheduled for a mismatch")
398		}	432		}
			433	+	runStub(t, n20, "SHA-256 mismatch")
399		ents, _ := os.ReadDir(f.dir)	434		ents, _ := os.ReadDir(f.dir)
400		for _, e := range ents {	435		for _, e := range ents {
401		if strings.HasPrefix(e.Name(), ".tmp-") {	436		if strings.HasPrefix(e.Name(), ".tmp-") {

@@ -432,12 +467,9 @@ func TestSyncOfflineKeepsTheRunnerGoing(t *testing.T) {
432		if !e.Unavailable {	467		if !e.Unavailable {
433		t.Fatalf("%s available offline", n)	468		t.Fatalf("%s available offline", n)
434		}	469		}
435	-	stub, _ := os.ReadFile(filepath.Join(e.Dir, "bin", "node"))	470	+	runStub(t, e, "is not available on this runner")
436	-	if !strings.Contains(string(stub), "is not available on this runner") \|\| !strings.HasPrefix(string(stub), "#!/bin/sh") {
437	-	t.Fatalf("stub: %q", stub)
438	-	}
439		}	471		}
440	-	if !f.m.state["node20"].retry {	472	+	if !f.m.wantsRetry() {
441		t.Fatal("a network failure must be retried")	473		t.Fatal("a network failure must be retried")
442		}	474		}
443		}	475		}

@@ -504,7 +536,7 @@ func TestSyncKeepsOnlyPinnedVersions(t *testing.T) {
504		for _, e := range ents {	536		for _, e := range ents {
505		got = append(got, e.Name())	537		got = append(got, e.Name())
506		}	538		}
507	-	want := ".unavailable node20-1.0.0-x64 node24-2.0.0-x64"	539	+	want := "node20-1.0.0-x64 node20_alpine-1.0.0-x64 node24-2.0.0-x64"
508		if strings.Join(got, " ") != want {	540		if strings.Join(got, " ") != want {
509		t.Fatalf("kept %v, want %s", got, want)	541		t.Fatalf("kept %v, want %s", got, want)
510		}	542		}

@@ -587,7 +619,7 @@ func TestFailedDownloadIsRetriedInTheBackground(t *testing.T) {
587		}))	619		}))
588		defer srv.Close()	620		defer srv.Close()
589		m := &Manager{Dir: f.dir, Arch: "x64", Client: srv.Client(), RetryEvery: time.Millisecond}	621		m := &Manager{Dir: f.dir, Arch: "x64", Client: srv.Client(), RetryEvery: time.Millisecond}
590	-	m.Table = []Artifact{{Name: "node24", Version: "2.0.0", Arch: "x64", URL: srv.URL + "/a-linux-x64.tar.gz", SHA256: f.sum}}	622	+	m.Table = []Artifact{{Name: "node24", Version: "2.0.0", Arch: "x64", URL: srv.URL + "/a-linux-x64.tar.gz", SHA256: f.sum, StripTop: true}}
591		ctx, cancel := context.WithCancel(context.Background())	623		ctx, cancel := context.WithCancel(context.Background())
592		defer cancel()	624		defer cancel()
593		m.Start(ctx, time.Minute)	625		m.Start(ctx, time.Minute)

@@ -630,3 +662,288 @@ func TestStartDoesNotWaitLongerThanAsked(t *testing.T) {
630		}	662		}
631		t.Fatal("the background sync did not stop")	663		t.Fatal("the background sync did not stop")
632		}	664		}
			665	+
			666	+	// ---- the directories of unavailable tools
			667	+
			668	+	func TestStubIsBuiltOnceAndNeverRewrittenInPlace(t *testing.T) {
			669	+	m := &Manager{Dir: filepath.Join(t.TempDir(), "e"), Arch: "x64"}
			670	+	e := m.unavailable("node24", "why", false)
			671	+	node := filepath.Join(e.ext.Dir, "bin", "node")
			672	+	file := filepath.Join(e.ext.Dir, sandbox.ExternalUnavailableFile)
			673	+	ino := func(p string) uint64 {
			674	+	st, err := os.Stat(p)
			675	+	if err != nil {
			676	+	t.Fatal(err)
			677	+	}
			678	+	return st.Sys().(*syscall.Stat_t).Ino
			679	+	}
			680	+	n0, f0 := ino(node), ino(file)
			681	+
			682	+	// Again, from this process and from a fresh one (a restart): same files.
			683	+	m.unavailable("node24", "why", false)
			684	+	(&Manager{Dir: m.Dir, Arch: "x64"}).unavailable("node24", "why", false)
			685	+	if ino(node) != n0 \|\| ino(file) != f0 {
			686	+	t.Fatal("an existing, correct stub was rewritten")
			687	+	}
			688	+
			689	+	// A damaged or missing file is replaced by a rename (a new inode, so a reader
			690	+	// of the old one never sees it change), with the right mode.
			691	+	if err := os.WriteFile(node, nil, 0o755); err != nil {
			692	+	t.Fatal(err)
			693	+	}
			694	+	if err := os.Remove(file); err != nil {
			695	+	t.Fatal(err)
			696	+	}
			697	+	(&Manager{Dir: m.Dir, Arch: "x64"}).unavailable("node24", "why", false)
			698	+	if b, _ := os.ReadFile(node); string(b) != unavailableStub {
			699	+	t.Fatalf("stub not repaired: %q", b)
			700	+	}
			701	+	if b, _ := os.ReadFile(file); string(b) != "why\n" {
			702	+	t.Fatalf("reason not repaired: %q", b)
			703	+	}
			704	+	if st, _ := os.Stat(node); st.Mode().Perm() != 0o755 \|\| ino(node) == n0 {
			705	+	t.Fatalf("repair in place: %v", st.Mode())
			706	+	}
			707	+	runStub(t, e.ext, "why")
			708	+	left, _ := filepath.Glob(filepath.Join(e.ext.Dir, "", ".tmp-"))
			709	+	left2, _ := filepath.Glob(filepath.Join(e.ext.Dir, ".tmp-*"))
			710	+	if len(left)+len(left2) != 0 {
			711	+	t.Fatalf("temporary files left: %v %v", left, left2)
			712	+	}
			713	+	}
			714	+
			715	+	// While Externals() and Sync() run in several goroutines, a reader that walks
			716	+	// the directories of the unavailable tools (as the build of a VM's externals
			717	+	// disk does) never sees a missing, empty or partial file.
			718	+	func TestConcurrentExternalsAndSyncNeverShowAHalfWrittenStub(t *testing.T) {
			719	+	f := newFixture(t)
			720	+	f.srv.Close() // offline: every tool is unavailable
			721	+	ctx := context.Background()
			722	+	stop := make(chan struct{})
			723	+	var wg sync.WaitGroup
			724	+	var bad atomic.Value
			725	+	check := func(dir string) {
			726	+	for _, c := range []struct{ name, want string }{{"bin/node", unavailableStub}} {
			727	+	b, err := os.ReadFile(filepath.Join(dir, c.name))
			728	+	if err != nil {
			729	+	bad.Store(fmt.Sprintf("%s/%s: %v", dir, c.name, err))
			730	+	} else if string(b) != c.want {
			731	+	bad.Store(fmt.Sprintf("%s/%s: %q", dir, c.name, b))
			732	+	}
			733	+	}
			734	+	b, err := os.ReadFile(filepath.Join(dir, sandbox.ExternalUnavailableFile))
			735	+	if err != nil \|\| len(b) == 0 \|\| !strings.HasSuffix(string(b), "\n") {
			736	+	bad.Store(fmt.Sprintf("%s/UNAVAILABLE: %q %v", dir, b, err))
			737	+	}
			738	+	}
			739	+	for i := 0; i < 4; i++ {
			740	+	wg.Add(1)
			741	+	go func() {
			742	+	defer wg.Done()
			743	+	for {
			744	+	select {
			745	+	case <-stop:
			746	+	return
			747	+	default:
			748	+	}
			749	+	for _, e := range f.m.Externals() {
			750	+	check(e.Dir) // what a driver would hand to the disk build
			751	+	}
			752	+	}
			753	+	}()
			754	+	}
			755	+	wg.Add(1)
			756	+	go func() {
			757	+	defer wg.Done()
			758	+	for {
			759	+	select {
			760	+	case <-stop:
			761	+	return
			762	+	default:
			763	+	}
			764	+	dirs, _ := filepath.Glob(filepath.Join(f.dir, unavailableDir, "*"))
			765	+	for _, d := range dirs {
			766	+	if !strings.Contains(filepath.Base(d), ".tmp-") {
			767	+	check(d)
			768	+	}
			769	+	}
			770	+	}
			771	+	}()
			772	+	for i := 0; i < 5; i++ {
			773	+	_ = f.m.Sync(ctx)
			774	+	}
			775	+	close(stop)
			776	+	wg.Wait()
			777	+	if v := bad.Load(); v != nil {
			778	+	t.Fatalf("a reader saw an incomplete stub: %v", v)
			779	+	}
			780	+	}
			781	+
			782	+	func TestPruneKeepsStubDirectoriesHandedOutInTheLastHour(t *testing.T) {
			783	+	f := newFixture(t)
			784	+	f.srv.Close()
			785	+	_ = f.m.Sync(context.Background()) // offline: stubs for the three tools
			786	+	root := filepath.Join(f.dir, unavailableDir)
			787	+	mk := func(name string, age time.Duration) string {
			788	+	p := filepath.Join(root, name)
			789	+	if err := os.MkdirAll(filepath.Join(p, "bin"), 0o755); err != nil {
			790	+	t.Fatal(err)
			791	+	}
			792	+	old := time.Now().Add(-age)
			793	+	if err := os.Chtimes(p, old, old); err != nil {
			794	+	t.Fatal(err)
			795	+	}
			796	+	return p
			797	+	}
			798	+	fresh := mk("node24-unavailable-aaaaaaaaaaaaaaaa", 5*time.Minute) // handed to a running job
			799	+	stale := mk("node24-unavailable-bbbbbbbbbbbbbbbb", 2*time.Hour)
			800	+
			801	+	// A reason that is not current any more but was handed out again just now
			802	+	// (Externals touches what it returns) survives; one that was not, does not.
			803	+	cur := f.m.Externals()[0]
			804	+	old := time.Now().Add(-3 * time.Hour)
			805	+	if err := os.Chtimes(cur.Dir, old, old); err != nil {
			806	+	t.Fatal(err)
			807	+	}
			808	+	f.m.Externals() // handed out: touched
			809	+	if st, _ := os.Stat(cur.Dir); time.Since(st.ModTime()) > time.Minute {
			810	+	t.Fatalf("Externals did not touch %s", cur.Dir)
			811	+	}
			812	+	// The set changes (the reasons are not the same any more), the touched
			813	+	// directory is no longer current.
			814	+	f.m.mu.Lock()
			815	+	f.m.state = map[string]entry{}
			816	+	f.m.mu.Unlock()
			817	+	f.m.prune(zerowrap.Logger{}, map[string]bool{})
			818	+	for _, p := range []string{fresh, cur.Dir} {
			819	+	if _, err := os.Stat(p); err != nil {
			820	+	t.Errorf("%s was pruned although handed out in the last hour", p)
			821	+	}
			822	+	}
			823	+	if _, err := os.Stat(stale); err == nil {
			824	+	t.Error("a stale stub directory was kept")
			825	+	}
			826	+	}
			827	+
			828	+	// ---- retries
			829	+
			830	+	func TestRetriesRunOnATickerWithoutAnyJob(t *testing.T) {
			831	+	f := newFixture(t)
			832	+	var up atomic.Bool
			833	+	var calls atomic.Int64
			834	+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			835	+	calls.Add(1)
			836	+	if !up.Load() {
			837	+	http.Error(w, "down", http.StatusServiceUnavailable)
			838	+	return
			839	+	}
			840	+	_, _ = w.Write(f.archive)
			841	+	}))
			842	+	defer srv.Close()
			843	+	m := &Manager{Dir: f.dir, Arch: "x64", Client: srv.Client(), RetryEvery: 20 * time.Millisecond}
			844	+	m.Table = []Artifact{{Name: "node24", Version: "2.0.0", Arch: "x64", URL: srv.URL + "/a-linux-x64.tar.gz", SHA256: f.sum, StripTop: true}}
			845	+	ctx, cancel := context.WithCancel(context.Background())
			846	+	m.Start(ctx, time.Minute)
			847	+
			848	+	// Nobody calls Externals(): the ticker alone retries.
			849	+	deadline := time.Now().Add(10 * time.Second)
			850	+	for calls.Load() < 3 && time.Now().Before(deadline) {
			851	+	time.Sleep(10 * time.Millisecond)
			852	+	}
			853	+	if calls.Load() < 3 {
			854	+	t.Fatalf("%d attempts without any job", calls.Load())
			855	+	}
			856	+	up.Store(true)
			857	+	for time.Now().Before(deadline) {
			858	+	m.mu.Lock()
			859	+	ok := !m.state["node24"].ext.Unavailable
			860	+	m.mu.Unlock()
			861	+	if ok {
			862	+	break
			863	+	}
			864	+	time.Sleep(10 * time.Millisecond)
			865	+	}
			866	+	if m.Externals()[0].Unavailable {
			867	+	t.Fatal("never recovered")
			868	+	}
			869	+
			870	+	// Once everything is there the ticker stops downloading, and with the context
			871	+	// it stops altogether.
			872	+	before := calls.Load()
			873	+	time.Sleep(150 * time.Millisecond)
			874	+	if calls.Load() != before {
			875	+	t.Fatal("the ticker keeps requesting what it has")
			876	+	}
			877	+	cancel()
			878	+	m.mu.Lock()
			879	+	m.state["node24"] = entry{ext: sandbox.External{Name: "node24", Unavailable: true}, retry: true}
			880	+	m.mu.Unlock()
			881	+	time.Sleep(150 * time.Millisecond)
			882	+	if calls.Load() != before {
			883	+	t.Fatal("a retry ran after the context ended")
			884	+	}
			885	+	}
			886	+
			887	+	// A verdict (wrong checksum) is not retried by the ticker.
			888	+	func TestNoRetryForAVerdict(t *testing.T) {
			889	+	f := newFixture(t)
			890	+	f.table[0].SHA256 = strings.Repeat("0", 64)
			891	+	m := &Manager{Dir: f.dir, Arch: "x64", Table: f.table[:1], Client: f.srv.Client(), RetryEvery: 10 * time.Millisecond}
			892	+	ctx, cancel := context.WithCancel(context.Background())
			893	+	defer cancel()
			894	+	m.Start(ctx, time.Minute)
			895	+	hits := f.hits.Load()
			896	+	time.Sleep(150 * time.Millisecond)
			897	+	if f.hits.Load() != hits {
			898	+	t.Fatal("a checksum mismatch was retried")
			899	+	}
			900	+	}
			901	+
			902	+	// ---- the real archives, when they are at hand
			903	+
			904	+	// The actions/alpine_nodejs archives have no top directory (./bin/node). Set
			905	+	// ERMITE_RUNNER_TEST_ALPINE_DIR, or leave the files in /tmp/alpine-node.
			906	+	func TestRealAlpineArchives(t *testing.T) {
			907	+	dir := os.Getenv("ERMITE_RUNNER_TEST_ALPINE_DIR")
			908	+	if dir == "" {
			909	+	dir = "/tmp/alpine-node"
			910	+	}
			911	+	ran := false
			912	+	for _, a := range Pinned {
			913	+	if !strings.HasSuffix(a.Name, "_alpine") {
			914	+	continue
			915	+	}
			916	+	archive := filepath.Join(dir, filepath.Base(a.URL))
			917	+	b, err := os.ReadFile(archive)
			918	+	if err != nil {
			919	+	t.Logf("%s: %v", a.Name, err)
			920	+	continue
			921	+	}
			922	+	ran = true
			923	+	t.Run(a.Name, func(t *testing.T) {
			924	+	sum := sha256.Sum256(b)
			925	+	if err := VerifySum(hex.EncodeToString(sum[:]), a.SHA256); err != nil {
			926	+	t.Fatalf("the pinned checksum is not the file's: %v", err)
			927	+	}
			928	+	dst := t.TempDir()
			929	+	if err := extractTarGz(archive, dst, a.StripTop); err != nil {
			930	+	t.Fatal(err)
			931	+	}
			932	+	if !isReady(dst) {
			933	+	t.Fatal("no executable bin/node")
			934	+	}
			935	+	if st, err := os.Stat(filepath.Join(dst, "LICENSE")); err != nil \|\| st.Mode().Perm() != 0o644 {
			936	+	t.Fatalf("LICENSE: %v %v", st, err)
			937	+	}
			938	+	// Taking the first component off would lose bin/node.
			939	+	wrong := t.TempDir()
			940	+	_ = extractTarGz(archive, wrong, true)
			941	+	if isReady(wrong) {
			942	+	t.Fatal("the alpine archive must not be unpacked with the top directory stripped")
			943	+	}
			944	+	})
			945	+	}
			946	+	if !ran {
			947	+	t.Skipf("no alpine archive in %s", dir)
			948	+	}
			949	+	}

M internal/externals/extract.go +23 -23


@@ -23,9 +23,10 @@ const (
23	23		maxUnpack = 2 << 30
24	24		)
25	25
26		-	// extractTarGz unpacks the .tar.gz at archive into the existing directory dest,
27		-	// dropping the first path component (Node.js archives hold one top directory,
28		-	// node-vX-linux-x64). It refuses what could write or point outside dest:
	26	+	// extractTarGz unpacks the .tar.gz at archive into the existing directory dest.
	27	+	// With stripTop the first path component is dropped (the nodejs.org archives
	28	+	// hold one top directory, node-vX-linux-x64); without it the names are taken as
	29	+	// they are (the actions/alpine_nodejs archives start at ./bin/node). It refuses what could write or point outside dest:
29	30		//
30	31		// - absolute names and names with ".." components;
31	32		// - anything but directories, regular files, symbolic links and hard links;

@@ -37,7 +38,7 @@ const (
37	38		// Writes go through os.Root, which also refuses to follow a link out of dest.
38	39		// File modes are reduced to 0755 / 0644: the owner of the files is whoever runs
39	40		// the runner, and the sandboxes only read and execute them.
40		-	func extractTarGz(archive, dest string) error {
	41	+	func extractTarGz(archive, dest string, stripTop bool) error {
41	42		f, err := os.Open(archive)
42	43		if err != nil {
43	44		return err

@@ -72,11 +73,11 @@ func extractTarGz(archive, dest string) error {
72	73		if h.Typeflag == tar.TypeXGlobalHeader {
73	74		continue
74	75		}
75		-	name, err := stripTop(h.Name)
	76	+	name, err := cleanName(h.Name, stripTop)
76	77		if err != nil {
77	78		return err
78	79		}
79		-	if name == "" { // the top directory itself
	80	+	if name == "" { // the top directory itself, or "./"
80	81		continue
81	82		}
82	83		if under(links, name) {

@@ -91,7 +92,7 @@ func extractTarGz(archive, dest string) error {
91	92		}
92	93		err = writeFile(root, name, tr, h.Mode)
93	94		case tar.TypeSymlink:
94		-	if err = checkLinkTarget(h.Name, h.Linkname); err == nil {
	95	+	if err = checkLinkTarget(name, h.Linkname); err == nil {
95	96		if err = root.MkdirAll(path.Dir(name), 0o755); err == nil {
96	97		if err = root.Symlink(h.Linkname, name); err == nil {
97	98		links[name] = true

@@ -100,7 +101,7 @@ func extractTarGz(archive, dest string) error {
100	101		}
101	102		case tar.TypeLink:
102	103		var old string
103		-	if old, err = stripTop(h.Linkname); err == nil {
	104	+	if old, err = cleanName(h.Linkname, stripTop); err == nil {
104	105		if old == "" \|\| under(links, old) {
105	106		err = fmt.Errorf("hard link %q points at %q", h.Name, h.Linkname)
106	107		} else if err = root.MkdirAll(path.Dir(name), 0o755); err == nil {

@@ -139,9 +140,10 @@ func writeFile(root *os.Root, name string, r io.Reader, mode int64) error {
139	140		return f.Close()
140	141		}
141	142
142		-	// stripTop validates an archive path and removes its first component. It
143		-	// returns "" for the top directory itself.
144		-	func stripTop(name string) (string, error) {
	143	+	// cleanName validates an archive path and returns it relative to the tree: "./"
	144	+	// prefixes and empty or "." components are dropped, and with stripTop so is the
	145	+	// first component. It returns "" for the top of the tree itself.
	146	+	func cleanName(name string, stripTop bool) (string, error) {
145	147		if name == "" \|\| strings.ContainsRune(name, 0) \|\| strings.ContainsRune(name, '\\') {
146	148		return "", fmt.Errorf("invalid entry name %q", name)
147	149		}

@@ -158,22 +160,26 @@ func stripTop(name string) (string, error) {
158	160		parts = append(parts, p)
159	161		}
160	162		}
161		-	if len(parts) <= 1 {
162		-	return "", nil
	163	+	if stripTop {
	164	+	if len(parts) <= 1 {
	165	+	return "", nil
	166	+	}
	167	+	parts = parts[1:]
163	168		}
164		-	return strings.Join(parts[1:], "/"), nil
	169	+	return strings.Join(parts, "/"), nil
165	170		}
166	171
167	172		// checkLinkTarget accepts a relative target whose ".." components all come
168		-	// first. Below the top of the tree that lexical rule is exact, because no entry
169		-	// is ever unpacked below a symbolic link.
	173	+	// first and that stays inside the tree. name is the cleaned path of the link.
	174	+	// The lexical rule is exact because no entry is ever unpacked below a symbolic
	175	+	// link.
170	176		func checkLinkTarget(name, target string) error {
171	177		if target == "" \|\| strings.ContainsRune(target, 0) \|\| strings.HasPrefix(target, "/") {
172	178		return fmt.Errorf("symbolic link %q has the target %q: absolute or empty", name, target)
173	179		}
174	180		// depth is how many directories lie between the tree's top and the link.
175	181		depth := 0
176		-	if dir := path.Dir(mustStrip(name)); dir != "." {
	182	+	if dir := path.Dir(name); dir != "." {
177	183		depth = strings.Count(dir, "/") + 1
178	184		}
179	185		up, seenName := 0, false

@@ -195,12 +201,6 @@ func checkLinkTarget(name, target string) error {
195	201		return nil
196	202		}
197	203
198		-	// mustStrip is stripTop for a name that was already validated.
199		-	func mustStrip(name string) string {
200		-	s, _ := stripTop(name)
201		-	return s
202		-	}
203		-
204	204		// under reports whether name, or one of its parents, is an unpacked symbolic link.
205	205		func under(links map[string]bool, name string) bool {
206	206		for p := name; p != "." && p != "/" && p != ""; p = path.Dir(p) {

 	maxUnpack  = 2 << 30
 )
-// extractTarGz unpacks the .tar.gz at archive into the existing directory dest,
+// extractTarGz unpacks the .tar.gz at archive into the existing directory dest.
-// dropping the first path component (Node.js archives hold one top directory,
+// With stripTop the first path component is dropped (the nodejs.org archives
-// node-vX-linux-x64). It refuses what could write or point outside dest:
+// hold one top directory, node-vX-linux-x64); without it the names are taken as
+// they are (the actions/alpine_nodejs archives start at ./bin/node). It refuses what could write or point outside dest:
 //
 //   - absolute names and names with ".." components;
 //   - anything but directories, regular files, symbolic links and hard links;
 // Writes go through os.Root, which also refuses to follow a link out of dest.
 // File modes are reduced to 0755 / 0644: the owner of the files is whoever runs
 // the runner, and the sandboxes only read and execute them.
-func extractTarGz(archive, dest string) error {
+func extractTarGz(archive, dest string, stripTop bool) error {
 	f, err := os.Open(archive)
 	if err != nil {
 		return err
 		if h.Typeflag == tar.TypeXGlobalHeader {
 			continue
 		}
-		name, err := stripTop(h.Name)
+		name, err := cleanName(h.Name, stripTop)
 		if err != nil {
 			return err
 		}
-		if name == "" { // the top directory itself
+		if name == "" { // the top directory itself, or "./"
 			continue
 		}
 		if under(links, name) {
 			}
 			err = writeFile(root, name, tr, h.Mode)
 		case tar.TypeSymlink:
-			if err = checkLinkTarget(h.Name, h.Linkname); err == nil {
+			if err = checkLinkTarget(name, h.Linkname); err == nil {
 				if err = root.MkdirAll(path.Dir(name), 0o755); err == nil {
 					if err = root.Symlink(h.Linkname, name); err == nil {
 						links[name] = true
 			}
 		case tar.TypeLink:
 			var old string
-			if old, err = stripTop(h.Linkname); err == nil {
+			if old, err = cleanName(h.Linkname, stripTop); err == nil {
 				if old == "" || under(links, old) {
 					err = fmt.Errorf("hard link %q points at %q", h.Name, h.Linkname)
 				} else if err = root.MkdirAll(path.Dir(name), 0o755); err == nil {
 	return f.Close()
 }
-// stripTop validates an archive path and removes its first component. It
+// cleanName validates an archive path and returns it relative to the tree: "./"
-// returns "" for the top directory itself.
+// prefixes and empty or "." components are dropped, and with stripTop so is the
-func stripTop(name string) (string, error) {
+// first component. It returns "" for the top of the tree itself.
+func cleanName(name string, stripTop bool) (string, error) {
 	if name == "" || strings.ContainsRune(name, 0) || strings.ContainsRune(name, '\\') {
 		return "", fmt.Errorf("invalid entry name %q", name)
 	}
 			parts = append(parts, p)
 		}
 	}
-	if len(parts) <= 1 {
+	if stripTop {
-		return "", nil
+		if len(parts) <= 1 {
+			return "", nil
+		}
+		parts = parts[1:]
 	}
-	return strings.Join(parts[1:], "/"), nil
+	return strings.Join(parts, "/"), nil
 }
 // checkLinkTarget accepts a relative target whose ".." components all come
-// first. Below the top of the tree that lexical rule is exact, because no entry
+// first and that stays inside the tree. name is the cleaned path of the link.
-// is ever unpacked below a symbolic link.
+// The lexical rule is exact because no entry is ever unpacked below a symbolic
+// link.
 func checkLinkTarget(name, target string) error {
 	if target == "" || strings.ContainsRune(target, 0) || strings.HasPrefix(target, "/") {
 		return fmt.Errorf("symbolic link %q has the target %q: absolute or empty", name, target)
 	}
 	// depth is how many directories lie between the tree's top and the link.
 	depth := 0
-	if dir := path.Dir(mustStrip(name)); dir != "." {
+	if dir := path.Dir(name); dir != "." {
 		depth = strings.Count(dir, "/") + 1
 	}
 	up, seenName := 0, false
 	return nil
 }
-// mustStrip is stripTop for a name that was already validated.
-func mustStrip(name string) string {
-	s, _ := stripTop(name)
-	return s
-}
 // under reports whether name, or one of its parents, is an unpacked symbolic link.
 func under(links map[string]bool, name string) bool {
 	for p := name; p != "." && p != "/" && p != ""; p = path.Dir(p) {

M internal/externals/manager.go +196 -51


@@ -52,15 +52,18 @@ type Manager struct {
52	52		// Client downloads the archives; nil means a client with a 15 minute limit
53	53		// that follows redirects to https locations only.
54	54		Client *http.Client
55		-	// RetryEvery is how long a failed download waits before the next attempt
56		-	// (default 5 minutes).
	55	+	// RetryEvery is the interval of the background retries of the downloads that
	56	+	// failed for a reason that may pass (default 5 minutes).
57	57		RetryEvery time.Duration
58	58
59	59		mu sync.Mutex
60	60		state map[string]entry // by artifact name
61		-	bg context.Context // the context of Start, for retries
62		-	lastTry time.Time
63	61		syncing bool
	62	+
	63	+	// stubMu serialises the creation of the directories of unavailable tools;
	64	+	// stubs remembers the ones built (or checked) by this process.
	65	+	stubMu sync.Mutex
	66	+	stubs map[string]bool
64	67		}
65	68
66	69		type entry struct {

@@ -79,13 +82,11 @@ func New(stateDir string) (m *Manager, ok bool) {
79	82		// runner with a working network normally takes its first job with everything in
80	83		// place, and one without never waits long. When the wait ends the download goes
81	84		// on; jobs that start meanwhile see the tools that are not done yet as
82		-	// unavailable ("still being downloaded"). Retries of failed downloads use ctx
83		-	// too: they stop with it.
	85	+	// unavailable ("still being downloaded"). Downloads that failed for a reason
	86	+	// that may pass (network, HTTP status) are retried every RetryEvery, jobs or
	87	+	// not, until ctx ends.
84	88		func (m *Manager) Start(ctx context.Context, wait time.Duration) {
85	89		log := zerowrap.FromCtx(ctx)
86		-	m.mu.Lock()
87		-	m.bg = ctx
88		-	m.mu.Unlock()
89	90		done := make(chan struct{})
90	91		go func() {
91	92		defer close(done)

@@ -93,6 +94,7 @@ func (m *Manager) Start(ctx context.Context, wait time.Duration) {
93	94		log.Warn().Err(err).Msg("some external tools are not available: jobs that need them fail with the reason; the download is retried")
94	95		}
95	96		}()
	97	+	go m.retryLoop(ctx)
96	98		select {
97	99		case <-done:
98	100		case <-time.After(wait):

@@ -101,6 +103,38 @@ func (m *Manager) Start(ctx context.Context, wait time.Duration) {
101	103		}
102	104		}
103	105
	106	+	// retryLoop calls Sync every RetryEvery while a tool is missing for a reason
	107	+	// that may pass.
	108	+	func (m *Manager) retryLoop(ctx context.Context) {
	109	+	every := m.RetryEvery
	110	+	if every == 0 {
	111	+	every = 5 * time.Minute
	112	+	}
	113	+	t := time.NewTicker(every)
	114	+	defer t.Stop()
	115	+	for {
	116	+	select {
	117	+	case <-ctx.Done():
	118	+	return
	119	+	case <-t.C:
	120	+	}
	121	+	if m.wantsRetry() {
	122	+	_ = m.Sync(ctx)
	123	+	}
	124	+	}
	125	+	}
	126	+
	127	+	func (m *Manager) wantsRetry() bool {
	128	+	m.mu.Lock()
	129	+	defer m.mu.Unlock()
	130	+	for _, e := range m.state {
	131	+	if e.retry {
	132	+	return true
	133	+	}
	134	+	}
	135	+	return false
	136	+	}
	137	+
104	138		// Sync makes the externals match the table: it downloads what is missing,
105	139		// removes everything else from Dir, and records the outcome per tool as it goes.
106	140		// It returns an error describing the artifacts that could not be provided; the

@@ -116,7 +150,7 @@ func (m *Manager) Sync(ctx context.Context) error {
116	150		m.state = map[string]entry{}
117	151		}
118	152		m.mu.Unlock()
119		-	defer func() { m.mu.Lock(); m.syncing = false; m.lastTry = time.Now(); m.mu.Unlock() }()
	153	+	defer func() { m.mu.Lock(); m.syncing = false; m.mu.Unlock() }()
120	154
121	155		log := zerowrap.FromCtx(ctx)
122	156		var errs []error

@@ -130,8 +164,6 @@ func (m *Manager) Sync(ctx context.Context) error {
130	164		switch {
131	165		case !ok:
132	166		e = m.unavailable(name, fmt.Sprintf("there is no %s build for linux-%s", name, m.Arch), false)
133		-	case a.SHA256 == "":
134		-	e = m.unavailable(name, fmt.Sprintf("%s (Node.js %s) is not available yet on this runner: %s", a.Name, a.Version, a.Pending), false)
135	167		default:
136	168		var err error
137	169		if e, err = m.ensure(ctx, a); err != nil {

@@ -154,28 +186,29 @@ func (m *Manager) Sync(ctx context.Context) error {
154	186		}
155	187
156	188		// Externals implements sandbox.ExternalsSource. A tool that has no outcome yet
157		-	// is reported as being downloaded. When a download failed for a reason that may
158		-	// pass, a retry is started in the background (at most one per RetryEvery).
	189	+	// is reported as being downloaded. It has no side effect on the downloads (the
	190	+	// retries run on a timer, see Start); the directories of unavailable tools are
	191	+	// built at most once and only touched here.
159	192		func (m *Manager) Externals() []sandbox.External {
160	193		m.mu.Lock()
161		-	defer m.mu.Unlock()
162		-	var out []sandbox.External
163		-	retry := false
164		-	for _, n := range m.names() {
165		-	e, ok := m.state[n]
166		-	if !ok {
167		-	e = m.unavailable(n, n+" is still being downloaded on this runner: run the job again in a few minutes", false)
168		-	}
169		-	retry = retry \|\| e.retry
170		-	out = append(out, e.ext)
171		-	}
172		-	every := m.RetryEvery
173		-	if every == 0 {
174		-	every = 5 * time.Minute
	194	+	names := m.names()
	195	+	got := make([]entry, len(names))
	196	+	have := make([]bool, len(names))
	197	+	for i, n := range names {
	198	+	got[i], have[i] = m.state[n]
175	199		}
176		-	if retry && !m.syncing && m.bg != nil && m.bg.Err() == nil && time.Since(m.lastTry) >= every {
177		-	m.lastTry = time.Now() // no second attempt before the next interval
178		-	go func(ctx context.Context) { _ = m.Sync(ctx) }(m.bg)
	200	+	m.mu.Unlock()
	201	+
	202	+	// Outside the lock: the first call for a reason writes files.
	203	+	out := make([]sandbox.External, len(names))
	204	+	for i, n := range names {
	205	+	switch {
	206	+	case !have[i]:
	207	+	got[i] = m.unavailable(n, n+" is still being downloaded on this runner: run the job again in a few minutes", false)
	208	+	case got[i].ext.Unavailable:
	209	+	touch(got[i].ext.Dir) // handed out again: not stale
	210	+	}
	211	+	out[i] = got[i].ext
179	212		}
180	213		return out
181	214		}

@@ -204,35 +237,147 @@ func (m *Manager) artifact(name string) (Artifact, bool) {
204	237
205	238		const unavailableDir = ".unavailable"
206	239
207		-	// unavailable builds the entry of a tool that is missing: a directory with the
208		-	// reason in ExternalUnavailableFile and a bin/node stub that prints it. Its name carries a hash of the reason, so
209		-	// that the ID (and so a VM's disk) changes only when the reason does.
	240	+	// unavailableStub is the bin/node of a tool that is missing: it prints the
	241	+	// reason (the file UNAVAILABLE next to bin/) on stderr and fails, so that a job
	242	+	// which runs the tool anyway (a node action, a run step) gets the reason rather
	243	+	// than "No such file or directory". It does not contain the reason.
	244	+	const unavailableStub = "#!/bin/sh\ncat \"$(dirname \"$0\")/../" + sandbox.ExternalUnavailableFile + "\" >&2\nexit 1\n"
	245	+
	246	+	// unavailable returns the entry of a tool that is missing: a directory with the
	247	+	// reason in ExternalUnavailableFile and the bin/node stub. Its name carries a
	248	+	// hash of the reason, so that the ID (and so a VM's disk) changes only when the
	249	+	// reason does. The directory is built once per process, complete, under a
	250	+	// temporary name and renamed into place; files that exist with the right
	251	+	// content are never rewritten, and a missing or different one is replaced
	252	+	// through a rename, so a reader (the build of a VM's externals disk, a job that
	253	+	// runs the stub) never sees a file half written. Each call touches the
	254	+	// directory: the pruning keeps what was handed out in the last hour.
210	255		func (m *Manager) unavailable(name, reason string, retry bool) entry {
211	256		sum := sha256.Sum256([]byte(name + "\x00" + reason))
212	257		id := "unavailable-" + hex.EncodeToString(sum[:8])
213	258		dir := filepath.Join(m.Dir, unavailableDir, name+"-"+id)
214	259		e := entry{retry: retry, ext: sandbox.External{Name: name, ID: id, Dir: dir, Unavailable: true}}
	260	+
	261	+	m.stubMu.Lock()
	262	+	defer m.stubMu.Unlock()
	263	+	if _, err := os.Lstat(filepath.Join(dir, "bin", "node")); err != nil \|\| !m.stubs[dir] {
	264	+	// Best effort: on failure the entry stays marked unavailable and the
	265	+	// sandbox sees whatever there is.
	266	+	if err := buildStub(dir, reason); err == nil {
	267	+	if m.stubs == nil {
	268	+	m.stubs = map[string]bool{}
	269	+	}
	270	+	m.stubs[dir] = true
	271	+	}
	272	+	}
	273	+	touch(dir)
	274	+	return e
	275	+	}
	276	+
	277	+	func touch(dir string) {
	278	+	now := time.Now()
	279	+	_ = os.Chtimes(dir, now, now)
	280	+	}
	281	+
	282	+	// buildStub makes dir hold the UNAVAILABLE file and the bin/node stub, with
	283	+	// explicit modes (the umask of the runner must not hide them from the user of
	284	+	// the sandbox).
	285	+	func buildStub(dir, reason string) error {
	286	+	parent := filepath.Dir(dir)
	287	+	if err := os.MkdirAll(parent, 0o755); err != nil {
	288	+	return err
	289	+	}
	290	+	if err := os.Chmod(parent, 0o755); err != nil {
	291	+	return err
	292	+	}
	293	+	files := []stubFile{
	294	+	{sandbox.ExternalUnavailableFile, reason + "\n", 0o644},
	295	+	{"bin/node", unavailableStub, 0o755},
	296	+	}
	297	+	if _, err := os.Lstat(dir); errors.Is(err, os.ErrNotExist) {
	298	+	tmp, err := os.MkdirTemp(parent, ".tmp-")
	299	+	if err != nil {
	300	+	return err
	301	+	}
	302	+	defer func() { _ = os.RemoveAll(tmp) }()
	303	+	if err := os.Mkdir(filepath.Join(tmp, "bin"), 0o755); err != nil {
	304	+	return err
	305	+	}
	306	+	for _, d := range []string{tmp, filepath.Join(tmp, "bin")} {
	307	+	if err := os.Chmod(d, 0o755); err != nil {
	308	+	return err
	309	+	}
	310	+	}
	311	+	for _, f := range files {
	312	+	if err := f.write(tmp); err != nil {
	313	+	return err
	314	+	}
	315	+	}
	316	+	if err := os.Rename(tmp, dir); err == nil {
	317	+	return nil
	318	+	}
	319	+	// Lost a race with another process, or the directory appeared: repair it.
	320	+	}
215	321		if err := os.MkdirAll(filepath.Join(dir, "bin"), 0o755); err != nil {
216		-	return e // still marked unavailable; the sandbox then has an empty directory
	322	+	return err
217	323		}
218		-	// Modes are set explicitly: the umask of the runner must not hide the files
219		-	// from the user of the sandbox.
220		-	for _, d := range []string{filepath.Dir(dir), dir, filepath.Join(dir, "bin")} {
221		-	_ = os.Chmod(d, 0o755)
	324	+	for _, f := range files {
	325	+	if f.current(dir) {
	326	+	continue
	327	+	}
	328	+	if err := f.write(dir); err != nil {
	329	+	return err
	330	+	}
222	331		}
223		-	write := func(name, content string, mode os.FileMode) {
224		-	p := filepath.Join(dir, name)
225		-	if os.WriteFile(p, []byte(content), mode) == nil {
226		-	_ = os.Chmod(p, mode)
	332	+	return nil
	333	+	}
	334	+
	335	+	type stubFile struct {
	336	+	name, content string
	337	+	mode os.FileMode
	338	+	}
	339	+
	340	+	// current reports whether the file exists with this content and mode.
	341	+	func (f stubFile) current(dir string) bool {
	342	+	p := filepath.Join(dir, f.name)
	343	+	st, err := os.Lstat(p)
	344	+	if err != nil \|\| !st.Mode().IsRegular() \|\| st.Mode().Perm() != f.mode {
	345	+	return false
	346	+	}
	347	+	b, err := os.ReadFile(p)
	348	+	return err == nil && string(b) == f.content
	349	+	}
	350	+
	351	+	// write creates the file through a temporary file in the same directory and a
	352	+	// rename, never in place.
	353	+	func (f stubFile) write(dir string) error {
	354	+	p := filepath.Join(dir, f.name)
	355	+	tmp, err := os.CreateTemp(filepath.Dir(p), ".tmp-")
	356	+	if err != nil {
	357	+	return err
	358	+	}
	359	+	ok := false
	360	+	defer func() {
	361	+	if !ok {
	362	+	_ = os.Remove(tmp.Name())
227	363		}
	364	+	}()
	365	+	if _, err := tmp.WriteString(f.content); err != nil {
	366	+	_ = tmp.Close()
	367	+	return err
228	368		}
229		-	write(sandbox.ExternalUnavailableFile, reason+"\n", 0o644)
230		-	// A stub where the executable would be, so that a job which runs it anyway
231		-	// (a node action, a run step) fails with the reason instead of "No such file
232		-	// or directory". The reason goes through a quoted here-document: it is data.
233		-	stub := "#!/bin/sh\ncat >&2 <<'ERMITE_UNAVAILABLE'\n" + strings.ReplaceAll(reason, "\nERMITE_UNAVAILABLE", "") + "\nERMITE_UNAVAILABLE\nexit 1\n"
234		-	write("bin/node", stub, 0o755)
235		-	return e
	369	+	if err := tmp.Chmod(f.mode); err != nil {
	370	+	_ = tmp.Close()
	371	+	return err
	372	+	}
	373	+	if err := tmp.Close(); err != nil {
	374	+	return err
	375	+	}
	376	+	if err := os.Rename(tmp.Name(), p); err != nil {
	377	+	return err
	378	+	}
	379	+	ok = true
	380	+	return nil
236	381		}
237	382
238	383		// ensure returns the ready entry of a, downloading it when needed. On failure it

@@ -290,7 +435,7 @@ func (m *Manager) fetch(ctx context.Context, a Artifact, final string) error {
290	435		if err := os.Mkdir(tree, 0o755); err != nil {
291	436		return err
292	437		}
293		-	if err := extractTarGz(archive, tree); err != nil {
	438	+	if err := extractTarGz(archive, tree, a.StripTop); err != nil {
294	439		return &verdictError{fmt.Errorf("unpack: %w", err)}
295	440		}
296	441		if !isReady(tree) {


@@ -52,15 +52,18 @@ type Manager struct {
52		// Client downloads the archives; nil means a client with a 15 minute limit	52		// Client downloads the archives; nil means a client with a 15 minute limit
53		// that follows redirects to https locations only.	53		// that follows redirects to https locations only.
54		Client *http.Client	54		Client *http.Client
55	-	// RetryEvery is how long a failed download waits before the next attempt	55	+	// RetryEvery is the interval of the background retries of the downloads that
56	-	// (default 5 minutes).	56	+	// failed for a reason that may pass (default 5 minutes).
57		RetryEvery time.Duration	57		RetryEvery time.Duration
58			58
59		mu sync.Mutex	59		mu sync.Mutex
60		state map[string]entry // by artifact name	60		state map[string]entry // by artifact name
61	-	bg context.Context // the context of Start, for retries
62	-	lastTry time.Time
63		syncing bool	61		syncing bool
			62	+
			63	+	// stubMu serialises the creation of the directories of unavailable tools;
			64	+	// stubs remembers the ones built (or checked) by this process.
			65	+	stubMu sync.Mutex
			66	+	stubs map[string]bool
64		}	67		}
65			68
66		type entry struct {	69		type entry struct {

@@ -79,13 +82,11 @@ func New(stateDir string) (m *Manager, ok bool) {
79		// runner with a working network normally takes its first job with everything in	82		// runner with a working network normally takes its first job with everything in
80		// place, and one without never waits long. When the wait ends the download goes	83		// place, and one without never waits long. When the wait ends the download goes
81		// on; jobs that start meanwhile see the tools that are not done yet as	84		// on; jobs that start meanwhile see the tools that are not done yet as
82	-	// unavailable ("still being downloaded"). Retries of failed downloads use ctx	85	+	// unavailable ("still being downloaded"). Downloads that failed for a reason
83	-	// too: they stop with it.	86	+	// that may pass (network, HTTP status) are retried every RetryEvery, jobs or
			87	+	// not, until ctx ends.
84		func (m *Manager) Start(ctx context.Context, wait time.Duration) {	88		func (m *Manager) Start(ctx context.Context, wait time.Duration) {
85		log := zerowrap.FromCtx(ctx)	89		log := zerowrap.FromCtx(ctx)
86	-	m.mu.Lock()
87	-	m.bg = ctx
88	-	m.mu.Unlock()
89		done := make(chan struct{})	90		done := make(chan struct{})
90		go func() {	91		go func() {
91		defer close(done)	92		defer close(done)

@@ -93,6 +94,7 @@ func (m *Manager) Start(ctx context.Context, wait time.Duration) {
93		log.Warn().Err(err).Msg("some external tools are not available: jobs that need them fail with the reason; the download is retried")	94		log.Warn().Err(err).Msg("some external tools are not available: jobs that need them fail with the reason; the download is retried")
94		}	95		}
95		}()	96		}()
			97	+	go m.retryLoop(ctx)
96		select {	98		select {
97		case <-done:	99		case <-done:
98		case <-time.After(wait):	100		case <-time.After(wait):

@@ -101,6 +103,38 @@ func (m *Manager) Start(ctx context.Context, wait time.Duration) {
101		}	103		}
102		}	104		}
103			105
			106	+	// retryLoop calls Sync every RetryEvery while a tool is missing for a reason
			107	+	// that may pass.
			108	+	func (m *Manager) retryLoop(ctx context.Context) {
			109	+	every := m.RetryEvery
			110	+	if every == 0 {
			111	+	every = 5 * time.Minute
			112	+	}
			113	+	t := time.NewTicker(every)
			114	+	defer t.Stop()
			115	+	for {
			116	+	select {
			117	+	case <-ctx.Done():
			118	+	return
			119	+	case <-t.C:
			120	+	}
			121	+	if m.wantsRetry() {
			122	+	_ = m.Sync(ctx)
			123	+	}
			124	+	}
			125	+	}
			126	+
			127	+	func (m *Manager) wantsRetry() bool {
			128	+	m.mu.Lock()
			129	+	defer m.mu.Unlock()
			130	+	for _, e := range m.state {
			131	+	if e.retry {
			132	+	return true
			133	+	}
			134	+	}
			135	+	return false
			136	+	}
			137	+
104		// Sync makes the externals match the table: it downloads what is missing,	138		// Sync makes the externals match the table: it downloads what is missing,
105		// removes everything else from Dir, and records the outcome per tool as it goes.	139		// removes everything else from Dir, and records the outcome per tool as it goes.
106		// It returns an error describing the artifacts that could not be provided; the	140		// It returns an error describing the artifacts that could not be provided; the

@@ -116,7 +150,7 @@ func (m *Manager) Sync(ctx context.Context) error {
116		m.state = map[string]entry{}	150		m.state = map[string]entry{}
117		}	151		}
118		m.mu.Unlock()	152		m.mu.Unlock()
119	-	defer func() { m.mu.Lock(); m.syncing = false; m.lastTry = time.Now(); m.mu.Unlock() }()	153	+	defer func() { m.mu.Lock(); m.syncing = false; m.mu.Unlock() }()
120			154
121		log := zerowrap.FromCtx(ctx)	155		log := zerowrap.FromCtx(ctx)
122		var errs []error	156		var errs []error

@@ -130,8 +164,6 @@ func (m *Manager) Sync(ctx context.Context) error {
130		switch {	164		switch {
131		case !ok:	165		case !ok:
132		e = m.unavailable(name, fmt.Sprintf("there is no %s build for linux-%s", name, m.Arch), false)	166		e = m.unavailable(name, fmt.Sprintf("there is no %s build for linux-%s", name, m.Arch), false)
133	-	case a.SHA256 == "":
134	-	e = m.unavailable(name, fmt.Sprintf("%s (Node.js %s) is not available yet on this runner: %s", a.Name, a.Version, a.Pending), false)
135		default:	167		default:
136		var err error	168		var err error
137		if e, err = m.ensure(ctx, a); err != nil {	169		if e, err = m.ensure(ctx, a); err != nil {

@@ -154,28 +186,29 @@ func (m *Manager) Sync(ctx context.Context) error {
154		}	186		}
155			187
156		// Externals implements sandbox.ExternalsSource. A tool that has no outcome yet	188		// Externals implements sandbox.ExternalsSource. A tool that has no outcome yet
157	-	// is reported as being downloaded. When a download failed for a reason that may	189	+	// is reported as being downloaded. It has no side effect on the downloads (the
158	-	// pass, a retry is started in the background (at most one per RetryEvery).	190	+	// retries run on a timer, see Start); the directories of unavailable tools are
			191	+	// built at most once and only touched here.
159		func (m *Manager) Externals() []sandbox.External {	192		func (m *Manager) Externals() []sandbox.External {
160		m.mu.Lock()	193		m.mu.Lock()
161	-	defer m.mu.Unlock()	194	+	names := m.names()
162	-	var out []sandbox.External	195	+	got := make([]entry, len(names))
163	-	retry := false	196	+	have := make([]bool, len(names))
164	-	for _, n := range m.names() {	197	+	for i, n := range names {
165	-	e, ok := m.state[n]	198	+	got[i], have[i] = m.state[n]
166	-	if !ok {
167	-	e = m.unavailable(n, n+" is still being downloaded on this runner: run the job again in a few minutes", false)
168	-	}
169	-	retry = retry \|\| e.retry
170	-	out = append(out, e.ext)
171	-	}
172	-	every := m.RetryEvery
173	-	if every == 0 {
174	-	every = 5 * time.Minute
175		}	199		}
176	-	if retry && !m.syncing && m.bg != nil && m.bg.Err() == nil && time.Since(m.lastTry) >= every {	200	+	m.mu.Unlock()
177	-	m.lastTry = time.Now() // no second attempt before the next interval	201	+
178	-	go func(ctx context.Context) { _ = m.Sync(ctx) }(m.bg)	202	+	// Outside the lock: the first call for a reason writes files.
			203	+	out := make([]sandbox.External, len(names))
			204	+	for i, n := range names {
			205	+	switch {
			206	+	case !have[i]:
			207	+	got[i] = m.unavailable(n, n+" is still being downloaded on this runner: run the job again in a few minutes", false)
			208	+	case got[i].ext.Unavailable:
			209	+	touch(got[i].ext.Dir) // handed out again: not stale
			210	+	}
			211	+	out[i] = got[i].ext
179		}	212		}
180		return out	213		return out
181		}	214		}

@@ -204,35 +237,147 @@ func (m *Manager) artifact(name string) (Artifact, bool) {
204			237
205		const unavailableDir = ".unavailable"	238		const unavailableDir = ".unavailable"
206			239
207	-	// unavailable builds the entry of a tool that is missing: a directory with the	240	+	// unavailableStub is the bin/node of a tool that is missing: it prints the
208	-	// reason in ExternalUnavailableFile and a bin/node stub that prints it. Its name carries a hash of the reason, so	241	+	// reason (the file UNAVAILABLE next to bin/) on stderr and fails, so that a job
209	-	// that the ID (and so a VM's disk) changes only when the reason does.	242	+	// which runs the tool anyway (a node action, a run step) gets the reason rather
			243	+	// than "No such file or directory". It does not contain the reason.
			244	+	const unavailableStub = "#!/bin/sh\ncat \"$(dirname \"$0\")/../" + sandbox.ExternalUnavailableFile + "\" >&2\nexit 1\n"
			245	+
			246	+	// unavailable returns the entry of a tool that is missing: a directory with the
			247	+	// reason in ExternalUnavailableFile and the bin/node stub. Its name carries a
			248	+	// hash of the reason, so that the ID (and so a VM's disk) changes only when the
			249	+	// reason does. The directory is built once per process, complete, under a
			250	+	// temporary name and renamed into place; files that exist with the right
			251	+	// content are never rewritten, and a missing or different one is replaced
			252	+	// through a rename, so a reader (the build of a VM's externals disk, a job that
			253	+	// runs the stub) never sees a file half written. Each call touches the
			254	+	// directory: the pruning keeps what was handed out in the last hour.
210		func (m *Manager) unavailable(name, reason string, retry bool) entry {	255		func (m *Manager) unavailable(name, reason string, retry bool) entry {
211		sum := sha256.Sum256([]byte(name + "\x00" + reason))	256		sum := sha256.Sum256([]byte(name + "\x00" + reason))
212		id := "unavailable-" + hex.EncodeToString(sum[:8])	257		id := "unavailable-" + hex.EncodeToString(sum[:8])
213		dir := filepath.Join(m.Dir, unavailableDir, name+"-"+id)	258		dir := filepath.Join(m.Dir, unavailableDir, name+"-"+id)
214		e := entry{retry: retry, ext: sandbox.External{Name: name, ID: id, Dir: dir, Unavailable: true}}	259		e := entry{retry: retry, ext: sandbox.External{Name: name, ID: id, Dir: dir, Unavailable: true}}
			260	+
			261	+	m.stubMu.Lock()
			262	+	defer m.stubMu.Unlock()
			263	+	if _, err := os.Lstat(filepath.Join(dir, "bin", "node")); err != nil \|\| !m.stubs[dir] {
			264	+	// Best effort: on failure the entry stays marked unavailable and the
			265	+	// sandbox sees whatever there is.
			266	+	if err := buildStub(dir, reason); err == nil {
			267	+	if m.stubs == nil {
			268	+	m.stubs = map[string]bool{}
			269	+	}
			270	+	m.stubs[dir] = true
			271	+	}
			272	+	}
			273	+	touch(dir)
			274	+	return e
			275	+	}
			276	+
			277	+	func touch(dir string) {
			278	+	now := time.Now()
			279	+	_ = os.Chtimes(dir, now, now)
			280	+	}
			281	+
			282	+	// buildStub makes dir hold the UNAVAILABLE file and the bin/node stub, with
			283	+	// explicit modes (the umask of the runner must not hide them from the user of
			284	+	// the sandbox).
			285	+	func buildStub(dir, reason string) error {
			286	+	parent := filepath.Dir(dir)
			287	+	if err := os.MkdirAll(parent, 0o755); err != nil {
			288	+	return err
			289	+	}
			290	+	if err := os.Chmod(parent, 0o755); err != nil {
			291	+	return err
			292	+	}
			293	+	files := []stubFile{
			294	+	{sandbox.ExternalUnavailableFile, reason + "\n", 0o644},
			295	+	{"bin/node", unavailableStub, 0o755},
			296	+	}
			297	+	if _, err := os.Lstat(dir); errors.Is(err, os.ErrNotExist) {
			298	+	tmp, err := os.MkdirTemp(parent, ".tmp-")
			299	+	if err != nil {
			300	+	return err
			301	+	}
			302	+	defer func() { _ = os.RemoveAll(tmp) }()
			303	+	if err := os.Mkdir(filepath.Join(tmp, "bin"), 0o755); err != nil {
			304	+	return err
			305	+	}
			306	+	for _, d := range []string{tmp, filepath.Join(tmp, "bin")} {
			307	+	if err := os.Chmod(d, 0o755); err != nil {
			308	+	return err
			309	+	}
			310	+	}
			311	+	for _, f := range files {
			312	+	if err := f.write(tmp); err != nil {
			313	+	return err
			314	+	}
			315	+	}
			316	+	if err := os.Rename(tmp, dir); err == nil {
			317	+	return nil
			318	+	}
			319	+	// Lost a race with another process, or the directory appeared: repair it.
			320	+	}
215		if err := os.MkdirAll(filepath.Join(dir, "bin"), 0o755); err != nil {	321		if err := os.MkdirAll(filepath.Join(dir, "bin"), 0o755); err != nil {
216	-	return e // still marked unavailable; the sandbox then has an empty directory	322	+	return err
217		}	323		}
218	-	// Modes are set explicitly: the umask of the runner must not hide the files	324	+	for _, f := range files {
219	-	// from the user of the sandbox.	325	+	if f.current(dir) {
220	-	for _, d := range []string{filepath.Dir(dir), dir, filepath.Join(dir, "bin")} {	326	+	continue
221	-	_ = os.Chmod(d, 0o755)	327	+	}
			328	+	if err := f.write(dir); err != nil {
			329	+	return err
			330	+	}
222		}	331		}
223	-	write := func(name, content string, mode os.FileMode) {	332	+	return nil
224	-	p := filepath.Join(dir, name)	333	+	}
225	-	if os.WriteFile(p, []byte(content), mode) == nil {	334	+
226	-	_ = os.Chmod(p, mode)	335	+	type stubFile struct {
			336	+	name, content string
			337	+	mode os.FileMode
			338	+	}
			339	+
			340	+	// current reports whether the file exists with this content and mode.
			341	+	func (f stubFile) current(dir string) bool {
			342	+	p := filepath.Join(dir, f.name)
			343	+	st, err := os.Lstat(p)
			344	+	if err != nil \|\| !st.Mode().IsRegular() \|\| st.Mode().Perm() != f.mode {
			345	+	return false
			346	+	}
			347	+	b, err := os.ReadFile(p)
			348	+	return err == nil && string(b) == f.content
			349	+	}
			350	+
			351	+	// write creates the file through a temporary file in the same directory and a
			352	+	// rename, never in place.
			353	+	func (f stubFile) write(dir string) error {
			354	+	p := filepath.Join(dir, f.name)
			355	+	tmp, err := os.CreateTemp(filepath.Dir(p), ".tmp-")
			356	+	if err != nil {
			357	+	return err
			358	+	}
			359	+	ok := false
			360	+	defer func() {
			361	+	if !ok {
			362	+	_ = os.Remove(tmp.Name())
227		}	363		}
			364	+	}()
			365	+	if _, err := tmp.WriteString(f.content); err != nil {
			366	+	_ = tmp.Close()
			367	+	return err
228		}	368		}
229	-	write(sandbox.ExternalUnavailableFile, reason+"\n", 0o644)	369	+	if err := tmp.Chmod(f.mode); err != nil {
230	-	// A stub where the executable would be, so that a job which runs it anyway	370	+	_ = tmp.Close()
231	-	// (a node action, a run step) fails with the reason instead of "No such file	371	+	return err
232	-	// or directory". The reason goes through a quoted here-document: it is data.	372	+	}
233	-	stub := "#!/bin/sh\ncat >&2 <<'ERMITE_UNAVAILABLE'\n" + strings.ReplaceAll(reason, "\nERMITE_UNAVAILABLE", "") + "\nERMITE_UNAVAILABLE\nexit 1\n"	373	+	if err := tmp.Close(); err != nil {
234	-	write("bin/node", stub, 0o755)	374	+	return err
235	-	return e	375	+	}
			376	+	if err := os.Rename(tmp.Name(), p); err != nil {
			377	+	return err
			378	+	}
			379	+	ok = true
			380	+	return nil
236		}	381		}
237			382
238		// ensure returns the ready entry of a, downloading it when needed. On failure it	383		// ensure returns the ready entry of a, downloading it when needed. On failure it

@@ -290,7 +435,7 @@ func (m *Manager) fetch(ctx context.Context, a Artifact, final string) error {
290		if err := os.Mkdir(tree, 0o755); err != nil {	435		if err := os.Mkdir(tree, 0o755); err != nil {
291		return err	436		return err
292		}	437		}
293	-	if err := extractTarGz(archive, tree); err != nil {	438	+	if err := extractTarGz(archive, tree, a.StripTop); err != nil {
294		return &verdictError{fmt.Errorf("unpack: %w", err)}	439		return &verdictError{fmt.Errorf("unpack: %w", err)}
295		}	440		}
296		if !isReady(tree) {	441		if !isReady(tree) {

M internal/externals/table.go +15 -17


@@ -8,8 +8,9 @@ import "runtime"
8	8		// the archive. The pins mirror actions/runner's Misc/externals.sh (node20
9	9		// 20.20.2, node24 24.21.0), so a workflow sees the Node.js it would see on a
10	10		// GitHub-hosted runner. The checksums of the nodejs.org builds come from the
11		-	// official SHASUMS256.txt of each release, and the archives are the .tar.gz
12		-	// ones (the Go standard library has no xz).
	11	+	// official SHASUMS256.txt of each release (those of the alpine builds are the
	12	+	// digests of the GitHub release assets), and the archives are the .tar.gz ones
	13	+	// (the Go standard library has no xz).
13	14		type Artifact struct {
14	15		// Name is the directory of the GitHub layout, below /home/runner/externals:
15	16		// node20, node24, node20_alpine, node24_alpine.

@@ -20,13 +21,11 @@ type Artifact struct {
20	21		Arch string
21	22		// URL is the https location of the .tar.gz.
22	23		URL string
23		-	// SHA256 is the lower-case hex SHA-256 of the archive. Empty means that the
24		-	// checksum has not been recorded yet: the artifact is then reported as
25		-	// unavailable (Pending says why) and nothing is downloaded.
	24	+	// SHA256 is the lower-case hex SHA-256 of the archive.
26	25		SHA256 string
27		-	// Pending is the explanation shown to a job that needs an artifact without
28		-	// a recorded checksum.
29		-	Pending string
	26	+	// StripTop: the archive has one top directory to drop (nodejs.org); the
	27	+	// actions/alpine_nodejs archives start at ./bin/node.
	28	+	StripTop bool
30	29		}
31	30
32	31		// dirName is the directory of the artifact below the externals directory.

@@ -45,20 +44,18 @@ const (
45	44
46	45		func nodejs(name, version, arch, sum string) Artifact {
47	46		return Artifact{
48		-	Name: name, Version: version, Arch: arch, SHA256: sum,
	47	+	Name: name, Version: version, Arch: arch, SHA256: sum, StripTop: true,
49	48		URL: nodeDist + "v" + version + "/node-v" + version + "-linux-" + arch + ".tar.gz",
50	49		}
51	50		}
52	51
	52	+	// alpine is a build of actions/alpine_nodejs (musl, x64 only), for job
	53	+	// containers based on Alpine.
53	54		func alpine(name, version, sum string) Artifact {
54		-	a := Artifact{
	55	+	return Artifact{
55	56		Name: name, Version: version, Arch: "x64", SHA256: sum,
56	57		URL: alpineDist + "v" + version + "/node-v" + version + "-alpine-x64.tar.gz",
57	58		}
58		-	if sum == "" {
59		-	a.Pending = "the checksum of the actions/alpine_nodejs build has not been recorded in this runner yet"
60		-	}
61		-	return a
62	59		}
63	60
64	61		// Pinned is every artifact the runner knows, for all architectures.

@@ -67,9 +64,10 @@ var Pinned = []Artifact{
67	64		nodejs("node20", node20Version, "arm64", "47ef73d543ecf6eb19435f6c03a0ac4809b3bf0dd6b26c7c571efc2a6572a74d"),
68	65		nodejs("node24", node24Version, "x64", "6e1db87ef58b8819e5d5402eff1536491b18edd8eb7bee5ef7897876e88dc5ff"),
69	66		nodejs("node24", node24Version, "arm64", "724282c3b43aec998aa9527380465b45d229e021b58035f5f4f63095eabfe5d5"),
70		-	// actions/runner ships the alpine builds for x64 only.
71		-	alpine("node20_alpine", node20Version, ""),
72		-	alpine("node24_alpine", node24Version, ""),
	67	+	// actions/runner ships the alpine builds for x64 only. The checksums are
	68	+	// those of the GitHub release assets.
	69	+	alpine("node20_alpine", node20Version, "f21a2253025a5d1a14332a0b1ed48871689c5ca9aa37a6141428944b75de7d91"),
	70	+	alpine("node24_alpine", node24Version, "2a4a9c6af9b6f1dd67e4c460474a6fbb64fafffe49e1166af80715bfd4e50775"),
73	71		}
74	72
75	73		// HostArch is GitHub's name for the architecture of this host ("" when the

 // the archive. The pins mirror actions/runner's Misc/externals.sh (node20
 // 20.20.2, node24 24.21.0), so a workflow sees the Node.js it would see on a
 // GitHub-hosted runner. The checksums of the nodejs.org builds come from the
-// official SHASUMS256.txt of each release, and the archives are the .tar.gz
+// official SHASUMS256.txt of each release (those of the alpine builds are the
-// ones (the Go standard library has no xz).
+// digests of the GitHub release assets), and the archives are the .tar.gz ones
+// (the Go standard library has no xz).
 type Artifact struct {
 	// Name is the directory of the GitHub layout, below /home/runner/externals:
 	// node20, node24, node20_alpine, node24_alpine.
 	Arch string
 	// URL is the https location of the .tar.gz.
 	URL string
-	// SHA256 is the lower-case hex SHA-256 of the archive. Empty means that the
+	// SHA256 is the lower-case hex SHA-256 of the archive.
-	// checksum has not been recorded yet: the artifact is then reported as
-	// unavailable (Pending says why) and nothing is downloaded.
 	SHA256 string
-	// Pending is the explanation shown to a job that needs an artifact without
+	// StripTop: the archive has one top directory to drop (nodejs.org); the
-	// a recorded checksum.
+	// actions/alpine_nodejs archives start at ./bin/node.
-	Pending string
+	StripTop bool
 }
 // dirName is the directory of the artifact below the externals directory.
 func nodejs(name, version, arch, sum string) Artifact {
 	return Artifact{
-		Name: name, Version: version, Arch: arch, SHA256: sum,
+		Name: name, Version: version, Arch: arch, SHA256: sum, StripTop: true,
 		URL: nodeDist + "v" + version + "/node-v" + version + "-linux-" + arch + ".tar.gz",
 	}
 }
+// alpine is a build of actions/alpine_nodejs (musl, x64 only), for job
+// containers based on Alpine.
 func alpine(name, version, sum string) Artifact {
-	a := Artifact{
+	return Artifact{
 		Name: name, Version: version, Arch: "x64", SHA256: sum,
 		URL: alpineDist + "v" + version + "/node-v" + version + "-alpine-x64.tar.gz",
 	}
-	if sum == "" {
-		a.Pending = "the checksum of the actions/alpine_nodejs build has not been recorded in this runner yet"
-	}
-	return a
 }
 // Pinned is every artifact the runner knows, for all architectures.
 	nodejs("node20", node20Version, "arm64", "47ef73d543ecf6eb19435f6c03a0ac4809b3bf0dd6b26c7c571efc2a6572a74d"),
 	nodejs("node24", node24Version, "x64", "6e1db87ef58b8819e5d5402eff1536491b18edd8eb7bee5ef7897876e88dc5ff"),
 	nodejs("node24", node24Version, "arm64", "724282c3b43aec998aa9527380465b45d229e021b58035f5f4f63095eabfe5d5"),
-	// actions/runner ships the alpine builds for x64 only.
+	// actions/runner ships the alpine builds for x64 only. The checksums are
-	alpine("node20_alpine", node20Version, ""),
+	// those of the GitHub release assets.
-	alpine("node24_alpine", node24Version, ""),
+	alpine("node20_alpine", node20Version, "f21a2253025a5d1a14332a0b1ed48871689c5ca9aa37a6141428944b75de7d91"),
+	alpine("node24_alpine", node24Version, "2a4a9c6af9b6f1dd67e4c460474a6fbb64fafffe49e1166af80715bfd4e50775"),
 }
 // HostArch is GitHub's name for the architecture of this host ("" when the