feat(ci): artifacts v4 storage and API (#39)

commit: 81486c6b3883d2964b1bfcad6eea2e0d89c53f64
author: brice · 6d ago · 2026-10-16 10:36 UTC
committer: bnema · 6d ago · 2026-10-16 10:36 UTC
parent: 0d0b8b2
signature: SSH UnverifiedSHA256:VOCA5lYARHcbKDZ7GhTV45VmcjWnJ6ZQIBazj17LivA The key is unknown

46 files · +8560 -521

M README.md +8 -1

 | `ERMITE_OWNER` | `owner` | owner login that CI shows in `github.repository` (`<owner>/<repo>`) and `github.repository_owner`; follows the rules of repository names |
 | `ERMITE_CI_LOG_RETENTION` | `90d` | how long CI runs, jobs, checks and logs are kept: `Nd` (days) or a Go duration such as `720h`, at least one hour |
 | `ERMITE_SECRETS_KEY` | empty | 32 random bytes in base64 (`head -c 32 /dev/urandom \| base64`) that encrypt CI secrets at rest (AES-256-GCM); keep it in the deployment secret store like `ERMITE_ADMIN_TOKEN`. Without it `ermite secret set` fails and jobs receive no secrets (variables and environments still work). The server refuses to start with another key than the one the stored secrets were sealed with; losing the key loses every secret (set them again), a backup without the key reveals none |
+| `ERMITE_CI_ARTIFACTS` | `1` | `0` turns the CI artifacts off: no artifact service, and artifact actions fail as on a GitHub Enterprise Server without artifacts v4 |
+| `ERMITE_CI_ARTIFACT_RETENTION` | `90d` | default and longest retention of an artifact (`retention-days`), at least one day: `Nd` or a Go duration |
+| `ERMITE_CI_ARTIFACT_QUOTA` | `10GiB` | space of the artifacts of one repository, uploads in progress included: bytes, or a number with `KiB`, `MiB`, `GiB`, `TiB` |
+| `ERMITE_CI_ARTIFACT_MAX` | `5GiB` | size of one artifact, same syntax |
 | `ERMITE_CI_POLL_TIMEOUT` | `50s` | how long the server holds a runner's `FetchJob` call when no job is queued, from `1s` to `55s`; set it below the response timeout of a reverse proxy in front of ermite (Gordon cuts a response that has no headers after 30 s, so use `25s` behind it) |
 `ermite version` prints the version (and the server's when a server URL is configured, see Remote CLI).
 ermite mr merge <repo> <n> --head <id> [--squash] [--delete-branch] | close <repo> <n>
 ermite run list <repo> [--workflow …] [--branch …] [--status …] | view <repo> <id> [--attempt n] [--log [--job …]]
 ermite run watch <repo> <id> [--log] | cancel <repo> <id> | rerun <repo> <id> [--failed]
+ermite run artifacts <repo> <id> | download <repo> <id> [--name N]… [--dir D]
 ermite runner list | remove <id>
 ermite web login                # prints a one-time sign-in link (60 s)
 ermite web logout --all         # signs every browser out
 `ermite run list|view|watch|cancel|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. `concurrency:` is supported at the workflow level (evaluated when the run starts) and at the job level (evaluated when the job is planned, once per matrix leg): a group is per repository and compared without regard to case, `queue: single` (the default) lets one run or job go and keeps one waiting, a newer waiting entry cancels the older one (`Canceling since a higher priority waiting request for '<group>' exists`), and `cancel-in-progress` (a boolean or an expression) also cancels the running one. With `queue: max`, up to 100 entries wait and run first in, first out; one more is cancelled (`Canceling since the concurrency group '<group>' already has 100 waiting requests`, ermite's wording). In a group that mixes both (GitHub does not say), ermite applies the rule of the entry that starts waiting: a `single` entry cancels every waiting entry, `max` ones included, and a `max` entry joins the end of the line. A job held back by a group shows `pending` in the API, the web pages and the CLI. A group that evaluates to nothing fails the job. `queue: max` with `cancel-in-progress: true` written as such makes the workflow invalid (its push records a failed run with the error, as GitHub's validation does); when either is an expression that evaluates to that pair, the job fails with the same message. The waiting line is stored with its head also in the field older ermite versions read, so a downgrade keeps the first waiting entry and cancels the others. Reusable workflows (`jobs.<id>.uses`) run: `./.github/workflows/x.yml` (the caller's repository, at the commit of the run), `owner/repo/.github/workflows/x.yml@ref` of a repository of this server when `owner` is `ERMITE_OWNER` and the actor of the run can read it (else the call fails as for a repository that does not exist), and otherwise a public repository on github.com, whose ref is resolved like the ref of an action and whose file is downloaded at that commit (at most 1 MiB). ermite expands the call when the job is planned: the called workflow must declare `on: workflow_call`, `with:` is checked against its inputs (types, `required`, defaults), `secrets:` passes the listed secrets, renamed as written, or all of them with `secrets: inherit` (a secret that is not passed is not there), and its `outputs` are available to the later jobs of the caller as `needs.<job>.outputs`. The jobs of the called workflow are named `<calling job> / <job>` and keyed `<calling job id>/<job id>`, wait for the `needs` of the calling job, and are repeated for each leg of its `strategy`; `permissions` can only be lowered by the called workflow. A call may nest four levels deep. `concurrency:` on the calling job and the workflow-level `concurrency:` of the called workflow (evaluated with the caller's `github` and the call's `inputs`) apply to the whole call, which holds the group until it ends, with `cancel-in-progress` as for any job. A workflow that uses another feature ermite does not support yet fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
-**Artifacts** are not supported. `actions/upload-artifact` and `actions/download-artifact` v4 and later (`@actions/artifact` v2) refuse any server that is not github.com, `*.ghe.com` or `*.localhost` and fail with `GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ and download-artifact@v4+ are not currently supported on GHES.`, as on a GitHub Enterprise Server. The cache (`actions/cache`, `setup-go`/`setup-node` `cache:`) is served by the runner and works.
+**Artifacts** (artifacts v4, `actions/upload-artifact` and `actions/download-artifact` v4 and later) are stored by ermite and served to the jobs by the runner, which translates the toolkit's protocol (ermite-api `JobArtifactService`, feature `ci.artifacts`). A job reaches the artifacts of its own run only, with its job token, while it runs; every job of the run, in any attempt, sees the finalized ones, and an artifact uploaded again by a re-run replaces the one of the earlier attempt. Contents are files under `$ERMITE_DATA_DIR/ci/artifacts`, named by ids (never by artifact name). Limits are checked before a byte is accepted: `ERMITE_CI_ARTIFACT_MAX` per artifact, `ERMITE_CI_ARTIFACT_QUOTA` per repository (uploads in progress included); `retention-days` is kept between one day and `ERMITE_CI_ARTIFACT_RETENTION`. A job has at most 32 unfinished uploads and a run at most 1000 artifacts. An upload not finished when its job ends is dropped. Artifacts go when they expire, with their run (retention) and with their repository. `ermite run artifacts <repo> <run>` lists them and `ermite run download <repo> <run> [--name N]… [--dir D]` downloads them (a zip is extracted into `D`, or `D/<name>` when there are several); the run page lists them with download links. They are never served anonymously, even for a public repository: listing and downloading need `repo:read`, deleting `ci:write`. Not covered: artifacts v3 (`@actions/artifact` v1), the REST endpoints of artifacts (`/actions/runs/{id}/artifacts`, `/actions/artifacts/{id}/zip`: `download-artifact` with `run-id` and `github-token` gets a 404).
+The cache (`actions/cache`, `setup-go`/`setup-node` `cache:`) is served by the runner and works.
 ## Web sign-in

A gen/ermite/storage/ci_artifact.pb.go +452 -0 generated file: collapsed

load diff

A internal/adapters/ciartifacts/ciartifacts.go +330 -0

@@ -0,0 +1,330 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	// Package ciartifacts implements ports.CIArtifactFiles under
	4	+	// $ERMITE_DATA_DIR/ci/artifacts, keyed by the global run id like the logs (a
	5	+	// repository rename never moves them):
	6	+	//
	7	+	// <run:020d>/<id:020d> the content of a committed artifact
	8	+	// <run:020d>/<id:020d>.blocks/<f> the staged blocks of a pending one
	9	+	// <run:020d>/.tmp-<id:020d>-* a content being assembled
	10	+	//
	11	+	// Paths are made of ids and of block file names the use case builds from
	12	+	// hexadecimal characters: an artifact name never reaches the file system.
	13	+	// Files are 0600 in 0700 directories, synced before they count, and a content
	14	+	// appears by rename, whole or not at all.
	15	+	package ciartifacts
	16	+
	17	+	import (
	18	+	"context"
	19	+	"crypto/sha256"
	20	+	"encoding/hex"
	21	+	"errors"
	22	+	"fmt"
	23	+	"io"
	24	+	"io/fs"
	25	+	"os"
	26	+	"path/filepath"
	27	+	"slices"
	28	+	"strconv"
	29	+	"strings"
	30	+
	31	+	"bnema.dev/ermite/internal/domain"
	32	+	"bnema.dev/ermite/internal/ports"
	33	+	)
	34	+
	35	+	// Dir is the directory of the artifacts under the data directory.
	36	+	func Dir(dataDir string) string { return filepath.Join(dataDir, "ci", "artifacts") }
	37	+
	38	+	const (
	39	+	idFormat = "%020d"
	40	+	blocksSuffix = ".blocks"
	41	+	tmpPrefix = ".tmp-"
	42	+	// maxFileName bounds a block file name (well under the 255 bytes of a
	43	+	// name on Linux).
	44	+	maxFileName = 200
	45	+	)
	46	+
	47	+	// Store is the file-backed artifact store.
	48	+	type Store struct{ root string }
	49	+
	50	+	var _ ports.CIArtifactFiles = (*Store)(nil)
	51	+
	52	+	// New returns a store rooted at dir, which need not exist yet.
	53	+	func New(dir string) *Store { return &Store{root: dir} }
	54	+
	55	+	func validRef(ref ports.CIArtifactRef) bool { return ref.RunID > 0 && ref.ID > 0 }
	56	+
	57	+	// validFile accepts the block file names of the use case: lowercase
	58	+	// hexadecimal characters and dots, not starting with a dot.
	59	+	func validFile(name string) bool {
	60	+	if name == "" \|\| len(name) > maxFileName \|\| name[0] == '.' {
	61	+	return false
	62	+	}
	63	+	return strings.Trim(name, "0123456789abcdef.") == ""
	64	+	}
	65	+
	66	+	func (s *Store) runDir(runID int64) string {
	67	+	return filepath.Join(s.root, fmt.Sprintf(idFormat, runID))
	68	+	}
	69	+
	70	+	func (s *Store) contentPath(ref ports.CIArtifactRef) string {
	71	+	return filepath.Join(s.runDir(ref.RunID), fmt.Sprintf(idFormat, ref.ID))
	72	+	}
	73	+
	74	+	func (s *Store) blocksDir(ref ports.CIArtifactRef) string {
	75	+	return s.contentPath(ref) + blocksSuffix
	76	+	}
	77	+
	78	+	// syncDir makes the entries of a directory durable.
	79	+	func syncDir(dir string) error {
	80	+	d, err := os.Open(dir)
	81	+	if err != nil {
	82	+	return err
	83	+	}
	84	+	serr := d.Sync()
	85	+	return errors.Join(serr, d.Close())
	86	+	}
	87	+
	88	+	// ctxReader stops a copy when its context ends.
	89	+	type ctxReader struct {
	90	+	ctx context.Context
	91	+	r io.Reader
	92	+	}
	93	+
	94	+	func (c ctxReader) Read(p []byte) (int, error) {
	95	+	if err := c.ctx.Err(); err != nil {
	96	+	return 0, err
	97	+	}
	98	+	return c.r.Read(p)
	99	+	}
	100	+
	101	+	// PutBlock writes a new block file, created exclusively.
	102	+	func (s *Store) PutBlock(ctx context.Context, ref ports.CIArtifactRef, file string, src io.Reader) (int64, error) {
	103	+	if !validRef(ref) \|\| !validFile(file) {
	104	+	return 0, fmt.Errorf("%w: block file %q", domain.ErrInvalid, file)
	105	+	}
	106	+	dir := s.blocksDir(ref)
	107	+	if err := os.MkdirAll(dir, 0o700); err != nil {
	108	+	return 0, fmt.Errorf("create block directory: %w", err)
	109	+	}
	110	+	path := filepath.Join(dir, file)
	111	+	f, err := os.OpenFile(path, os.O_WRONLY\|os.O_CREATE\|os.O_EXCL, 0o600)
	112	+	if err != nil {
	113	+	return 0, fmt.Errorf("create block: %w", err)
	114	+	}
	115	+	n, werr := io.Copy(f, ctxReader{ctx, src})
	116	+	var serr error
	117	+	if werr == nil {
	118	+	serr = f.Sync()
	119	+	}
	120	+	cerr := f.Close()
	121	+	if err := errors.Join(werr, serr, cerr); err != nil {
	122	+	_ = os.Remove(path)
	123	+	if werr != nil {
	124	+	return 0, werr // the source's error, as it is
	125	+	}
	126	+	return 0, fmt.Errorf("write block: %w", err)
	127	+	}
	128	+	if err := syncDir(dir); err != nil {
	129	+	_ = os.Remove(path)
	130	+	return 0, fmt.Errorf("sync block directory: %w", err)
	131	+	}
	132	+	return n, nil
	133	+	}
	134	+
	135	+	// RemoveBlock deletes one block file.
	136	+	func (s *Store) RemoveBlock(_ context.Context, ref ports.CIArtifactRef, file string) error {
	137	+	if !validRef(ref) \|\| !validFile(file) {
	138	+	return nil
	139	+	}
	140	+	if err := os.Remove(filepath.Join(s.blocksDir(ref), file)); err != nil && !errors.Is(err, fs.ErrNotExist) {
	141	+	return fmt.Errorf("remove block: %w", err)
	142	+	}
	143	+	return nil
	144	+	}
	145	+
	146	+	// Assemble writes the blocks into a temporary file, syncs it and renames it
	147	+	// over the content.
	148	+	func (s *Store) Assemble(ctx context.Context, ref ports.CIArtifactRef, files []string) (int64, string, error) {
	149	+	if !validRef(ref) {
	150	+	return 0, "", domain.ErrNotFound
	151	+	}
	152	+	for _, f := range files {
	153	+	if !validFile(f) {
	154	+	return 0, "", fmt.Errorf("%w: block file %q", domain.ErrInvalid, f)
	155	+	}
	156	+	}
	157	+	dir := s.runDir(ref.RunID)
	158	+	if err := os.MkdirAll(dir, 0o700); err != nil {
	159	+	return 0, "", fmt.Errorf("create artifact directory: %w", err)
	160	+	}
	161	+	out, err := os.CreateTemp(dir, tmpPrefix+fmt.Sprintf(idFormat, ref.ID)+"-*")
	162	+	if err != nil {
	163	+	return 0, "", fmt.Errorf("create artifact: %w", err)
	164	+	}
	165	+	tmp := out.Name()
	166	+	h := sha256.New()
	167	+	size, err := s.concat(ctx, ref, files, io.MultiWriter(out, h))
	168	+	if err == nil {
	169	+	err = out.Sync()
	170	+	}
	171	+	if cerr := out.Close(); err == nil {
	172	+	err = cerr
	173	+	}
	174	+	if err == nil {
	175	+	err = os.Rename(tmp, s.contentPath(ref))
	176	+	}
	177	+	if err != nil {
	178	+	_ = os.Remove(tmp)
	179	+	if errors.Is(err, domain.ErrNotFound) \|\| ctx.Err() != nil {
	180	+	return 0, "", err
	181	+	}
	182	+	return 0, "", fmt.Errorf("assemble artifact: %w", err)
	183	+	}
	184	+	if err := syncDir(dir); err != nil {
	185	+	return 0, "", fmt.Errorf("sync artifact directory: %w", err)
	186	+	}
	187	+	return size, "sha256:" + hex.EncodeToString(h.Sum(nil)), nil
	188	+	}
	189	+
	190	+	func (s *Store) concat(ctx context.Context, ref ports.CIArtifactRef, files []string, w io.Writer) (int64, error) {
	191	+	var size int64
	192	+	for _, name := range files {
	193	+	f, err := os.Open(filepath.Join(s.blocksDir(ref), name))
	194	+	if errors.Is(err, fs.ErrNotExist) {
	195	+	return 0, fmt.Errorf("%w: block file %s", domain.ErrNotFound, name)
	196	+	}
	197	+	if err != nil {
	198	+	return 0, err
	199	+	}
	200	+	n, err := io.Copy(w, ctxReader{ctx, f})
	201	+	_ = f.Close()
	202	+	if err != nil {
	203	+	return 0, err
	204	+	}
	205	+	size += n
	206	+	}
	207	+	return size, nil
	208	+	}
	209	+
	210	+	// Open opens the content at offset.
	211	+	func (s *Store) Open(_ context.Context, ref ports.CIArtifactRef, offset int64) (io.ReadCloser, error) {
	212	+	if !validRef(ref) \|\| offset < 0 {
	213	+	return nil, domain.ErrNotFound
	214	+	}
	215	+	f, err := os.Open(s.contentPath(ref))
	216	+	if errors.Is(err, fs.ErrNotExist) {
	217	+	return nil, domain.ErrNotFound
	218	+	}
	219	+	if err != nil {
	220	+	return nil, fmt.Errorf("open artifact: %w", err)
	221	+	}
	222	+	if _, err := f.Seek(offset, io.SeekStart); err != nil {
	223	+	_ = f.Close()
	224	+	return nil, fmt.Errorf("seek artifact: %w", err)
	225	+	}
	226	+	return f, nil
	227	+	}
	228	+
	229	+	// Remove deletes the content, the blocks and any assembly in progress.
	230	+	func (s *Store) Remove(ctx context.Context, ref ports.CIArtifactRef) error {
	231	+	if !validRef(ref) {
	232	+	return nil
	233	+	}
	234	+	var errs []error
	235	+	if err := os.Remove(s.contentPath(ref)); err != nil && !errors.Is(err, fs.ErrNotExist) {
	236	+	errs = append(errs, err)
	237	+	}
	238	+	errs = append(errs, s.RemoveBlocks(ctx, ref))
	239	+	tmps, _ := filepath.Glob(filepath.Join(s.runDir(ref.RunID), tmpPrefix+fmt.Sprintf(idFormat, ref.ID)+"-*"))
	240	+	for _, t := range tmps {
	241	+	if err := os.Remove(t); err != nil && !errors.Is(err, fs.ErrNotExist) {
	242	+	errs = append(errs, err)
	243	+	}
	244	+	}
	245	+	if err := errors.Join(errs...); err != nil {
	246	+	return fmt.Errorf("remove artifact: %w", err)
	247	+	}
	248	+	return nil
	249	+	}
	250	+
	251	+	// RemoveBlocks deletes the staging directory of the artifact.
	252	+	func (s *Store) RemoveBlocks(_ context.Context, ref ports.CIArtifactRef) error {
	253	+	if !validRef(ref) {
	254	+	return nil
	255	+	}
	256	+	if err := os.RemoveAll(s.blocksDir(ref)); err != nil {
	257	+	return fmt.Errorf("remove blocks: %w", err)
	258	+	}
	259	+	return nil
	260	+	}
	261	+
	262	+	// RemoveRuns deletes the directories of the runs.
	263	+	func (s *Store) RemoveRuns(_ context.Context, runIDs []int64) error {
	264	+	var errs []error
	265	+	for _, id := range runIDs {
	266	+	if id < 1 {
	267	+	continue
	268	+	}
	269	+	if err := os.RemoveAll(s.runDir(id)); err != nil {
	270	+	errs = append(errs, err)
	271	+	}
	272	+	}
	273	+	return errors.Join(errs...)
	274	+	}
	275	+
	276	+	// RunIDs lists the run directories.
	277	+	func (s *Store) RunIDs(_ context.Context) ([]int64, error) {
	278	+	entries, err := os.ReadDir(s.root)
	279	+	if errors.Is(err, fs.ErrNotExist) {
	280	+	return nil, nil
	281	+	}
	282	+	if err != nil {
	283	+	return nil, fmt.Errorf("list artifact directories: %w", err)
	284	+	}
	285	+	var ids []int64
	286	+	for _, e := range entries {
	287	+	if id, ok := parseID(e.Name()); ok && e.IsDir() {
	288	+	ids = append(ids, id)
	289	+	}
	290	+	}
	291	+	return ids, nil
	292	+	}
	293	+
	294	+	// ArtifactIDs lists the artifacts that have a content, blocks or an assembly
	295	+	// in progress in the run directory.
	296	+	func (s *Store) ArtifactIDs(_ context.Context, runID int64) ([]int64, error) {
	297	+	if runID < 1 {
	298	+	return nil, nil
	299	+	}
	300	+	entries, err := os.ReadDir(s.runDir(runID))
	301	+	if errors.Is(err, fs.ErrNotExist) {
	302	+	return nil, nil
	303	+	}
	304	+	if err != nil {
	305	+	return nil, fmt.Errorf("list artifacts: %w", err)
	306	+	}
	307	+	var ids []int64
	308	+	for _, e := range entries {
	309	+	name := e.Name()
	310	+	if rest, ok := strings.CutPrefix(name, tmpPrefix); ok {
	311	+	name, _, _ = strings.Cut(rest, "-")
	312	+	} else {
	313	+	name = strings.TrimSuffix(name, blocksSuffix)
	314	+	}
	315	+	if id, ok := parseID(name); ok && !slices.Contains(ids, id) {
	316	+	ids = append(ids, id)
	317	+	}
	318	+	}
	319	+	slices.Sort(ids)
	320	+	return ids, nil
	321	+	}
	322	+
	323	+	// parseID reads a zero-padded id.
	324	+	func parseID(name string) (int64, bool) {
	325	+	if len(name) != 20 {
	326	+	return 0, false
	327	+	}
	328	+	id, err := strconv.ParseInt(name, 10, 64)
	329	+	return id, err == nil && id > 0
	330	+	}

A internal/adapters/ciartifacts/ciartifacts_test.go +183 -0

@@ -0,0 +1,183 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package ciartifacts_test
	4	+
	5	+	import (
	6	+	"crypto/sha256"
	7	+	"encoding/hex"
	8	+	"errors"
	9	+	"io"
	10	+	"os"
	11	+	"path/filepath"
	12	+	"strings"
	13	+	"testing"
	14	+
	15	+	"github.com/stretchr/testify/assert"
	16	+	"github.com/stretchr/testify/require"
	17	+
	18	+	"bnema.dev/ermite/internal/adapters/ciartifacts"
	19	+	"bnema.dev/ermite/internal/domain"
	20	+	"bnema.dev/ermite/internal/ports"
	21	+	)
	22	+
	23	+	var ref = ports.CIArtifactRef{RunID: 7, ID: 42}
	24	+
	25	+	func digestOf(s string) string {
	26	+	sum := sha256.Sum256([]byte(s))
	27	+	return "sha256:" + hex.EncodeToString(sum[:])
	28	+	}
	29	+
	30	+	func entries(t *testing.T, dir string) []string {
	31	+	t.Helper()
	32	+	es, err := os.ReadDir(dir)
	33	+	if errors.Is(err, os.ErrNotExist) {
	34	+	return nil
	35	+	}
	36	+	require.NoError(t, err)
	37	+	var out []string
	38	+	for _, e := range es {
	39	+	out = append(out, e.Name())
	40	+	}
	41	+	return out
	42	+	}
	43	+
	44	+	func TestBlocksAssembleInTheOrderOfTheList(t *testing.T) {
	45	+	ctx := t.Context()
	46	+	root := t.TempDir()
	47	+	s := ciartifacts.New(root)
	48	+
	49	+	// Uploaded out of order; the list decides.
	50	+	for file, data := range map[string]string{"02.aa": "world", "01.bb": "hello ", "03.cc": "!"} {
	51	+	n, err := s.PutBlock(ctx, ref, file, strings.NewReader(data))
	52	+	require.NoError(t, err)
	53	+	assert.EqualValues(t, len(data), n)
	54	+	}
	55	+	size, digest, err := s.Assemble(ctx, ref, []string{"01.bb", "02.aa", "03.cc", "03.cc"})
	56	+	require.NoError(t, err)
	57	+	assert.EqualValues(t, len("hello world!!"), size)
	58	+	assert.Equal(t, digestOf("hello world!!"), digest, "a repeated id is repeated content")
	59	+
	60	+	// The documented layout, named by ids only.
	61	+	b, err := os.ReadFile(filepath.Join(root, "00000000000000000007", "00000000000000000042"))
	62	+	require.NoError(t, err)
	63	+	assert.Equal(t, "hello world!!", string(b))
	64	+	for _, e := range entries(t, filepath.Join(root, "00000000000000000007")) {
	65	+	assert.False(t, strings.HasPrefix(e, ".tmp-"), "no temporary file left: %s", e)
	66	+	}
	67	+
	68	+	// Offset reads.
	69	+	r, err := s.Open(ctx, ref, 6)
	70	+	require.NoError(t, err)
	71	+	rest, err := io.ReadAll(r)
	72	+	require.NoError(t, err)
	73	+	require.NoError(t, r.Close())
	74	+	assert.Equal(t, "world!!", string(rest))
	75	+
	76	+	// A new assembly replaces the content whole.
	77	+	size, digest, err = s.Assemble(ctx, ref, nil)
	78	+	require.NoError(t, err)
	79	+	assert.Zero(t, size, "an empty list is an empty content")
	80	+	assert.Equal(t, digestOf(""), digest)
	81	+
	82	+	ids, err := s.ArtifactIDs(ctx, 7)
	83	+	require.NoError(t, err)
	84	+	assert.Equal(t, []int64{42}, ids)
	85	+	runs, err := s.RunIDs(ctx)
	86	+	require.NoError(t, err)
	87	+	assert.Equal(t, []int64{7}, runs)
	88	+
	89	+	require.NoError(t, s.RemoveBlocks(ctx, ref))
	90	+	assert.Equal(t, []string{"00000000000000000042"}, entries(t, filepath.Join(root, "00000000000000000007")))
	91	+	require.NoError(t, s.Remove(ctx, ref))
	92	+	assert.Empty(t, entries(t, filepath.Join(root, "00000000000000000007")))
	93	+	_, err = s.Open(ctx, ref, 0)
	94	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	95	+	require.NoError(t, s.RemoveRuns(ctx, []int64{7}))
	96	+	assert.Empty(t, entries(t, root))
	97	+	}
	98	+
	99	+	func TestAMissingBlockIsNotFoundAndLeavesNoContent(t *testing.T) {
	100	+	ctx := t.Context()
	101	+	root := t.TempDir()
	102	+	s := ciartifacts.New(root)
	103	+	_, err := s.PutBlock(ctx, ref, "01.aa", strings.NewReader("x"))
	104	+	require.NoError(t, err)
	105	+	_, _, err = s.Assemble(ctx, ref, []string{"01.aa", "02.aa"})
	106	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	107	+	_, err = s.Open(ctx, ref, 0)
	108	+	assert.ErrorIs(t, err, domain.ErrNotFound, "no content appears")
	109	+	for _, e := range entries(t, filepath.Join(root, "00000000000000000007")) {
	110	+	assert.False(t, strings.HasPrefix(e, ".tmp-"), e)
	111	+	}
	112	+	}
	113	+
	114	+	type failingReader struct{ n int }
	115	+
	116	+	func (f *failingReader) Read(p []byte) (int, error) {
	117	+	if f.n == 0 {
	118	+	return 0, errors.New("stream broken")
	119	+	}
	120	+	f.n--
	121	+	p[0] = 'x'
	122	+	return 1, nil
	123	+	}
	124	+
	125	+	func TestABrokenSourceLeavesNoBlock(t *testing.T) {
	126	+	ctx := t.Context()
	127	+	root := t.TempDir()
	128	+	s := ciartifacts.New(root)
	129	+	_, err := s.PutBlock(ctx, ref, "01.aa", &failingReader{n: 3})
	130	+	require.EqualError(t, err, "stream broken", "the source's error as it is")
	131	+	assert.Empty(t, entries(t, filepath.Join(root, "00000000000000000007", "00000000000000000042.blocks")))
	132	+
	133	+	// A file name is never reused: the second put of a name fails.
	134	+	_, err = s.PutBlock(ctx, ref, "01.aa", strings.NewReader("a"))
	135	+	require.NoError(t, err)
	136	+	_, err = s.PutBlock(ctx, ref, "01.aa", strings.NewReader("b"))
	137	+	assert.Error(t, err)
	138	+	}
	139	+
	140	+	func TestNamesOutsideTheRulesNeverReachTheFileSystem(t *testing.T) {
	141	+	ctx := t.Context()
	142	+	root := t.TempDir()
	143	+	s := ciartifacts.New(filepath.Join(root, "store"))
	144	+	for _, name := range []string{"", "../x", "a/b", ".hidden", "..", "ZZ", "01.aa/../../x", strings.Repeat("a", 201)} {
	145	+	_, err := s.PutBlock(ctx, ref, name, strings.NewReader("x"))
	146	+	assert.ErrorIs(t, err, domain.ErrInvalid, "%q", name)
	147	+	_, _, err = s.Assemble(ctx, ref, []string{name})
	148	+	assert.Error(t, err, "%q", name)
	149	+	}
	150	+	for _, bad := range []ports.CIArtifactRef{{RunID: 0, ID: 1}, {RunID: 1, ID: -1}} {
	151	+	_, err := s.PutBlock(ctx, bad, "01.aa", strings.NewReader("x"))
	152	+	assert.ErrorIs(t, err, domain.ErrInvalid)
	153	+	_, err = s.Open(ctx, bad, 0)
	154	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	155	+	}
	156	+	for _, e := range entries(t, root) {
	157	+	assert.Equal(t, "store", e, "nothing written beside the store")
	158	+	}
	159	+	}
	160	+
	161	+	func TestOrphanListingSeesEveryKindOfFile(t *testing.T) {
	162	+	ctx := t.Context()
	163	+	root := t.TempDir()
	164	+	s := ciartifacts.New(root)
	165	+	run := filepath.Join(root, "00000000000000000009")
	166	+	require.NoError(t, os.MkdirAll(filepath.Join(run, "00000000000000000003.blocks"), 0o700))
	167	+	require.NoError(t, os.WriteFile(filepath.Join(run, "00000000000000000001"), nil, 0o600))
	168	+	require.NoError(t, os.WriteFile(filepath.Join(run, ".tmp-00000000000000000002-123"), nil, 0o600))
	169	+	require.NoError(t, os.WriteFile(filepath.Join(run, "junk"), nil, 0o600))
	170	+	require.NoError(t, os.MkdirAll(filepath.Join(root, "not-a-run"), 0o700))
	171	+
	172	+	ids, err := s.ArtifactIDs(ctx, 9)
	173	+	require.NoError(t, err)
	174	+	assert.Equal(t, []int64{1, 2, 3}, ids)
	175	+	runs, err := s.RunIDs(ctx)
	176	+	require.NoError(t, err)
	177	+	assert.Equal(t, []int64{9}, runs)
	178	+
	179	+	// Remove takes the assembly in progress too.
	180	+	require.NoError(t, s.Remove(ctx, ports.CIArtifactRef{RunID: 9, ID: 2}))
	181	+	ids, _ = s.ArtifactIDs(ctx, 9)
	182	+	assert.Equal(t, []int64{1, 3}, ids)
	183	+	}

M internal/adapters/client/client.go +5 -0


@@ -70,6 +70,11 @@ func (c *Client) Runs() ermiteconnect.RunServiceClient {
70	return ermiteconnect.NewRunServiceClient(c.http, c.base, c.opts...)	70		return ermiteconnect.NewRunServiceClient(c.http, c.base, c.opts...)
71	}	71		}
72		72
		73	+	// Artifacts returns the ArtifactService client.
		74	+	func (c *Client) Artifacts() ermiteconnect.ArtifactServiceClient {
		75	+	return ermiteconnect.NewArtifactServiceClient(c.http, c.base, c.opts...)
		76	+	}
		77	+
73	// Tokens returns the TokenService client.	78		// Tokens returns the TokenService client.
74	func (c *Client) Tokens() ermiteconnect.TokenServiceClient {	79		func (c *Client) Tokens() ermiteconnect.TokenServiceClient {
75	return ermiteconnect.NewTokenServiceClient(c.http, c.base, c.opts...)	80		return ermiteconnect.NewTokenServiceClient(c.http, c.base, c.opts...)

M internal/adapters/client/client_test.go +3 -3


@@ -139,16 +139,16 @@ func TestDescribeBoundsLongMessages(t *testing.T) {
139		func TestMissingFeatures(t *testing.T) {	139		func TestMissingFeatures(t *testing.T) {
140		assert.Empty(t, client.MissingFeatures(client.UsedFeatures()))	140		assert.Empty(t, client.MissingFeatures(client.UsedFeatures()))
141		assert.Equal(t,	141		assert.Equal(t,
142	-	[]string{client.FeatureEventsResume, client.FeatureMergeRequests, client.FeatureMergeDeleteBranch, client.FeatureSigningKey, client.FeatureTokenBranchPrefixes, client.FeatureCIRuns, client.FeatureCIRunners, client.FeatureCISecrets, client.FeatureCIDispatch},	142	+	[]string{client.FeatureEventsResume, client.FeatureMergeRequests, client.FeatureMergeDeleteBranch, client.FeatureSigningKey, client.FeatureTokenBranchPrefixes, client.FeatureCIRuns, client.FeatureCIRunners, client.FeatureCISecrets, client.FeatureCIDispatch, client.FeatureCIArtifacts},
143		client.MissingFeatures([]string{client.FeatureRepoUpdate, client.FeatureSession, client.FeaturePasskeys, "other"}))	143		client.MissingFeatures([]string{client.FeatureRepoUpdate, client.FeatureSession, client.FeaturePasskeys, "other"}))
144		assert.Equal(t, []string{client.FeatureSigningKey}, client.MissingFeatures([]string{	144		assert.Equal(t, []string{client.FeatureSigningKey}, client.MissingFeatures([]string{
145		client.FeatureRepoUpdate, client.FeatureEventsResume, client.FeatureMergeRequests, client.FeatureMergeDeleteBranch, client.FeatureSession,	145		client.FeatureRepoUpdate, client.FeatureEventsResume, client.FeatureMergeRequests, client.FeatureMergeDeleteBranch, client.FeatureSession,
146		client.FeatureTokenBranchPrefixes, client.FeatureCIRuns, client.FeatureCIRunners, client.FeatureCISecrets, client.FeatureCIDispatch,	146		client.FeatureTokenBranchPrefixes, client.FeatureCIRuns, client.FeatureCIRunners, client.FeatureCISecrets, client.FeatureCIDispatch,
147	-	client.FeaturePasskeys,	147	+	client.FeatureCIArtifacts, client.FeaturePasskeys,
148		}))	148		}))
149		assert.Equal(t, []string{client.FeaturePasskeys}, client.MissingFeatures([]string{	149		assert.Equal(t, []string{client.FeaturePasskeys}, client.MissingFeatures([]string{
150		client.FeatureRepoUpdate, client.FeatureEventsResume, client.FeatureMergeRequests, client.FeatureMergeDeleteBranch, client.FeatureSigningKey, client.FeatureSession,	150		client.FeatureRepoUpdate, client.FeatureEventsResume, client.FeatureMergeRequests, client.FeatureMergeDeleteBranch, client.FeatureSigningKey, client.FeatureSession,
151	-	client.FeatureTokenBranchPrefixes, client.FeatureCIRuns, client.FeatureCIRunners, client.FeatureCISecrets, client.FeatureCIDispatch,	151	+	client.FeatureTokenBranchPrefixes, client.FeatureCIRuns, client.FeatureCIRunners, client.FeatureCISecrets, client.FeatureCIDispatch, client.FeatureCIArtifacts,
152		}))	152		}))
153		assert.Equal(t, client.UsedFeatures(), client.MissingFeatures(nil))	153		assert.Equal(t, client.UsedFeatures(), client.MissingFeatures(nil))
154		}	154		}

M internal/adapters/client/features.go +4 -1

 	FeatureCISecrets = "ci.secrets"
 	// FeatureCIDispatch is RunService.DispatchWorkflow (`ermite workflow run`).
 	FeatureCIDispatch = "ci.dispatch"
+	// FeatureCIArtifacts is ArtifactService (`ermite run artifacts`,
+	// `run download`).
+	FeatureCIArtifacts = "ci.artifacts"
 )
 // UsedFeatures lists the features the CLI uses, in the order they are reported.
 func UsedFeatures() []string {
-	return []string{FeatureRepoUpdate, FeatureEventsResume, FeatureMergeRequests, FeatureMergeDeleteBranch, FeatureSigningKey, FeatureSession, FeatureTokenBranchPrefixes, FeatureCIRuns, FeatureCIRunners, FeatureCISecrets, FeatureCIDispatch, FeaturePasskeys}
+	return []string{FeatureRepoUpdate, FeatureEventsResume, FeatureMergeRequests, FeatureMergeDeleteBranch, FeatureSigningKey, FeatureSession, FeatureTokenBranchPrefixes, FeatureCIRuns, FeatureCIRunners, FeatureCISecrets, FeatureCIDispatch, FeatureCIArtifacts, FeaturePasskeys}
 }
 // MissingFeatures returns the features the CLI uses that have is lacking.

M internal/adapters/kvstore/ci.go +2 -1

 }
 // ciKeys lists every key of the snapshot that belongs to the repository's CI
-// runs, checks and numbers.
+// runs, checks, numbers and artifacts.
 func ciKeys(ctx context.Context, snap *kv.Snapshot, repo domain.RepoName) [][]byte {
 	c := &CI{}
 	var keys [][]byte
 		keys = append(keys, slices.Clone(key))
 	}
 	keys = append(keys, repoReaderKeys(ctx, snap, repo)...)
+	keys = append(keys, artifactKeys(ctx, snap, repo)...)
 	for _, p := range []string{checkPrefix + repo.String() + "/", runNumPrefix + repo.String() + "/", concurPrefix + repo.String() + "/", concurMarkPrefixAll + repo.String() + "/", refTipsPrefix(repo)} {
 		for k := range snap.Prefix(ctx, []byte(p)) {
 			keys = append(keys, slices.Clone(k))

A internal/adapters/kvstore/ci_artifact.go +372 -0

@@ -0,0 +1,372 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package kvstore
	4	+
	5	+	import (
	6	+	"context"
	7	+	"encoding/hex"
	8	+	"errors"
	9	+	"fmt"
	10	+	"slices"
	11	+	"strconv"
	12	+	"strings"
	13	+	"unicode/utf8"
	14	+
	15	+	kv "bnema.dev/ermite-kv"
	16	+	"google.golang.org/protobuf/proto"
	17	+
	18	+	storagepb "bnema.dev/ermite/gen/ermite/storage"
	19	+	"bnema.dev/ermite/internal/domain"
	20	+	"bnema.dev/ermite/internal/ports"
	21	+	)
	22	+
	23	+	// Key layout of the CI artifacts (CI-4 D3). New prefixes only: an older build
	24	+	// ignores them, so there is no schema change.
	25	+	//
	26	+	// artifact/<repo>/<run:020d>/<id:020d> CIArtifact
	27	+	// artifactid/<id:020d> index: "<repo>/<run:020d>"
	28	+	// artifactblk/<id:020d>/<hex(block id)> CIArtifactBlock: a staged block
	29	+	// artifactuse/<repo> CIArtifactUsage: bytes charged
	30	+	// artifactseq Sequence: last artifact id
	31	+	//
	32	+	// Every write of the artifacts of a repository requires and rewrites its
	33	+	// usage record (ports.CIArtifactWrite). The records of a repository go with it
	34	+	// (ciKeys); those of a run that retention removed are deleted by the sweep of
	35	+	// the artifacts, which refunds them.
	36	+	const (
	37	+	artifactPrefix = "artifact/"
	38	+	artifactIDPrefix = "artifactid/"
	39	+	artifactBlkPrefix = "artifactblk/"
	40	+	artifactUsePrefix = "artifactuse/"
	41	+	artifactSeqKey = "artifactseq"
	42	+	)
	43	+
	44	+	func artifactRunPrefix(repo domain.RepoName, runID int64) string {
	45	+	return artifactPrefix + repo.String() + "/" + fmt.Sprintf(idFormat, runID) + "/"
	46	+	}
	47	+	func artifactKey(repo domain.RepoName, runID, id int64) []byte {
	48	+	return []byte(artifactRunPrefix(repo, runID) + fmt.Sprintf(idFormat, id))
	49	+	}
	50	+	func artifactIDKey(id int64) []byte { return []byte(artifactIDPrefix + fmt.Sprintf(idFormat, id)) }
	51	+	func artifactBlocksPrefix(id int64) string {
	52	+	return artifactBlkPrefix + fmt.Sprintf(idFormat, id) + "/"
	53	+	}
	54	+	func artifactBlockKey(id int64, block []byte) []byte {
	55	+	return []byte(artifactBlocksPrefix(id) + hex.EncodeToString(block))
	56	+	}
	57	+	func artifactUseKey(repo domain.RepoName) []byte { return []byte(artifactUsePrefix + repo.String()) }
	58	+
	59	+	// CIArtifacts is the ermite-kv backed ports.CIArtifactStore. It shares the
	60	+	// database of the Store it comes from.
	61	+	type CIArtifacts struct {
	62	+	db *kv.DB
	63	+	}
	64	+
	65	+	var _ ports.CIArtifactStore = (*CIArtifacts)(nil)
	66	+
	67	+	// Artifacts returns the artifact store over the same database. It is valid
	68	+	// until the Store is closed.
	69	+	func (s Store) Artifacts() CIArtifacts { return &CIArtifacts{db: s.db} }
	70	+
	71	+	// --- encoding ---
	72	+
	73	+	var artifactStates = []struct {
	74	+	d domain.CIArtifactState
	75	+	p storagepb.CIArtifactState
	76	+	}{
	77	+	{domain.CIArtifactPending, storagepb.CIArtifactState_CI_ARTIFACT_STATE_PENDING},
	78	+	{domain.CIArtifactCommitted, storagepb.CIArtifactState_CI_ARTIFACT_STATE_COMMITTED},
	79	+	{domain.CIArtifactFinal, storagepb.CIArtifactState_CI_ARTIFACT_STATE_FINAL},
	80	+	}
	81	+
	82	+	func encodeArtifact(a domain.CIArtifact) ([]byte, error) {
	83	+	if a.Repo.IsZero() \|\| a.ID < 1 \|\| a.RunID < 1 {
	84	+	return nil, fmt.Errorf("%w: artifact without id, run or repository", domain.ErrInvalid)
	85	+	}
	86	+	if !utf8.ValidString(a.Name) \|\| !utf8.ValidString(a.MediaType) \|\| !utf8.ValidString(a.JobID) {
	87	+	return nil, fmt.Errorf("%w: artifact strings must be valid UTF-8", domain.ErrInvalid)
	88	+	}
	89	+	pb := &storagepb.CIArtifact{
	90	+	Id: a.ID, Repo: a.Repo.String(), RunId: a.RunID, Attempt: a.Attempt, JobId: a.JobID,
	91	+	Name: a.Name, MediaType: a.MediaType, Size: a.Size, Digest: a.Digest,
	92	+	CreatedAt: encodeTime(a.CreatedAt), ExpiresAt: encodeTime(a.ExpiresAt),
	93	+	Charged: a.Charged, ListDigest: a.ListDigest,
	94	+	}
	95	+	if a.Assignment != (domain.TokenHash{}) {
	96	+	pb.Assignment = a.Assignment[:]
	97	+	}
	98	+	for _, s := range artifactStates {
	99	+	if s.d == a.State {
	100	+	pb.State = s.p
	101	+	}
	102	+	}
	103	+	if pb.State == storagepb.CIArtifactState_CI_ARTIFACT_STATE_UNSPECIFIED {
	104	+	return nil, fmt.Errorf("%w: artifact state %q", domain.ErrInvalid, a.State)
	105	+	}
	106	+	return mustMarshal(pb), nil
	107	+	}
	108	+
	109	+	//hexcheck:ignore suspicious-business-logic-in-adapter decodes a record into the domain type: a storage concern, no policy
	110	+	func decodeArtifact(item kv.Item) (domain.CIArtifact, error) {
	111	+	var pb storagepb.CIArtifact
	112	+	if err := proto.Unmarshal(item.Value, &pb); err != nil {
	113	+	return domain.CIArtifact{}, fmt.Errorf("decode artifact: %w", err)
	114	+	}
	115	+	repo, err := domain.ParseRepoName(pb.GetRepo())
	116	+	if err != nil {
	117	+	return domain.CIArtifact{}, fmt.Errorf("decode artifact: %w", err)
	118	+	}
	119	+	a := domain.CIArtifact{
	120	+	ID: pb.GetId(), Repo: repo, RunID: pb.GetRunId(), Attempt: pb.GetAttempt(), JobID: pb.GetJobId(),
	121	+	Name: pb.GetName(), MediaType: pb.GetMediaType(), Size: pb.GetSize(), Digest: pb.GetDigest(),
	122	+	CreatedAt: decodeTime(pb.GetCreatedAt()), ExpiresAt: decodeTime(pb.GetExpiresAt()),
	123	+	Charged: pb.GetCharged(), ListDigest: pb.GetListDigest(), Version: item.Version,
	124	+	}
	125	+	copy(a.Assignment[:], pb.GetAssignment())
	126	+	for _, s := range artifactStates {
	127	+	if s.p == pb.GetState() {
	128	+	a.State = s.d
	129	+	}
	130	+	}
	131	+	if a.State == "" {
	132	+	return domain.CIArtifact{}, fmt.Errorf("decode artifact %d: unknown state %v", a.ID, pb.GetState())
	133	+	}
	134	+	return a, nil
	135	+	}
	136	+
	137	+	// --- reads ---
	138	+
	139	+	// artifactIDTries bounds the attempts of NextArtifactID: each conflict means
	140	+	// another call took an id, and uploads begin many at once.
	141	+	const artifactIDTries = 1000
	142	+
	143	+	// NextArtifactID takes the next id in its own batch.
	144	+	//
	145	+	//hexcheck:ignore suspicious-business-logic-in-adapter takes the next number under a version precondition: a storage concern, no policy
	146	+	func (c *CIArtifacts) NextArtifactID(ctx context.Context) (int64, error) {
	147	+	ci := &CI{db: c.db}
	148	+	for range artifactIDTries {
	149	+	last, item, ok, err := ci.readSeq(ctx, []byte(artifactSeqKey))
	150	+	if err != nil {
	151	+	return 0, err
	152	+	}
	153	+	var b kv.Batch
	154	+	stageSeq(&b, []byte(artifactSeqKey), item, ok, last+1)
	155	+	_, err = c.db.Apply(ctx, &b)
	156	+	if errors.Is(err, kv.ErrConflict) {
	157	+	continue
	158	+	}
	159	+	if err != nil {
	160	+	return 0, fmt.Errorf("take an artifact id: %w", err)
	161	+	}
	162	+	return last + 1, nil
	163	+	}
	164	+	return 0, fmt.Errorf("take an artifact id: %w", domain.ErrConflict)
	165	+	}
	166	+
	167	+	// ArtifactUsage reads artifactuse/<repo>.
	168	+	func (c *CIArtifacts) ArtifactUsage(ctx context.Context, repo domain.RepoName) (int64, uint64, error) {
	169	+	item, ok := c.db.Get(ctx, artifactUseKey(repo))
	170	+	if !ok {
	171	+	return 0, 0, nil
	172	+	}
	173	+	var pb storagepb.CIArtifactUsage
	174	+	if err := proto.Unmarshal(item.Value, &pb); err != nil {
	175	+	return 0, 0, fmt.Errorf("decode artifact usage: %w", err)
	176	+	}
	177	+	return pb.GetUsed(), item.Version, nil
	178	+	}
	179	+
	180	+	// GetArtifact resolves the id index, then reads the record.
	181	+	func (c *CIArtifacts) GetArtifact(ctx context.Context, id int64) (domain.CIArtifact, error) {
	182	+	snap := c.db.Snapshot(ctx)
	183	+	idx, ok := snap.Get(ctx, artifactIDKey(id))
	184	+	if !ok \|\| id < 1 {
	185	+	return domain.CIArtifact{}, domain.ErrNotFound
	186	+	}
	187	+	repo, runID, err := decodeArtifactIndex(idx.Value)
	188	+	if err != nil {
	189	+	return domain.CIArtifact{}, err
	190	+	}
	191	+	item, ok := snap.Get(ctx, artifactKey(repo, runID, id))
	192	+	if !ok {
	193	+	return domain.CIArtifact{}, domain.ErrNotFound
	194	+	}
	195	+	return decodeArtifact(item)
	196	+	}
	197	+
	198	+	func encodeArtifactIndex(a domain.CIArtifact) []byte {
	199	+	return []byte(a.Repo.String() + "/" + fmt.Sprintf(idFormat, a.RunID))
	200	+	}
	201	+
	202	+	func decodeArtifactIndex(val []byte) (domain.RepoName, int64, error) {
	203	+	name, run, ok := strings.Cut(string(val), "/")
	204	+	if !ok {
	205	+	return domain.RepoName{}, 0, fmt.Errorf("decode artifact index %q", val)
	206	+	}
	207	+	repo, err := domain.ParseRepoName(name)
	208	+	if err != nil {
	209	+	return domain.RepoName{}, 0, fmt.Errorf("decode artifact index: %w", err)
	210	+	}
	211	+	runID, err := strconv.ParseInt(run, 10, 64)
	212	+	if err != nil {
	213	+	return domain.RepoName{}, 0, fmt.Errorf("decode artifact index: %w", err)
	214	+	}
	215	+	return repo, runID, nil
	216	+	}
	217	+
	218	+	// RunArtifacts lists artifact/<repo>/<run>/, which is in id order.
	219	+	func (c *CIArtifacts) RunArtifacts(ctx context.Context, repo domain.RepoName, runID int64) ([]domain.CIArtifact, error) {
	220	+	return c.list(ctx, artifactRunPrefix(repo, runID))
	221	+	}
	222	+
	223	+	// AllArtifacts lists artifact/.
	224	+	func (c *CIArtifacts) AllArtifacts(ctx context.Context) ([]domain.CIArtifact, error) {
	225	+	return c.list(ctx, artifactPrefix)
	226	+	}
	227	+
	228	+	func (c *CIArtifacts) list(ctx context.Context, prefix string) ([]domain.CIArtifact, error) {
	229	+	var out []domain.CIArtifact
	230	+	for key, item := range c.db.Snapshot(ctx).Prefix(ctx, []byte(prefix)) {
	231	+	a, err := decodeArtifact(item)
	232	+	if err != nil {
	233	+	return nil, fmt.Errorf("decode %s: %w", key, err)
	234	+	}
	235	+	out = append(out, a)
	236	+	}
	237	+	return out, nil
	238	+	}
	239	+
	240	+	// ArtifactBlocks lists artifactblk/<id>/.
	241	+	func (c *CIArtifacts) ArtifactBlocks(ctx context.Context, id int64) ([]ports.CIArtifactBlock, error) {
	242	+	prefix := artifactBlocksPrefix(id)
	243	+	var out []ports.CIArtifactBlock
	244	+	for key, item := range c.db.Snapshot(ctx).Prefix(ctx, []byte(prefix)) {
	245	+	block, err := hex.DecodeString(strings.TrimPrefix(string(key), prefix))
	246	+	if err != nil {
	247	+	return nil, fmt.Errorf("decode %s: %w", key, err)
	248	+	}
	249	+	var pb storagepb.CIArtifactBlock
	250	+	if err := proto.Unmarshal(item.Value, &pb); err != nil {
	251	+	return nil, fmt.Errorf("decode %s: %w", key, err)
	252	+	}
	253	+	out = append(out, ports.CIArtifactBlock{ArtifactID: id, ID: block, Size: pb.GetSize(), File: pb.GetFile()})
	254	+	}
	255	+	return out, nil
	256	+	}
	257	+
	258	+	// BlockArtifactIDs lists the artifact ids under artifactblk/.
	259	+	func (c *CIArtifacts) BlockArtifactIDs(ctx context.Context) ([]int64, error) {
	260	+	var ids []int64
	261	+	for key := range c.db.Snapshot(ctx).Prefix(ctx, []byte(artifactBlkPrefix)) {
	262	+	raw, _, _ := strings.Cut(strings.TrimPrefix(string(key), artifactBlkPrefix), "/")
	263	+	id, err := strconv.ParseInt(raw, 10, 64)
	264	+	if err != nil {
	265	+	return nil, fmt.Errorf("decode %s: %w", key, err)
	266	+	}
	267	+	if len(ids) == 0 \|\| ids[len(ids)-1] != id {
	268	+	ids = append(ids, id)
	269	+	}
	270	+	}
	271	+	return ids, nil
	272	+	}
	273	+
	274	+	// --- writes ---
	275	+
	276	+	// ApplyArtifacts writes w in one batch.
	277	+	//
	278	+	//hexcheck:ignore suspicious-business-logic-in-adapter stages records and their derived keys under version preconditions: a storage concern, no policy
	279	+	func (c *CIArtifacts) ApplyArtifacts(ctx context.Context, w ports.CIArtifactWrite) error {
	280	+	var b kv.Batch
	281	+	guard := false
	282	+	if w.UsageVersion == 0 {
	283	+	b.RequireAbsent(artifactUseKey(w.Repo))
	284	+	guard = true
	285	+	} else {
	286	+	b.RequireVersion(artifactUseKey(w.Repo), w.UsageVersion)
	287	+	}
	288	+	b.Put(artifactUseKey(w.Repo), mustMarshal(&storagepb.CIArtifactUsage{Used: w.Used}))
	289	+	snap := c.db.Snapshot(ctx)
	290	+	for _, a := range w.Put {
	291	+	if a.Repo != w.Repo {
	292	+	return fmt.Errorf("%w: artifact %d of another repository", domain.ErrInvalid, a.ID)
	293	+	}
	294	+	val, err := encodeArtifact(a)
	295	+	if err != nil {
	296	+	return err
	297	+	}
	298	+	stage(&b, artifactKey(a.Repo, a.RunID, a.ID), val, a.Version)
	299	+	if a.Version == 0 {
	300	+	b.RequireAbsent(artifactIDKey(a.ID))
	301	+	b.Put(artifactIDKey(a.ID), encodeArtifactIndex(a))
	302	+	guard = true
	303	+	}
	304	+	}
	305	+	for _, a := range w.Delete {
	306	+	if a.Repo != w.Repo \|\| a.Version == 0 {
	307	+	return fmt.Errorf("%w: delete of artifact %d", domain.ErrInvalid, a.ID)
	308	+	}
	309	+	key := artifactKey(a.Repo, a.RunID, a.ID)
	310	+	b.RequireVersion(key, a.Version)
	311	+	b.Delete(key)
	312	+	b.Delete(artifactIDKey(a.ID))
	313	+	for k := range snap.Prefix(ctx, []byte(artifactBlocksPrefix(a.ID))) {
	314	+	b.Delete(slices.Clone(k))
	315	+	}
	316	+	}
	317	+	for _, id := range w.DropBlocks {
	318	+	for k := range snap.Prefix(ctx, []byte(artifactBlocksPrefix(id))) {
	319	+	b.Delete(slices.Clone(k))
	320	+	}
	321	+	}
	322	+	for _, blk := range w.PutBlocks {
	323	+	if !utf8.ValidString(blk.File) {
	324	+	return fmt.Errorf("%w: block file name", domain.ErrInvalid)
	325	+	}
	326	+	b.Put(artifactBlockKey(blk.ArtifactID, blk.ID), mustMarshal(&storagepb.CIArtifactBlock{Size: blk.Size, File: blk.File}))
	327	+	}
	328	+	if guard {
	329	+	if err := guardRepoKey(ctx, c.db, &b, w.Repo); err != nil {
	330	+	return err
	331	+	}
	332	+	}
	333	+	_, err := c.db.Apply(ctx, &b)
	334	+	if errors.Is(err, kv.ErrConflict) {
	335	+	return domain.ErrConflict
	336	+	}
	337	+	if err != nil {
	338	+	return fmt.Errorf("write the artifacts of %q: %w", w.Repo, err)
	339	+	}
	340	+	return nil
	341	+	}
	342	+
	343	+	// artifactKeys lists every key of the snapshot that belongs to the artifacts
	344	+	// of the repository: records, their id index and blocks, and the usage record.
	345	+	func artifactKeys(ctx context.Context, snap *kv.Snapshot, repo domain.RepoName) [][]byte {
	346	+	keys := [][]byte{artifactUseKey(repo)}
	347	+	for key, item := range snap.Prefix(ctx, []byte(artifactPrefix+repo.String()+"/")) {
	348	+	keys = append(keys, slices.Clone(key))
	349	+	if a, err := decodeArtifact(item); err == nil {
	350	+	keys = append(keys, artifactIDKey(a.ID))
	351	+	for k := range snap.Prefix(ctx, []byte(artifactBlocksPrefix(a.ID))) {
	352	+	keys = append(keys, slices.Clone(k))
	353	+	}
	354	+	}
	355	+	}
	356	+	return keys
	357	+	}
	358	+
	359	+	// DeleteBlocks removes artifactblk/<id>/.
	360	+	func (c *CIArtifacts) DeleteBlocks(ctx context.Context, id int64) error {
	361	+	var b kv.Batch
	362	+	for k := range c.db.Snapshot(ctx).Prefix(ctx, []byte(artifactBlocksPrefix(id))) {
	363	+	b.Delete(slices.Clone(k))
	364	+	}
	365	+	if b.Len() == 0 {
	366	+	return nil
	367	+	}
	368	+	if _, err := c.db.Apply(ctx, &b); err != nil {
	369	+	return fmt.Errorf("delete the blocks of artifact %d: %w", id, err)
	370	+	}
	371	+	return nil
	372	+	}

A internal/adapters/kvstore/ci_artifact_test.go +172 -0

@@ -0,0 +1,172 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package kvstore_test
	4	+
	5	+	import (
	6	+	"context"
	7	+	"sync"
	8	+	"testing"
	9	+	"time"
	10	+
	11	+	"github.com/stretchr/testify/assert"
	12	+	"github.com/stretchr/testify/require"
	13	+
	14	+	"bnema.dev/ermite/internal/domain"
	15	+	"bnema.dev/ermite/internal/ports"
	16	+	)
	17	+
	18	+	func pendingArtifact(repo domain.RepoName, run, id int64, name string) domain.CIArtifact {
	19	+	return domain.CIArtifact{
	20	+	ID: id, Repo: repo, RunID: run, Attempt: 1, JobID: "1-0-0", Assignment: domain.HashTokenSecret("ermitejob_x"),
	21	+	Name: name, MediaType: domain.CIArtifactZip, State: domain.CIArtifactPending,
	22	+	CreatedAt: created, ExpiresAt: created.Add(24 * time.Hour),
	23	+	}
	24	+	}
	25	+
	26	+	func TestCIArtifactsRoundTripAndIndexes(t *testing.T) {
	27	+	ctx := context.Background()
	28	+	s, _, n := ciRepo(t)
	29	+	arts := s.Artifacts()
	30	+
	31	+	used, ver, err := arts.ArtifactUsage(ctx, n)
	32	+	require.NoError(t, err)
	33	+	assert.Zero(t, used)
	34	+	assert.Zero(t, ver)
	35	+
	36	+	id1, err := arts.NextArtifactID(ctx)
	37	+	require.NoError(t, err)
	38	+	id2, err := arts.NextArtifactID(ctx)
	39	+	require.NoError(t, err)
	40	+	assert.Greater(t, id2, id1, "ids increase")
	41	+
	42	+	a := pendingArtifact(n, 7, id1, "dist")
	43	+	b := pendingArtifact(n, 7, id2, "logs")
	44	+	require.NoError(t, arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Put: []domain.CIArtifact{a, b}}))
	45	+
	46	+	got, err := arts.GetArtifact(ctx, id1)
	47	+	require.NoError(t, err)
	48	+	assert.Equal(t, "dist", got.Name)
	49	+	assert.Equal(t, a.Assignment, got.Assignment)
	50	+	assert.Equal(t, domain.CIArtifactPending, got.State)
	51	+	assert.True(t, got.ExpiresAt.Equal(a.ExpiresAt))
	52	+	assert.NotZero(t, got.Version)
	53	+
	54	+	list, err := arts.RunArtifacts(ctx, n, 7)
	55	+	require.NoError(t, err)
	56	+	require.Len(t, list, 2)
	57	+	assert.Equal(t, []int64{id1, id2}, []int64{list[0].ID, list[1].ID}, "by id")
	58	+	other, err := arts.RunArtifacts(ctx, n, 8)
	59	+	require.NoError(t, err)
	60	+	assert.Empty(t, other)
	61	+
	62	+	// A second create of the same id is a conflict.
	63	+	err = arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, UsageVersion: 0, Put: []domain.CIArtifact{a}})
	64	+	assert.ErrorIs(t, err, domain.ErrConflict)
	65	+
	66	+	// Blocks, and the usage precondition.
	67	+	_, ver, err = arts.ArtifactUsage(ctx, n)
	68	+	require.NoError(t, err)
	69	+	got.Charged = 10
	70	+	blk := ports.CIArtifactBlock{ArtifactID: id1, ID: []byte{0, 1, 0xff}, Size: 10, File: "0001ff.ab"}
	71	+	require.NoError(t, arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Used: 10, UsageVersion: ver, Put: []domain.CIArtifact{got}, PutBlocks: []ports.CIArtifactBlock{blk}}))
	72	+	err = arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Used: 99, UsageVersion: ver})
	73	+	assert.ErrorIs(t, err, domain.ErrConflict, "the usage moved since it was read")
	74	+	used, _, err = arts.ArtifactUsage(ctx, n)
	75	+	require.NoError(t, err)
	76	+	assert.EqualValues(t, 10, used)
	77	+	blocks, err := arts.ArtifactBlocks(ctx, id1)
	78	+	require.NoError(t, err)
	79	+	assert.Equal(t, []ports.CIArtifactBlock{blk}, blocks)
	80	+	ids, err := arts.BlockArtifactIDs(ctx)
	81	+	require.NoError(t, err)
	82	+	assert.Equal(t, []int64{id1}, ids)
	83	+
	84	+	// A stale record version is a conflict too.
	85	+	_, ver, _ = arts.ArtifactUsage(ctx, n)
	86	+	err = arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Used: 10, UsageVersion: ver, Put: []domain.CIArtifact{got}})
	87	+	assert.ErrorIs(t, err, domain.ErrConflict)
	88	+
	89	+	// Delete takes the record, its index and its blocks.
	90	+	cur, err := arts.GetArtifact(ctx, id1)
	91	+	require.NoError(t, err)
	92	+	_, ver, _ = arts.ArtifactUsage(ctx, n)
	93	+	require.NoError(t, arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Used: 0, UsageVersion: ver, Delete: []domain.CIArtifact{cur}}))
	94	+	_, err = arts.GetArtifact(ctx, id1)
	95	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	96	+	blocks, err = arts.ArtifactBlocks(ctx, id1)
	97	+	require.NoError(t, err)
	98	+	assert.Empty(t, blocks)
	99	+	all, err := arts.AllArtifacts(ctx)
	100	+	require.NoError(t, err)
	101	+	require.Len(t, all, 1)
	102	+	assert.Equal(t, id2, all[0].ID)
	103	+
	104	+	_, err = arts.GetArtifact(ctx, 0)
	105	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	106	+	}
	107	+
	108	+	func TestCIArtifactIDsAreUniqueUnderConcurrency(t *testing.T) {
	109	+	ctx := context.Background()
	110	+	s, _, _ := ciRepo(t)
	111	+	arts := s.Artifacts()
	112	+	var mu sync.Mutex
	113	+	seen := map[int64]bool{}
	114	+	var wg sync.WaitGroup
	115	+	for range 8 {
	116	+	wg.Go(func() {
	117	+	for range 10 {
	118	+	id, err := arts.NextArtifactID(ctx)
	119	+	assert.NoError(t, err)
	120	+	mu.Lock()
	121	+	assert.False(t, seen[id], "id %d given twice", id)
	122	+	seen[id] = true
	123	+	mu.Unlock()
	124	+	}
	125	+	})
	126	+	}
	127	+	wg.Wait()
	128	+	assert.Len(t, seen, 80)
	129	+	}
	130	+
	131	+	func TestCIArtifactsGoWithTheRepository(t *testing.T) {
	132	+	ctx := context.Background()
	133	+	s, _, n := ciRepo(t)
	134	+	arts := s.Artifacts()
	135	+	a := pendingArtifact(n, 3, 1, "dist")
	136	+	blk := ports.CIArtifactBlock{ArtifactID: 1, ID: []byte("b"), Size: 4, File: "62.00"}
	137	+	require.NoError(t, arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Used: 4, Put: []domain.CIArtifact{a}, PutBlocks: []ports.CIArtifactBlock{blk}}))
	138	+
	139	+	require.NoError(t, s.Delete(ctx, n))
	140	+	_, err := arts.GetArtifact(ctx, 1)
	141	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	142	+	all, err := arts.AllArtifacts(ctx)
	143	+	require.NoError(t, err)
	144	+	assert.Empty(t, all)
	145	+	ids, err := arts.BlockArtifactIDs(ctx)
	146	+	require.NoError(t, err)
	147	+	assert.Empty(t, ids)
	148	+	used, ver, err := arts.ArtifactUsage(ctx, n)
	149	+	require.NoError(t, err)
	150	+	assert.Zero(t, used)
	151	+	assert.Zero(t, ver)
	152	+
	153	+	// No write recreates keys under the deleted repository.
	154	+	err = arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Put: []domain.CIArtifact{pendingArtifact(n, 3, 2, "x")}})
	155	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	156	+	all, _ = arts.AllArtifacts(ctx)
	157	+	assert.Empty(t, all)
	158	+	}
	159	+
	160	+	func TestCIArtifactDeleteBlocks(t *testing.T) {
	161	+	ctx := context.Background()
	162	+	s, _, n := ciRepo(t)
	163	+	arts := s.Artifacts()
	164	+	a := pendingArtifact(n, 3, 5, "dist")
	165	+	blks := []ports.CIArtifactBlock{{ArtifactID: 5, ID: []byte("a"), Size: 1, File: "61.00"}, {ArtifactID: 5, ID: []byte("b"), Size: 1, File: "62.00"}}
	166	+	require.NoError(t, arts.ApplyArtifacts(ctx, ports.CIArtifactWrite{Repo: n, Used: 2, Put: []domain.CIArtifact{a}, PutBlocks: blks}))
	167	+	require.NoError(t, arts.DeleteBlocks(ctx, 5))
	168	+	got, err := arts.ArtifactBlocks(ctx, 5)
	169	+	require.NoError(t, err)
	170	+	assert.Empty(t, got)
	171	+	require.NoError(t, arts.DeleteBlocks(ctx, 5), "nothing left is fine")
	172	+	}

M internal/adapters/kvstore/repos.go +1 -1


@@ -123,7 +123,7 @@ const deleteTries = 32
123			123
124		// Delete removes the repository and everything keyed to it: its merge	124		// Delete removes the repository and everything keyed to it: its merge
125		// requests, their comments and read positions, and its number sequence, and	125		// requests, their comments and read positions, and its number sequence, and
126	-	// its CI runs, jobs, checks and run numbers. The	126	+	// its CI runs, jobs, checks, run numbers and artifacts. The
127		// Git data is moved to the trash by the caller; the merge requests are not	127		// Git data is moved to the trash by the caller; the merge requests are not
128		// kept anywhere, like the repository metadata, so a repository created again	128		// kept anywhere, like the repository metadata, so a repository created again
129		// under the same name starts clean, with its numbers starting at 1. A missing	129		// under the same name starts clean, with its numbers starting at 1. A missing

M internal/adapters/rpc/api.go +21 -4

 // bounded: a diff has its own limits and blobs are streamed.
 const MaxRequestBytes = 1 << 20
-// RunnerMaxRequestBytes bounds the size of a request message of RunnerService:
+// RunnerMaxRequestBytes bounds the size of a request message of RunnerService
-// a job summary of exactly 1 MiB (UploadSummary) must be accepted, so the limit
+// and JobArtifactService: a job summary of exactly 1 MiB (UploadSummary) and a
-// leaves room for the envelope (job id, step number, framing).
+// block message with 1 MiB of data must be accepted, so the limit leaves room
+// for the envelope (job id, step number, block id, framing).
 const RunnerMaxRequestBytes = MaxRequestBytes + 64<<10
 // Features are the optional capabilities of the contract that this server
 var Features = []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.dispatch", "ci.cache"}
 // FeaturePasskeys is listed by GetVersion when passkeys are available.
+// FeatureCIArtifacts (artifact.go) when the artifact services are served.
 const FeaturePasskeys = "passkeys"
 // FeatureCIOIDC is listed by GetVersion when the server issues OIDC ID tokens
 	// issues none (GetVersion then does not list "ci.oidc"). Same rule as
 	// Sessions about typed nil pointers.
 	OIDCKeys ports.OIDCKeyService
+	// JobArtifacts and Artifacts serve JobArtifactService and ArtifactService,
+	// and GetVersion lists "ci.artifacts", when both are set and Runner is.
+	// Same rule as Sessions about typed nil pointers.
+	JobArtifacts ports.JobArtifactService
+	Artifacts    ports.ArtifactService
 }
 // API is the set of Connect handlers of the contract.
 	if settings {
 		features = append(features, FeatureCISecrets)
 	}
+	artifacts := cfg.JobArtifacts != nil && cfg.Artifacts != nil && cfg.Runner != nil
+	if artifacts {
+		features = append(features, FeatureCIArtifacts)
+	}
 	if cfg.OIDCKeys != nil {
 		features = append(features, FeatureCIOIDC)
 	}
 	}
 	if cfg.Runner != nil {
 		runnerOpts := append(slices.Clone(opts), connect.WithReadMaxBytes(RunnerMaxRequestBytes))
-		mux.Handle(ermiteconnect.NewRunnerServiceHandler(&runnerService{runners: cfg.Runner, limiter: cfg.Limiter}, runnerOpts...))
+		runners := &runnerService{runners: cfg.Runner, limiter: cfg.Limiter}
+		mux.Handle(ermiteconnect.NewRunnerServiceHandler(runners, runnerOpts...))
+		if artifacts {
+			// A block message carries up to 1 MiB of data, like a summary.
+			mux.Handle(ermiteconnect.NewJobArtifactServiceHandler(&jobArtifactService{artifacts: cfg.JobArtifacts, runner: runners}, runnerOpts...))
+			mux.Handle(ermiteconnect.NewArtifactServiceHandler(&artifactService{artifacts: cfg.Artifacts}, opts...))
+		}
 	}
 	if cfg.Runs != nil {
 		mux.Handle(ermiteconnect.NewRunServiceHandler(&runService{runs: cfg.Runs}, opts...))

A internal/adapters/rpc/artifact.go +193 -0

@@ -0,0 +1,193 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package rpc
	4	+
	5	+	import (
	6	+	"context"
	7	+	"fmt"
	8	+	"io"
	9	+
	10	+	"connectrpc.com/connect"
	11	+
	12	+	pb "bnema.dev/ermite-api/gen/ermite"
	13	+	"bnema.dev/ermite/internal/domain"
	14	+	"bnema.dev/ermite/internal/ports"
	15	+	)
	16	+
	17	+	// FeatureCIArtifacts is listed by GetVersion when the artifact services are
	18	+	// served.
	19	+	const FeatureCIArtifacts = "ci.artifacts"
	20	+
	21	+	func artifactToPB(a domain.CIArtifact) *pb.Artifact {
	22	+	return &pb.Artifact{
	23	+	Id: a.ID, Repo: a.Repo.String(), RunId: a.RunID, Attempt: a.Attempt, JobId: a.JobID,
	24	+	Name: a.Name, Size: a.Size, Digest: a.Digest, MediaType: a.MediaType,
	25	+	CreatedAt: timestamp(a.CreatedAt), ExpiresAt: timestamp(a.ExpiresAt),
	26	+	}
	27	+	}
	28	+
	29	+	// jobArtifactService implements ermiteconnect.JobArtifactServiceHandler.
	30	+	type jobArtifactService struct {
	31	+	artifacts ports.JobArtifactService
	32	+	runner *runnerService // failed: the failures of job tokens count like the runner's
	33	+	}
	34	+
	35	+	func (s jobArtifactService) CreateJobArtifact(ctx context.Context, req connect.Request[pb.CreateJobArtifactRequest]) (*connect.Response[pb.CreateJobArtifactResponse], error) {
	36	+	out, err := s.artifacts.CreateJobArtifact(ctx, secretFrom(ctx), ports.CreateJobArtifactParams{
	37	+	JobID: req.Msg.GetJobId(), Name: req.Msg.GetName(), MediaType: req.Msg.GetMediaType(), ExpiresAt: goTime(req.Msg.GetExpiresAt()),
	38	+	})
	39	+	if err != nil {
	40	+	return nil, s.runner.failed(ctx, err)
	41	+	}
	42	+	return connect.NewResponse(&pb.CreateJobArtifactResponse{ArtifactId: out.ID, ExpiresAt: timestamp(out.ExpiresAt)}), nil
	43	+	}
	44	+
	45	+	// blockReader reads the data of a block stream: the first message's, then
	46	+	// the following ones', which must carry data only.
	47	+	type blockReader struct {
	48	+	stream *connect.ClientStream[pb.UploadJobArtifactBlockRequest]
	49	+	buf []byte
	50	+	}
	51	+
	52	+	func checkBlockData(data []byte) error {
	53	+	if len(data) > domain.CIArtifactChunk {
	54	+	return fmt.Errorf("%w: a message carries at most %d bytes of data", domain.ErrInvalid, domain.CIArtifactChunk)
	55	+	}
	56	+	return nil
	57	+	}
	58	+
	59	+	func (r *blockReader) Read(p []byte) (int, error) {
	60	+	for len(r.buf) == 0 {
	61	+	if !r.stream.Receive() {
	62	+	if err := r.stream.Err(); err != nil {
	63	+	return 0, err
	64	+	}
	65	+	return 0, io.EOF
	66	+	}
	67	+	m := r.stream.Msg()
	68	+	if m.GetJobId() != "" \|\| m.GetArtifactId() != 0 \|\| len(m.GetBlockId()) > 0 {
	69	+	return 0, fmt.Errorf("%w: job_id, artifact_id and block_id are in the first message only", domain.ErrInvalid)
	70	+	}
	71	+	if err := checkBlockData(m.GetData()); err != nil {
	72	+	return 0, err
	73	+	}
	74	+	r.buf = m.GetData()
	75	+	}
	76	+	n := copy(p, r.buf)
	77	+	r.buf = r.buf[n:]
	78	+	return n, nil
	79	+	}
	80	+
	81	+	func (s jobArtifactService) UploadJobArtifactBlock(ctx context.Context, stream connect.ClientStream[pb.UploadJobArtifactBlockRequest]) (*connect.Response[pb.UploadJobArtifactBlockResponse], error) {
	82	+	if !stream.Receive() {
	83	+	err := stream.Err()
	84	+	if err == nil {
	85	+	err = fmt.Errorf("%w: the stream has no message", domain.ErrInvalid)
	86	+	}
	87	+	return nil, s.runner.failed(ctx, err)
	88	+	}
	89	+	first := stream.Msg()
	90	+	if err := checkBlockData(first.GetData()); err != nil {
	91	+	return nil, s.runner.failed(ctx, err)
	92	+	}
	93	+	r := &blockReader{stream: stream, buf: first.GetData()}
	94	+	size, err := s.artifacts.UploadJobArtifactBlock(ctx, secretFrom(ctx), first.GetJobId(), first.GetArtifactId(), first.GetBlockId(), r)
	95	+	if err != nil {
	96	+	return nil, s.runner.failed(ctx, err)
	97	+	}
	98	+	return connect.NewResponse(&pb.UploadJobArtifactBlockResponse{Size: size}), nil
	99	+	}
	100	+
	101	+	func (s jobArtifactService) CommitJobArtifact(ctx context.Context, req connect.Request[pb.CommitJobArtifactRequest]) (*connect.Response[pb.CommitJobArtifactResponse], error) {
	102	+	size, digest, err := s.artifacts.CommitJobArtifact(ctx, secretFrom(ctx), req.Msg.GetJobId(), req.Msg.GetArtifactId(), req.Msg.GetBlockIds())
	103	+	if err != nil {
	104	+	return nil, s.runner.failed(ctx, err)
	105	+	}
	106	+	return connect.NewResponse(&pb.CommitJobArtifactResponse{Size: size, Digest: digest}), nil
	107	+	}
	108	+
	109	+	func (s jobArtifactService) FinalizeJobArtifact(ctx context.Context, req connect.Request[pb.FinalizeJobArtifactRequest]) (*connect.Response[pb.FinalizeJobArtifactResponse], error) {
	110	+	a, err := s.artifacts.FinalizeJobArtifact(ctx, secretFrom(ctx), req.Msg.GetJobId(), req.Msg.GetName(), req.Msg.GetSize(), req.Msg.GetDigest())
	111	+	if err != nil {
	112	+	return nil, s.runner.failed(ctx, err)
	113	+	}
	114	+	return connect.NewResponse(&pb.FinalizeJobArtifactResponse{Artifact: artifactToPB(a)}), nil
	115	+	}
	116	+
	117	+	func (s jobArtifactService) ListJobArtifacts(ctx context.Context, req connect.Request[pb.ListJobArtifactsRequest]) (*connect.Response[pb.ListJobArtifactsResponse], error) {
	118	+	arts, err := s.artifacts.ListJobArtifacts(ctx, secretFrom(ctx), req.Msg.GetJobId(), req.Msg.GetName(), req.Msg.GetArtifactId())
	119	+	if err != nil {
	120	+	return nil, s.runner.failed(ctx, err)
	121	+	}
	122	+	out := &pb.ListJobArtifactsResponse{Artifacts: make([]*pb.Artifact, 0, len(arts))}
	123	+	for _, a := range arts {
	124	+	out.Artifacts = append(out.Artifacts, artifactToPB(a))
	125	+	}
	126	+	return connect.NewResponse(out), nil
	127	+	}
	128	+
	129	+	func (s jobArtifactService) DownloadJobArtifact(ctx context.Context, req connect.Request[pb.DownloadJobArtifactRequest], stream *connect.ServerStream[pb.DownloadJobArtifactResponse]) error {
	130	+	err := s.artifacts.DownloadJobArtifact(ctx, secretFrom(ctx), req.Msg.GetJobId(), req.Msg.GetArtifactId(), req.Msg.GetOffset(), func(c ports.ArtifactChunk) error {
	131	+	m := &pb.DownloadJobArtifactResponse{Chunk: c.Data}
	132	+	if c.Artifact != nil {
	133	+	m.Artifact = artifactToPB(*c.Artifact)
	134	+	}
	135	+	return stream.Send(m)
	136	+	})
	137	+	return s.runner.failed(ctx, err)
	138	+	}
	139	+
	140	+	func (s jobArtifactService) DeleteJobArtifact(ctx context.Context, req connect.Request[pb.DeleteJobArtifactRequest]) (*connect.Response[pb.DeleteJobArtifactResponse], error) {
	141	+	id, err := s.artifacts.DeleteJobArtifact(ctx, secretFrom(ctx), req.Msg.GetJobId(), req.Msg.GetName())
	142	+	if err != nil {
	143	+	return nil, s.runner.failed(ctx, err)
	144	+	}
	145	+	return connect.NewResponse(&pb.DeleteJobArtifactResponse{ArtifactId: id}), nil
	146	+	}
	147	+
	148	+	// artifactService implements ermiteconnect.ArtifactServiceHandler.
	149	+	type artifactService struct {
	150	+	artifacts ports.ArtifactService
	151	+	}
	152	+
	153	+	func (s artifactService) ListArtifacts(ctx context.Context, req connect.Request[pb.ListArtifactsRequest]) (*connect.Response[pb.ListArtifactsResponse], error) {
	154	+	name, err := repoOf(req.Msg.GetRepo())
	155	+	if err != nil {
	156	+	return nil, ciError(ctx, err)
	157	+	}
	158	+	res, err := s.artifacts.ListArtifacts(ctx, caller(ctx), name, req.Msg.GetRunId())
	159	+	if err != nil {
	160	+	return nil, ciError(ctx, err)
	161	+	}
	162	+	out := &pb.ListArtifactsResponse{UsedBytes: res.Used, QuotaBytes: res.Quota, Artifacts: make([]*pb.Artifact, 0, len(res.Artifacts))}
	163	+	for _, a := range res.Artifacts {
	164	+	out.Artifacts = append(out.Artifacts, artifactToPB(a))
	165	+	}
	166	+	return connect.NewResponse(out), nil
	167	+	}
	168	+
	169	+	func (s artifactService) DownloadArtifact(ctx context.Context, req connect.Request[pb.DownloadArtifactRequest], stream *connect.ServerStream[pb.DownloadArtifactResponse]) error {
	170	+	name, err := repoOf(req.Msg.GetRepo())
	171	+	if err != nil {
	172	+	return ciError(ctx, err)
	173	+	}
	174	+	err = s.artifacts.DownloadArtifact(ctx, caller(ctx), name, req.Msg.GetArtifactId(), req.Msg.GetOffset(), func(c ports.ArtifactChunk) error {
	175	+	m := &pb.DownloadArtifactResponse{Chunk: c.Data}
	176	+	if c.Artifact != nil {
	177	+	m.Artifact = artifactToPB(*c.Artifact)
	178	+	}
	179	+	return stream.Send(m)
	180	+	})
	181	+	return ciError(ctx, err)
	182	+	}
	183	+
	184	+	func (s artifactService) DeleteArtifact(ctx context.Context, req connect.Request[pb.DeleteArtifactRequest]) (*connect.Response[pb.DeleteArtifactResponse], error) {
	185	+	name, err := repoOf(req.Msg.GetRepo())
	186	+	if err != nil {
	187	+	return nil, ciError(ctx, err)
	188	+	}
	189	+	if err := s.artifacts.DeleteArtifact(ctx, caller(ctx), name, req.Msg.GetArtifactId()); err != nil {
	190	+	return nil, ciError(ctx, err)
	191	+	}
	192	+	return connect.NewResponse(&pb.DeleteArtifactResponse{}), nil
	193	+	}

M internal/adapters/rpc/ci.go +8 -0


@@ -36,6 +36,14 @@ var runnerSecretProcedures = map[string]bool{
36	// ResolveActions and UploadSummary are job calls too.	36		// ResolveActions and UploadSummary are job calls too.
37	ermiteconnect.RunnerServiceResolveActionsProcedure: true,	37		ermiteconnect.RunnerServiceResolveActionsProcedure: true,
38	ermiteconnect.RunnerServiceUploadSummaryProcedure: true,	38		ermiteconnect.RunnerServiceUploadSummaryProcedure: true,
		39	+	// So is all of JobArtifactService.
		40	+	ermiteconnect.JobArtifactServiceCreateJobArtifactProcedure: true,
		41	+	ermiteconnect.JobArtifactServiceUploadJobArtifactBlockProcedure: true,
		42	+	ermiteconnect.JobArtifactServiceCommitJobArtifactProcedure: true,
		43	+	ermiteconnect.JobArtifactServiceFinalizeJobArtifactProcedure: true,
		44	+	ermiteconnect.JobArtifactServiceListJobArtifactsProcedure: true,
		45	+	ermiteconnect.JobArtifactServiceDownloadJobArtifactProcedure: true,
		46	+	ermiteconnect.JobArtifactServiceDeleteJobArtifactProcedure: true,
39	}	47		}
40		48
41	// ciError maps an error of the CI use cases. A call on a job that ended gets	49		// ciError maps an error of the CI use cases. A call on a job that ended gets

M internal/adapters/web/actions.templ +44 -0


@@ -86,6 +86,9 @@ templ RunDetail(l Layout, p RunPage) {
86	<div id="run-body">	86		<div id="run-body">
87	@RunBody(p)	87		@RunBody(p)
88	</div>	88		</div>
		89	+	if p.Artifacts != nil {
		90	+	@RunArtifacts(*p.Artifacts)
		91	+	}
89	@RunSummaries(p.Summaries, p.SummariesMore)	92		@RunSummaries(p.Summaries, p.SummariesMore)
90	}	93		}
91	}	94		}

@@ -202,6 +205,47 @@ templ JobAnnotations(anns []AnnotationView, more bool) {
202	</section>	205		</section>
203	}	206		}
204		207
		208	+	// RunArtifacts is the artifacts of a run, with the space the artifacts of
		209	+	// the repository use. Names come from the workflow and are text.
		210	+	templ RunArtifacts(a ArtifactsView) {
		211	+	<section class="artifacts" aria-labelledby="run-artifacts">
		212	+	<h3 id="run-artifacts">Artifacts <span class="count">{ itoa(len(a.Rows)) }</span></h3>
		213	+	<table class="list" aria-label="Artifacts">
		214	+	<thead>
		215	+	<tr><th class="c-name">name</th><th class="c-sub">size</th><th class="c-who">job</th><th class="c-when">expires</th><th></th></tr>
		216	+	</thead>
		217	+	<tbody>
		218	+	for _, r := range a.Rows {
		219	+	<tr>
		220	+	<td class="c-name"><a href={ templ.SafeURL(r.Href) } download>{ r.Name }</a></td>
		221	+	<td class="c-sub dim">{ r.Size }</td>
		222	+	<td class="c-who">
		223	+	if r.JobHref != "" {
		224	+	<a href={ templ.SafeURL(r.JobHref) }>{ r.Job }</a>
		225	+	} else {
		226	+	<span class="dim">{ r.Job }</span>
		227	+	}
		228	+	</td>
		229	+	<td class="c-when dim">
		230	+	if r.ExpiresAt != "" {
		231	+	<time datetime={ r.ExpiresAt }>{ r.Expires }</time>
		232	+	} else {
		233	+	{ r.Expires }
		234	+	}
		235	+	</td>
		236	+	<td>
		237	+	if r.DeleteAction != "" {
		238	+	<form method="post" action={ templ.SafeURL(r.DeleteAction) }><button type="submit" class="btn" aria-label={ "Delete " + r.Name }>Delete</button></form>
		239	+	}
		240	+	</td>
		241	+	</tr>
		242	+	}
		243	+	</tbody>
		244	+	</table>
		245	+	<p class="mute">{ a.Usage }</p>
		246	+	</section>
		247	+	}
		248	+
205	// RunSummaries is the job summaries of a run, one block per step that wrote	249		// RunSummaries is the job summaries of a run, one block per step that wrote
206	// one. The Markdown is rendered by the README renderer: raw HTML dropped, URLs	250		// one. The Markdown is rendered by the README renderer: raw HTML dropped, URLs
207	// filtered, images shown as their alt text.	251		// filtered, images shown as their alt text.

A internal/adapters/web/actions_artifacts.go +270 -0

@@ -0,0 +1,270 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package web
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"mime"
	9	+	"net/http"
	10	+	"strconv"
	11	+	"strings"
	12	+	"time"
	13	+	"unicode"
	14	+
	15	+	"github.com/bnema/zerowrap"
	16	+
	17	+	"bnema.dev/ermite/internal/domain"
	18	+	"bnema.dev/ermite/internal/ports"
	19	+	)
	20	+
	21	+	// The artifacts of a run (CI-4 D3), read through ports.ArtifactService. Only a
	22	+	// signed-in caller may list or download them, even on a public repository.
	23	+	//
	24	+	// GET /<repo>/actions/runs/<id>/artifacts/<artifact id> the content
	25	+	// POST /<repo>/actions/runs/<id>/artifacts/<artifact id>/delete (the owner)
	26	+
	27	+	// artifactTimeout bounds the download of one artifact.
	28	+	const artifactTimeout = 30 * time.Minute
	29	+
	30	+	func artifactHref(repo string, runID, id int64) string {
	31	+	return runHref(repo, runID) + "/artifacts/" + strconv.FormatInt(id, 10)
	32	+	}
	33	+
	34	+	// ArtifactRow is an artifact of the run page.
	35	+	type ArtifactRow struct {
	36	+	Name, Href, Size string
	37	+	// Expires is the expiry for the eye, ExpiresAt as RFC 3339.
	38	+	Expires, ExpiresAt string
	39	+	// Job is the job that uploaded it, JobHref its page ("" when it is not a
	40	+	// job of the attempt shown).
	41	+	Job, JobHref string
	42	+	// DeleteAction is the form that deletes it; "" when the visitor cannot.
	43	+	DeleteAction string
	44	+	}
	45	+
	46	+	// ArtifactsView is the artifacts section of the run page.
	47	+	type ArtifactsView struct {
	48	+	Rows []ArtifactRow
	49	+	// Usage is the space the artifacts of the repository use.
	50	+	Usage string
	51	+	}
	52	+
	53	+	// runArtifacts is the artifacts section of a run, nil when there is nothing to
	54	+	// show: no artifacts on this server, a visitor who cannot list them, no
	55	+	// artifact, or a failed read, which is logged; the run page never fails for it.
	56	+	//
	57	+	//hexcheck:ignore suspicious-business-logic-in-adapter builds the artifacts section from the use case results; which errors hide it is presentation, no rule is decided here
	58	+	func (s Server) runArtifacts(ctx context.Context, repo domain.RepoName, p RunPage) ArtifactsView {
	59	+	if s.cfg.Artifacts == nil \|\| !SignedIn(ctx) {
	60	+	return nil
	61	+	}
	62	+	runID := p.detail.Run.ID
	63	+	list, err := s.cfg.Artifacts.ListArtifacts(ctx, principal(ctx), repo, runID)
	64	+	if err != nil {
	65	+	if !errors.Is(err, domain.ErrNotFound) && !errors.Is(err, domain.ErrUnauthenticated) &&
	66	+	!errors.Is(err, domain.ErrForbidden) && !errors.Is(err, context.Canceled) {
	67	+	log := zerowrap.FromCtx(ctx)
	68	+	log.Warn().Err(err).Str("repo", repo.String()).Int64("run", runID).Msg("list the artifacts of the run")
	69	+	}
	70	+	return nil
	71	+	}
	72	+	if len(list.Artifacts) == 0 {
	73	+	return nil
	74	+	}
	75	+	jobs := map[string]string{}
	76	+	for _, j := range p.detail.Jobs {
	77	+	jobs[j.ID] = j.Name
	78	+	}
	79	+	name := repo.String()
	80	+	v := &ArtifactsView{Usage: humanSize(list.Used) + " used"}
	81	+	if list.Quota > 0 {
	82	+	v.Usage = humanSize(list.Used) + " of " + humanSize(list.Quota) + " used"
	83	+	}
	84	+	for _, a := range list.Artifacts {
	85	+	row := ArtifactRow{
	86	+	Name: visibleText(a.Name), Href: artifactHref(name, runID, a.ID), Size: humanSize(a.Size),
	87	+	Expires: a.ExpiresAt.UTC().Format("2006-01-02 15:04 UTC"), ExpiresAt: rfc3339(a.ExpiresAt),
	88	+	Job: visibleText(a.JobID),
	89	+	}
	90	+	if a.ExpiresAt.IsZero() {
	91	+	row.Expires = "never"
	92	+	}
	93	+	if n, ok := jobs[a.JobID]; ok {
	94	+	row.Job, row.JobHref = visibleText(n), jobHref(name, runID, a.JobID)
	95	+	}
	96	+	row.DeleteAction = artifactHref(name, runID, a.ID) + "/delete"
	97	+	v.Rows = append(v.Rows, row)
	98	+	}
	99	+	return v
	100	+	}
	101	+
	102	+	func parseArtifactPath(r *http.Request) (int64, error) {
	103	+	id, err := strconv.ParseInt(r.PathValue("artifact"), 10, 64)
	104	+	if err != nil \|\| id < 1 {
	105	+	return 0, errNotFound
	106	+	}
	107	+	return id, nil
	108	+	}
	109	+
	110	+	// artifactDownload is GET /<repo>/actions/runs/<id>/artifacts/<artifact id>:
	111	+	// it streams the content as an attachment. The media type the uploader chose
	112	+	// is never served: a zip is application/zip, anything else
	113	+	// application/octet-stream, and the answer is sandboxed and not sniffed, so
	114	+	// that nothing uploaded by a job can run in the origin of the forge.
	115	+	//
	116	+	//hexcheck:ignore suspicious-business-logic-in-adapter streams what the use case sends; the headers are about the response, no rule is decided here
	117	+	func (s Server) artifactDownload(w http.ResponseWriter, r http.Request) {
	118	+	ctx, cancel := context.WithTimeout(r.Context(), artifactTimeout)
	119	+	defer cancel()
	120	+	repo, runID, err := s.runTarget(r)
	121	+	if err != nil {
	122	+	s.fail(ctx, w, r, err)
	123	+	return
	124	+	}
	125	+	id, err := parseArtifactPath(r)
	126	+	if err != nil {
	127	+	s.fail(ctx, w, r, err)
	128	+	return
	129	+	}
	130	+	var started, headOnly bool
	131	+	var writeErr error
	132	+	err = s.cfg.Artifacts.DownloadArtifact(ctx, principal(ctx), repo, id, 0, func(c ports.ArtifactChunk) error {
	133	+	if !started {
	134	+	if c.Artifact == nil {
	135	+	return errors.New("web: the download did not start with the artifact")
	136	+	}
	137	+	if c.Artifact.RunID != runID {
	138	+	return errNotFound // an artifact of another run of the repository
	139	+	}
	140	+	h := w.Header()
	141	+	artifactHeaders(h, *c.Artifact)
	142	+	cacheHeaders(ctx, h, noStore, sessionCache)
	143	+	w.WriteHeader(http.StatusOK)
	144	+	started = true
	145	+	if r.Method == http.MethodHead {
	146	+	headOnly = true
	147	+	return errStopDownload
	148	+	}
	149	+	}
	150	+	if len(c.Data) == 0 {
	151	+	return nil
	152	+	}
	153	+	if _, werr := w.Write(c.Data); werr != nil {
	154	+	writeErr = werr
	155	+	return werr
	156	+	}
	157	+	return nil
	158	+	})
	159	+	switch {
	160	+	case err == nil, headOnly, writeErr != nil:
	161	+	// Done, or the visitor is gone.
	162	+	case !started:
	163	+	s.artifactFailed(ctx, w, r, err, runHref(repo.String(), runID))
	164	+	default:
	165	+	// The headers are gone: cut the answer short, so that the client
	166	+	// sees it is incomplete.
	167	+	log := zerowrap.FromCtx(ctx)
	168	+	log.Error().Err(err).Str("repo", repo.String()).Int64("artifact", id).Msg("artifact download failed")
	169	+	panic(http.ErrAbortHandler)
	170	+	}
	171	+	}
	172	+
	173	+	// errStopDownload ends the download of a HEAD request after the headers.
	174	+	var errStopDownload = errors.New("web: download stopped")
	175	+
	176	+	// artifactFailed answers a download refused before any byte was sent.
	177	+	func (s Server) artifactFailed(ctx context.Context, w http.ResponseWriter, r http.Request, err error, back string) {
	178	+	if errors.Is(err, domain.ErrUnauthenticated) \|\| errors.Is(err, domain.ErrForbidden) {
	179	+	s.reject(ctx, w, r, http.StatusForbidden, "forbidden", "sign in to download the artifacts of a run.", back)
	180	+	return
	181	+	}
	182	+	s.fail(ctx, w, r, err)
	183	+	}
	184	+
	185	+	// artifactHeaders are the headers of the content of a.
	186	+	func artifactHeaders(h http.Header, a domain.CIArtifact) {
	187	+	name := attachmentName(a.Name)
	188	+	typ := "application/octet-stream"
	189	+	if a.MediaType == domain.CIArtifactZip {
	190	+	typ = domain.CIArtifactZip
	191	+	name += ".zip"
	192	+	}
	193	+	h.Set("Content-Type", typ)
	194	+	h.Set("Content-Disposition", contentDisposition(name))
	195	+	h.Set("X-Content-Type-Options", "nosniff")
	196	+	h.Set("Content-Security-Policy", rawCSP)
	197	+	h.Set("Content-Length", strconv.FormatInt(a.Size, 10))
	198	+	}
	199	+
	200	+	// attachmentName is an artifact name fit for a file name: without control
	201	+	// characters, quotes, backslashes or slashes (which the domain already
	202	+	// refuses), "artifact" when nothing is left.
	203	+	func attachmentName(name string) string {
	204	+	name = strings.Map(func(r rune) rune {
	205	+	if unicode.IsControl(r) \|\| r == '"' \|\| r == '\\' \|\| r == '/' \|\| r == unicode.ReplacementChar {
	206	+	return -1
	207	+	}
	208	+	return r
	209	+	}, name)
	210	+	name = strings.TrimSpace(name)
	211	+	if name == "" \|\| name == "." \|\| name == ".." {
	212	+	return "artifact"
	213	+	}
	214	+	return name
	215	+	}
	216	+
	217	+	// contentDisposition is an attachment named name: a quoted ASCII filename,
	218	+	// and for a name that is not ASCII its RFC 5987 form as well, with the ASCII
	219	+	// one as the fallback (non-ASCII characters replaced by '_').
	220	+	func contentDisposition(name string) string {
	221	+	ascii := true
	222	+	fallback := strings.Map(func(r rune) rune {
	223	+	if r > unicode.MaxASCII {
	224	+	ascii = false
	225	+	return '_'
	226	+	}
	227	+	return r
	228	+	}, name)
	229	+	cd := mime.FormatMediaType("attachment", map[string]string{"filename": fallback})
	230	+	if cd == "" {
	231	+	cd = "attachment"
	232	+	}
	233	+	if !ascii {
	234	+	cd += "; filename*=UTF-8''" + rfc5987(name)
	235	+	}
	236	+	return cd
	237	+	}
	238	+
	239	+	// rfc5987 percent-encodes s but for the attr-char of RFC 5987.
	240	+	func rfc5987(s string) string {
	241	+	const hexDigits = "0123456789ABCDEF"
	242	+	var b strings.Builder
	243	+	for i := 0; i < len(s); i++ {
	244	+	c := s[i]
	245	+	if ('a' <= c && c <= 'z') \|\| ('A' <= c && c <= 'Z') \|\| ('0' <= c && c <= '9') \|\| strings.IndexByte("!#$&+-.^_`\|~", c) >= 0 {
	246	+	b.WriteByte(c)
	247	+	continue
	248	+	}
	249	+	b.WriteByte('%')
	250	+	b.WriteByte(hexDigits[c>>4])
	251	+	b.WriteByte(hexDigits[c&15])
	252	+	}
	253	+	return b.String()
	254	+	}
	255	+
	256	+	// artifactDelete is POST /<repo>/actions/runs/<id>/artifacts/<artifact id>/delete.
	257	+	func (s Server) artifactDelete(ctx context.Context, r http.Request) (string, error) {
	258	+	repo, runID, err := s.runTarget(r)
	259	+	if err != nil {
	260	+	return "", err
	261	+	}
	262	+	id, err := parseArtifactPath(r)
	263	+	if err != nil {
	264	+	return "", err
	265	+	}
	266	+	if err := s.cfg.Artifacts.DeleteArtifact(ctx, principal(ctx), repo, id); err != nil {
	267	+	return "", err
	268	+	}
	269	+	return runHref(repo.String(), runID), nil
	270	+	}

A internal/adapters/web/actions_artifacts_test.go +253 -0

@@ -0,0 +1,253 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package web
	4	+
	5	+	import (
	6	+	"context"
	7	+	"net/http"
	8	+	"strings"
	9	+	"sync"
	10	+	"testing"
	11	+	"time"
	12	+
	13	+	"github.com/stretchr/testify/assert"
	14	+	"github.com/stretchr/testify/mock"
	15	+	"github.com/stretchr/testify/require"
	16	+
	17	+	"bnema.dev/ermite/internal/domain"
	18	+	"bnema.dev/ermite/internal/ports"
	19	+	"bnema.dev/ermite/internal/ports/mocks"
	20	+	)
	21	+
	22	+	// fakeArtifacts drives a mock ArtifactService that refuses anonymous callers,
	23	+	// like the use case, and serves one artifact. It counts the lists and records
	24	+	// the deletions.
	25	+	type fakeArtifacts struct {
	26	+	m *mocks.ArtifactService
	27	+	mu sync.Mutex
	28	+	list ports.ArtifactList
	29	+	listErr error
	30	+	art domain.CIArtifact
	31	+	data []byte
	32	+	lists int
	33	+	deleted []int64
	34	+	}
	35	+
	36	+	func newFakeArtifacts(t testing.T, name, mediaType string, data []byte) fakeArtifacts {
	37	+	t.Helper()
	38	+	repo := repoName(t, "proj")
	39	+	a := domain.CIArtifact{
	40	+	ID: 7, Repo: repo, RunID: 1, Attempt: 1, JobID: "1-0-0", Name: name, MediaType: mediaType,
	41	+	Size: int64(len(data)), State: domain.CIArtifactFinal,
	42	+	CreatedAt: fixedNow, ExpiresAt: fixedNow.Add(90 * 24 * time.Hour),
	43	+	}
	44	+	f := &fakeArtifacts{
	45	+	m: mocks.NewArtifactService(t), art: a, data: data,
	46	+	list: ports.ArtifactList{Artifacts: []domain.CIArtifact{a}, Used: 3 << 20, Quota: 10 << 30},
	47	+	}
	48	+	f.m.EXPECT().ListArtifacts(mock.Anything, mock.Anything, repo, mock.Anything).RunAndReturn(
	49	+	func(_ context.Context, p domain.Principal, _ domain.RepoName, runID int64) (ports.ArtifactList, error) {
	50	+	f.mu.Lock()
	51	+	defer f.mu.Unlock()
	52	+	f.lists++
	53	+	switch {
	54	+	case !p.IsOwner():
	55	+	return ports.ArtifactList{}, domain.ErrUnauthenticated
	56	+	case f.listErr != nil:
	57	+	return ports.ArtifactList{}, f.listErr
	58	+	case runID != f.art.RunID:
	59	+	return ports.ArtifactList{}, nil
	60	+	}
	61	+	return f.list, nil
	62	+	}).Maybe()
	63	+	f.m.EXPECT().DownloadArtifact(mock.Anything, mock.Anything, repo, mock.Anything, mock.Anything, mock.Anything).RunAndReturn(
	64	+	func(_ context.Context, p domain.Principal, _ domain.RepoName, id, offset int64, send func(ports.ArtifactChunk) error) error {
	65	+	if !p.IsOwner() {
	66	+	return domain.ErrUnauthenticated
	67	+	}
	68	+	if id != f.art.ID \|\| offset != 0 {
	69	+	return domain.ErrNotFound
	70	+	}
	71	+	a := f.art
	72	+	if err := send(ports.ArtifactChunk{Artifact: &a}); err != nil {
	73	+	return err
	74	+	}
	75	+	half := len(f.data) / 2
	76	+	for _, d := range [][]byte{f.data[:half], f.data[half:]} {
	77	+	if err := send(ports.ArtifactChunk{Data: d}); err != nil {
	78	+	return err
	79	+	}
	80	+	}
	81	+	return nil
	82	+	}).Maybe()
	83	+	f.m.EXPECT().DeleteArtifact(mock.Anything, mock.Anything, repo, mock.Anything).RunAndReturn(
	84	+	func(_ context.Context, p domain.Principal, _ domain.RepoName, id int64) error {
	85	+	if !p.IsOwner() {
	86	+	return domain.ErrUnauthenticated
	87	+	}
	88	+	f.mu.Lock()
	89	+	defer f.mu.Unlock()
	90	+	f.deleted = append(f.deleted, id)
	91	+	return nil
	92	+	}).Maybe()
	93	+	return f
	94	+	}
	95	+
	96	+	// artifactsServer serves the run 1 of proj, with its job 1-0-0, and the
	97	+	// artifacts of arts; the session token "tok" signs the owner in.
	98	+	func artifactsServer(t testing.T, arts fakeArtifacts) *passkeyFixture {
	99	+	t.Helper()
	100	+	repo := repoName(t, "proj")
	101	+	runs := mocks.NewRunService(t)
	102	+	job := ports.JobView{CIJob: domain.CIJob{
	103	+	ID: "1-0-0", RunID: 1, Attempt: 1, RunAttempt: 1, Repo: repo, Key: "build", Name: "build",
	104	+	Phase: domain.CIPhaseEnded, Conclusion: domain.CISuccess,
	105	+	}}
	106	+	runs.EXPECT().GetRun(mock.Anything, mock.Anything, repo, int64(1), int32(0)).Return(ports.RunDetail{
	107	+	Run: domain.CIRun{Repo: repo, ID: 1, Attempt: 1, Number: 1, WorkflowName: "ci", Sha: strings.Repeat("a", 40), Status: domain.CICompleted, Conclusion: domain.CISuccess},
	108	+	Jobs: []ports.JobView{job}, LatestAttempt: 1,
	109	+	}, nil).Maybe()
	110	+	runs.EXPECT().ListAnnotations(mock.Anything, mock.Anything, repo, int64(1), int32(1), "").Return(ports.AnnotationList{}, nil).Maybe()
	111	+	runs.EXPECT().GetStepSummaries(mock.Anything, mock.Anything, repo, mock.Anything, mock.Anything, mock.Anything).Return(ports.SummaryList{}, nil).Maybe()
	112	+	browse := mocks.NewBrowseService(t)
	113	+	browse.EXPECT().ListRefs(mock.Anything, mock.Anything, repo, mock.Anything).Return(ports.RefPage{}, nil).Maybe()
	114	+	cfg := Config{
	115	+	Repos: reposServing(t, publicRepo(t, "proj")), Browse: browse, Runs: runs,
	116	+	PublicURL: forgeOrigin, Sessions: newSessions(t, "tok"), Limiter: &blockingLimiter{},
	117	+	Now: func() time.Time { return fixedNow },
	118	+	}
	119	+	if arts != nil {
	120	+	cfg.Artifacts = arts.m
	121	+	}
	122	+	return &passkeyFixture{srv: newTestServer(t, cfg)}
	123	+	}
	124	+
	125	+	func TestRunPageArtifactsForTheOwner(t *testing.T) {
	126	+	t.Parallel()
	127	+	arts := newFakeArtifacts(t, "dist", domain.CIArtifactZip, []byte("PK"))
	128	+	f := artifactsServer(t, arts)
	129	+	rec := f.do(t, req{method: "GET", target: "/proj/actions/runs/1", cookie: "tok"})
	130	+	require.Equal(t, http.StatusOK, rec.Code)
	131	+	body := rec.Body.String()
	132	+	assert.Contains(t, body, `id="run-artifacts"`)
	133	+	assert.Contains(t, body, `href="/proj/actions/runs/1/artifacts/7"`)
	134	+	assert.Contains(t, body, ">dist</a>")
	135	+	assert.Contains(t, body, "2 B", "the size")
	136	+	assert.Contains(t, body, `datetime="2027-01-10T12:00:00Z"`, "the expiry")
	137	+	assert.Contains(t, body, `href="/proj/actions/runs/1/job/1-0-0">build</a>`, "the job that uploaded it")
	138	+	assert.Contains(t, body, "3.0 MiB of 10.0 GiB used")
	139	+	assert.Contains(t, body, `action="/proj/actions/runs/1/artifacts/7/delete"`)
	140	+	}
	141	+
	142	+	func TestRunPageArtifactsHidden(t *testing.T) {
	143	+	t.Parallel()
	144	+	// Anonymous: no section, and the use case is not even asked.
	145	+	arts := newFakeArtifacts(t, "dist", domain.CIArtifactZip, []byte("PK"))
	146	+	f := artifactsServer(t, arts)
	147	+	rec := f.do(t, req{method: "GET", target: "/proj/actions/runs/1"})
	148	+	require.Equal(t, http.StatusOK, rec.Code)
	149	+	assert.NotContains(t, rec.Body.String(), "run-artifacts")
	150	+	assert.Zero(t, arts.lists)
	151	+
	152	+	// A refusal or a failure of the list leaves the page whole, without it.
	153	+	for _, err := range []error{domain.ErrForbidden, domain.ErrNotFound, assert.AnError} {
	154	+	arts := newFakeArtifacts(t, "dist", domain.CIArtifactZip, []byte("PK"))
	155	+	arts.listErr = err
	156	+	rec := artifactsServer(t, arts).do(t, req{method: "GET", target: "/proj/actions/runs/1", cookie: "tok"})
	157	+	require.Equal(t, http.StatusOK, rec.Code, err)
	158	+	assert.NotContains(t, rec.Body.String(), "run-artifacts", err)
	159	+	}
	160	+
	161	+	// No artifacts on the server: no section and no route.
	162	+	f = artifactsServer(t, nil)
	163	+	rec = f.do(t, req{method: "GET", target: "/proj/actions/runs/1", cookie: "tok"})
	164	+	require.Equal(t, http.StatusOK, rec.Code)
	165	+	assert.NotContains(t, rec.Body.String(), "run-artifacts")
	166	+	assert.Equal(t, http.StatusNotFound, f.do(t, req{method: "GET", target: "/proj/actions/runs/1/artifacts/7", cookie: "tok"}).Code)
	167	+	}
	168	+
	169	+	func TestArtifactDownloadZip(t *testing.T) {
	170	+	t.Parallel()
	171	+	data := []byte("PK\x03\x04 the archive")
	172	+	f := artifactsServer(t, newFakeArtifacts(t, "dist", domain.CIArtifactZip, data))
	173	+	rec := f.do(t, req{method: "GET", target: "/proj/actions/runs/1/artifacts/7", cookie: "tok"})
	174	+	require.Equal(t, http.StatusOK, rec.Code)
	175	+	h := rec.Header()
	176	+	assert.Equal(t, "application/zip", h.Get("Content-Type"))
	177	+	assert.Equal(t, `attachment; filename=dist.zip`, h.Get("Content-Disposition"))
	178	+	assert.Equal(t, "nosniff", h.Get("X-Content-Type-Options"))
	179	+	assert.Contains(t, h.Get("Content-Security-Policy"), "sandbox")
	180	+	assert.Equal(t, itoa(len(data)), h.Get("Content-Length"))
	181	+	assert.Equal(t, sessionCache, h.Get("Cache-Control"))
	182	+	assert.Equal(t, data, rec.Body.Bytes())
	183	+
	184	+	// HEAD: the headers only.
	185	+	rec = f.do(t, req{method: "HEAD", target: "/proj/actions/runs/1/artifacts/7", cookie: "tok"})
	186	+	require.Equal(t, http.StatusOK, rec.Code)
	187	+	assert.Equal(t, itoa(len(data)), rec.Header().Get("Content-Length"))
	188	+	assert.Empty(t, rec.Body.Bytes())
	189	+	}
	190	+
	191	+	// The media type an uploader chose is never served: it could be text/html.
	192	+	func TestArtifactDownloadOtherType(t *testing.T) {
	193	+	t.Parallel()
	194	+	data := []byte("<script>alert(1)</script>")
	195	+	f := artifactsServer(t, newFakeArtifacts(t, "report é.html", "text/html", data))
	196	+	rec := f.do(t, req{method: "GET", target: "/proj/actions/runs/1/artifacts/7", cookie: "tok"})
	197	+	require.Equal(t, http.StatusOK, rec.Code)
	198	+	h := rec.Header()
	199	+	assert.Equal(t, "application/octet-stream", h.Get("Content-Type"))
	200	+	assert.Equal(t, `attachment; filename="report _.html"; filename*=UTF-8''report%20%C3%A9.html`, h.Get("Content-Disposition"))
	201	+	assert.Equal(t, "nosniff", h.Get("X-Content-Type-Options"))
	202	+	assert.Contains(t, h.Get("Content-Security-Policy"), "sandbox")
	203	+	assert.Equal(t, data, rec.Body.Bytes())
	204	+	}
	205	+
	206	+	func TestArtifactDownloadRefused(t *testing.T) {
	207	+	t.Parallel()
	208	+	f := artifactsServer(t, newFakeArtifacts(t, "dist", domain.CIArtifactZip, []byte("PK")))
	209	+	// Anonymous, even on a public repository.
	210	+	rec := f.do(t, req{method: "GET", target: "/proj/actions/runs/1/artifacts/7"})
	211	+	assert.Equal(t, http.StatusForbidden, rec.Code)
	212	+	assert.Empty(t, rec.Header().Get("Content-Disposition"))
	213	+	// Not a number, unknown, or an artifact of another run.
	214	+	for _, target := range []string{
	215	+	"/proj/actions/runs/1/artifacts/x", "/proj/actions/runs/1/artifacts/0", "/proj/actions/runs/1/artifacts/-7",
	216	+	"/proj/actions/runs/1/artifacts/8", "/proj/actions/runs/2/artifacts/7", "/proj/actions/runs/x/artifacts/7",
	217	+	} {
	218	+	rec := f.do(t, req{method: "GET", target: target, cookie: "tok"})
	219	+	assert.Equal(t, http.StatusNotFound, rec.Code, target)
	220	+	assert.Empty(t, rec.Header().Get("Content-Disposition"), target)
	221	+	}
	222	+	}
	223	+
	224	+	func TestArtifactDelete(t *testing.T) {
	225	+	t.Parallel()
	226	+	arts := newFakeArtifacts(t, "dist", domain.CIArtifactZip, []byte("PK"))
	227	+	f := artifactsServer(t, arts)
	228	+	target := "/proj/actions/runs/1/artifacts/7/delete"
	229	+	form := "application/x-www-form-urlencoded"
	230	+	// The gate: a session and the Origin of the forge.
	231	+	assert.Equal(t, http.StatusForbidden, f.do(t, req{method: "POST", target: target, origin: forgeOrigin, contentType: form}).Code)
	232	+	assert.Equal(t, http.StatusForbidden, f.do(t, req{method: "POST", target: target, origin: "https://evil.example", cookie: "tok", contentType: form}).Code)
	233	+	assert.Empty(t, arts.deleted)
	234	+
	235	+	rec := f.do(t, req{method: "POST", target: target, origin: forgeOrigin, cookie: "tok", contentType: form})
	236	+	assert.Equal(t, http.StatusSeeOther, rec.Code)
	237	+	assert.Equal(t, "/proj/actions/runs/1", rec.Header().Get("Location"))
	238	+	assert.Equal(t, []int64{7}, arts.deleted)
	239	+	}
	240	+
	241	+	func TestContentDisposition(t *testing.T) {
	242	+	t.Parallel()
	243	+	for in, want := range map[string]string{
	244	+	"dist.zip": `attachment; filename=dist.zip`,
	245	+	"a b.txt": `attachment; filename="a b.txt"`,
	246	+	"日本.zip": `attachment; filename=__.zip; filename*=UTF-8''%E6%97%A5%E6%9C%AC.zip`,
	247	+	attachmentName(""): `attachment; filename=artifact`,
	248	+	} {
	249	+	assert.Equal(t, want, contentDisposition(in), in)
	250	+	}
	251	+	assert.Equal(t, "ab", attachmentName("a\"\x00\r\n\\b"))
	252	+	assert.Equal(t, "artifact", attachmentName(".."))
	253	+	}

M internal/adapters/web/actions_pages.go +6 -1

 //	GET  /<repo>/actions/runs/<id>/job/<job id>   one job: steps and live log
 //	GET  /<repo>/actions/runs/<id>/job/<job id>/raw   the log as text
 //	POST /<repo>/actions/runs/<id>/cancel | rerun | rerun-failed   (the owner)
+//	GET  /<repo>/actions/runs/<id>/artifacts/<artifact id>   (actions_artifacts.go)
 const (
 	// actionsPageSize is the number of runs one page lists.
 	Summaries []JobSummaries
 	// SummariesMore says that some summaries are not shown (the page budget).
 	SummariesMore bool
-	Active        bool
+	// Artifacts are those of the run; only the full page has them, nil when
+	// the visitor cannot list them or there is none.
+	Artifacts *ArtifactsView
+	Active    bool
 	// detail is what the page was built from.
 	detail ports.RunDetail
 	// Forms are the owner's actions; nil/empty when the visitor cannot.
 	if p.Summaries, p.SummariesMore, err = s.jobSummaries(ctx, d, p.detail.Run, p.detail.Jobs, p.Attempt == p.Latest, maxSummaryPageBytes); err != nil {
 		return nil, 0, err
 	}
+	p.Artifacts = s.runArtifacts(ctx, d.repo.Name, p)
 	l := s.repoLayout(ctx, d, "actions")
 	l.Title = "run " + p.ID + " · " + d.name() + " · " + s.instance
 	l.Live, l.LiveURL = true, liveURL("repo:"+d.name(), "view=run&n="+p.ID)

M internal/adapters/web/actions_templ.go +746 -494 generated file: collapsed

load diff

M internal/adapters/web/summary.go +3 -1

 		return itoa(int(n)) + " B"
 	case n < 1<<20:
 		return fmt.Sprintf("%.1f KiB", float64(n)/1024)
+	case n < 1<<30:
+		return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20))
 	}
-	return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20))
+	return fmt.Sprintf("%.1f GiB", float64(n)/(1<<30))
 }
 // recentActions is the newest runs for the rail, as the caller of the request

M internal/adapters/web/web.go +8 -0


@@ -58,6 +58,10 @@ type Config struct {
58	// commits and merge requests. Without it there are no such pages. Leave it	58		// commits and merge requests. Without it there are no such pages. Leave it
59	// unset rather than assigning a typed nil pointer.	59		// unset rather than assigning a typed nil pointer.
60	Runs ports.RunService	60		Runs ports.RunService
		61	+	// Artifacts serves the artifacts of the runs: a section of the run page,
		62	+	// their download and their deletion. It needs Runs. Without it there is
		63	+	// neither. Leave it unset rather than assigning a typed nil pointer.
		64	+	Artifacts ports.ArtifactService
61	// Events is the source of the event stream behind /-/events. Without	65		// Events is the source of the event stream behind /-/events. Without
62	// it the pages are not live and the route answers 404. Leave it unset	66		// it the pages are not live and the route answers 404. Leave it unset
63	// rather than assigning a typed nil pointer.	67		// rather than assigning a typed nil pointer.

@@ -192,6 +196,10 @@ func (s Server) routes() http.ServeMux {
192	mux.HandleFunc("POST /{repo}/actions/runs/{id}/cancel", s.ciAction(s.runCancel))	196		mux.HandleFunc("POST /{repo}/actions/runs/{id}/cancel", s.ciAction(s.runCancel))
193	mux.HandleFunc("POST /{repo}/actions/runs/{id}/rerun", s.ciAction(s.runRerun(false)))	197		mux.HandleFunc("POST /{repo}/actions/runs/{id}/rerun", s.ciAction(s.runRerun(false)))
194	mux.HandleFunc("POST /{repo}/actions/runs/{id}/rerun-failed", s.ciAction(s.runRerun(true)))	198		mux.HandleFunc("POST /{repo}/actions/runs/{id}/rerun-failed", s.ciAction(s.runRerun(true)))
		199	+	if s.cfg.Artifacts != nil {
		200	+	mux.HandleFunc("GET /{repo}/actions/runs/{id}/artifacts/{artifact}", s.artifactDownload)
		201	+	mux.HandleFunc("POST /{repo}/actions/runs/{id}/artifacts/{artifact}/delete", s.ciAction(s.artifactDelete))
		202	+	}
195	}	203		}
196	mux.HandleFunc(catchAll, s.page(s.notFound))	204		mux.HandleFunc(catchAll, s.page(s.notFound))
197	return mux	205		return mux

M internal/app/api_test.go +1 -1


@@ -196,7 +196,7 @@ func TestAC2ReadAccess(t *testing.T) {
196		for _, who := range allPrincipals {	196		for _, who := range allPrincipals {
197		resp, err := r.who[who].server.GetVersion(ctx, connect.NewRequest(&pb.GetVersionRequest{}))	197		resp, err := r.who[who].server.GetVersion(ctx, connect.NewRequest(&pb.GetVersionRequest{}))
198		require.NoError(t, err, who)	198		require.NoError(t, err, who)
199	-	assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.dispatch", "ci.cache", "ci.secrets"}, resp.Msg.GetFeatures(), who)	199	+	assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.dispatch", "ci.cache", "ci.secrets", "ci.artifacts"}, resp.Msg.GetFeatures(), who)
200		assert.NotEmpty(t, resp.Msg.GetVersion())	200		assert.NotEmpty(t, resp.Msg.GetVersion())
201		}	201		}
202		})	202		})

A internal/app/ci_artifacts_test.go +789 -0

@@ -0,0 +1,789 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package app
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"context"
	8	+	"crypto/sha256"
	9	+	"encoding/hex"
	10	+	"errors"
	11	+	"fmt"
	12	+	"net/http"
	13	+	"os"
	14	+	"path/filepath"
	15	+	"strings"
	16	+	"sync"
	17	+	"sync/atomic"
	18	+	"testing"
	19	+	"time"
	20	+
	21	+	"connectrpc.com/connect"
	22	+	"github.com/stretchr/testify/assert"
	23	+	"github.com/stretchr/testify/require"
	24	+	"google.golang.org/protobuf/types/known/timestamppb"
	25	+
	26	+	pb "bnema.dev/ermite-api/gen/ermite"
	27	+	"bnema.dev/ermite-api/gen/ermite/ermiteconnect"
	28	+	"bnema.dev/ermite/internal/adapters/ciartifacts"
	29	+	"bnema.dev/ermite/internal/domain"
	30	+	"bnema.dev/ermite/internal/ports"
	31	+	"bnema.dev/ermite/internal/usecase"
	32	+	)
	33	+
	34	+	// CI-4 D3: the artifacts of the runs, as the runner and the owners call them.
	35	+
	36	+	const artifactWorkflow = `name: ci
	37	+	on:
	38	+	push:
	39	+	branches: [main]
	40	+	jobs:
	41	+	up:
	42	+	runs-on: ubuntu-latest
	43	+	steps:
	44	+	- run: echo up
	45	+	down:
	46	+	runs-on: ubuntu-latest
	47	+	steps:
	48	+	- run: echo down
	49	+	`
	50	+
	51	+	// artifactAPI are the artifact clients of one caller.
	52	+	type artifactAPI struct {
	53	+	job ermiteconnect.JobArtifactServiceClient
	54	+	owner ermiteconnect.ArtifactServiceClient
	55	+	}
	56	+
	57	+	func (c *ciHarness) artifacts(secret string) artifactAPI {
	58	+	hc := &http.Client{Transport: bearerTransport{secret: secret, next: http.DefaultTransport}}
	59	+	return artifactAPI{ermiteconnect.NewJobArtifactServiceClient(hc, c.url("/-/api")), ermiteconnect.NewArtifactServiceClient(hc, c.url("/-/api"))}
	60	+	}
	61	+
	62	+	// strangerArtifacts are clients whose refused calls do not count against the
	63	+	// test's client address.
	64	+	func (c *ciHarness) strangerArtifacts(secret string) artifactAPI {
	65	+	hc := &http.Client{Transport: bearerTransport{secret: secret, freshClient: new(atomic.Uint32), next: http.DefaultTransport}}
	66	+	return artifactAPI{ermiteconnect.NewJobArtifactServiceClient(hc, c.url("/-/api")), ermiteconnect.NewArtifactServiceClient(hc, c.url("/-/api"))}
	67	+	}
	68	+
	69	+	// jobArtifacts is the job side of one job.
	70	+	type jobArtifacts struct {
	71	+	c *ciHarness
	72	+	job *pb.Job
	73	+	api artifactAPI
	74	+	}
	75	+
	76	+	func (c ciHarness) jobArtifacts(j pb.Job) *jobArtifacts {
	77	+	return &jobArtifacts{c: c, job: j, api: c.artifacts(j.GetJobToken())}
	78	+	}
	79	+
	80	+	func (j jobArtifacts) create(name string, expires timestamppb.Timestamp) (*pb.CreateJobArtifactResponse, error) {
	81	+	res, err := j.api.job.CreateJobArtifact(j.c.ctx, connect.NewRequest(&pb.CreateJobArtifactRequest{JobId: j.job.GetJobId(), Name: name, ExpiresAt: expires}))
	82	+	if err != nil {
	83	+	return nil, err
	84	+	}
	85	+	return res.Msg, nil
	86	+	}
	87	+
	88	+	func (j *jobArtifacts) mustCreate(name string) int64 {
	89	+	j.c.t.Helper()
	90	+	res, err := j.create(name, nil)
	91	+	require.NoError(j.c.t, err)
	92	+	return res.GetArtifactId()
	93	+	}
	94	+
	95	+	// block uploads data as one block, in messages of at most 1 MiB.
	96	+	func (j *jobArtifacts) block(id int64, blockID string, data []byte) (int64, error) {
	97	+	return putBlock(j.c.ctx, j.api, j.job.GetJobId(), id, []byte(blockID), data)
	98	+	}
	99	+
	100	+	func putBlock(ctx context.Context, api artifactAPI, jobID string, id int64, blockID, data []byte) (int64, error) {
	101	+	stream := api.job.UploadJobArtifactBlock(ctx)
	102	+	first := &pb.UploadJobArtifactBlockRequest{JobId: jobID, ArtifactId: id, BlockId: blockID}
	103	+	n := min(len(data), domain.CIArtifactChunk)
	104	+	first.Data, data = data[:n], data[n:]
	105	+	if err := stream.Send(first); err != nil {
	106	+	data = nil // the server answered early: CloseAndReceive has the error
	107	+	}
	108	+	for len(data) > 0 {
	109	+	n := min(len(data), domain.CIArtifactChunk)
	110	+	if err := stream.Send(&pb.UploadJobArtifactBlockRequest{Data: data[:n]}); err != nil {
	111	+	break // the server answered early: CloseAndReceive has the error
	112	+	}
	113	+	data = data[n:]
	114	+	}
	115	+	res, err := stream.CloseAndReceive()
	116	+	if err != nil {
	117	+	return 0, err
	118	+	}
	119	+	return res.Msg.GetSize(), nil
	120	+	}
	121	+
	122	+	func (j jobArtifacts) commit(id int64, blocks ...string) (pb.CommitJobArtifactResponse, error) {
	123	+	ids := make([][]byte, 0, len(blocks))
	124	+	for _, b := range blocks {
	125	+	ids = append(ids, []byte(b))
	126	+	}
	127	+	res, err := j.api.job.CommitJobArtifact(j.c.ctx, connect.NewRequest(&pb.CommitJobArtifactRequest{JobId: j.job.GetJobId(), ArtifactId: id, BlockIds: ids}))
	128	+	if err != nil {
	129	+	return nil, err
	130	+	}
	131	+	return res.Msg, nil
	132	+	}
	133	+
	134	+	func (j jobArtifacts) finalize(name, digest string) (pb.Artifact, error) {
	135	+	res, err := j.api.job.FinalizeJobArtifact(j.c.ctx, connect.NewRequest(&pb.FinalizeJobArtifactRequest{JobId: j.job.GetJobId(), Name: name, Digest: digest}))
	136	+	if err != nil {
	137	+	return nil, err
	138	+	}
	139	+	return res.Msg.GetArtifact(), nil
	140	+	}
	141	+
	142	+	func (j jobArtifacts) list(name string) ([]pb.Artifact, error) {
	143	+	res, err := j.api.job.ListJobArtifacts(j.c.ctx, connect.NewRequest(&pb.ListJobArtifactsRequest{JobId: j.job.GetJobId(), Name: name}))
	144	+	if err != nil {
	145	+	return nil, err
	146	+	}
	147	+	return res.Msg.GetArtifacts(), nil
	148	+	}
	149	+
	150	+	func (j jobArtifacts) download(id, offset int64) (pb.Artifact, []byte, error) {
	151	+	stream, err := j.api.job.DownloadJobArtifact(j.c.ctx, connect.NewRequest(&pb.DownloadJobArtifactRequest{JobId: j.job.GetJobId(), ArtifactId: id, Offset: offset}))
	152	+	if err != nil {
	153	+	return nil, nil, err
	154	+	}
	155	+	defer func() { _ = stream.Close() }()
	156	+	var a *pb.Artifact
	157	+	var buf bytes.Buffer
	158	+	for stream.Receive() {
	159	+	if m := stream.Msg(); m.GetArtifact() != nil {
	160	+	a = m.GetArtifact()
	161	+	}
	162	+	buf.Write(stream.Msg().GetChunk())
	163	+	}
	164	+	return a, buf.Bytes(), stream.Err()
	165	+	}
	166	+
	167	+	func (j *jobArtifacts) remove(name string) error {
	168	+	_, err := j.api.job.DeleteJobArtifact(j.c.ctx, connect.NewRequest(&pb.DeleteJobArtifactRequest{JobId: j.job.GetJobId(), Name: name}))
	169	+	return err
	170	+	}
	171	+
	172	+	// upload runs a whole upload of one block.
	173	+	func (j jobArtifacts) upload(name string, data []byte) pb.Artifact {
	174	+	j.c.t.Helper()
	175	+	id := j.mustCreate(name)
	176	+	_, err := j.block(id, "only", data)
	177	+	require.NoError(j.c.t, err)
	178	+	_, err = j.commit(id, "only")
	179	+	require.NoError(j.c.t, err)
	180	+	a, err := j.finalize(name, "")
	181	+	require.NoError(j.c.t, err)
	182	+	return a
	183	+	}
	184	+
	185	+	func digestOf(b []byte) string {
	186	+	sum := sha256.Sum256(b)
	187	+	return "sha256:" + hex.EncodeToString(sum[:])
	188	+	}
	189	+
	190	+	func pattern(n int, seed byte) []byte {
	191	+	b := make([]byte, n)
	192	+	for i := range b {
	193	+	b[i] = seed + byte(i%251)
	194	+	}
	195	+	return b
	196	+	}
	197	+
	198	+	func (c *ciHarness) usage() int64 {
	199	+	c.t.Helper()
	200	+	used, _, err := c.forge.store.Artifacts().ArtifactUsage(c.ctx, c.name(c.repo))
	201	+	require.NoError(c.t, err)
	202	+	return used
	203	+	}
	204	+
	205	+	// artifactFiles lists every file under ci/artifacts.
	206	+	func (c *ciHarness) artifactFiles() []string {
	207	+	c.t.Helper()
	208	+	var out []string
	209	+	root := ciartifacts.Dir(c.data)
	210	+	_ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
	211	+	if err == nil && !d.IsDir() {
	212	+	rel, _ := filepath.Rel(root, path)
	213	+	out = append(out, rel)
	214	+	}
	215	+	return nil
	216	+	})
	217	+	return out
	218	+	}
	219	+
	220	+	func endedDetail(t *testing.T, err error) {
	221	+	t.Helper()
	222	+	require.Equal(t, connect.CodeFailedPrecondition, connect.CodeOf(err), "%v", err)
	223	+	var ce *connect.Error
	224	+	require.True(t, errors.As(err, &ce))
	225	+	found := false
	226	+	for _, d := range ce.Details() {
	227	+	if _, derr := d.Value(); derr == nil && strings.HasSuffix(d.Type(), "JobEnded") {
	228	+	found = true
	229	+	}
	230	+	}
	231	+	assert.True(t, found, "a JobEnded detail")
	232	+	}
	233	+
	234	+	// newArtifactRun pushes the workflow and returns the jobs of the run, given and
	235	+	// acknowledged.
	236	+	func (c ciHarness) newArtifactRun(r fakeRunner, files map[string]string) (runID int64, up, down *pb.Job) {
	237	+	c.t.Helper()
	238	+	before := len(c.runs())
	239	+	c.push("main", files, "")
	240	+	c.waitFor("a run", func() bool { return len(c.runs()) == before+1 })
	241	+	runID = c.runs()[0].GetId()
	242	+	a, b := r.await(), r.await()
	243	+	r.heartbeat(a)
	244	+	r.heartbeat(b)
	245	+	if strings.HasSuffix(a.GetJobKey(), "down") {
	246	+	a, b = b, a
	247	+	}
	248	+	return runID, a, b
	249	+	}
	250	+
	251	+	func TestCIArtifactsPassBetweenJobs(t *testing.T) {
	252	+	c := newCIHarness(t, nil)
	253	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	254	+	_, upJob, downJob := c.newArtifactRun(runner, map[string]string{".github/workflows/ci.yml": artifactWorkflow})
	255	+	up, down := c.jobArtifacts(upJob), c.jobArtifacts(downJob)
	256	+
	257	+	ver, err := c.root.server.GetVersion(c.ctx, connect.NewRequest(&pb.GetVersionRequest{}))
	258	+	require.NoError(t, err)
	259	+	assert.Contains(t, ver.Msg.GetFeatures(), "ci.artifacts")
	260	+
	261	+	// Blocks of 1.5 MiB (two messages each), uploaded at once in any order.
	262	+	blocks := [][]byte{pattern(3<<19, 1), pattern(3<<19, 2), pattern(3<<19, 3), pattern(10, 4)}
	263	+	var content []byte
	264	+	for _, b := range blocks {
	265	+	content = append(content, b...)
	266	+	}
	267	+	created, err := up.create("dist", nil)
	268	+	require.NoError(t, err)
	269	+	id := created.GetArtifactId()
	270	+	retention := domain.CIRetentionDays * 24 * time.Hour
	271	+	assert.WithinDuration(t, time.Now().Add(retention), created.GetExpiresAt().AsTime(), time.Minute, "the default retention")
	272	+
	273	+	t.Run("an upload in progress is invisible", func(t *testing.T) {
	274	+	got, err := down.list("")
	275	+	require.NoError(t, err)
	276	+	assert.Empty(t, got)
	277	+	})
	278	+
	279	+	var wg sync.WaitGroup
	280	+	for i := len(blocks) - 1; i >= 0; i-- {
	281	+	wg.Go(func() {
	282	+	n, err := up.block(id, fmt.Sprint("b", i), blocks[i])
	283	+	assert.NoError(t, err)
	284	+	assert.EqualValues(t, len(blocks[i]), n)
	285	+	})
	286	+	}
	287	+	wg.Wait()
	288	+	// A retried block replaces the first one, and is charged once.
	289	+	_, err = up.block(id, "b3", blocks[3])
	290	+	require.NoError(t, err)
	291	+	assert.EqualValues(t, len(content), c.usage(), "pending blocks are charged")
	292	+
	293	+	commit, err := up.commit(id, "b0", "b1", "b2", "b3")
	294	+	require.NoError(t, err)
	295	+	assert.EqualValues(t, len(content), commit.GetSize())
	296	+	assert.Equal(t, digestOf(content), commit.GetDigest())
	297	+	again, err := up.commit(id, "b0", "b1", "b2", "b3")
	298	+	require.NoError(t, err, "a repeat returns the same result")
	299	+	assert.Equal(t, commit.GetDigest(), again.GetDigest())
	300	+	_, err = up.commit(id, "b0")
	301	+	assert.Equal(t, connect.CodeFailedPrecondition, connect.CodeOf(err), "another list on a committed artifact")
	302	+	_, err = up.block(id, "b9", []byte("late"))
	303	+	assert.Equal(t, connect.CodeFailedPrecondition, connect.CodeOf(err), "no block after the commit")
	304	+	assert.EqualValues(t, len(content), c.usage(), "the content replaces the charge of the blocks")
	305	+	for _, f := range c.artifactFiles() {
	306	+	assert.NotContains(t, f, ".blocks", "the blocks are dropped at the commit")
	307	+	}
	308	+
	309	+	_, err = up.finalize("dist", digestOf([]byte("other")))
	310	+	assert.Equal(t, connect.CodeInvalidArgument, connect.CodeOf(err), "a digest that differs")
	311	+	a, err := up.finalize("dist", strings.ToUpper(digestOf(content)[7:]))
	312	+	assert.Equal(t, connect.CodeInvalidArgument, connect.CodeOf(err), "a digest without its prefix")
	313	+	require.Nil(t, a)
	314	+	a, err = up.finalize("dist", digestOf(content))
	315	+	require.NoError(t, err)
	316	+	assert.Equal(t, id, a.GetId())
	317	+	assert.Equal(t, upJob.GetJobId(), a.GetJobId())
	318	+	assert.Equal(t, domain.CIArtifactZip, a.GetMediaType())
	319	+	repeat, err := up.finalize("dist", "")
	320	+	require.NoError(t, err, "a repeat returns the same artifact")
	321	+	assert.Equal(t, id, repeat.GetId())
	322	+
	323	+	t.Run("another job of the run lists and downloads it", func(t *testing.T) {
	324	+	got, err := down.list("")
	325	+	require.NoError(t, err)
	326	+	require.Len(t, got, 1)
	327	+	assert.Equal(t, "dist", got[0].GetName())
	328	+	assert.EqualValues(t, len(content), got[0].GetSize())
	329	+	meta, body, err := down.download(id, 0)
	330	+	require.NoError(t, err)
	331	+	assert.Equal(t, "dist", meta.GetName())
	332	+	assert.True(t, bytes.Equal(content, body))
	333	+	_, body, err = down.download(id, 3<<19+5)
	334	+	require.NoError(t, err, "a resumed download")
	335	+	assert.True(t, bytes.Equal(content[3<<19+5:], body))
	336	+	_, body, err = down.download(id, int64(len(content)))
	337	+	require.NoError(t, err)
	338	+	assert.Empty(t, body)
	339	+	_, _, err = down.download(id, int64(len(content))+1)
	340	+	assert.Equal(t, connect.CodeOutOfRange, connect.CodeOf(err))
	341	+	})
	342	+
	343	+	t.Run("a name is unique in the attempt; overwrite deletes first", func(t *testing.T) {
	344	+	_, err := down.create("dist", nil)
	345	+	assert.Equal(t, connect.CodeAlreadyExists, connect.CodeOf(err))
	346	+	require.NoError(t, down.remove("dist"))
	347	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(down.remove("dist")))
	348	+	assert.Zero(t, c.usage(), "deleting refunds")
	349	+	b := down.upload("dist", []byte("v2"))
	350	+	assert.Greater(t, b.GetId(), id, "ids increase, never reused")
	351	+	_, body, err := up.download(b.GetId(), 0)
	352	+	require.NoError(t, err)
	353	+	assert.Equal(t, "v2", string(body))
	354	+	})
	355	+
	356	+	t.Run("names are checked, never used as paths", func(t *testing.T) {
	357	+	for _, name := range []string{"", "../x", "a/b", `a\b`, "a:b", "nul\x00", strings.Repeat("n", 257)} {
	358	+	_, err := up.create(name, nil)
	359	+	assert.Equal(t, connect.CodeInvalidArgument, connect.CodeOf(err), "%q", name)
	360	+	}
	361	+	})
	362	+
	363	+	t.Run("the owners: repo:read to list and download, ci:write to delete", func(t *testing.T) {
	364	+	res, err := c.artifacts(adminToken).owner.ListArtifacts(c.ctx, connect.NewRequest(&pb.ListArtifactsRequest{Repo: c.repo, RunId: a.GetRunId()}))
	365	+	require.NoError(t, err)
	366	+	require.Len(t, res.Msg.GetArtifacts(), 1)
	367	+	assert.EqualValues(t, 2, res.Msg.GetUsedBytes())
	368	+	assert.EqualValues(t, domain.DefaultCIArtifactQuota, res.Msg.GetQuotaBytes())
	369	+
	370	+	// Anonymous: a private repository does not exist, a public one needs a
	371	+	// signed-in caller.
	372	+	anon := c.strangerArtifacts("")
	373	+	_, err = anon.owner.ListArtifacts(c.ctx, connect.NewRequest(&pb.ListArtifactsRequest{Repo: c.repo, RunId: a.GetRunId()}))
	374	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(err))
	375	+	public := domain.VisibilityPublic
	376	+	_, err = c.forge.Repos.Update(c.ctx, domain.Root(), c.name(c.repo), ports.UpdateRepoParams{Visibility: &public})
	377	+	require.NoError(t, err)
	378	+	_, err = anon.owner.ListArtifacts(c.ctx, connect.NewRequest(&pb.ListArtifactsRequest{Repo: c.repo, RunId: a.GetRunId()}))
	379	+	assert.Equal(t, connect.CodeUnauthenticated, connect.CodeOf(err))
	380	+	stream, err := anon.owner.DownloadArtifact(c.ctx, connect.NewRequest(&pb.DownloadArtifactRequest{Repo: c.repo, ArtifactId: res.Msg.GetArtifacts()[0].GetId()}))
	381	+	if err == nil {
	382	+	for stream.Receive() {
	383	+	t.Error("anonymous got content")
	384	+	}
	385	+	err = stream.Err()
	386	+	}
	387	+	assert.Equal(t, connect.CodeUnauthenticated, connect.CodeOf(err))
	388	+
	389	+	// A token with ci:write only reads the repository (public) but not its
	390	+	// artifacts; it may delete them.
	391	+	tok, err := c.root.token.CreateToken(c.ctx, connect.NewRequest(&pb.CreateTokenRequest{Label: "ci", Scopes: []pb.Scope{pb.Scope_SCOPE_CI_WRITE}}))
	392	+	require.NoError(t, err)
	393	+	ciw := c.strangerArtifacts(tok.Msg.GetSecret())
	394	+	_, err = ciw.owner.ListArtifacts(c.ctx, connect.NewRequest(&pb.ListArtifactsRequest{Repo: c.repo, RunId: a.GetRunId()}))
	395	+	assert.Equal(t, connect.CodePermissionDenied, connect.CodeOf(err))
	396	+	_, err = anon.owner.DeleteArtifact(c.ctx, connect.NewRequest(&pb.DeleteArtifactRequest{Repo: c.repo, ArtifactId: res.Msg.GetArtifacts()[0].GetId()}))
	397	+	assert.Equal(t, connect.CodeUnauthenticated, connect.CodeOf(err))
	398	+
	399	+	root := c.artifacts(adminToken)
	400	+	dl, err := root.owner.DownloadArtifact(c.ctx, connect.NewRequest(&pb.DownloadArtifactRequest{Repo: c.repo, ArtifactId: res.Msg.GetArtifacts()[0].GetId()}))
	401	+	require.NoError(t, err)
	402	+	var body []byte
	403	+	for dl.Receive() {
	404	+	body = append(body, dl.Msg().GetChunk()...)
	405	+	}
	406	+	require.NoError(t, dl.Err())
	407	+	assert.Equal(t, "v2", string(body))
	408	+	_, err = ciw.owner.DeleteArtifact(c.ctx, connect.NewRequest(&pb.DeleteArtifactRequest{Repo: c.repo, ArtifactId: res.Msg.GetArtifacts()[0].GetId()}))
	409	+	require.NoError(t, err)
	410	+	got, err := up.list("")
	411	+	require.NoError(t, err)
	412	+	assert.Empty(t, got)
	413	+	assert.Zero(t, c.usage())
	414	+	})
	415	+	}
	416	+
	417	+	// AC3: a job token of another run reaches nothing of this run, and the calls
	418	+	// are refused once the job ended.
	419	+	func TestCIArtifactsAreIsolatedPerRunAndJob(t *testing.T) {
	420	+	c := newCIHarness(t, nil)
	421	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	422	+	runA, upA, downA := c.newArtifactRun(runner, map[string]string{".github/workflows/ci.yml": artifactWorkflow})
	423	+	_, upB, _ := c.newArtifactRun(runner, map[string]string{"README": "two"})
	424	+	a, b := c.jobArtifacts(upA), c.jobArtifacts(upB)
	425	+
	426	+	secret := a.upload("secret", []byte("of run A"))
	427	+	pendingID := a.mustCreate("wip")
	428	+	_, err := a.block(pendingID, "x", []byte("half"))
	429	+	require.NoError(t, err)
	430	+
	431	+	t.Run("a job of run B", func(t *testing.T) {
	432	+	got, err := b.list("")
	433	+	require.NoError(t, err)
	434	+	assert.Empty(t, got, "lists its own run only")
	435	+	_, _, err = b.download(secret.GetId(), 0)
	436	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(err))
	437	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(b.remove("secret")))
	438	+	_, err = b.block(pendingID, "y", []byte("intrusion"))
	439	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(err))
	440	+	_, err = b.commit(pendingID, "x")
	441	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(err))
	442	+	_, err = b.finalize("wip", "")
	443	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(err))
	444	+	})
	445	+
	446	+	t.Run("another job of the same run uploads nothing into the job's upload", func(t *testing.T) {
	447	+	d := c.jobArtifacts(downA)
	448	+	_, err := d.block(pendingID, "y", []byte("intrusion"))
	449	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(err))
	450	+	_, err = d.commit(pendingID, "x")
	451	+	assert.Equal(t, connect.CodeNotFound, connect.CodeOf(err))
	452	+	})
	453	+
	454	+	t.Run("job ids in requests must be the token's", func(t *testing.T) {
	455	+	wrong := &jobArtifacts{c: c, job: &pb.Job{JobId: upA.GetJobId(), JobToken: upB.GetJobToken()}, api: c.strangerArtifacts(upB.GetJobToken())}
	456	+	_, err := wrong.list("")
	457	+	assert.Equal(t, connect.CodePermissionDenied, connect.CodeOf(err))
	458	+	_, _, err = wrong.download(secret.GetId(), 0)
	459	+	assert.Equal(t, connect.CodePermissionDenied, connect.CodeOf(err))
	460	+	assert.Equal(t, connect.CodePermissionDenied, connect.CodeOf(wrong.remove("secret")))
	461	+	_, err = wrong.block(pendingID, "z", []byte("x"))
	462	+	assert.Equal(t, connect.CodePermissionDenied, connect.CodeOf(err))
	463	+	})
	464	+
	465	+	t.Run("no token, a runner token, a repository token", func(t *testing.T) {
	466	+	for name, secret := range map[string]string{"none": "", "runner": runner.secret, "garbage": "ermitejob_nope", "admin": adminToken} {
	467	+	s := &jobArtifacts{c: c, job: upA, api: c.strangerArtifacts(secret)}
	468	+	_, err := s.list("")
	469	+	assert.Equal(t, connect.CodeUnauthenticated, connect.CodeOf(err), name)
	470	+	_, err = s.block(pendingID, "z", []byte("x"))
	471	+	assert.Equal(t, connect.CodeUnauthenticated, connect.CodeOf(err), name)
	472	+	}
	473	+	})
	474	+
	475	+	t.Run("the end of the job discards its uploads and refuses its token", func(t *testing.T) {
	476	+	before := c.usage()
	477	+	runner.runJob(upA, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	478	+	assert.EqualValues(t, before-int64(len("half")), c.usage(), "the unfinished upload is refunded")
	479	+	_, err := c.forge.store.Artifacts().GetArtifact(c.ctx, pendingID)
	480	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	481	+	for _, f := range c.artifactFiles() {
	482	+	assert.NotContains(t, f, fmt.Sprintf("%020d", pendingID), "its files are gone")
	483	+	}
	484	+	_, err = a.list("")
	485	+	endedDetail(t, err)
	486	+	_, err = a.create("late", nil)
	487	+	endedDetail(t, err)
	488	+	_, _, err = a.download(secret.GetId(), 0)
	489	+	endedDetail(t, err)
	490	+
	491	+	// The finalized artifact stays for the other jobs.
	492	+	got, err := c.jobArtifacts(downA).list("secret")
	493	+	require.NoError(t, err)
	494	+	require.Len(t, got, 1)
	495	+	assert.Equal(t, runA, got[0].GetRunId())
	496	+	})
	497	+	}
	498	+
	499	+	// shiftClock is the real time moved by an offset the test sets.
	500	+	type shiftClock struct{ off atomic.Int64 }
	501	+
	502	+	func (s *shiftClock) Now() time.Time { return time.Now().Add(time.Duration(s.off.Load())) }
	503	+	func (s *shiftClock) advance(d time.Duration) { s.off.Add(int64(d)) }
	504	+	func (s *shiftClock) since(t time.Time) time.Duration { return s.Now().Sub(t) }
	505	+
	506	+	func TestCIArtifactLimits(t *testing.T) {
	507	+	const mib = 1 << 20
	508	+	c := newCIHarness(t, func(cfg *usecase.CIConfig) {
	509	+	cfg.ArtifactLimits = ports.ArtifactLimits{Retention: 10 * 24 * time.Hour, Quota: 3 * mib, MaxSize: 2 * mib}
	510	+	})
	511	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	512	+	_, upJob, downJob := c.newArtifactRun(runner, map[string]string{".github/workflows/ci.yml": artifactWorkflow})
	513	+	up, down := c.jobArtifacts(upJob), c.jobArtifacts(downJob)
	514	+
	515	+	t.Run("the retention is kept between one day and the longest", func(t *testing.T) {
	516	+	res, err := up.create("soon", timestamppb.New(time.Now().Add(time.Hour)))
	517	+	require.NoError(t, err)
	518	+	assert.WithinDuration(t, time.Now().Add(24*time.Hour), res.GetExpiresAt().AsTime(), time.Minute)
	519	+	res, err = up.create("late", timestamppb.New(time.Now().Add(100024time.Hour)))
	520	+	require.NoError(t, err)
	521	+	assert.WithinDuration(t, time.Now().Add(1024time.Hour), res.GetExpiresAt().AsTime(), time.Minute)
	522	+	res, err = up.create("mid", timestamppb.New(time.Now().Add(524time.Hour)))
	523	+	require.NoError(t, err)
	524	+	assert.WithinDuration(t, time.Now().Add(524time.Hour), res.GetExpiresAt().AsTime(), time.Second)
	525	+	})
	526	+
	527	+	id := up.mustCreate("big")
	528	+	t.Run("a block over the size of one artifact is refused and nothing is kept", func(t *testing.T) {
	529	+	_, err := up.block(id, "a", pattern(2*mib+1, 0))
	530	+	require.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err))
	531	+	assert.Contains(t, err.Error(), "2 MiB")
	532	+	assert.Zero(t, c.usage())
	533	+	assert.Empty(t, c.artifactFiles())
	534	+	})
	535	+
	536	+	t.Run("blocks add up against the size of one artifact", func(t *testing.T) {
	537	+	_, err := up.block(id, "a", pattern(3*mib/2, 0))
	538	+	require.NoError(t, err)
	539	+	_, err = up.block(id, "b", pattern(mib, 0))
	540	+	require.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err))
	541	+	assert.EqualValues(t, 3*mib/2, c.usage())
	542	+	_, err = up.commit(id, "a", "a")
	543	+	require.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err), "a repeated block counts twice")
	544	+	_, err = up.commit(id, "a", "nope")
	545	+	assert.Equal(t, connect.CodeInvalidArgument, connect.CodeOf(err), "a block never uploaded")
	546	+	_, err = up.finalize("big", "")
	547	+	assert.Equal(t, connect.CodeFailedPrecondition, connect.CodeOf(err), "not committed")
	548	+	})
	549	+
	550	+	t.Run("the quota of the repository counts the uploads in progress", func(t *testing.T) {
	551	+	other := down.mustCreate("other")
	552	+	_, err := down.block(other, "a", pattern(mib, 1))
	553	+	require.NoError(t, err)
	554	+	_, err = down.block(other, "b", pattern(mib, 1))
	555	+	require.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err))
	556	+	assert.Contains(t, err.Error(), "quota (3 MiB)")
	557	+	// A retry of a stored block within the quota replaces it.
	558	+	_, err = down.block(other, "a", pattern(mib/2, 2))
	559	+	require.NoError(t, err)
	560	+	assert.EqualValues(t, 2*mib, c.usage())
	561	+	_, err = down.block(other, "b", pattern(mib, 1))
	562	+	require.NoError(t, err)
	563	+	_, err = down.create("third", nil)
	564	+	assert.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err), "the space is used up")
	565	+	})
	566	+	}
	567	+
	568	+	func TestCIArtifactsSweep(t *testing.T) {
	569	+	clk := &shiftClock{}
	570	+	c := newCIHarness(t, func(cfg *usecase.CIConfig) {
	571	+	cfg.Clock = clk
	572	+	cfg.Tick = 20 * time.Millisecond
	573	+	cfg.ArtifactLimits.Retention = 2 * 24 * time.Hour
	574	+	})
	575	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	576	+	runID, upJob, _ := c.newArtifactRun(runner, map[string]string{".github/workflows/ci.yml": artifactWorkflow})
	577	+	up := c.jobArtifacts(upJob)
	578	+	kept := up.upload("kept", []byte("content"))
	579	+
	580	+	// A crash of the server between the end of the job and its cleanup: an
	581	+	// unfinished upload stays recorded with its block; a crash between a file
	582	+	// and its record: files that no record names.
	583	+	crashed := up.mustCreate("crashed")
	584	+	_, err := up.block(crashed, "x", []byte("stale"))
	585	+	require.NoError(t, err)
	586	+	stray := filepath.Join(ciartifacts.Dir(c.data), fmt.Sprintf("%020d", runID), fmt.Sprintf("%020d", int64(999999)))
	587	+	require.NoError(t, os.WriteFile(stray, []byte("orphan"), 0o600))
	588	+	strayRun := filepath.Join(ciartifacts.Dir(c.data), fmt.Sprintf("%020d", int64(888888)))
	589	+	require.NoError(t, os.MkdirAll(strayRun, 0o700))
	590	+	start := clk.Now()
	591	+
	592	+	// The day after: the upload is older than a day and swept, the stray files
	593	+	// go, the finalized artifact stays.
	594	+	clk.advance(25 * time.Hour)
	595	+	c.waitFor("the sweep", func() bool {
	596	+	_, err := c.forge.store.Artifacts().GetArtifact(c.ctx, crashed)
	597	+	return errors.Is(err, domain.ErrNotFound)
	598	+	})
	599	+	c.waitFor("the stray files", func() bool {
	600	+	_, err1 := os.Stat(stray)
	601	+	_, err2 := os.Stat(strayRun)
	602	+	return os.IsNotExist(err1) && os.IsNotExist(err2)
	603	+	})
	604	+	assert.EqualValues(t, len("content"), c.usage(), "the unfinished upload is refunded")
	605	+	_, err = c.forge.store.Artifacts().GetArtifact(c.ctx, kept.GetId())
	606	+	require.NoError(t, err)
	607	+	for _, f := range c.artifactFiles() {
	608	+	assert.Equal(t, filepath.Join(fmt.Sprintf("%020d", runID), fmt.Sprintf("%020d", kept.GetId())), f)
	609	+	}
	610	+
	611	+	// Past its expiry it is invisible at once, then swept.
	612	+	clk.advance(24 * time.Hour)
	613	+	res, err := c.artifacts(adminToken).owner.ListArtifacts(c.ctx, connect.NewRequest(&pb.ListArtifactsRequest{Repo: c.repo, RunId: runID}))
	614	+	require.NoError(t, err)
	615	+	assert.Empty(t, res.Msg.GetArtifacts(), "expired")
	616	+	c.waitFor("the expired artifact swept", func() bool { return c.usage() == 0 && len(c.artifactFiles()) == 0 })
	617	+	assert.Greater(t, clk.since(start), 48*time.Hour)
	618	+	}
	619	+
	620	+	// Deleting the repository takes the records and the files of its artifacts.
	621	+	func TestCIArtifactsGoWithTheRepository(t *testing.T) {
	622	+	c := newCIHarness(t, nil)
	623	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	624	+	_, upJob, _ := c.newArtifactRun(runner, map[string]string{".github/workflows/ci.yml": artifactWorkflow})
	625	+	a := c.jobArtifacts(upJob).upload("dist", []byte("x"))
	626	+	require.NotEmpty(t, c.artifactFiles())
	627	+
	628	+	require.NoError(t, c.forge.Repos.Delete(c.ctx, domain.Root(), c.name(c.repo)))
	629	+	_, err := c.forge.store.Artifacts().GetArtifact(c.ctx, a.GetId())
	630	+	assert.ErrorIs(t, err, domain.ErrNotFound)
	631	+	assert.Empty(t, c.artifactFiles())
	632	+	}
	633	+
	634	+	// ERMITE_CI_ARTIFACTS=0 serves neither service.
	635	+	func TestCIArtifactsOff(t *testing.T) {
	636	+	h := newHarnessWith(t, harnessOpts{env: map[string]string{"ERMITE_CI_ARTIFACTS": "0"}}, nil)
	637	+	ver, err := h.api("").server.GetVersion(h.ctx, connect.NewRequest(&pb.GetVersionRequest{}))
	638	+	require.NoError(t, err)
	639	+	assert.NotContains(t, ver.Msg.GetFeatures(), "ci.artifacts")
	640	+	c := &ciHarness{harness: h}
	641	+	_, err = c.artifacts(adminToken).owner.ListArtifacts(h.ctx, connect.NewRequest(&pb.ListArtifactsRequest{Repo: "x", RunId: 1}))
	642	+	assert.Equal(t, connect.CodeUnimplemented, connect.CodeOf(err))
	643	+	_, err = c.artifacts("ermitejob_x").job.ListJobArtifacts(h.ctx, connect.NewRequest(&pb.ListJobArtifactsRequest{JobId: "1-0-0"}))
	644	+	assert.Equal(t, connect.CodeUnimplemented, connect.CodeOf(err))
	645	+	}
	646	+
	647	+	// Concurrent writes never pass the limits: the bytes of a block are reserved
	648	+	// before they are written, against the charges and the other writes.
	649	+	func TestCIArtifactLimitsHoldUnderConcurrency(t *testing.T) {
	650	+	const kib, mib = 1 << 10, 1 << 20
	651	+	c := newCIHarness(t, func(cfg *usecase.CIConfig) {
	652	+	cfg.ArtifactLimits = ports.ArtifactLimits{Quota: 3 * mib, MaxSize: 2 * mib}
	653	+	})
	654	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	655	+	_, upJob, _ := c.newArtifactRun(runner, map[string]string{".github/workflows/ci.yml": artifactWorkflow})
	656	+	up := c.jobArtifacts(upJob)
	657	+
	658	+	blockFiles := func() (total int64) {
	659	+	for _, f := range c.artifactFiles() {
	660	+	if info, err := os.Stat(filepath.Join(ciartifacts.Dir(c.data), f)); err == nil {
	661	+	total += info.Size()
	662	+	}
	663	+	}
	664	+	return total
	665	+	}
	666	+
	667	+	// peak samples the bytes on disk while fn runs.
	668	+	peak := func(fn func()) int64 {
	669	+	var top atomic.Int64
	670	+	done := make(chan struct{})
	671	+	go func() {
	672	+	for {
	673	+	if n := blockFiles(); n > top.Load() {
	674	+	top.Store(n)
	675	+	}
	676	+	select {
	677	+	case <-done:
	678	+	return
	679	+	default:
	680	+	}
	681	+	}
	682	+	}()
	683	+	fn()
	684	+	close(done)
	685	+	return top.Load()
	686	+	}
	687	+
	688	+	t.Run("one artifact: parallel blocks stop at the size of one artifact", func(t *testing.T) {
	689	+	id := up.mustCreate("one")
	690	+	var stored atomic.Int64
	691	+	var wg sync.WaitGroup
	692	+	top := peak(func() {
	693	+	for i := range 12 {
	694	+	wg.Go(func() {
	695	+	n, err := up.block(id, fmt.Sprint("b", i), pattern(512*kib, byte(i)))
	696	+	if err == nil {
	697	+	stored.Add(n)
	698	+	return
	699	+	}
	700	+	assert.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err))
	701	+	})
	702	+	}
	703	+	wg.Wait()
	704	+	})
	705	+	assert.LessOrEqual(t, top, int64(2*mib), "nothing beyond the limit ever reached the disk")
	706	+	assert.LessOrEqual(t, stored.Load(), int64(2*mib), "never past the size of one artifact")
	707	+	assert.Positive(t, stored.Load())
	708	+	assert.Equal(t, stored.Load(), c.usage(), "what is stored is what is charged")
	709	+	assert.Equal(t, stored.Load(), blockFiles(), "no refused block left on disk")
	710	+	})
	711	+
	712	+	t.Run("several artifacts: parallel blocks stop at the quota", func(t *testing.T) {
	713	+	ids := []int64{up.mustCreate("two"), up.mustCreate("three")}
	714	+	before := c.usage()
	715	+	var stored atomic.Int64
	716	+	var wg sync.WaitGroup
	717	+	for i := range 12 {
	718	+	wg.Go(func() {
	719	+	n, err := up.block(ids[i%2], fmt.Sprint("b", i), pattern(256*kib, byte(i)))
	720	+	if err == nil {
	721	+	stored.Add(n)
	722	+	return
	723	+	}
	724	+	assert.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err))
	725	+	})
	726	+	}
	727	+	wg.Wait()
	728	+	assert.LessOrEqual(t, c.usage(), int64(3*mib), "never past the quota")
	729	+	assert.Positive(t, stored.Load())
	730	+	assert.Equal(t, before+stored.Load(), c.usage())
	731	+	assert.Equal(t, c.usage(), blockFiles(), "no refused block left on disk")
	732	+	})
	733	+	}
	734	+
	735	+	func TestCIArtifactCaps(t *testing.T) {
	736	+	c := newCIHarness(t, nil)
	737	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	738	+	_, upJob, downJob := c.newArtifactRun(runner, map[string]string{".github/workflows/ci.yml": artifactWorkflow})
	739	+	up, down := c.jobArtifacts(upJob), c.jobArtifacts(downJob)
	740	+
	741	+	t.Run("a job has a bounded number of unfinished uploads", func(t *testing.T) {
	742	+	for i := range domain.CIArtifactMaxPendingPerJob {
	743	+	up.mustCreate(fmt.Sprint("p", i))
	744	+	}
	745	+	_, err := up.create("one-more", nil)
	746	+	assert.Equal(t, connect.CodeResourceExhausted, connect.CodeOf(err))
	747	+	_, err = up.create("p0", nil)
	748	+	require.NoError(t, err, "a new create of a pending name replaces it")
	749	+	down.mustCreate("other job")
	750	+	})
	751	+
	752	+	t.Run("a commit repeated while it runs, or after its caller left, gives one result", func(t *testing.T) {
	753	+	content := pattern(6<<20, 9)
	754	+	id := down.mustCreate("big")
	755	+	_, err := down.block(id, "a", content[:3<<20])
	756	+	require.NoError(t, err)
	757	+	_, err = down.block(id, "b", content[3<<20:])
	758	+	require.NoError(t, err)
	759	+
	760	+	// A caller that gives up at once: the commit still completes.
	761	+	gone, cancel := context.WithTimeout(c.ctx, time.Millisecond)
	762	+	_, _ = down.api.job.CommitJobArtifact(gone, connect.NewRequest(&pb.CommitJobArtifactRequest{JobId: downJob.GetJobId(), ArtifactId: id, BlockIds: [][]byte{[]byte("a"), []byte("b")}}))
	763	+	cancel()
	764	+
	765	+	var wg sync.WaitGroup
	766	+	results := make([]*pb.CommitJobArtifactResponse, 4)
	767	+	for i := range results {
	768	+	wg.Go(func() {
	769	+	res, err := down.commit(id, "a", "b")
	770	+	assert.NoError(t, err)
	771	+	results[i] = res
	772	+	})
	773	+	}
	774	+	wg.Wait()
	775	+	for _, r := range results {
	776	+	require.NotNil(t, r)
	777	+	assert.Equal(t, digestOf(content), r.GetDigest())
	778	+	assert.EqualValues(t, len(content), r.GetSize())
	779	+	}
	780	+	_, body, err := func() (*pb.Artifact, []byte, error) {
	781	+	if _, err := down.finalize("big", ""); err != nil {
	782	+	return nil, nil, err
	783	+	}
	784	+	return up.download(id, 0)
	785	+	}()
	786	+	require.NoError(t, err)
	787	+	assert.True(t, bytes.Equal(content, body))
	788	+	})
	789	+	}

M internal/app/cli_e2e_test.go +1 -1


@@ -326,7 +326,7 @@ func TestAC6CLIEndToEnd(t *testing.T) {
326		ver := decode[versionOut](t, res.stdout)	326		ver := decode[versionOut](t, res.stdout)
327		assert.Equal(t, c.url, ver.Server.URL)	327		assert.Equal(t, c.url, ver.Server.URL)
328		assert.NotEmpty(t, ver.Server.Version)	328		assert.NotEmpty(t, ver.Server.Version)
329	-	assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.dispatch", "ci.cache", "ci.secrets"}, ver.Server.Features)	329	+	assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.dispatch", "ci.cache", "ci.secrets", "ci.artifacts"}, ver.Server.Features)
330		assert.Equal(t, []string{"passkeys"}, ver.Server.MissingFeatures)	330		assert.Equal(t, []string{"passkeys"}, ver.Server.MissingFeatures)
331		stdout, stderr, code = c.bin(t, secret, "version")	331		stdout, stderr, code = c.bin(t, secret, "version")
332		require.Equal(t, 0, code, stderr)	332		require.Equal(t, 0, code, stderr)

M internal/app/cli_run.go +5 -3

 func newRunCmd(env *cliEnv) *cobra.Command {
 	run := &cobra.Command{
 		Use:   "run",
-		Short: "List, read, watch, cancel and re-run CI runs",
+		Short: "List, read, watch, cancel and re-run CI runs, and download their artifacts",
 		Long: "Work with the CI runs of a repository (the workflows that .github/workflows start on a push).\n" +
-			"Reading needs the access of reading the repository; cancel and rerun need the ci:write scope.",
+			"Reading needs the access of reading the repository; cancel and rerun need the ci:write scope.\n" +
+			"Artifacts are never served anonymously, even for a public repository: they need a token.",
 	}
-	run.AddCommand(newRunListCmd(env), newRunViewCmd(env), newRunWatchCmd(env), newRunCancelCmd(env), newRunRerunCmd(env))
+	run.AddCommand(newRunListCmd(env), newRunViewCmd(env), newRunWatchCmd(env), newRunCancelCmd(env), newRunRerunCmd(env),
+		newRunArtifactsCmd(env), newRunDownloadCmd(env))
 	return run
 }

A internal/app/cli_run_artifacts.go +411 -0

@@ -0,0 +1,411 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package app
	4	+
	5	+	import (
	6	+	"archive/zip"
	7	+	"context"
	8	+	"crypto/rand"
	9	+	"encoding/hex"
	10	+	"errors"
	11	+	"fmt"
	12	+	"io"
	13	+	"io/fs"
	14	+	"os"
	15	+	"path"
	16	+	"path/filepath"
	17	+	"slices"
	18	+	"strconv"
	19	+	"strings"
	20	+
	21	+	"connectrpc.com/connect"
	22	+	"github.com/spf13/cobra"
	23	+
	24	+	"bnema.dev/ermite-api/gen/ermite"
	25	+	"bnema.dev/ermite/internal/adapters/client"
	26	+	)
	27	+
	28	+	// artifactZip is the media type of an artifact that is a zip archive.
	29	+	const artifactZip = "application/zip"
	30	+
	31	+	// maxExtractBytes bounds what the extraction of one archive writes: a small
	32	+	// archive can declare a huge content.
	33	+	var maxExtractBytes int64 = 32 << 30
	34	+
	35	+	// maxZipEntries bounds the entries of one archive.
	36	+	const maxZipEntries = 100_000
	37	+
	38	+	// jsonArtifact is the stable JSON shape of an artifact.
	39	+	type jsonArtifact struct {
	40	+	ID int64 `json:"id"`
	41	+	Name string `json:"name"`
	42	+	Size int64 `json:"size"`
	43	+	Digest string `json:"digest"`
	44	+	MediaType string `json:"media_type"`
	45	+	RunID int64 `json:"run_id"`
	46	+	Attempt int32 `json:"attempt"`
	47	+	JobID string `json:"job_id"`
	48	+	CreatedAt *string `json:"created_at"`
	49	+	ExpiresAt *string `json:"expires_at"`
	50	+	}
	51	+
	52	+	type jsonArtifactList struct {
	53	+	Artifacts []jsonArtifact `json:"artifacts"`
	54	+	UsedBytes int64 `json:"used_bytes"`
	55	+	QuotaBytes int64 `json:"quota_bytes"`
	56	+	}
	57	+
	58	+	func toJSONArtifact(a *ermite.Artifact) jsonArtifact {
	59	+	return jsonArtifact{
	60	+	ID: a.GetId(), Name: a.GetName(), Size: a.GetSize(), Digest: a.GetDigest(), MediaType: a.GetMediaType(),
	61	+	RunID: a.GetRunId(), Attempt: a.GetAttempt(), JobID: a.GetJobId(),
	62	+	CreatedAt: jsonTime(a.GetCreatedAt()), ExpiresAt: jsonTime(a.GetExpiresAt()),
	63	+	}
	64	+	}
	65	+
	66	+	// humanBytes is a size for people, in binary units.
	67	+	func humanBytes(n int64) string {
	68	+	const unit = 1024
	69	+	if n < unit {
	70	+	return strconv.FormatInt(n, 10) + " B"
	71	+	}
	72	+	div, exp := int64(unit), 0
	73	+	for m := n / unit; m >= unit && exp < 4; m /= unit {
	74	+	div *= unit
	75	+	exp++
	76	+	}
	77	+	return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTP"[exp])
	78	+	}
	79	+
	80	+	// listRunArtifacts reads the artifacts of a run. A server without them
	81	+	// answers unimplemented, said in words here.
	82	+	func listRunArtifacts(ctx context.Context, c client.Client, repo string, runID int64) (ermite.ListArtifactsResponse, error) {
	83	+	res, err := c.Artifacts().ListArtifacts(ctx, connect.NewRequest(&ermite.ListArtifactsRequest{Repo: repo, RunId: runID}))
	84	+	if connect.CodeOf(err) == connect.CodeUnimplemented {
	85	+	return nil, fmt.Errorf("the server does not keep artifacts (it lacks the ci.artifacts feature)")
	86	+	}
	87	+	if err != nil {
	88	+	return nil, err
	89	+	}
	90	+	return res.Msg, nil
	91	+	}
	92	+
	93	+	func newRunArtifactsCmd(env cliEnv) cobra.Command {
	94	+	cmd := &cobra.Command{
	95	+	Use: "artifacts <repo> <run id>",
	96	+	Short: "List the artifacts of a run",
	97	+	Long: "List the artifacts that the jobs of a run uploaded (actions/upload-artifact), with the space the\n" +
	98	+	"artifacts of the repository use. Needs a token that reads the repository, even a public one.\n" +
	99	+	"JSON fields (--json): artifacts (id, name, size, digest, media_type, run_id, attempt, job_id,\n" +
	100	+	"created_at, expires_at), used_bytes and quota_bytes.",
	101	+	Args: cobra.ExactArgs(2),
	102	+	ValidArgsFunction: completeRepoNames(env),
	103	+	}
	104	+	asJSON := addJSONFlag(cmd)
	105	+	cmd.RunE = func(cmd *cobra.Command, args []string) error {
	106	+	id, err := parseRunID(args[1])
	107	+	if err != nil {
	108	+	return err
	109	+	}
	110	+	c, err := env.client(cmd)
	111	+	if err != nil {
	112	+	return err
	113	+	}
	114	+	res, err := listRunArtifacts(env.ctx(cmd), c, args[0], id)
	115	+	if err != nil {
	116	+	return err
	117	+	}
	118	+	if *asJSON {
	119	+	out := jsonArtifactList{Artifacts: []jsonArtifact{}, UsedBytes: res.GetUsedBytes(), QuotaBytes: res.GetQuotaBytes()}
	120	+	for _, a := range res.GetArtifacts() {
	121	+	out.Artifacts = append(out.Artifacts, toJSONArtifact(a))
	122	+	}
	123	+	return writeJSON(cmd.OutOrStdout(), out)
	124	+	}
	125	+	w := cmd.OutOrStdout()
	126	+	if len(res.GetArtifacts()) == 0 {
	127	+	noticef(cmd, "run %d has no artifacts", id)
	128	+	} else {
	129	+	rows := make([][]string, 0, len(res.GetArtifacts()))
	130	+	for _, a := range res.GetArtifacts() {
	131	+	rows = append(rows, []string{
	132	+	strconv.FormatInt(a.GetId(), 10), a.GetName(), humanBytes(a.GetSize()), shortTime(a.GetExpiresAt(), "-"),
	133	+	})
	134	+	}
	135	+	if err := table(w, []string{"ID", "NAME", "SIZE", "EXPIRES"}, rows); err != nil {
	136	+	return err
	137	+	}
	138	+	}
	139	+	_, _ = fmt.Fprintln(w, quotaLine(res.GetUsedBytes(), res.GetQuotaBytes()))
	140	+	return nil
	141	+	}
	142	+	return cmd
	143	+	}
	144	+
	145	+	// quotaLine says how much of its space the artifacts of the repository use.
	146	+	func quotaLine(used, quota int64) string {
	147	+	if quota <= 0 {
	148	+	return "artifacts of the repository: " + humanBytes(used) + " used"
	149	+	}
	150	+	return "artifacts of the repository: " + humanBytes(used) + " of " + humanBytes(quota) + " used"
	151	+	}
	152	+
	153	+	func newRunDownloadCmd(env cliEnv) cobra.Command {
	154	+	var names []string
	155	+	var dir string
	156	+	cmd := &cobra.Command{
	157	+	Use: "download <repo> <run id>",
	158	+	Short: "Download the artifacts of a run",
	159	+	Long: "Download the artifacts of a run, all of them or those that --name chooses (repeat it for several).\n" +
	160	+	"A zip artifact is extracted: into --dir when one artifact is downloaded, into <dir>/<name> when\n" +
	161	+	"several are. An artifact that is not a zip (uploaded with archive: false) is saved as <dir>/<name>.\n" +
	162	+	"Existing files are never overwritten. Needs a token that reads the repository, even a public one.",
	163	+	Args: cobra.ExactArgs(2),
	164	+	ValidArgsFunction: completeRepoNames(env),
	165	+	}
	166	+	cmd.Flags().StringArrayVarP(&names, "name", "n", nil, "download the artifact of this name (repeatable)")
	167	+	cmd.Flags().StringVarP(&dir, "dir", "D", ".", "the directory to download into")
	168	+	cmd.RunE = func(cmd *cobra.Command, args []string) error {
	169	+	id, err := parseRunID(args[1])
	170	+	if err != nil {
	171	+	return err
	172	+	}
	173	+	c, err := env.client(cmd)
	174	+	if err != nil {
	175	+	return err
	176	+	}
	177	+	ctx := env.ctx(cmd)
	178	+	res, err := listRunArtifacts(ctx, c, args[0], id)
	179	+	if err != nil {
	180	+	return err
	181	+	}
	182	+	picked, err := pickArtifacts(res.GetArtifacts(), names)
	183	+	if err != nil {
	184	+	return err
	185	+	}
	186	+	if len(picked) == 0 {
	187	+	return fmt.Errorf("run %d has no artifacts", id)
	188	+	}
	189	+	for _, a := range picked {
	190	+	if err := checkArtifactName(a.GetName()); err != nil {
	191	+	return err
	192	+	}
	193	+	}
	194	+	if err := os.MkdirAll(dir, 0o755); err != nil {
	195	+	return err
	196	+	}
	197	+	root, err := os.OpenRoot(dir)
	198	+	if err != nil {
	199	+	return err
	200	+	}
	201	+	defer func() { _ = root.Close() }()
	202	+	for _, a := range picked {
	203	+	to, err := downloadArtifact(ctx, c, args[0], a, root, len(picked) > 1)
	204	+	if err != nil {
	205	+	return fmt.Errorf("artifact %q: %w", clean(a.GetName()), err)
	206	+	}
	207	+	noticef(cmd, "downloaded %s to %s", clean(a.GetName()), clean(filepath.Join(dir, to)))
	208	+	}
	209	+	return nil
	210	+	}
	211	+	return cmd
	212	+	}
	213	+
	214	+	// pickArtifacts selects the artifacts of the names, every one when there is
	215	+	// none. A name that no artifact has is an error.
	216	+	func pickArtifacts(all []ermite.Artifact, names []string) ([]ermite.Artifact, error) {
	217	+	if len(names) == 0 {
	218	+	return all, nil
	219	+	}
	220	+	var out []*ermite.Artifact
	221	+	for _, n := range names {
	222	+	i := slices.IndexFunc(all, func(a *ermite.Artifact) bool { return a.GetName() == n })
	223	+	if i < 0 {
	224	+	return nil, fmt.Errorf("the run has no artifact %q", clean(n))
	225	+	}
	226	+	if !slices.Contains(out, all[i]) {
	227	+	out = append(out, all[i])
	228	+	}
	229	+	}
	230	+	return out, nil
	231	+	}
	232	+
	233	+	// checkArtifactName refuses a name that is not a plain file name: the server's
	234	+	// rules exclude separators, but the name comes from a job.
	235	+	func checkArtifactName(name string) error {
	236	+	if name == "" \|\| name == "." \|\| name == ".." \|\| strings.ContainsAny(name, "/\\\x00") \|\| !filepath.IsLocal(name) {
	237	+	return fmt.Errorf("artifact %q: the name is not usable as a file name", clean(name))
	238	+	}
	239	+	return nil
	240	+	}
	241	+
	242	+	// downloadArtifact streams an artifact into a temporary file of root, then
	243	+	// extracts it (a zip) or renames it to its name. It returns the path written,
	244	+	// relative to root.
	245	+	func downloadArtifact(ctx context.Context, c client.Client, repo string, a ermite.Artifact, root *os.Root, several bool) (string, error) {
	246	+	var rnd [8]byte
	247	+	_, _ = rand.Read(rnd[:])
	248	+	tmp := ".ermite-download-" + hex.EncodeToString(rnd[:])
	249	+	f, err := root.OpenFile(tmp, os.O_RDWR\|os.O_CREATE\|os.O_EXCL, 0o600)
	250	+	if err != nil {
	251	+	return "", err
	252	+	}
	253	+	defer func() {
	254	+	_ = f.Close()
	255	+	_ = root.Remove(tmp)
	256	+	}()
	257	+	got, err := streamArtifact(ctx, c, repo, a.GetId(), f)
	258	+	if err != nil {
	259	+	return "", err
	260	+	}
	261	+	if got.GetMediaType() != artifactZip {
	262	+	if err := f.Close(); err != nil {
	263	+	return "", err
	264	+	}
	265	+	if _, err := root.Lstat(a.GetName()); err == nil {
	266	+	return "", fmt.Errorf("%s already exists", a.GetName())
	267	+	} else if !errors.Is(err, fs.ErrNotExist) {
	268	+	return "", err
	269	+	}
	270	+	return a.GetName(), root.Rename(tmp, a.GetName())
	271	+	}
	272	+	into, to := root, "."
	273	+	if several {
	274	+	to = a.GetName()
	275	+	if err := root.MkdirAll(to, 0o755); err != nil {
	276	+	return "", err
	277	+	}
	278	+	if into, err = root.OpenRoot(to); err != nil {
	279	+	return "", err
	280	+	}
	281	+	defer func() { _ = into.Close() }()
	282	+	}
	283	+	size, err := f.Seek(0, io.SeekEnd)
	284	+	if err != nil {
	285	+	return "", err
	286	+	}
	287	+	if err := extractZip(f, size, into, maxExtractBytes); err != nil {
	288	+	return "", err
	289	+	}
	290	+	return to, nil
	291	+	}
	292	+
	293	+	// streamArtifact writes the content of an artifact to w and returns the
	294	+	// artifact, checking that the content has its size.
	295	+	func streamArtifact(ctx context.Context, c client.Client, repo string, id int64, w io.Writer) (ermite.Artifact, error) {
	296	+	stream, err := c.Artifacts().DownloadArtifact(ctx, connect.NewRequest(&ermite.DownloadArtifactRequest{Repo: repo, ArtifactId: id}))
	297	+	if err != nil {
	298	+	return nil, err
	299	+	}
	300	+	defer func() { _ = stream.Close() }()
	301	+	var a *ermite.Artifact
	302	+	var n int64
	303	+	for stream.Receive() {
	304	+	msg := stream.Msg()
	305	+	if a == nil {
	306	+	if a = msg.GetArtifact(); a == nil {
	307	+	return nil, errors.New("the server did not describe the artifact")
	308	+	}
	309	+	}
	310	+	chunk := msg.GetChunk()
	311	+	if n+int64(len(chunk)) > a.GetSize() {
	312	+	return nil, errors.New("the server sent more than the size of the artifact")
	313	+	}
	314	+	if _, err := w.Write(chunk); err != nil {
	315	+	return nil, err
	316	+	}
	317	+	n += int64(len(chunk))
	318	+	}
	319	+	if err := stream.Err(); err != nil {
	320	+	return nil, err
	321	+	}
	322	+	if a == nil \|\| n != a.GetSize() {
	323	+	return nil, errors.New("the download ended before the end of the artifact")
	324	+	}
	325	+	return a, nil
	326	+	}
	327	+
	328	+	// extractZip writes the files of the archive into root. Every entry must be
	329	+	// a local path (no absolute path, no "..") and a directory or a regular file:
	330	+	// a symbolic link or any other kind fails the whole extraction. Files that
	331	+	// exist are not overwritten, and the extraction writes at most limit bytes.
	332	+	// os.Root keeps every write inside the directory, whatever links it holds.
	333	+	func extractZip(r io.ReaderAt, size int64, root *os.Root, limit int64) error {
	334	+	zr, err := zip.NewReader(r, size)
	335	+	if err != nil && !errors.Is(err, zip.ErrInsecurePath) {
	336	+	return fmt.Errorf("not a zip archive: %w", err)
	337	+	}
	338	+	if len(zr.File) > maxZipEntries {
	339	+	return fmt.Errorf("the archive has more than %d entries", maxZipEntries)
	340	+	}
	341	+	var total uint64
	342	+	for _, f := range zr.File {
	343	+	if err := checkZipEntry(f); err != nil {
	344	+	return err
	345	+	}
	346	+	total += f.UncompressedSize64
	347	+	}
	348	+	if limit < 0 \|\| total > uint64(limit) {
	349	+	return fmt.Errorf("the archive expands to more than %s", humanBytes(limit))
	350	+	}
	351	+	remaining := limit
	352	+	for _, f := range zr.File {
	353	+	name := path.Clean(f.Name)
	354	+	if f.FileInfo().IsDir() {
	355	+	if err := root.MkdirAll(filepath.FromSlash(name), 0o755); err != nil {
	356	+	return err
	357	+	}
	358	+	continue
	359	+	}
	360	+	if dir := path.Dir(name); dir != "." {
	361	+	if err := root.MkdirAll(filepath.FromSlash(dir), 0o755); err != nil {
	362	+	return err
	363	+	}
	364	+	}
	365	+	n, err := extractZipFile(f, root, filepath.FromSlash(name), remaining)
	366	+	if err != nil {
	367	+	return err
	368	+	}
	369	+	remaining -= n
	370	+	}
	371	+	return nil
	372	+	}
	373	+
	374	+	// checkZipEntry refuses an entry that is not a local directory or regular file.
	375	+	func checkZipEntry(f *zip.File) error {
	376	+	name := f.Name
	377	+	if name == "" \|\| strings.HasPrefix(name, "/") \|\| strings.Contains(name, "\\") \|\| strings.ContainsRune(name, 0) \|\|
	378	+	slices.Contains(strings.Split(name, "/"), "..") \|\| !filepath.IsLocal(filepath.FromSlash(path.Clean(name))) {
	379	+	return fmt.Errorf("the archive has an unsafe path %q", clean(name))
	380	+	}
	381	+	mode := f.Mode()
	382	+	if !mode.IsDir() && !mode.IsRegular() {
	383	+	return fmt.Errorf("the archive has %q, which is not a regular file or a directory", clean(name))
	384	+	}
	385	+	return nil
	386	+	}
	387	+
	388	+	// extractZipFile writes one file of the archive, at most limit bytes.
	389	+	func extractZipFile(f zip.File, root os.Root, name string, limit int64) (int64, error) {
	390	+	perm := os.FileMode(0o644)
	391	+	if f.Mode()&0o100 != 0 {
	392	+	perm = 0o755
	393	+	}
	394	+	rc, err := f.Open()
	395	+	if err != nil {
	396	+	return 0, err
	397	+	}
	398	+	defer func() { _ = rc.Close() }()
	399	+	out, err := root.OpenFile(name, os.O_WRONLY\|os.O_CREATE\|os.O_EXCL, perm)
	400	+	if err != nil {
	401	+	return 0, err
	402	+	}
	403	+	n, err := io.Copy(out, io.LimitReader(rc, limit+1))
	404	+	if cerr := out.Close(); err == nil {
	405	+	err = cerr
	406	+	}
	407	+	if err == nil && n > limit {
	408	+	err = fmt.Errorf("the archive expands to more than declared")
	409	+	}
	410	+	return n, err
	411	+	}

A internal/app/cli_run_artifacts_test.go +292 -0

@@ -0,0 +1,292 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package app
	4	+
	5	+	import (
	6	+	"archive/zip"
	7	+	"bytes"
	8	+	"compress/flate"
	9	+	"context"
	10	+	"fmt"
	11	+	"net/http"
	12	+	"net/http/httptest"
	13	+	"os"
	14	+	"path/filepath"
	15	+	"testing"
	16	+
	17	+	"connectrpc.com/connect"
	18	+	"github.com/stretchr/testify/assert"
	19	+	"github.com/stretchr/testify/require"
	20	+
	21	+	"bnema.dev/ermite-api/gen/ermite"
	22	+	"bnema.dev/ermite-api/gen/ermite/ermiteconnect"
	23	+	)
	24	+
	25	+	// fakeArtifactAPI serves the artifacts of run 1 of proj.
	26	+	type fakeArtifactAPI struct {
	27	+	ermiteconnect.UnimplementedArtifactServiceHandler
	28	+	arts []*ermite.Artifact
	29	+	content map[int64][]byte
	30	+	}
	31	+
	32	+	func (f fakeArtifactAPI) ListArtifacts(_ context.Context, r connect.Request[ermite.ListArtifactsRequest]) (*connect.Response[ermite.ListArtifactsResponse], error) {
	33	+	if r.Msg.GetRepo() != "proj" \|\| r.Msg.GetRunId() != 1 {
	34	+	return nil, connect.NewError(connect.CodeNotFound, nil)
	35	+	}
	36	+	return connect.NewResponse(&ermite.ListArtifactsResponse{Artifacts: f.arts, UsedBytes: 3 << 20, QuotaBytes: 10 << 30}), nil
	37	+	}
	38	+
	39	+	func (f fakeArtifactAPI) DownloadArtifact(_ context.Context, r connect.Request[ermite.DownloadArtifactRequest], st *connect.ServerStream[ermite.DownloadArtifactResponse]) error {
	40	+	for _, a := range f.arts {
	41	+	if a.GetId() != r.Msg.GetArtifactId() {
	42	+	continue
	43	+	}
	44	+	if err := st.Send(&ermite.DownloadArtifactResponse{Artifact: a}); err != nil {
	45	+	return err
	46	+	}
	47	+	data := f.content[a.GetId()]
	48	+	for len(data) > 0 {
	49	+	n := min(len(data), 7)
	50	+	if err := st.Send(&ermite.DownloadArtifactResponse{Chunk: data[:n]}); err != nil {
	51	+	return err
	52	+	}
	53	+	data = data[n:]
	54	+	}
	55	+	return nil
	56	+	}
	57	+	return connect.NewError(connect.CodeNotFound, nil)
	58	+	}
	59	+
	60	+	func (f *fakeArtifactAPI) add(name, mediaType string, data []byte) {
	61	+	if f.content == nil {
	62	+	f.content = map[int64][]byte{}
	63	+	}
	64	+	id := int64(len(f.arts) + 1)
	65	+	f.arts = append(f.arts, &ermite.Artifact{
	66	+	Id: id, Repo: "proj", RunId: 1, Attempt: 1, JobId: "1-0-0", Name: name, MediaType: mediaType,
	67	+	Size: int64(len(data)), ExpiresAt: ts("2027-01-08T12:00:00Z"),
	68	+	})
	69	+	f.content[id] = data
	70	+	}
	71	+
	72	+	func (f fakeArtifactAPI) serve(t testing.T) string {
	73	+	t.Helper()
	74	+	mux := http.NewServeMux()
	75	+	p, h := ermiteconnect.NewArtifactServiceHandler(f)
	76	+	mux.Handle("/-/api"+p, http.StripPrefix("/-/api", h))
	77	+	srv := httptest.NewServer(mux)
	78	+	t.Cleanup(srv.Close)
	79	+	return srv.URL
	80	+	}
	81	+
	82	+	type zipEntry struct {
	83	+	name, body string
	84	+	mode os.FileMode
	85	+	}
	86	+
	87	+	func zipOf(t *testing.T, entries ...zipEntry) []byte {
	88	+	t.Helper()
	89	+	var b bytes.Buffer
	90	+	zw := zip.NewWriter(&b)
	91	+	for _, e := range entries {
	92	+	h := &zip.FileHeader{Name: e.name, Method: zip.Deflate}
	93	+	if e.mode != 0 {
	94	+	h.SetMode(e.mode)
	95	+	}
	96	+	w, err := zw.CreateHeader(h)
	97	+	require.NoError(t, err)
	98	+	_, err = w.Write([]byte(e.body))
	99	+	require.NoError(t, err)
	100	+	}
	101	+	require.NoError(t, zw.Close())
	102	+	return b.Bytes()
	103	+	}
	104	+
	105	+	func readFile(t *testing.T, p string) string {
	106	+	t.Helper()
	107	+	b, err := os.ReadFile(p)
	108	+	require.NoError(t, err)
	109	+	return string(b)
	110	+	}
	111	+
	112	+	// noTemp checks that no temporary file of a download is left in dir.
	113	+	func noTemp(t *testing.T, dir string) {
	114	+	t.Helper()
	115	+	left, err := filepath.Glob(filepath.Join(dir, ".ermite-download-*"))
	116	+	require.NoError(t, err)
	117	+	assert.Empty(t, left)
	118	+	}
	119	+
	120	+	func TestRunArtifactsList(t *testing.T) {
	121	+	f := &fakeArtifactAPI{}
	122	+	f.add("dist", "application/zip", []byte("PK"))
	123	+	f.add("report.txt", "text/plain", bytes.Repeat([]byte("x"), 1536))
	124	+	url := f.serve(t)
	125	+	out := runCLI(t, envFor(url), "", "run", "artifacts", "proj", "1").ok(t)
	126	+	assert.Contains(t, out.stdout, "ID")
	127	+	assert.Contains(t, out.stdout, "EXPIRES")
	128	+	assert.Regexp(t, `1\s+dist\s+2 B\s+2027-01-08 12:00`, out.stdout)
	129	+	assert.Regexp(t, `2\s+report\.txt\s+1\.5 KiB`, out.stdout)
	130	+	assert.Contains(t, out.stdout, "3.0 MiB of 10.0 GiB used")
	131	+
	132	+	js := decode[jsonArtifactList](t, runCLI(t, envFor(url), "", "run", "artifacts", "proj", "1", "--json").ok(t).stdout)
	133	+	require.Len(t, js.Artifacts, 2)
	134	+	assert.Equal(t, "report.txt", js.Artifacts[1].Name)
	135	+	assert.Equal(t, int64(10<<30), js.QuotaBytes)
	136	+
	137	+	assert.Error(t, runCLI(t, envFor(url), "", "run", "artifacts", "proj", "2").err)
	138	+	}
	139	+
	140	+	func TestRunDownloadOneZip(t *testing.T) {
	141	+	f := &fakeArtifactAPI{}
	142	+	f.add("dist", "application/zip", zipOf(t,
	143	+	zipEntry{name: "bin/"}, zipEntry{name: "bin/app", body: "binary", mode: 0o755}, zipEntry{name: "README", body: "read me"}))
	144	+	dir := filepath.Join(t.TempDir(), "out")
	145	+	runCLI(t, envFor(f.serve(t)), "", "run", "download", "proj", "1", "--dir", dir).ok(t)
	146	+	assert.Equal(t, "binary", readFile(t, filepath.Join(dir, "bin", "app")))
	147	+	assert.Equal(t, "read me", readFile(t, filepath.Join(dir, "README")))
	148	+	st, err := os.Stat(filepath.Join(dir, "bin", "app"))
	149	+	require.NoError(t, err)
	150	+	assert.NotZero(t, st.Mode()&0o100, "the executable bit")
	151	+	noTemp(t, dir)
	152	+	}
	153	+
	154	+	func TestRunDownloadSeveral(t *testing.T) {
	155	+	f := &fakeArtifactAPI{}
	156	+	f.add("linux", "application/zip", zipOf(t, zipEntry{name: "app", body: "linux"}))
	157	+	f.add("darwin", "application/zip", zipOf(t, zipEntry{name: "app", body: "darwin"}))
	158	+	f.add("notes.txt", "text/plain", []byte("plain file"))
	159	+	url := f.serve(t)
	160	+
	161	+	dir := t.TempDir()
	162	+	runCLI(t, envFor(url), "", "run", "download", "proj", "1", "--dir", dir).ok(t)
	163	+	assert.Equal(t, "linux", readFile(t, filepath.Join(dir, "linux", "app")))
	164	+	assert.Equal(t, "darwin", readFile(t, filepath.Join(dir, "darwin", "app")))
	165	+	assert.Equal(t, "plain file", readFile(t, filepath.Join(dir, "notes.txt")), "saved as is")
	166	+	noTemp(t, dir)
	167	+
	168	+	// Again into the same directory: nothing is overwritten.
	169	+	assert.Error(t, runCLI(t, envFor(url), "", "run", "download", "proj", "1", "--dir", dir).err)
	170	+	noTemp(t, dir)
	171	+
	172	+	// --name chooses; two names are several, one is extracted into the dir.
	173	+	two := t.TempDir()
	174	+	runCLI(t, envFor(url), "", "run", "download", "proj", "1", "--dir", two, "--name", "darwin", "-n", "notes.txt").ok(t)
	175	+	assert.Equal(t, "darwin", readFile(t, filepath.Join(two, "darwin", "app")))
	176	+	assert.Equal(t, "plain file", readFile(t, filepath.Join(two, "notes.txt")))
	177	+	assert.NoDirExists(t, filepath.Join(two, "linux"))
	178	+
	179	+	one := t.TempDir()
	180	+	runCLI(t, envFor(url), "", "run", "download", "proj", "1", "--dir", one, "--name", "linux").ok(t)
	181	+	assert.Equal(t, "linux", readFile(t, filepath.Join(one, "app")))
	182	+
	183	+	unknown := runCLI(t, envFor(url), "", "run", "download", "proj", "1", "--dir", t.TempDir(), "--name", "nope")
	184	+	require.Error(t, unknown.err)
	185	+	assert.Contains(t, unknown.err.Error(), `no artifact "nope"`)
	186	+	}
	187	+
	188	+	func TestRunDownloadRefusesUnsafeArchives(t *testing.T) {
	189	+	for name, entries := range map[string][]zipEntry{
	190	+	"dot-dot": {{name: "ok", body: "x"}, {name: "../escape", body: "x"}},
	191	+	"deep": {{name: "a/../../escape", body: "x"}},
	192	+	"absolute": {{name: "/tmp/escape", body: "x"}},
	193	+	"symlink": {{name: "link", body: "/etc/passwd", mode: os.ModeSymlink \| 0o777}},
	194	+	"backslash": {{name: `a\..\..\escape`, body: "x"}},
	195	+	"dir-link": {{name: "d", body: "/", mode: os.ModeSymlink \| 0o777}, {name: "d/escape", body: "x"}},
	196	+	"device": {{name: "dev", body: "", mode: os.ModeDevice \| 0o666}},
	197	+	} {
	198	+	t.Run(name, func(t *testing.T) {
	199	+	f := &fakeArtifactAPI{}
	200	+	f.add("evil", "application/zip", zipOf(t, entries...))
	201	+	parent := t.TempDir()
	202	+	dir := filepath.Join(parent, "out")
	203	+	res := runCLI(t, envFor(f.serve(t)), "", "run", "download", "proj", "1", "--dir", dir)
	204	+	require.Error(t, res.err)
	205	+	assert.NoFileExists(t, filepath.Join(parent, "escape"))
	206	+	assert.NoFileExists(t, filepath.Join(dir, "link"))
	207	+	assert.NoFileExists(t, filepath.Join(dir, "ok"), "nothing is written from a refused archive")
	208	+	noTemp(t, dir)
	209	+	})
	210	+	}
	211	+	}
	212	+
	213	+	func TestExtractZipBoundsTheExpansion(t *testing.T) {
	214	+	data := zipOf(t, zipEntry{name: "big", body: string(bytes.Repeat([]byte("0"), 1<<20))})
	215	+	dir := t.TempDir()
	216	+	root, err := os.OpenRoot(dir)
	217	+	require.NoError(t, err)
	218	+	defer func() { _ = root.Close() }()
	219	+	err = extractZip(bytes.NewReader(data), int64(len(data)), root, 1<<10)
	220	+	require.Error(t, err)
	221	+	assert.Contains(t, err.Error(), "expands to more than")
	222	+	assert.NoFileExists(t, filepath.Join(dir, "big"))
	223	+	}
	224	+
	225	+	// A setuid or setgid bit and other mode bits of an entry are dropped: files
	226	+	// are 0644, or 0755 when the owner could execute them.
	227	+	func TestExtractZipKeepsOnlyPlainModes(t *testing.T) {
	228	+	data := zipOf(t, zipEntry{name: "suid", body: "x", mode: os.ModeSetuid \| os.ModeSetgid \| 0o4777}, zipEntry{name: "plain", body: "y", mode: 0o666})
	229	+	dir := t.TempDir()
	230	+	root, err := os.OpenRoot(dir)
	231	+	require.NoError(t, err)
	232	+	defer func() { _ = root.Close() }()
	233	+	require.NoError(t, extractZip(bytes.NewReader(data), int64(len(data)), root, 1<<20))
	234	+	for name, want := range map[string]os.FileMode{"suid": 0o755, "plain": 0o644} {
	235	+	info, err := os.Stat(filepath.Join(dir, name))
	236	+	require.NoError(t, err)
	237	+	assert.Zero(t, info.Mode()&(os.ModeSetuid\|os.ModeSetgid\|os.ModeSticky), name)
	238	+	assert.Equal(t, want, info.Mode().Perm()&want, name)
	239	+	assert.Zero(t, info.Mode().Perm()&^0o755, name)
	240	+	}
	241	+	}
	242	+
	243	+	// An entry whose header understates its size (a zip bomb that lies) is cut at
	244	+	// the limit: the declared sizes are not trusted alone.
	245	+	func TestExtractZipStopsALyingEntry(t *testing.T) {
	246	+	var deflated bytes.Buffer
	247	+	fw, err := flate.NewWriter(&deflated, flate.BestCompression)
	248	+	require.NoError(t, err)
	249	+	_, err = fw.Write(bytes.Repeat([]byte{0}, 4<<20))
	250	+	require.NoError(t, err)
	251	+	require.NoError(t, fw.Close())
	252	+	var b bytes.Buffer
	253	+	zw := zip.NewWriter(&b)
	254	+	w, err := zw.CreateRaw(&zip.FileHeader{Name: "bomb", Method: zip.Deflate, CompressedSize64: uint64(deflated.Len()), UncompressedSize64: 10})
	255	+	require.NoError(t, err)
	256	+	_, err = w.Write(deflated.Bytes())
	257	+	require.NoError(t, err)
	258	+	require.NoError(t, zw.Close())
	259	+
	260	+	dir := t.TempDir()
	261	+	root, err := os.OpenRoot(dir)
	262	+	require.NoError(t, err)
	263	+	defer func() { _ = root.Close() }()
	264	+	require.Error(t, extractZip(bytes.NewReader(b.Bytes()), int64(b.Len()), root, 1<<20))
	265	+	if info, err := os.Stat(filepath.Join(dir, "bomb")); err == nil {
	266	+	assert.LessOrEqual(t, info.Size(), int64(1<<20)+1, "never more than the limit written")
	267	+	}
	268	+	}
	269	+
	270	+	func TestExtractZipBoundsTheEntries(t *testing.T) {
	271	+	entries := make([]zipEntry, maxZipEntries+1)
	272	+	for i := range entries {
	273	+	entries[i] = zipEntry{name: fmt.Sprint("f", i)}
	274	+	}
	275	+	data := zipOf(t, entries...)
	276	+	dir := t.TempDir()
	277	+	root, err := os.OpenRoot(dir)
	278	+	require.NoError(t, err)
	279	+	defer func() { _ = root.Close() }()
	280	+	err = extractZip(bytes.NewReader(data), int64(len(data)), root, 1<<30)
	281	+	require.ErrorContains(t, err, "entries")
	282	+	assert.NoFileExists(t, filepath.Join(dir, "f0"))
	283	+	}
	284	+
	285	+	func TestCheckArtifactName(t *testing.T) {
	286	+	for _, n := range []string{"", ".", "..", "a/b", `a\b`, "/etc/passwd", "../x", "a\x00b"} {
	287	+	assert.Error(t, checkArtifactName(n), n)
	288	+	}
	289	+	assert.NoError(t, checkArtifactName("dist-linux.tar"))
	290	+	assert.Equal(t, "1.5 KiB", humanBytes(1536))
	291	+	assert.Equal(t, "10.0 GiB", humanBytes(10<<30))
	292	+	}

M internal/app/cli_version.go +1 -0


@@ -120,5 +120,6 @@ var featureUse = map[string]string{
120	client.FeatureCIRunners: "`ermite runner` needs: the commands will fail on this server",	120		client.FeatureCIRunners: "`ermite runner` needs: the commands will fail on this server",
121	client.FeatureCISecrets: "`ermite secret`, `variable` and `environment` need: the commands will fail on this server",	121		client.FeatureCISecrets: "`ermite secret`, `variable` and `environment` need: the commands will fail on this server",
122	client.FeatureCIDispatch: "`ermite workflow run` needs: the command will fail on this server",	122		client.FeatureCIDispatch: "`ermite workflow run` needs: the command will fail on this server",
		123	+	client.FeatureCIArtifacts: "`ermite run artifacts` and `run download` need: the commands will fail on this server",
123	client.FeaturePasskeys: "`ermite web passkey` needs: the commands will fail on this server (passkeys need an https public URL with a DNS host name)",	124		client.FeaturePasskeys: "`ermite web passkey` needs: the commands will fail on this server (passkeys need an https public URL with a DNS host name)",
124	}	125		}

M internal/app/cli_version_test.go +5 -5


@@ -17,7 +17,7 @@ import (
17		)	17		)
18			18
19		// allFeatures are the features this client uses.	19		// allFeatures are the features this client uses.
20	-	var allFeatures = []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "token.branch_prefixes", "ci.runs", "ci.runners", "ci.secrets", "ci.dispatch", "passkeys"}	20	+	var allFeatures = []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "token.branch_prefixes", "ci.runs", "ci.runners", "ci.secrets", "ci.dispatch", "ci.artifacts", "passkeys"}
21			21
22		// Every feature the client uses needs a sentence for the version warning.	22		// Every feature the client uses needs a sentence for the version warning.
23		func TestEveryUsedFeatureExplainsWhatBreaks(t *testing.T) {	23		func TestEveryUsedFeatureExplainsWhatBreaks(t *testing.T) {

@@ -45,15 +45,15 @@ func TestVersionWithServer(t *testing.T) {
45		f := versionFake(allFeatures...)	45		f := versionFake(allFeatures...)
46		url := f.serve(t)	46		url := f.serve(t)
47		res := runCLI(t, envFor(url), "", "version").ok(t)	47		res := runCLI(t, envFor(url), "", "version").ok(t)
48	-	assert.Equal(t, "client ermite v0.0.1\nserver v0.4.0 at "+url+"\nfeatures repo.update, events.resume, merge_requests, merge_requests.delete_branch, signing_key, session, token.branch_prefixes, ci.runs, ci.runners, ci.secrets, ci.dispatch, passkeys\n", res.stdout)	48	+	assert.Equal(t, "client ermite v0.0.1\nserver v0.4.0 at "+url+"\nfeatures repo.update, events.resume, merge_requests, merge_requests.delete_branch, signing_key, session, token.branch_prefixes, ci.runs, ci.runners, ci.secrets, ci.dispatch, ci.artifacts, passkeys\n", res.stdout)
49		assert.Empty(t, res.stderr, "no warning when every feature is there")	49		assert.Empty(t, res.stderr, "no warning when every feature is there")
50			50
51		res = runCLI(t, envFor(url), "", "version", "--json").ok(t)	51		res = runCLI(t, envFor(url), "", "version", "--json").ok(t)
52	-	assert.JSONEq(t, `{"client":{"version":"v0.0.1"},"server":{"url":"`+url+`","version":"v0.4.0","features":["repo.update","events.resume","merge_requests","merge_requests.delete_branch","signing_key","session","token.branch_prefixes","ci.runs","ci.runners","ci.secrets","ci.dispatch","passkeys"],"missing_features":[]}}`, res.stdout)	52	+	assert.JSONEq(t, `{"client":{"version":"v0.0.1"},"server":{"url":"`+url+`","version":"v0.4.0","features":["repo.update","events.resume","merge_requests","merge_requests.delete_branch","signing_key","session","token.branch_prefixes","ci.runs","ci.runners","ci.secrets","ci.dispatch","ci.artifacts","passkeys"],"missing_features":[]}}`, res.stdout)
53		}	53		}
54			54
55		func TestVersionWarnsAboutMissingFeatures(t *testing.T) {	55		func TestVersionWarnsAboutMissingFeatures(t *testing.T) {
56	-	f := versionFake("repo.update", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "token.branch_prefixes", "ci.runs", "ci.runners", "ci.secrets", "ci.dispatch", "passkeys")	56	+	f := versionFake("repo.update", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "token.branch_prefixes", "ci.runs", "ci.runners", "ci.secrets", "ci.dispatch", "ci.artifacts", "passkeys")
57		url := f.serve(t)	57		url := f.serve(t)
58		res := runCLI(t, envFor(url), "", "version").ok(t)	58		res := runCLI(t, envFor(url), "", "version").ok(t)
59		assert.Contains(t, res.stdout, "v0.4.0")	59		assert.Contains(t, res.stdout, "v0.4.0")

@@ -61,7 +61,7 @@ func TestVersionWarnsAboutMissingFeatures(t *testing.T) {
61		assert.NotContains(t, res.stderr, "repo.update")	61		assert.NotContains(t, res.stderr, "repo.update")
62			62
63		res = runCLI(t, envFor(url), "", "version", "--json").ok(t)	63		res = runCLI(t, envFor(url), "", "version", "--json").ok(t)
64	-	assert.JSONEq(t, `{"client":{"version":"v0.0.1"},"server":{"url":"`+url+`","version":"v0.4.0","features":["repo.update","merge_requests","merge_requests.delete_branch","signing_key","session","token.branch_prefixes","ci.runs","ci.runners","ci.secrets","ci.dispatch","passkeys"],"missing_features":["events.resume"]}}`, res.stdout)	64	+	assert.JSONEq(t, `{"client":{"version":"v0.0.1"},"server":{"url":"`+url+`","version":"v0.4.0","features":["repo.update","merge_requests","merge_requests.delete_branch","signing_key","session","token.branch_prefixes","ci.runs","ci.runners","ci.secrets","ci.dispatch","ci.artifacts","passkeys"],"missing_features":["events.resume"]}}`, res.stdout)
65		assert.Contains(t, res.stderr, "events.resume", "the warning also goes to stderr with --json")	65		assert.Contains(t, res.stderr, "events.resume", "the warning also goes to stderr with --json")
66			66
67		f = versionFake()	67		f = versionFake()

M internal/app/forge.go +20 -0


@@ -16,6 +16,7 @@ import (
16		16
17	"github.com/bnema/zerowrap"	17		"github.com/bnema/zerowrap"
18		18
		19	+	"bnema.dev/ermite/internal/adapters/ciartifacts"
19	"bnema.dev/ermite/internal/adapters/cilogs"	20		"bnema.dev/ermite/internal/adapters/cilogs"
20	"bnema.dev/ermite/internal/adapters/cisummaries"	21		"bnema.dev/ermite/internal/adapters/cisummaries"
21	"bnema.dev/ermite/internal/adapters/ermitegit"	22		"bnema.dev/ermite/internal/adapters/ermitegit"

@@ -277,6 +278,7 @@ func newForge(ctx context.Context, cfg config.Config, opts options) (_ *forge, e
277	Workflows: opts.workflows(githubraw.New(githubraw.Config{})),	278		Workflows: opts.workflows(githubraw.New(githubraw.Config{})),
278	OIDC: f.OIDC,	279		OIDC: f.OIDC,
279	}	280		}
		281	+	artifacts(&ciCfg, cfg, store)
280	f.pullRequests(&ciCfg, store, git, committer)	282		f.pullRequests(&ciCfg, store, git, committer)
281	if opts.ci != nil {	283		if opts.ci != nil {
282	opts.ci(&ciCfg)	284		opts.ci(&ciCfg)

@@ -338,6 +340,11 @@ func newForge(ctx context.Context, cfg config.Config, opts options) (_ *forge, e
338	if f.OIDC != nil {	340		if f.OIDC != nil {
339	apiOIDC = f.OIDC	341		apiOIDC = f.OIDC
340	}	342		}
		343	+	var jobArtifacts ports.JobArtifactService
		344	+	var artifacts ports.ArtifactService
		345	+	if cfg.CIArtifacts {
		346	+	jobArtifacts, artifacts = f.CI, f.CI
		347	+	}
341	f.API = rpc.New(rpc.Config{	348		f.API = rpc.New(rpc.Config{
342	Auth: f.Auth,	349		Auth: f.Auth,
343	Repos: f.Repos,	350		Repos: f.Repos,

@@ -354,6 +361,8 @@ func newForge(ctx context.Context, cfg config.Config, opts options) (_ *forge, e
354	Variables: f.Settings,	361		Variables: f.Settings,
355	Environments: f.Settings,	362		Environments: f.Settings,
356	OIDCKeys: apiOIDC,	363		OIDCKeys: apiOIDC,
		364	+	JobArtifacts: jobArtifacts,
		365	+	Artifacts: artifacts,
357	Version: serverVersion(),	366		Version: serverVersion(),
358	TrustedProxies: cfg.TrustedProxies,	367		TrustedProxies: cfg.TrustedProxies,
359	Limiter: limiter,	368		Limiter: limiter,

@@ -473,6 +482,17 @@ func (f *forge) Close(ctx context.Context) {
473	}	482		}
474	}	483		}
475		484
		485	+	// artifacts gives the CI the stores and limits of the artifacts (CI-4 D3):
		486	+	// records in the kv store, contents under ci/artifacts. ERMITE_CI_ARTIFACTS=0
		487	+	// gives none, and the API then serves no artifact service.
		488	+	func artifacts(ci usecase.CIConfig, cfg config.Config, store kvstore.Store) {
		489	+	if !cfg.CIArtifacts {
		490	+	return
		491	+	}
		492	+	ci.Artifacts, ci.ArtifactFiles = store.Artifacts(), ciartifacts.New(ciartifacts.Dir(cfg.DataDir))
		493	+	ci.ArtifactLimits = ports.ArtifactLimits{Retention: cfg.CIArtifactRetention, Quota: cfg.CIArtifactQuota, MaxSize: cfg.CIArtifactMax}
		494	+	}
		495	+
476	// pullRequests gives the CI the merge requests, the refs/pull refs and the	496		// pullRequests gives the CI the merge requests, the refs/pull refs and the
477	// identity of the test merge commits (CI-3, D5).	497		// identity of the test merge commits (CI-3, D5).
478	func (f forge) pullRequests(cfg usecase.CIConfig, store kvstore.Store, git gitcli.Git, committer domain.Identity) {	498		func (f forge) pullRequests(cfg usecase.CIConfig, store kvstore.Store, git gitcli.Git, committer domain.Identity) {

M internal/app/web.go +7 -0


@@ -35,11 +35,18 @@ func newWeb(cfg config.Config, f forge, limiter clientlimit.Limiter) (http.Han
35	if f.Passkeys != nil {	35		if f.Passkeys != nil {
36	passkeys = webPasskeys{f.Passkeys}	36		passkeys = webPasskeys{f.Passkeys}
37	}	37		}
		38	+	// Without artifacts (ERMITE_CI_ARTIFACTS=0) the run page has no section
		39	+	// for them and no download route.
		40	+	var artifacts ports.ArtifactService
		41	+	if cfg.CIArtifacts {
		42	+	artifacts = f.CI
		43	+	}
38	return web.New(web.Config{	44		return web.New(web.Config{
39	Repos: f.Repos,	45		Repos: f.Repos,
40	Browse: f.Browse,	46		Browse: f.Browse,
41	MergeRequests: f.MergeRequests,	47		MergeRequests: f.MergeRequests,
42	Runs: f.CI,	48		Runs: f.CI,
		49	+	Artifacts: artifacts,
43	Events: f.EventLog,	50		Events: f.EventLog,
44	Owner: cfg.Owner,	51		Owner: cfg.Owner,
45	PublicURL: cfg.PublicURL,	52		PublicURL: cfg.PublicURL,

M internal/config/config.go +71 -0


@@ -47,6 +47,15 @@ const (
47	// random bytes in base64, provided by the deployment secret store like	47		// random bytes in base64, provided by the deployment secret store like
48	// ERMITE_ADMIN_TOKEN. Without it secrets cannot be set and jobs get none.	48		// ERMITE_ADMIN_TOKEN. Without it secrets cannot be set and jobs get none.
49	EnvSecretsKey = "ERMITE_SECRETS_KEY"	49		EnvSecretsKey = "ERMITE_SECRETS_KEY"
		50	+	// EnvCIArtifacts turns the CI artifacts (artifacts v4) off when "0" or
		51	+	// "false"; they are on by default.
		52	+	EnvCIArtifacts = "ERMITE_CI_ARTIFACTS"
		53	+	// EnvCIArtifactRetention is the default and longest retention of an
		54	+	// artifact, EnvCIArtifactQuota the space of the artifacts of one
		55	+	// repository, EnvCIArtifactMax the size of one artifact.
		56	+	EnvCIArtifactRetention = "ERMITE_CI_ARTIFACT_RETENTION"
		57	+	EnvCIArtifactQuota = "ERMITE_CI_ARTIFACT_QUOTA"
		58	+	EnvCIArtifactMax = "ERMITE_CI_ARTIFACT_MAX"
50	)	59		)
51		60
52	// SecretsKeyLen is the length of the secrets key in bytes (AES-256).	61		// SecretsKeyLen is the length of the secrets key in bytes (AES-256).

@@ -124,6 +133,13 @@ type Config struct {
124	// when the variable is unset. It is a secret: never log it or put it in an	133		// when the variable is unset. It is a secret: never log it or put it in an
125	// error.	134		// error.
126	SecretsKey []byte	135		SecretsKey []byte
		136	+	// CIArtifacts is false when ERMITE_CI_ARTIFACTS turns the artifacts off.
		137	+	CIArtifacts bool
		138	+	// CIArtifactRetention, CIArtifactQuota and CIArtifactMax are the limits of
		139	+	// the artifacts; 0 (variable unset) leaves the default to the CI use case.
		140	+	CIArtifactRetention time.Duration
		141	+	CIArtifactQuota int64
		142	+	CIArtifactMax int64
127	}	143		}
128		144
129	// Error lists every configuration problem found, in variable order.	145		// Error lists every configuration problem found, in variable order.

@@ -294,12 +310,67 @@ func Load(get func(string) string) (Config, error) {
294	}	310		}
295	}	311		}
296		312
		313	+	switch raw := strings.ToLower(strings.TrimSpace(get(EnvCIArtifacts))); raw {
		314	+	case "", "1", "true":
		315	+	c.CIArtifacts = true
		316	+	case "0", "false":
		317	+	default:
		318	+	bad(EnvCIArtifacts, "must be 0, 1, false or true (got %q)", raw)
		319	+	}
		320	+	if raw := strings.TrimSpace(get(EnvCIArtifactRetention)); raw != "" {
		321	+	d, err := parseRetention(raw)
		322	+	switch {
		323	+	case err != nil:
		324	+	bad(EnvCIArtifactRetention, "%v", err)
		325	+	case d < 24*time.Hour:
		326	+	bad(EnvCIArtifactRetention, "must be at least one day (got %q)", raw)
		327	+	default:
		328	+	c.CIArtifactRetention = d
		329	+	}
		330	+	}
		331	+	for _, v := range []struct {
		332	+	env string
		333	+	dst *int64
		334	+	}{{EnvCIArtifactQuota, &c.CIArtifactQuota}, {EnvCIArtifactMax, &c.CIArtifactMax}} {
		335	+	if raw := strings.TrimSpace(get(v.env)); raw != "" {
		336	+	n, err := parseSize(raw)
		337	+	if err != nil {
		338	+	bad(v.env, "%v", err)
		339	+	} else {
		340	+	*v.dst = n
		341	+	}
		342	+	}
		343	+	}
		344	+
297	if len(problems) > 0 {	345		if len(problems) > 0 {
298	return Config{}, &Error{Problems: problems}	346		return Config{}, &Error{Problems: problems}
299	}	347		}
300	return c, nil	348		return c, nil
301	}	349		}
302		350
		351	+	// sizeUnits are the suffixes of parseSize, longest first.
		352	+	var sizeUnits = []struct {
		353	+	suffix string
		354	+	mult int64
		355	+	}{{"TiB", 1 << 40}, {"GiB", 1 << 30}, {"MiB", 1 << 20}, {"KiB", 1 << 10}, {"B", 1}}
		356	+
		357	+	// parseSize reads a size in bytes: a number with an optional suffix B, KiB,
		358	+	// MiB, GiB or TiB ("10GiB", "512 MiB"), from 1 MiB to 1 PiB.
		359	+	func parseSize(raw string) (int64, error) {
		360	+	num, mult := raw, int64(1)
		361	+	for _, u := range sizeUnits {
		362	+	if n, ok := strings.CutSuffix(raw, u.suffix); ok {
		363	+	num, mult = strings.TrimSpace(n), u.mult
		364	+	break
		365	+	}
		366	+	}
		367	+	n, err := strconv.ParseInt(num, 10, 64)
		368	+	if err != nil \|\| n < 1 \|\| n > (1<<50)/mult \|\| n*mult < 1<<20 {
		369	+	return 0, fmt.Errorf("must be a size from 1MiB to 1024TiB such as 10GiB (got %q)", raw)
		370	+	}
		371	+	return n * mult, nil
		372	+	}
		373	+
303	// decodeSecretsKey reads 32 bytes in standard or URL-safe base64, with or	374		// decodeSecretsKey reads 32 bytes in standard or URL-safe base64, with or
304	// without padding.	375		// without padding.
305	func decodeSecretsKey(raw string) ([]byte, bool) {	376		func decodeSecretsKey(raw string) ([]byte, bool) {

M internal/config/config_test.go +41 -0


@@ -319,3 +319,44 @@ func TestSecretsKey(t *testing.T) {
319	assert.NotContains(t, err.Error(), bad, "the value is never repeated: "+name)	319		assert.NotContains(t, err.Error(), bad, "the value is never repeated: "+name)
320	}	320		}
321	}	321		}
		322	+
		323	+	func TestCIArtifacts(t *testing.T) {
		324	+	c, err := config.Load(lookup(valid(nil)))
		325	+	require.NoError(t, err)
		326	+	assert.True(t, c.CIArtifacts, "on by default")
		327	+	assert.Zero(t, c.CIArtifactRetention, "unset: the defaults are left to the CI use case")
		328	+	assert.Zero(t, c.CIArtifactQuota)
		329	+	assert.Zero(t, c.CIArtifactMax)
		330	+
		331	+	c, err = config.Load(lookup(valid(map[string]string{
		332	+	config.EnvCIArtifacts: "0", config.EnvCIArtifactRetention: "30d",
		333	+	config.EnvCIArtifactQuota: "20GiB", config.EnvCIArtifactMax: "512 MiB",
		334	+	})))
		335	+	require.NoError(t, err)
		336	+	assert.False(t, c.CIArtifacts)
		337	+	assert.Equal(t, 3024time.Hour, c.CIArtifactRetention)
		338	+	assert.EqualValues(t, 20<<30, c.CIArtifactQuota)
		339	+	assert.EqualValues(t, 512<<20, c.CIArtifactMax)
		340	+
		341	+	for in, want := range map[string]int64{"1048576": 1 << 20, "2048KiB": 2 << 20, "1TiB": 1 << 40, "3 GiB": 3 << 30, "5368709120B": 5 << 30} {
		342	+	c, err := config.Load(lookup(valid(map[string]string{config.EnvCIArtifactMax: in})))
		343	+	require.NoError(t, err, in)
		344	+	assert.Equal(t, want, c.CIArtifactMax, in)
		345	+	}
		346	+
		347	+	for env, in := range map[string]string{
		348	+	config.EnvCIArtifacts: "maybe",
		349	+	config.EnvCIArtifactRetention: "12h",
		350	+	config.EnvCIArtifactQuota: "10GB",
		351	+	config.EnvCIArtifactMax: "1KiB",
		352	+	} {
		353	+	_, err := config.Load(lookup(valid(map[string]string{env: in})))
		354	+	var ce *config.Error
		355	+	require.True(t, errors.As(err, &ce), env)
		356	+	assert.Contains(t, ce.Problems[0], env)
		357	+	}
		358	+	for _, in := range []string{"0", "-1GiB", "2048TiB", "9223372036854775807GiB", "GiB"} {
		359	+	_, err := config.Load(lookup(valid(map[string]string{config.EnvCIArtifactQuota: in})))
		360	+	require.Error(t, err, in)
		361	+	}
		362	+	}

A internal/domain/ci_artifact.go +175 -0

@@ -0,0 +1,175 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package domain
	4	+
	5	+	import (
	6	+	"fmt"
	7	+	"strings"
	8	+	"time"
	9	+	"unicode"
	10	+	"unicode/utf8"
	11	+	)
	12	+
	13	+	// Limits and defaults of the CI artifacts (artifacts v4, CI-4 D3).
	14	+	const (
	15	+	// CIArtifactMaxNameLen is the longest artifact name, in characters.
	16	+	CIArtifactMaxNameLen = 256
	17	+	// CIArtifactMaxMediaTypeLen is the longest media type of an artifact.
	18	+	CIArtifactMaxMediaTypeLen = 255
	19	+	// CIArtifactMaxBlockIDLen is the longest block id (Azure's, decoded).
	20	+	CIArtifactMaxBlockIDLen = 64
	21	+	// CIArtifactMaxBlocks bounds the blocks of a commit and the blocks staged
	22	+	// for one artifact.
	23	+	CIArtifactMaxBlocks = 50_000
	24	+	// CIArtifactMaxPendingPerJob bounds the unfinished uploads of a job, and
	25	+	// CIArtifactMaxPerRun the artifacts of a run, unfinished ones included
	26	+	// (the toolkit uploads one artifact at a time).
	27	+	CIArtifactMaxPendingPerJob = 32
	28	+	CIArtifactMaxPerRun = 1000
	29	+	// CIArtifactChunk is the largest piece of data of one message, both ways.
	30	+	CIArtifactChunk = 1 << 20
	31	+	// CIArtifactMinRetention is the shortest retention of an artifact.
	32	+	CIArtifactMinRetention = 24 * time.Hour
	33	+	// CIArtifactPendingMax is how long an upload may stay unfinished before the
	34	+	// sweep deletes it, whatever its job does.
	35	+	CIArtifactPendingMax = 24 * time.Hour
	36	+	// DefaultCIArtifactQuota is the space the artifacts of one repository may
	37	+	// use, uploads in progress included.
	38	+	DefaultCIArtifactQuota = 10 << 30
	39	+	// DefaultCIArtifactMaxSize is the largest artifact.
	40	+	DefaultCIArtifactMaxSize = 5 << 30
	41	+	// CIArtifactZip is the media type of an archive.
	42	+	CIArtifactZip = "application/zip"
	43	+	)
	44	+
	45	+	// CIArtifactState is where an artifact is in its upload.
	46	+	type CIArtifactState string
	47	+
	48	+	const (
	49	+	// CIArtifactPending is created, its blocks being uploaded.
	50	+	CIArtifactPending CIArtifactState = "pending"
	51	+	// CIArtifactCommitted has its content; not visible yet.
	52	+	CIArtifactCommitted CIArtifactState = "committed"
	53	+	// CIArtifactFinal is finalized: visible to the run and its owners.
	54	+	CIArtifactFinal CIArtifactState = "final"
	55	+	)
	56	+
	57	+	// CIArtifact is a file that a job of a run uploaded with
	58	+	// actions/upload-artifact. It belongs to the run: every job of the run, in any
	59	+	// attempt, sees the finalized ones.
	60	+	type CIArtifact struct {
	61	+	// ID is global, monotonic and never reused.
	62	+	ID int64
	63	+	Repo RepoName
	64	+	RunID int64
	65	+	// Attempt is the attempt of the run of the job that uploads it.
	66	+	Attempt int32
	67	+	JobID string
	68	+	// Assignment is the hash of the job token of the assignment that uploads
	69	+	// it: a pending artifact belongs to that assignment only.
	70	+	Assignment TokenHash
	71	+	Name string
	72	+	MediaType string
	73	+	Size int64
	74	+	// Digest is "sha256:<hex>" of the content, known from the commit.
	75	+	Digest string
	76	+	State CIArtifactState
	77	+	CreatedAt time.Time
	78	+	ExpiresAt time.Time
	79	+	// Charged is what the artifact counts against the quota of its
	80	+	// repository: the blocks staged while pending, then the content.
	81	+	Charged int64
	82	+	// ListDigest identifies the block list of the commit.
	83	+	ListDigest []byte
	84	+	// Version is the store version of the record; 0 for a new one.
	85	+	Version uint64
	86	+	}
	87	+
	88	+	// Final reports whether the artifact is finalized.
	89	+	func (a CIArtifact) Final() bool { return a.State == CIArtifactFinal }
	90	+
	91	+	// Visible reports whether the artifact is finalized and not expired at now.
	92	+	func (a CIArtifact) Visible(now time.Time) bool { return a.Final() && now.Before(a.ExpiresAt) }
	93	+
	94	+	// UploadedBy reports whether the job, at its run attempt, uploads or uploaded
	95	+	// the artifact. It says nothing of the assignment.
	96	+	func (a CIArtifact) UploadedBy(j CIJob) bool {
	97	+	return a.RunID == j.RunID && a.Attempt == j.RunAttempt && a.JobID == j.ID
	98	+	}
	99	+
	100	+	// PendingOf reports whether the artifact is an unfinished upload of the
	101	+	// current assignment of the job.
	102	+	func (a CIArtifact) PendingOf(j CIJob) bool {
	103	+	return !a.Final() && a.UploadedBy(j) && a.Assignment == j.TokenHash && j.HasToken()
	104	+	}
	105	+
	106	+	// ParseArtifactName checks an artifact name: 1 to 256 characters of valid
	107	+	// UTF-8, without control characters, '"', ':', '<', '>', '\|', '*', '?', '\'
	108	+	// or '/' (the characters @actions/artifact refuses). A name is never used as a
	109	+	// path: these rules are the toolkit's.
	110	+	func ParseArtifactName(name string) error {
	111	+	if name == "" \|\| !utf8.ValidString(name) \|\| utf8.RuneCountInString(name) > CIArtifactMaxNameLen {
	112	+	return fmt.Errorf("%w: an artifact name has 1 to %d characters of valid UTF-8", ErrInvalid, CIArtifactMaxNameLen)
	113	+	}
	114	+	for _, r := range name {
	115	+	if unicode.IsControl(r) \|\| strings.ContainsRune(`":<>\|*?\/`, r) {
	116	+	return fmt.Errorf("%w: an artifact name has no control character and none of \" : < > \| * ? \\ /", ErrInvalid)
	117	+	}
	118	+	}
	119	+	return nil
	120	+	}
	121	+
	122	+	// ParseArtifactMediaType checks the media type of an artifact uploaded as is
	123	+	// and returns it, CIArtifactZip for an empty one: at most 255 printable ASCII
	124	+	// characters.
	125	+	func ParseArtifactMediaType(mt string) (string, error) {
	126	+	if mt == "" {
	127	+	return CIArtifactZip, nil
	128	+	}
	129	+	if len(mt) > CIArtifactMaxMediaTypeLen {
	130	+	return "", fmt.Errorf("%w: a media type has at most %d characters", ErrInvalid, CIArtifactMaxMediaTypeLen)
	131	+	}
	132	+	for i := 0; i < len(mt); i++ {
	133	+	if mt[i] < 0x20 \|\| mt[i] > 0x7e {
	134	+	return "", fmt.Errorf("%w: a media type has printable ASCII characters only", ErrInvalid)
	135	+	}
	136	+	}
	137	+	return mt, nil
	138	+	}
	139	+
	140	+	// ParseArtifactDigest checks a digest given by an uploader: "sha256:" and 64
	141	+	// hexadecimal characters, returned in lower case. Empty is no digest.
	142	+	func ParseArtifactDigest(d string) (string, error) {
	143	+	if d == "" {
	144	+	return "", nil
	145	+	}
	146	+	hex, ok := strings.CutPrefix(strings.ToLower(d), "sha256:")
	147	+	if !ok \|\| len(hex) != 64 \|\| strings.Trim(hex, "0123456789abcdef") != "" {
	148	+	return "", fmt.Errorf("%w: a digest is \"sha256:\" and 64 hexadecimal characters", ErrInvalid)
	149	+	}
	150	+	return "sha256:" + hex, nil
	151	+	}
	152	+
	153	+	// CheckArtifactBlockID checks a block id: 1 to 64 bytes.
	154	+	func CheckArtifactBlockID(id []byte) error {
	155	+	if len(id) == 0 \|\| len(id) > CIArtifactMaxBlockIDLen {
	156	+	return fmt.Errorf("%w: a block id has 1 to %d bytes", ErrInvalid, CIArtifactMaxBlockIDLen)
	157	+	}
	158	+	return nil
	159	+	}
	160	+
	161	+	// ArtifactExpiry is the expiry an artifact gets: the one asked for (zero:
	162	+	// none) kept between one day from now and the longest retention, now plus the
	163	+	// retention without one. A retention below one day counts as one day.
	164	+	func ArtifactExpiry(asked, now time.Time, retention time.Duration) time.Time {
	165	+	retention = max(retention, CIArtifactMinRetention)
	166	+	switch {
	167	+	case asked.IsZero():
	168	+	return now.Add(retention)
	169	+	case asked.Before(now.Add(CIArtifactMinRetention)):
	170	+	return now.Add(CIArtifactMinRetention)
	171	+	case asked.After(now.Add(retention)):
	172	+	return now.Add(retention)
	173	+	}
	174	+	return asked
	175	+	}

A internal/domain/ci_artifact_test.go +78 -0

@@ -0,0 +1,78 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package domain_test
	4	+
	5	+	import (
	6	+	"strings"
	7	+	"testing"
	8	+	"time"
	9	+
	10	+	"github.com/stretchr/testify/assert"
	11	+	"github.com/stretchr/testify/require"
	12	+
	13	+	"bnema.dev/ermite/internal/domain"
	14	+	)
	15	+
	16	+	func TestParseArtifactName(t *testing.T) {
	17	+	for _, ok := range []string{"dist", "my artifact (linux) v1.2", "é", strings.Repeat("é", 256), "..", "a.b-c_d"} {
	18	+	assert.NoError(t, domain.ParseArtifactName(ok), "%q", ok)
	19	+	}
	20	+	for _, bad := range []string{"", strings.Repeat("a", 257), "a/b", `a\b`, "a:b", `a"b`, "a<b", "a>b", "a\|b", "a*b", "a?b", "a\nb", "a\x00b", "\xff"} {
	21	+	assert.ErrorIs(t, domain.ParseArtifactName(bad), domain.ErrInvalid, "%q", bad)
	22	+	}
	23	+	}
	24	+
	25	+	func TestParseArtifactMediaTypeAndDigest(t *testing.T) {
	26	+	mt, err := domain.ParseArtifactMediaType("")
	27	+	require.NoError(t, err)
	28	+	assert.Equal(t, domain.CIArtifactZip, mt)
	29	+	mt, err = domain.ParseArtifactMediaType("text/plain; charset=utf-8")
	30	+	require.NoError(t, err)
	31	+	assert.Equal(t, "text/plain; charset=utf-8", mt)
	32	+	for _, bad := range []string{strings.Repeat("a", 256), "text/plain\r\nX-Evil: 1", "tëxt/plain"} {
	33	+	_, err := domain.ParseArtifactMediaType(bad)
	34	+	assert.ErrorIs(t, err, domain.ErrInvalid, "%q", bad)
	35	+	}
	36	+
	37	+	hex := strings.Repeat("ab", 32)
	38	+	d, err := domain.ParseArtifactDigest("SHA256:" + strings.ToUpper(hex))
	39	+	require.NoError(t, err)
	40	+	assert.Equal(t, "sha256:"+hex, d)
	41	+	d, err = domain.ParseArtifactDigest("")
	42	+	require.NoError(t, err)
	43	+	assert.Empty(t, d)
	44	+	for _, bad := range []string{hex, "sha256:" + hex[:62], "sha1:" + hex, "sha256:" + strings.Repeat("zz", 32)} {
	45	+	_, err := domain.ParseArtifactDigest(bad)
	46	+	assert.ErrorIs(t, err, domain.ErrInvalid, bad)
	47	+	}
	48	+	}
	49	+
	50	+	func TestArtifactExpiry(t *testing.T) {
	51	+	now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
	52	+	day := 24 * time.Hour
	53	+	assert.Equal(t, now.Add(90day), domain.ArtifactExpiry(time.Time{}, now, 90day), "unset: the retention")
	54	+	assert.Equal(t, now.Add(day), domain.ArtifactExpiry(now.Add(time.Hour), now, 90*day), "at least a day")
	55	+	assert.Equal(t, now.Add(day), domain.ArtifactExpiry(now.Add(-time.Hour), now, 90*day), "a time in the past")
	56	+	assert.Equal(t, now.Add(90day), domain.ArtifactExpiry(now.Add(400day), now, 90*day), "at most the retention")
	57	+	assert.Equal(t, now.Add(3day), domain.ArtifactExpiry(now.Add(3day), now, 90*day))
	58	+	assert.Equal(t, now.Add(day), domain.ArtifactExpiry(time.Time{}, now, time.Hour), "a retention below a day counts as a day")
	59	+	}
	60	+
	61	+	func TestArtifactOwnership(t *testing.T) {
	62	+	tok := domain.HashTokenSecret("ermitejob_a")
	63	+	j := domain.CIJob{ID: "5-0-0", RunID: 5, RunAttempt: 2, TokenHash: tok}
	64	+	a := domain.CIArtifact{RunID: 5, Attempt: 2, JobID: "5-0-0", Assignment: tok, State: domain.CIArtifactPending}
	65	+	assert.True(t, a.PendingOf(j))
	66	+	other := j
	67	+	other.TokenHash = domain.HashTokenSecret("ermitejob_b")
	68	+	assert.False(t, a.PendingOf(other), "another assignment of the job")
	69	+	assert.True(t, a.UploadedBy(other))
	70	+	next := j
	71	+	next.RunAttempt = 3
	72	+	assert.False(t, a.UploadedBy(next), "another attempt")
	73	+	a.State = domain.CIArtifactFinal
	74	+	assert.False(t, a.PendingOf(j))
	75	+	a.ExpiresAt = time.Now().Add(time.Hour)
	76	+	assert.True(t, a.Visible(time.Now()))
	77	+	assert.False(t, a.Visible(a.ExpiresAt))
	78	+	}

A internal/ports/ci_artifact.go +169 -0

@@ -0,0 +1,169 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package ports
	4	+
	5	+	import (
	6	+	"context"
	7	+	"io"
	8	+	"time"
	9	+
	10	+	"bnema.dev/ermite/internal/domain"
	11	+	)
	12	+
	13	+	// CIArtifactBlock is a staged block of a pending artifact.
	14	+	type CIArtifactBlock struct {
	15	+	ArtifactID int64
	16	+	// ID is the uploader's block id (1 to 64 bytes).
	17	+	ID []byte
	18	+	Size int64
	19	+	// File is the name of the block's file in CIArtifactFiles.
	20	+	File string
	21	+	}
	22	+
	23	+	// CIArtifactWrite is a change of the artifacts of one repository, written in
	24	+	// one atomic batch. The batch requires the usage record of the repository at
	25	+	// UsageVersion (0: none stored) and stores Used in it: every write of the
	26	+	// artifacts of a repository goes through that record, so two writes never both
	27	+	// succeed on the same state, and a write fails with domain.ErrConflict when
	28	+	// anything of the repository's artifacts changed since UsageVersion was read.
	29	+	//
	30	+	// Put stores records: one with Version 0 must not exist yet (the batch then
	31	+	// also requires the repository, domain.ErrNotFound once it is deleted), any
	32	+	// other must be at its Version. Delete removes records at their Version with
	33	+	// their staged blocks. PutBlocks stores blocks (replacing one of the same id),
	34	+	// DropBlocks removes all the blocks of the artifacts named.
	35	+	type CIArtifactWrite struct {
	36	+	Repo domain.RepoName
	37	+	Used int64
	38	+	UsageVersion uint64
	39	+	Put []domain.CIArtifact
	40	+	Delete []domain.CIArtifact
	41	+	PutBlocks []CIArtifactBlock
	42	+	DropBlocks []int64
	43	+	}
	44	+
	45	+	// CIArtifactStore keeps the records of the CI artifacts, never their contents.
	46	+	// *kvstore.CIArtifacts implements it; the records go with their repository.
	47	+	type CIArtifactStore interface {
	48	+	// NextArtifactID takes the next artifact id: global, increasing, never
	49	+	// reused.
	50	+	NextArtifactID(ctx context.Context) (int64, error)
	51	+	// ArtifactUsage returns the bytes charged to the artifacts of the
	52	+	// repository and the version of the record (0, 0 when there is none).
	53	+	ArtifactUsage(ctx context.Context, repo domain.RepoName) (used int64, version uint64, err error)
	54	+	// GetArtifact returns an artifact by id, or domain.ErrNotFound.
	55	+	GetArtifact(ctx context.Context, id int64) (domain.CIArtifact, error)
	56	+	// RunArtifacts returns the artifacts of a run in every state, by id.
	57	+	RunArtifacts(ctx context.Context, repo domain.RepoName, runID int64) ([]domain.CIArtifact, error)
	58	+	// AllArtifacts returns every artifact, for the sweep.
	59	+	AllArtifacts(ctx context.Context) ([]domain.CIArtifact, error)
	60	+	// ArtifactBlocks returns the staged blocks of an artifact.
	61	+	ArtifactBlocks(ctx context.Context, id int64) ([]CIArtifactBlock, error)
	62	+	// BlockArtifactIDs returns the artifacts that have staged blocks, for the
	63	+	// sweep of blocks left by a deleted artifact.
	64	+	BlockArtifactIDs(ctx context.Context) ([]int64, error)
	65	+	// ApplyArtifacts writes w in one batch.
	66	+	ApplyArtifacts(ctx context.Context, w CIArtifactWrite) error
	67	+	// DeleteBlocks removes the staged blocks of an artifact that no longer
	68	+	// exists (left by a repository deleted during an upload).
	69	+	DeleteBlocks(ctx context.Context, id int64) error
	70	+	}
	71	+
	72	+	// CIArtifactRef names the files of one artifact.
	73	+	type CIArtifactRef struct {
	74	+	RunID int64
	75	+	ID int64
	76	+	}
	77	+
	78	+	// CIArtifactFiles keeps the contents of the CI artifacts and their staged
	79	+	// blocks, never in the metadata store. Files are named by ids only.
	80	+	type CIArtifactFiles interface {
	81	+	// PutBlock writes src into a new block file of the artifact, synced, and
	82	+	// returns its size. A failure of src or of the write leaves no file.
	83	+	PutBlock(ctx context.Context, ref CIArtifactRef, file string, src io.Reader) (int64, error)
	84	+	// RemoveBlock deletes one block file; a missing one is not an error.
	85	+	RemoveBlock(ctx context.Context, ref CIArtifactRef, file string) error
	86	+	// Assemble concatenates the block files in order into the content of the
	87	+	// artifact, which replaces any earlier one whole, and returns its size and
	88	+	// digest ("sha256:<hex>"). A missing block file is domain.ErrNotFound.
	89	+	Assemble(ctx context.Context, ref CIArtifactRef, files []string) (size int64, digest string, err error)
	90	+	// Open returns the content from offset, or domain.ErrNotFound.
	91	+	Open(ctx context.Context, ref CIArtifactRef, offset int64) (io.ReadCloser, error)
	92	+	// Remove deletes the content and the blocks of the artifact.
	93	+	Remove(ctx context.Context, ref CIArtifactRef) error
	94	+	// RemoveBlocks deletes the blocks of the artifact.
	95	+	RemoveBlocks(ctx context.Context, ref CIArtifactRef) error
	96	+	// RemoveRuns deletes the files of the runs.
	97	+	RemoveRuns(ctx context.Context, runIDs []int64) error
	98	+	// RunIDs lists the runs that have a directory, ArtifactIDs the artifacts
	99	+	// that have files in one, for the orphan sweep.
	100	+	RunIDs(ctx context.Context) ([]int64, error)
	101	+	ArtifactIDs(ctx context.Context, runID int64) ([]int64, error)
	102	+	}
	103	+
	104	+	// ArtifactLimits are the settings of the artifacts of the server.
	105	+	type ArtifactLimits struct {
	106	+	// Retention is the default and longest retention.
	107	+	Retention time.Duration
	108	+	// Quota is the space of the artifacts of one repository, MaxSize the size
	109	+	// of one artifact.
	110	+	Quota int64
	111	+	MaxSize int64
	112	+	}
	113	+
	114	+	// CreatedArtifact is the answer of CreateJobArtifact.
	115	+	type CreatedArtifact struct {
	116	+	ID int64
	117	+	ExpiresAt time.Time
	118	+	}
	119	+
	120	+	// CreateJobArtifactParams begin an upload.
	121	+	type CreateJobArtifactParams struct {
	122	+	JobID string
	123	+	Name string
	124	+	MediaType string
	125	+	// ExpiresAt is the expiry asked for; zero for the default.
	126	+	ExpiresAt time.Time
	127	+	}
	128	+
	129	+	// ArtifactChunk is one message of a download: the artifact in the first one,
	130	+	// then data.
	131	+	type ArtifactChunk struct {
	132	+	Artifact *domain.CIArtifact
	133	+	Data []byte
	134	+	}
	135	+
	136	+	// JobArtifactService is the artifacts side of the runner protocol. The secret
	137	+	// is the job token; jobID must be its job (domain.ErrForbidden otherwise), the
	138	+	// job must not have ended (a *domain.JobEndedError), and the run is always the
	139	+	// job's. *usecase.CI implements it.
	140	+	type JobArtifactService interface {
	141	+	CreateJobArtifact(ctx context.Context, secret string, params CreateJobArtifactParams) (CreatedArtifact, error)
	142	+	// UploadJobArtifactBlock stores data as a block of a pending artifact of
	143	+	// the job. An error of data is returned as it is.
	144	+	UploadJobArtifactBlock(ctx context.Context, secret, jobID string, artifactID int64, blockID []byte, data io.Reader) (int64, error)
	145	+	CommitJobArtifact(ctx context.Context, secret, jobID string, artifactID int64, blockIDs [][]byte) (size int64, digest string, err error)
	146	+	FinalizeJobArtifact(ctx context.Context, secret, jobID, name string, size int64, digest string) (domain.CIArtifact, error)
	147	+	ListJobArtifacts(ctx context.Context, secret, jobID, name string, artifactID int64) ([]domain.CIArtifact, error)
	148	+	// DownloadJobArtifact calls send with the artifact, then with the content
	149	+	// from offset in chunks of at most domain.CIArtifactChunk bytes.
	150	+	DownloadJobArtifact(ctx context.Context, secret, jobID string, artifactID, offset int64, send func(ArtifactChunk) error) error
	151	+	DeleteJobArtifact(ctx context.Context, secret, jobID, name string) (int64, error)
	152	+	}
	153	+
	154	+	// ArtifactList are the artifacts of a run with the space of the repository.
	155	+	type ArtifactList struct {
	156	+	Artifacts []domain.CIArtifact
	157	+	Used int64
	158	+	Quota int64
	159	+	}
	160	+
	161	+	// ArtifactService is the owner side of the artifacts. Listing and downloading
	162	+	// need a signed-in caller with repo:read, even on a public repository;
	163	+	// deleting needs ci:write. *usecase.CI implements it.
	164	+	type ArtifactService interface {
	165	+	ListArtifacts(ctx context.Context, p domain.Principal, repo domain.RepoName, runID int64) (ArtifactList, error)
	166	+	// DownloadArtifact is DownloadJobArtifact for an owner.
	167	+	DownloadArtifact(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID, offset int64, send func(ArtifactChunk) error) error
	168	+	DeleteArtifact(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID int64) error
	169	+	}

A internal/ports/mocks/ArtifactService.go +277 -0

@@ -0,0 +1,277 @@
	1	+	// Code generated by mockery; DO NOT EDIT.
	2	+	// github.com/vektra/mockery
	3	+	// template: testify
	4	+
	5	+	package mocks
	6	+
	7	+	import (
	8	+	"context"
	9	+
	10	+	"bnema.dev/ermite/internal/domain"
	11	+	"bnema.dev/ermite/internal/ports"
	12	+	mock "github.com/stretchr/testify/mock"
	13	+	)
	14	+
	15	+	// NewArtifactService creates a new instance of ArtifactService. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
	16	+	// The first argument is typically a *testing.T value.
	17	+	func NewArtifactService(t interface {
	18	+	mock.TestingT
	19	+	Cleanup(func())
	20	+	}) *ArtifactService {
	21	+	if helper, ok := t.(interface{ Helper() }); ok {
	22	+	helper.Helper()
	23	+	}
	24	+
	25	+	mock := &ArtifactService{}
	26	+	mock.Mock.Test(t)
	27	+
	28	+	t.Cleanup(func() {
	29	+	if helper, ok := t.(interface{ Helper() }); ok {
	30	+	helper.Helper()
	31	+	}
	32	+	mock.AssertExpectations(t)
	33	+	})
	34	+
	35	+	return mock
	36	+	}
	37	+
	38	+	// ArtifactService is an autogenerated mock type for the ArtifactService type
	39	+	type ArtifactService struct {
	40	+	mock.Mock
	41	+	}
	42	+
	43	+	type ArtifactService_Expecter struct {
	44	+	mock *mock.Mock
	45	+	}
	46	+
	47	+	func (_m ArtifactService) EXPECT() ArtifactService_Expecter {
	48	+	return &ArtifactService_Expecter{mock: &_m.Mock}
	49	+	}
	50	+
	51	+	// DeleteArtifact provides a mock function for the type ArtifactService
	52	+	func (_mock *ArtifactService) DeleteArtifact(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID int64) error {
	53	+	ret := _mock.Called(ctx, p, repo, artifactID)
	54	+
	55	+	if len(ret) == 0 {
	56	+	panic("no return value specified for DeleteArtifact")
	57	+	}
	58	+
	59	+	var r0 error
	60	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.Principal, domain.RepoName, int64) error); ok {
	61	+	r0 = returnFunc(ctx, p, repo, artifactID)
	62	+	} else {
	63	+	r0 = ret.Error(0)
	64	+	}
	65	+	return r0
	66	+	}
	67	+
	68	+	// ArtifactService_DeleteArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteArtifact'
	69	+	type ArtifactService_DeleteArtifact_Call struct {
	70	+	*mock.Call
	71	+	}
	72	+
	73	+	// DeleteArtifact is a helper method to define mock.On call
	74	+	// - ctx context.Context
	75	+	// - p domain.Principal
	76	+	// - repo domain.RepoName
	77	+	// - artifactID int64
	78	+	func (_e ArtifactService_Expecter) DeleteArtifact(ctx any, p any, repo any, artifactID any) ArtifactService_DeleteArtifact_Call {
	79	+	return &ArtifactService_DeleteArtifact_Call{Call: _e.mock.On("DeleteArtifact", ctx, p, repo, artifactID)}
	80	+	}
	81	+
	82	+	func (_c ArtifactService_DeleteArtifact_Call) Run(run func(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID int64)) ArtifactService_DeleteArtifact_Call {
	83	+	_c.Call.Run(func(args mock.Arguments) {
	84	+	var arg0 context.Context
	85	+	if args[0] != nil {
	86	+	arg0 = args[0].(context.Context)
	87	+	}
	88	+	var arg1 domain.Principal
	89	+	if args[1] != nil {
	90	+	arg1 = args[1].(domain.Principal)
	91	+	}
	92	+	var arg2 domain.RepoName
	93	+	if args[2] != nil {
	94	+	arg2 = args[2].(domain.RepoName)
	95	+	}
	96	+	var arg3 int64
	97	+	if args[3] != nil {
	98	+	arg3 = args[3].(int64)
	99	+	}
	100	+	run(
	101	+	arg0,
	102	+	arg1,
	103	+	arg2,
	104	+	arg3,
	105	+	)
	106	+	})
	107	+	return _c
	108	+	}
	109	+
	110	+	func (_c ArtifactService_DeleteArtifact_Call) Return(err error) ArtifactService_DeleteArtifact_Call {
	111	+	_c.Call.Return(err)
	112	+	return _c
	113	+	}
	114	+
	115	+	func (_c ArtifactService_DeleteArtifact_Call) RunAndReturn(run func(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID int64) error) ArtifactService_DeleteArtifact_Call {
	116	+	_c.Call.Return(run)
	117	+	return _c
	118	+	}
	119	+
	120	+	// DownloadArtifact provides a mock function for the type ArtifactService
	121	+	func (_mock *ArtifactService) DownloadArtifact(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID int64, offset int64, send func(ports.ArtifactChunk) error) error {
	122	+	ret := _mock.Called(ctx, p, repo, artifactID, offset, send)
	123	+
	124	+	if len(ret) == 0 {
	125	+	panic("no return value specified for DownloadArtifact")
	126	+	}
	127	+
	128	+	var r0 error
	129	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.Principal, domain.RepoName, int64, int64, func(ports.ArtifactChunk) error) error); ok {
	130	+	r0 = returnFunc(ctx, p, repo, artifactID, offset, send)
	131	+	} else {
	132	+	r0 = ret.Error(0)
	133	+	}
	134	+	return r0
	135	+	}
	136	+
	137	+	// ArtifactService_DownloadArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DownloadArtifact'
	138	+	type ArtifactService_DownloadArtifact_Call struct {
	139	+	*mock.Call
	140	+	}
	141	+
	142	+	// DownloadArtifact is a helper method to define mock.On call
	143	+	// - ctx context.Context
	144	+	// - p domain.Principal
	145	+	// - repo domain.RepoName
	146	+	// - artifactID int64
	147	+	// - offset int64
	148	+	// - send func(ports.ArtifactChunk) error
	149	+	func (_e ArtifactService_Expecter) DownloadArtifact(ctx any, p any, repo any, artifactID any, offset any, send any) ArtifactService_DownloadArtifact_Call {
	150	+	return &ArtifactService_DownloadArtifact_Call{Call: _e.mock.On("DownloadArtifact", ctx, p, repo, artifactID, offset, send)}
	151	+	}
	152	+
	153	+	func (_c ArtifactService_DownloadArtifact_Call) Run(run func(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID int64, offset int64, send func(ports.ArtifactChunk) error)) ArtifactService_DownloadArtifact_Call {
	154	+	_c.Call.Run(func(args mock.Arguments) {
	155	+	var arg0 context.Context
	156	+	if args[0] != nil {
	157	+	arg0 = args[0].(context.Context)
	158	+	}
	159	+	var arg1 domain.Principal
	160	+	if args[1] != nil {
	161	+	arg1 = args[1].(domain.Principal)
	162	+	}
	163	+	var arg2 domain.RepoName
	164	+	if args[2] != nil {
	165	+	arg2 = args[2].(domain.RepoName)
	166	+	}
	167	+	var arg3 int64
	168	+	if args[3] != nil {
	169	+	arg3 = args[3].(int64)
	170	+	}
	171	+	var arg4 int64
	172	+	if args[4] != nil {
	173	+	arg4 = args[4].(int64)
	174	+	}
	175	+	var arg5 func(ports.ArtifactChunk) error
	176	+	if args[5] != nil {
	177	+	arg5 = args[5].(func(ports.ArtifactChunk) error)
	178	+	}
	179	+	run(
	180	+	arg0,
	181	+	arg1,
	182	+	arg2,
	183	+	arg3,
	184	+	arg4,
	185	+	arg5,
	186	+	)
	187	+	})
	188	+	return _c
	189	+	}
	190	+
	191	+	func (_c ArtifactService_DownloadArtifact_Call) Return(err error) ArtifactService_DownloadArtifact_Call {
	192	+	_c.Call.Return(err)
	193	+	return _c
	194	+	}
	195	+
	196	+	func (_c ArtifactService_DownloadArtifact_Call) RunAndReturn(run func(ctx context.Context, p domain.Principal, repo domain.RepoName, artifactID int64, offset int64, send func(ports.ArtifactChunk) error) error) ArtifactService_DownloadArtifact_Call {
	197	+	_c.Call.Return(run)
	198	+	return _c
	199	+	}
	200	+
	201	+	// ListArtifacts provides a mock function for the type ArtifactService
	202	+	func (_mock *ArtifactService) ListArtifacts(ctx context.Context, p domain.Principal, repo domain.RepoName, runID int64) (ports.ArtifactList, error) {
	203	+	ret := _mock.Called(ctx, p, repo, runID)
	204	+
	205	+	if len(ret) == 0 {
	206	+	panic("no return value specified for ListArtifacts")
	207	+	}
	208	+
	209	+	var r0 ports.ArtifactList
	210	+	var r1 error
	211	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.Principal, domain.RepoName, int64) (ports.ArtifactList, error)); ok {
	212	+	return returnFunc(ctx, p, repo, runID)
	213	+	}
	214	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.Principal, domain.RepoName, int64) ports.ArtifactList); ok {
	215	+	r0 = returnFunc(ctx, p, repo, runID)
	216	+	} else {
	217	+	r0 = ret.Get(0).(ports.ArtifactList)
	218	+	}
	219	+	if returnFunc, ok := ret.Get(1).(func(context.Context, domain.Principal, domain.RepoName, int64) error); ok {
	220	+	r1 = returnFunc(ctx, p, repo, runID)
	221	+	} else {
	222	+	r1 = ret.Error(1)
	223	+	}
	224	+	return r0, r1
	225	+	}
	226	+
	227	+	// ArtifactService_ListArtifacts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ListArtifacts'
	228	+	type ArtifactService_ListArtifacts_Call struct {
	229	+	*mock.Call
	230	+	}
	231	+
	232	+	// ListArtifacts is a helper method to define mock.On call
	233	+	// - ctx context.Context
	234	+	// - p domain.Principal
	235	+	// - repo domain.RepoName
	236	+	// - runID int64
	237	+	func (_e ArtifactService_Expecter) ListArtifacts(ctx any, p any, repo any, runID any) ArtifactService_ListArtifacts_Call {
	238	+	return &ArtifactService_ListArtifacts_Call{Call: _e.mock.On("ListArtifacts", ctx, p, repo, runID)}
	239	+	}
	240	+
	241	+	func (_c ArtifactService_ListArtifacts_Call) Run(run func(ctx context.Context, p domain.Principal, repo domain.RepoName, runID int64)) ArtifactService_ListArtifacts_Call {
	242	+	_c.Call.Run(func(args mock.Arguments) {
	243	+	var arg0 context.Context
	244	+	if args[0] != nil {
	245	+	arg0 = args[0].(context.Context)
	246	+	}
	247	+	var arg1 domain.Principal
	248	+	if args[1] != nil {
	249	+	arg1 = args[1].(domain.Principal)
	250	+	}
	251	+	var arg2 domain.RepoName
	252	+	if args[2] != nil {
	253	+	arg2 = args[2].(domain.RepoName)
	254	+	}
	255	+	var arg3 int64
	256	+	if args[3] != nil {
	257	+	arg3 = args[3].(int64)
	258	+	}
	259	+	run(
	260	+	arg0,
	261	+	arg1,
	262	+	arg2,
	263	+	arg3,
	264	+	)
	265	+	})
	266	+	return _c
	267	+	}
	268	+
	269	+	func (_c ArtifactService_ListArtifacts_Call) Return(artifactList ports.ArtifactList, err error) ArtifactService_ListArtifacts_Call {
	270	+	_c.Call.Return(artifactList, err)
	271	+	return _c
	272	+	}
	273	+
	274	+	func (_c ArtifactService_ListArtifacts_Call) RunAndReturn(run func(ctx context.Context, p domain.Principal, repo domain.RepoName, runID int64) (ports.ArtifactList, error)) ArtifactService_ListArtifacts_Call {
	275	+	_c.Call.Return(run)
	276	+	return _c
	277	+	}

A internal/ports/mocks/CIArtifactFiles.go +643 -0

@@ -0,0 +1,643 @@
	1	+	// Code generated by mockery; DO NOT EDIT.
	2	+	// github.com/vektra/mockery
	3	+	// template: testify
	4	+
	5	+	package mocks
	6	+
	7	+	import (
	8	+	"context"
	9	+	"io"
	10	+
	11	+	"bnema.dev/ermite/internal/ports"
	12	+	mock "github.com/stretchr/testify/mock"
	13	+	)
	14	+
	15	+	// NewCIArtifactFiles creates a new instance of CIArtifactFiles. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
	16	+	// The first argument is typically a *testing.T value.
	17	+	func NewCIArtifactFiles(t interface {
	18	+	mock.TestingT
	19	+	Cleanup(func())
	20	+	}) *CIArtifactFiles {
	21	+	if helper, ok := t.(interface{ Helper() }); ok {
	22	+	helper.Helper()
	23	+	}
	24	+
	25	+	mock := &CIArtifactFiles{}
	26	+	mock.Mock.Test(t)
	27	+
	28	+	t.Cleanup(func() {
	29	+	if helper, ok := t.(interface{ Helper() }); ok {
	30	+	helper.Helper()
	31	+	}
	32	+	mock.AssertExpectations(t)
	33	+	})
	34	+
	35	+	return mock
	36	+	}
	37	+
	38	+	// CIArtifactFiles is an autogenerated mock type for the CIArtifactFiles type
	39	+	type CIArtifactFiles struct {
	40	+	mock.Mock
	41	+	}
	42	+
	43	+	type CIArtifactFiles_Expecter struct {
	44	+	mock *mock.Mock
	45	+	}
	46	+
	47	+	func (_m CIArtifactFiles) EXPECT() CIArtifactFiles_Expecter {
	48	+	return &CIArtifactFiles_Expecter{mock: &_m.Mock}
	49	+	}
	50	+
	51	+	// ArtifactIDs provides a mock function for the type CIArtifactFiles
	52	+	func (_mock *CIArtifactFiles) ArtifactIDs(ctx context.Context, runID int64) ([]int64, error) {
	53	+	ret := _mock.Called(ctx, runID)
	54	+
	55	+	if len(ret) == 0 {
	56	+	panic("no return value specified for ArtifactIDs")
	57	+	}
	58	+
	59	+	var r0 []int64
	60	+	var r1 error
	61	+	if returnFunc, ok := ret.Get(0).(func(context.Context, int64) ([]int64, error)); ok {
	62	+	return returnFunc(ctx, runID)
	63	+	}
	64	+	if returnFunc, ok := ret.Get(0).(func(context.Context, int64) []int64); ok {
	65	+	r0 = returnFunc(ctx, runID)
	66	+	} else {
	67	+	if ret.Get(0) != nil {
	68	+	r0 = ret.Get(0).([]int64)
	69	+	}
	70	+	}
	71	+	if returnFunc, ok := ret.Get(1).(func(context.Context, int64) error); ok {
	72	+	r1 = returnFunc(ctx, runID)
	73	+	} else {
	74	+	r1 = ret.Error(1)
	75	+	}
	76	+	return r0, r1
	77	+	}
	78	+
	79	+	// CIArtifactFiles_ArtifactIDs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ArtifactIDs'
	80	+	type CIArtifactFiles_ArtifactIDs_Call struct {
	81	+	*mock.Call
	82	+	}
	83	+
	84	+	// ArtifactIDs is a helper method to define mock.On call
	85	+	// - ctx context.Context
	86	+	// - runID int64
	87	+	func (_e CIArtifactFiles_Expecter) ArtifactIDs(ctx any, runID any) CIArtifactFiles_ArtifactIDs_Call {
	88	+	return &CIArtifactFiles_ArtifactIDs_Call{Call: _e.mock.On("ArtifactIDs", ctx, runID)}
	89	+	}
	90	+
	91	+	func (_c CIArtifactFiles_ArtifactIDs_Call) Run(run func(ctx context.Context, runID int64)) CIArtifactFiles_ArtifactIDs_Call {
	92	+	_c.Call.Run(func(args mock.Arguments) {
	93	+	var arg0 context.Context
	94	+	if args[0] != nil {
	95	+	arg0 = args[0].(context.Context)
	96	+	}
	97	+	var arg1 int64
	98	+	if args[1] != nil {
	99	+	arg1 = args[1].(int64)
	100	+	}
	101	+	run(
	102	+	arg0,
	103	+	arg1,
	104	+	)
	105	+	})
	106	+	return _c
	107	+	}
	108	+
	109	+	func (_c CIArtifactFiles_ArtifactIDs_Call) Return(int64s []int64, err error) CIArtifactFiles_ArtifactIDs_Call {
	110	+	_c.Call.Return(int64s, err)
	111	+	return _c
	112	+	}
	113	+
	114	+	func (_c CIArtifactFiles_ArtifactIDs_Call) RunAndReturn(run func(ctx context.Context, runID int64) ([]int64, error)) CIArtifactFiles_ArtifactIDs_Call {
	115	+	_c.Call.Return(run)
	116	+	return _c
	117	+	}
	118	+
	119	+	// Assemble provides a mock function for the type CIArtifactFiles
	120	+	func (_mock *CIArtifactFiles) Assemble(ctx context.Context, ref ports.CIArtifactRef, files []string) (int64, string, error) {
	121	+	ret := _mock.Called(ctx, ref, files)
	122	+
	123	+	if len(ret) == 0 {
	124	+	panic("no return value specified for Assemble")
	125	+	}
	126	+
	127	+	var r0 int64
	128	+	var r1 string
	129	+	var r2 error
	130	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef, []string) (int64, string, error)); ok {
	131	+	return returnFunc(ctx, ref, files)
	132	+	}
	133	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef, []string) int64); ok {
	134	+	r0 = returnFunc(ctx, ref, files)
	135	+	} else {
	136	+	r0 = ret.Get(0).(int64)
	137	+	}
	138	+	if returnFunc, ok := ret.Get(1).(func(context.Context, ports.CIArtifactRef, []string) string); ok {
	139	+	r1 = returnFunc(ctx, ref, files)
	140	+	} else {
	141	+	r1 = ret.Get(1).(string)
	142	+	}
	143	+	if returnFunc, ok := ret.Get(2).(func(context.Context, ports.CIArtifactRef, []string) error); ok {
	144	+	r2 = returnFunc(ctx, ref, files)
	145	+	} else {
	146	+	r2 = ret.Error(2)
	147	+	}
	148	+	return r0, r1, r2
	149	+	}
	150	+
	151	+	// CIArtifactFiles_Assemble_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Assemble'
	152	+	type CIArtifactFiles_Assemble_Call struct {
	153	+	*mock.Call
	154	+	}
	155	+
	156	+	// Assemble is a helper method to define mock.On call
	157	+	// - ctx context.Context
	158	+	// - ref ports.CIArtifactRef
	159	+	// - files []string
	160	+	func (_e CIArtifactFiles_Expecter) Assemble(ctx any, ref any, files any) CIArtifactFiles_Assemble_Call {
	161	+	return &CIArtifactFiles_Assemble_Call{Call: _e.mock.On("Assemble", ctx, ref, files)}
	162	+	}
	163	+
	164	+	func (_c CIArtifactFiles_Assemble_Call) Run(run func(ctx context.Context, ref ports.CIArtifactRef, files []string)) CIArtifactFiles_Assemble_Call {
	165	+	_c.Call.Run(func(args mock.Arguments) {
	166	+	var arg0 context.Context
	167	+	if args[0] != nil {
	168	+	arg0 = args[0].(context.Context)
	169	+	}
	170	+	var arg1 ports.CIArtifactRef
	171	+	if args[1] != nil {
	172	+	arg1 = args[1].(ports.CIArtifactRef)
	173	+	}
	174	+	var arg2 []string
	175	+	if args[2] != nil {
	176	+	arg2 = args[2].([]string)
	177	+	}
	178	+	run(
	179	+	arg0,
	180	+	arg1,
	181	+	arg2,
	182	+	)
	183	+	})
	184	+	return _c
	185	+	}
	186	+
	187	+	func (_c CIArtifactFiles_Assemble_Call) Return(size int64, digest string, err error) CIArtifactFiles_Assemble_Call {
	188	+	_c.Call.Return(size, digest, err)
	189	+	return _c
	190	+	}
	191	+
	192	+	func (_c CIArtifactFiles_Assemble_Call) RunAndReturn(run func(ctx context.Context, ref ports.CIArtifactRef, files []string) (int64, string, error)) CIArtifactFiles_Assemble_Call {
	193	+	_c.Call.Return(run)
	194	+	return _c
	195	+	}
	196	+
	197	+	// Open provides a mock function for the type CIArtifactFiles
	198	+	func (_mock *CIArtifactFiles) Open(ctx context.Context, ref ports.CIArtifactRef, offset int64) (io.ReadCloser, error) {
	199	+	ret := _mock.Called(ctx, ref, offset)
	200	+
	201	+	if len(ret) == 0 {
	202	+	panic("no return value specified for Open")
	203	+	}
	204	+
	205	+	var r0 io.ReadCloser
	206	+	var r1 error
	207	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef, int64) (io.ReadCloser, error)); ok {
	208	+	return returnFunc(ctx, ref, offset)
	209	+	}
	210	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef, int64) io.ReadCloser); ok {
	211	+	r0 = returnFunc(ctx, ref, offset)
	212	+	} else {
	213	+	if ret.Get(0) != nil {
	214	+	r0 = ret.Get(0).(io.ReadCloser)
	215	+	}
	216	+	}
	217	+	if returnFunc, ok := ret.Get(1).(func(context.Context, ports.CIArtifactRef, int64) error); ok {
	218	+	r1 = returnFunc(ctx, ref, offset)
	219	+	} else {
	220	+	r1 = ret.Error(1)
	221	+	}
	222	+	return r0, r1
	223	+	}
	224	+
	225	+	// CIArtifactFiles_Open_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Open'
	226	+	type CIArtifactFiles_Open_Call struct {
	227	+	*mock.Call
	228	+	}
	229	+
	230	+	// Open is a helper method to define mock.On call
	231	+	// - ctx context.Context
	232	+	// - ref ports.CIArtifactRef
	233	+	// - offset int64
	234	+	func (_e CIArtifactFiles_Expecter) Open(ctx any, ref any, offset any) CIArtifactFiles_Open_Call {
	235	+	return &CIArtifactFiles_Open_Call{Call: _e.mock.On("Open", ctx, ref, offset)}
	236	+	}
	237	+
	238	+	func (_c CIArtifactFiles_Open_Call) Run(run func(ctx context.Context, ref ports.CIArtifactRef, offset int64)) CIArtifactFiles_Open_Call {
	239	+	_c.Call.Run(func(args mock.Arguments) {
	240	+	var arg0 context.Context
	241	+	if args[0] != nil {
	242	+	arg0 = args[0].(context.Context)
	243	+	}
	244	+	var arg1 ports.CIArtifactRef
	245	+	if args[1] != nil {
	246	+	arg1 = args[1].(ports.CIArtifactRef)
	247	+	}
	248	+	var arg2 int64
	249	+	if args[2] != nil {
	250	+	arg2 = args[2].(int64)
	251	+	}
	252	+	run(
	253	+	arg0,
	254	+	arg1,
	255	+	arg2,
	256	+	)
	257	+	})
	258	+	return _c
	259	+	}
	260	+
	261	+	func (_c CIArtifactFiles_Open_Call) Return(readCloser io.ReadCloser, err error) CIArtifactFiles_Open_Call {
	262	+	_c.Call.Return(readCloser, err)
	263	+	return _c
	264	+	}
	265	+
	266	+	func (_c CIArtifactFiles_Open_Call) RunAndReturn(run func(ctx context.Context, ref ports.CIArtifactRef, offset int64) (io.ReadCloser, error)) CIArtifactFiles_Open_Call {
	267	+	_c.Call.Return(run)
	268	+	return _c
	269	+	}
	270	+
	271	+	// PutBlock provides a mock function for the type CIArtifactFiles
	272	+	func (_mock *CIArtifactFiles) PutBlock(ctx context.Context, ref ports.CIArtifactRef, file string, src io.Reader) (int64, error) {
	273	+	ret := _mock.Called(ctx, ref, file, src)
	274	+
	275	+	if len(ret) == 0 {
	276	+	panic("no return value specified for PutBlock")
	277	+	}
	278	+
	279	+	var r0 int64
	280	+	var r1 error
	281	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef, string, io.Reader) (int64, error)); ok {
	282	+	return returnFunc(ctx, ref, file, src)
	283	+	}
	284	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef, string, io.Reader) int64); ok {
	285	+	r0 = returnFunc(ctx, ref, file, src)
	286	+	} else {
	287	+	r0 = ret.Get(0).(int64)
	288	+	}
	289	+	if returnFunc, ok := ret.Get(1).(func(context.Context, ports.CIArtifactRef, string, io.Reader) error); ok {
	290	+	r1 = returnFunc(ctx, ref, file, src)
	291	+	} else {
	292	+	r1 = ret.Error(1)
	293	+	}
	294	+	return r0, r1
	295	+	}
	296	+
	297	+	// CIArtifactFiles_PutBlock_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'PutBlock'
	298	+	type CIArtifactFiles_PutBlock_Call struct {
	299	+	*mock.Call
	300	+	}
	301	+
	302	+	// PutBlock is a helper method to define mock.On call
	303	+	// - ctx context.Context
	304	+	// - ref ports.CIArtifactRef
	305	+	// - file string
	306	+	// - src io.Reader
	307	+	func (_e CIArtifactFiles_Expecter) PutBlock(ctx any, ref any, file any, src any) CIArtifactFiles_PutBlock_Call {
	308	+	return &CIArtifactFiles_PutBlock_Call{Call: _e.mock.On("PutBlock", ctx, ref, file, src)}
	309	+	}
	310	+
	311	+	func (_c CIArtifactFiles_PutBlock_Call) Run(run func(ctx context.Context, ref ports.CIArtifactRef, file string, src io.Reader)) CIArtifactFiles_PutBlock_Call {
	312	+	_c.Call.Run(func(args mock.Arguments) {
	313	+	var arg0 context.Context
	314	+	if args[0] != nil {
	315	+	arg0 = args[0].(context.Context)
	316	+	}
	317	+	var arg1 ports.CIArtifactRef
	318	+	if args[1] != nil {
	319	+	arg1 = args[1].(ports.CIArtifactRef)
	320	+	}
	321	+	var arg2 string
	322	+	if args[2] != nil {
	323	+	arg2 = args[2].(string)
	324	+	}
	325	+	var arg3 io.Reader
	326	+	if args[3] != nil {
	327	+	arg3 = args[3].(io.Reader)
	328	+	}
	329	+	run(
	330	+	arg0,
	331	+	arg1,
	332	+	arg2,
	333	+	arg3,
	334	+	)
	335	+	})
	336	+	return _c
	337	+	}
	338	+
	339	+	func (_c CIArtifactFiles_PutBlock_Call) Return(n int64, err error) CIArtifactFiles_PutBlock_Call {
	340	+	_c.Call.Return(n, err)
	341	+	return _c
	342	+	}
	343	+
	344	+	func (_c CIArtifactFiles_PutBlock_Call) RunAndReturn(run func(ctx context.Context, ref ports.CIArtifactRef, file string, src io.Reader) (int64, error)) CIArtifactFiles_PutBlock_Call {
	345	+	_c.Call.Return(run)
	346	+	return _c
	347	+	}
	348	+
	349	+	// Remove provides a mock function for the type CIArtifactFiles
	350	+	func (_mock *CIArtifactFiles) Remove(ctx context.Context, ref ports.CIArtifactRef) error {
	351	+	ret := _mock.Called(ctx, ref)
	352	+
	353	+	if len(ret) == 0 {
	354	+	panic("no return value specified for Remove")
	355	+	}
	356	+
	357	+	var r0 error
	358	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef) error); ok {
	359	+	r0 = returnFunc(ctx, ref)
	360	+	} else {
	361	+	r0 = ret.Error(0)
	362	+	}
	363	+	return r0
	364	+	}
	365	+
	366	+	// CIArtifactFiles_Remove_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Remove'
	367	+	type CIArtifactFiles_Remove_Call struct {
	368	+	*mock.Call
	369	+	}
	370	+
	371	+	// Remove is a helper method to define mock.On call
	372	+	// - ctx context.Context
	373	+	// - ref ports.CIArtifactRef
	374	+	func (_e CIArtifactFiles_Expecter) Remove(ctx any, ref any) CIArtifactFiles_Remove_Call {
	375	+	return &CIArtifactFiles_Remove_Call{Call: _e.mock.On("Remove", ctx, ref)}
	376	+	}
	377	+
	378	+	func (_c CIArtifactFiles_Remove_Call) Run(run func(ctx context.Context, ref ports.CIArtifactRef)) CIArtifactFiles_Remove_Call {
	379	+	_c.Call.Run(func(args mock.Arguments) {
	380	+	var arg0 context.Context
	381	+	if args[0] != nil {
	382	+	arg0 = args[0].(context.Context)
	383	+	}
	384	+	var arg1 ports.CIArtifactRef
	385	+	if args[1] != nil {
	386	+	arg1 = args[1].(ports.CIArtifactRef)
	387	+	}
	388	+	run(
	389	+	arg0,
	390	+	arg1,
	391	+	)
	392	+	})
	393	+	return _c
	394	+	}
	395	+
	396	+	func (_c CIArtifactFiles_Remove_Call) Return(err error) CIArtifactFiles_Remove_Call {
	397	+	_c.Call.Return(err)
	398	+	return _c
	399	+	}
	400	+
	401	+	func (_c CIArtifactFiles_Remove_Call) RunAndReturn(run func(ctx context.Context, ref ports.CIArtifactRef) error) CIArtifactFiles_Remove_Call {
	402	+	_c.Call.Return(run)
	403	+	return _c
	404	+	}
	405	+
	406	+	// RemoveBlock provides a mock function for the type CIArtifactFiles
	407	+	func (_mock *CIArtifactFiles) RemoveBlock(ctx context.Context, ref ports.CIArtifactRef, file string) error {
	408	+	ret := _mock.Called(ctx, ref, file)
	409	+
	410	+	if len(ret) == 0 {
	411	+	panic("no return value specified for RemoveBlock")
	412	+	}
	413	+
	414	+	var r0 error
	415	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef, string) error); ok {
	416	+	r0 = returnFunc(ctx, ref, file)
	417	+	} else {
	418	+	r0 = ret.Error(0)
	419	+	}
	420	+	return r0
	421	+	}
	422	+
	423	+	// CIArtifactFiles_RemoveBlock_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'RemoveBlock'
	424	+	type CIArtifactFiles_RemoveBlock_Call struct {
	425	+	*mock.Call
	426	+	}
	427	+
	428	+	// RemoveBlock is a helper method to define mock.On call
	429	+	// - ctx context.Context
	430	+	// - ref ports.CIArtifactRef
	431	+	// - file string
	432	+	func (_e CIArtifactFiles_Expecter) RemoveBlock(ctx any, ref any, file any) CIArtifactFiles_RemoveBlock_Call {
	433	+	return &CIArtifactFiles_RemoveBlock_Call{Call: _e.mock.On("RemoveBlock", ctx, ref, file)}
	434	+	}
	435	+
	436	+	func (_c CIArtifactFiles_RemoveBlock_Call) Run(run func(ctx context.Context, ref ports.CIArtifactRef, file string)) CIArtifactFiles_RemoveBlock_Call {
	437	+	_c.Call.Run(func(args mock.Arguments) {
	438	+	var arg0 context.Context
	439	+	if args[0] != nil {
	440	+	arg0 = args[0].(context.Context)
	441	+	}
	442	+	var arg1 ports.CIArtifactRef
	443	+	if args[1] != nil {
	444	+	arg1 = args[1].(ports.CIArtifactRef)
	445	+	}
	446	+	var arg2 string
	447	+	if args[2] != nil {
	448	+	arg2 = args[2].(string)
	449	+	}
	450	+	run(
	451	+	arg0,
	452	+	arg1,
	453	+	arg2,
	454	+	)
	455	+	})
	456	+	return _c
	457	+	}
	458	+
	459	+	func (_c CIArtifactFiles_RemoveBlock_Call) Return(err error) CIArtifactFiles_RemoveBlock_Call {
	460	+	_c.Call.Return(err)
	461	+	return _c
	462	+	}
	463	+
	464	+	func (_c CIArtifactFiles_RemoveBlock_Call) RunAndReturn(run func(ctx context.Context, ref ports.CIArtifactRef, file string) error) CIArtifactFiles_RemoveBlock_Call {
	465	+	_c.Call.Return(run)
	466	+	return _c
	467	+	}
	468	+
	469	+	// RemoveBlocks provides a mock function for the type CIArtifactFiles
	470	+	func (_mock *CIArtifactFiles) RemoveBlocks(ctx context.Context, ref ports.CIArtifactRef) error {
	471	+	ret := _mock.Called(ctx, ref)
	472	+
	473	+	if len(ret) == 0 {
	474	+	panic("no return value specified for RemoveBlocks")
	475	+	}
	476	+
	477	+	var r0 error
	478	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactRef) error); ok {
	479	+	r0 = returnFunc(ctx, ref)
	480	+	} else {
	481	+	r0 = ret.Error(0)
	482	+	}
	483	+	return r0
	484	+	}
	485	+
	486	+	// CIArtifactFiles_RemoveBlocks_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'RemoveBlocks'
	487	+	type CIArtifactFiles_RemoveBlocks_Call struct {
	488	+	*mock.Call
	489	+	}
	490	+
	491	+	// RemoveBlocks is a helper method to define mock.On call
	492	+	// - ctx context.Context
	493	+	// - ref ports.CIArtifactRef
	494	+	func (_e CIArtifactFiles_Expecter) RemoveBlocks(ctx any, ref any) CIArtifactFiles_RemoveBlocks_Call {
	495	+	return &CIArtifactFiles_RemoveBlocks_Call{Call: _e.mock.On("RemoveBlocks", ctx, ref)}
	496	+	}
	497	+
	498	+	func (_c CIArtifactFiles_RemoveBlocks_Call) Run(run func(ctx context.Context, ref ports.CIArtifactRef)) CIArtifactFiles_RemoveBlocks_Call {
	499	+	_c.Call.Run(func(args mock.Arguments) {
	500	+	var arg0 context.Context
	501	+	if args[0] != nil {
	502	+	arg0 = args[0].(context.Context)
	503	+	}
	504	+	var arg1 ports.CIArtifactRef
	505	+	if args[1] != nil {
	506	+	arg1 = args[1].(ports.CIArtifactRef)
	507	+	}
	508	+	run(
	509	+	arg0,
	510	+	arg1,
	511	+	)
	512	+	})
	513	+	return _c
	514	+	}
	515	+
	516	+	func (_c CIArtifactFiles_RemoveBlocks_Call) Return(err error) CIArtifactFiles_RemoveBlocks_Call {
	517	+	_c.Call.Return(err)
	518	+	return _c
	519	+	}
	520	+
	521	+	func (_c CIArtifactFiles_RemoveBlocks_Call) RunAndReturn(run func(ctx context.Context, ref ports.CIArtifactRef) error) CIArtifactFiles_RemoveBlocks_Call {
	522	+	_c.Call.Return(run)
	523	+	return _c
	524	+	}
	525	+
	526	+	// RemoveRuns provides a mock function for the type CIArtifactFiles
	527	+	func (_mock *CIArtifactFiles) RemoveRuns(ctx context.Context, runIDs []int64) error {
	528	+	ret := _mock.Called(ctx, runIDs)
	529	+
	530	+	if len(ret) == 0 {
	531	+	panic("no return value specified for RemoveRuns")
	532	+	}
	533	+
	534	+	var r0 error
	535	+	if returnFunc, ok := ret.Get(0).(func(context.Context, []int64) error); ok {
	536	+	r0 = returnFunc(ctx, runIDs)
	537	+	} else {
	538	+	r0 = ret.Error(0)
	539	+	}
	540	+	return r0
	541	+	}
	542	+
	543	+	// CIArtifactFiles_RemoveRuns_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'RemoveRuns'
	544	+	type CIArtifactFiles_RemoveRuns_Call struct {
	545	+	*mock.Call
	546	+	}
	547	+
	548	+	// RemoveRuns is a helper method to define mock.On call
	549	+	// - ctx context.Context
	550	+	// - runIDs []int64
	551	+	func (_e CIArtifactFiles_Expecter) RemoveRuns(ctx any, runIDs any) CIArtifactFiles_RemoveRuns_Call {
	552	+	return &CIArtifactFiles_RemoveRuns_Call{Call: _e.mock.On("RemoveRuns", ctx, runIDs)}
	553	+	}
	554	+
	555	+	func (_c CIArtifactFiles_RemoveRuns_Call) Run(run func(ctx context.Context, runIDs []int64)) CIArtifactFiles_RemoveRuns_Call {
	556	+	_c.Call.Run(func(args mock.Arguments) {
	557	+	var arg0 context.Context
	558	+	if args[0] != nil {
	559	+	arg0 = args[0].(context.Context)
	560	+	}
	561	+	var arg1 []int64
	562	+	if args[1] != nil {
	563	+	arg1 = args[1].([]int64)
	564	+	}
	565	+	run(
	566	+	arg0,
	567	+	arg1,
	568	+	)
	569	+	})
	570	+	return _c
	571	+	}
	572	+
	573	+	func (_c CIArtifactFiles_RemoveRuns_Call) Return(err error) CIArtifactFiles_RemoveRuns_Call {
	574	+	_c.Call.Return(err)
	575	+	return _c
	576	+	}
	577	+
	578	+	func (_c CIArtifactFiles_RemoveRuns_Call) RunAndReturn(run func(ctx context.Context, runIDs []int64) error) CIArtifactFiles_RemoveRuns_Call {
	579	+	_c.Call.Return(run)
	580	+	return _c
	581	+	}
	582	+
	583	+	// RunIDs provides a mock function for the type CIArtifactFiles
	584	+	func (_mock *CIArtifactFiles) RunIDs(ctx context.Context) ([]int64, error) {
	585	+	ret := _mock.Called(ctx)
	586	+
	587	+	if len(ret) == 0 {
	588	+	panic("no return value specified for RunIDs")
	589	+	}
	590	+
	591	+	var r0 []int64
	592	+	var r1 error
	593	+	if returnFunc, ok := ret.Get(0).(func(context.Context) ([]int64, error)); ok {
	594	+	return returnFunc(ctx)
	595	+	}
	596	+	if returnFunc, ok := ret.Get(0).(func(context.Context) []int64); ok {
	597	+	r0 = returnFunc(ctx)
	598	+	} else {
	599	+	if ret.Get(0) != nil {
	600	+	r0 = ret.Get(0).([]int64)
	601	+	}
	602	+	}
	603	+	if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
	604	+	r1 = returnFunc(ctx)
	605	+	} else {
	606	+	r1 = ret.Error(1)
	607	+	}
	608	+	return r0, r1
	609	+	}
	610	+
	611	+	// CIArtifactFiles_RunIDs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'RunIDs'
	612	+	type CIArtifactFiles_RunIDs_Call struct {
	613	+	*mock.Call
	614	+	}
	615	+
	616	+	// RunIDs is a helper method to define mock.On call
	617	+	// - ctx context.Context
	618	+	func (_e CIArtifactFiles_Expecter) RunIDs(ctx any) CIArtifactFiles_RunIDs_Call {
	619	+	return &CIArtifactFiles_RunIDs_Call{Call: _e.mock.On("RunIDs", ctx)}
	620	+	}
	621	+
	622	+	func (_c CIArtifactFiles_RunIDs_Call) Run(run func(ctx context.Context)) CIArtifactFiles_RunIDs_Call {
	623	+	_c.Call.Run(func(args mock.Arguments) {
	624	+	var arg0 context.Context
	625	+	if args[0] != nil {
	626	+	arg0 = args[0].(context.Context)
	627	+	}
	628	+	run(
	629	+	arg0,
	630	+	)
	631	+	})
	632	+	return _c
	633	+	}
	634	+
	635	+	func (_c CIArtifactFiles_RunIDs_Call) Return(int64s []int64, err error) CIArtifactFiles_RunIDs_Call {
	636	+	_c.Call.Return(int64s, err)
	637	+	return _c
	638	+	}
	639	+
	640	+	func (_c CIArtifactFiles_RunIDs_Call) RunAndReturn(run func(ctx context.Context) ([]int64, error)) CIArtifactFiles_RunIDs_Call {
	641	+	_c.Call.Return(run)
	642	+	return _c
	643	+	}

A internal/ports/mocks/CIArtifactStore.go +627 -0

@@ -0,0 +1,627 @@
	1	+	// Code generated by mockery; DO NOT EDIT.
	2	+	// github.com/vektra/mockery
	3	+	// template: testify
	4	+
	5	+	package mocks
	6	+
	7	+	import (
	8	+	"context"
	9	+
	10	+	"bnema.dev/ermite/internal/domain"
	11	+	"bnema.dev/ermite/internal/ports"
	12	+	mock "github.com/stretchr/testify/mock"
	13	+	)
	14	+
	15	+	// NewCIArtifactStore creates a new instance of CIArtifactStore. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
	16	+	// The first argument is typically a *testing.T value.
	17	+	func NewCIArtifactStore(t interface {
	18	+	mock.TestingT
	19	+	Cleanup(func())
	20	+	}) *CIArtifactStore {
	21	+	if helper, ok := t.(interface{ Helper() }); ok {
	22	+	helper.Helper()
	23	+	}
	24	+
	25	+	mock := &CIArtifactStore{}
	26	+	mock.Mock.Test(t)
	27	+
	28	+	t.Cleanup(func() {
	29	+	if helper, ok := t.(interface{ Helper() }); ok {
	30	+	helper.Helper()
	31	+	}
	32	+	mock.AssertExpectations(t)
	33	+	})
	34	+
	35	+	return mock
	36	+	}
	37	+
	38	+	// CIArtifactStore is an autogenerated mock type for the CIArtifactStore type
	39	+	type CIArtifactStore struct {
	40	+	mock.Mock
	41	+	}
	42	+
	43	+	type CIArtifactStore_Expecter struct {
	44	+	mock *mock.Mock
	45	+	}
	46	+
	47	+	func (_m CIArtifactStore) EXPECT() CIArtifactStore_Expecter {
	48	+	return &CIArtifactStore_Expecter{mock: &_m.Mock}
	49	+	}
	50	+
	51	+	// AllArtifacts provides a mock function for the type CIArtifactStore
	52	+	func (_mock *CIArtifactStore) AllArtifacts(ctx context.Context) ([]domain.CIArtifact, error) {
	53	+	ret := _mock.Called(ctx)
	54	+
	55	+	if len(ret) == 0 {
	56	+	panic("no return value specified for AllArtifacts")
	57	+	}
	58	+
	59	+	var r0 []domain.CIArtifact
	60	+	var r1 error
	61	+	if returnFunc, ok := ret.Get(0).(func(context.Context) ([]domain.CIArtifact, error)); ok {
	62	+	return returnFunc(ctx)
	63	+	}
	64	+	if returnFunc, ok := ret.Get(0).(func(context.Context) []domain.CIArtifact); ok {
	65	+	r0 = returnFunc(ctx)
	66	+	} else {
	67	+	if ret.Get(0) != nil {
	68	+	r0 = ret.Get(0).([]domain.CIArtifact)
	69	+	}
	70	+	}
	71	+	if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
	72	+	r1 = returnFunc(ctx)
	73	+	} else {
	74	+	r1 = ret.Error(1)
	75	+	}
	76	+	return r0, r1
	77	+	}
	78	+
	79	+	// CIArtifactStore_AllArtifacts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'AllArtifacts'
	80	+	type CIArtifactStore_AllArtifacts_Call struct {
	81	+	*mock.Call
	82	+	}
	83	+
	84	+	// AllArtifacts is a helper method to define mock.On call
	85	+	// - ctx context.Context
	86	+	func (_e CIArtifactStore_Expecter) AllArtifacts(ctx any) CIArtifactStore_AllArtifacts_Call {
	87	+	return &CIArtifactStore_AllArtifacts_Call{Call: _e.mock.On("AllArtifacts", ctx)}
	88	+	}
	89	+
	90	+	func (_c CIArtifactStore_AllArtifacts_Call) Run(run func(ctx context.Context)) CIArtifactStore_AllArtifacts_Call {
	91	+	_c.Call.Run(func(args mock.Arguments) {
	92	+	var arg0 context.Context
	93	+	if args[0] != nil {
	94	+	arg0 = args[0].(context.Context)
	95	+	}
	96	+	run(
	97	+	arg0,
	98	+	)
	99	+	})
	100	+	return _c
	101	+	}
	102	+
	103	+	func (_c CIArtifactStore_AllArtifacts_Call) Return(cIArtifacts []domain.CIArtifact, err error) CIArtifactStore_AllArtifacts_Call {
	104	+	_c.Call.Return(cIArtifacts, err)
	105	+	return _c
	106	+	}
	107	+
	108	+	func (_c CIArtifactStore_AllArtifacts_Call) RunAndReturn(run func(ctx context.Context) ([]domain.CIArtifact, error)) CIArtifactStore_AllArtifacts_Call {
	109	+	_c.Call.Return(run)
	110	+	return _c
	111	+	}
	112	+
	113	+	// ApplyArtifacts provides a mock function for the type CIArtifactStore
	114	+	func (_mock *CIArtifactStore) ApplyArtifacts(ctx context.Context, w ports.CIArtifactWrite) error {
	115	+	ret := _mock.Called(ctx, w)
	116	+
	117	+	if len(ret) == 0 {
	118	+	panic("no return value specified for ApplyArtifacts")
	119	+	}
	120	+
	121	+	var r0 error
	122	+	if returnFunc, ok := ret.Get(0).(func(context.Context, ports.CIArtifactWrite) error); ok {
	123	+	r0 = returnFunc(ctx, w)
	124	+	} else {
	125	+	r0 = ret.Error(0)
	126	+	}
	127	+	return r0
	128	+	}
	129	+
	130	+	// CIArtifactStore_ApplyArtifacts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ApplyArtifacts'
	131	+	type CIArtifactStore_ApplyArtifacts_Call struct {
	132	+	*mock.Call
	133	+	}
	134	+
	135	+	// ApplyArtifacts is a helper method to define mock.On call
	136	+	// - ctx context.Context
	137	+	// - w ports.CIArtifactWrite
	138	+	func (_e CIArtifactStore_Expecter) ApplyArtifacts(ctx any, w any) CIArtifactStore_ApplyArtifacts_Call {
	139	+	return &CIArtifactStore_ApplyArtifacts_Call{Call: _e.mock.On("ApplyArtifacts", ctx, w)}
	140	+	}
	141	+
	142	+	func (_c CIArtifactStore_ApplyArtifacts_Call) Run(run func(ctx context.Context, w ports.CIArtifactWrite)) CIArtifactStore_ApplyArtifacts_Call {
	143	+	_c.Call.Run(func(args mock.Arguments) {
	144	+	var arg0 context.Context
	145	+	if args[0] != nil {
	146	+	arg0 = args[0].(context.Context)
	147	+	}
	148	+	var arg1 ports.CIArtifactWrite
	149	+	if args[1] != nil {
	150	+	arg1 = args[1].(ports.CIArtifactWrite)
	151	+	}
	152	+	run(
	153	+	arg0,
	154	+	arg1,
	155	+	)
	156	+	})
	157	+	return _c
	158	+	}
	159	+
	160	+	func (_c CIArtifactStore_ApplyArtifacts_Call) Return(err error) CIArtifactStore_ApplyArtifacts_Call {
	161	+	_c.Call.Return(err)
	162	+	return _c
	163	+	}
	164	+
	165	+	func (_c CIArtifactStore_ApplyArtifacts_Call) RunAndReturn(run func(ctx context.Context, w ports.CIArtifactWrite) error) CIArtifactStore_ApplyArtifacts_Call {
	166	+	_c.Call.Return(run)
	167	+	return _c
	168	+	}
	169	+
	170	+	// ArtifactBlocks provides a mock function for the type CIArtifactStore
	171	+	func (_mock *CIArtifactStore) ArtifactBlocks(ctx context.Context, id int64) ([]ports.CIArtifactBlock, error) {
	172	+	ret := _mock.Called(ctx, id)
	173	+
	174	+	if len(ret) == 0 {
	175	+	panic("no return value specified for ArtifactBlocks")
	176	+	}
	177	+
	178	+	var r0 []ports.CIArtifactBlock
	179	+	var r1 error
	180	+	if returnFunc, ok := ret.Get(0).(func(context.Context, int64) ([]ports.CIArtifactBlock, error)); ok {
	181	+	return returnFunc(ctx, id)
	182	+	}
	183	+	if returnFunc, ok := ret.Get(0).(func(context.Context, int64) []ports.CIArtifactBlock); ok {
	184	+	r0 = returnFunc(ctx, id)
	185	+	} else {
	186	+	if ret.Get(0) != nil {
	187	+	r0 = ret.Get(0).([]ports.CIArtifactBlock)
	188	+	}
	189	+	}
	190	+	if returnFunc, ok := ret.Get(1).(func(context.Context, int64) error); ok {
	191	+	r1 = returnFunc(ctx, id)
	192	+	} else {
	193	+	r1 = ret.Error(1)
	194	+	}
	195	+	return r0, r1
	196	+	}
	197	+
	198	+	// CIArtifactStore_ArtifactBlocks_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ArtifactBlocks'
	199	+	type CIArtifactStore_ArtifactBlocks_Call struct {
	200	+	*mock.Call
	201	+	}
	202	+
	203	+	// ArtifactBlocks is a helper method to define mock.On call
	204	+	// - ctx context.Context
	205	+	// - id int64
	206	+	func (_e CIArtifactStore_Expecter) ArtifactBlocks(ctx any, id any) CIArtifactStore_ArtifactBlocks_Call {
	207	+	return &CIArtifactStore_ArtifactBlocks_Call{Call: _e.mock.On("ArtifactBlocks", ctx, id)}
	208	+	}
	209	+
	210	+	func (_c CIArtifactStore_ArtifactBlocks_Call) Run(run func(ctx context.Context, id int64)) CIArtifactStore_ArtifactBlocks_Call {
	211	+	_c.Call.Run(func(args mock.Arguments) {
	212	+	var arg0 context.Context
	213	+	if args[0] != nil {
	214	+	arg0 = args[0].(context.Context)
	215	+	}
	216	+	var arg1 int64
	217	+	if args[1] != nil {
	218	+	arg1 = args[1].(int64)
	219	+	}
	220	+	run(
	221	+	arg0,
	222	+	arg1,
	223	+	)
	224	+	})
	225	+	return _c
	226	+	}
	227	+
	228	+	func (_c CIArtifactStore_ArtifactBlocks_Call) Return(cIArtifactBlocks []ports.CIArtifactBlock, err error) CIArtifactStore_ArtifactBlocks_Call {
	229	+	_c.Call.Return(cIArtifactBlocks, err)
	230	+	return _c
	231	+	}
	232	+
	233	+	func (_c CIArtifactStore_ArtifactBlocks_Call) RunAndReturn(run func(ctx context.Context, id int64) ([]ports.CIArtifactBlock, error)) CIArtifactStore_ArtifactBlocks_Call {
	234	+	_c.Call.Return(run)
	235	+	return _c
	236	+	}
	237	+
	238	+	// ArtifactUsage provides a mock function for the type CIArtifactStore
	239	+	func (_mock *CIArtifactStore) ArtifactUsage(ctx context.Context, repo domain.RepoName) (int64, uint64, error) {
	240	+	ret := _mock.Called(ctx, repo)
	241	+
	242	+	if len(ret) == 0 {
	243	+	panic("no return value specified for ArtifactUsage")
	244	+	}
	245	+
	246	+	var r0 int64
	247	+	var r1 uint64
	248	+	var r2 error
	249	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.RepoName) (int64, uint64, error)); ok {
	250	+	return returnFunc(ctx, repo)
	251	+	}
	252	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.RepoName) int64); ok {
	253	+	r0 = returnFunc(ctx, repo)
	254	+	} else {
	255	+	r0 = ret.Get(0).(int64)
	256	+	}
	257	+	if returnFunc, ok := ret.Get(1).(func(context.Context, domain.RepoName) uint64); ok {
	258	+	r1 = returnFunc(ctx, repo)
	259	+	} else {
	260	+	r1 = ret.Get(1).(uint64)
	261	+	}
	262	+	if returnFunc, ok := ret.Get(2).(func(context.Context, domain.RepoName) error); ok {
	263	+	r2 = returnFunc(ctx, repo)
	264	+	} else {
	265	+	r2 = ret.Error(2)
	266	+	}
	267	+	return r0, r1, r2
	268	+	}
	269	+
	270	+	// CIArtifactStore_ArtifactUsage_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ArtifactUsage'
	271	+	type CIArtifactStore_ArtifactUsage_Call struct {
	272	+	*mock.Call
	273	+	}
	274	+
	275	+	// ArtifactUsage is a helper method to define mock.On call
	276	+	// - ctx context.Context
	277	+	// - repo domain.RepoName
	278	+	func (_e CIArtifactStore_Expecter) ArtifactUsage(ctx any, repo any) CIArtifactStore_ArtifactUsage_Call {
	279	+	return &CIArtifactStore_ArtifactUsage_Call{Call: _e.mock.On("ArtifactUsage", ctx, repo)}
	280	+	}
	281	+
	282	+	func (_c CIArtifactStore_ArtifactUsage_Call) Run(run func(ctx context.Context, repo domain.RepoName)) CIArtifactStore_ArtifactUsage_Call {
	283	+	_c.Call.Run(func(args mock.Arguments) {
	284	+	var arg0 context.Context
	285	+	if args[0] != nil {
	286	+	arg0 = args[0].(context.Context)
	287	+	}
	288	+	var arg1 domain.RepoName
	289	+	if args[1] != nil {
	290	+	arg1 = args[1].(domain.RepoName)
	291	+	}
	292	+	run(
	293	+	arg0,
	294	+	arg1,
	295	+	)
	296	+	})
	297	+	return _c
	298	+	}
	299	+
	300	+	func (_c CIArtifactStore_ArtifactUsage_Call) Return(used int64, version uint64, err error) CIArtifactStore_ArtifactUsage_Call {
	301	+	_c.Call.Return(used, version, err)
	302	+	return _c
	303	+	}
	304	+
	305	+	func (_c CIArtifactStore_ArtifactUsage_Call) RunAndReturn(run func(ctx context.Context, repo domain.RepoName) (int64, uint64, error)) CIArtifactStore_ArtifactUsage_Call {
	306	+	_c.Call.Return(run)
	307	+	return _c
	308	+	}
	309	+
	310	+	// BlockArtifactIDs provides a mock function for the type CIArtifactStore
	311	+	func (_mock *CIArtifactStore) BlockArtifactIDs(ctx context.Context) ([]int64, error) {
	312	+	ret := _mock.Called(ctx)
	313	+
	314	+	if len(ret) == 0 {
	315	+	panic("no return value specified for BlockArtifactIDs")
	316	+	}
	317	+
	318	+	var r0 []int64
	319	+	var r1 error
	320	+	if returnFunc, ok := ret.Get(0).(func(context.Context) ([]int64, error)); ok {
	321	+	return returnFunc(ctx)
	322	+	}
	323	+	if returnFunc, ok := ret.Get(0).(func(context.Context) []int64); ok {
	324	+	r0 = returnFunc(ctx)
	325	+	} else {
	326	+	if ret.Get(0) != nil {
	327	+	r0 = ret.Get(0).([]int64)
	328	+	}
	329	+	}
	330	+	if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
	331	+	r1 = returnFunc(ctx)
	332	+	} else {
	333	+	r1 = ret.Error(1)
	334	+	}
	335	+	return r0, r1
	336	+	}
	337	+
	338	+	// CIArtifactStore_BlockArtifactIDs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'BlockArtifactIDs'
	339	+	type CIArtifactStore_BlockArtifactIDs_Call struct {
	340	+	*mock.Call
	341	+	}
	342	+
	343	+	// BlockArtifactIDs is a helper method to define mock.On call
	344	+	// - ctx context.Context
	345	+	func (_e CIArtifactStore_Expecter) BlockArtifactIDs(ctx any) CIArtifactStore_BlockArtifactIDs_Call {
	346	+	return &CIArtifactStore_BlockArtifactIDs_Call{Call: _e.mock.On("BlockArtifactIDs", ctx)}
	347	+	}
	348	+
	349	+	func (_c CIArtifactStore_BlockArtifactIDs_Call) Run(run func(ctx context.Context)) CIArtifactStore_BlockArtifactIDs_Call {
	350	+	_c.Call.Run(func(args mock.Arguments) {
	351	+	var arg0 context.Context
	352	+	if args[0] != nil {
	353	+	arg0 = args[0].(context.Context)
	354	+	}
	355	+	run(
	356	+	arg0,
	357	+	)
	358	+	})
	359	+	return _c
	360	+	}
	361	+
	362	+	func (_c CIArtifactStore_BlockArtifactIDs_Call) Return(int64s []int64, err error) CIArtifactStore_BlockArtifactIDs_Call {
	363	+	_c.Call.Return(int64s, err)
	364	+	return _c
	365	+	}
	366	+
	367	+	func (_c CIArtifactStore_BlockArtifactIDs_Call) RunAndReturn(run func(ctx context.Context) ([]int64, error)) CIArtifactStore_BlockArtifactIDs_Call {
	368	+	_c.Call.Return(run)
	369	+	return _c
	370	+	}
	371	+
	372	+	// DeleteBlocks provides a mock function for the type CIArtifactStore
	373	+	func (_mock *CIArtifactStore) DeleteBlocks(ctx context.Context, id int64) error {
	374	+	ret := _mock.Called(ctx, id)
	375	+
	376	+	if len(ret) == 0 {
	377	+	panic("no return value specified for DeleteBlocks")
	378	+	}
	379	+
	380	+	var r0 error
	381	+	if returnFunc, ok := ret.Get(0).(func(context.Context, int64) error); ok {
	382	+	r0 = returnFunc(ctx, id)
	383	+	} else {
	384	+	r0 = ret.Error(0)
	385	+	}
	386	+	return r0
	387	+	}
	388	+
	389	+	// CIArtifactStore_DeleteBlocks_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteBlocks'
	390	+	type CIArtifactStore_DeleteBlocks_Call struct {
	391	+	*mock.Call
	392	+	}
	393	+
	394	+	// DeleteBlocks is a helper method to define mock.On call
	395	+	// - ctx context.Context
	396	+	// - id int64
	397	+	func (_e CIArtifactStore_Expecter) DeleteBlocks(ctx any, id any) CIArtifactStore_DeleteBlocks_Call {
	398	+	return &CIArtifactStore_DeleteBlocks_Call{Call: _e.mock.On("DeleteBlocks", ctx, id)}
	399	+	}
	400	+
	401	+	func (_c CIArtifactStore_DeleteBlocks_Call) Run(run func(ctx context.Context, id int64)) CIArtifactStore_DeleteBlocks_Call {
	402	+	_c.Call.Run(func(args mock.Arguments) {
	403	+	var arg0 context.Context
	404	+	if args[0] != nil {
	405	+	arg0 = args[0].(context.Context)
	406	+	}
	407	+	var arg1 int64
	408	+	if args[1] != nil {
	409	+	arg1 = args[1].(int64)
	410	+	}
	411	+	run(
	412	+	arg0,
	413	+	arg1,
	414	+	)
	415	+	})
	416	+	return _c
	417	+	}
	418	+
	419	+	func (_c CIArtifactStore_DeleteBlocks_Call) Return(err error) CIArtifactStore_DeleteBlocks_Call {
	420	+	_c.Call.Return(err)
	421	+	return _c
	422	+	}
	423	+
	424	+	func (_c CIArtifactStore_DeleteBlocks_Call) RunAndReturn(run func(ctx context.Context, id int64) error) CIArtifactStore_DeleteBlocks_Call {
	425	+	_c.Call.Return(run)
	426	+	return _c
	427	+	}
	428	+
	429	+	// GetArtifact provides a mock function for the type CIArtifactStore
	430	+	func (_mock *CIArtifactStore) GetArtifact(ctx context.Context, id int64) (domain.CIArtifact, error) {
	431	+	ret := _mock.Called(ctx, id)
	432	+
	433	+	if len(ret) == 0 {
	434	+	panic("no return value specified for GetArtifact")
	435	+	}
	436	+
	437	+	var r0 domain.CIArtifact
	438	+	var r1 error
	439	+	if returnFunc, ok := ret.Get(0).(func(context.Context, int64) (domain.CIArtifact, error)); ok {
	440	+	return returnFunc(ctx, id)
	441	+	}
	442	+	if returnFunc, ok := ret.Get(0).(func(context.Context, int64) domain.CIArtifact); ok {
	443	+	r0 = returnFunc(ctx, id)
	444	+	} else {
	445	+	r0 = ret.Get(0).(domain.CIArtifact)
	446	+	}
	447	+	if returnFunc, ok := ret.Get(1).(func(context.Context, int64) error); ok {
	448	+	r1 = returnFunc(ctx, id)
	449	+	} else {
	450	+	r1 = ret.Error(1)
	451	+	}
	452	+	return r0, r1
	453	+	}
	454	+
	455	+	// CIArtifactStore_GetArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetArtifact'
	456	+	type CIArtifactStore_GetArtifact_Call struct {
	457	+	*mock.Call
	458	+	}
	459	+
	460	+	// GetArtifact is a helper method to define mock.On call
	461	+	// - ctx context.Context
	462	+	// - id int64
	463	+	func (_e CIArtifactStore_Expecter) GetArtifact(ctx any, id any) CIArtifactStore_GetArtifact_Call {
	464	+	return &CIArtifactStore_GetArtifact_Call{Call: _e.mock.On("GetArtifact", ctx, id)}
	465	+	}
	466	+
	467	+	func (_c CIArtifactStore_GetArtifact_Call) Run(run func(ctx context.Context, id int64)) CIArtifactStore_GetArtifact_Call {
	468	+	_c.Call.Run(func(args mock.Arguments) {
	469	+	var arg0 context.Context
	470	+	if args[0] != nil {
	471	+	arg0 = args[0].(context.Context)
	472	+	}
	473	+	var arg1 int64
	474	+	if args[1] != nil {
	475	+	arg1 = args[1].(int64)
	476	+	}
	477	+	run(
	478	+	arg0,
	479	+	arg1,
	480	+	)
	481	+	})
	482	+	return _c
	483	+	}
	484	+
	485	+	func (_c CIArtifactStore_GetArtifact_Call) Return(cIArtifact domain.CIArtifact, err error) CIArtifactStore_GetArtifact_Call {
	486	+	_c.Call.Return(cIArtifact, err)
	487	+	return _c
	488	+	}
	489	+
	490	+	func (_c CIArtifactStore_GetArtifact_Call) RunAndReturn(run func(ctx context.Context, id int64) (domain.CIArtifact, error)) CIArtifactStore_GetArtifact_Call {
	491	+	_c.Call.Return(run)
	492	+	return _c
	493	+	}
	494	+
	495	+	// NextArtifactID provides a mock function for the type CIArtifactStore
	496	+	func (_mock *CIArtifactStore) NextArtifactID(ctx context.Context) (int64, error) {
	497	+	ret := _mock.Called(ctx)
	498	+
	499	+	if len(ret) == 0 {
	500	+	panic("no return value specified for NextArtifactID")
	501	+	}
	502	+
	503	+	var r0 int64
	504	+	var r1 error
	505	+	if returnFunc, ok := ret.Get(0).(func(context.Context) (int64, error)); ok {
	506	+	return returnFunc(ctx)
	507	+	}
	508	+	if returnFunc, ok := ret.Get(0).(func(context.Context) int64); ok {
	509	+	r0 = returnFunc(ctx)
	510	+	} else {
	511	+	r0 = ret.Get(0).(int64)
	512	+	}
	513	+	if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
	514	+	r1 = returnFunc(ctx)
	515	+	} else {
	516	+	r1 = ret.Error(1)
	517	+	}
	518	+	return r0, r1
	519	+	}
	520	+
	521	+	// CIArtifactStore_NextArtifactID_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'NextArtifactID'
	522	+	type CIArtifactStore_NextArtifactID_Call struct {
	523	+	*mock.Call
	524	+	}
	525	+
	526	+	// NextArtifactID is a helper method to define mock.On call
	527	+	// - ctx context.Context
	528	+	func (_e CIArtifactStore_Expecter) NextArtifactID(ctx any) CIArtifactStore_NextArtifactID_Call {
	529	+	return &CIArtifactStore_NextArtifactID_Call{Call: _e.mock.On("NextArtifactID", ctx)}
	530	+	}
	531	+
	532	+	func (_c CIArtifactStore_NextArtifactID_Call) Run(run func(ctx context.Context)) CIArtifactStore_NextArtifactID_Call {
	533	+	_c.Call.Run(func(args mock.Arguments) {
	534	+	var arg0 context.Context
	535	+	if args[0] != nil {
	536	+	arg0 = args[0].(context.Context)
	537	+	}
	538	+	run(
	539	+	arg0,
	540	+	)
	541	+	})
	542	+	return _c
	543	+	}
	544	+
	545	+	func (_c CIArtifactStore_NextArtifactID_Call) Return(n int64, err error) CIArtifactStore_NextArtifactID_Call {
	546	+	_c.Call.Return(n, err)
	547	+	return _c
	548	+	}
	549	+
	550	+	func (_c CIArtifactStore_NextArtifactID_Call) RunAndReturn(run func(ctx context.Context) (int64, error)) CIArtifactStore_NextArtifactID_Call {
	551	+	_c.Call.Return(run)
	552	+	return _c
	553	+	}
	554	+
	555	+	// RunArtifacts provides a mock function for the type CIArtifactStore
	556	+	func (_mock *CIArtifactStore) RunArtifacts(ctx context.Context, repo domain.RepoName, runID int64) ([]domain.CIArtifact, error) {
	557	+	ret := _mock.Called(ctx, repo, runID)
	558	+
	559	+	if len(ret) == 0 {
	560	+	panic("no return value specified for RunArtifacts")
	561	+	}
	562	+
	563	+	var r0 []domain.CIArtifact
	564	+	var r1 error
	565	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.RepoName, int64) ([]domain.CIArtifact, error)); ok {
	566	+	return returnFunc(ctx, repo, runID)
	567	+	}
	568	+	if returnFunc, ok := ret.Get(0).(func(context.Context, domain.RepoName, int64) []domain.CIArtifact); ok {
	569	+	r0 = returnFunc(ctx, repo, runID)
	570	+	} else {
	571	+	if ret.Get(0) != nil {
	572	+	r0 = ret.Get(0).([]domain.CIArtifact)
	573	+	}
	574	+	}
	575	+	if returnFunc, ok := ret.Get(1).(func(context.Context, domain.RepoName, int64) error); ok {
	576	+	r1 = returnFunc(ctx, repo, runID)
	577	+	} else {
	578	+	r1 = ret.Error(1)
	579	+	}
	580	+	return r0, r1
	581	+	}
	582	+
	583	+	// CIArtifactStore_RunArtifacts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'RunArtifacts'
	584	+	type CIArtifactStore_RunArtifacts_Call struct {
	585	+	*mock.Call
	586	+	}
	587	+
	588	+	// RunArtifacts is a helper method to define mock.On call
	589	+	// - ctx context.Context
	590	+	// - repo domain.RepoName
	591	+	// - runID int64
	592	+	func (_e CIArtifactStore_Expecter) RunArtifacts(ctx any, repo any, runID any) CIArtifactStore_RunArtifacts_Call {
	593	+	return &CIArtifactStore_RunArtifacts_Call{Call: _e.mock.On("RunArtifacts", ctx, repo, runID)}
	594	+	}
	595	+
	596	+	func (_c CIArtifactStore_RunArtifacts_Call) Run(run func(ctx context.Context, repo domain.RepoName, runID int64)) CIArtifactStore_RunArtifacts_Call {
	597	+	_c.Call.Run(func(args mock.Arguments) {
	598	+	var arg0 context.Context
	599	+	if args[0] != nil {
	600	+	arg0 = args[0].(context.Context)
	601	+	}
	602	+	var arg1 domain.RepoName
	603	+	if args[1] != nil {
	604	+	arg1 = args[1].(domain.RepoName)
	605	+	}
	606	+	var arg2 int64
	607	+	if args[2] != nil {
	608	+	arg2 = args[2].(int64)
	609	+	}
	610	+	run(
	611	+	arg0,
	612	+	arg1,
	613	+	arg2,
	614	+	)
	615	+	})
	616	+	return _c
	617	+	}
	618	+
	619	+	func (_c CIArtifactStore_RunArtifacts_Call) Return(cIArtifacts []domain.CIArtifact, err error) CIArtifactStore_RunArtifacts_Call {
	620	+	_c.Call.Return(cIArtifacts, err)
	621	+	return _c
	622	+	}
	623	+
	624	+	func (_c CIArtifactStore_RunArtifacts_Call) RunAndReturn(run func(ctx context.Context, repo domain.RepoName, runID int64) ([]domain.CIArtifact, error)) CIArtifactStore_RunArtifacts_Call {
	625	+	_c.Call.Return(run)
	626	+	return _c
	627	+	}

A internal/ports/mocks/JobArtifactService.go +637 -0

@@ -0,0 +1,637 @@
	1	+	// Code generated by mockery; DO NOT EDIT.
	2	+	// github.com/vektra/mockery
	3	+	// template: testify
	4	+
	5	+	package mocks
	6	+
	7	+	import (
	8	+	"context"
	9	+	"io"
	10	+
	11	+	"bnema.dev/ermite/internal/domain"
	12	+	"bnema.dev/ermite/internal/ports"
	13	+	mock "github.com/stretchr/testify/mock"
	14	+	)
	15	+
	16	+	// NewJobArtifactService creates a new instance of JobArtifactService. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
	17	+	// The first argument is typically a *testing.T value.
	18	+	func NewJobArtifactService(t interface {
	19	+	mock.TestingT
	20	+	Cleanup(func())
	21	+	}) *JobArtifactService {
	22	+	if helper, ok := t.(interface{ Helper() }); ok {
	23	+	helper.Helper()
	24	+	}
	25	+
	26	+	mock := &JobArtifactService{}
	27	+	mock.Mock.Test(t)
	28	+
	29	+	t.Cleanup(func() {
	30	+	if helper, ok := t.(interface{ Helper() }); ok {
	31	+	helper.Helper()
	32	+	}
	33	+	mock.AssertExpectations(t)
	34	+	})
	35	+
	36	+	return mock
	37	+	}
	38	+
	39	+	// JobArtifactService is an autogenerated mock type for the JobArtifactService type
	40	+	type JobArtifactService struct {
	41	+	mock.Mock
	42	+	}
	43	+
	44	+	type JobArtifactService_Expecter struct {
	45	+	mock *mock.Mock
	46	+	}
	47	+
	48	+	func (_m JobArtifactService) EXPECT() JobArtifactService_Expecter {
	49	+	return &JobArtifactService_Expecter{mock: &_m.Mock}
	50	+	}
	51	+
	52	+	// CommitJobArtifact provides a mock function for the type JobArtifactService
	53	+	func (_mock *JobArtifactService) CommitJobArtifact(ctx context.Context, secret string, jobID string, artifactID int64, blockIDs [][]byte) (int64, string, error) {
	54	+	ret := _mock.Called(ctx, secret, jobID, artifactID, blockIDs)
	55	+
	56	+	if len(ret) == 0 {
	57	+	panic("no return value specified for CommitJobArtifact")
	58	+	}
	59	+
	60	+	var r0 int64
	61	+	var r1 string
	62	+	var r2 error
	63	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, int64, [][]byte) (int64, string, error)); ok {
	64	+	return returnFunc(ctx, secret, jobID, artifactID, blockIDs)
	65	+	}
	66	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, int64, [][]byte) int64); ok {
	67	+	r0 = returnFunc(ctx, secret, jobID, artifactID, blockIDs)
	68	+	} else {
	69	+	r0 = ret.Get(0).(int64)
	70	+	}
	71	+	if returnFunc, ok := ret.Get(1).(func(context.Context, string, string, int64, [][]byte) string); ok {
	72	+	r1 = returnFunc(ctx, secret, jobID, artifactID, blockIDs)
	73	+	} else {
	74	+	r1 = ret.Get(1).(string)
	75	+	}
	76	+	if returnFunc, ok := ret.Get(2).(func(context.Context, string, string, int64, [][]byte) error); ok {
	77	+	r2 = returnFunc(ctx, secret, jobID, artifactID, blockIDs)
	78	+	} else {
	79	+	r2 = ret.Error(2)
	80	+	}
	81	+	return r0, r1, r2
	82	+	}
	83	+
	84	+	// JobArtifactService_CommitJobArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'CommitJobArtifact'
	85	+	type JobArtifactService_CommitJobArtifact_Call struct {
	86	+	*mock.Call
	87	+	}
	88	+
	89	+	// CommitJobArtifact is a helper method to define mock.On call
	90	+	// - ctx context.Context
	91	+	// - secret string
	92	+	// - jobID string
	93	+	// - artifactID int64
	94	+	// - blockIDs [][]byte
	95	+	func (_e JobArtifactService_Expecter) CommitJobArtifact(ctx any, secret any, jobID any, artifactID any, blockIDs any) JobArtifactService_CommitJobArtifact_Call {
	96	+	return &JobArtifactService_CommitJobArtifact_Call{Call: _e.mock.On("CommitJobArtifact", ctx, secret, jobID, artifactID, blockIDs)}
	97	+	}
	98	+
	99	+	func (_c JobArtifactService_CommitJobArtifact_Call) Run(run func(ctx context.Context, secret string, jobID string, artifactID int64, blockIDs [][]byte)) JobArtifactService_CommitJobArtifact_Call {
	100	+	_c.Call.Run(func(args mock.Arguments) {
	101	+	var arg0 context.Context
	102	+	if args[0] != nil {
	103	+	arg0 = args[0].(context.Context)
	104	+	}
	105	+	var arg1 string
	106	+	if args[1] != nil {
	107	+	arg1 = args[1].(string)
	108	+	}
	109	+	var arg2 string
	110	+	if args[2] != nil {
	111	+	arg2 = args[2].(string)
	112	+	}
	113	+	var arg3 int64
	114	+	if args[3] != nil {
	115	+	arg3 = args[3].(int64)
	116	+	}
	117	+	var arg4 [][]byte
	118	+	if args[4] != nil {
	119	+	arg4 = args[4].([][]byte)
	120	+	}
	121	+	run(
	122	+	arg0,
	123	+	arg1,
	124	+	arg2,
	125	+	arg3,
	126	+	arg4,
	127	+	)
	128	+	})
	129	+	return _c
	130	+	}
	131	+
	132	+	func (_c JobArtifactService_CommitJobArtifact_Call) Return(size int64, digest string, err error) JobArtifactService_CommitJobArtifact_Call {
	133	+	_c.Call.Return(size, digest, err)
	134	+	return _c
	135	+	}
	136	+
	137	+	func (_c JobArtifactService_CommitJobArtifact_Call) RunAndReturn(run func(ctx context.Context, secret string, jobID string, artifactID int64, blockIDs [][]byte) (int64, string, error)) JobArtifactService_CommitJobArtifact_Call {
	138	+	_c.Call.Return(run)
	139	+	return _c
	140	+	}
	141	+
	142	+	// CreateJobArtifact provides a mock function for the type JobArtifactService
	143	+	func (_mock *JobArtifactService) CreateJobArtifact(ctx context.Context, secret string, params ports.CreateJobArtifactParams) (ports.CreatedArtifact, error) {
	144	+	ret := _mock.Called(ctx, secret, params)
	145	+
	146	+	if len(ret) == 0 {
	147	+	panic("no return value specified for CreateJobArtifact")
	148	+	}
	149	+
	150	+	var r0 ports.CreatedArtifact
	151	+	var r1 error
	152	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, ports.CreateJobArtifactParams) (ports.CreatedArtifact, error)); ok {
	153	+	return returnFunc(ctx, secret, params)
	154	+	}
	155	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, ports.CreateJobArtifactParams) ports.CreatedArtifact); ok {
	156	+	r0 = returnFunc(ctx, secret, params)
	157	+	} else {
	158	+	r0 = ret.Get(0).(ports.CreatedArtifact)
	159	+	}
	160	+	if returnFunc, ok := ret.Get(1).(func(context.Context, string, ports.CreateJobArtifactParams) error); ok {
	161	+	r1 = returnFunc(ctx, secret, params)
	162	+	} else {
	163	+	r1 = ret.Error(1)
	164	+	}
	165	+	return r0, r1
	166	+	}
	167	+
	168	+	// JobArtifactService_CreateJobArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'CreateJobArtifact'
	169	+	type JobArtifactService_CreateJobArtifact_Call struct {
	170	+	*mock.Call
	171	+	}
	172	+
	173	+	// CreateJobArtifact is a helper method to define mock.On call
	174	+	// - ctx context.Context
	175	+	// - secret string
	176	+	// - params ports.CreateJobArtifactParams
	177	+	func (_e JobArtifactService_Expecter) CreateJobArtifact(ctx any, secret any, params any) JobArtifactService_CreateJobArtifact_Call {
	178	+	return &JobArtifactService_CreateJobArtifact_Call{Call: _e.mock.On("CreateJobArtifact", ctx, secret, params)}
	179	+	}
	180	+
	181	+	func (_c JobArtifactService_CreateJobArtifact_Call) Run(run func(ctx context.Context, secret string, params ports.CreateJobArtifactParams)) JobArtifactService_CreateJobArtifact_Call {
	182	+	_c.Call.Run(func(args mock.Arguments) {
	183	+	var arg0 context.Context
	184	+	if args[0] != nil {
	185	+	arg0 = args[0].(context.Context)
	186	+	}
	187	+	var arg1 string
	188	+	if args[1] != nil {
	189	+	arg1 = args[1].(string)
	190	+	}
	191	+	var arg2 ports.CreateJobArtifactParams
	192	+	if args[2] != nil {
	193	+	arg2 = args[2].(ports.CreateJobArtifactParams)
	194	+	}
	195	+	run(
	196	+	arg0,
	197	+	arg1,
	198	+	arg2,
	199	+	)
	200	+	})
	201	+	return _c
	202	+	}
	203	+
	204	+	func (_c JobArtifactService_CreateJobArtifact_Call) Return(createdArtifact ports.CreatedArtifact, err error) JobArtifactService_CreateJobArtifact_Call {
	205	+	_c.Call.Return(createdArtifact, err)
	206	+	return _c
	207	+	}
	208	+
	209	+	func (_c JobArtifactService_CreateJobArtifact_Call) RunAndReturn(run func(ctx context.Context, secret string, params ports.CreateJobArtifactParams) (ports.CreatedArtifact, error)) JobArtifactService_CreateJobArtifact_Call {
	210	+	_c.Call.Return(run)
	211	+	return _c
	212	+	}
	213	+
	214	+	// DeleteJobArtifact provides a mock function for the type JobArtifactService
	215	+	func (_mock *JobArtifactService) DeleteJobArtifact(ctx context.Context, secret string, jobID string, name string) (int64, error) {
	216	+	ret := _mock.Called(ctx, secret, jobID, name)
	217	+
	218	+	if len(ret) == 0 {
	219	+	panic("no return value specified for DeleteJobArtifact")
	220	+	}
	221	+
	222	+	var r0 int64
	223	+	var r1 error
	224	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, string) (int64, error)); ok {
	225	+	return returnFunc(ctx, secret, jobID, name)
	226	+	}
	227	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, string) int64); ok {
	228	+	r0 = returnFunc(ctx, secret, jobID, name)
	229	+	} else {
	230	+	r0 = ret.Get(0).(int64)
	231	+	}
	232	+	if returnFunc, ok := ret.Get(1).(func(context.Context, string, string, string) error); ok {
	233	+	r1 = returnFunc(ctx, secret, jobID, name)
	234	+	} else {
	235	+	r1 = ret.Error(1)
	236	+	}
	237	+	return r0, r1
	238	+	}
	239	+
	240	+	// JobArtifactService_DeleteJobArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteJobArtifact'
	241	+	type JobArtifactService_DeleteJobArtifact_Call struct {
	242	+	*mock.Call
	243	+	}
	244	+
	245	+	// DeleteJobArtifact is a helper method to define mock.On call
	246	+	// - ctx context.Context
	247	+	// - secret string
	248	+	// - jobID string
	249	+	// - name string
	250	+	func (_e JobArtifactService_Expecter) DeleteJobArtifact(ctx any, secret any, jobID any, name any) JobArtifactService_DeleteJobArtifact_Call {
	251	+	return &JobArtifactService_DeleteJobArtifact_Call{Call: _e.mock.On("DeleteJobArtifact", ctx, secret, jobID, name)}
	252	+	}
	253	+
	254	+	func (_c JobArtifactService_DeleteJobArtifact_Call) Run(run func(ctx context.Context, secret string, jobID string, name string)) JobArtifactService_DeleteJobArtifact_Call {
	255	+	_c.Call.Run(func(args mock.Arguments) {
	256	+	var arg0 context.Context
	257	+	if args[0] != nil {
	258	+	arg0 = args[0].(context.Context)
	259	+	}
	260	+	var arg1 string
	261	+	if args[1] != nil {
	262	+	arg1 = args[1].(string)
	263	+	}
	264	+	var arg2 string
	265	+	if args[2] != nil {
	266	+	arg2 = args[2].(string)
	267	+	}
	268	+	var arg3 string
	269	+	if args[3] != nil {
	270	+	arg3 = args[3].(string)
	271	+	}
	272	+	run(
	273	+	arg0,
	274	+	arg1,
	275	+	arg2,
	276	+	arg3,
	277	+	)
	278	+	})
	279	+	return _c
	280	+	}
	281	+
	282	+	func (_c JobArtifactService_DeleteJobArtifact_Call) Return(n int64, err error) JobArtifactService_DeleteJobArtifact_Call {
	283	+	_c.Call.Return(n, err)
	284	+	return _c
	285	+	}
	286	+
	287	+	func (_c JobArtifactService_DeleteJobArtifact_Call) RunAndReturn(run func(ctx context.Context, secret string, jobID string, name string) (int64, error)) JobArtifactService_DeleteJobArtifact_Call {
	288	+	_c.Call.Return(run)
	289	+	return _c
	290	+	}
	291	+
	292	+	// DownloadJobArtifact provides a mock function for the type JobArtifactService
	293	+	func (_mock *JobArtifactService) DownloadJobArtifact(ctx context.Context, secret string, jobID string, artifactID int64, offset int64, send func(ports.ArtifactChunk) error) error {
	294	+	ret := _mock.Called(ctx, secret, jobID, artifactID, offset, send)
	295	+
	296	+	if len(ret) == 0 {
	297	+	panic("no return value specified for DownloadJobArtifact")
	298	+	}
	299	+
	300	+	var r0 error
	301	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, int64, int64, func(ports.ArtifactChunk) error) error); ok {
	302	+	r0 = returnFunc(ctx, secret, jobID, artifactID, offset, send)
	303	+	} else {
	304	+	r0 = ret.Error(0)
	305	+	}
	306	+	return r0
	307	+	}
	308	+
	309	+	// JobArtifactService_DownloadJobArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DownloadJobArtifact'
	310	+	type JobArtifactService_DownloadJobArtifact_Call struct {
	311	+	*mock.Call
	312	+	}
	313	+
	314	+	// DownloadJobArtifact is a helper method to define mock.On call
	315	+	// - ctx context.Context
	316	+	// - secret string
	317	+	// - jobID string
	318	+	// - artifactID int64
	319	+	// - offset int64
	320	+	// - send func(ports.ArtifactChunk) error
	321	+	func (_e JobArtifactService_Expecter) DownloadJobArtifact(ctx any, secret any, jobID any, artifactID any, offset any, send any) JobArtifactService_DownloadJobArtifact_Call {
	322	+	return &JobArtifactService_DownloadJobArtifact_Call{Call: _e.mock.On("DownloadJobArtifact", ctx, secret, jobID, artifactID, offset, send)}
	323	+	}
	324	+
	325	+	func (_c JobArtifactService_DownloadJobArtifact_Call) Run(run func(ctx context.Context, secret string, jobID string, artifactID int64, offset int64, send func(ports.ArtifactChunk) error)) JobArtifactService_DownloadJobArtifact_Call {
	326	+	_c.Call.Run(func(args mock.Arguments) {
	327	+	var arg0 context.Context
	328	+	if args[0] != nil {
	329	+	arg0 = args[0].(context.Context)
	330	+	}
	331	+	var arg1 string
	332	+	if args[1] != nil {
	333	+	arg1 = args[1].(string)
	334	+	}
	335	+	var arg2 string
	336	+	if args[2] != nil {
	337	+	arg2 = args[2].(string)
	338	+	}
	339	+	var arg3 int64
	340	+	if args[3] != nil {
	341	+	arg3 = args[3].(int64)
	342	+	}
	343	+	var arg4 int64
	344	+	if args[4] != nil {
	345	+	arg4 = args[4].(int64)
	346	+	}
	347	+	var arg5 func(ports.ArtifactChunk) error
	348	+	if args[5] != nil {
	349	+	arg5 = args[5].(func(ports.ArtifactChunk) error)
	350	+	}
	351	+	run(
	352	+	arg0,
	353	+	arg1,
	354	+	arg2,
	355	+	arg3,
	356	+	arg4,
	357	+	arg5,
	358	+	)
	359	+	})
	360	+	return _c
	361	+	}
	362	+
	363	+	func (_c JobArtifactService_DownloadJobArtifact_Call) Return(err error) JobArtifactService_DownloadJobArtifact_Call {
	364	+	_c.Call.Return(err)
	365	+	return _c
	366	+	}
	367	+
	368	+	func (_c JobArtifactService_DownloadJobArtifact_Call) RunAndReturn(run func(ctx context.Context, secret string, jobID string, artifactID int64, offset int64, send func(ports.ArtifactChunk) error) error) JobArtifactService_DownloadJobArtifact_Call {
	369	+	_c.Call.Return(run)
	370	+	return _c
	371	+	}
	372	+
	373	+	// FinalizeJobArtifact provides a mock function for the type JobArtifactService
	374	+	func (_mock *JobArtifactService) FinalizeJobArtifact(ctx context.Context, secret string, jobID string, name string, size int64, digest string) (domain.CIArtifact, error) {
	375	+	ret := _mock.Called(ctx, secret, jobID, name, size, digest)
	376	+
	377	+	if len(ret) == 0 {
	378	+	panic("no return value specified for FinalizeJobArtifact")
	379	+	}
	380	+
	381	+	var r0 domain.CIArtifact
	382	+	var r1 error
	383	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, string, int64, string) (domain.CIArtifact, error)); ok {
	384	+	return returnFunc(ctx, secret, jobID, name, size, digest)
	385	+	}
	386	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, string, int64, string) domain.CIArtifact); ok {
	387	+	r0 = returnFunc(ctx, secret, jobID, name, size, digest)
	388	+	} else {
	389	+	r0 = ret.Get(0).(domain.CIArtifact)
	390	+	}
	391	+	if returnFunc, ok := ret.Get(1).(func(context.Context, string, string, string, int64, string) error); ok {
	392	+	r1 = returnFunc(ctx, secret, jobID, name, size, digest)
	393	+	} else {
	394	+	r1 = ret.Error(1)
	395	+	}
	396	+	return r0, r1
	397	+	}
	398	+
	399	+	// JobArtifactService_FinalizeJobArtifact_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'FinalizeJobArtifact'
	400	+	type JobArtifactService_FinalizeJobArtifact_Call struct {
	401	+	*mock.Call
	402	+	}
	403	+
	404	+	// FinalizeJobArtifact is a helper method to define mock.On call
	405	+	// - ctx context.Context
	406	+	// - secret string
	407	+	// - jobID string
	408	+	// - name string
	409	+	// - size int64
	410	+	// - digest string
	411	+	func (_e JobArtifactService_Expecter) FinalizeJobArtifact(ctx any, secret any, jobID any, name any, size any, digest any) JobArtifactService_FinalizeJobArtifact_Call {
	412	+	return &JobArtifactService_FinalizeJobArtifact_Call{Call: _e.mock.On("FinalizeJobArtifact", ctx, secret, jobID, name, size, digest)}
	413	+	}
	414	+
	415	+	func (_c JobArtifactService_FinalizeJobArtifact_Call) Run(run func(ctx context.Context, secret string, jobID string, name string, size int64, digest string)) JobArtifactService_FinalizeJobArtifact_Call {
	416	+	_c.Call.Run(func(args mock.Arguments) {
	417	+	var arg0 context.Context
	418	+	if args[0] != nil {
	419	+	arg0 = args[0].(context.Context)
	420	+	}
	421	+	var arg1 string
	422	+	if args[1] != nil {
	423	+	arg1 = args[1].(string)
	424	+	}
	425	+	var arg2 string
	426	+	if args[2] != nil {
	427	+	arg2 = args[2].(string)
	428	+	}
	429	+	var arg3 string
	430	+	if args[3] != nil {
	431	+	arg3 = args[3].(string)
	432	+	}
	433	+	var arg4 int64
	434	+	if args[4] != nil {
	435	+	arg4 = args[4].(int64)
	436	+	}
	437	+	var arg5 string
	438	+	if args[5] != nil {
	439	+	arg5 = args[5].(string)
	440	+	}
	441	+	run(
	442	+	arg0,
	443	+	arg1,
	444	+	arg2,
	445	+	arg3,
	446	+	arg4,
	447	+	arg5,
	448	+	)
	449	+	})
	450	+	return _c
	451	+	}
	452	+
	453	+	func (_c JobArtifactService_FinalizeJobArtifact_Call) Return(cIArtifact domain.CIArtifact, err error) JobArtifactService_FinalizeJobArtifact_Call {
	454	+	_c.Call.Return(cIArtifact, err)
	455	+	return _c
	456	+	}
	457	+
	458	+	func (_c JobArtifactService_FinalizeJobArtifact_Call) RunAndReturn(run func(ctx context.Context, secret string, jobID string, name string, size int64, digest string) (domain.CIArtifact, error)) JobArtifactService_FinalizeJobArtifact_Call {
	459	+	_c.Call.Return(run)
	460	+	return _c
	461	+	}
	462	+
	463	+	// ListJobArtifacts provides a mock function for the type JobArtifactService
	464	+	func (_mock *JobArtifactService) ListJobArtifacts(ctx context.Context, secret string, jobID string, name string, artifactID int64) ([]domain.CIArtifact, error) {
	465	+	ret := _mock.Called(ctx, secret, jobID, name, artifactID)
	466	+
	467	+	if len(ret) == 0 {
	468	+	panic("no return value specified for ListJobArtifacts")
	469	+	}
	470	+
	471	+	var r0 []domain.CIArtifact
	472	+	var r1 error
	473	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, string, int64) ([]domain.CIArtifact, error)); ok {
	474	+	return returnFunc(ctx, secret, jobID, name, artifactID)
	475	+	}
	476	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, string, int64) []domain.CIArtifact); ok {
	477	+	r0 = returnFunc(ctx, secret, jobID, name, artifactID)
	478	+	} else {
	479	+	if ret.Get(0) != nil {
	480	+	r0 = ret.Get(0).([]domain.CIArtifact)
	481	+	}
	482	+	}
	483	+	if returnFunc, ok := ret.Get(1).(func(context.Context, string, string, string, int64) error); ok {
	484	+	r1 = returnFunc(ctx, secret, jobID, name, artifactID)
	485	+	} else {
	486	+	r1 = ret.Error(1)
	487	+	}
	488	+	return r0, r1
	489	+	}
	490	+
	491	+	// JobArtifactService_ListJobArtifacts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ListJobArtifacts'
	492	+	type JobArtifactService_ListJobArtifacts_Call struct {
	493	+	*mock.Call
	494	+	}
	495	+
	496	+	// ListJobArtifacts is a helper method to define mock.On call
	497	+	// - ctx context.Context
	498	+	// - secret string
	499	+	// - jobID string
	500	+	// - name string
	501	+	// - artifactID int64
	502	+	func (_e JobArtifactService_Expecter) ListJobArtifacts(ctx any, secret any, jobID any, name any, artifactID any) JobArtifactService_ListJobArtifacts_Call {
	503	+	return &JobArtifactService_ListJobArtifacts_Call{Call: _e.mock.On("ListJobArtifacts", ctx, secret, jobID, name, artifactID)}
	504	+	}
	505	+
	506	+	func (_c JobArtifactService_ListJobArtifacts_Call) Run(run func(ctx context.Context, secret string, jobID string, name string, artifactID int64)) JobArtifactService_ListJobArtifacts_Call {
	507	+	_c.Call.Run(func(args mock.Arguments) {
	508	+	var arg0 context.Context
	509	+	if args[0] != nil {
	510	+	arg0 = args[0].(context.Context)
	511	+	}
	512	+	var arg1 string
	513	+	if args[1] != nil {
	514	+	arg1 = args[1].(string)
	515	+	}
	516	+	var arg2 string
	517	+	if args[2] != nil {
	518	+	arg2 = args[2].(string)
	519	+	}
	520	+	var arg3 string
	521	+	if args[3] != nil {
	522	+	arg3 = args[3].(string)
	523	+	}
	524	+	var arg4 int64
	525	+	if args[4] != nil {
	526	+	arg4 = args[4].(int64)
	527	+	}
	528	+	run(
	529	+	arg0,
	530	+	arg1,
	531	+	arg2,
	532	+	arg3,
	533	+	arg4,
	534	+	)
	535	+	})
	536	+	return _c
	537	+	}
	538	+
	539	+	func (_c JobArtifactService_ListJobArtifacts_Call) Return(cIArtifacts []domain.CIArtifact, err error) JobArtifactService_ListJobArtifacts_Call {
	540	+	_c.Call.Return(cIArtifacts, err)
	541	+	return _c
	542	+	}
	543	+
	544	+	func (_c JobArtifactService_ListJobArtifacts_Call) RunAndReturn(run func(ctx context.Context, secret string, jobID string, name string, artifactID int64) ([]domain.CIArtifact, error)) JobArtifactService_ListJobArtifacts_Call {
	545	+	_c.Call.Return(run)
	546	+	return _c
	547	+	}
	548	+
	549	+	// UploadJobArtifactBlock provides a mock function for the type JobArtifactService
	550	+	func (_mock *JobArtifactService) UploadJobArtifactBlock(ctx context.Context, secret string, jobID string, artifactID int64, blockID []byte, data io.Reader) (int64, error) {
	551	+	ret := _mock.Called(ctx, secret, jobID, artifactID, blockID, data)
	552	+
	553	+	if len(ret) == 0 {
	554	+	panic("no return value specified for UploadJobArtifactBlock")
	555	+	}
	556	+
	557	+	var r0 int64
	558	+	var r1 error
	559	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, int64, []byte, io.Reader) (int64, error)); ok {
	560	+	return returnFunc(ctx, secret, jobID, artifactID, blockID, data)
	561	+	}
	562	+	if returnFunc, ok := ret.Get(0).(func(context.Context, string, string, int64, []byte, io.Reader) int64); ok {
	563	+	r0 = returnFunc(ctx, secret, jobID, artifactID, blockID, data)
	564	+	} else {
	565	+	r0 = ret.Get(0).(int64)
	566	+	}
	567	+	if returnFunc, ok := ret.Get(1).(func(context.Context, string, string, int64, []byte, io.Reader) error); ok {
	568	+	r1 = returnFunc(ctx, secret, jobID, artifactID, blockID, data)
	569	+	} else {
	570	+	r1 = ret.Error(1)
	571	+	}
	572	+	return r0, r1
	573	+	}
	574	+
	575	+	// JobArtifactService_UploadJobArtifactBlock_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UploadJobArtifactBlock'
	576	+	type JobArtifactService_UploadJobArtifactBlock_Call struct {
	577	+	*mock.Call
	578	+	}
	579	+
	580	+	// UploadJobArtifactBlock is a helper method to define mock.On call
	581	+	// - ctx context.Context
	582	+	// - secret string
	583	+	// - jobID string
	584	+	// - artifactID int64
	585	+	// - blockID []byte
	586	+	// - data io.Reader
	587	+	func (_e JobArtifactService_Expecter) UploadJobArtifactBlock(ctx any, secret any, jobID any, artifactID any, blockID any, data any) JobArtifactService_UploadJobArtifactBlock_Call {
	588	+	return &JobArtifactService_UploadJobArtifactBlock_Call{Call: _e.mock.On("UploadJobArtifactBlock", ctx, secret, jobID, artifactID, blockID, data)}
	589	+	}
	590	+
	591	+	func (_c JobArtifactService_UploadJobArtifactBlock_Call) Run(run func(ctx context.Context, secret string, jobID string, artifactID int64, blockID []byte, data io.Reader)) JobArtifactService_UploadJobArtifactBlock_Call {
	592	+	_c.Call.Run(func(args mock.Arguments) {
	593	+	var arg0 context.Context
	594	+	if args[0] != nil {
	595	+	arg0 = args[0].(context.Context)
	596	+	}
	597	+	var arg1 string
	598	+	if args[1] != nil {
	599	+	arg1 = args[1].(string)
	600	+	}
	601	+	var arg2 string
	602	+	if args[2] != nil {
	603	+	arg2 = args[2].(string)
	604	+	}
	605	+	var arg3 int64
	606	+	if args[3] != nil {
	607	+	arg3 = args[3].(int64)
	608	+	}
	609	+	var arg4 []byte
	610	+	if args[4] != nil {
	611	+	arg4 = args[4].([]byte)
	612	+	}
	613	+	var arg5 io.Reader
	614	+	if args[5] != nil {
	615	+	arg5 = args[5].(io.Reader)
	616	+	}
	617	+	run(
	618	+	arg0,
	619	+	arg1,
	620	+	arg2,
	621	+	arg3,
	622	+	arg4,
	623	+	arg5,
	624	+	)
	625	+	})
	626	+	return _c
	627	+	}
	628	+
	629	+	func (_c JobArtifactService_UploadJobArtifactBlock_Call) Return(n int64, err error) JobArtifactService_UploadJobArtifactBlock_Call {
	630	+	_c.Call.Return(n, err)
	631	+	return _c
	632	+	}
	633	+
	634	+	func (_c JobArtifactService_UploadJobArtifactBlock_Call) RunAndReturn(run func(ctx context.Context, secret string, jobID string, artifactID int64, blockID []byte, data io.Reader) (int64, error)) JobArtifactService_UploadJobArtifactBlock_Call {
	635	+	_c.Call.Return(run)
	636	+	return _c
	637	+	}

M internal/usecase/ci.go +20 -0


@@ -124,6 +124,15 @@ type CIConfig struct {
124	PublicURL string	124		PublicURL string
125	// Retention is how long runs, jobs, checks and logs are kept.	125		// Retention is how long runs, jobs, checks and logs are kept.
126	Retention time.Duration	126		Retention time.Duration
		127	+	// Artifacts and ArtifactFiles keep the artifacts of the runs (CI-4 D3):
		128	+	// the records and the contents. Without both the artifact services refuse
		129	+	// every call. ArtifactLimits are their settings; a zero field takes its
		130	+	// default (the run retention of domain.CIRetentionDays,
		131	+	// domain.DefaultCIArtifactQuota, domain.DefaultCIArtifactMaxSize). Leave
		132	+	// them unset rather than assigning typed nil pointers.
		133	+	Artifacts ports.CIArtifactStore
		134	+	ArtifactFiles ports.CIArtifactFiles
		135	+	ArtifactLimits ports.ArtifactLimits
127		136
128	// The durations below default to the contract's; tests shorten them.	137		// The durations below default to the contract's; tests shorten them.
129	PollTimeout time.Duration	138		PollTimeout time.Duration

@@ -186,6 +195,13 @@ func (c CIConfig) withDefaults() CIConfig {
186	def(&c.TimeoutSlack, domain.CITimeoutSlack)	195		def(&c.TimeoutSlack, domain.CITimeoutSlack)
187	def(&c.QueueTimeout, domain.CIQueueTimeout)	196		def(&c.QueueTimeout, domain.CIQueueTimeout)
188	def(&c.Retention, domain.CIRetentionDays24time.Hour)	197		def(&c.Retention, domain.CIRetentionDays24time.Hour)
		198	+	def(&c.ArtifactLimits.Retention, domain.CIRetentionDays24time.Hour)
		199	+	if c.ArtifactLimits.Quota <= 0 {
		200	+	c.ArtifactLimits.Quota = domain.DefaultCIArtifactQuota
		201	+	}
		202	+	if c.ArtifactLimits.MaxSize <= 0 {
		203	+	c.ArtifactLimits.MaxSize = domain.DefaultCIArtifactMaxSize
		204	+	}
189	if c.RemoteConcurrency <= 0 {	205		if c.RemoteConcurrency <= 0 {
190	c.RemoteConcurrency = DefaultCIRemoteConcurrency	206		c.RemoteConcurrency = DefaultCIRemoteConcurrency
191	}	207		}

@@ -254,6 +270,10 @@ type CI struct {
254	streams int	270		streams int
255	perClient map[string]int	271		perClient map[string]int
256	closed bool	272		closed bool
		273	+	// artifactLocks serialize the commits of each artifact (ci_artifacts.go).
		274	+	artifactLocks map[int64]*artifactLock
		275	+	// reservations are the artifact bytes being written (ci_artifacts.go).
		276	+	reservations artifactReservations
257	}	277		}
258		278
259	var (	279		var (

A internal/usecase/ci_artifacts.go +1037 -0

@@ -0,0 +1,1037 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package usecase
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"context"
	8	+	"crypto/rand"
	9	+	"crypto/sha256"
	10	+	"encoding/binary"
	11	+	"encoding/hex"
	12	+	"errors"
	13	+	"fmt"
	14	+	"io"
	15	+	"strconv"
	16	+	"sync"
	17	+	"time"
	18	+
	19	+	"github.com/bnema/zerowrap"
	20	+
	21	+	"bnema.dev/ermite/internal/domain"
	22	+	"bnema.dev/ermite/internal/ports"
	23	+	)
	24	+
	25	+	// The artifacts of the runs (artifacts v4, CI-4 D3). The runner translates the
	26	+	// toolkit's protocol and calls JobArtifactService with the job token; owners
	27	+	// use ArtifactService. The records are in CIConfig.Artifacts, the contents in
	28	+	// CIConfig.ArtifactFiles, named by ids only.
	29	+	//
	30	+	// Every write of the artifacts of a repository goes through its usage record
	31	+	// (ports.CIArtifactWrite): a decision is made on the records read after it and
	32	+	// written under its version, or made again. So the quota, the unique names of
	33	+	// a run and the states of an upload hold with concurrent calls, and the bytes
	34	+	// of a write are counted before it is accepted. The files are written before
	35	+	// the record that refers to them and removed after the record that dropped
	36	+	// them: a crash leaves at most files that no record names, which the sweep
	37	+	// removes, never a record without its file.
	38	+
	39	+	var (
	40	+	_ ports.JobArtifactService = (*CI)(nil)
	41	+	_ ports.ArtifactService = (*CI)(nil)
	42	+	)
	43	+
	44	+	const (
	45	+	// artifactTries bounds the retries of a write of the artifacts of a
	46	+	// repository that other writes keep changing.
	47	+	artifactTries = 256
	48	+	// commitTries bounds the assemblies of one commit whose blocks change
	49	+	// meanwhile.
	50	+	commitTries = 5
	51	+	)
	52	+
	53	+	var (
	54	+	// errNoArtifactWrite ends updateArtifacts without a write.
	55	+	errNoArtifactWrite = errors.New("no write")
	56	+	// errArtifactChanged makes a commit start again.
	57	+	errArtifactChanged = errors.New("artifact changed")
	58	+	)
	59	+
	60	+	func (c *CI) artifactsOn() bool { return c.cfg.Artifacts != nil && c.cfg.ArtifactFiles != nil }
	61	+
	62	+	func artifactRef(a domain.CIArtifact) ports.CIArtifactRef {
	63	+	return ports.CIArtifactRef{RunID: a.RunID, ID: a.ID}
	64	+	}
	65	+
	66	+	// artifactJob authenticates the job token of an artifact call: the job of
	67	+	// jobID, which has not ended.
	68	+	func (c *CI) artifactJob(ctx context.Context, secret, jobID string) (domain.CIJob, error) {
	69	+	if !c.artifactsOn() {
	70	+	return domain.CIJob{}, fmt.Errorf("%w: artifacts are off on this server", domain.ErrPreconditionFailed)
	71	+	}
	72	+	j, err := c.jobFor(ctx, secret, jobID)
	73	+	if err != nil {
	74	+	return domain.CIJob{}, err
	75	+	}
	76	+	if j.Ended() {
	77	+	return domain.CIJob{}, endedError(j)
	78	+	}
	79	+	return j, nil
	80	+	}
	81	+
	82	+	// updateArtifacts reads the usage of the repository, lets fn fill the write
	83	+	// from the records it reads then, and applies it; on a conflict it starts
	84	+	// again. fn returns errNoArtifactWrite to write nothing.
	85	+	func (c CI) updateArtifacts(ctx context.Context, repo domain.RepoName, fn func(used int64, w ports.CIArtifactWrite) error) error {
	86	+	for range artifactTries {
	87	+	used, version, err := c.cfg.Artifacts.ArtifactUsage(ctx, repo)
	88	+	if err != nil {
	89	+	return err
	90	+	}
	91	+	w := ports.CIArtifactWrite{Repo: repo, Used: used, UsageVersion: version}
	92	+	if err := fn(used, &w); err != nil {
	93	+	return err
	94	+	}
	95	+	err = c.cfg.Artifacts.ApplyArtifacts(ctx, w)
	96	+	if errors.Is(err, domain.ErrConflict) {
	97	+	continue
	98	+	}
	99	+	return err
	100	+	}
	101	+	return fmt.Errorf("%w: the artifacts of %s keep changing, try again", domain.ErrUnavailable, repo)
	102	+	}
	103	+
	104	+	// --- limits ---
	105	+
	106	+	// sizeText writes a limit for a message: in GiB or MiB when it is a whole
	107	+	// number of them.
	108	+	func sizeText(n int64) string {
	109	+	switch {
	110	+	case n >= 1<<30 && n%(1<<30) == 0:
	111	+	return strconv.FormatInt(n>>30, 10) + " GiB"
	112	+	case n >= 1<<20 && n%(1<<20) == 0:
	113	+	return strconv.FormatInt(n>>20, 10) + " MiB"
	114	+	}
	115	+	return strconv.FormatInt(n, 10) + " bytes"
	116	+	}
	117	+
	118	+	func (c *CI) errArtifactTooLarge() error {
	119	+	return fmt.Errorf("%w: the artifact would exceed the size limit of one artifact (%s)", domain.ErrResourceExhausted, sizeText(c.cfg.ArtifactLimits.MaxSize))
	120	+	}
	121	+
	122	+	func (c *CI) errQuota(repo domain.RepoName) error {
	123	+	return fmt.Errorf("%w: the artifacts of %s would exceed their quota (%s)", domain.ErrResourceExhausted, repo, sizeText(c.cfg.ArtifactLimits.Quota))
	124	+	}
	125	+
	126	+	// --- reservations ---
	127	+
	128	+	// artifactReservations are the bytes being written and not charged yet:
	129	+	// blocks streaming in, the repeats of a commit being assembled. A write
	130	+	// reserves its bytes before they reach the disk, under the lock of its
	131	+	// repository and against the charges stored and the other reservations, and
	132	+	// releases them once its record is written or refused. So concurrent writes
	133	+	// never put more on disk than the limits leave, and the write of the record
	134	+	// checks them again. The size of one artifact and the quota are both limits of
	135	+	// one repository: repositories never wait for one another.
	136	+	type artifactReservations struct {
	137	+	mu sync.Mutex
	138	+	repos map[domain.RepoName]*repoReservations
	139	+	}
	140	+
	141	+	// repoReservations are the reservations of one repository. An entry stays
	142	+	// once made (a few words per repository that ever uploaded); its maps empty.
	143	+	type repoReservations struct {
	144	+	mu sync.Mutex
	145	+	total int64
	146	+	art map[int64]int64
	147	+	}
	148	+
	149	+	func (r artifactReservations) of(repo domain.RepoName) repoReservations {
	150	+	r.mu.Lock()
	151	+	defer r.mu.Unlock()
	152	+	if r.repos == nil {
	153	+	r.repos = map[domain.RepoName]*repoReservations{}
	154	+	}
	155	+	rr := r.repos[repo]
	156	+	if rr == nil {
	157	+	rr = &repoReservations{art: map[int64]int64{}}
	158	+	r.repos[repo] = rr
	159	+	}
	160	+	return rr
	161	+	}
	162	+
	163	+	// reserve takes n bytes for artifact a, or returns the limit it would cross.
	164	+	func (c *CI) reserve(ctx context.Context, a domain.CIArtifact, n int64) error {
	165	+	r := c.reservations.of(a.Repo)
	166	+	r.mu.Lock()
	167	+	defer r.mu.Unlock()
	168	+	used, _, err := c.cfg.Artifacts.ArtifactUsage(ctx, a.Repo)
	169	+	if err != nil {
	170	+	return err
	171	+	}
	172	+	cur, err := c.cfg.Artifacts.GetArtifact(ctx, a.ID)
	173	+	if err != nil {
	174	+	return err
	175	+	}
	176	+	limits := c.cfg.ArtifactLimits
	177	+	switch {
	178	+	case cur.Charged+r.art[a.ID]+n > limits.MaxSize:
	179	+	return c.errArtifactTooLarge()
	180	+	case used+r.total+n > limits.Quota:
	181	+	return c.errQuota(a.Repo)
	182	+	}
	183	+	r.total += n
	184	+	r.art[a.ID] += n
	185	+	return nil
	186	+	}
	187	+
	188	+	// release gives back n bytes reserved for a.
	189	+	func (c *CI) release(a domain.CIArtifact, n int64) {
	190	+	if n == 0 {
	191	+	return
	192	+	}
	193	+	r := c.reservations.of(a.Repo)
	194	+	r.mu.Lock()
	195	+	defer r.mu.Unlock()
	196	+	r.total = max(r.total-n, 0)
	197	+	if r.art[a.ID] -= n; r.art[a.ID] <= 0 {
	198	+	delete(r.art, a.ID)
	199	+	}
	200	+	}
	201	+
	202	+	// reserveAhead is how much a reservingReader reserves at once, so that a
	203	+	// large block takes one reservation per MiB, not one per read.
	204	+	const reserveAhead = domain.CIArtifactChunk
	205	+
	206	+	// reservingReader passes on the data of r, each read covered by a
	207	+	// reservation first: a read that the limits refuse fails without being passed
	208	+	// on, so a block is never written beyond them. It reserves reserveAhead at a
	209	+	// time, or just what a read needs when the limits leave less; reserved is all
	210	+	// it holds (written or not), released whole after the record is written.
	211	+	type reservingReader struct {
	212	+	ctx context.Context
	213	+	c *CI
	214	+	a domain.CIArtifact
	215	+	r io.Reader
	216	+	reserved int64
	217	+	left int64 // reserved and not yet passed on
	218	+	}
	219	+
	220	+	func (rr *reservingReader) Read(p []byte) (int, error) {
	221	+	n, err := rr.r.Read(p)
	222	+	if need := int64(n) - rr.left; need > 0 {
	223	+	got := max(need, reserveAhead)
	224	+	if rerr := rr.c.reserve(rr.ctx, rr.a, got); rerr != nil {
	225	+	if got == need {
	226	+	return 0, rerr
	227	+	}
	228	+	got = need
	229	+	if rerr := rr.c.reserve(rr.ctx, rr.a, got); rerr != nil {
	230	+	return 0, rerr
	231	+	}
	232	+	}
	233	+	rr.reserved += got
	234	+	rr.left += got
	235	+	}
	236	+	rr.left -= int64(n)
	237	+	return n, err
	238	+	}
	239	+
	240	+	// --- locks of the commits ---
	241	+
	242	+	// lockArtifact serializes the commits of an artifact: two commits must not
	243	+	// assemble its content at once. It returns the unlock function.
	244	+	func (c *CI) lockArtifact(id int64) func() {
	245	+	c.mu.Lock()
	246	+	if c.artifactLocks == nil {
	247	+	c.artifactLocks = map[int64]*artifactLock{}
	248	+	}
	249	+	l := c.artifactLocks[id]
	250	+	if l == nil {
	251	+	l = &artifactLock{}
	252	+	c.artifactLocks[id] = l
	253	+	}
	254	+	l.users++
	255	+	c.mu.Unlock()
	256	+	l.Lock()
	257	+	return func() {
	258	+	l.Unlock()
	259	+	c.mu.Lock()
	260	+	if l.users--; l.users == 0 {
	261	+	delete(c.artifactLocks, id)
	262	+	}
	263	+	c.mu.Unlock()
	264	+	}
	265	+	}
	266	+
	267	+	type artifactLock struct {
	268	+	sync.Mutex
	269	+	users int
	270	+	}
	271	+
	272	+	// --- removal ---
	273	+
	274	+	// removeArtifacts deletes the artifacts of a run that pick chooses, judged on
	275	+	// the records as they are when the write is made, refunds them, then removes
	276	+	// their files. It returns the deleted ones.
	277	+	func (c *CI) removeArtifacts(ctx context.Context, repo domain.RepoName, runID int64, pick func(domain.CIArtifact) bool) ([]domain.CIArtifact, error) {
	278	+	var removed []domain.CIArtifact
	279	+	err := c.updateArtifacts(ctx, repo, func(used int64, w *ports.CIArtifactWrite) error {
	280	+	arts, err := c.cfg.Artifacts.RunArtifacts(ctx, repo, runID)
	281	+	if err != nil {
	282	+	return err
	283	+	}
	284	+	removed = removed[:0]
	285	+	for _, a := range arts {
	286	+	if pick(a) {
	287	+	removed = append(removed, a)
	288	+	w.Used -= a.Charged
	289	+	}
	290	+	}
	291	+	if len(removed) == 0 {
	292	+	return errNoArtifactWrite
	293	+	}
	294	+	w.Delete = removed
	295	+	w.Used = max(w.Used, 0)
	296	+	return nil
	297	+	})
	298	+	if errors.Is(err, errNoArtifactWrite) {
	299	+	return nil, nil
	300	+	}
	301	+	if err != nil {
	302	+	return nil, err
	303	+	}
	304	+	c.removeArtifactFiles(ctx, removed)
	305	+	return removed, nil
	306	+	}
	307	+
	308	+	// removeArtifactFiles removes the files of deleted artifacts. A failure is
	309	+	// logged: the sweep removes the files that no record names.
	310	+	func (c *CI) removeArtifactFiles(ctx context.Context, arts []domain.CIArtifact) {
	311	+	for _, a := range arts {
	312	+	if err := c.cfg.ArtifactFiles.Remove(ctx, artifactRef(a)); err != nil {
	313	+	ctxLog(ctx).Warn().Err(err).Int64("artifact_id", a.ID).Msg("ci: remove the files of an artifact; the sweep will")
	314	+	}
	315	+	}
	316	+	}
	317	+
	318	+	// dropIfEnded discards the uploads of the job's assignment when the job ended
	319	+	// or was given again while the call ran: the cleanup of the end may have run
	320	+	// before the call wrote its record.
	321	+	func (c *CI) dropIfEnded(ctx context.Context, j domain.CIJob) {
	322	+	cur, err := c.cfg.Store.GetJob(ctx, j.RunID, j.RunAttempt, j.ID)
	323	+	if err == nil && !cur.Ended() && cur.TokenHash == j.TokenHash {
	324	+	return
	325	+	}
	326	+	c.dropPending(ctx, j, func(a domain.CIArtifact) bool { return a.Assignment == j.TokenHash })
	327	+	}
	328	+
	329	+	// dropPending discards the unfinished uploads of a job, of any of its
	330	+	// assignments, that pick chooses.
	331	+	func (c *CI) dropPending(ctx context.Context, j domain.CIJob, pick func(domain.CIArtifact) bool) {
	332	+	removed, err := c.removeArtifacts(ctx, j.Repo, j.RunID, func(a domain.CIArtifact) bool {
	333	+	return !a.Final() && a.UploadedBy(j) && pick(a)
	334	+	})
	335	+	if err != nil && !errors.Is(err, domain.ErrNotFound) {
	336	+	ctxLog(ctx).Error().Err(err).Str("job_id", j.ID).Msg("ci: discard the unfinished artifacts of a job; the sweep will")
	337	+	return
	338	+	}
	339	+	if len(removed) > 0 {
	340	+	ctxLog(ctx).Info().Str("job_id", j.ID).Int64("run_id", j.RunID).Int("artifacts", len(removed)).Msg("ci: unfinished artifacts discarded")
	341	+	}
	342	+	}
	343	+
	344	+	// artifactsJobEnded discards the unfinished uploads of a job that ended. It is
	345	+	// called after the end is written.
	346	+	func (c *CI) artifactsJobEnded(ctx context.Context, j domain.CIJob) {
	347	+	if !c.artifactsOn() {
	348	+	return
	349	+	}
	350	+	c.dropPending(ctx, j, func(domain.CIArtifact) bool { return true })
	351	+	}
	352	+
	353	+	// --- JobArtifactService ---
	354	+
	355	+	// CreateJobArtifact begins an upload: a pending artifact with a new id. A
	356	+	// pending artifact of the same job and name is discarded.
	357	+	func (c *CI) CreateJobArtifact(ctx context.Context, secret string, params ports.CreateJobArtifactParams) (ports.CreatedArtifact, error) {
	358	+	j, err := c.artifactJob(ctx, secret, params.JobID)
	359	+	if err != nil {
	360	+	return ports.CreatedArtifact{}, err
	361	+	}
	362	+	if err := domain.ParseArtifactName(params.Name); err != nil {
	363	+	return ports.CreatedArtifact{}, err
	364	+	}
	365	+	mediaType, err := domain.ParseArtifactMediaType(params.MediaType)
	366	+	if err != nil {
	367	+	return ports.CreatedArtifact{}, err
	368	+	}
	369	+	now := c.now()
	370	+	id, err := c.cfg.Artifacts.NextArtifactID(ctx)
	371	+	if err != nil {
	372	+	return ports.CreatedArtifact{}, err
	373	+	}
	374	+	a := domain.CIArtifact{
	375	+	ID: id, Repo: j.Repo, RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID, Assignment: j.TokenHash,
	376	+	Name: params.Name, MediaType: mediaType, State: domain.CIArtifactPending,
	377	+	CreatedAt: now, ExpiresAt: domain.ArtifactExpiry(params.ExpiresAt, now, c.cfg.ArtifactLimits.Retention),
	378	+	}
	379	+	var discarded []domain.CIArtifact
	380	+	err = c.updateArtifacts(ctx, j.Repo, func(used int64, w *ports.CIArtifactWrite) error {
	381	+	if used >= c.cfg.ArtifactLimits.Quota {
	382	+	return c.errQuota(j.Repo)
	383	+	}
	384	+	arts, err := c.cfg.Artifacts.RunArtifacts(ctx, j.Repo, j.RunID)
	385	+	if err != nil {
	386	+	return err
	387	+	}
	388	+	discarded = discarded[:0]
	389	+	pending := 0
	390	+	for _, o := range arts {
	391	+	switch {
	392	+	case o.Name != a.Name:
	393	+	if !o.Final() && o.UploadedBy(j) {
	394	+	pending++
	395	+	}
	396	+	case o.Visible(now) && o.Attempt == j.RunAttempt:
	397	+	return fmt.Errorf("%w: an artifact with this name already exists on the workflow run", domain.ErrAlreadyExists)
	398	+	case !o.Final() && o.UploadedBy(j):
	399	+	discarded = append(discarded, o) // a new create is a new upload
	400	+	w.Used -= o.Charged
	401	+	}
	402	+	}
	403	+	switch {
	404	+	case pending >= domain.CIArtifactMaxPendingPerJob:
	405	+	return fmt.Errorf("%w: a job has at most %d unfinished artifact uploads", domain.ErrResourceExhausted, domain.CIArtifactMaxPendingPerJob)
	406	+	case len(arts)-len(discarded) >= domain.CIArtifactMaxPerRun:
	407	+	return fmt.Errorf("%w: a run has at most %d artifacts", domain.ErrResourceExhausted, domain.CIArtifactMaxPerRun)
	408	+	}
	409	+	w.Put, w.Delete = []domain.CIArtifact{a}, discarded
	410	+	return nil
	411	+	})
	412	+	if err != nil {
	413	+	return ports.CreatedArtifact{}, err
	414	+	}
	415	+	c.removeArtifactFiles(ctx, discarded)
	416	+	log := zerowrap.FromCtx(ctx)
	417	+	log.Info().Int64("artifact_id", id).Int64("run_id", j.RunID).Str("job_id", j.ID).Msg("artifact upload begun")
	418	+	c.dropIfEnded(ctx, j)
	419	+	return ports.CreatedArtifact{ID: id, ExpiresAt: a.ExpiresAt}, nil
	420	+	}
	421	+
	422	+	// pendingArtifact returns an unfinished upload of the job's assignment, or
	423	+	// domain.ErrNotFound.
	424	+	func (c *CI) pendingArtifact(ctx context.Context, j domain.CIJob, id int64) (domain.CIArtifact, error) {
	425	+	a, err := c.cfg.Artifacts.GetArtifact(ctx, id)
	426	+	if err != nil {
	427	+	return domain.CIArtifact{}, err
	428	+	}
	429	+	if a.Repo != j.Repo \|\| !a.PendingOf(j) {
	430	+	return domain.CIArtifact{}, fmt.Errorf("%w: artifact %d is not an upload of this job", domain.ErrNotFound, id)
	431	+	}
	432	+	return a, nil
	433	+	}
	434	+
	435	+	func errCommitted(id int64) error {
	436	+	return fmt.Errorf("%w: artifact %d is already committed", domain.ErrPreconditionFailed, id)
	437	+	}
	438	+
	439	+	func newBlockFile(blockID []byte) string {
	440	+	var raw [8]byte
	441	+	_, _ = rand.Read(raw[:])
	442	+	return hex.EncodeToString(blockID) + "." + hex.EncodeToString(raw[:])
	443	+	}
	444	+
	445	+	// UploadJobArtifactBlock stores a block: the data goes to a new file, counted
	446	+	// against the limits as it comes, then the record names it in place of the
	447	+	// block of the same id, if any.
	448	+	func (c *CI) UploadJobArtifactBlock(ctx context.Context, secret, jobID string, artifactID int64, blockID []byte, data io.Reader) (int64, error) {
	449	+	j, err := c.artifactJob(ctx, secret, jobID)
	450	+	if err != nil {
	451	+	return 0, err
	452	+	}
	453	+	if err := domain.CheckArtifactBlockID(blockID); err != nil {
	454	+	return 0, err
	455	+	}
	456	+	a, err := c.pendingArtifact(ctx, j, artifactID)
	457	+	if err != nil {
	458	+	return 0, err
	459	+	}
	460	+	if a.State != domain.CIArtifactPending {
	461	+	return 0, errCommitted(a.ID)
	462	+	}
	463	+	// Every byte is reserved before it is written, against the charges and
	464	+	// the other writes in flight (a block it replaces counts until the record
	465	+	// drops it): nothing beyond the limits ever reaches the disk.
	466	+	src := &reservingReader{ctx: ctx, c: c, a: a, r: data}
	467	+	defer func() { c.release(a, src.reserved) }()
	468	+	limits := c.cfg.ArtifactLimits
	469	+	ref, file := artifactRef(a), newBlockFile(blockID)
	470	+	size, err := c.cfg.ArtifactFiles.PutBlock(ctx, ref, file, src)
	471	+	if err != nil {
	472	+	return 0, err
	473	+	}
	474	+	var old *ports.CIArtifactBlock
	475	+	err = c.updateArtifacts(ctx, a.Repo, func(used int64, w *ports.CIArtifactWrite) error {
	476	+	cur, err := c.pendingArtifact(ctx, j, a.ID)
	477	+	if err != nil {
	478	+	return err
	479	+	}
	480	+	if cur.State != domain.CIArtifactPending {
	481	+	return errCommitted(cur.ID)
	482	+	}
	483	+	blocks, err := c.cfg.Artifacts.ArtifactBlocks(ctx, cur.ID)
	484	+	if err != nil {
	485	+	return err
	486	+	}
	487	+	old = nil
	488	+	for i := range blocks {
	489	+	if bytes.Equal(blocks[i].ID, blockID) {
	490	+	old = &blocks[i]
	491	+	}
	492	+	}
	493	+	var replaced int64
	494	+	if old != nil {
	495	+	replaced = old.Size
	496	+	} else if len(blocks) >= domain.CIArtifactMaxBlocks {
	497	+	return fmt.Errorf("%w: an artifact has at most %d blocks", domain.ErrInvalid, domain.CIArtifactMaxBlocks)
	498	+	}
	499	+	cur.Charged += size - replaced
	500	+	w.Used += size - replaced
	501	+	switch {
	502	+	case cur.Charged > limits.MaxSize:
	503	+	return c.errArtifactTooLarge()
	504	+	case w.Used > limits.Quota:
	505	+	return c.errQuota(cur.Repo)
	506	+	}
	507	+	w.Put = []domain.CIArtifact{cur}
	508	+	w.PutBlocks = []ports.CIArtifactBlock{{ArtifactID: cur.ID, ID: blockID, Size: size, File: file}}
	509	+	return nil
	510	+	})
	511	+	if err != nil {
	512	+	if rerr := c.cfg.ArtifactFiles.RemoveBlock(ctx, ref, file); rerr != nil {
	513	+	ctxLog(ctx).Warn().Err(rerr).Int64("artifact_id", a.ID).Msg("ci: remove a refused block; the sweep will")
	514	+	}
	515	+	return 0, err
	516	+	}
	517	+	if old != nil {
	518	+	if err := c.cfg.ArtifactFiles.RemoveBlock(ctx, ref, old.File); err != nil {
	519	+	ctxLog(ctx).Warn().Err(err).Int64("artifact_id", a.ID).Msg("ci: remove a replaced block")
	520	+	}
	521	+	}
	522	+	c.dropIfEnded(ctx, j)
	523	+	return size, nil
	524	+	}
	525	+
	526	+	// blockListDigest identifies a block list: each id with its length.
	527	+	func blockListDigest(ids [][]byte) []byte {
	528	+	h := sha256.New()
	529	+	var n [2]byte
	530	+	for _, id := range ids {
	531	+	binary.BigEndian.PutUint16(n[:], uint16(len(id))) //nolint:gosec // at most 64
	532	+	h.Write(n[:])
	533	+	h.Write(id)
	534	+	}
	535	+	return h.Sum(nil)
	536	+	}
	537	+
	538	+	// CommitJobArtifact assembles the content of a pending artifact from its
	539	+	// blocks, then records it and drops the blocks. The size counts against the
	540	+	// limits with the repeats of the list.
	541	+	func (c *CI) CommitJobArtifact(ctx context.Context, secret, jobID string, artifactID int64, blockIDs [][]byte) (int64, string, error) {
	542	+	j, err := c.artifactJob(ctx, secret, jobID)
	543	+	if err != nil {
	544	+	return 0, "", err
	545	+	}
	546	+	if len(blockIDs) > domain.CIArtifactMaxBlocks {
	547	+	return 0, "", fmt.Errorf("%w: a commit names at most %d blocks", domain.ErrInvalid, domain.CIArtifactMaxBlocks)
	548	+	}
	549	+	for _, id := range blockIDs {
	550	+	if err := domain.CheckArtifactBlockID(id); err != nil {
	551	+	return 0, "", err
	552	+	}
	553	+	}
	554	+	list := blockListDigest(blockIDs)
	555	+	// A retry of the runner waits here for the commit in progress and gets
	556	+	// its result. The commit goes on when its caller leaves: the runner gives
	557	+	// up on a long assembly and asks again.
	558	+	unlock := c.lockArtifact(artifactID)
	559	+	defer unlock()
	560	+	ctx = context.WithoutCancel(ctx)
	561	+	limits := c.cfg.ArtifactLimits
	562	+	for range commitTries {
	563	+	a, err := c.cfg.Artifacts.GetArtifact(ctx, artifactID)
	564	+	if err != nil {
	565	+	return 0, "", err
	566	+	}
	567	+	if a.Repo != j.Repo \|\| !a.UploadedBy(j) \|\| a.Assignment != j.TokenHash {
	568	+	return 0, "", fmt.Errorf("%w: artifact %d is not an upload of this job", domain.ErrNotFound, artifactID)
	569	+	}
	570	+	if a.State != domain.CIArtifactPending {
	571	+	if bytes.Equal(a.ListDigest, list) {
	572	+	return a.Size, a.Digest, nil // a repeat
	573	+	}
	574	+	return 0, "", fmt.Errorf("%w: artifact %d is committed with another block list", domain.ErrPreconditionFailed, a.ID)
	575	+	}
	576	+	blocks, err := c.cfg.Artifacts.ArtifactBlocks(ctx, a.ID)
	577	+	if err != nil {
	578	+	return 0, "", err
	579	+	}
	580	+	byID := make(map[string]ports.CIArtifactBlock, len(blocks))
	581	+	for _, b := range blocks {
	582	+	byID[string(b.ID)] = b
	583	+	}
	584	+	files := make([]string, 0, len(blockIDs))
	585	+	var total int64
	586	+	for _, id := range blockIDs {
	587	+	b, ok := byID[string(id)]
	588	+	if !ok {
	589	+	return 0, "", fmt.Errorf("%w: block %x was never uploaded", domain.ErrInvalid, id)
	590	+	}
	591	+	files = append(files, b.File)
	592	+	total += b.Size
	593	+	}
	594	+	if total > limits.MaxSize {
	595	+	return 0, "", c.errArtifactTooLarge()
	596	+	}
	597	+	// The repeats of the list make the content larger than the blocks
	598	+	// charged: that excess is reserved before the content is written.
	599	+	excess := max(total-a.Charged, 0)
	600	+	if excess > 0 {
	601	+	if err := c.reserve(ctx, a, excess); err != nil {
	602	+	return 0, "", err
	603	+	}
	604	+	}
	605	+	size, digest, err := c.cfg.ArtifactFiles.Assemble(ctx, artifactRef(a), files)
	606	+	if err != nil {
	607	+	c.release(a, excess)
	608	+	}
	609	+	if errors.Is(err, domain.ErrNotFound) {
	610	+	continue // a block was replaced meanwhile
	611	+	}
	612	+	if err != nil {
	613	+	return 0, "", err
	614	+	}
	615	+	err = c.updateArtifacts(ctx, a.Repo, func(used int64, w *ports.CIArtifactWrite) error {
	616	+	cur, err := c.cfg.Artifacts.GetArtifact(ctx, a.ID)
	617	+	if err != nil {
	618	+	return err
	619	+	}
	620	+	if cur.Version != a.Version {
	621	+	return errArtifactChanged
	622	+	}
	623	+	w.Used += size - cur.Charged
	624	+	if w.Used > limits.Quota {
	625	+	return c.errQuota(cur.Repo)
	626	+	}
	627	+	cur.State, cur.Size, cur.Digest, cur.ListDigest, cur.Charged = domain.CIArtifactCommitted, size, digest, list, size
	628	+	w.Put, w.DropBlocks = []domain.CIArtifact{cur}, []int64{cur.ID}
	629	+	return nil
	630	+	})
	631	+	c.release(a, excess)
	632	+	if errors.Is(err, errArtifactChanged) {
	633	+	continue
	634	+	}
	635	+	if errors.Is(err, domain.ErrNotFound) {
	636	+	c.removeArtifactFiles(ctx, []domain.CIArtifact{a}) // discarded meanwhile
	637	+	}
	638	+	if err != nil {
	639	+	return 0, "", err
	640	+	}
	641	+	if err := c.cfg.ArtifactFiles.RemoveBlocks(ctx, artifactRef(a)); err != nil {
	642	+	ctxLog(ctx).Warn().Err(err).Int64("artifact_id", a.ID).Msg("ci: remove the blocks of a committed artifact")
	643	+	}
	644	+	c.dropIfEnded(ctx, j)
	645	+	return size, digest, nil
	646	+	}
	647	+	return 0, "", fmt.Errorf("%w: the blocks of artifact %d keep changing, try again", domain.ErrUnavailable, artifactID)
	648	+	}
	649	+
	650	+	// FinalizeJobArtifact makes the committed upload of the job with that name
	651	+	// visible. An artifact of that name from an earlier attempt is deleted in the
	652	+	// same write; one finalized in this attempt is already_exists.
	653	+	func (c *CI) FinalizeJobArtifact(ctx context.Context, secret, jobID, name string, size int64, digest string) (domain.CIArtifact, error) {
	654	+	j, err := c.artifactJob(ctx, secret, jobID)
	655	+	if err != nil {
	656	+	return domain.CIArtifact{}, err
	657	+	}
	658	+	if err := domain.ParseArtifactName(name); err != nil {
	659	+	return domain.CIArtifact{}, err
	660	+	}
	661	+	digest, err = domain.ParseArtifactDigest(digest)
	662	+	if err != nil {
	663	+	return domain.CIArtifact{}, err
	664	+	}
	665	+	now := c.now()
	666	+	var out domain.CIArtifact
	667	+	var replaced []domain.CIArtifact
	668	+	err = c.updateArtifacts(ctx, j.Repo, func(used int64, w *ports.CIArtifactWrite) error {
	669	+	arts, err := c.cfg.Artifacts.RunArtifacts(ctx, j.Repo, j.RunID)
	670	+	if err != nil {
	671	+	return err
	672	+	}
	673	+	var mine, done *domain.CIArtifact
	674	+	var finals []domain.CIArtifact
	675	+	for i, a := range arts {
	676	+	switch {
	677	+	case a.Name != name:
	678	+	case a.PendingOf(j):
	679	+	if mine == nil \|\| a.ID > mine.ID {
	680	+	mine = &arts[i]
	681	+	}
	682	+	case a.Final():
	683	+	finals = append(finals, a)
	684	+	if a.UploadedBy(j) && a.Visible(now) {
	685	+	done = &arts[i]
	686	+	}
	687	+	}
	688	+	}
	689	+	switch {
	690	+	case mine == nil && done != nil:
	691	+	out = *done // a repeat
	692	+	return errNoArtifactWrite
	693	+	case mine == nil:
	694	+	return fmt.Errorf("%w: this job has no upload of an artifact named %q", domain.ErrNotFound, name)
	695	+	case mine.State != domain.CIArtifactCommitted:
	696	+	return fmt.Errorf("%w: artifact %q is not committed", domain.ErrPreconditionFailed, name)
	697	+	case digest != "" && digest != mine.Digest:
	698	+	return fmt.Errorf("%w: the digest %s is not the digest of the content (%s)", domain.ErrInvalid, digest, mine.Digest)
	699	+	}
	700	+	replaced = replaced[:0]
	701	+	for _, f := range finals {
	702	+	if f.Visible(now) && f.Attempt == j.RunAttempt {
	703	+	return fmt.Errorf("%w: an artifact with this name already exists on the workflow run", domain.ErrAlreadyExists)
	704	+	}
	705	+	replaced = append(replaced, f) // an earlier attempt's, or expired
	706	+	w.Used -= f.Charged
	707	+	}
	708	+	out = *mine
	709	+	out.State = domain.CIArtifactFinal
	710	+	w.Put, w.Delete = []domain.CIArtifact{out}, replaced
	711	+	w.Used = max(w.Used, 0)
	712	+	return nil
	713	+	})
	714	+	if errors.Is(err, errNoArtifactWrite) {
	715	+	return out, nil
	716	+	}
	717	+	if err != nil {
	718	+	return domain.CIArtifact{}, err
	719	+	}
	720	+	c.removeArtifactFiles(ctx, replaced)
	721	+	log := zerowrap.FromCtx(ctx)
	722	+	ev := log.Info()
	723	+	if size != out.Size {
	724	+	ev = ev.Int64("uploader_size", size)
	725	+	}
	726	+	ev.Int64("artifact_id", out.ID).Int64("run_id", out.RunID).Str("job_id", j.ID).Int64("size", out.Size).Int("replaced", len(replaced)).Msg("artifact finalized")
	727	+	return out, nil
	728	+	}
	729	+
	730	+	// ListJobArtifacts lists the visible artifacts of the job's run.
	731	+	func (c *CI) ListJobArtifacts(ctx context.Context, secret, jobID, name string, artifactID int64) ([]domain.CIArtifact, error) {
	732	+	j, err := c.artifactJob(ctx, secret, jobID)
	733	+	if err != nil {
	734	+	return nil, err
	735	+	}
	736	+	arts, err := c.visibleArtifacts(ctx, j.Repo, j.RunID)
	737	+	if err != nil {
	738	+	return nil, err
	739	+	}
	740	+	out := arts[:0]
	741	+	for _, a := range arts {
	742	+	if (name == "" \|\| a.Name == name) && (artifactID == 0 \|\| a.ID == artifactID) {
	743	+	out = append(out, a)
	744	+	}
	745	+	}
	746	+	return out, nil
	747	+	}
	748	+
	749	+	func (c *CI) visibleArtifacts(ctx context.Context, repo domain.RepoName, runID int64) ([]domain.CIArtifact, error) {
	750	+	arts, err := c.cfg.Artifacts.RunArtifacts(ctx, repo, runID)
	751	+	if err != nil {
	752	+	return nil, err
	753	+	}
	754	+	now := c.now()
	755	+	out := make([]domain.CIArtifact, 0, len(arts))
	756	+	for _, a := range arts {
	757	+	if a.Visible(now) {
	758	+	out = append(out, a)
	759	+	}
	760	+	}
	761	+	return out, nil
	762	+	}
	763	+
	764	+	// DownloadJobArtifact streams a visible artifact of the job's run.
	765	+	func (c *CI) DownloadJobArtifact(ctx context.Context, secret, jobID string, artifactID, offset int64, send func(ports.ArtifactChunk) error) error {
	766	+	j, err := c.artifactJob(ctx, secret, jobID)
	767	+	if err != nil {
	768	+	return err
	769	+	}
	770	+	a, err := c.cfg.Artifacts.GetArtifact(ctx, artifactID)
	771	+	if err != nil {
	772	+	return err
	773	+	}
	774	+	if a.Repo != j.Repo \|\| a.RunID != j.RunID \|\| !a.Visible(c.now()) {
	775	+	return fmt.Errorf("%w: artifact %d is not an artifact of this run", domain.ErrNotFound, artifactID)
	776	+	}
	777	+	return c.streamArtifact(ctx, a, offset, send)
	778	+	}
	779	+
	780	+	// streamArtifact sends the artifact, then its content from offset.
	781	+	func (c *CI) streamArtifact(ctx context.Context, a domain.CIArtifact, offset int64, send func(ports.ArtifactChunk) error) error {
	782	+	if offset < 0 \|\| offset > a.Size {
	783	+	return fmt.Errorf("%w: offset %d is beyond the size %d", domain.ErrOutOfRange, offset, a.Size)
	784	+	}
	785	+	f, err := c.cfg.ArtifactFiles.Open(ctx, artifactRef(a), offset)
	786	+	if err != nil {
	787	+	return err
	788	+	}
	789	+	defer func() { _ = f.Close() }()
	790	+	if err := send(ports.ArtifactChunk{Artifact: &a}); err != nil {
	791	+	return err
	792	+	}
	793	+	left := a.Size - offset
	794	+	buf := make([]byte, min(left, domain.CIArtifactChunk))
	795	+	for left > 0 {
	796	+	n, err := io.ReadFull(f, buf[:min(left, int64(len(buf)))])
	797	+	if err != nil {
	798	+	return fmt.Errorf("read artifact %d: %w", a.ID, err)
	799	+	}
	800	+	left -= int64(n)
	801	+	if err := send(ports.ArtifactChunk{Data: buf[:n]}); err != nil {
	802	+	return err
	803	+	}
	804	+	}
	805	+	return nil
	806	+	}
	807	+
	808	+	// DeleteJobArtifact deletes the visible artifact of that name of the job's
	809	+	// run.
	810	+	func (c *CI) DeleteJobArtifact(ctx context.Context, secret, jobID, name string) (int64, error) {
	811	+	j, err := c.artifactJob(ctx, secret, jobID)
	812	+	if err != nil {
	813	+	return 0, err
	814	+	}
	815	+	if err := domain.ParseArtifactName(name); err != nil {
	816	+	return 0, err
	817	+	}
	818	+	now := c.now()
	819	+	removed, err := c.removeArtifacts(ctx, j.Repo, j.RunID, func(a domain.CIArtifact) bool {
	820	+	return a.Name == name && a.Visible(now)
	821	+	})
	822	+	if err != nil {
	823	+	return 0, err
	824	+	}
	825	+	if len(removed) == 0 {
	826	+	return 0, fmt.Errorf("%w: no artifact named %q on this run", domain.ErrNotFound, name)
	827	+	}
	828	+	log := zerowrap.FromCtx(ctx)
	829	+	log.Info().Int64("artifact_id", removed[0].ID).Int64("run_id", j.RunID).Str("job_id", j.ID).Msg("artifact deleted by a job")
	830	+	return removed[0].ID, nil
	831	+	}
	832	+
	833	+	// --- ArtifactService ---
	834	+
	835	+	// artifactRepo is the access rule of reading artifacts: a signed-in caller
	836	+	// with repo:read, even on a public repository (GitHub serves no artifact
	837	+	// anonymously).
	838	+	func (c *CI) artifactRepo(ctx context.Context, p domain.Principal, name domain.RepoName) (domain.Repo, error) {
	839	+	if !c.artifactsOn() {
	840	+	return domain.Repo{}, fmt.Errorf("%w: artifacts are off on this server", domain.ErrPreconditionFailed)
	841	+	}
	842	+	repo, err := c.readableRepo(ctx, p, name)
	843	+	if err != nil {
	844	+	return domain.Repo{}, err
	845	+	}
	846	+	if err := scopeAccess(p, repo, domain.ScopeRepoRead); err != nil {
	847	+	return domain.Repo{}, err
	848	+	}
	849	+	return repo, nil
	850	+	}
	851	+
	852	+	// ListArtifacts lists the visible artifacts of a run with the space of the
	853	+	// repository.
	854	+	func (c *CI) ListArtifacts(ctx context.Context, p domain.Principal, name domain.RepoName, runID int64) (ports.ArtifactList, error) {
	855	+	if _, err := c.artifactRepo(ctx, p, name); err != nil {
	856	+	return ports.ArtifactList{}, err
	857	+	}
	858	+	if _, err := c.runOf(ctx, name, runID); err != nil {
	859	+	return ports.ArtifactList{}, err
	860	+	}
	861	+	arts, err := c.visibleArtifacts(ctx, name, runID)
	862	+	if err != nil {
	863	+	return ports.ArtifactList{}, err
	864	+	}
	865	+	used, _, err := c.cfg.Artifacts.ArtifactUsage(ctx, name)
	866	+	if err != nil {
	867	+	return ports.ArtifactList{}, err
	868	+	}
	869	+	return ports.ArtifactList{Artifacts: arts, Used: used, Quota: c.cfg.ArtifactLimits.Quota}, nil
	870	+	}
	871	+
	872	+	// ownerArtifact returns a visible artifact of the repository.
	873	+	func (c *CI) ownerArtifact(ctx context.Context, name domain.RepoName, id int64) (domain.CIArtifact, error) {
	874	+	a, err := c.cfg.Artifacts.GetArtifact(ctx, id)
	875	+	if err != nil {
	876	+	return domain.CIArtifact{}, err
	877	+	}
	878	+	if a.Repo != name \|\| !a.Visible(c.now()) {
	879	+	return domain.CIArtifact{}, fmt.Errorf("%w: artifact %d is not an artifact of %s", domain.ErrNotFound, id, name)
	880	+	}
	881	+	return a, nil
	882	+	}
	883	+
	884	+	// DownloadArtifact streams a visible artifact of the repository.
	885	+	func (c *CI) DownloadArtifact(ctx context.Context, p domain.Principal, name domain.RepoName, artifactID, offset int64, send func(ports.ArtifactChunk) error) error {
	886	+	if _, err := c.artifactRepo(ctx, p, name); err != nil {
	887	+	return err
	888	+	}
	889	+	a, err := c.ownerArtifact(ctx, name, artifactID)
	890	+	if err != nil {
	891	+	return err
	892	+	}
	893	+	return c.streamArtifact(ctx, a, offset, send)
	894	+	}
	895	+
	896	+	// DeleteArtifact deletes an artifact. It needs ci:write on the repository and,
	897	+	// like the other writes on a run, the branch of the run within the prefixes of
	898	+	// a limited token.
	899	+	func (c *CI) DeleteArtifact(ctx context.Context, p domain.Principal, name domain.RepoName, artifactID int64) error {
	900	+	if !c.artifactsOn() {
	901	+	return fmt.Errorf("%w: artifacts are off on this server", domain.ErrPreconditionFailed)
	902	+	}
	903	+	if _, err := c.writableRepo(ctx, p, name); err != nil {
	904	+	return err
	905	+	}
	906	+	a, err := c.ownerArtifact(ctx, name, artifactID)
	907	+	if err != nil {
	908	+	return err
	909	+	}
	910	+	run, err := c.runOf(ctx, name, a.RunID)
	911	+	if err != nil {
	912	+	return err
	913	+	}
	914	+	if err := runWriteAccess(p, run); err != nil {
	915	+	return err
	916	+	}
	917	+	removed, err := c.removeArtifacts(ctx, name, a.RunID, func(o domain.CIArtifact) bool { return o.ID == a.ID && o.Final() })
	918	+	if err != nil {
	919	+	return err
	920	+	}
	921	+	if len(removed) == 0 {
	922	+	return fmt.Errorf("%w: artifact %d is not an artifact of %s", domain.ErrNotFound, artifactID, name)
	923	+	}
	924	+	log := zerowrap.FromCtx(ctx)
	925	+	log.Info().Int64("artifact_id", a.ID).Int64("run_id", a.RunID).Str("repo", name.String()).
	926	+	Str("principal", p.Label()).Str("token_id", p.TokenID()).Msg("artifact deleted")
	927	+	return nil
	928	+	}
	929	+
	930	+	// --- sweep ---
	931	+
	932	+	// sweepArtifacts deletes the expired artifacts, those of runs that no longer
	933	+	// exist, and the unfinished uploads of jobs that ended, were given again or
	934	+	// began more than domain.CIArtifactPendingMax ago; then the blocks and files
	935	+	// that no record names (a crash between a file and its record, a repository
	936	+	// deleted while the server was down). Every failure is logged and the sweep
	937	+	// goes on.
	938	+	func (c *CI) sweepArtifacts(ctx context.Context) {
	939	+	if !c.artifactsOn() {
	940	+	return
	941	+	}
	942	+	log := ctxLog(ctx)
	943	+	all, err := c.cfg.Artifacts.AllArtifacts(ctx)
	944	+	if err != nil {
	945	+	log.Error().Err(err).Msg("ci: list the artifacts")
	946	+	return
	947	+	}
	948	+	type runKey struct {
	949	+	repo domain.RepoName
	950	+	run int64
	951	+	}
	952	+	runs := map[runKey]bool{}
	953	+	for _, a := range all {
	954	+	runs[runKey{a.Repo, a.RunID}] = true
	955	+	}
	956	+	removed := 0
	957	+	for k := range runs {
	958	+	gone := false
	959	+	if _, err := c.cfg.Store.GetRunSource(ctx, k.run); errors.Is(err, domain.ErrNotFound) {
	960	+	gone = true
	961	+	}
	962	+	now := c.now()
	963	+	arts, err := c.removeArtifacts(ctx, k.repo, k.run, func(a domain.CIArtifact) bool {
	964	+	return gone \|\| c.artifactDead(ctx, a, now)
	965	+	})
	966	+	if err != nil && !errors.Is(err, domain.ErrNotFound) {
	967	+	log.Error().Err(err).Int64("run_id", k.run).Msg("ci: remove the expired artifacts of a run")
	968	+	}
	969	+	removed += len(arts)
	970	+	}
	971	+	if removed > 0 {
	972	+	log.Info().Int("artifacts", removed).Msg("ci: expired artifacts removed")
	973	+	}
	974	+	c.sweepArtifactOrphans(ctx)
	975	+	}
	976	+
	977	+	// artifactDead reports whether the sweep deletes an artifact of a run that
	978	+	// exists.
	979	+	func (c *CI) artifactDead(ctx context.Context, a domain.CIArtifact, now time.Time) bool {
	980	+	if a.Final() {
	981	+	return !now.Before(a.ExpiresAt)
	982	+	}
	983	+	if now.Sub(a.CreatedAt) >= domain.CIArtifactPendingMax {
	984	+	return true
	985	+	}
	986	+	j, err := c.cfg.Store.GetJob(ctx, a.RunID, a.Attempt, a.JobID)
	987	+	switch {
	988	+	case errors.Is(err, domain.ErrNotFound):
	989	+	return true
	990	+	case err != nil:
	991	+	return false
	992	+	}
	993	+	return j.Ended() \|\| j.TokenHash != a.Assignment
	994	+	}
	995	+
	996	+	// sweepArtifactOrphans removes the blocks and files that no record names.
	997	+	func (c *CI) sweepArtifactOrphans(ctx context.Context) {
	998	+	log := ctxLog(ctx)
	999	+	if ids, err := c.cfg.Artifacts.BlockArtifactIDs(ctx); err != nil {
	1000	+	log.Error().Err(err).Msg("ci: list the staged artifact blocks")
	1001	+	} else {
	1002	+	for _, id := range ids {
	1003	+	if _, err := c.cfg.Artifacts.GetArtifact(ctx, id); !errors.Is(err, domain.ErrNotFound) {
	1004	+	continue
	1005	+	}
	1006	+	if err := c.cfg.Artifacts.DeleteBlocks(ctx, id); err != nil {
	1007	+	log.Error().Err(err).Int64("artifact_id", id).Msg("ci: remove the blocks of an artifact that no longer exists")
	1008	+	}
	1009	+	}
	1010	+	}
	1011	+	c.removeOrphans(ctx, "artifacts", "artifact directories", c.cfg.ArtifactFiles.RunIDs, c.cfg.ArtifactFiles.RemoveRuns)
	1012	+	runIDs, err := c.cfg.ArtifactFiles.RunIDs(ctx)
	1013	+	if err != nil {
	1014	+	log.Error().Err(err).Msg("ci: list the artifact directories")
	1015	+	return
	1016	+	}
	1017	+	for _, run := range runIDs {
	1018	+	ids, err := c.cfg.ArtifactFiles.ArtifactIDs(ctx, run)
	1019	+	if err != nil {
	1020	+	log.Error().Err(err).Int64("run_id", run).Msg("ci: list the artifact files of a run")
	1021	+	continue
	1022	+	}
	1023	+	for _, id := range ids {
	1024	+	a, err := c.cfg.Artifacts.GetArtifact(ctx, id)
	1025	+	if err == nil && a.RunID == run {
	1026	+	continue
	1027	+	}
	1028	+	if err != nil && !errors.Is(err, domain.ErrNotFound) {
	1029	+	continue
	1030	+	}
	1031	+	ref := ports.CIArtifactRef{RunID: run, ID: id}
	1032	+	if err := c.cfg.ArtifactFiles.Remove(ctx, ref); err != nil {
	1033	+	log.Error().Err(err).Int64("artifact_id", id).Msg("ci: remove the files of an artifact that no longer exists")
	1034	+	}
	1035	+	}
	1036	+	}
	1037	+	}

A internal/usecase/ci_artifacts_reserve_test.go +89 -0

@@ -0,0 +1,89 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package usecase
	4	+
	5	+	import (
	6	+	"bytes"
	7	+	"io"
	8	+	"testing"
	9	+	"testing/iotest"
	10	+
	11	+	"github.com/stretchr/testify/assert"
	12	+	"github.com/stretchr/testify/mock"
	13	+	"github.com/stretchr/testify/require"
	14	+
	15	+	"bnema.dev/ermite/internal/domain"
	16	+	"bnema.dev/ermite/internal/ports"
	17	+	"bnema.dev/ermite/internal/ports/mocks"
	18	+	)
	19	+
	20	+	// smallReads hands out at most 32 KiB per read, like io.Copy's buffer.
	21	+	type smallReads struct{ r io.Reader }
	22	+
	23	+	func (s smallReads) Read(p []byte) (int, error) { return s.r.Read(p[:min(len(p), 32<<10)]) }
	24	+
	25	+	func reserveFixture(t testing.T, limits ports.ArtifactLimits) (CI, *mocks.CIArtifactStore, domain.CIArtifact) {
	26	+	t.Helper()
	27	+	repo, err := domain.ParseRepoName("proj")
	28	+	require.NoError(t, err)
	29	+	a := domain.CIArtifact{ID: 7, Repo: repo, RunID: 1, State: domain.CIArtifactPending}
	30	+	store := mocks.NewCIArtifactStore(t)
	31	+	store.EXPECT().ArtifactUsage(mock.Anything, repo).Return(0, 1, nil).Maybe()
	32	+	store.EXPECT().GetArtifact(mock.Anything, a.ID).Return(a, nil).Maybe()
	33	+	return &CI{cfg: CIConfig{Artifacts: store, ArtifactLimits: limits}}, store, a
	34	+	}
	35	+
	36	+	// A large block takes one reservation per MiB, not one per read.
	37	+	func TestReservationsPerMiB(t *testing.T) {
	38	+	c, store, a := reserveFixture(t, ports.ArtifactLimits{Quota: 1 << 30, MaxSize: 1 << 30})
	39	+	rr := &reservingReader{ctx: t.Context(), c: c, a: a, r: smallReads{bytes.NewReader(make([]byte, 8<<20))}}
	40	+	n, err := io.Copy(io.Discard, rr)
	41	+	require.NoError(t, err)
	42	+	assert.EqualValues(t, 8<<20, n)
	43	+	calls := 0
	44	+	for _, call := range store.Calls {
	45	+	if call.Method == "ArtifactUsage" {
	46	+	calls++
	47	+	}
	48	+	}
	49	+	assert.LessOrEqual(t, calls, 8, "one reservation per MiB for 256 reads")
	50	+	assert.EqualValues(t, 8<<20, rr.reserved, "never more reserved than needed at the end of a whole MiB")
	51	+
	52	+	c.release(a, rr.reserved)
	53	+	assert.Zero(t, c.reservations.of(a.Repo).total, "all given back")
	54	+	}
	55	+
	56	+	// Near the limit a read reserves just what it needs: an upload that fits is
	57	+	// never refused because of the reservation ahead.
	58	+	func TestReservationsNearTheLimit(t *testing.T) {
	59	+	const limit = 1<<20 + 100
	60	+	c, _, a := reserveFixture(t, ports.ArtifactLimits{Quota: 1 << 30, MaxSize: limit})
	61	+	rr := &reservingReader{ctx: t.Context(), c: c, a: a, r: iotest.HalfReader(bytes.NewReader(make([]byte, limit)))}
	62	+	_, err := io.Copy(io.Discard, rr)
	63	+	require.NoError(t, err)
	64	+	assert.LessOrEqual(t, rr.reserved, int64(limit))
	65	+
	66	+	// One byte more than the limit: the read that crosses it fails and is not
	67	+	// passed on.
	68	+	c.release(a, rr.reserved)
	69	+	rr = &reservingReader{ctx: t.Context(), c: c, a: a, r: bytes.NewReader(make([]byte, limit+1))}
	70	+	n, err := io.Copy(io.Discard, rr)
	71	+	require.ErrorIs(t, err, domain.ErrResourceExhausted)
	72	+	assert.LessOrEqual(t, n, int64(limit))
	73	+	}
	74	+
	75	+	// Repositories do not share a lock or a quota.
	76	+	func TestReservationsArePerRepository(t *testing.T) {
	77	+	c, _, a := reserveFixture(t, ports.ArtifactLimits{Quota: 1 << 20, MaxSize: 1 << 20})
	78	+	require.NoError(t, c.reserve(t.Context(), a, 1<<20))
	79	+	assert.ErrorIs(t, c.reserve(t.Context(), a, 1), domain.ErrResourceExhausted)
	80	+
	81	+	other, err := domain.ParseRepoName("other")
	82	+	require.NoError(t, err)
	83	+	b := domain.CIArtifact{ID: 8, Repo: other, RunID: 2, State: domain.CIArtifactPending}
	84	+	store := c.cfg.Artifacts.(*mocks.CIArtifactStore)
	85	+	store.EXPECT().ArtifactUsage(mock.Anything, other).Return(0, 1, nil)
	86	+	store.EXPECT().GetArtifact(mock.Anything, b.ID).Return(b, nil)
	87	+	require.NoError(t, c.reserve(t.Context(), b, 1<<20), "the quota of another repository is its own")
	88	+	assert.NotSame(t, c.reservations.of(a.Repo), c.reservations.of(b.Repo))
	89	+	}

M internal/usecase/ci_concurrency.go +1 -0


@@ -679,6 +679,7 @@ func (rs *runState) releaseEnded(ctx context.Context, changed []int, runDone boo
679	if !j.Ended() {	679		if !j.Ended() {
680	continue	680		continue
681	}	681		}
		682	+	c.artifactsJobEnded(ctx, j)
682	e := domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID}	683		e := domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID}
683	groups := []string{}	684		groups := []string{}
684	if j.Concurrency.Group != "" && !j.Concurrency.FromRun {	685		if j.Concurrency.Group != "" && !j.Concurrency.FromRun {

M internal/usecase/ci_watch.go +10 -4

 }
 // sweep removes the runs older than the retention, and the logs of runs that
-// no longer exist (a repository deleted while the server was down). Every
+// no longer exist (a repository deleted while the server was down), then
+// sweeps the artifacts (expired, of removed runs, unfinished). Every
 // failure is logged and the next step still runs: a log directory that cannot
 // be removed is tried again at the next sweep.
 func (c *CI) sweep(ctx context.Context) {
 	}
 	c.removeOrphans(ctx, "logs", "log directories", c.cfg.Logs.RunIDs, c.cfg.Logs.RemoveRuns)
 	c.removeOrphans(ctx, "summaries", "summary directories", c.cfg.Summaries.RunIDs, c.cfg.Summaries.RemoveRuns)
+	c.sweepArtifacts(ctx)
 }
 // removeOrphans deletes what a file store (logs, summaries) holds for runs that
 	c.sched.forgetDirty(repo)
 }
-// RemoveLogs deletes the logs and the summaries of runs of a repository that
+// RemoveLogs deletes the logs, the summaries and the artifact files of runs of
-// was deleted.
+// a repository that was deleted (its artifact records went with it).
 func (c *CI) RemoveLogs(ctx context.Context, runIDs []int64) error {
 	for _, id := range runIDs {
 		c.locks.Delete(id)
 	}
-	return errors.Join(c.cfg.Logs.RemoveRuns(ctx, runIDs), c.cfg.Summaries.RemoveRuns(ctx, runIDs))
+	err := errors.Join(c.cfg.Logs.RemoveRuns(ctx, runIDs), c.cfg.Summaries.RemoveRuns(ctx, runIDs))
+	if c.artifactsOn() {
+		err = errors.Join(err, c.cfg.ArtifactFiles.RemoveRuns(ctx, runIDs))
+	}
+	return err
 }

A proto/ermite/storage/ci_artifact.proto +69 -0

@@ -0,0 +1,69 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	// Internal, unversioned storage schema of the CI artifacts (artifacts v4,
	4	+	// CI-4 D3). The contents are not stored here: they are files under
	5	+	// $ERMITE_DATA_DIR/ci/artifacts.
	6	+
	7	+	syntax = "proto3";
	8	+
	9	+	package ermite.storage;
	10	+
	11	+	import "google/protobuf/timestamp.proto";
	12	+
	13	+	option go_package = "bnema.dev/ermite/gen/ermite/storage;storagepb";
	14	+
	15	+	// CIArtifactState is where an artifact is in its upload.
	16	+	enum CIArtifactState {
	17	+	CI_ARTIFACT_STATE_UNSPECIFIED = 0;
	18	+	// Created; its blocks are being uploaded.
	19	+	CI_ARTIFACT_STATE_PENDING = 1;
	20	+	// Its content is assembled from the blocks; not visible yet.
	21	+	CI_ARTIFACT_STATE_COMMITTED = 2;
	22	+	// Finalized: visible to the run and its owners.
	23	+	CI_ARTIFACT_STATE_FINAL = 3;
	24	+	}
	25	+
	26	+	// CIArtifact is stored under "artifact/<repo>/<run:020d>/<id:020d>", with the
	27	+	// index "artifactid/<id:020d>" (value "<repo>/<run:020d>").
	28	+	message CIArtifact {
	29	+	int64 id = 1;
	30	+	string repo = 2;
	31	+	int64 run_id = 3;
	32	+	// attempt: the attempt of the run whose job uploads it.
	33	+	int32 attempt = 4;
	34	+	string job_id = 5;
	35	+	// assignment: the hash of the job token of the assignment that uploads it;
	36	+	// only that assignment may change a pending artifact.
	37	+	bytes assignment = 6;
	38	+	string name = 7;
	39	+	string media_type = 8;
	40	+	int64 size = 9;
	41	+	// digest: "sha256:<hex>", set from the commit.
	42	+	string digest = 10;
	43	+	CIArtifactState state = 11;
	44	+	google.protobuf.Timestamp created_at = 12;
	45	+	google.protobuf.Timestamp expires_at = 13;
	46	+	// charged: the bytes counted against the quota of the repository (the
	47	+	// staged blocks while pending, the content once committed).
	48	+	int64 charged = 14;
	49	+	// list_digest: the SHA-256 of the block list of the commit, to recognize a
	50	+	// repeated commit.
	51	+	bytes list_digest = 15;
	52	+	}
	53	+
	54	+	// CIArtifactBlock is stored under "artifactblk/<artifact:020d>/<hex(block id)>":
	55	+	// a staged block of a pending artifact.
	56	+	message CIArtifactBlock {
	57	+	int64 size = 1;
	58	+	// file: the name of the block's file in the staging directory of the
	59	+	// artifact (hexadecimal characters and a dot only).
	60	+	string file = 2;
	61	+	}
	62	+
	63	+	// CIArtifactUsage is stored under "artifactuse/<repo>": the bytes charged to
	64	+	// the artifacts of the repository, pending uploads included. Every write of
	65	+	// the artifacts of the repository rewrites it under its version, so that two
	66	+	// writes never both decide on the same state.
	67	+	message CIArtifactUsage {
	68	+	int64 used = 1;
	69	+	}