e4fb10d feat(ci): create/delete triggers, concurrency queue, schedule timezone (#42) · ermite

M README.md +6 -4


@@ -115,7 +115,7 @@ A merge request proposes a branch for its target (the default branch unless set)
115	115
116	116		## CI
117	117
118		-	ermite runs the `.github/workflows/*.yml` workflows of a repository on `push` (a branch or a tag, and the merge of a merge request), on `pull_request` and `pull_request_target` (the merge requests, below), `workflow_dispatch`, `schedule` and `workflow_run` (below) and shows each job as a check on the commit, the branch list and the merge request. CI is on once a runner is registered; there is no switch. Before that a push makes no run and no check.
	118	+	ermite runs the `.github/workflows/*.yml` workflows of a repository on `push` (a branch or a tag, and the merge of a merge request), on `pull_request` and `pull_request_target` (the merge requests, below), `workflow_dispatch`, `schedule`, `workflow_run`, `create` and `delete` (below) and shows each job as a check on the commit, the branch list and the merge request. CI is on once a runner is registered; there is no switch. Before that a push makes no run and no check.
119	119
120	120		Runs execute on [ermite-runner](https://git.bnema.dev/bnema/ermite-runner) hosts, which call ermite over its API. Create a token for the runner host with `ermite token create <label> --preset runner` (the `runner:register` scope, which also gives access to the code and the secrets of every repository: give it only to runners you operate), then register the runner with it. `ermite runner list` shows the runners and whether they are online; `ermite runner remove <id>` stops one. A job goes to a runner that has every label of its `runs-on`; a job no runner takes within 24 hours fails.
121	121

@@ -131,17 +131,19 @@ An action of `ERMITE_OWNER/<repo>` that exists on this server wins over the same
131	131
132	132		`workflow_dispatch`. `ermite workflow run <file> --repo <repo> [--ref <branch or tag>] [-f name=value …]` starts a run of a workflow that declares `workflow_dispatch` (the API is `RunService.DispatchWorkflow`, feature `ci.dispatch`; it needs `ci:write`). The file is its path or its name under `.github/workflows`, and it is read at the ref (the default branch when `--ref` is left out). The inputs are checked against the workflow: an undeclared input, a missing `required` input without a default, a `boolean` other than `true` or `false`, a `number` that is not a decimal number, a `choice` outside its `options`, and an `environment` that is not a valid environment name are refused; a `choice` without default takes its first option. A workflow has at most 25 inputs and their values at most 65,535 characters. In the run, `github.event.inputs` holds the values as strings and the `inputs` context as their declared types (`boolean`, `number`, else string). The actor is the caller. The run is trusted (secrets, write token) only when both the caller is the owner and the ref it runs on was moved to its current commit by the owner (see `schedule`): the owner dispatching an agent's branch gets an untrusted run, like the owner re-running an agent's run, and a ref with no record (pushed before ermite recorded pushers) is untrusted too. This holds for a tag as for a branch, and for a job token as for the owner. A token limited to branch prefixes can dispatch only on a branch under them and its run is never trusted. A job can dispatch with its repository token when its `permissions:` give `actions: write` (GitHub's exception to "events made with `GITHUB_TOKEN` start no run"), even without `contents: read`; a token that does not hold the permission is refused.
133	133
134		-	`schedule`. `on.schedule[].cron` (five fields, in UTC; `timezone` is ignored) is read on the default branch only. Every minute the scheduler starts the workflows whose cron matches that minute, on the latest commit of the default branch; the payload is `{"schedule": "<cron>"}`. A minute is handled once: ticks missed while ermite was down or busy are not made up, and a workflow with several matching entries starts one run. The actor is whoever last moved the default branch to its current commit (a push or a merge), and the run is trusted only if that was the owner. ermite records who moves each branch and tag before it moves (in the pre-receive decision of a push, and in the merge under its lock; a record that cannot be written refuses the push), together with the commit it is for: the record counts only while the ref still points to that commit, so a late write cannot leave the owner's record on a tip an agent moved there (owner A, agent B, agent A). A tip with no record, a record for another commit (a push refused after its record was written) or an unreadable record runs untrusted, without secrets. If the owner and a token push the very same commit at the same time, the owner's record may remain: the content is the same and the owner pushed it. At most 64 repositories are looked at per second, in rotation, so a pass over more takes several seconds and never starves one. The schedules of a repository are read again when its default branch moves, and go with the repository.
	134	+	`schedule`. `on.schedule[].cron` (five fields) is read at the wall clock of the entry's `timezone`, an IANA time zone name such as `America/New_York` (UTC without one; the time zone database is built into ermite). A name that is not one makes the workflow invalid: its push records a failed run with the error. On a daylight saving spring-forward day, an entry whose time is skipped runs at the next valid time (a 02:30 entry runs at 03:00), as GitHub documents; on a fall-back day, a time that happens twice runs once, at its first occurrence (GitHub does not document it; this is cron's behaviour). The entries are read on the default branch only. Every minute the scheduler starts the workflows whose cron matches that minute, on the latest commit of the default branch; the payload is `{"schedule": "<cron>"}`. A minute is handled once: ticks missed while ermite was down or busy are not made up, and a workflow with several matching entries starts one run. The actor is whoever last moved the default branch to its current commit (a push or a merge), and the run is trusted only if that was the owner. ermite records who moves each branch and tag before it moves (in the pre-receive decision of a push, and in the merge under its lock; a record that cannot be written refuses the push), together with the commit it is for: the record counts only while the ref still points to that commit, so a late write cannot leave the owner's record on a tip an agent moved there (owner A, agent B, agent A). A tip with no record, a record for another commit (a push refused after its record was written) or an unreadable record runs untrusted, without secrets. If the owner and a token push the very same commit at the same time, the owner's record may remain: the content is the same and the owner pushed it. At most 64 repositories are looked at per second, in rotation, so a pass over more takes several seconds and never starves one. The schedules of a repository are read again when its default branch moves, and go with the repository.
135	135
136	136		`workflow_run`. A workflow that lists `workflow_run: workflows: [<name>]` runs when a run of a workflow with that `name` is `requested` (created) or `completed` (`types:` picks one; `in_progress` never happens). `branches` and `branches-ignore` apply to the head branch of the triggering run (the source branch of a merge request). The workflow is read from the default branch and runs on its latest commit; the payload has `workflow_run` (`id`, `name`, `head_branch`, `head_sha`, `event`, `status`, `conclusion`, `run_number`, `run_attempt`, `path`, `actor`) and `workflow`. The run inherits the actor and the trust of the triggering run and never gains any: a workflow_run of an untrusted run has no secrets. A chain stops after three levels of `workflow_run`.
137	137
138		-	The event of a run is `push`, `pull_request`, `pull_request_target`, `workflow_dispatch`, `schedule` or `workflow_run`: `ermite run list` and `view` and the Actions pages show it.
	138	+	`create` and `delete`. A push (or a merge that deletes its source branch) that creates or deletes a branch or a tag starts the workflows that declare `create` or `delete`, besides its `push` workflows. A `create` run is on the created ref: `github.ref` is it, `github.sha` its commit, and the workflow is read from that commit. A `delete` run is on the default branch, its latest commit, and its workflow is read from there: the deleted ref is gone. The payload has `ref` (the short name), `ref_type` (`branch` or `tag`), `pusher_type` (`user`), `repository` and `sender`; `create` adds `master_branch` (the default branch) and `description` (`null`). There are no filters, and `[skip ci]` does not apply. A push that creates (or deletes) more than three tags starts no `create` (or `delete`) run for them, as on GitHub. A `create` run is trusted only when the pusher is the owner and the record of the created ref names the commit it runs (see `schedule`); a `delete` run only when the deleter is the owner and the default branch tip was pushed by the owner, as for `pull_request_target` (the deleted ref's record went with it). A job token starts neither.
	139	+
	140	+	The event of a run is `push`, `pull_request`, `pull_request_target`, `workflow_dispatch`, `schedule`, `workflow_run`, `create` or `delete`: `ermite run list` and `view` and the Actions pages show it.
139	141
140	142		The Actions that run here call a small part of GitHub's REST API, served at `github.api_url` (`/-/github/api`): `GET /repos/{owner}/{repo}`, `/repos/{owner}/{repo}/hash-algorithm`, `/repos/{owner}/{repo}/tarball[/{ref}]` (streamed with 200, not redirected; a `HEAD` answers the headers without building the archive) and `GET /repos/{owner}/{repo}/pulls/{number}/files` (what `dorny/paths-filter` reads on a `pull_request`: up to 3000 files, `page` and `per_page` of at most 100, 30 by default, with the `Link` header; a job token needs `pull-requests: read`, else `403 Resource not accessible by integration`). The owner in these paths and in the Git URLs is matched without regard to case, as on GitHub. Any other path is GitHub's `404 {"message":"Not Found",…}`, so an action that needs more fails as on a GitHub instance without that feature.
141	143
142	144		Actions from github.com are resolved with `git ls-remote https://github.com/<owner>/<repo>` (no credentials, 20 s, 16 MiB of output, at most four at once, 60 s for a whole call). ermite hands git only the proxy settings of its own environment (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, `ALL_PROXY` and their lower-case forms), so a server that reaches the Internet through a proxy sets them for `ermite serve`; nothing else of its environment, no credential, reaches git.
143	145
144		-	`ermite run list\|view\|watch\|cancel\|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. `concurrency:` is supported at the workflow level (evaluated when the run starts) and at the job level (evaluated when the job is planned, once per matrix leg): a group is per repository and compared without regard to case, `queue: single` (the default) lets one run or job go and keeps one waiting, a newer waiting entry cancels the older one (`Canceling since a higher priority waiting request for '<group>' exists`), and `cancel-in-progress` (a boolean or an expression) also cancels the running one. A job held back by a group shows `pending` in the API, the web pages and the CLI. A group that evaluates to nothing fails the job, and `queue: max` is refused with `not supported yet: queue: max`. Reusable workflows (`jobs.<id>.uses`) run: `./.github/workflows/x.yml` (the caller's repository, at the commit of the run), `owner/repo/.github/workflows/x.yml@ref` of a repository of this server when `owner` is `ERMITE_OWNER` and the actor of the run can read it (else the call fails as for a repository that does not exist), and otherwise a public repository on github.com, whose ref is resolved like the ref of an action and whose file is downloaded at that commit (at most 1 MiB). ermite expands the call when the job is planned: the called workflow must declare `on: workflow_call`, `with:` is checked against its inputs (types, `required`, defaults), `secrets:` passes the listed secrets, renamed as written, or all of them with `secrets: inherit` (a secret that is not passed is not there), and its `outputs` are available to the later jobs of the caller as `needs.<job>.outputs`. The jobs of the called workflow are named `<calling job> / <job>` and keyed `<calling job id>/<job id>`, wait for the `needs` of the calling job, and are repeated for each leg of its `strategy`; `permissions` can only be lowered by the called workflow. A call may nest four levels deep. `concurrency:` on the calling job and the workflow-level `concurrency:` of the called workflow (evaluated with the caller's `github` and the call's `inputs`) apply to the whole call, which holds the group until it ends, with `cancel-in-progress` as for any job. A workflow that uses another feature ermite does not support yet fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
	146	+	`ermite run list\|view\|watch\|cancel\|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. `concurrency:` is supported at the workflow level (evaluated when the run starts) and at the job level (evaluated when the job is planned, once per matrix leg): a group is per repository and compared without regard to case, `queue: single` (the default) lets one run or job go and keeps one waiting, a newer waiting entry cancels the older one (`Canceling since a higher priority waiting request for '<group>' exists`), and `cancel-in-progress` (a boolean or an expression) also cancels the running one. With `queue: max`, up to 100 entries wait and run first in, first out; one more is cancelled (`Canceling since the concurrency group '<group>' already has 100 waiting requests`, ermite's wording). In a group that mixes both (GitHub does not say), ermite applies the rule of the entry that starts waiting: a `single` entry cancels every waiting entry, `max` ones included, and a `max` entry joins the end of the line. A job held back by a group shows `pending` in the API, the web pages and the CLI. A group that evaluates to nothing fails the job. `queue: max` with `cancel-in-progress: true` written as such makes the workflow invalid (its push records a failed run with the error, as GitHub's validation does); when either is an expression that evaluates to that pair, the job fails with the same message. The waiting line is stored with its head also in the field older ermite versions read, so a downgrade keeps the first waiting entry and cancels the others. Reusable workflows (`jobs.<id>.uses`) run: `./.github/workflows/x.yml` (the caller's repository, at the commit of the run), `owner/repo/.github/workflows/x.yml@ref` of a repository of this server when `owner` is `ERMITE_OWNER` and the actor of the run can read it (else the call fails as for a repository that does not exist), and otherwise a public repository on github.com, whose ref is resolved like the ref of an action and whose file is downloaded at that commit (at most 1 MiB). ermite expands the call when the job is planned: the called workflow must declare `on: workflow_call`, `with:` is checked against its inputs (types, `required`, defaults), `secrets:` passes the listed secrets, renamed as written, or all of them with `secrets: inherit` (a secret that is not passed is not there), and its `outputs` are available to the later jobs of the caller as `needs.<job>.outputs`. The jobs of the called workflow are named `<calling job> / <job>` and keyed `<calling job id>/<job id>`, wait for the `needs` of the calling job, and are repeated for each leg of its `strategy`; `permissions` can only be lowered by the called workflow. A call may nest four levels deep. `concurrency:` on the calling job and the workflow-level `concurrency:` of the called workflow (evaluated with the caller's `github` and the call's `inputs`) apply to the whole call, which holds the group until it ends, with `cancel-in-progress` as for any job. A workflow that uses another feature ermite does not support yet fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
145	147
146	148		Artifacts are not supported. `actions/upload-artifact` and `actions/download-artifact` v4 and later (`@actions/artifact` v2) refuse any server that is not github.com, `.ghe.com` or `.localhost` and fail with `GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ and download-artifact@v4+ are not currently supported on GHES.`, as on a GitHub Enterprise Server. The cache (`actions/cache`, `setup-go`/`setup-node` `cache:`) is served by the runner and works.
147	149

 ## CI
-ermite runs the `.github/workflows/*.yml` workflows of a repository on `push` (a branch or a tag, and the merge of a merge request), on `pull_request` and `pull_request_target` (the merge requests, below), `workflow_dispatch`, `schedule` and `workflow_run` (below) and shows each job as a check on the commit, the branch list and the merge request. CI is on once a runner is registered; there is no switch. Before that a push makes no run and no check.
+ermite runs the `.github/workflows/*.yml` workflows of a repository on `push` (a branch or a tag, and the merge of a merge request), on `pull_request` and `pull_request_target` (the merge requests, below), `workflow_dispatch`, `schedule`, `workflow_run`, `create` and `delete` (below) and shows each job as a check on the commit, the branch list and the merge request. CI is on once a runner is registered; there is no switch. Before that a push makes no run and no check.
 Runs execute on [ermite-runner](https://git.bnema.dev/bnema/ermite-runner) hosts, which call ermite over its API. Create a token for the runner host with `ermite token create <label> --preset runner` (the `runner:register` scope, which also gives access to the code and the secrets of every repository: give it only to runners you operate), then register the runner with it. `ermite runner list` shows the runners and whether they are online; `ermite runner remove <id>` stops one. A job goes to a runner that has every label of its `runs-on`; a job no runner takes within 24 hours fails.
 **`workflow_dispatch`.** `ermite workflow run <file> --repo <repo> [--ref <branch or tag>] [-f name=value …]` starts a run of a workflow that declares `workflow_dispatch` (the API is `RunService.DispatchWorkflow`, feature `ci.dispatch`; it needs `ci:write`). The file is its path or its name under `.github/workflows`, and it is read at the ref (the default branch when `--ref` is left out). The inputs are checked against the workflow: an undeclared input, a missing `required` input without a default, a `boolean` other than `true` or `false`, a `number` that is not a decimal number, a `choice` outside its `options`, and an `environment` that is not a valid environment name are refused; a `choice` without default takes its first option. A workflow has at most 25 inputs and their values at most 65,535 characters. In the run, `github.event.inputs` holds the values as strings and the `inputs` context as their declared types (`boolean`, `number`, else string). The actor is the caller. The run is trusted (secrets, write token) only when **both** the caller is the owner **and** the ref it runs on was moved to its current commit by the owner (see `schedule`): the owner dispatching an agent's branch gets an untrusted run, like the owner re-running an agent's run, and a ref with no record (pushed before ermite recorded pushers) is untrusted too. This holds for a tag as for a branch, and for a job token as for the owner. A token limited to branch prefixes can dispatch only on a branch under them and its run is never trusted. A job can dispatch with its repository token when its `permissions:` give `actions: write` (GitHub's exception to "events made with `GITHUB_TOKEN` start no run"), even without `contents: read`; a token that does not hold the permission is refused.
-**`schedule`.** `on.schedule[].cron` (five fields, in UTC; `timezone` is ignored) is read on the **default branch** only. Every minute the scheduler starts the workflows whose cron matches that minute, on the latest commit of the default branch; the payload is `{"schedule": "<cron>"}`. A minute is handled once: ticks missed while ermite was down or busy are not made up, and a workflow with several matching entries starts one run. The actor is **whoever last moved the default branch to its current commit** (a push or a merge), and the run is trusted only if that was the owner. ermite records who moves each branch and tag **before** it moves (in the pre-receive decision of a push, and in the merge under its lock; a record that cannot be written refuses the push), together with the commit it is for: the record counts only while the ref still points to that commit, so a late write cannot leave the owner's record on a tip an agent moved there (owner A, agent B, agent A). A tip with no record, a record for another commit (a push refused after its record was written) or an unreadable record runs untrusted, without secrets. If the owner and a token push the very same commit at the same time, the owner's record may remain: the content is the same and the owner pushed it. At most 64 repositories are looked at per second, in rotation, so a pass over more takes several seconds and never starves one. The schedules of a repository are read again when its default branch moves, and go with the repository.
+**`schedule`.** `on.schedule[].cron` (five fields) is read at the wall clock of the entry's `timezone`, an IANA time zone name such as `America/New_York` (UTC without one; the time zone database is built into ermite). A name that is not one makes the workflow invalid: its push records a failed run with the error. On a daylight saving spring-forward day, an entry whose time is skipped runs at the next valid time (a 02:30 entry runs at 03:00), as GitHub documents; on a fall-back day, a time that happens twice runs once, at its first occurrence (GitHub does not document it; this is cron's behaviour). The entries are read on the **default branch** only. Every minute the scheduler starts the workflows whose cron matches that minute, on the latest commit of the default branch; the payload is `{"schedule": "<cron>"}`. A minute is handled once: ticks missed while ermite was down or busy are not made up, and a workflow with several matching entries starts one run. The actor is **whoever last moved the default branch to its current commit** (a push or a merge), and the run is trusted only if that was the owner. ermite records who moves each branch and tag **before** it moves (in the pre-receive decision of a push, and in the merge under its lock; a record that cannot be written refuses the push), together with the commit it is for: the record counts only while the ref still points to that commit, so a late write cannot leave the owner's record on a tip an agent moved there (owner A, agent B, agent A). A tip with no record, a record for another commit (a push refused after its record was written) or an unreadable record runs untrusted, without secrets. If the owner and a token push the very same commit at the same time, the owner's record may remain: the content is the same and the owner pushed it. At most 64 repositories are looked at per second, in rotation, so a pass over more takes several seconds and never starves one. The schedules of a repository are read again when its default branch moves, and go with the repository.
 **`workflow_run`.** A workflow that lists `workflow_run: workflows: [<name>]` runs when a run of a workflow with that `name` is `requested` (created) or `completed` (`types:` picks one; `in_progress` never happens). `branches` and `branches-ignore` apply to the head branch of the triggering run (the source branch of a merge request). The workflow is read from the **default branch** and runs on its latest commit; the payload has `workflow_run` (`id`, `name`, `head_branch`, `head_sha`, `event`, `status`, `conclusion`, `run_number`, `run_attempt`, `path`, `actor`) and `workflow`. The run inherits the actor and the trust of the triggering run and never gains any: a workflow_run of an untrusted run has no secrets. A chain stops after three levels of `workflow_run`.
-The event of a run is `push`, `pull_request`, `pull_request_target`, `workflow_dispatch`, `schedule` or `workflow_run`: `ermite run list` and `view` and the Actions pages show it.
+**`create` and `delete`.** A push (or a merge that deletes its source branch) that creates or deletes a branch or a tag starts the workflows that declare `create` or `delete`, besides its `push` workflows. A `create` run is on the created ref: `github.ref` is it, `github.sha` its commit, and the workflow is read from that commit. A `delete` run is on the **default branch**, its latest commit, and its workflow is read from there: the deleted ref is gone. The payload has `ref` (the short name), `ref_type` (`branch` or `tag`), `pusher_type` (`user`), `repository` and `sender`; `create` adds `master_branch` (the default branch) and `description` (`null`). There are no filters, and `[skip ci]` does not apply. A push that creates (or deletes) more than three tags starts no `create` (or `delete`) run for them, as on GitHub. A `create` run is trusted only when the pusher is the owner **and** the record of the created ref names the commit it runs (see `schedule`); a `delete` run only when the deleter is the owner **and** the default branch tip was pushed by the owner, as for `pull_request_target` (the deleted ref's record went with it). A job token starts neither.
+The event of a run is `push`, `pull_request`, `pull_request_target`, `workflow_dispatch`, `schedule`, `workflow_run`, `create` or `delete`: `ermite run list` and `view` and the Actions pages show it.
 The Actions that run here call a small part of GitHub's REST API, served at `github.api_url` (`/-/github/api`): `GET /repos/{owner}/{repo}`, `/repos/{owner}/{repo}/hash-algorithm`, `/repos/{owner}/{repo}/tarball[/{ref}]` (streamed with 200, not redirected; a `HEAD` answers the headers without building the archive) and `GET /repos/{owner}/{repo}/pulls/{number}/files` (what `dorny/paths-filter` reads on a `pull_request`: up to 3000 files, `page` and `per_page` of at most 100, 30 by default, with the `Link` header; a job token needs `pull-requests: read`, else `403 Resource not accessible by integration`). The owner in these paths and in the Git URLs is matched without regard to case, as on GitHub. Any other path is GitHub's `404 {"message":"Not Found",…}`, so an action that needs more fails as on a GitHub instance without that feature.
 Actions from github.com are resolved with `git ls-remote https://github.com/<owner>/<repo>` (no credentials, 20 s, 16 MiB of output, at most four at once, 60 s for a whole call). ermite hands git only the proxy settings of its own environment (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, `ALL_PROXY` and their lower-case forms), so a server that reaches the Internet through a proxy sets them for `ermite serve`; nothing else of its environment, no credential, reaches git.
-`ermite run list|view|watch|cancel|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. `concurrency:` is supported at the workflow level (evaluated when the run starts) and at the job level (evaluated when the job is planned, once per matrix leg): a group is per repository and compared without regard to case, `queue: single` (the default) lets one run or job go and keeps one waiting, a newer waiting entry cancels the older one (`Canceling since a higher priority waiting request for '<group>' exists`), and `cancel-in-progress` (a boolean or an expression) also cancels the running one. A job held back by a group shows `pending` in the API, the web pages and the CLI. A group that evaluates to nothing fails the job, and `queue: max` is refused with `not supported yet: queue: max`. Reusable workflows (`jobs.<id>.uses`) run: `./.github/workflows/x.yml` (the caller's repository, at the commit of the run), `owner/repo/.github/workflows/x.yml@ref` of a repository of this server when `owner` is `ERMITE_OWNER` and the actor of the run can read it (else the call fails as for a repository that does not exist), and otherwise a public repository on github.com, whose ref is resolved like the ref of an action and whose file is downloaded at that commit (at most 1 MiB). ermite expands the call when the job is planned: the called workflow must declare `on: workflow_call`, `with:` is checked against its inputs (types, `required`, defaults), `secrets:` passes the listed secrets, renamed as written, or all of them with `secrets: inherit` (a secret that is not passed is not there), and its `outputs` are available to the later jobs of the caller as `needs.<job>.outputs`. The jobs of the called workflow are named `<calling job> / <job>` and keyed `<calling job id>/<job id>`, wait for the `needs` of the calling job, and are repeated for each leg of its `strategy`; `permissions` can only be lowered by the called workflow. A call may nest four levels deep. `concurrency:` on the calling job and the workflow-level `concurrency:` of the called workflow (evaluated with the caller's `github` and the call's `inputs`) apply to the whole call, which holds the group until it ends, with `cancel-in-progress` as for any job. A workflow that uses another feature ermite does not support yet fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
+`ermite run list|view|watch|cancel|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. `concurrency:` is supported at the workflow level (evaluated when the run starts) and at the job level (evaluated when the job is planned, once per matrix leg): a group is per repository and compared without regard to case, `queue: single` (the default) lets one run or job go and keeps one waiting, a newer waiting entry cancels the older one (`Canceling since a higher priority waiting request for '<group>' exists`), and `cancel-in-progress` (a boolean or an expression) also cancels the running one. With `queue: max`, up to 100 entries wait and run first in, first out; one more is cancelled (`Canceling since the concurrency group '<group>' already has 100 waiting requests`, ermite's wording). In a group that mixes both (GitHub does not say), ermite applies the rule of the entry that starts waiting: a `single` entry cancels every waiting entry, `max` ones included, and a `max` entry joins the end of the line. A job held back by a group shows `pending` in the API, the web pages and the CLI. A group that evaluates to nothing fails the job. `queue: max` with `cancel-in-progress: true` written as such makes the workflow invalid (its push records a failed run with the error, as GitHub's validation does); when either is an expression that evaluates to that pair, the job fails with the same message. The waiting line is stored with its head also in the field older ermite versions read, so a downgrade keeps the first waiting entry and cancels the others. Reusable workflows (`jobs.<id>.uses`) run: `./.github/workflows/x.yml` (the caller's repository, at the commit of the run), `owner/repo/.github/workflows/x.yml@ref` of a repository of this server when `owner` is `ERMITE_OWNER` and the actor of the run can read it (else the call fails as for a repository that does not exist), and otherwise a public repository on github.com, whose ref is resolved like the ref of an action and whose file is downloaded at that commit (at most 1 MiB). ermite expands the call when the job is planned: the called workflow must declare `on: workflow_call`, `with:` is checked against its inputs (types, `required`, defaults), `secrets:` passes the listed secrets, renamed as written, or all of them with `secrets: inherit` (a secret that is not passed is not there), and its `outputs` are available to the later jobs of the caller as `needs.<job>.outputs`. The jobs of the called workflow are named `<calling job> / <job>` and keyed `<calling job id>/<job id>`, wait for the `needs` of the calling job, and are repeated for each leg of its `strategy`; `permissions` can only be lowered by the called workflow. A call may nest four levels deep. `concurrency:` on the calling job and the workflow-level `concurrency:` of the called workflow (evaluated with the caller's `github` and the call's `inputs`) apply to the whole call, which holds the group until it ends, with `cancel-in-progress` as for any job. A workflow that uses another feature ermite does not support yet fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
 **Artifacts** are not supported. `actions/upload-artifact` and `actions/download-artifact` v4 and later (`@actions/artifact` v2) refuse any server that is not github.com, `*.ghe.com` or `*.localhost` and fail with `GHESNotSupportedError: @actions/artifact v2.0.0+, upload-artifact@v4+ and download-artifact@v4+ are not currently supported on GHES.`, as on a GitHub Enterprise Server. The cache (`actions/cache`, `setup-go`/`setup-node` `cache:`) is served by the runner and works.

M gen/ermite/storage/ci.pb.go +70 -22


@@ -2148,7 +2148,9 @@ type CIConcurrency struct {
2148	2148		FromRun bool `protobuf:"varint,5,opt,name=from_run,json=fromRun,proto3" json:"from_run,omitempty"`
2149	2149		// preempted: cancelled because a newer entry of the group asked for it; the
2150	2150		// job ends with GitHub's annotation.
2151		-	Preempted bool `protobuf:"varint,6,opt,name=preempted,proto3" json:"preempted,omitempty"`
	2151	+	Preempted bool `protobuf:"varint,6,opt,name=preempted,proto3" json:"preempted,omitempty"`
	2152	+	// queue_max: queue: max (false is queue: single, the default).
	2153	+	QueueMax bool `protobuf:"varint,7,opt,name=queue_max,json=queueMax,proto3" json:"queue_max,omitempty"`
2152	2154		unknownFields protoimpl.UnknownFields
2153	2155		sizeCache protoimpl.SizeCache
2154	2156		}

@@ -2225,6 +2227,13 @@ func (x *CIConcurrency) GetPreempted() bool {
2225	2227		return false
2226	2228		}
2227	2229
	2230	+	func (x *CIConcurrency) GetQueueMax() bool {
	2231	+	if x != nil {
	2232	+	return x.QueueMax
	2233	+	}
	2234	+	return false
	2235	+	}
	2236	+
2228	2237		// CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.
2229	2238		type CIConcurrencyEntry struct {
2230	2239		state protoimpl.MessageState `protogen:"open.v1"`

@@ -2234,7 +2243,9 @@ type CIConcurrencyEntry struct {
2234	2243		Cancel bool `protobuf:"varint,4,opt,name=cancel,proto3" json:"cancel,omitempty"`
2235	2244		// canceling: a waiting entry with cancel-in-progress asked this running one
2236	2245		// to stop; the entry stays until it has ended.
2237		-	Canceling bool `protobuf:"varint,5,opt,name=canceling,proto3" json:"canceling,omitempty"`
	2246	+	Canceling bool `protobuf:"varint,5,opt,name=canceling,proto3" json:"canceling,omitempty"`
	2247	+	// queue_max: the entry was queued with queue: max.
	2248	+	QueueMax bool `protobuf:"varint,6,opt,name=queue_max,json=queueMax,proto3" json:"queue_max,omitempty"`
2238	2249		unknownFields protoimpl.UnknownFields
2239	2250		sizeCache protoimpl.SizeCache
2240	2251		}

@@ -2304,13 +2315,26 @@ func (x *CIConcurrencyEntry) GetCanceling() bool {
2304	2315		return false
2305	2316		}
2306	2317
	2318	+	func (x *CIConcurrencyEntry) GetQueueMax() bool {
	2319	+	if x != nil {
	2320	+	return x.QueueMax
	2321	+	}
	2322	+	return false
	2323	+	}
	2324	+
2307	2325		// CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
2308		-	// running entry and the pending one of a group (queue: single). A group with
2309		-	// neither has no key.
	2326	+	// running entry and the waiting ones of a group. A group with neither has no
	2327	+	// key.
2310	2328		type CIConcurrencyRecord struct {
2311		-	state protoimpl.MessageState `protogen:"open.v1"`
2312		-	Running *CIConcurrencyEntry `protobuf:"bytes,1,opt,name=running,proto3" json:"running,omitempty"`
2313		-	Pending *CIConcurrencyEntry `protobuf:"bytes,2,opt,name=pending,proto3" json:"pending,omitempty"`
	2329	+	state protoimpl.MessageState `protogen:"open.v1"`
	2330	+	Running *CIConcurrencyEntry `protobuf:"bytes,1,opt,name=running,proto3" json:"running,omitempty"`
	2331	+	// pending is the head of queue, written for the binaries that predate queue
	2332	+	// (a downgrade keeps the first waiting entry; the others, which their record
	2333	+	// no longer names, are cancelled as replaced). It is read when queue is
	2334	+	// empty: a record stored by such a binary.
	2335	+	Pending *CIConcurrencyEntry `protobuf:"bytes,2,opt,name=pending,proto3" json:"pending,omitempty"`
	2336	+	// queue holds the waiting entries, first in first out.
	2337	+	Queue []*CIConcurrencyEntry `protobuf:"bytes,4,rep,name=queue,proto3" json:"queue,omitempty"`
2314	2338		unknownFields protoimpl.UnknownFields
2315	2339		sizeCache protoimpl.SizeCache
2316	2340		}

@@ -2359,6 +2383,13 @@ func (x CIConcurrencyRecord) GetPending() CIConcurrencyEntry {
2359	2383		return nil
2360	2384		}
2361	2385
	2386	+	func (x CIConcurrencyRecord) GetQueue() []CIConcurrencyEntry {
	2387	+	if x != nil {
	2388	+	return x.Queue
	2389	+	}
	2390	+	return nil
	2391	+	}
	2392	+
2362	2393		// CIRefTip is stored under "cireftip/<repo>/<full ref name>" (a branch or a
2363	2394		// tag): who moves the ref to the object sha, so that a scheduled or dispatched
2364	2395		// run on it takes their actor and trust (domain.CIRefTip). It is written before

@@ -2453,8 +2484,12 @@ type CIConcurrencyMark struct {
2453	2484		state protoimpl.MessageState `protogen:"open.v1"`
2454	2485		Entry *CIConcurrencyEntry `protobuf:"bytes,1,opt,name=entry,proto3" json:"entry,omitempty"`
2455	2486		// group as the newest request wrote it, for the cancel annotation.
2456		-	Group string `protobuf:"bytes,2,opt,name=group,proto3" json:"group,omitempty"`
2457		-	State CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
	2487	+	Group string `protobuf:"bytes,2,opt,name=group,proto3" json:"group,omitempty"`
	2488	+	State CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
	2489	+	// queue_full: with state REPLACED, the entry was a queue: max entry that
	2490	+	// found the waiting line full (its annotation says so). Stored as REPLACED so
	2491	+	// that a binary that predates it still cancels the entry.
	2492	+	QueueFull bool `protobuf:"varint,4,opt,name=queue_full,json=queueFull,proto3" json:"queue_full,omitempty"`
2458	2493		unknownFields protoimpl.UnknownFields
2459	2494		sizeCache protoimpl.SizeCache
2460	2495		}

@@ -2510,6 +2545,13 @@ func (x *CIConcurrencyMark) GetState() CIConcurrencyState {
2510	2545		return CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED
2511	2546		}
2512	2547
	2548	+	func (x *CIConcurrencyMark) GetQueueFull() bool {
	2549	+	if x != nil {
	2550	+	return x.QueueFull
	2551	+	}
	2552	+	return false
	2553	+	}
	2554	+
2513	2555		var File_ermite_storage_ci_proto protoreflect.FileDescriptor
2514	2556
2515	2557		const file_ermite_storage_ci_proto_rawDesc = "" +

@@ -2721,34 +2763,39 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
2721	2763		"\n" +
2722	2764		"end_column\x18\b \x01(\x05R\tendColumn\"S\n" +
2723	2765		"\x11CIStepAnnotations\x12>\n" +
2724		-	"\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations\"\xce\x01\n" +
	2766	+	"\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations\"\xeb\x01\n" +
2725	2767		"\rCIConcurrency\x12\x14\n" +
2726	2768		"\x05group\x18\x01 \x01(\tR\x05group\x12\x16\n" +
2727	2769		"\x06cancel\x18\x02 \x01(\bR\x06cancel\x128\n" +
2728	2770		"\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state\x12\x1c\n" +
2729	2771		"\tevaluated\x18\x04 \x01(\bR\tevaluated\x12\x19\n" +
2730	2772		"\bfrom_run\x18\x05 \x01(\bR\afromRun\x12\x1c\n" +
2731		-	"\tpreempted\x18\x06 \x01(\bR\tpreempted\"\x92\x01\n" +
	2773	+	"\tpreempted\x18\x06 \x01(\bR\tpreempted\x12\x1b\n" +
	2774	+	"\tqueue_max\x18\a \x01(\bR\bqueueMax\"\xaf\x01\n" +
2732	2775		"\x12CIConcurrencyEntry\x12\x15\n" +
2733	2776		"\x06run_id\x18\x01 \x01(\x03R\x05runId\x12\x18\n" +
2734	2777		"\aattempt\x18\x02 \x01(\x05R\aattempt\x12\x15\n" +
2735	2778		"\x06job_id\x18\x03 \x01(\tR\x05jobId\x12\x16\n" +
2736	2779		"\x06cancel\x18\x04 \x01(\bR\x06cancel\x12\x1c\n" +
2737		-	"\tcanceling\x18\x05 \x01(\bR\tcanceling\"\x97\x01\n" +
	2780	+	"\tcanceling\x18\x05 \x01(\bR\tcanceling\x12\x1b\n" +
	2781	+	"\tqueue_max\x18\x06 \x01(\bR\bqueueMax\"\xd1\x01\n" +
2738	2782		"\x13CIConcurrencyRecord\x12<\n" +
2739	2783		"\arunning\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\arunning\x12<\n" +
2740		-	"\apending\x18\x02 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\apendingJ\x04\b\x03\x10\x04\"\x95\x01\n" +
	2784	+	"\apending\x18\x02 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\apending\x128\n" +
	2785	+	"\x05queue\x18\x04 \x03(\v2\".ermite.storage.CIConcurrencyEntryR\x05queueJ\x04\b\x03\x10\x04\"\x95\x01\n" +
2741	2786		"\bCIRefTip\x12\x10\n" +
2742	2787		"\x03sha\x18\x01 \x01(\tR\x03sha\x12\x14\n" +
2743	2788		"\x05actor\x18\x02 \x01(\tR\x05actor\x12\x18\n" +
2744	2789		"\atrusted\x18\x03 \x01(\bR\atrusted\x12\x1f\n" +
2745	2790		"\vpusher_kind\x18\x04 \x01(\tR\n" +
2746	2791		"pusherKind\x12&\n" +
2747		-	"\x0fpusher_token_id\x18\x05 \x01(\tR\rpusherTokenId\"\x9d\x01\n" +
	2792	+	"\x0fpusher_token_id\x18\x05 \x01(\tR\rpusherTokenId\"\xbc\x01\n" +
2748	2793		"\x11CIConcurrencyMark\x128\n" +
2749	2794		"\x05entry\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\x05entry\x12\x14\n" +
2750	2795		"\x05group\x18\x02 \x01(\tR\x05group\x128\n" +
2751		-	"\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state*\x86\x01\n" +
	2796	+	"\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state\x12\x1d\n" +
	2797	+	"\n" +
	2798	+	"queue_full\x18\x04 \x01(\bR\tqueueFull*\x86\x01\n" +
2752	2799		"\bCIStatus\x12\x19\n" +
2753	2800		"\x15CI_STATUS_UNSPECIFIED\x10\x00\x12\x14\n" +
2754	2801		"\x10CI_STATUS_QUEUED\x10\x01\x12\x19\n" +

@@ -2890,13 +2937,14 @@ var file_ermite_storage_ci_proto_depIdxs = []int32{
2890	2937		7, // 43: ermite.storage.CIConcurrency.state:type_name -> ermite.storage.CIConcurrencyState
2891	2938		20, // 44: ermite.storage.CIConcurrencyRecord.running:type_name -> ermite.storage.CIConcurrencyEntry
2892	2939		20, // 45: ermite.storage.CIConcurrencyRecord.pending:type_name -> ermite.storage.CIConcurrencyEntry
2893		-	20, // 46: ermite.storage.CIConcurrencyMark.entry:type_name -> ermite.storage.CIConcurrencyEntry
2894		-	7, // 47: ermite.storage.CIConcurrencyMark.state:type_name -> ermite.storage.CIConcurrencyState
2895		-	48, // [48:48] is the sub-list for method output_type
2896		-	48, // [48:48] is the sub-list for method input_type
2897		-	48, // [48:48] is the sub-list for extension type_name
2898		-	48, // [48:48] is the sub-list for extension extendee
2899		-	0, // [0:48] is the sub-list for field type_name
	2940	+	20, // 46: ermite.storage.CIConcurrencyRecord.queue:type_name -> ermite.storage.CIConcurrencyEntry
	2941	+	20, // 47: ermite.storage.CIConcurrencyMark.entry:type_name -> ermite.storage.CIConcurrencyEntry
	2942	+	7, // 48: ermite.storage.CIConcurrencyMark.state:type_name -> ermite.storage.CIConcurrencyState
	2943	+	49, // [49:49] is the sub-list for method output_type
	2944	+	49, // [49:49] is the sub-list for method input_type
	2945	+	49, // [49:49] is the sub-list for extension type_name
	2946	+	49, // [49:49] is the sub-list for extension extendee
	2947	+	0, // [0:49] is the sub-list for field type_name
2900	2948		}
2901	2949
2902	2950		func init() { file_ermite_storage_ci_proto_init() }

 	FromRun bool `protobuf:"varint,5,opt,name=from_run,json=fromRun,proto3" json:"from_run,omitempty"`
 	// preempted: cancelled because a newer entry of the group asked for it; the
 	// job ends with GitHub's annotation.
-	Preempted     bool `protobuf:"varint,6,opt,name=preempted,proto3" json:"preempted,omitempty"`
+	Preempted bool `protobuf:"varint,6,opt,name=preempted,proto3" json:"preempted,omitempty"`
+	// queue_max: queue: max (false is queue: single, the default).
+	QueueMax      bool `protobuf:"varint,7,opt,name=queue_max,json=queueMax,proto3" json:"queue_max,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
 	return false
 }
+func (x *CIConcurrency) GetQueueMax() bool {
+	if x != nil {
+		return x.QueueMax
+	}
+	return false
+}
 // CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.
 type CIConcurrencyEntry struct {
 	state   protoimpl.MessageState `protogen:"open.v1"`
 	Cancel  bool                   `protobuf:"varint,4,opt,name=cancel,proto3" json:"cancel,omitempty"`
 	// canceling: a waiting entry with cancel-in-progress asked this running one
 	// to stop; the entry stays until it has ended.
-	Canceling     bool `protobuf:"varint,5,opt,name=canceling,proto3" json:"canceling,omitempty"`
+	Canceling bool `protobuf:"varint,5,opt,name=canceling,proto3" json:"canceling,omitempty"`
+	// queue_max: the entry was queued with queue: max.
+	QueueMax      bool `protobuf:"varint,6,opt,name=queue_max,json=queueMax,proto3" json:"queue_max,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
 	return false
 }
+func (x *CIConcurrencyEntry) GetQueueMax() bool {
+	if x != nil {
+		return x.QueueMax
+	}
+	return false
+}
 // CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
-// running entry and the pending one of a group (queue: single). A group with
+// running entry and the waiting ones of a group. A group with neither has no
-// neither has no key.
+// key.
 type CIConcurrencyRecord struct {
-	state         protoimpl.MessageState `protogen:"open.v1"`
+	state   protoimpl.MessageState `protogen:"open.v1"`
-	Running       *CIConcurrencyEntry    `protobuf:"bytes,1,opt,name=running,proto3" json:"running,omitempty"`
+	Running *CIConcurrencyEntry    `protobuf:"bytes,1,opt,name=running,proto3" json:"running,omitempty"`
-	Pending       *CIConcurrencyEntry    `protobuf:"bytes,2,opt,name=pending,proto3" json:"pending,omitempty"`
+	// pending is the head of queue, written for the binaries that predate queue
+	// (a downgrade keeps the first waiting entry; the others, which their record
+	// no longer names, are cancelled as replaced). It is read when queue is
+	// empty: a record stored by such a binary.
+	Pending *CIConcurrencyEntry `protobuf:"bytes,2,opt,name=pending,proto3" json:"pending,omitempty"`
+	// queue holds the waiting entries, first in first out.
+	Queue         []*CIConcurrencyEntry `protobuf:"bytes,4,rep,name=queue,proto3" json:"queue,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
 	return nil
 }
+func (x *CIConcurrencyRecord) GetQueue() []*CIConcurrencyEntry {
+	if x != nil {
+		return x.Queue
+	}
+	return nil
+}
 // CIRefTip is stored under "cireftip/<repo>/<full ref name>" (a branch or a
 // tag): who moves the ref to the object sha, so that a scheduled or dispatched
 // run on it takes their actor and trust (domain.CIRefTip). It is written before
 	state protoimpl.MessageState `protogen:"open.v1"`
 	Entry *CIConcurrencyEntry    `protobuf:"bytes,1,opt,name=entry,proto3" json:"entry,omitempty"`
 	// group as the newest request wrote it, for the cancel annotation.
-	Group         string             `protobuf:"bytes,2,opt,name=group,proto3" json:"group,omitempty"`
+	Group string             `protobuf:"bytes,2,opt,name=group,proto3" json:"group,omitempty"`
-	State         CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
+	State CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
+	// queue_full: with state REPLACED, the entry was a queue: max entry that
+	// found the waiting line full (its annotation says so). Stored as REPLACED so
+	// that a binary that predates it still cancels the entry.
+	QueueFull     bool `protobuf:"varint,4,opt,name=queue_full,json=queueFull,proto3" json:"queue_full,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
 	return CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED
 }
+func (x *CIConcurrencyMark) GetQueueFull() bool {
+	if x != nil {
+		return x.QueueFull
+	}
+	return false
+}
 var File_ermite_storage_ci_proto protoreflect.FileDescriptor
 const file_ermite_storage_ci_proto_rawDesc = "" +
 	"\n" +
 	"end_column\x18\b \x01(\x05R\tendColumn\"S\n" +
 	"\x11CIStepAnnotations\x12>\n" +
-	"\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations\"\xce\x01\n" +
+	"\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations\"\xeb\x01\n" +
 	"\rCIConcurrency\x12\x14\n" +
 	"\x05group\x18\x01 \x01(\tR\x05group\x12\x16\n" +
 	"\x06cancel\x18\x02 \x01(\bR\x06cancel\x128\n" +
 	"\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state\x12\x1c\n" +
 	"\tevaluated\x18\x04 \x01(\bR\tevaluated\x12\x19\n" +
 	"\bfrom_run\x18\x05 \x01(\bR\afromRun\x12\x1c\n" +
-	"\tpreempted\x18\x06 \x01(\bR\tpreempted\"\x92\x01\n" +
+	"\tpreempted\x18\x06 \x01(\bR\tpreempted\x12\x1b\n" +
+	"\tqueue_max\x18\a \x01(\bR\bqueueMax\"\xaf\x01\n" +
 	"\x12CIConcurrencyEntry\x12\x15\n" +
 	"\x06run_id\x18\x01 \x01(\x03R\x05runId\x12\x18\n" +
 	"\aattempt\x18\x02 \x01(\x05R\aattempt\x12\x15\n" +
 	"\x06job_id\x18\x03 \x01(\tR\x05jobId\x12\x16\n" +
 	"\x06cancel\x18\x04 \x01(\bR\x06cancel\x12\x1c\n" +
-	"\tcanceling\x18\x05 \x01(\bR\tcanceling\"\x97\x01\n" +
+	"\tcanceling\x18\x05 \x01(\bR\tcanceling\x12\x1b\n" +
+	"\tqueue_max\x18\x06 \x01(\bR\bqueueMax\"\xd1\x01\n" +
 	"\x13CIConcurrencyRecord\x12<\n" +
 	"\arunning\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\arunning\x12<\n" +
-	"\apending\x18\x02 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\apendingJ\x04\b\x03\x10\x04\"\x95\x01\n" +
+	"\apending\x18\x02 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\apending\x128\n" +
+	"\x05queue\x18\x04 \x03(\v2\".ermite.storage.CIConcurrencyEntryR\x05queueJ\x04\b\x03\x10\x04\"\x95\x01\n" +
 	"\bCIRefTip\x12\x10\n" +
 	"\x03sha\x18\x01 \x01(\tR\x03sha\x12\x14\n" +
 	"\x05actor\x18\x02 \x01(\tR\x05actor\x12\x18\n" +
 	"\atrusted\x18\x03 \x01(\bR\atrusted\x12\x1f\n" +
 	"\vpusher_kind\x18\x04 \x01(\tR\n" +
 	"pusherKind\x12&\n" +
-	"\x0fpusher_token_id\x18\x05 \x01(\tR\rpusherTokenId\"\x9d\x01\n" +
+	"\x0fpusher_token_id\x18\x05 \x01(\tR\rpusherTokenId\"\xbc\x01\n" +
 	"\x11CIConcurrencyMark\x128\n" +
 	"\x05entry\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\x05entry\x12\x14\n" +
 	"\x05group\x18\x02 \x01(\tR\x05group\x128\n" +
-	"\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state*\x86\x01\n" +
+	"\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state\x12\x1d\n" +
+	"\n" +
+	"queue_full\x18\x04 \x01(\bR\tqueueFull*\x86\x01\n" +
 	"\bCIStatus\x12\x19\n" +
 	"\x15CI_STATUS_UNSPECIFIED\x10\x00\x12\x14\n" +
 	"\x10CI_STATUS_QUEUED\x10\x01\x12\x19\n" +
 	7,  // 43: ermite.storage.CIConcurrency.state:type_name -> ermite.storage.CIConcurrencyState
 	20, // 44: ermite.storage.CIConcurrencyRecord.running:type_name -> ermite.storage.CIConcurrencyEntry
 	20, // 45: ermite.storage.CIConcurrencyRecord.pending:type_name -> ermite.storage.CIConcurrencyEntry
-	20, // 46: ermite.storage.CIConcurrencyMark.entry:type_name -> ermite.storage.CIConcurrencyEntry
+	20, // 46: ermite.storage.CIConcurrencyRecord.queue:type_name -> ermite.storage.CIConcurrencyEntry
-	7,  // 47: ermite.storage.CIConcurrencyMark.state:type_name -> ermite.storage.CIConcurrencyState
+	20, // 47: ermite.storage.CIConcurrencyMark.entry:type_name -> ermite.storage.CIConcurrencyEntry
-	48, // [48:48] is the sub-list for method output_type
+	7,  // 48: ermite.storage.CIConcurrencyMark.state:type_name -> ermite.storage.CIConcurrencyState
-	48, // [48:48] is the sub-list for method input_type
+	49, // [49:49] is the sub-list for method output_type
-	48, // [48:48] is the sub-list for extension type_name
+	49, // [49:49] is the sub-list for method input_type
-	48, // [48:48] is the sub-list for extension extendee
+	49, // [49:49] is the sub-list for extension type_name
-	0,  // [0:48] is the sub-list for field type_name
+	49, // [49:49] is the sub-list for extension extendee
+	0,  // [0:49] is the sub-list for field type_name
 }
 func init() { file_ermite_storage_ci_proto_init() }

M internal/adapters/kvstore/ci_concurrency.go +41 -16


@@ -49,11 +49,19 @@ var concurrencyStates = []struct {
49	49		{domain.CIConcurrencyHolding, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING},
50	50		}
51	51
	52	+	// markStates are the states a mark may hold on top of concurrencyStates.
	53	+	var markStates = append(slices.Clone(concurrencyStates), []struct {
	54	+	d domain.CIConcurrencyState
	55	+	p storagepb.CIConcurrencyState
	56	+	}{
	57	+	{domain.CIConcurrencyReplaced, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED},
	58	+	}...)
	59	+
52	60		func encodeConcurrency(c domain.CIConcurrency) *storagepb.CIConcurrency {
53	61		if c == (domain.CIConcurrency{}) {
54	62		return nil
55	63		}
56		-	out := &storagepb.CIConcurrency{Group: c.Group, Cancel: c.Cancel, Evaluated: c.Evaluated, FromRun: c.FromRun, Preempted: c.Preempted}
	64	+	out := &storagepb.CIConcurrency{Group: c.Group, Cancel: c.Cancel, Evaluated: c.Evaluated, FromRun: c.FromRun, Preempted: c.Preempted, QueueMax: c.Max}
57	65		for _, e := range concurrencyStates {
58	66		if e.d == c.State {
59	67		out.State = e.p

@@ -66,7 +74,7 @@ func decodeConcurrency(pb *storagepb.CIConcurrency) domain.CIConcurrency {
66	74		if pb == nil {
67	75		return domain.CIConcurrency{}
68	76		}
69		-	c := domain.CIConcurrency{Group: pb.GetGroup(), Cancel: pb.GetCancel(), Evaluated: pb.GetEvaluated(), FromRun: pb.GetFromRun(), Preempted: pb.GetPreempted()}
	77	+	c := domain.CIConcurrency{Group: pb.GetGroup(), Cancel: pb.GetCancel(), Evaluated: pb.GetEvaluated(), FromRun: pb.GetFromRun(), Preempted: pb.GetPreempted(), Max: pb.GetQueueMax()}
70	78		for _, e := range concurrencyStates {
71	79		if e.p == pb.GetState() {
72	80		c.State = e.d

@@ -79,11 +87,20 @@ func encodeConcurrencyEntry(e domain.CIConcurrencyEntry) storagepb.CIConcurren
79	87		if e == nil {
80	88		return nil
81	89		}
82		-	return &storagepb.CIConcurrencyEntry{RunId: e.RunID, Attempt: e.Attempt, JobId: e.JobID, Cancel: e.Cancel, Canceling: e.Canceling}
	90	+	return &storagepb.CIConcurrencyEntry{RunId: e.RunID, Attempt: e.Attempt, JobId: e.JobID, Cancel: e.Cancel, Canceling: e.Canceling, QueueMax: e.Max}
83	91		}
84	92
	93	+	// encodeConcurrencyRecord writes the waiting entries to queue, and the head of
	94	+	// the line to pending too, for a binary that predates queue (a downgrade).
85	95		func encodeConcurrencyRecord(r domain.CIConcurrencyRecord) []byte {
86		-	return mustMarshal(&storagepb.CIConcurrencyRecord{Running: encodeConcurrencyEntry(r.Running), Pending: encodeConcurrencyEntry(r.Pending)})
	96	+	pb := &storagepb.CIConcurrencyRecord{Running: encodeConcurrencyEntry(r.Running)}
	97	+	for i := range r.Pending {
	98	+	pb.Queue = append(pb.Queue, encodeConcurrencyEntry(&r.Pending[i]))
	99	+	}
	100	+	if len(r.Pending) > 0 {
	101	+	pb.Pending = encodeConcurrencyEntry(&r.Pending[0])
	102	+	}
	103	+	return mustMarshal(pb)
87	104		}
88	105
89	106		//hexcheck:ignore suspicious-business-logic-in-adapter maps the stored record to the domain record field by field: no rule is decided here

@@ -92,14 +109,22 @@ func decodeConcurrencyRecord(val []byte) (domain.CIConcurrencyRecord, error) {
92	109		if err := proto.Unmarshal(val, &pb); err != nil {
93	110		return domain.CIConcurrencyRecord{}, fmt.Errorf("decode concurrency group: %w", err)
94	111		}
95		-	return domain.CIConcurrencyRecord{Running: decodeConcurrencyEntry(pb.GetRunning()), Pending: decodeConcurrencyEntry(pb.GetPending())}, nil
	112	+	rec := domain.CIConcurrencyRecord{Running: decodeConcurrencyEntry(pb.GetRunning())}
	113	+	for _, q := range pb.GetQueue() {
	114	+	rec.Pending = append(rec.Pending, *decodeConcurrencyEntry(q))
	115	+	}
	116	+	// A record stored before queue: max holds its only waiting entry in pending.
	117	+	if p := decodeConcurrencyEntry(pb.GetPending()); p != nil && len(rec.Pending) == 0 {
	118	+	rec.Pending = []domain.CIConcurrencyEntry{*p}
	119	+	}
	120	+	return rec, nil
96	121		}
97	122
98	123		func decodeConcurrencyEntry(pb storagepb.CIConcurrencyEntry) domain.CIConcurrencyEntry {
99	124		if pb == nil {
100	125		return nil
101	126		}
102		-	return &domain.CIConcurrencyEntry{RunID: pb.GetRunId(), Attempt: pb.GetAttempt(), JobID: pb.GetJobId(), Cancel: pb.GetCancel(), Canceling: pb.GetCanceling()}
	127	+	return &domain.CIConcurrencyEntry{RunID: pb.GetRunId(), Attempt: pb.GetAttempt(), JobID: pb.GetJobId(), Cancel: pb.GetCancel(), Canceling: pb.GetCanceling(), Max: pb.GetQueueMax()}
103	128		}
104	129
105	130		// concurMarkPrefix is where the marks of a group are: the group key is escaped,

@@ -110,13 +135,13 @@ func concurMarkPrefix(repo domain.RepoName, group string) string {
110	135
111	136		func encodeConcurrencyMark(m domain.CIConcurrencyMark) []byte {
112	137		pb := &storagepb.CIConcurrencyMark{Entry: encodeConcurrencyEntry(&m.Entry), Group: m.Group}
113		-	for _, e := range concurrencyStates {
	138	+	for _, e := range markStates {
114	139		if e.d == m.State {
115	140		pb.State = e.p
116	141		}
117	142		}
118		-	if m.State == domain.CIConcurrencyReplaced {
119		-	pb.State = storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED
	143	+	if m.State == domain.CIConcurrencyRejected {
	144	+	pb.State, pb.QueueFull = storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED, true
120	145		}
121	146		return mustMarshal(pb)
122	147		}

@@ -131,13 +156,13 @@ func decodeConcurrencyMark(val []byte) (domain.CIConcurrencyMark, error) {
131	156		if e := decodeConcurrencyEntry(pb.GetEntry()); e != nil {
132	157		m.Entry = *e
133	158		}
134		-	switch pb.GetState() {
135		-	case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING:
136		-	m.State = domain.CIConcurrencyPending
137		-	case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING:
138		-	m.State = domain.CIConcurrencyHolding
139		-	case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED:
140		-	m.State = domain.CIConcurrencyReplaced
	159	+	for _, e := range markStates {
	160	+	if e.p == pb.GetState() {
	161	+	m.State = e.d
	162	+	}
	163	+	}
	164	+	if m.State == domain.CIConcurrencyReplaced && pb.GetQueueFull() {
	165	+	m.State = domain.CIConcurrencyRejected
141	166		}
142	167		return m, nil
143	168		}

 	{domain.CIConcurrencyHolding, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING},
 }
+// markStates are the states a mark may hold on top of concurrencyStates.
+var markStates = append(slices.Clone(concurrencyStates), []struct {
+	d domain.CIConcurrencyState
+	p storagepb.CIConcurrencyState
+}{
+	{domain.CIConcurrencyReplaced, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED},
+}...)
 func encodeConcurrency(c domain.CIConcurrency) *storagepb.CIConcurrency {
 	if c == (domain.CIConcurrency{}) {
 		return nil
 	}
-	out := &storagepb.CIConcurrency{Group: c.Group, Cancel: c.Cancel, Evaluated: c.Evaluated, FromRun: c.FromRun, Preempted: c.Preempted}
+	out := &storagepb.CIConcurrency{Group: c.Group, Cancel: c.Cancel, Evaluated: c.Evaluated, FromRun: c.FromRun, Preempted: c.Preempted, QueueMax: c.Max}
 	for _, e := range concurrencyStates {
 		if e.d == c.State {
 			out.State = e.p
 	if pb == nil {
 		return domain.CIConcurrency{}
 	}
-	c := domain.CIConcurrency{Group: pb.GetGroup(), Cancel: pb.GetCancel(), Evaluated: pb.GetEvaluated(), FromRun: pb.GetFromRun(), Preempted: pb.GetPreempted()}
+	c := domain.CIConcurrency{Group: pb.GetGroup(), Cancel: pb.GetCancel(), Evaluated: pb.GetEvaluated(), FromRun: pb.GetFromRun(), Preempted: pb.GetPreempted(), Max: pb.GetQueueMax()}
 	for _, e := range concurrencyStates {
 		if e.p == pb.GetState() {
 			c.State = e.d
 	if e == nil {
 		return nil
 	}
-	return &storagepb.CIConcurrencyEntry{RunId: e.RunID, Attempt: e.Attempt, JobId: e.JobID, Cancel: e.Cancel, Canceling: e.Canceling}
+	return &storagepb.CIConcurrencyEntry{RunId: e.RunID, Attempt: e.Attempt, JobId: e.JobID, Cancel: e.Cancel, Canceling: e.Canceling, QueueMax: e.Max}
 }
+// encodeConcurrencyRecord writes the waiting entries to queue, and the head of
+// the line to pending too, for a binary that predates queue (a downgrade).
 func encodeConcurrencyRecord(r domain.CIConcurrencyRecord) []byte {
-	return mustMarshal(&storagepb.CIConcurrencyRecord{Running: encodeConcurrencyEntry(r.Running), Pending: encodeConcurrencyEntry(r.Pending)})
+	pb := &storagepb.CIConcurrencyRecord{Running: encodeConcurrencyEntry(r.Running)}
+	for i := range r.Pending {
+		pb.Queue = append(pb.Queue, encodeConcurrencyEntry(&r.Pending[i]))
+	}
+	if len(r.Pending) > 0 {
+		pb.Pending = encodeConcurrencyEntry(&r.Pending[0])
+	}
+	return mustMarshal(pb)
 }
 //hexcheck:ignore suspicious-business-logic-in-adapter maps the stored record to the domain record field by field: no rule is decided here
 	if err := proto.Unmarshal(val, &pb); err != nil {
 		return domain.CIConcurrencyRecord{}, fmt.Errorf("decode concurrency group: %w", err)
 	}
-	return domain.CIConcurrencyRecord{Running: decodeConcurrencyEntry(pb.GetRunning()), Pending: decodeConcurrencyEntry(pb.GetPending())}, nil
+	rec := domain.CIConcurrencyRecord{Running: decodeConcurrencyEntry(pb.GetRunning())}
+	for _, q := range pb.GetQueue() {
+		rec.Pending = append(rec.Pending, *decodeConcurrencyEntry(q))
+	}
+	// A record stored before queue: max holds its only waiting entry in pending.
+	if p := decodeConcurrencyEntry(pb.GetPending()); p != nil && len(rec.Pending) == 0 {
+		rec.Pending = []domain.CIConcurrencyEntry{*p}
+	}
+	return rec, nil
 }
 func decodeConcurrencyEntry(pb *storagepb.CIConcurrencyEntry) *domain.CIConcurrencyEntry {
 	if pb == nil {
 		return nil
 	}
-	return &domain.CIConcurrencyEntry{RunID: pb.GetRunId(), Attempt: pb.GetAttempt(), JobID: pb.GetJobId(), Cancel: pb.GetCancel(), Canceling: pb.GetCanceling()}
+	return &domain.CIConcurrencyEntry{RunID: pb.GetRunId(), Attempt: pb.GetAttempt(), JobID: pb.GetJobId(), Cancel: pb.GetCancel(), Canceling: pb.GetCanceling(), Max: pb.GetQueueMax()}
 }
 // concurMarkPrefix is where the marks of a group are: the group key is escaped,
 func encodeConcurrencyMark(m domain.CIConcurrencyMark) []byte {
 	pb := &storagepb.CIConcurrencyMark{Entry: encodeConcurrencyEntry(&m.Entry), Group: m.Group}
-	for _, e := range concurrencyStates {
+	for _, e := range markStates {
 		if e.d == m.State {
 			pb.State = e.p
 		}
 	}
-	if m.State == domain.CIConcurrencyReplaced {
+	if m.State == domain.CIConcurrencyRejected {
-		pb.State = storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED
+		pb.State, pb.QueueFull = storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED, true
 	}
 	return mustMarshal(pb)
 }
 	if e := decodeConcurrencyEntry(pb.GetEntry()); e != nil {
 		m.Entry = *e
 	}
-	switch pb.GetState() {
+	for _, e := range markStates {
-	case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING:
+		if e.p == pb.GetState() {
-		m.State = domain.CIConcurrencyPending
+			m.State = e.d
-	case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING:
+		}
-		m.State = domain.CIConcurrencyHolding
+	}
-	case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED:
+	if m.State == domain.CIConcurrencyReplaced && pb.GetQueueFull() {
-		m.State = domain.CIConcurrencyReplaced
+		m.State = domain.CIConcurrencyRejected
 	}
 	return m, nil
 }

M internal/adapters/kvstore/ci_concurrency_test.go +78 -14


@@ -9,7 +9,10 @@ import (
9	9
10	10		"github.com/stretchr/testify/assert"
11	11		"github.com/stretchr/testify/require"
	12	+	"google.golang.org/protobuf/proto"
12	13
	14	+	storagepb "bnema.dev/ermite/gen/ermite/storage"
	15	+	"bnema.dev/ermite/internal/adapters/kvstore"
13	16		"bnema.dev/ermite/internal/domain"
14	17		)
15	18

@@ -47,14 +50,14 @@ func markOf(e domain.CIConcurrencyEntry, state domain.CIConcurrencyState) domain
47	50		func TestCIConcurrencyGroupRecord(t *testing.T) {
48	51		_, ci, n := ciRepo(t)
49	52		update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
50		-	s.Record.Running, s.Record.Pending = entry(1), &domain.CIConcurrencyEntry{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}
	53	+	s.Record.Running, s.Record.Pending = entry(1), []domain.CIConcurrencyEntry{{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}}
51	54		})
52	55		got := read(t, ci, n, "DEPLOY")
53	56		require.NotNil(t, got.Record.Running)
54		-	require.NotNil(t, got.Record.Pending)
	57	+	require.Len(t, got.Record.Pending, 1)
55	58		assert.EqualValues(t, 1, got.Record.Running.RunID)
56		-	assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
57		-	assert.True(t, got.Record.Pending.Cancel)
	59	+	assert.Equal(t, "2-0-0", got.Record.Pending[0].JobID)
	60	+	assert.True(t, got.Record.Pending[0].Cancel)
58	61		assert.True(t, read(t, ci, n, "other").IsEmpty(), "another group is another record")
59	62
60	63		update(t, ci, n, "deploy", func(s domain.CIConcurrencyGroupState) { s = domain.CIConcurrencyGroupState{} })

@@ -70,7 +73,7 @@ func TestCIConcurrencyMarksAreWrittenWithTheRecord(t *testing.T) {
70	73		_, ci, n := ciRepo(t)
71	74		run, wait := entry(1), entry(2)
72	75		update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
73		-	s.Record.Running, s.Record.Pending = &run, &wait
	76	+	s.Record.Running, s.Record.Pending = &run, []domain.CIConcurrencyEntry{wait}
74	77		s.Marks = map[string]domain.CIConcurrencyMark{
75	78		run.Key(): markOf(run, domain.CIConcurrencyHolding),
76	79		wait.Key(): markOf(wait, domain.CIConcurrencyPending),

@@ -83,12 +86,13 @@ func TestCIConcurrencyMarksAreWrittenWithTheRecord(t *testing.T) {
83	86
84	87		// The waiting entry is replaced: unnamed and marked in one transaction.
85	88		update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
86		-	s.Record.Pending = entry(3)
	89	+	s.Record.Pending = []domain.CIConcurrencyEntry{*entry(3)}
87	90		s.Marks[wait.Key()] = markOf(wait, domain.CIConcurrencyReplaced)
88	91		s.Marks[entry(3).Key()] = markOf(*entry(3), domain.CIConcurrencyPending)
89	92		})
90	93		got = read(t, ci, n, "Deploy")
91		-	assert.EqualValues(t, 3, got.Record.Pending.RunID)
	94	+	require.Len(t, got.Record.Pending, 1)
	95	+	assert.EqualValues(t, 3, got.Record.Pending[0].RunID)
92	96		assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[wait.Key()].State)
93	97		assert.Len(t, got.Marks, 3)
94	98

@@ -196,7 +200,7 @@ func TestCIConcurrencyAMarkChangeIsGatedByTheRecord(t *testing.T) {
196	200		require.NoError(t, ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
197	201		calls++
198	202		if calls == 1 {
199		-	update(t, ci, n, "g", func(o *domain.CIConcurrencyGroupState) { o.Record.Pending = entry(2) })
	203	+	update(t, ci, n, "g", func(o domain.CIConcurrencyGroupState) { o.Record.Pending = []domain.CIConcurrencyEntry{entry(2)} })
200	204		}
201	205		m := s.Marks[entry(1).Key()]
202	206		m.State = domain.CIConcurrencyReplaced

@@ -205,7 +209,7 @@ func TestCIConcurrencyAMarkChangeIsGatedByTheRecord(t *testing.T) {
205	209		}))
206	210		assert.Equal(t, 2, calls, "the first attempt was made on a stale record")
207	211		got := read(t, ci, n, "g")
208		-	require.NotNil(t, got.Record.Pending, "the concurrent change is kept")
	212	+	require.Len(t, got.Record.Pending, 1, "the concurrent change is kept")
209	213		assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[entry(1).Key()].State)
210	214		}
211	215

@@ -238,7 +242,9 @@ func TestCIConcurrencyUnchangedRecordKeepsItsPrecondition(t *testing.T) {
238	242		// copy that shares the entries.
239	243		func TestCIConcurrencyInPlaceMutationIsWritten(t *testing.T) {
240	244		_, ci, n := ciRepo(t)
241		-	update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) { s.Record.Running, s.Record.Pending = entry(1), entry(2) })
	245	+	update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
	246	+	s.Record.Running, s.Record.Pending = entry(1), []domain.CIConcurrencyEntry{*entry(2)}
	247	+	})
242	248
243	249		update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
244	250		s.Record.Running.Canceling = true // through the pointer, the struct is returned as given

@@ -248,12 +254,12 @@ func TestCIConcurrencyInPlaceMutationIsWritten(t *testing.T) {
248	254		assert.True(t, got.Record.Running.Canceling, "the in-place change of the running entry is stored")
249	255
250	256		update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
251		-	s.Record.Pending.JobID, s.Record.Pending.Cancel = "2-0-0", true
	257	+	s.Record.Pending[0].JobID, s.Record.Pending[0].Cancel = "2-0-0", true
252	258		})
253	259		got = read(t, ci, n, "g")
254		-	require.NotNil(t, got.Record.Pending)
255		-	assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
256		-	assert.True(t, got.Record.Pending.Cancel)
	260	+	require.Len(t, got.Record.Pending, 1)
	261	+	assert.Equal(t, "2-0-0", got.Record.Pending[0].JobID)
	262	+	assert.True(t, got.Record.Pending[0].Cancel)
257	263		}
258	264
259	265		// ListConcurrency returns every group with its repository and canonical key.

@@ -292,3 +298,61 @@ func TestCIConcurrencyGroupsAreDeletedWithTheRepository(t *testing.T) {
292	298		require.NoError(t, err)
293	299		assert.Empty(t, groups)
294	300		}
	301	+
	302	+	// queue: max: the waiting line is stored in order with the queue mode of each
	303	+	// entry, and a rejected mark is kept as such.
	304	+	func TestCIConcurrencyQueueRoundTrip(t *testing.T) {
	305	+	_, ci, n := ciRepo(t)
	306	+	line := []domain.CIConcurrencyEntry{
	307	+	{RunID: 4, Attempt: 1, Max: true},
	308	+	{RunID: 2, Attempt: 1, JobID: "2-0-0", Max: true},
	309	+	{RunID: 3, Attempt: 2},
	310	+	}
	311	+	rejected := domain.CIConcurrencyEntry{RunID: 5, Attempt: 1, Max: true}
	312	+	update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
	313	+	s.Record.Running, s.Record.Pending = entry(1), line
	314	+	s.Marks = map[string]domain.CIConcurrencyMark{rejected.Key(): markOf(rejected, domain.CIConcurrencyRejected)}
	315	+	})
	316	+	got := read(t, ci, n, "deploy")
	317	+	assert.Equal(t, line, got.Record.Pending, "the order and the queue modes are kept")
	318	+	assert.Equal(t, domain.CIConcurrencyRejected, got.Marks[rejected.Key()].State)
	319	+	assert.True(t, got.Marks[rejected.Key()].Entry.Max)
	320	+
	321	+	// Stored as replaced with queue_full: a binary that predates it still
	322	+	// cancels the entry.
	323	+	var pb storagepb.CIConcurrencyMark
	324	+	require.NoError(t, proto.Unmarshal(kvstore.EncodeConcurrencyMark(markOf(rejected, domain.CIConcurrencyRejected)), &pb))
	325	+	assert.Equal(t, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED, pb.GetState())
	326	+	assert.True(t, pb.GetQueueFull())
	327	+	}
	328	+
	329	+	// A record stored before queue: max holds its only waiting entry in the
	330	+	// pending field: it reads as a line of one, and is written back in the new
	331	+	// layout.
	332	+	func TestCIConcurrencyRecordOldLayout(t *testing.T) {
	333	+	old, err := proto.Marshal(&storagepb.CIConcurrencyRecord{
	334	+	Running: &storagepb.CIConcurrencyEntry{RunId: 1, Attempt: 1},
	335	+	Pending: &storagepb.CIConcurrencyEntry{RunId: 2, Attempt: 1, JobId: "2-0-0", Cancel: true},
	336	+	})
	337	+	require.NoError(t, err)
	338	+	rec, err := kvstore.DecodeConcurrencyRecord(old)
	339	+	require.NoError(t, err)
	340	+	require.NotNil(t, rec.Running)
	341	+	assert.EqualValues(t, 1, rec.Running.RunID)
	342	+	assert.Equal(t, []domain.CIConcurrencyEntry{{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}}, rec.Pending)
	343	+
	344	+	var pb storagepb.CIConcurrencyRecord
	345	+	require.NoError(t, proto.Unmarshal(kvstore.EncodeConcurrencyRecord(rec), &pb))
	346	+	assert.EqualValues(t, 2, pb.GetPending().GetRunId(), "the head of the line is written to the old field too, for a downgrade")
	347	+	assert.Len(t, pb.GetQueue(), 1)
	348	+
	349	+	// A line of two: an older binary sees its head only.
	350	+	rec.Pending = append(rec.Pending, domain.CIConcurrencyEntry{RunID: 3, Attempt: 1, Max: true})
	351	+	pb.Reset()
	352	+	require.NoError(t, proto.Unmarshal(kvstore.EncodeConcurrencyRecord(rec), &pb))
	353	+	assert.EqualValues(t, 2, pb.GetPending().GetRunId())
	354	+	assert.Len(t, pb.GetQueue(), 2)
	355	+	back, err := kvstore.DecodeConcurrencyRecord(kvstore.EncodeConcurrencyRecord(rec))
	356	+	require.NoError(t, err)
	357	+	assert.Equal(t, rec, back, "the queue wins over the old field")
	358	+	}

 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"google.golang.org/protobuf/proto"
+	storagepb "bnema.dev/ermite/gen/ermite/storage"
+	"bnema.dev/ermite/internal/adapters/kvstore"
 	"bnema.dev/ermite/internal/domain"
 )
 func TestCIConcurrencyGroupRecord(t *testing.T) {
 	_, ci, n := ciRepo(t)
 	update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
-		s.Record.Running, s.Record.Pending = entry(1), &domain.CIConcurrencyEntry{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}
+		s.Record.Running, s.Record.Pending = entry(1), []domain.CIConcurrencyEntry{{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}}
 	})
 	got := read(t, ci, n, "DEPLOY")
 	require.NotNil(t, got.Record.Running)
-	require.NotNil(t, got.Record.Pending)
+	require.Len(t, got.Record.Pending, 1)
 	assert.EqualValues(t, 1, got.Record.Running.RunID)
-	assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
+	assert.Equal(t, "2-0-0", got.Record.Pending[0].JobID)
-	assert.True(t, got.Record.Pending.Cancel)
+	assert.True(t, got.Record.Pending[0].Cancel)
 	assert.True(t, read(t, ci, n, "other").IsEmpty(), "another group is another record")
 	update(t, ci, n, "deploy", func(s *domain.CIConcurrencyGroupState) { *s = domain.CIConcurrencyGroupState{} })
 	_, ci, n := ciRepo(t)
 	run, wait := *entry(1), *entry(2)
 	update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
-		s.Record.Running, s.Record.Pending = &run, &wait
+		s.Record.Running, s.Record.Pending = &run, []domain.CIConcurrencyEntry{wait}
 		s.Marks = map[string]domain.CIConcurrencyMark{
 			run.Key():  markOf(run, domain.CIConcurrencyHolding),
 			wait.Key(): markOf(wait, domain.CIConcurrencyPending),
 	// The waiting entry is replaced: unnamed and marked in one transaction.
 	update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
-		s.Record.Pending = entry(3)
+		s.Record.Pending = []domain.CIConcurrencyEntry{*entry(3)}
 		s.Marks[wait.Key()] = markOf(wait, domain.CIConcurrencyReplaced)
 		s.Marks[entry(3).Key()] = markOf(*entry(3), domain.CIConcurrencyPending)
 	})
 	got = read(t, ci, n, "Deploy")
-	assert.EqualValues(t, 3, got.Record.Pending.RunID)
+	require.Len(t, got.Record.Pending, 1)
+	assert.EqualValues(t, 3, got.Record.Pending[0].RunID)
 	assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[wait.Key()].State)
 	assert.Len(t, got.Marks, 3)
 	require.NoError(t, ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
 		calls++
 		if calls == 1 {
-			update(t, ci, n, "g", func(o *domain.CIConcurrencyGroupState) { o.Record.Pending = entry(2) })
+			update(t, ci, n, "g", func(o *domain.CIConcurrencyGroupState) { o.Record.Pending = []domain.CIConcurrencyEntry{*entry(2)} })
 		}
 		m := s.Marks[entry(1).Key()]
 		m.State = domain.CIConcurrencyReplaced
 	}))
 	assert.Equal(t, 2, calls, "the first attempt was made on a stale record")
 	got := read(t, ci, n, "g")
-	require.NotNil(t, got.Record.Pending, "the concurrent change is kept")
+	require.Len(t, got.Record.Pending, 1, "the concurrent change is kept")
 	assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[entry(1).Key()].State)
 }
 // copy that shares the entries.
 func TestCIConcurrencyInPlaceMutationIsWritten(t *testing.T) {
 	_, ci, n := ciRepo(t)
-	update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) { s.Record.Running, s.Record.Pending = entry(1), entry(2) })
+	update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
+		s.Record.Running, s.Record.Pending = entry(1), []domain.CIConcurrencyEntry{*entry(2)}
+	})
 	update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
 		s.Record.Running.Canceling = true // through the pointer, the struct is returned as given
 	assert.True(t, got.Record.Running.Canceling, "the in-place change of the running entry is stored")
 	update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
-		s.Record.Pending.JobID, s.Record.Pending.Cancel = "2-0-0", true
+		s.Record.Pending[0].JobID, s.Record.Pending[0].Cancel = "2-0-0", true
 	})
 	got = read(t, ci, n, "g")
-	require.NotNil(t, got.Record.Pending)
+	require.Len(t, got.Record.Pending, 1)
-	assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
+	assert.Equal(t, "2-0-0", got.Record.Pending[0].JobID)
-	assert.True(t, got.Record.Pending.Cancel)
+	assert.True(t, got.Record.Pending[0].Cancel)
 }
 // ListConcurrency returns every group with its repository and canonical key.
 	require.NoError(t, err)
 	assert.Empty(t, groups)
 }
+// queue: max: the waiting line is stored in order with the queue mode of each
+// entry, and a rejected mark is kept as such.
+func TestCIConcurrencyQueueRoundTrip(t *testing.T) {
+	_, ci, n := ciRepo(t)
+	line := []domain.CIConcurrencyEntry{
+		{RunID: 4, Attempt: 1, Max: true},
+		{RunID: 2, Attempt: 1, JobID: "2-0-0", Max: true},
+		{RunID: 3, Attempt: 2},
+	}
+	rejected := domain.CIConcurrencyEntry{RunID: 5, Attempt: 1, Max: true}
+	update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
+		s.Record.Running, s.Record.Pending = entry(1), line
+		s.Marks = map[string]domain.CIConcurrencyMark{rejected.Key(): markOf(rejected, domain.CIConcurrencyRejected)}
+	})
+	got := read(t, ci, n, "deploy")
+	assert.Equal(t, line, got.Record.Pending, "the order and the queue modes are kept")
+	assert.Equal(t, domain.CIConcurrencyRejected, got.Marks[rejected.Key()].State)
+	assert.True(t, got.Marks[rejected.Key()].Entry.Max)
+	// Stored as replaced with queue_full: a binary that predates it still
+	// cancels the entry.
+	var pb storagepb.CIConcurrencyMark
+	require.NoError(t, proto.Unmarshal(kvstore.EncodeConcurrencyMark(markOf(rejected, domain.CIConcurrencyRejected)), &pb))
+	assert.Equal(t, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED, pb.GetState())
+	assert.True(t, pb.GetQueueFull())
+}
+// A record stored before queue: max holds its only waiting entry in the
+// pending field: it reads as a line of one, and is written back in the new
+// layout.
+func TestCIConcurrencyRecordOldLayout(t *testing.T) {
+	old, err := proto.Marshal(&storagepb.CIConcurrencyRecord{
+		Running: &storagepb.CIConcurrencyEntry{RunId: 1, Attempt: 1},
+		Pending: &storagepb.CIConcurrencyEntry{RunId: 2, Attempt: 1, JobId: "2-0-0", Cancel: true},
+	})
+	require.NoError(t, err)
+	rec, err := kvstore.DecodeConcurrencyRecord(old)
+	require.NoError(t, err)
+	require.NotNil(t, rec.Running)
+	assert.EqualValues(t, 1, rec.Running.RunID)
+	assert.Equal(t, []domain.CIConcurrencyEntry{{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}}, rec.Pending)
+	var pb storagepb.CIConcurrencyRecord
+	require.NoError(t, proto.Unmarshal(kvstore.EncodeConcurrencyRecord(rec), &pb))
+	assert.EqualValues(t, 2, pb.GetPending().GetRunId(), "the head of the line is written to the old field too, for a downgrade")
+	assert.Len(t, pb.GetQueue(), 1)
+	// A line of two: an older binary sees its head only.
+	rec.Pending = append(rec.Pending, domain.CIConcurrencyEntry{RunID: 3, Attempt: 1, Max: true})
+	pb.Reset()
+	require.NoError(t, proto.Unmarshal(kvstore.EncodeConcurrencyRecord(rec), &pb))
+	assert.EqualValues(t, 2, pb.GetPending().GetRunId())
+	assert.Len(t, pb.GetQueue(), 2)
+	back, err := kvstore.DecodeConcurrencyRecord(kvstore.EncodeConcurrencyRecord(rec))
+	require.NoError(t, err)
+	assert.Equal(t, rec, back, "the queue wins over the old field")
+}

M internal/adapters/kvstore/export_test.go +8 -0


@@ -8,3 +8,11 @@ func SetAfterSettingsSnapshot(f func()) (restore func()) {
8	8		afterSettingsSnapshot = f
9	9		return func() { afterSettingsSnapshot = nil }
10	10		}
	11	+
	12	+	// DecodeConcurrencyRecord, EncodeConcurrencyRecord and EncodeConcurrencyMark
	13	+	// expose the stored layout of a concurrency group to the tests.
	14	+	var (
	15	+	DecodeConcurrencyRecord = decodeConcurrencyRecord
	16	+	EncodeConcurrencyRecord = encodeConcurrencyRecord
	17	+	EncodeConcurrencyMark = encodeConcurrencyMark
	18	+	)


@@ -8,3 +8,11 @@ func SetAfterSettingsSnapshot(f func()) (restore func()) {
8	afterSettingsSnapshot = f	8		afterSettingsSnapshot = f
9	return func() { afterSettingsSnapshot = nil }	9		return func() { afterSettingsSnapshot = nil }
10	}	10		}
		11	+
		12	+	// DecodeConcurrencyRecord, EncodeConcurrencyRecord and EncodeConcurrencyMark
		13	+	// expose the stored layout of a concurrency group to the tests.
		14	+	var (
		15	+	DecodeConcurrencyRecord = decodeConcurrencyRecord
		16	+	EncodeConcurrencyRecord = encodeConcurrencyRecord
		17	+	EncodeConcurrencyMark = encodeConcurrencyMark
		18	+	)

M internal/app/ci_concurrency_test.go +165 -29


@@ -267,8 +267,8 @@ func TestCIConcurrencyFreedByLostJob(t *testing.T) {
267	267		assert.Equal(t, r2.GetId(), job2.GetRunId())
268	268		}
269	269
270		-	// A group that evaluates to nothing fails the job as GitHub does, and
271		-	// queue: max is refused.
	270	+	// A group that evaluates to nothing fails the job as GitHub does, and so does
	271	+	// queue: max with cancel-in-progress, literal or evaluated to true.
272	272		func TestCIConcurrencyErrors(t *testing.T) {
273	273		c, runner := concurrencyHarness(t)
274	274		_ = runner

@@ -276,18 +276,24 @@ func TestCIConcurrencyErrors(t *testing.T) {
276	276		c.waitJobs("failed", r1, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
277	277		assert.Contains(t, c.jobsOf(r1)[0].GetError(), "concurrency group name cannot be empty")
278	278
	279	+	// The literal pair is a workflow validation error (GitHub's docs): the run
	280	+	// fails with no job, like an invalid workflow file, at job and workflow level.
279	281		r3 := c.pushN(3, workflowAt("", " concurrency:\n group: g\n queue: max\n cancel-in-progress: true\n"))
280		-	c.waitJobs("failed", r3, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
281		-	assert.Equal(t, "'queue: max' cannot be combined with 'cancel-in-progress: true'.", c.jobsOf(r3)[0].GetError())
282		-
283		-	r2 := c.pushN(2, workflowAt("", " concurrency:\n group: g\n queue: max\n"))
	282	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, r3.GetConclusion())
	283	+	assert.Empty(t, c.jobsOf(r3))
	284	+	assert.Equal(t, ".github/workflows/ci.yml (Line: 12, Col: 27): 'queue: max' cannot be combined with 'cancel-in-progress: true'.", c.getRun(r3.GetId()).GetRun().GetError())
	285	+	r4 := c.pushN(4, workflowAt("concurrency: {group: g, queue: max, cancel-in-progress: true}\n", ""))
	286	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, r4.GetConclusion())
	287	+	assert.Contains(t, c.getRun(r4.GetId()).GetRun().GetError(), "'queue: max' cannot be combined with 'cancel-in-progress: true'.")
	288	+
	289	+	r2 := c.pushN(2, workflowAt("", " concurrency:\n group: g\n queue: max\n cancel-in-progress: ${{ github.ref == 'refs/heads/main' }}\n"))
284	290		c.waitJobs("failed", r2, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
285		-	assert.Equal(t, "not supported yet: queue: max", strings.TrimSpace(c.jobsOf(r2)[0].GetError()))
	291	+	assert.Equal(t, "'queue: max' cannot be combined with 'cancel-in-progress: true'.", strings.TrimSpace(c.jobsOf(r2)[0].GetError()))
286	292		}
287	293
288	294		// holdGroup writes a group behind the use case's back, as a crash would leave
289	295		// it: the entries are named and marked whatever the state of their runs.
290		-	func (c ciHarness) holdGroup(group string, running, pending domain.CIConcurrencyEntry) {
	296	+	func (c ciHarness) holdGroup(group string, running domain.CIConcurrencyEntry, pending ...domain.CIConcurrencyEntry) {
291	297		c.t.Helper()
292	298		require.NoError(c.t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), group, func(st *domain.CIConcurrencyGroupState) error {
293	299		st.Record.Running, st.Record.Pending = running, pending

@@ -295,8 +301,8 @@ func (c ciHarness) holdGroup(group string, running, pending domain.CIConcurren
295	301		if running != nil {
296	302		st.Marks[running.Key()] = domain.CIConcurrencyMark{Entry: *running, Group: group, State: domain.CIConcurrencyHolding}
297	303		}
298		-	if pending != nil {
299		-	st.Marks[pending.Key()] = domain.CIConcurrencyMark{Entry: *pending, Group: group, State: domain.CIConcurrencyPending}
	304	+	for _, p := range pending {
	305	+	st.Marks[p.Key()] = domain.CIConcurrencyMark{Entry: p, Group: group, State: domain.CIConcurrencyPending}
300	306		}
301	307		return nil
302	308		}))

@@ -329,7 +335,7 @@ func TestCIConcurrencyHealsAfterACrash(t *testing.T) {
329	335		runner.runJob(runner.await(), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
330	336		c.waitFor("the first run completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
331	337		c.waitFor("the group is released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
332		-	c.holdGroup("Deploy", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil) // the release that did not happen
	338	+	c.holdGroup("Deploy", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}) // the release that did not happen
333	339
334	340		r2 := c.pushN(2, workflowAt(wfGroup, ""))
335	341		job2 := runner.await()

@@ -348,7 +354,7 @@ func TestCIConcurrencyHealsAfterRunDeletion(t *testing.T) {
348	354		c.waitFor("completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
349	355		c.waitFor("released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
350	356		ghost := &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}
351		-	c.holdGroup("Deploy", ghost, nil)
	357	+	c.holdGroup("Deploy", ghost)
352	358		ids, err := c.forge.store.CI().Sweep(c.ctx, time.Now().Add(time.Hour))
353	359		require.NoError(t, err)
354	360		require.Contains(t, ids, r1.GetId(), "the run is deleted")

@@ -415,8 +421,8 @@ func TestCIConcurrencyCancelSurvivesAFailedDelivery(t *testing.T) {
415	421		r3 := c.pushN(3, workflowAt(wfGroup, ""))
416	422		c.waitFor("the first delivery failed", func() bool { return fs.hits.Load() == 1 })
417	423		rec := c.groupRecord("Deploy")
418		-	require.NotNil(t, rec.Pending)
419		-	assert.Equal(t, r3.GetId(), rec.Pending.RunID)
	424	+	require.Len(t, rec.Pending, 1)
	425	+	assert.Equal(t, r3.GetId(), rec.Pending[0].RunID)
420	426		c.waitFor("the replaced run is cancelled by the retry", func() bool { return c.conclusionOf(r2) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
421	427		assert.True(t, c.cancelAnnotation(r2, "Deploy"))
422	428		runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)

@@ -449,7 +455,7 @@ func TestCIConcurrencyReconcileCancelsEveryReplacedVictim(t *testing.T) {
449	455		next := c.pushN(3+i, workflowAt(wfGroup, ""))
450	456		c.waitFor("the newer run takes the waiting place", func() bool {
451	457		rec := c.groupRecord("Deploy")
452		-	return rec.Pending != nil && rec.Pending.RunID == next.GetId()
	458	+	return len(rec.Pending) == 1 && rec.Pending[0].RunID == next.GetId()
453	459		})
454	460		c.waitJobs("the newer run is pending", next, allStatus(pb.CiStatus_CI_STATUS_PENDING))
455	461		victims = append(victims, prev)

@@ -539,9 +545,9 @@ func TestCIConcurrencyCrashBetweenGroupAndEntry(t *testing.T) {
539	545		_, err = store.Apply(c.ctx, ports.CIWrite{Runs: []domain.CIRun{run}, Jobs: jobs})
540	546		require.NoError(t, err)
541	547		rec := c.groupRecord("Deploy")
542		-	require.NotNil(t, rec.Pending)
543		-	require.Equal(t, r2.GetId(), rec.Pending.RunID, "the group names the run")
544		-	assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", *rec.Pending), "and marks it, in the same batch")
	548	+	require.Len(t, rec.Pending, 1)
	549	+	require.Equal(t, r2.GetId(), rec.Pending[0].RunID, "the group names the run")
	550	+	assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", rec.Pending[0]), "and marks it, in the same batch")
545	551
546	552		c.forge.CI.ReconcileConcurrency(c.ctx)
547	553		c.waitFor("the entry is marked pending again", func() bool {

@@ -551,8 +557,8 @@ func TestCIConcurrencyCrashBetweenGroupAndEntry(t *testing.T) {
551	557		time.Sleep(200 * time.Millisecond)
552	558		assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2), "a named entry is not a replaced one")
553	559		rec = c.groupRecord("Deploy")
554		-	require.NotNil(t, rec.Pending)
555		-	assert.Equal(t, r2.GetId(), rec.Pending.RunID)
	560	+	require.Len(t, rec.Pending, 1)
	561	+	assert.Equal(t, r2.GetId(), rec.Pending[0].RunID)
556	562
557	563		runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
558	564		assert.Equal(t, r2.GetId(), runner.await().GetRunId())

@@ -713,7 +719,7 @@ func TestCIConcurrencyReconcileAtRestart(t *testing.T) {
713	719		run.Status, run.Conclusion, run.CompletedAt = domain.CICompleted, domain.CISuccess, time.Now()
714	720		_, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs, Runs: []domain.CIRun{run}})
715	721		require.NoError(t, err)
716		-	c.holdGroup("Stale", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil)
	722	+	c.holdGroup("Stale", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1})
717	723		rec := c.groupRecord("Deploy")
718	724		require.NotNil(t, rec.Running)
719	725		require.Equal(t, r1.GetId(), rec.Running.RunID, "the record still holds the ended run")

@@ -745,7 +751,7 @@ func TestCIConcurrencyReconcileAtRestart(t *testing.T) {
745	751		assert.Equal(t, "DEPLOY", groups[0].Group)
746	752		require.NotNil(t, groups[0].State.Record.Running)
747	753		assert.Equal(t, r2.GetId(), groups[0].State.Record.Running.RunID)
748		-	assert.Nil(t, groups[0].State.Record.Pending)
	754	+	assert.Empty(t, groups[0].State.Record.Pending)
749	755		}
750	756
751	757		// Both writes of a run's own record fail when it queues (its "pending" state),

@@ -776,7 +782,7 @@ func TestCIConcurrencyReplacedRunWithLostWritesIsCancelled(t *testing.T) {
776	782		b := c.pushN(2, workflowAt(wfGroup, ""))
777	783		c.waitFor("B is named by the group", func() bool {
778	784		rec := c.groupRecord("Deploy")
779		-	return rec.Pending != nil && rec.Pending.RunID == b.GetId()
	785	+	return len(rec.Pending) == 1 && rec.Pending[0].RunID == b.GetId()
780	786		})
781	787		bEntry := domain.CIConcurrencyEntry{RunID: b.GetId(), Attempt: 1}
782	788		assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", bEntry), "the group batch marked it")

@@ -785,8 +791,8 @@ func TestCIConcurrencyReplacedRunWithLostWritesIsCancelled(t *testing.T) {
785	791		c.pushN(3, workflowAt(wfGroup, ""))
786	792		c.waitFor("B is marked replaced", func() bool { return c.markState("Deploy", bEntry) == domain.CIConcurrencyReplaced })
787	793		rec := c.groupRecord("Deploy")
788		-	require.NotNil(t, rec.Pending)
789		-	assert.NotEqual(t, b.GetId(), rec.Pending.RunID, "B is no longer named")
	794	+	require.Len(t, rec.Pending, 1)
	795	+	assert.NotEqual(t, b.GetId(), rec.Pending[0].RunID, "B is no longer named")
790	796		assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(b))
791	797
792	798		fs.fails.Store(0)

@@ -844,8 +850,8 @@ func TestCIJobConcurrencyReplacedJobNeverAcquiresAgain(t *testing.T) {
844	850		rec = c.groupRecord("jobgroup")
845	851		require.NotNil(t, rec.Running)
846	852		assert.Equal(t, rC.GetId(), rec.Running.RunID, "C still holds the group")
847		-	if rec.Pending != nil {
848		-	assert.NotEqual(t, rB.GetId(), rec.Pending.RunID, "B did not take the waiting place back")
	853	+	if len(rec.Pending) > 0 {
	854	+	assert.NotEqual(t, rB.GetId(), rec.Pending[0].RunID, "B did not take the waiting place back")
849	855		}
850	856		// Nothing of B's grouped job is given to a runner.
851	857		for range 5 {

@@ -943,8 +949,8 @@ func TestCIJobConcurrencyCrashBetweenGroupAndJob(t *testing.T) {
943	949		_, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs})
944	950		require.NoError(t, err)
945	951		rec := c.groupRecord("jobgroup")
946		-	require.NotNil(t, rec.Pending)
947		-	assert.Equal(t, entry.JobID, rec.Pending.JobID, "the group names the job")
	952	+	require.Len(t, rec.Pending, 1)
	953	+	assert.Equal(t, entry.JobID, rec.Pending[0].JobID, "the group names the job")
948	954		assert.Equal(t, domain.CIConcurrencyPending, c.markState("jobgroup", entry), "and its mark says so")
949	955
950	956		c.forge.CI.ReconcileConcurrency(c.ctx)

@@ -1110,3 +1116,133 @@ func TestCIConcurrencyPrunesStaleMarksInBoundedBatches(t *testing.T) {
1110	1116		assert.Contains(t, st.Marks, domain.CIConcurrencyEntry{RunID: r.GetId(), Attempt: 1}.Key())
1111	1117		runner.runJob(job, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1112	1118		}
	1119	+
	1120	+	func (c ciHarness) hasAnnotation(run pb.Run, msg string) bool {
	1121	+	for _, j := range c.jobsOf(run) {
	1122	+	for _, a := range c.annotations(run.GetId(), j.GetId()).GetAnnotations() {
	1123	+	if a.GetAnnotation().GetMessage() == msg {
	1124	+	return true
	1125	+	}
	1126	+	}
	1127	+	}
	1128	+	return false
	1129	+	}
	1130	+
	1131	+	// pendingRuns is the waiting line of a group, as run ids.
	1132	+	func (c *ciHarness) pendingRuns(group string) []int64 {
	1133	+	var ids []int64
	1134	+	for _, p := range c.groupRecord(group).Pending {
	1135	+	ids = append(ids, p.RunID)
	1136	+	}
	1137	+	return ids
	1138	+	}
	1139	+
	1140	+	func (c ciHarness) completed(run pb.Run) bool {
	1141	+	return c.getRun(run.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
	1142	+	}
	1143	+
	1144	+	// queue: max at the workflow and at the job level: three runs of one group
	1145	+	// run one after the other in the order they started waiting, none cancelled.
	1146	+	func TestCIConcurrencyQueueMaxRunsInOrder(t *testing.T) {
	1147	+	for _, tt := range []struct{ name, top, job string }{
	1148	+	{"workflow", "concurrency:\n group: Deploy\n queue: max\n", ""},
	1149	+	{"job", "", " concurrency:\n group: Deploy\n queue: max\n"},
	1150	+	} {
	1151	+	t.Run(tt.name, func(t *testing.T) {
	1152	+	c, runner := concurrencyHarness(t)
	1153	+	wf := workflowAt(tt.top, tt.job)
	1154	+	r1 := c.pushN(1, wf)
	1155	+	job1 := runner.await()
	1156	+	require.Equal(t, r1.GetId(), job1.GetRunId())
	1157	+	r2 := c.pushN(2, wf)
	1158	+	c.waitJobs("the second run is pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
	1159	+	r3 := c.pushN(3, wf)
	1160	+	c.waitJobs("the third run is pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
	1161	+	assert.Equal(t, []int64{r2.GetId(), r3.GetId()}, c.pendingRuns("Deploy"), "both wait, in order")
	1162	+	rec := c.groupRecord("Deploy")
	1163	+	assert.True(t, rec.Running.Max && rec.Pending[0].Max && rec.Pending[1].Max, "each entry records its queue mode")
	1164	+
	1165	+	runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1166	+	job2 := runner.await()
	1167	+	assert.Equal(t, r2.GetId(), job2.GetRunId(), "the head of the line goes first")
	1168	+	assert.Equal(t, []int64{r3.GetId()}, c.pendingRuns("Deploy"))
	1169	+	runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1170	+	job3 := runner.await()
	1171	+	assert.Equal(t, r3.GetId(), job3.GetRunId())
	1172	+	runner.runJob(job3, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1173	+	for _, r := range []*pb.Run{r1, r2, r3} {
	1174	+	c.waitFor("the run completes", func() bool { return c.completed(r) })
	1175	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_SUCCESS, c.conclusionOf(r))
	1176	+	}
	1177	+	c.waitFor("the group is free", func() bool { return c.groupRecord("Deploy").IsEmpty() })
	1178	+	})
	1179	+	}
	1180	+	}
	1181	+
	1182	+	// A queue: max entry that finds the line full is cancelled with ermite's
	1183	+	// annotation; the waiting ones stay. The cap is lowered to 2 here (it is
	1184	+	// domain.CIMaxConcurrencyQueue, 100, by default; see the domain tests).
	1185	+	func TestCIConcurrencyQueueMaxFullLineCancelsTheNewcomer(t *testing.T) {
	1186	+	c := newCIHarness(t, func(cfg *usecase.CIConfig) {
	1187	+	cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300time.Millisecond, 20time.Millisecond, 100*time.Millisecond
	1188	+	cfg.ConcurrencyQueueMax = 2
	1189	+	})
	1190	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	1191	+	wf := workflowAt("concurrency:\n group: Deploy\n queue: max\n", "")
	1192	+	r1 := c.pushN(1, wf)
	1193	+	job1 := runner.await()
	1194	+	r2 := c.pushN(2, wf)
	1195	+	c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
	1196	+	r3 := c.pushN(3, wf)
	1197	+	c.waitJobs("pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
	1198	+
	1199	+	r4 := c.pushN(4, wf)
	1200	+	c.waitFor("the newcomer is cancelled", func() bool { return c.completed(r4) })
	1201	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r4))
	1202	+	assert.True(t, c.hasAnnotation(r4, domain.ConcurrencyQueueFullMessage("Deploy")))
	1203	+	assert.False(t, c.cancelAnnotation(r4, "Deploy"))
	1204	+	assert.Equal(t, []int64{r2.GetId(), r3.GetId()}, c.pendingRuns("Deploy"), "the line is kept")
	1205	+	assert.False(t, c.groupRecord("Deploy").Names(domain.CIConcurrencyEntry{RunID: r4.GetId(), Attempt: 1}), "the newcomer was never named")
	1206	+	assert.False(t, c.completed(r1))
	1207	+	assert.False(t, c.completed(r2))
	1208	+	assert.False(t, c.completed(r3))
	1209	+
	1210	+	runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1211	+	job2 := runner.await()
	1212	+	assert.Equal(t, r2.GetId(), job2.GetRunId())
	1213	+	runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1214	+	job3 := runner.await()
	1215	+	assert.Equal(t, r3.GetId(), job3.GetRunId())
	1216	+	runner.runJob(job3, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1217	+	c.waitFor("the last run completes", func() bool { return c.completed(r3) })
	1218	+	}
	1219	+
	1220	+	// ermite's rule for a mixed group: a queue: single entry that starts waiting
	1221	+	// cancels every waiting entry, queue: max ones included, and is then the only
	1222	+	// one; the running entry is left alone (no cancel-in-progress).
	1223	+	func TestCIConcurrencyQueueSingleCancelsTheMaxLine(t *testing.T) {
	1224	+	c, runner := concurrencyHarness(t)
	1225	+	maxWF := workflowAt("concurrency:\n group: Deploy\n queue: max\n", "")
	1226	+	r1 := c.pushN(1, maxWF)
	1227	+	job1 := runner.await()
	1228	+	r2 := c.pushN(2, maxWF)
	1229	+	c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
	1230	+	r3 := c.pushN(3, maxWF)
	1231	+	c.waitJobs("pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
	1232	+
	1233	+	r4 := c.pushN(4, workflowAt(wfGroup, ""))
	1234	+	c.waitJobs("the single run is pending", r4, allStatus(pb.CiStatus_CI_STATUS_PENDING))
	1235	+	for _, r := range []*pb.Run{r2, r3} {
	1236	+	c.waitFor("the max run is cancelled", func() bool { return c.completed(r) })
	1237	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r))
	1238	+	assert.True(t, c.cancelAnnotation(r, "Deploy"))
	1239	+	}
	1240	+	assert.Equal(t, []int64{r4.GetId()}, c.pendingRuns("Deploy"))
	1241	+	assert.False(t, c.completed(r1), "the running run is left alone")
	1242	+
	1243	+	runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1244	+	job4 := runner.await()
	1245	+	assert.Equal(t, r4.GetId(), job4.GetRunId())
	1246	+	runner.runJob(job4, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
	1247	+	c.waitFor("the single run completes", func() bool { return c.completed(r4) })
	1248	+	}


@@ -267,8 +267,8 @@ func TestCIConcurrencyFreedByLostJob(t *testing.T) {
267		assert.Equal(t, r2.GetId(), job2.GetRunId())	267		assert.Equal(t, r2.GetId(), job2.GetRunId())
268		}	268		}
269			269
270	-	// A group that evaluates to nothing fails the job as GitHub does, and	270	+	// A group that evaluates to nothing fails the job as GitHub does, and so does
271	-	// queue: max is refused.	271	+	// queue: max with cancel-in-progress, literal or evaluated to true.
272		func TestCIConcurrencyErrors(t *testing.T) {	272		func TestCIConcurrencyErrors(t *testing.T) {
273		c, runner := concurrencyHarness(t)	273		c, runner := concurrencyHarness(t)
274		_ = runner	274		_ = runner

@@ -276,18 +276,24 @@ func TestCIConcurrencyErrors(t *testing.T) {
276		c.waitJobs("failed", r1, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))	276		c.waitJobs("failed", r1, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
277		assert.Contains(t, c.jobsOf(r1)[0].GetError(), "concurrency group name cannot be empty")	277		assert.Contains(t, c.jobsOf(r1)[0].GetError(), "concurrency group name cannot be empty")
278			278
			279	+	// The literal pair is a workflow validation error (GitHub's docs): the run
			280	+	// fails with no job, like an invalid workflow file, at job and workflow level.
279		r3 := c.pushN(3, workflowAt("", " concurrency:\n group: g\n queue: max\n cancel-in-progress: true\n"))	281		r3 := c.pushN(3, workflowAt("", " concurrency:\n group: g\n queue: max\n cancel-in-progress: true\n"))
280	-	c.waitJobs("failed", r3, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))	282	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, r3.GetConclusion())
281	-	assert.Equal(t, "'queue: max' cannot be combined with 'cancel-in-progress: true'.", c.jobsOf(r3)[0].GetError())	283	+	assert.Empty(t, c.jobsOf(r3))
282	-		284	+	assert.Equal(t, ".github/workflows/ci.yml (Line: 12, Col: 27): 'queue: max' cannot be combined with 'cancel-in-progress: true'.", c.getRun(r3.GetId()).GetRun().GetError())
283	-	r2 := c.pushN(2, workflowAt("", " concurrency:\n group: g\n queue: max\n"))	285	+	r4 := c.pushN(4, workflowAt("concurrency: {group: g, queue: max, cancel-in-progress: true}\n", ""))
			286	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, r4.GetConclusion())
			287	+	assert.Contains(t, c.getRun(r4.GetId()).GetRun().GetError(), "'queue: max' cannot be combined with 'cancel-in-progress: true'.")
			288	+
			289	+	r2 := c.pushN(2, workflowAt("", " concurrency:\n group: g\n queue: max\n cancel-in-progress: ${{ github.ref == 'refs/heads/main' }}\n"))
284		c.waitJobs("failed", r2, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))	290		c.waitJobs("failed", r2, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
285	-	assert.Equal(t, "not supported yet: queue: max", strings.TrimSpace(c.jobsOf(r2)[0].GetError()))	291	+	assert.Equal(t, "'queue: max' cannot be combined with 'cancel-in-progress: true'.", strings.TrimSpace(c.jobsOf(r2)[0].GetError()))
286		}	292		}
287			293
288		// holdGroup writes a group behind the use case's back, as a crash would leave	294		// holdGroup writes a group behind the use case's back, as a crash would leave
289		// it: the entries are named and marked whatever the state of their runs.	295		// it: the entries are named and marked whatever the state of their runs.
290	-	func (c ciHarness) holdGroup(group string, running, pending domain.CIConcurrencyEntry) {	296	+	func (c ciHarness) holdGroup(group string, running domain.CIConcurrencyEntry, pending ...domain.CIConcurrencyEntry) {
291		c.t.Helper()	297		c.t.Helper()
292		require.NoError(c.t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), group, func(st *domain.CIConcurrencyGroupState) error {	298		require.NoError(c.t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), group, func(st *domain.CIConcurrencyGroupState) error {
293		st.Record.Running, st.Record.Pending = running, pending	299		st.Record.Running, st.Record.Pending = running, pending

@@ -295,8 +301,8 @@ func (c ciHarness) holdGroup(group string, running, pending domain.CIConcurren
295		if running != nil {	301		if running != nil {
296		st.Marks[running.Key()] = domain.CIConcurrencyMark{Entry: *running, Group: group, State: domain.CIConcurrencyHolding}	302		st.Marks[running.Key()] = domain.CIConcurrencyMark{Entry: *running, Group: group, State: domain.CIConcurrencyHolding}
297		}	303		}
298	-	if pending != nil {	304	+	for _, p := range pending {
299	-	st.Marks[pending.Key()] = domain.CIConcurrencyMark{Entry: *pending, Group: group, State: domain.CIConcurrencyPending}	305	+	st.Marks[p.Key()] = domain.CIConcurrencyMark{Entry: p, Group: group, State: domain.CIConcurrencyPending}
300		}	306		}
301		return nil	307		return nil
302		}))	308		}))

@@ -329,7 +335,7 @@ func TestCIConcurrencyHealsAfterACrash(t *testing.T) {
329		runner.runJob(runner.await(), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)	335		runner.runJob(runner.await(), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
330		c.waitFor("the first run completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })	336		c.waitFor("the first run completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
331		c.waitFor("the group is released", func() bool { return c.groupRecord("Deploy").IsEmpty() })	337		c.waitFor("the group is released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
332	-	c.holdGroup("Deploy", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil) // the release that did not happen	338	+	c.holdGroup("Deploy", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}) // the release that did not happen
333			339
334		r2 := c.pushN(2, workflowAt(wfGroup, ""))	340		r2 := c.pushN(2, workflowAt(wfGroup, ""))
335		job2 := runner.await()	341		job2 := runner.await()

@@ -348,7 +354,7 @@ func TestCIConcurrencyHealsAfterRunDeletion(t *testing.T) {
348		c.waitFor("completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })	354		c.waitFor("completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
349		c.waitFor("released", func() bool { return c.groupRecord("Deploy").IsEmpty() })	355		c.waitFor("released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
350		ghost := &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}	356		ghost := &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}
351	-	c.holdGroup("Deploy", ghost, nil)	357	+	c.holdGroup("Deploy", ghost)
352		ids, err := c.forge.store.CI().Sweep(c.ctx, time.Now().Add(time.Hour))	358		ids, err := c.forge.store.CI().Sweep(c.ctx, time.Now().Add(time.Hour))
353		require.NoError(t, err)	359		require.NoError(t, err)
354		require.Contains(t, ids, r1.GetId(), "the run is deleted")	360		require.Contains(t, ids, r1.GetId(), "the run is deleted")

@@ -415,8 +421,8 @@ func TestCIConcurrencyCancelSurvivesAFailedDelivery(t *testing.T) {
415		r3 := c.pushN(3, workflowAt(wfGroup, ""))	421		r3 := c.pushN(3, workflowAt(wfGroup, ""))
416		c.waitFor("the first delivery failed", func() bool { return fs.hits.Load() == 1 })	422		c.waitFor("the first delivery failed", func() bool { return fs.hits.Load() == 1 })
417		rec := c.groupRecord("Deploy")	423		rec := c.groupRecord("Deploy")
418	-	require.NotNil(t, rec.Pending)	424	+	require.Len(t, rec.Pending, 1)
419	-	assert.Equal(t, r3.GetId(), rec.Pending.RunID)	425	+	assert.Equal(t, r3.GetId(), rec.Pending[0].RunID)
420		c.waitFor("the replaced run is cancelled by the retry", func() bool { return c.conclusionOf(r2) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })	426		c.waitFor("the replaced run is cancelled by the retry", func() bool { return c.conclusionOf(r2) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
421		assert.True(t, c.cancelAnnotation(r2, "Deploy"))	427		assert.True(t, c.cancelAnnotation(r2, "Deploy"))
422		runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)	428		runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)

@@ -449,7 +455,7 @@ func TestCIConcurrencyReconcileCancelsEveryReplacedVictim(t *testing.T) {
449		next := c.pushN(3+i, workflowAt(wfGroup, ""))	455		next := c.pushN(3+i, workflowAt(wfGroup, ""))
450		c.waitFor("the newer run takes the waiting place", func() bool {	456		c.waitFor("the newer run takes the waiting place", func() bool {
451		rec := c.groupRecord("Deploy")	457		rec := c.groupRecord("Deploy")
452	-	return rec.Pending != nil && rec.Pending.RunID == next.GetId()	458	+	return len(rec.Pending) == 1 && rec.Pending[0].RunID == next.GetId()
453		})	459		})
454		c.waitJobs("the newer run is pending", next, allStatus(pb.CiStatus_CI_STATUS_PENDING))	460		c.waitJobs("the newer run is pending", next, allStatus(pb.CiStatus_CI_STATUS_PENDING))
455		victims = append(victims, prev)	461		victims = append(victims, prev)

@@ -539,9 +545,9 @@ func TestCIConcurrencyCrashBetweenGroupAndEntry(t *testing.T) {
539		_, err = store.Apply(c.ctx, ports.CIWrite{Runs: []domain.CIRun{run}, Jobs: jobs})	545		_, err = store.Apply(c.ctx, ports.CIWrite{Runs: []domain.CIRun{run}, Jobs: jobs})
540		require.NoError(t, err)	546		require.NoError(t, err)
541		rec := c.groupRecord("Deploy")	547		rec := c.groupRecord("Deploy")
542	-	require.NotNil(t, rec.Pending)	548	+	require.Len(t, rec.Pending, 1)
543	-	require.Equal(t, r2.GetId(), rec.Pending.RunID, "the group names the run")	549	+	require.Equal(t, r2.GetId(), rec.Pending[0].RunID, "the group names the run")
544	-	assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", *rec.Pending), "and marks it, in the same batch")	550	+	assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", rec.Pending[0]), "and marks it, in the same batch")
545			551
546		c.forge.CI.ReconcileConcurrency(c.ctx)	552		c.forge.CI.ReconcileConcurrency(c.ctx)
547		c.waitFor("the entry is marked pending again", func() bool {	553		c.waitFor("the entry is marked pending again", func() bool {

@@ -551,8 +557,8 @@ func TestCIConcurrencyCrashBetweenGroupAndEntry(t *testing.T) {
551		time.Sleep(200 * time.Millisecond)	557		time.Sleep(200 * time.Millisecond)
552		assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2), "a named entry is not a replaced one")	558		assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2), "a named entry is not a replaced one")
553		rec = c.groupRecord("Deploy")	559		rec = c.groupRecord("Deploy")
554	-	require.NotNil(t, rec.Pending)	560	+	require.Len(t, rec.Pending, 1)
555	-	assert.Equal(t, r2.GetId(), rec.Pending.RunID)	561	+	assert.Equal(t, r2.GetId(), rec.Pending[0].RunID)
556			562
557		runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)	563		runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
558		assert.Equal(t, r2.GetId(), runner.await().GetRunId())	564		assert.Equal(t, r2.GetId(), runner.await().GetRunId())

@@ -713,7 +719,7 @@ func TestCIConcurrencyReconcileAtRestart(t *testing.T) {
713		run.Status, run.Conclusion, run.CompletedAt = domain.CICompleted, domain.CISuccess, time.Now()	719		run.Status, run.Conclusion, run.CompletedAt = domain.CICompleted, domain.CISuccess, time.Now()
714		_, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs, Runs: []domain.CIRun{run}})	720		_, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs, Runs: []domain.CIRun{run}})
715		require.NoError(t, err)	721		require.NoError(t, err)
716	-	c.holdGroup("Stale", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil)	722	+	c.holdGroup("Stale", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1})
717		rec := c.groupRecord("Deploy")	723		rec := c.groupRecord("Deploy")
718		require.NotNil(t, rec.Running)	724		require.NotNil(t, rec.Running)
719		require.Equal(t, r1.GetId(), rec.Running.RunID, "the record still holds the ended run")	725		require.Equal(t, r1.GetId(), rec.Running.RunID, "the record still holds the ended run")

@@ -745,7 +751,7 @@ func TestCIConcurrencyReconcileAtRestart(t *testing.T) {
745		assert.Equal(t, "DEPLOY", groups[0].Group)	751		assert.Equal(t, "DEPLOY", groups[0].Group)
746		require.NotNil(t, groups[0].State.Record.Running)	752		require.NotNil(t, groups[0].State.Record.Running)
747		assert.Equal(t, r2.GetId(), groups[0].State.Record.Running.RunID)	753		assert.Equal(t, r2.GetId(), groups[0].State.Record.Running.RunID)
748	-	assert.Nil(t, groups[0].State.Record.Pending)	754	+	assert.Empty(t, groups[0].State.Record.Pending)
749		}	755		}
750			756
751		// Both writes of a run's own record fail when it queues (its "pending" state),	757		// Both writes of a run's own record fail when it queues (its "pending" state),

@@ -776,7 +782,7 @@ func TestCIConcurrencyReplacedRunWithLostWritesIsCancelled(t *testing.T) {
776		b := c.pushN(2, workflowAt(wfGroup, ""))	782		b := c.pushN(2, workflowAt(wfGroup, ""))
777		c.waitFor("B is named by the group", func() bool {	783		c.waitFor("B is named by the group", func() bool {
778		rec := c.groupRecord("Deploy")	784		rec := c.groupRecord("Deploy")
779	-	return rec.Pending != nil && rec.Pending.RunID == b.GetId()	785	+	return len(rec.Pending) == 1 && rec.Pending[0].RunID == b.GetId()
780		})	786		})
781		bEntry := domain.CIConcurrencyEntry{RunID: b.GetId(), Attempt: 1}	787		bEntry := domain.CIConcurrencyEntry{RunID: b.GetId(), Attempt: 1}
782		assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", bEntry), "the group batch marked it")	788		assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", bEntry), "the group batch marked it")

@@ -785,8 +791,8 @@ func TestCIConcurrencyReplacedRunWithLostWritesIsCancelled(t *testing.T) {
785		c.pushN(3, workflowAt(wfGroup, ""))	791		c.pushN(3, workflowAt(wfGroup, ""))
786		c.waitFor("B is marked replaced", func() bool { return c.markState("Deploy", bEntry) == domain.CIConcurrencyReplaced })	792		c.waitFor("B is marked replaced", func() bool { return c.markState("Deploy", bEntry) == domain.CIConcurrencyReplaced })
787		rec := c.groupRecord("Deploy")	793		rec := c.groupRecord("Deploy")
788	-	require.NotNil(t, rec.Pending)	794	+	require.Len(t, rec.Pending, 1)
789	-	assert.NotEqual(t, b.GetId(), rec.Pending.RunID, "B is no longer named")	795	+	assert.NotEqual(t, b.GetId(), rec.Pending[0].RunID, "B is no longer named")
790		assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(b))	796		assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(b))
791			797
792		fs.fails.Store(0)	798		fs.fails.Store(0)

@@ -844,8 +850,8 @@ func TestCIJobConcurrencyReplacedJobNeverAcquiresAgain(t *testing.T) {
844		rec = c.groupRecord("jobgroup")	850		rec = c.groupRecord("jobgroup")
845		require.NotNil(t, rec.Running)	851		require.NotNil(t, rec.Running)
846		assert.Equal(t, rC.GetId(), rec.Running.RunID, "C still holds the group")	852		assert.Equal(t, rC.GetId(), rec.Running.RunID, "C still holds the group")
847	-	if rec.Pending != nil {	853	+	if len(rec.Pending) > 0 {
848	-	assert.NotEqual(t, rB.GetId(), rec.Pending.RunID, "B did not take the waiting place back")	854	+	assert.NotEqual(t, rB.GetId(), rec.Pending[0].RunID, "B did not take the waiting place back")
849		}	855		}
850		// Nothing of B's grouped job is given to a runner.	856		// Nothing of B's grouped job is given to a runner.
851		for range 5 {	857		for range 5 {

@@ -943,8 +949,8 @@ func TestCIJobConcurrencyCrashBetweenGroupAndJob(t *testing.T) {
943		_, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs})	949		_, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs})
944		require.NoError(t, err)	950		require.NoError(t, err)
945		rec := c.groupRecord("jobgroup")	951		rec := c.groupRecord("jobgroup")
946	-	require.NotNil(t, rec.Pending)	952	+	require.Len(t, rec.Pending, 1)
947	-	assert.Equal(t, entry.JobID, rec.Pending.JobID, "the group names the job")	953	+	assert.Equal(t, entry.JobID, rec.Pending[0].JobID, "the group names the job")
948		assert.Equal(t, domain.CIConcurrencyPending, c.markState("jobgroup", entry), "and its mark says so")	954		assert.Equal(t, domain.CIConcurrencyPending, c.markState("jobgroup", entry), "and its mark says so")
949			955
950		c.forge.CI.ReconcileConcurrency(c.ctx)	956		c.forge.CI.ReconcileConcurrency(c.ctx)

@@ -1110,3 +1116,133 @@ func TestCIConcurrencyPrunesStaleMarksInBoundedBatches(t *testing.T) {
1110		assert.Contains(t, st.Marks, domain.CIConcurrencyEntry{RunID: r.GetId(), Attempt: 1}.Key())	1116		assert.Contains(t, st.Marks, domain.CIConcurrencyEntry{RunID: r.GetId(), Attempt: 1}.Key())
1111		runner.runJob(job, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)	1117		runner.runJob(job, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1112		}	1118		}
			1119	+
			1120	+	func (c ciHarness) hasAnnotation(run pb.Run, msg string) bool {
			1121	+	for _, j := range c.jobsOf(run) {
			1122	+	for _, a := range c.annotations(run.GetId(), j.GetId()).GetAnnotations() {
			1123	+	if a.GetAnnotation().GetMessage() == msg {
			1124	+	return true
			1125	+	}
			1126	+	}
			1127	+	}
			1128	+	return false
			1129	+	}
			1130	+
			1131	+	// pendingRuns is the waiting line of a group, as run ids.
			1132	+	func (c *ciHarness) pendingRuns(group string) []int64 {
			1133	+	var ids []int64
			1134	+	for _, p := range c.groupRecord(group).Pending {
			1135	+	ids = append(ids, p.RunID)
			1136	+	}
			1137	+	return ids
			1138	+	}
			1139	+
			1140	+	func (c ciHarness) completed(run pb.Run) bool {
			1141	+	return c.getRun(run.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
			1142	+	}
			1143	+
			1144	+	// queue: max at the workflow and at the job level: three runs of one group
			1145	+	// run one after the other in the order they started waiting, none cancelled.
			1146	+	func TestCIConcurrencyQueueMaxRunsInOrder(t *testing.T) {
			1147	+	for _, tt := range []struct{ name, top, job string }{
			1148	+	{"workflow", "concurrency:\n group: Deploy\n queue: max\n", ""},
			1149	+	{"job", "", " concurrency:\n group: Deploy\n queue: max\n"},
			1150	+	} {
			1151	+	t.Run(tt.name, func(t *testing.T) {
			1152	+	c, runner := concurrencyHarness(t)
			1153	+	wf := workflowAt(tt.top, tt.job)
			1154	+	r1 := c.pushN(1, wf)
			1155	+	job1 := runner.await()
			1156	+	require.Equal(t, r1.GetId(), job1.GetRunId())
			1157	+	r2 := c.pushN(2, wf)
			1158	+	c.waitJobs("the second run is pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
			1159	+	r3 := c.pushN(3, wf)
			1160	+	c.waitJobs("the third run is pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
			1161	+	assert.Equal(t, []int64{r2.GetId(), r3.GetId()}, c.pendingRuns("Deploy"), "both wait, in order")
			1162	+	rec := c.groupRecord("Deploy")
			1163	+	assert.True(t, rec.Running.Max && rec.Pending[0].Max && rec.Pending[1].Max, "each entry records its queue mode")
			1164	+
			1165	+	runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1166	+	job2 := runner.await()
			1167	+	assert.Equal(t, r2.GetId(), job2.GetRunId(), "the head of the line goes first")
			1168	+	assert.Equal(t, []int64{r3.GetId()}, c.pendingRuns("Deploy"))
			1169	+	runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1170	+	job3 := runner.await()
			1171	+	assert.Equal(t, r3.GetId(), job3.GetRunId())
			1172	+	runner.runJob(job3, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1173	+	for _, r := range []*pb.Run{r1, r2, r3} {
			1174	+	c.waitFor("the run completes", func() bool { return c.completed(r) })
			1175	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_SUCCESS, c.conclusionOf(r))
			1176	+	}
			1177	+	c.waitFor("the group is free", func() bool { return c.groupRecord("Deploy").IsEmpty() })
			1178	+	})
			1179	+	}
			1180	+	}
			1181	+
			1182	+	// A queue: max entry that finds the line full is cancelled with ermite's
			1183	+	// annotation; the waiting ones stay. The cap is lowered to 2 here (it is
			1184	+	// domain.CIMaxConcurrencyQueue, 100, by default; see the domain tests).
			1185	+	func TestCIConcurrencyQueueMaxFullLineCancelsTheNewcomer(t *testing.T) {
			1186	+	c := newCIHarness(t, func(cfg *usecase.CIConfig) {
			1187	+	cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300time.Millisecond, 20time.Millisecond, 100*time.Millisecond
			1188	+	cfg.ConcurrencyQueueMax = 2
			1189	+	})
			1190	+	runner := c.registerRunner("alpha", "ubuntu-latest")
			1191	+	wf := workflowAt("concurrency:\n group: Deploy\n queue: max\n", "")
			1192	+	r1 := c.pushN(1, wf)
			1193	+	job1 := runner.await()
			1194	+	r2 := c.pushN(2, wf)
			1195	+	c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
			1196	+	r3 := c.pushN(3, wf)
			1197	+	c.waitJobs("pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
			1198	+
			1199	+	r4 := c.pushN(4, wf)
			1200	+	c.waitFor("the newcomer is cancelled", func() bool { return c.completed(r4) })
			1201	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r4))
			1202	+	assert.True(t, c.hasAnnotation(r4, domain.ConcurrencyQueueFullMessage("Deploy")))
			1203	+	assert.False(t, c.cancelAnnotation(r4, "Deploy"))
			1204	+	assert.Equal(t, []int64{r2.GetId(), r3.GetId()}, c.pendingRuns("Deploy"), "the line is kept")
			1205	+	assert.False(t, c.groupRecord("Deploy").Names(domain.CIConcurrencyEntry{RunID: r4.GetId(), Attempt: 1}), "the newcomer was never named")
			1206	+	assert.False(t, c.completed(r1))
			1207	+	assert.False(t, c.completed(r2))
			1208	+	assert.False(t, c.completed(r3))
			1209	+
			1210	+	runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1211	+	job2 := runner.await()
			1212	+	assert.Equal(t, r2.GetId(), job2.GetRunId())
			1213	+	runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1214	+	job3 := runner.await()
			1215	+	assert.Equal(t, r3.GetId(), job3.GetRunId())
			1216	+	runner.runJob(job3, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1217	+	c.waitFor("the last run completes", func() bool { return c.completed(r3) })
			1218	+	}
			1219	+
			1220	+	// ermite's rule for a mixed group: a queue: single entry that starts waiting
			1221	+	// cancels every waiting entry, queue: max ones included, and is then the only
			1222	+	// one; the running entry is left alone (no cancel-in-progress).
			1223	+	func TestCIConcurrencyQueueSingleCancelsTheMaxLine(t *testing.T) {
			1224	+	c, runner := concurrencyHarness(t)
			1225	+	maxWF := workflowAt("concurrency:\n group: Deploy\n queue: max\n", "")
			1226	+	r1 := c.pushN(1, maxWF)
			1227	+	job1 := runner.await()
			1228	+	r2 := c.pushN(2, maxWF)
			1229	+	c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
			1230	+	r3 := c.pushN(3, maxWF)
			1231	+	c.waitJobs("pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
			1232	+
			1233	+	r4 := c.pushN(4, workflowAt(wfGroup, ""))
			1234	+	c.waitJobs("the single run is pending", r4, allStatus(pb.CiStatus_CI_STATUS_PENDING))
			1235	+	for _, r := range []*pb.Run{r2, r3} {
			1236	+	c.waitFor("the max run is cancelled", func() bool { return c.completed(r) })
			1237	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r))
			1238	+	assert.True(t, c.cancelAnnotation(r, "Deploy"))
			1239	+	}
			1240	+	assert.Equal(t, []int64{r4.GetId()}, c.pendingRuns("Deploy"))
			1241	+	assert.False(t, c.completed(r1), "the running run is left alone")
			1242	+
			1243	+	runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1244	+	job4 := runner.await()
			1245	+	assert.Equal(t, r4.GetId(), job4.GetRunId())
			1246	+	runner.runJob(job4, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
			1247	+	c.waitFor("the single run completes", func() bool { return c.completed(r4) })
			1248	+	}

M internal/app/ci_permissions_test.go +6 -4


@@ -133,13 +133,14 @@ func TestDefaultTokenCannotPush(t *testing.T) {
133	133		func TestJobTokenWithContentsWritePushesButNotWorkflows(t *testing.T) {
134	134		c := newCIHarness(t, nil)
135	135		runner := c.registerRunner("alpha", "ubuntu-latest")
136		-	wfYAML := "on:\n push:\n create:\npermissions:\n contents: write\njobs:\n j:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n"
	136	+	wfYAML := "on:\n push:\n create:\n delete:\npermissions:\n contents: write\njobs:\n j:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n"
137	137		c.push("main", map[string]string{".github/workflows/p.yml": wfYAML}, "")
138	138		c.push("feat", map[string]string{"feat.txt": "f"}, "")
139	139		job := runner.await()
140	140		tok := job.GetGithubToken()
141	141		require.Equal(t, "write", levelsOf(job)["contents"])
142		-	c.waitFor("two runs", func() bool { return len(c.runs()) == 2 })
	142	+	// push and create for main, then for feat.
	143	+	c.waitFor("four runs", func() bool { return len(c.runs()) == 4 })
143	144		runsBefore := len(c.runs())
144	145		ref := func(r string) string { return c.bareGit("proj", "rev-parse", "--verify", "-q", r) }
145	146		refs := func() string { return c.bareGit("proj", "for-each-ref", "--format=%(refname) %(objectname)") }

@@ -238,12 +239,13 @@ func TestJobTokenWithContentsWritePushesButNotWorkflows(t *testing.T) {
238	239		// None of the pushes of the job token started a run; a push of the owner
239	240		// does, and queues after them.
240	241		c.push("sentinel", map[string]string{"s.txt": "s"}, "")
241		-	c.waitFor("the owner's runs", func() bool { return len(c.runs()) >= runsBefore+2 })
	242	+	c.waitFor("the owner's runs", func() bool { return len(c.runs()) >= runsBefore+4 })
242	243		for _, r := range c.runs() {
243	244		assert.NotEqual(t, "refs/heads/orphan", r.GetRef(), "no run for a job token's push")
244	245		assert.NotEqual(t, "refs/tags/v0", r.GetRef())
	246	+	assert.NotEqual(t, "delete", r.GetEvent(), "no run for a job token's deletes")
245	247		}
246		-	assert.Len(t, c.runs(), runsBefore+2, "the owner's rworkflow and sentinel pushes only")
	248	+	assert.Len(t, c.runs(), runsBefore+4, "push and create of the owner's rworkflow and sentinel only")
247	249		}
248	250
249	251		func TestJobTokenCannotPushToAnotherRepository(t *testing.T) {

 func TestJobTokenWithContentsWritePushesButNotWorkflows(t *testing.T) {
 	c := newCIHarness(t, nil)
 	runner := c.registerRunner("alpha", "ubuntu-latest")
-	wfYAML := "on:\n  push:\n  create:\npermissions:\n  contents: write\njobs:\n  j:\n    runs-on: ubuntu-latest\n    steps:\n      - run: echo hi\n"
+	wfYAML := "on:\n  push:\n  create:\n  delete:\npermissions:\n  contents: write\njobs:\n  j:\n    runs-on: ubuntu-latest\n    steps:\n      - run: echo hi\n"
 	c.push("main", map[string]string{".github/workflows/p.yml": wfYAML}, "")
 	c.push("feat", map[string]string{"feat.txt": "f"}, "")
 	job := runner.await()
 	tok := job.GetGithubToken()
 	require.Equal(t, "write", levelsOf(job)["contents"])
-	c.waitFor("two runs", func() bool { return len(c.runs()) == 2 })
+	// push and create for main, then for feat.
+	c.waitFor("four runs", func() bool { return len(c.runs()) == 4 })
 	runsBefore := len(c.runs())
 	ref := func(r string) string { return c.bareGit("proj", "rev-parse", "--verify", "-q", r) }
 	refs := func() string { return c.bareGit("proj", "for-each-ref", "--format=%(refname) %(objectname)") }
 	// None of the pushes of the job token started a run; a push of the owner
 	// does, and queues after them.
 	c.push("sentinel", map[string]string{"s.txt": "s"}, "")
-	c.waitFor("the owner's runs", func() bool { return len(c.runs()) >= runsBefore+2 })
+	c.waitFor("the owner's runs", func() bool { return len(c.runs()) >= runsBefore+4 })
 	for _, r := range c.runs() {
 		assert.NotEqual(t, "refs/heads/orphan", r.GetRef(), "no run for a job token's push")
 		assert.NotEqual(t, "refs/tags/v0", r.GetRef())
+		assert.NotEqual(t, "delete", r.GetEvent(), "no run for a job token's deletes")
 	}
-	assert.Len(t, c.runs(), runsBefore+2, "the owner's rworkflow and sentinel pushes only")
+	assert.Len(t, c.runs(), runsBefore+4, "push and create of the owner's rworkflow and sentinel only")
 }
 func TestJobTokenCannotPushToAnotherRepository(t *testing.T) {

A internal/app/ci_refevents_test.go +213 -0

@@ -0,0 +1,213 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package app
	4	+
	5	+	import (
	6	+	"fmt"
	7	+	"testing"
	8	+
	9	+	"github.com/stretchr/testify/assert"
	10	+	"github.com/stretchr/testify/require"
	11	+
	12	+	pb "bnema.dev/ermite-api/gen/ermite"
	13	+	"bnema.dev/ermite/internal/domain"
	14	+	)
	15	+
	16	+	// CI-4 D6: the create and delete events.
	17	+
	18	+	const refEventWorkflow = `name: %s
	19	+	on: %s
	20	+	jobs:
	21	+	j:
	22	+	runs-on: ubuntu-latest
	23	+	steps:
	24	+	- run: echo ${{ secrets.API_TOKEN }} %s
	25	+	`
	26	+
	27	+	func refEventWF(event, mark string) map[string]string {
	28	+	return map[string]string{".github/workflows/" + event + ".yml": fmt.Sprintf(refEventWorkflow, event, event, mark)}
	29	+	}
	30	+
	31	+	// newRefEventHarness is a harness with a repository secret, the workflow of
	32	+	// event on main and the create run of main itself finished when event is create.
	33	+	func newRefEventHarness(t testing.T, event string) (ciHarness, *fakeRunner, string) {
	34	+	t.Helper()
	35	+	c := newSecretsHarness(t, secretsKeyA)
	36	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	37	+	c.setSecret(repoScope("proj"), "API_TOKEN", "repo-token")
	38	+	main := c.push("main", refEventWF(event, "from-main"), "")
	39	+	if event == "create" {
	40	+	job := c.finish(runner)
	41	+	assert.Equal(t, "refs/heads/main", job.GetRef(), "creating main is a create event")
	42	+	}
	43	+	return c, runner, main
	44	+	}
	45	+
	46	+	func eventOf(c ciHarness, j pb.Job) map[string]any {
	47	+	return c.ghContext(j)["event"].(map[string]any)
	48	+	}
	49	+
	50	+	func TestCreateRunsOnTheCreatedRef(t *testing.T) {
	51	+	c, runner, _ := newRefEventHarness(t, "create")
	52	+
	53	+	// A new branch: its own workflow file runs, on its commit.
	54	+	c.mustGit(c.wt, "checkout", "-q", "-b", "feat")
	55	+	feat := c.push("feat", refEventWF("create", "from-feat"), "")
	56	+	job := c.finish(runner)
	57	+	assert.Contains(t, job.GetWorkflowYaml(), "from-feat")
	58	+	assert.Equal(t, "refs/heads/feat", job.GetRef())
	59	+	assert.Equal(t, feat, job.GetSha())
	60	+	assert.True(t, job.GetTrusted())
	61	+	assert.Equal(t, "repo-token", job.GetSecrets()["API_TOKEN"])
	62	+	gh := c.ghContext(job)
	63	+	assert.Equal(t, "create", gh["event_name"])
	64	+	assert.Equal(t, "refs/heads/feat", gh["ref"])
	65	+	assert.Equal(t, "feat", gh["ref_name"])
	66	+	assert.Equal(t, "branch", gh["ref_type"])
	67	+	ev := eventOf(c, job)
	68	+	assert.Equal(t, "feat", ev["ref"])
	69	+	assert.Equal(t, "branch", ev["ref_type"])
	70	+	assert.Equal(t, "main", ev["master_branch"])
	71	+	assert.Equal(t, "user", ev["pusher_type"])
	72	+	assert.Contains(t, ev, "description")
	73	+	assert.Nil(t, ev["description"])
	74	+	assert.NotNil(t, ev["repository"])
	75	+	assert.NotNil(t, ev["sender"])
	76	+	assert.NotContains(t, ev, "commits", "not a push payload")
	77	+	assert.Equal(t, "refs/heads/feat", job.GetCache().GetWriteRef(), "a trusted run saves under its ref")
	78	+
	79	+	// A tag.
	80	+	c.mustGit(c.wt, "tag", "v1")
	81	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "v1")
	82	+	job = c.finish(runner)
	83	+	assert.Equal(t, "refs/tags/v1", job.GetRef())
	84	+	ev = eventOf(c, job)
	85	+	assert.Equal(t, "v1", ev["ref"])
	86	+	assert.Equal(t, "tag", ev["ref_type"])
	87	+
	88	+	// Moving an existing branch creates nothing.
	89	+	c.push("feat", map[string]string{"b.txt": "b\n"}, "")
	90	+	c.mustGit(c.wt, "checkout", "-q", "main")
	91	+	c.push("main", map[string]string{"sentinel.txt": "s\n"}, "")
	92	+	c.mustGit(c.wt, "checkout", "-q", "-b", "sentinel")
	93	+	c.push("sentinel", map[string]string{"s.txt": "s\n"}, "")
	94	+	job = c.finish(runner)
	95	+	assert.Equal(t, "refs/heads/sentinel", job.GetRef(), "the update of feat and main started no run")
	96	+	for _, r := range c.runs() {
	97	+	assert.Equal(t, "create", r.GetEvent())
	98	+	}
	99	+	assert.Len(t, c.runs(), 4, "main, feat, v1, sentinel")
	100	+	}
	101	+
	102	+	// More than three tags at once create no event, as on GitHub.
	103	+	func TestCreateSkipsMoreThanThreeTags(t *testing.T) {
	104	+	c, runner, _ := newRefEventHarness(t, "create")
	105	+	for _, tag := range []string{"t1", "t2", "t3", "t4"} {
	106	+	c.mustGit(c.wt, "tag", tag)
	107	+	}
	108	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "t1", "t2", "t3", "t4")
	109	+	c.mustGit(c.wt, "checkout", "-q", "-b", "sentinel")
	110	+	c.push("sentinel", map[string]string{"s.txt": "s\n"}, "")
	111	+	job := c.finish(runner)
	112	+	assert.Equal(t, "refs/heads/sentinel", job.GetRef())
	113	+	assert.Len(t, c.runs(), 2, "main and sentinel")
	114	+	}
	115	+
	116	+	func TestDeleteRunsTheDefaultBranchWorkflow(t *testing.T) {
	117	+	c, runner, main := newRefEventHarness(t, "delete")
	118	+	// The deleted branch carries another version of the workflow: it is not
	119	+	// the one that runs.
	120	+	c.mustGit(c.wt, "checkout", "-q", "-b", "feat")
	121	+	c.push("feat", refEventWF("delete", "from-feat"), "")
	122	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":feat")
	123	+
	124	+	job := c.finish(runner)
	125	+	assert.Contains(t, job.GetWorkflowYaml(), "from-main")
	126	+	assert.Equal(t, "refs/heads/main", job.GetRef())
	127	+	assert.Equal(t, main, job.GetSha())
	128	+	assert.True(t, job.GetTrusted())
	129	+	assert.Equal(t, "repo-token", job.GetSecrets()["API_TOKEN"])
	130	+	gh := c.ghContext(job)
	131	+	assert.Equal(t, "delete", gh["event_name"])
	132	+	assert.Equal(t, "refs/heads/main", gh["ref"])
	133	+	assert.Equal(t, main, gh["sha"])
	134	+	ev := eventOf(c, job)
	135	+	assert.Equal(t, "feat", ev["ref"])
	136	+	assert.Equal(t, "branch", ev["ref_type"])
	137	+	assert.Equal(t, "user", ev["pusher_type"])
	138	+	assert.NotContains(t, ev, "master_branch")
	139	+	assert.NotNil(t, ev["repository"])
	140	+	assert.NotNil(t, ev["sender"])
	141	+	assert.Equal(t, "refs/heads/main", job.GetCache().GetWriteRef())
	142	+
	143	+	runs := c.runs()
	144	+	require.Len(t, runs, 1)
	145	+	assert.Equal(t, "delete", runs[0].GetEvent())
	146	+	assert.Equal(t, "refs/heads/main", runs[0].GetRef())
	147	+
	148	+	// A tag.
	149	+	c.mustGit(c.wt, "tag", "v1", "main")
	150	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "v1")
	151	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":refs/tags/v1")
	152	+	ev = eventOf(c, c.finish(runner))
	153	+	assert.Equal(t, "v1", ev["ref"])
	154	+	assert.Equal(t, "tag", ev["ref_type"])
	155	+	}
	156	+
	157	+	// An agent creating or deleting runs untrusted; an owner's delete runs
	158	+	// untrusted when the default tip was not pushed by a trusted principal.
	159	+	func TestCreateAndDeleteTrust(t *testing.T) {
	160	+	c := newSecretsHarness(t, secretsKeyA)
	161	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	162	+	c.setSecret(repoScope("proj"), "API_TOKEN", "repo-token")
	163	+	files := refEventWF("create", "c")
	164	+	for k, v := range refEventWF("delete", "d") {
	165	+	files[k] = v
	166	+	}
	167	+	main := c.push("main", files, "")
	168	+	require.True(t, c.finish(runner).GetTrusted(), "the owner created main")
	169	+	secret, _ := c.branchToken(scopes(domain.ScopeRepoRead, domain.ScopeRepoWrite), []string{"agent/"}, "proj")
	170	+	agentURL := c.urlAuth(secret, "/proj.git")
	171	+
	172	+	// The agent creates a branch: untrusted.
	173	+	c.mustGit(c.wt, "checkout", "-q", "-b", "agent/x")
	174	+	c.writeFile(c.wt, "x.txt", []byte("x\n"))
	175	+	c.mustGit(c.wt, "push", "-q", agentURL, "agent/x")
	176	+	job := c.finish(runner)
	177	+	assert.Equal(t, "create", c.ghContext(job)["event_name"])
	178	+	assert.False(t, job.GetTrusted())
	179	+	assert.Empty(t, job.GetSecrets())
	180	+	assert.Equal(t, "refs/untrusted/heads/agent/x", job.GetCache().GetWriteRef())
	181	+
	182	+	// The agent deletes it: untrusted, and the default branch is not written.
	183	+	c.mustGit(c.wt, "push", "-q", agentURL, ":agent/x")
	184	+	job = c.finish(runner)
	185	+	assert.Equal(t, "delete", c.ghContext(job)["event_name"])
	186	+	assert.Equal(t, main, job.GetSha())
	187	+	assert.False(t, job.GetTrusted())
	188	+	assert.Empty(t, job.GetSecrets())
	189	+	assert.Empty(t, job.GetCache().GetWriteRef(), "an untrusted run never saves under the default branch")
	190	+
	191	+	// The owner deletes a branch the agent pushed: trusted, the deleted ref's
	192	+	// record does not matter (it is gone with the ref).
	193	+	c.mustGit(c.wt, "push", "-q", agentURL, "agent/x")
	194	+	c.finish(runner) // create
	195	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":agent/x")
	196	+	job = c.finish(runner)
	197	+	assert.Equal(t, "delete", c.ghContext(job)["event_name"])
	198	+	assert.True(t, job.GetTrusted())
	199	+	assert.Equal(t, "repo-token", job.GetSecrets()["API_TOKEN"])
	200	+
	201	+	// The owner deletes while main's record says an untrusted principal set
	202	+	// it: the default tip's code runs, so untrusted.
	203	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "agent/x:refs/heads/other")
	204	+	c.finish(runner) // create
	205	+	require.NoError(t, c.forge.store.CI().PutRefTip(c.ctx, domain.CIRefTip{
	206	+	Repo: c.name("proj"), Ref: "refs/heads/main", Sha: main, Actor: "claude", Trusted: false,
	207	+	}))
	208	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":other")
	209	+	job = c.finish(runner)
	210	+	assert.Equal(t, "delete", c.ghContext(job)["event_name"])
	211	+	assert.False(t, job.GetTrusted())
	212	+	assert.Empty(t, job.GetSecrets())
	213	+	}

@@ -0,0 +1,213 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package app
	4	+
	5	+	import (
	6	+	"fmt"
	7	+	"testing"
	8	+
	9	+	"github.com/stretchr/testify/assert"
	10	+	"github.com/stretchr/testify/require"
	11	+
	12	+	pb "bnema.dev/ermite-api/gen/ermite"
	13	+	"bnema.dev/ermite/internal/domain"
	14	+	)
	15	+
	16	+	// CI-4 D6: the create and delete events.
	17	+
	18	+	const refEventWorkflow = `name: %s
	19	+	on: %s
	20	+	jobs:
	21	+	j:
	22	+	runs-on: ubuntu-latest
	23	+	steps:
	24	+	- run: echo ${{ secrets.API_TOKEN }} %s
	25	+	`
	26	+
	27	+	func refEventWF(event, mark string) map[string]string {
	28	+	return map[string]string{".github/workflows/" + event + ".yml": fmt.Sprintf(refEventWorkflow, event, event, mark)}
	29	+	}
	30	+
	31	+	// newRefEventHarness is a harness with a repository secret, the workflow of
	32	+	// event on main and the create run of main itself finished when event is create.
	33	+	func newRefEventHarness(t testing.T, event string) (ciHarness, *fakeRunner, string) {
	34	+	t.Helper()
	35	+	c := newSecretsHarness(t, secretsKeyA)
	36	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	37	+	c.setSecret(repoScope("proj"), "API_TOKEN", "repo-token")
	38	+	main := c.push("main", refEventWF(event, "from-main"), "")
	39	+	if event == "create" {
	40	+	job := c.finish(runner)
	41	+	assert.Equal(t, "refs/heads/main", job.GetRef(), "creating main is a create event")
	42	+	}
	43	+	return c, runner, main
	44	+	}
	45	+
	46	+	func eventOf(c ciHarness, j pb.Job) map[string]any {
	47	+	return c.ghContext(j)["event"].(map[string]any)
	48	+	}
	49	+
	50	+	func TestCreateRunsOnTheCreatedRef(t *testing.T) {
	51	+	c, runner, _ := newRefEventHarness(t, "create")
	52	+
	53	+	// A new branch: its own workflow file runs, on its commit.
	54	+	c.mustGit(c.wt, "checkout", "-q", "-b", "feat")
	55	+	feat := c.push("feat", refEventWF("create", "from-feat"), "")
	56	+	job := c.finish(runner)
	57	+	assert.Contains(t, job.GetWorkflowYaml(), "from-feat")
	58	+	assert.Equal(t, "refs/heads/feat", job.GetRef())
	59	+	assert.Equal(t, feat, job.GetSha())
	60	+	assert.True(t, job.GetTrusted())
	61	+	assert.Equal(t, "repo-token", job.GetSecrets()["API_TOKEN"])
	62	+	gh := c.ghContext(job)
	63	+	assert.Equal(t, "create", gh["event_name"])
	64	+	assert.Equal(t, "refs/heads/feat", gh["ref"])
	65	+	assert.Equal(t, "feat", gh["ref_name"])
	66	+	assert.Equal(t, "branch", gh["ref_type"])
	67	+	ev := eventOf(c, job)
	68	+	assert.Equal(t, "feat", ev["ref"])
	69	+	assert.Equal(t, "branch", ev["ref_type"])
	70	+	assert.Equal(t, "main", ev["master_branch"])
	71	+	assert.Equal(t, "user", ev["pusher_type"])
	72	+	assert.Contains(t, ev, "description")
	73	+	assert.Nil(t, ev["description"])
	74	+	assert.NotNil(t, ev["repository"])
	75	+	assert.NotNil(t, ev["sender"])
	76	+	assert.NotContains(t, ev, "commits", "not a push payload")
	77	+	assert.Equal(t, "refs/heads/feat", job.GetCache().GetWriteRef(), "a trusted run saves under its ref")
	78	+
	79	+	// A tag.
	80	+	c.mustGit(c.wt, "tag", "v1")
	81	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "v1")
	82	+	job = c.finish(runner)
	83	+	assert.Equal(t, "refs/tags/v1", job.GetRef())
	84	+	ev = eventOf(c, job)
	85	+	assert.Equal(t, "v1", ev["ref"])
	86	+	assert.Equal(t, "tag", ev["ref_type"])
	87	+
	88	+	// Moving an existing branch creates nothing.
	89	+	c.push("feat", map[string]string{"b.txt": "b\n"}, "")
	90	+	c.mustGit(c.wt, "checkout", "-q", "main")
	91	+	c.push("main", map[string]string{"sentinel.txt": "s\n"}, "")
	92	+	c.mustGit(c.wt, "checkout", "-q", "-b", "sentinel")
	93	+	c.push("sentinel", map[string]string{"s.txt": "s\n"}, "")
	94	+	job = c.finish(runner)
	95	+	assert.Equal(t, "refs/heads/sentinel", job.GetRef(), "the update of feat and main started no run")
	96	+	for _, r := range c.runs() {
	97	+	assert.Equal(t, "create", r.GetEvent())
	98	+	}
	99	+	assert.Len(t, c.runs(), 4, "main, feat, v1, sentinel")
	100	+	}
	101	+
	102	+	// More than three tags at once create no event, as on GitHub.
	103	+	func TestCreateSkipsMoreThanThreeTags(t *testing.T) {
	104	+	c, runner, _ := newRefEventHarness(t, "create")
	105	+	for _, tag := range []string{"t1", "t2", "t3", "t4"} {
	106	+	c.mustGit(c.wt, "tag", tag)
	107	+	}
	108	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "t1", "t2", "t3", "t4")
	109	+	c.mustGit(c.wt, "checkout", "-q", "-b", "sentinel")
	110	+	c.push("sentinel", map[string]string{"s.txt": "s\n"}, "")
	111	+	job := c.finish(runner)
	112	+	assert.Equal(t, "refs/heads/sentinel", job.GetRef())
	113	+	assert.Len(t, c.runs(), 2, "main and sentinel")
	114	+	}
	115	+
	116	+	func TestDeleteRunsTheDefaultBranchWorkflow(t *testing.T) {
	117	+	c, runner, main := newRefEventHarness(t, "delete")
	118	+	// The deleted branch carries another version of the workflow: it is not
	119	+	// the one that runs.
	120	+	c.mustGit(c.wt, "checkout", "-q", "-b", "feat")
	121	+	c.push("feat", refEventWF("delete", "from-feat"), "")
	122	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":feat")
	123	+
	124	+	job := c.finish(runner)
	125	+	assert.Contains(t, job.GetWorkflowYaml(), "from-main")
	126	+	assert.Equal(t, "refs/heads/main", job.GetRef())
	127	+	assert.Equal(t, main, job.GetSha())
	128	+	assert.True(t, job.GetTrusted())
	129	+	assert.Equal(t, "repo-token", job.GetSecrets()["API_TOKEN"])
	130	+	gh := c.ghContext(job)
	131	+	assert.Equal(t, "delete", gh["event_name"])
	132	+	assert.Equal(t, "refs/heads/main", gh["ref"])
	133	+	assert.Equal(t, main, gh["sha"])
	134	+	ev := eventOf(c, job)
	135	+	assert.Equal(t, "feat", ev["ref"])
	136	+	assert.Equal(t, "branch", ev["ref_type"])
	137	+	assert.Equal(t, "user", ev["pusher_type"])
	138	+	assert.NotContains(t, ev, "master_branch")
	139	+	assert.NotNil(t, ev["repository"])
	140	+	assert.NotNil(t, ev["sender"])
	141	+	assert.Equal(t, "refs/heads/main", job.GetCache().GetWriteRef())
	142	+
	143	+	runs := c.runs()
	144	+	require.Len(t, runs, 1)
	145	+	assert.Equal(t, "delete", runs[0].GetEvent())
	146	+	assert.Equal(t, "refs/heads/main", runs[0].GetRef())
	147	+
	148	+	// A tag.
	149	+	c.mustGit(c.wt, "tag", "v1", "main")
	150	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "v1")
	151	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":refs/tags/v1")
	152	+	ev = eventOf(c, c.finish(runner))
	153	+	assert.Equal(t, "v1", ev["ref"])
	154	+	assert.Equal(t, "tag", ev["ref_type"])
	155	+	}
	156	+
	157	+	// An agent creating or deleting runs untrusted; an owner's delete runs
	158	+	// untrusted when the default tip was not pushed by a trusted principal.
	159	+	func TestCreateAndDeleteTrust(t *testing.T) {
	160	+	c := newSecretsHarness(t, secretsKeyA)
	161	+	runner := c.registerRunner("alpha", "ubuntu-latest")
	162	+	c.setSecret(repoScope("proj"), "API_TOKEN", "repo-token")
	163	+	files := refEventWF("create", "c")
	164	+	for k, v := range refEventWF("delete", "d") {
	165	+	files[k] = v
	166	+	}
	167	+	main := c.push("main", files, "")
	168	+	require.True(t, c.finish(runner).GetTrusted(), "the owner created main")
	169	+	secret, _ := c.branchToken(scopes(domain.ScopeRepoRead, domain.ScopeRepoWrite), []string{"agent/"}, "proj")
	170	+	agentURL := c.urlAuth(secret, "/proj.git")
	171	+
	172	+	// The agent creates a branch: untrusted.
	173	+	c.mustGit(c.wt, "checkout", "-q", "-b", "agent/x")
	174	+	c.writeFile(c.wt, "x.txt", []byte("x\n"))
	175	+	c.mustGit(c.wt, "push", "-q", agentURL, "agent/x")
	176	+	job := c.finish(runner)
	177	+	assert.Equal(t, "create", c.ghContext(job)["event_name"])
	178	+	assert.False(t, job.GetTrusted())
	179	+	assert.Empty(t, job.GetSecrets())
	180	+	assert.Equal(t, "refs/untrusted/heads/agent/x", job.GetCache().GetWriteRef())
	181	+
	182	+	// The agent deletes it: untrusted, and the default branch is not written.
	183	+	c.mustGit(c.wt, "push", "-q", agentURL, ":agent/x")
	184	+	job = c.finish(runner)
	185	+	assert.Equal(t, "delete", c.ghContext(job)["event_name"])
	186	+	assert.Equal(t, main, job.GetSha())
	187	+	assert.False(t, job.GetTrusted())
	188	+	assert.Empty(t, job.GetSecrets())
	189	+	assert.Empty(t, job.GetCache().GetWriteRef(), "an untrusted run never saves under the default branch")
	190	+
	191	+	// The owner deletes a branch the agent pushed: trusted, the deleted ref's
	192	+	// record does not matter (it is gone with the ref).
	193	+	c.mustGit(c.wt, "push", "-q", agentURL, "agent/x")
	194	+	c.finish(runner) // create
	195	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":agent/x")
	196	+	job = c.finish(runner)
	197	+	assert.Equal(t, "delete", c.ghContext(job)["event_name"])
	198	+	assert.True(t, job.GetTrusted())
	199	+	assert.Equal(t, "repo-token", job.GetSecrets()["API_TOKEN"])
	200	+
	201	+	// The owner deletes while main's record says an untrusted principal set
	202	+	// it: the default tip's code runs, so untrusted.
	203	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), "agent/x:refs/heads/other")
	204	+	c.finish(runner) // create
	205	+	require.NoError(t, c.forge.store.CI().PutRefTip(c.ctx, domain.CIRefTip{
	206	+	Repo: c.name("proj"), Ref: "refs/heads/main", Sha: main, Actor: "claude", Trusted: false,
	207	+	}))
	208	+	c.mustGit(c.wt, "push", "-q", c.urlAuth(adminToken, "/proj.git"), ":other")
	209	+	job = c.finish(runner)
	210	+	assert.Equal(t, "delete", c.ghContext(job)["event_name"])
	211	+	assert.False(t, job.GetTrusted())
	212	+	assert.Empty(t, job.GetSecrets())
	213	+	}

M internal/app/ci_schedule_test.go +32 -0


@@ -255,3 +255,35 @@ func TestScheduleBoundsTheWorkPerTick(t *testing.T) {
255	255		assert.Equal(t, time.Date(2026, 3, 4, 10, 1, 0, 0, time.UTC), ci.SchedulePassed(), "three ticks go through three repositories")
256	256		assert.Len(t, c.scheduled(), 1)
257	257		}
	258	+
	259	+	// CI-4 D6: the timezone of an entry is the wall clock its cron reads.
	260	+	func TestScheduleTimezone(t *testing.T) {
	261	+	c, clk, _ := newScheduleHarness(t, "2026-06-01 06:58:30")
	262	+	wf := map[string]string{".github/workflows/nightly.yml": "name: nightly\non:\n schedule:\n - cron: '0 9 * * *'\n timezone: Europe/Paris\njobs:\n j:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n"}
	263	+	c.push("main", wf, "")
	264	+	tick := func(at string) {
	265	+	clk.set(at)
	266	+	c.forge.CI.ScheduleNow(c.ctx)
	267	+	}
	268	+	tick("2026-06-01 06:59:10")
	269	+	tick("2026-06-01 09:00:10")
	270	+	assert.Empty(t, c.scheduled(), "09:00 UTC is 11:00 in Paris")
	271	+	tick("2026-06-02 07:00:10")
	272	+	require.Len(t, c.scheduled(), 1, "09:00 in Paris")
	273	+	}
	274	+
	275	+	// A timezone that is no IANA name makes the workflow invalid: its push records
	276	+	// a failed run with the error, and it is never scheduled.
	277	+	func TestScheduleInvalidTimezone(t *testing.T) {
	278	+	c, clk, _ := newScheduleHarness(t, "2026-06-01 06:58:30")
	279	+	wf := map[string]string{".github/workflows/nightly.yml": "name: nightly\non:\n push:\n schedule:\n - cron: '* * * * *'\n timezone: Mars/Olympus\njobs:\n j:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n"}
	280	+	c.push("main", wf, "")
	281	+	c.waitFor("the invalid run", func() bool { return len(c.runs()) == 1 })
	282	+	run := c.runs()[0]
	283	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, run.GetConclusion())
	284	+	assert.Equal(t, ".github/workflows/nightly.yml", run.GetName())
	285	+	assert.Contains(t, c.getRun(run.GetId()).GetRun().GetError(), `.github/workflows/nightly.yml: invalid schedule timezone "Mars/Olympus"`)
	286	+	clk.set("2026-06-01 07:00:10")
	287	+	c.forge.CI.ScheduleNow(c.ctx)
	288	+	assert.Empty(t, c.scheduled())
	289	+	}


@@ -255,3 +255,35 @@ func TestScheduleBoundsTheWorkPerTick(t *testing.T) {
255	assert.Equal(t, time.Date(2026, 3, 4, 10, 1, 0, 0, time.UTC), ci.SchedulePassed(), "three ticks go through three repositories")	255		assert.Equal(t, time.Date(2026, 3, 4, 10, 1, 0, 0, time.UTC), ci.SchedulePassed(), "three ticks go through three repositories")
256	assert.Len(t, c.scheduled(), 1)	256		assert.Len(t, c.scheduled(), 1)
257	}	257		}
		258	+
		259	+	// CI-4 D6: the timezone of an entry is the wall clock its cron reads.
		260	+	func TestScheduleTimezone(t *testing.T) {
		261	+	c, clk, _ := newScheduleHarness(t, "2026-06-01 06:58:30")
		262	+	wf := map[string]string{".github/workflows/nightly.yml": "name: nightly\non:\n schedule:\n - cron: '0 9 * * *'\n timezone: Europe/Paris\njobs:\n j:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n"}
		263	+	c.push("main", wf, "")
		264	+	tick := func(at string) {
		265	+	clk.set(at)
		266	+	c.forge.CI.ScheduleNow(c.ctx)
		267	+	}
		268	+	tick("2026-06-01 06:59:10")
		269	+	tick("2026-06-01 09:00:10")
		270	+	assert.Empty(t, c.scheduled(), "09:00 UTC is 11:00 in Paris")
		271	+	tick("2026-06-02 07:00:10")
		272	+	require.Len(t, c.scheduled(), 1, "09:00 in Paris")
		273	+	}
		274	+
		275	+	// A timezone that is no IANA name makes the workflow invalid: its push records
		276	+	// a failed run with the error, and it is never scheduled.
		277	+	func TestScheduleInvalidTimezone(t *testing.T) {
		278	+	c, clk, _ := newScheduleHarness(t, "2026-06-01 06:58:30")
		279	+	wf := map[string]string{".github/workflows/nightly.yml": "name: nightly\non:\n push:\n schedule:\n - cron: '* * * * *'\n timezone: Mars/Olympus\njobs:\n j:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n"}
		280	+	c.push("main", wf, "")
		281	+	c.waitFor("the invalid run", func() bool { return len(c.runs()) == 1 })
		282	+	run := c.runs()[0]
		283	+	assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, run.GetConclusion())
		284	+	assert.Equal(t, ".github/workflows/nightly.yml", run.GetName())
		285	+	assert.Contains(t, c.getRun(run.GetId()).GetRun().GetError(), `.github/workflows/nightly.yml: invalid schedule timezone "Mars/Olympus"`)
		286	+	clk.set("2026-06-01 07:00:10")
		287	+	c.forge.CI.ScheduleNow(c.ctx)
		288	+	assert.Empty(t, c.scheduled())
		289	+	}

M internal/domain/ci_concurrency.go +117 -11


@@ -3,6 +3,7 @@
3	3		package domain
4	4
5	5		import (
	6	+	"slices"
6	7		"strconv"
7	8		"unicode"
8	9		"unicode/utf16"

@@ -16,7 +17,7 @@ const (
16	17		// CIConcurrencyIdle: the group is evaluated but the entry has not asked for
17	18		// its turn yet.
18	19		CIConcurrencyIdle CIConcurrencyState = ""
19		-	// CIConcurrencyPending: the group record names this entry as the waiting
	20	+	// CIConcurrencyPending: the group record names this entry as a waiting
20	21		// one, behind the running entry.
21	22		CIConcurrencyPending CIConcurrencyState = "pending"
22	23		// CIConcurrencyHolding: this entry is the running one of its group.

@@ -24,13 +25,31 @@ const (
24	25		// CIConcurrencyReplaced: only in a mark. A newer entry replaced this one:
25	26		// it is owed a cancellation and never acquires the group.
26	27		CIConcurrencyReplaced CIConcurrencyState = "replaced"
	28	+	// CIConcurrencyRejected: only in a mark. A queue: max entry that arrived
	29	+	// while CIMaxConcurrencyQueue entries were waiting: like a replaced entry it
	30	+	// is owed a cancellation and never acquires the group, but its annotation is
	31	+	// ConcurrencyQueueFullMessage.
	32	+	CIConcurrencyRejected CIConcurrencyState = "rejected"
27	33		)
28	34
	35	+	// Owed reports whether a mark in this state is owed a cancellation: the entry
	36	+	// was replaced or rejected, and never acquires its group.
	37	+	func (s CIConcurrencyState) Owed() bool {
	38	+	return s == CIConcurrencyReplaced \|\| s == CIConcurrencyRejected
	39	+	}
	40	+
29	41		// CIMaxConcurrencyGroup is the longest concurrency group name GitHub accepts
30	42		// (actions/runner WorkflowTemplateConverter.cs:679-687, "must be less than 400
31	43		// characters").
32	44		const CIMaxConcurrencyGroup = 400
33	45
	46	+	// CIMaxConcurrencyQueue is how many entries may wait in a group with queue: max
	47	+	// (docs.github.com, data/reusables/actions/actions-group-concurrency.md: "Up to
	48	+	// 100 jobs or workflow runs can be pending in the concurrency group. When the
	49	+	// queue is full, any additional jobs or workflow runs are canceled."; the limits
	50	+	// page: "up to 100 jobs or workflow runs can be queued per concurrency group").
	51	+	const CIMaxConcurrencyQueue = 100
	52	+
34	53		// CIConcurrency is the evaluated "concurrency:" of a run (workflow level) or of
35	54		// a job.
36	55		type CIConcurrency struct {

@@ -39,6 +58,8 @@ type CIConcurrency struct {
39	58		Group string
40	59		// Cancel is the evaluated cancel-in-progress.
41	60		Cancel bool
	61	+	// Max is set for queue: max (the default, queue: single, leaves it false).
	62	+	Max bool
42	63		// State is written to the record of the entry only after the group record
43	64		// named it (acquire committed): an entry that waits in a group that does
44	65		// not name it any more was replaced, and is cancelled.

@@ -83,6 +104,13 @@ func ConcurrencyCancelMessage(group string) string {
83	104		return "Canceling since a higher priority waiting request for '" + group + "' exists"
84	105		}
85	106
	107	+	// ConcurrencyQueueFullMessage is the annotation of a queue: max entry cancelled
	108	+	// because CIMaxConcurrencyQueue entries already wait in its group. GitHub's text
	109	+	// is not documented: this one is ermite's.
	110	+	func ConcurrencyQueueFullMessage(group string) string {
	111	+	return "Canceling since the concurrency group '" + group + "' already has " + strconv.Itoa(CIMaxConcurrencyQueue) + " waiting requests"
	112	+	}
	113	+
86	114		// CIConcurrencyEntry is a run (JobID empty) or a job waiting for or holding a
87	115		// concurrency group.
88	116		type CIConcurrencyEntry struct {

@@ -96,6 +124,8 @@ type CIConcurrencyEntry struct {
96	124		// cancel-in-progress asked to stop; it stays named until it has ended, and
97	125		// the reconciliation asks it again.
98	126		Canceling bool
	127	+	// Max is the queue mode of the entry: queue: max when set, else single.
	128	+	Max bool
99	129		}
100	130
101	131		// Key identifies the entry in the marks of its group.

@@ -108,27 +138,103 @@ func (e CIConcurrencyEntry) Same(o CIConcurrencyEntry) bool {
108	138		return e.RunID == o.RunID && e.Attempt == o.Attempt && e.JobID == o.JobID
109	139		}
110	140
111		-	// CIConcurrencyRecord is the state of a group: at most one running entry and
112		-	// one pending entry (queue: single).
	141	+	// CIConcurrencyRecord is the state of a group: at most one running entry, and
	142	+	// the waiting entries in the order they started waiting (first in, first out).
	143	+	//
	144	+	// GitHub documents each queue mode alone; a group that mixes them follows
	145	+	// ermite's rules (Enqueue): a queue: single entry that starts waiting cancels
	146	+	// and replaces every waiting entry, whatever their mode, and is then the only
	147	+	// one; a queue: max entry joins the end of the line, or is rejected when
	148	+	// CIMaxConcurrencyQueue entries already wait. When the running entry leaves,
	149	+	// the head of the line takes its place (Promote); a waiting entry that leaves
	150	+	// keeps the order of the others.
113	151		type CIConcurrencyRecord struct {
114	152		Running *CIConcurrencyEntry
115		-	Pending *CIConcurrencyEntry
	153	+	Pending []CIConcurrencyEntry
116	154		}
117	155
118	156		// IsEmpty reports whether nothing runs or waits in the group.
119		-	func (r CIConcurrencyRecord) IsEmpty() bool { return r.Running == nil && r.Pending == nil }
	157	+	func (r CIConcurrencyRecord) IsEmpty() bool { return r.Running == nil && len(r.Pending) == 0 }
120	158
121		-	// Names reports whether the record names the entry as the running or the
122		-	// pending one.
	159	+	// Names reports whether the record names the entry as the running or a pending
	160	+	// one.
123	161		func (r CIConcurrencyRecord) Names(e CIConcurrencyEntry) bool {
124		-	return (r.Running != nil && r.Running.Same(e)) \|\| (r.Pending != nil && r.Pending.Same(e))
	162	+	return (r.Running != nil && r.Running.Same(e)) \|\| r.PendingIndex(e) >= 0
	163	+	}
	164	+
	165	+	// PendingIndex is the place of the entry in the waiting line, -1 if it does not
	166	+	// wait.
	167	+	func (r CIConcurrencyRecord) PendingIndex(e CIConcurrencyEntry) int {
	168	+	for i, p := range r.Pending {
	169	+	if p.Same(e) {
	170	+	return i
	171	+	}
	172	+	}
	173	+	return -1
	174	+	}
	175	+
	176	+	// RemovePending takes the entry out of the waiting line, keeping the order of
	177	+	// the others, and reports whether it was there.
	178	+	func (r *CIConcurrencyRecord) RemovePending(e CIConcurrencyEntry) bool {
	179	+	i := r.PendingIndex(e)
	180	+	if i < 0 {
	181	+	return false
	182	+	}
	183	+	r.Pending = append(r.Pending[:i:i], r.Pending[i+1:]...)
	184	+	if len(r.Pending) == 0 {
	185	+	r.Pending = nil
	186	+	}
	187	+	return true
	188	+	}
	189	+
	190	+	// Enqueue puts a new entry, which neither runs nor waits, in the waiting line.
	191	+	// A queue: single entry replaces every waiting entry (returned, in line order)
	192	+	// and is then the only one; a queue: max entry is appended, unless limit
	193	+	// entries already wait: then the record is left as it is and rejected is set.
	194	+	func (r *CIConcurrencyRecord) Enqueue(e CIConcurrencyEntry, limit int) (replaced []CIConcurrencyEntry, rejected bool) {
	195	+	if !e.Max {
	196	+	replaced = r.Pending
	197	+	r.Pending = []CIConcurrencyEntry{e}
	198	+	return replaced, false
	199	+	}
	200	+	if len(r.Pending) >= limit {
	201	+	return nil, true
	202	+	}
	203	+	r.Pending = append(r.Pending, e)
	204	+	return nil, false
	205	+	}
	206	+
	207	+	// Promote gives the free running place to the head of the waiting line, and
	208	+	// returns it (nil when the place is taken or nobody waits).
	209	+	func (r CIConcurrencyRecord) Promote() CIConcurrencyEntry {
	210	+	if r.Running != nil \|\| len(r.Pending) == 0 {
	211	+	return nil
	212	+	}
	213	+	head := r.Pending[0]
	214	+	r.Pending = r.Pending[1:]
	215	+	if len(r.Pending) == 0 {
	216	+	r.Pending = nil
	217	+	}
	218	+	r.Running = &head
	219	+	return r.Running
	220	+	}
	221	+
	222	+	// Clone is a copy of the record that shares nothing with it.
	223	+	func (r CIConcurrencyRecord) Clone() CIConcurrencyRecord {
	224	+	out := CIConcurrencyRecord{Pending: slices.Clone(r.Pending)}
	225	+	if r.Running != nil {
	226	+	e := *r.Running
	227	+	out.Running = &e
	228	+	}
	229	+	return out
125	230		}
126	231
127	232		// CIConcurrencyMark says where an entry stands in its group. The marks of a group
128	233		// are written in the same batch as its record: an entry is marked pending or
129		-	// holding if and only if the record names it as the pending or the running
130		-	// entry, and an entry that a newer one replaced is marked replaced in the batch
131		-	// that unnames it. So a replaced entry is found from the marks alone, whatever
	234	+	// holding if and only if the record names it as a pending or the running
	235	+	// entry, an entry that a newer one replaced is marked replaced in the batch
	236	+	// that unnames it, and a queue: max entry that found the line full is marked
	237	+	// rejected in the batch that refused it. So a replaced entry is found from the marks alone, whatever
132	238		// failed after that batch.
133	239		type CIConcurrencyMark struct {
134	240		Entry CIConcurrencyEntry


@@ -3,6 +3,7 @@
3		package domain	3		package domain
4			4
5		import (	5		import (
			6	+	"slices"
6		"strconv"	7		"strconv"
7		"unicode"	8		"unicode"
8		"unicode/utf16"	9		"unicode/utf16"

@@ -16,7 +17,7 @@ const (
16		// CIConcurrencyIdle: the group is evaluated but the entry has not asked for	17		// CIConcurrencyIdle: the group is evaluated but the entry has not asked for
17		// its turn yet.	18		// its turn yet.
18		CIConcurrencyIdle CIConcurrencyState = ""	19		CIConcurrencyIdle CIConcurrencyState = ""
19	-	// CIConcurrencyPending: the group record names this entry as the waiting	20	+	// CIConcurrencyPending: the group record names this entry as a waiting
20		// one, behind the running entry.	21		// one, behind the running entry.
21		CIConcurrencyPending CIConcurrencyState = "pending"	22		CIConcurrencyPending CIConcurrencyState = "pending"
22		// CIConcurrencyHolding: this entry is the running one of its group.	23		// CIConcurrencyHolding: this entry is the running one of its group.

@@ -24,13 +25,31 @@ const (
24		// CIConcurrencyReplaced: only in a mark. A newer entry replaced this one:	25		// CIConcurrencyReplaced: only in a mark. A newer entry replaced this one:
25		// it is owed a cancellation and never acquires the group.	26		// it is owed a cancellation and never acquires the group.
26		CIConcurrencyReplaced CIConcurrencyState = "replaced"	27		CIConcurrencyReplaced CIConcurrencyState = "replaced"
			28	+	// CIConcurrencyRejected: only in a mark. A queue: max entry that arrived
			29	+	// while CIMaxConcurrencyQueue entries were waiting: like a replaced entry it
			30	+	// is owed a cancellation and never acquires the group, but its annotation is
			31	+	// ConcurrencyQueueFullMessage.
			32	+	CIConcurrencyRejected CIConcurrencyState = "rejected"
27		)	33		)
28			34
			35	+	// Owed reports whether a mark in this state is owed a cancellation: the entry
			36	+	// was replaced or rejected, and never acquires its group.
			37	+	func (s CIConcurrencyState) Owed() bool {
			38	+	return s == CIConcurrencyReplaced \|\| s == CIConcurrencyRejected
			39	+	}
			40	+
29		// CIMaxConcurrencyGroup is the longest concurrency group name GitHub accepts	41		// CIMaxConcurrencyGroup is the longest concurrency group name GitHub accepts
30		// (actions/runner WorkflowTemplateConverter.cs:679-687, "must be less than 400	42		// (actions/runner WorkflowTemplateConverter.cs:679-687, "must be less than 400
31		// characters").	43		// characters").
32		const CIMaxConcurrencyGroup = 400	44		const CIMaxConcurrencyGroup = 400
33			45
			46	+	// CIMaxConcurrencyQueue is how many entries may wait in a group with queue: max
			47	+	// (docs.github.com, data/reusables/actions/actions-group-concurrency.md: "Up to
			48	+	// 100 jobs or workflow runs can be pending in the concurrency group. When the
			49	+	// queue is full, any additional jobs or workflow runs are canceled."; the limits
			50	+	// page: "up to 100 jobs or workflow runs can be queued per concurrency group").
			51	+	const CIMaxConcurrencyQueue = 100
			52	+
34		// CIConcurrency is the evaluated "concurrency:" of a run (workflow level) or of	53		// CIConcurrency is the evaluated "concurrency:" of a run (workflow level) or of
35		// a job.	54		// a job.
36		type CIConcurrency struct {	55		type CIConcurrency struct {

@@ -39,6 +58,8 @@ type CIConcurrency struct {
39		Group string	58		Group string
40		// Cancel is the evaluated cancel-in-progress.	59		// Cancel is the evaluated cancel-in-progress.
41		Cancel bool	60		Cancel bool
			61	+	// Max is set for queue: max (the default, queue: single, leaves it false).
			62	+	Max bool
42		// State is written to the record of the entry only after the group record	63		// State is written to the record of the entry only after the group record
43		// named it (acquire committed): an entry that waits in a group that does	64		// named it (acquire committed): an entry that waits in a group that does
44		// not name it any more was replaced, and is cancelled.	65		// not name it any more was replaced, and is cancelled.

@@ -83,6 +104,13 @@ func ConcurrencyCancelMessage(group string) string {
83		return "Canceling since a higher priority waiting request for '" + group + "' exists"	104		return "Canceling since a higher priority waiting request for '" + group + "' exists"
84		}	105		}
85			106
			107	+	// ConcurrencyQueueFullMessage is the annotation of a queue: max entry cancelled
			108	+	// because CIMaxConcurrencyQueue entries already wait in its group. GitHub's text
			109	+	// is not documented: this one is ermite's.
			110	+	func ConcurrencyQueueFullMessage(group string) string {
			111	+	return "Canceling since the concurrency group '" + group + "' already has " + strconv.Itoa(CIMaxConcurrencyQueue) + " waiting requests"
			112	+	}
			113	+
86		// CIConcurrencyEntry is a run (JobID empty) or a job waiting for or holding a	114		// CIConcurrencyEntry is a run (JobID empty) or a job waiting for or holding a
87		// concurrency group.	115		// concurrency group.
88		type CIConcurrencyEntry struct {	116		type CIConcurrencyEntry struct {

@@ -96,6 +124,8 @@ type CIConcurrencyEntry struct {
96		// cancel-in-progress asked to stop; it stays named until it has ended, and	124		// cancel-in-progress asked to stop; it stays named until it has ended, and
97		// the reconciliation asks it again.	125		// the reconciliation asks it again.
98		Canceling bool	126		Canceling bool
			127	+	// Max is the queue mode of the entry: queue: max when set, else single.
			128	+	Max bool
99		}	129		}
100			130
101		// Key identifies the entry in the marks of its group.	131		// Key identifies the entry in the marks of its group.

@@ -108,27 +138,103 @@ func (e CIConcurrencyEntry) Same(o CIConcurrencyEntry) bool {
108		return e.RunID == o.RunID && e.Attempt == o.Attempt && e.JobID == o.JobID	138		return e.RunID == o.RunID && e.Attempt == o.Attempt && e.JobID == o.JobID
109		}	139		}
110			140
111	-	// CIConcurrencyRecord is the state of a group: at most one running entry and	141	+	// CIConcurrencyRecord is the state of a group: at most one running entry, and
112	-	// one pending entry (queue: single).	142	+	// the waiting entries in the order they started waiting (first in, first out).
			143	+	//
			144	+	// GitHub documents each queue mode alone; a group that mixes them follows
			145	+	// ermite's rules (Enqueue): a queue: single entry that starts waiting cancels
			146	+	// and replaces every waiting entry, whatever their mode, and is then the only
			147	+	// one; a queue: max entry joins the end of the line, or is rejected when
			148	+	// CIMaxConcurrencyQueue entries already wait. When the running entry leaves,
			149	+	// the head of the line takes its place (Promote); a waiting entry that leaves
			150	+	// keeps the order of the others.
113		type CIConcurrencyRecord struct {	151		type CIConcurrencyRecord struct {
114		Running *CIConcurrencyEntry	152		Running *CIConcurrencyEntry
115	-	Pending *CIConcurrencyEntry	153	+	Pending []CIConcurrencyEntry
116		}	154		}
117			155
118		// IsEmpty reports whether nothing runs or waits in the group.	156		// IsEmpty reports whether nothing runs or waits in the group.
119	-	func (r CIConcurrencyRecord) IsEmpty() bool { return r.Running == nil && r.Pending == nil }	157	+	func (r CIConcurrencyRecord) IsEmpty() bool { return r.Running == nil && len(r.Pending) == 0 }
120			158
121	-	// Names reports whether the record names the entry as the running or the	159	+	// Names reports whether the record names the entry as the running or a pending
122	-	// pending one.	160	+	// one.
123		func (r CIConcurrencyRecord) Names(e CIConcurrencyEntry) bool {	161		func (r CIConcurrencyRecord) Names(e CIConcurrencyEntry) bool {
124	-	return (r.Running != nil && r.Running.Same(e)) \|\| (r.Pending != nil && r.Pending.Same(e))	162	+	return (r.Running != nil && r.Running.Same(e)) \|\| r.PendingIndex(e) >= 0
			163	+	}
			164	+
			165	+	// PendingIndex is the place of the entry in the waiting line, -1 if it does not
			166	+	// wait.
			167	+	func (r CIConcurrencyRecord) PendingIndex(e CIConcurrencyEntry) int {
			168	+	for i, p := range r.Pending {
			169	+	if p.Same(e) {
			170	+	return i
			171	+	}
			172	+	}
			173	+	return -1
			174	+	}
			175	+
			176	+	// RemovePending takes the entry out of the waiting line, keeping the order of
			177	+	// the others, and reports whether it was there.
			178	+	func (r *CIConcurrencyRecord) RemovePending(e CIConcurrencyEntry) bool {
			179	+	i := r.PendingIndex(e)
			180	+	if i < 0 {
			181	+	return false
			182	+	}
			183	+	r.Pending = append(r.Pending[:i:i], r.Pending[i+1:]...)
			184	+	if len(r.Pending) == 0 {
			185	+	r.Pending = nil
			186	+	}
			187	+	return true
			188	+	}
			189	+
			190	+	// Enqueue puts a new entry, which neither runs nor waits, in the waiting line.
			191	+	// A queue: single entry replaces every waiting entry (returned, in line order)
			192	+	// and is then the only one; a queue: max entry is appended, unless limit
			193	+	// entries already wait: then the record is left as it is and rejected is set.
			194	+	func (r *CIConcurrencyRecord) Enqueue(e CIConcurrencyEntry, limit int) (replaced []CIConcurrencyEntry, rejected bool) {
			195	+	if !e.Max {
			196	+	replaced = r.Pending
			197	+	r.Pending = []CIConcurrencyEntry{e}
			198	+	return replaced, false
			199	+	}
			200	+	if len(r.Pending) >= limit {
			201	+	return nil, true
			202	+	}
			203	+	r.Pending = append(r.Pending, e)
			204	+	return nil, false
			205	+	}
			206	+
			207	+	// Promote gives the free running place to the head of the waiting line, and
			208	+	// returns it (nil when the place is taken or nobody waits).
			209	+	func (r CIConcurrencyRecord) Promote() CIConcurrencyEntry {
			210	+	if r.Running != nil \|\| len(r.Pending) == 0 {
			211	+	return nil
			212	+	}
			213	+	head := r.Pending[0]
			214	+	r.Pending = r.Pending[1:]
			215	+	if len(r.Pending) == 0 {
			216	+	r.Pending = nil
			217	+	}
			218	+	r.Running = &head
			219	+	return r.Running
			220	+	}
			221	+
			222	+	// Clone is a copy of the record that shares nothing with it.
			223	+	func (r CIConcurrencyRecord) Clone() CIConcurrencyRecord {
			224	+	out := CIConcurrencyRecord{Pending: slices.Clone(r.Pending)}
			225	+	if r.Running != nil {
			226	+	e := *r.Running
			227	+	out.Running = &e
			228	+	}
			229	+	return out
125		}	230		}
126			231
127		// CIConcurrencyMark says where an entry stands in its group. The marks of a group	232		// CIConcurrencyMark says where an entry stands in its group. The marks of a group
128		// are written in the same batch as its record: an entry is marked pending or	233		// are written in the same batch as its record: an entry is marked pending or
129	-	// holding if and only if the record names it as the pending or the running	234	+	// holding if and only if the record names it as a pending or the running
130	-	// entry, and an entry that a newer one replaced is marked replaced in the batch	235	+	// entry, an entry that a newer one replaced is marked replaced in the batch
131	-	// that unnames it. So a replaced entry is found from the marks alone, whatever	236	+	// that unnames it, and a queue: max entry that found the line full is marked
			237	+	// rejected in the batch that refused it. So a replaced entry is found from the marks alone, whatever
132		// failed after that batch.	238		// failed after that batch.
133		type CIConcurrencyMark struct {	239		type CIConcurrencyMark struct {
134		Entry CIConcurrencyEntry	240		Entry CIConcurrencyEntry

M internal/domain/ci_concurrency_test.go +57 -0


@@ -7,6 +7,7 @@ import (
7	7		"testing"
8	8
9	9		"github.com/stretchr/testify/assert"
	10	+	"github.com/stretchr/testify/require"
10	11
11	12		"bnema.dev/ermite/internal/domain"
12	13		)

@@ -45,3 +46,59 @@ func TestConcurrencyGroupLengthCountsUTF16Units(t *testing.T) {
45	46		assert.Equal(t, 2, domain.ConcurrencyGroupLength("😀"), "a supplementary character is two")
46	47		assert.Equal(t, 400, domain.ConcurrencyGroupLength(strings.Repeat("😀", 200)))
47	48		}
	49	+
	50	+	// queue: max lines up to GitHub's 100 waiting entries in order and rejects one
	51	+	// more; a waiting entry that leaves keeps the order of the others; the head
	52	+	// takes the free running place.
	53	+	func TestConcurrencyQueueMax(t *testing.T) {
	54	+	t.Parallel()
	55	+	assert.Equal(t, 100, domain.CIMaxConcurrencyQueue)
	56	+	rec := domain.CIConcurrencyRecord{Running: &domain.CIConcurrencyEntry{RunID: 1, Attempt: 1}}
	57	+	for i := range domain.CIMaxConcurrencyQueue {
	58	+	replaced, rejected := rec.Enqueue(domain.CIConcurrencyEntry{RunID: int64(10 + i), Attempt: 1, Max: true}, domain.CIMaxConcurrencyQueue)
	59	+	require.False(t, rejected, "entry %d", i)
	60	+	require.Empty(t, replaced)
	61	+	}
	62	+	require.Len(t, rec.Pending, domain.CIMaxConcurrencyQueue)
	63	+	replaced, rejected := rec.Enqueue(domain.CIConcurrencyEntry{RunID: 999, Attempt: 1, Max: true}, domain.CIMaxConcurrencyQueue)
	64	+	assert.True(t, rejected, "the 101st waiting entry is rejected")
	65	+	assert.Empty(t, replaced)
	66	+	assert.Len(t, rec.Pending, domain.CIMaxConcurrencyQueue, "the line is left as it was")
	67	+	assert.False(t, rec.Names(domain.CIConcurrencyEntry{RunID: 999, Attempt: 1}))
	68	+
	69	+	assert.True(t, rec.RemovePending(domain.CIConcurrencyEntry{RunID: 11, Attempt: 1}))
	70	+	assert.False(t, rec.RemovePending(domain.CIConcurrencyEntry{RunID: 11, Attempt: 1}))
	71	+	assert.EqualValues(t, []int64{10, 12, 13}, []int64{rec.Pending[0].RunID, rec.Pending[1].RunID, rec.Pending[2].RunID})
	72	+
	73	+	assert.Nil(t, rec.Promote(), "the running place is taken")
	74	+	rec.Running = nil
	75	+	head := rec.Promote()
	76	+	require.NotNil(t, head)
	77	+	assert.EqualValues(t, 10, head.RunID, "first in, first out")
	78	+	assert.EqualValues(t, 12, rec.Pending[0].RunID)
	79	+	assert.Equal(t, 1, rec.PendingIndex(domain.CIConcurrencyEntry{RunID: 13, Attempt: 1}))
	80	+	}
	81	+
	82	+	// A queue: single entry replaces the whole line, queue: max entries included,
	83	+	// and is then the only waiting one; a max entry joins behind it.
	84	+	func TestConcurrencyQueueSingleClearsTheLine(t *testing.T) {
	85	+	t.Parallel()
	86	+	rec := domain.CIConcurrencyRecord{Running: &domain.CIConcurrencyEntry{RunID: 1, Attempt: 1}}
	87	+	for i := range 3 {
	88	+	rec.Enqueue(domain.CIConcurrencyEntry{RunID: int64(10 + i), Attempt: 1, Max: true}, 3)
	89	+	}
	90	+	single := domain.CIConcurrencyEntry{RunID: 20, Attempt: 1}
	91	+	replaced, rejected := rec.Enqueue(single, 3)
	92	+	assert.False(t, rejected)
	93	+	require.Len(t, replaced, 3)
	94	+	assert.EqualValues(t, 10, replaced[0].RunID)
	95	+	assert.Equal(t, []domain.CIConcurrencyEntry{single}, rec.Pending)
	96	+	maxEntry := domain.CIConcurrencyEntry{RunID: 21, Attempt: 1, Max: true}
	97	+	replaced, _ = rec.Enqueue(maxEntry, 3)
	98	+	assert.Empty(t, replaced)
	99	+	assert.Equal(t, []domain.CIConcurrencyEntry{single, maxEntry}, rec.Pending)
	100	+	assert.True(t, domain.CIConcurrencyRejected.Owed())
	101	+	assert.True(t, domain.CIConcurrencyReplaced.Owed())
	102	+	assert.False(t, domain.CIConcurrencyPending.Owed())
	103	+	assert.Equal(t, "Canceling since the concurrency group 'g' already has 100 waiting requests", domain.ConcurrencyQueueFullMessage("g"))
	104	+	}


@@ -7,6 +7,7 @@ import (
7	"testing"	7		"testing"
8		8
9	"github.com/stretchr/testify/assert"	9		"github.com/stretchr/testify/assert"
		10	+	"github.com/stretchr/testify/require"
10		11
11	"bnema.dev/ermite/internal/domain"	12		"bnema.dev/ermite/internal/domain"
12	)	13		)

@@ -45,3 +46,59 @@ func TestConcurrencyGroupLengthCountsUTF16Units(t *testing.T) {
45	assert.Equal(t, 2, domain.ConcurrencyGroupLength("😀"), "a supplementary character is two")	46		assert.Equal(t, 2, domain.ConcurrencyGroupLength("😀"), "a supplementary character is two")
46	assert.Equal(t, 400, domain.ConcurrencyGroupLength(strings.Repeat("😀", 200)))	47		assert.Equal(t, 400, domain.ConcurrencyGroupLength(strings.Repeat("😀", 200)))
47	}	48		}
		49	+
		50	+	// queue: max lines up to GitHub's 100 waiting entries in order and rejects one
		51	+	// more; a waiting entry that leaves keeps the order of the others; the head
		52	+	// takes the free running place.
		53	+	func TestConcurrencyQueueMax(t *testing.T) {
		54	+	t.Parallel()
		55	+	assert.Equal(t, 100, domain.CIMaxConcurrencyQueue)
		56	+	rec := domain.CIConcurrencyRecord{Running: &domain.CIConcurrencyEntry{RunID: 1, Attempt: 1}}
		57	+	for i := range domain.CIMaxConcurrencyQueue {
		58	+	replaced, rejected := rec.Enqueue(domain.CIConcurrencyEntry{RunID: int64(10 + i), Attempt: 1, Max: true}, domain.CIMaxConcurrencyQueue)
		59	+	require.False(t, rejected, "entry %d", i)
		60	+	require.Empty(t, replaced)
		61	+	}
		62	+	require.Len(t, rec.Pending, domain.CIMaxConcurrencyQueue)
		63	+	replaced, rejected := rec.Enqueue(domain.CIConcurrencyEntry{RunID: 999, Attempt: 1, Max: true}, domain.CIMaxConcurrencyQueue)
		64	+	assert.True(t, rejected, "the 101st waiting entry is rejected")
		65	+	assert.Empty(t, replaced)
		66	+	assert.Len(t, rec.Pending, domain.CIMaxConcurrencyQueue, "the line is left as it was")
		67	+	assert.False(t, rec.Names(domain.CIConcurrencyEntry{RunID: 999, Attempt: 1}))
		68	+
		69	+	assert.True(t, rec.RemovePending(domain.CIConcurrencyEntry{RunID: 11, Attempt: 1}))
		70	+	assert.False(t, rec.RemovePending(domain.CIConcurrencyEntry{RunID: 11, Attempt: 1}))
		71	+	assert.EqualValues(t, []int64{10, 12, 13}, []int64{rec.Pending[0].RunID, rec.Pending[1].RunID, rec.Pending[2].RunID})
		72	+
		73	+	assert.Nil(t, rec.Promote(), "the running place is taken")
		74	+	rec.Running = nil
		75	+	head := rec.Promote()
		76	+	require.NotNil(t, head)
		77	+	assert.EqualValues(t, 10, head.RunID, "first in, first out")
		78	+	assert.EqualValues(t, 12, rec.Pending[0].RunID)
		79	+	assert.Equal(t, 1, rec.PendingIndex(domain.CIConcurrencyEntry{RunID: 13, Attempt: 1}))
		80	+	}
		81	+
		82	+	// A queue: single entry replaces the whole line, queue: max entries included,
		83	+	// and is then the only waiting one; a max entry joins behind it.
		84	+	func TestConcurrencyQueueSingleClearsTheLine(t *testing.T) {
		85	+	t.Parallel()
		86	+	rec := domain.CIConcurrencyRecord{Running: &domain.CIConcurrencyEntry{RunID: 1, Attempt: 1}}
		87	+	for i := range 3 {
		88	+	rec.Enqueue(domain.CIConcurrencyEntry{RunID: int64(10 + i), Attempt: 1, Max: true}, 3)
		89	+	}
		90	+	single := domain.CIConcurrencyEntry{RunID: 20, Attempt: 1}
		91	+	replaced, rejected := rec.Enqueue(single, 3)
		92	+	assert.False(t, rejected)
		93	+	require.Len(t, replaced, 3)
		94	+	assert.EqualValues(t, 10, replaced[0].RunID)
		95	+	assert.Equal(t, []domain.CIConcurrencyEntry{single}, rec.Pending)
		96	+	maxEntry := domain.CIConcurrencyEntry{RunID: 21, Attempt: 1, Max: true}
		97	+	replaced, _ = rec.Enqueue(maxEntry, 3)
		98	+	assert.Empty(t, replaced)
		99	+	assert.Equal(t, []domain.CIConcurrencyEntry{single, maxEntry}, rec.Pending)
		100	+	assert.True(t, domain.CIConcurrencyRejected.Owed())
		101	+	assert.True(t, domain.CIConcurrencyReplaced.Owed())
		102	+	assert.False(t, domain.CIConcurrencyPending.Owed())
		103	+	assert.Equal(t, "Canceling since the concurrency group 'g' already has 100 waiting requests", domain.ConcurrencyQueueFullMessage("g"))
		104	+	}

M internal/domain/ci_cron.go +86 -2


@@ -7,11 +7,15 @@ import (
7	7		"strconv"
8	8		"strings"
9	9		"time"
	10	+
	11	+	// The time zones of schedules do not depend on the host's zoneinfo (the
	12	+	// image has none).
	13	+	_ "time/tzdata"
10	14		)
11	15
12	16		// Cron is a parsed "cron:" of a schedule trigger: five fields (minute, hour, day
13		-	// of month, month, day of week), evaluated in UTC (CI-3 D5; the "timezone" key
14		-	// is ignored). The syntax is that of GitHub's schedule: "*", numbers, lists
	17	+	// of month, month, day of week), matched against a wall clock (UTC, or the
	18	+	// "timezone" of the entry: CronSchedule). The syntax is that of GitHub's schedule: "*", numbers, lists
15	19		// "a,b", ranges "a-b", steps "*/n", "a/n" and "a-b/n", the names jan-dec and
16	20		// sun-sat in any case; the day of week is 0-6 (Sunday is 0).
17	21		//

@@ -132,3 +136,83 @@ func (c Cron) Matches(t time.Time) bool {
132	136		}
133	137		return domOK && dowOK
134	138		}
	139	+
	140	+	// CronSchedule is a cron entry with its time zone (CI-4 D6). GitHub runs the
	141	+	// entry at the wall clock of the zone, UTC when it has none. In a zone with
	142	+	// daylight saving time, an entry whose time is skipped by a spring-forward
	143	+	// transition runs at the next valid time (docs.github.com, "Events that trigger
	144	+	// workflows", schedule: "a 2:30 AM schedule advances to 3:00 AM"). The docs do
	145	+	// not say what a fall-back transition does: a wall clock that happens twice
	146	+	// runs once, at its first occurrence, as cron does.
	147	+	type CronSchedule struct {
	148	+	Cron Cron
	149	+	Loc *time.Location
	150	+	}
	151	+
	152	+	// ParseCronTimezone reads the "timezone" of a schedule entry: an IANA time zone
	153	+	// name, UTC when empty. "Local" (the server's zone) is not one.
	154	+	func ParseCronTimezone(name string) (*time.Location, error) {
	155	+	if name == "" {
	156	+	return time.UTC, nil
	157	+	}
	158	+	if name == "Local" \|\| strings.HasPrefix(name, "/") \|\| strings.Contains(name, "..") {
	159	+	return nil, fmt.Errorf("%w: invalid schedule timezone %q: expected an IANA time zone name such as America/New_York", ErrInvalid, name)
	160	+	}
	161	+	loc, err := time.LoadLocation(name)
	162	+	if err != nil {
	163	+	return nil, fmt.Errorf("%w: invalid schedule timezone %q: expected an IANA time zone name such as America/New_York", ErrInvalid, name)
	164	+	}
	165	+	return loc, nil
	166	+	}
	167	+
	168	+	// cronGapLimit bounds how far back a transition is looked for: the largest
	169	+	// clock change of the time zone database (Samoa skipped a whole day in 2011).
	170	+	const cronGapLimit = 25 * time.Hour
	171	+
	172	+	// Due reports whether the entry fires at the minute that starts at t.
	173	+	func (s CronSchedule) Due(t time.Time) bool {
	174	+	loc := s.Loc
	175	+	if loc == nil {
	176	+	loc = time.UTC
	177	+	}
	178	+	t = t.Truncate(time.Minute)
	179	+	wall := wallMinute(t, loc)
	180	+	if s.Cron.Matches(wall) {
	181	+	// A wall clock already seen in the last hours (a fall-back
	182	+	// transition repeats it) does not fire again.
	183	+	return !s.repeated(t, wall, loc)
	184	+	}
	185	+	// The first minute after a spring-forward gap runs the entries whose wall
	186	+	// clock fell in the gap.
	187	+	prev := wallMinute(t.Add(-time.Minute), loc)
	188	+	for w := prev.Add(time.Minute); w.Before(wall); w = w.Add(time.Minute) {
	189	+	if s.Cron.Matches(w) {
	190	+	return true
	191	+	}
	192	+	}
	193	+	return false
	194	+	}
	195	+
	196	+	// repeated reports whether the wall clock of t was already shown earlier: an
	197	+	// instant t-d has it when the offset there is the offset at t plus d, so only
	198	+	// the larger offsets of the last hours are candidates.
	199	+	func (s CronSchedule) repeated(t, wall time.Time, loc *time.Location) bool {
	200	+	if loc == time.UTC {
	201	+	return false
	202	+	}
	203	+	_, now := t.In(loc).Zone()
	204	+	for h := time.Hour; h <= cronGapLimit; h += time.Hour {
	205	+	_, off := t.Add(-h).In(loc).Zone()
	206	+	if back := time.Duration(off-now) * time.Second; back > 0 && wallMinute(t.Add(-back), loc).Equal(wall) {
	207	+	return true
	208	+	}
	209	+	}
	210	+	return false
	211	+	}
	212	+
	213	+	// wallMinute is the wall clock of t in loc, as a UTC time (Matches reads its
	214	+	// fields).
	215	+	func wallMinute(t time.Time, loc *time.Location) time.Time {
	216	+	l := t.In(loc)
	217	+	return time.Date(l.Year(), l.Month(), l.Day(), l.Hour(), l.Minute(), 0, 0, time.UTC)
	218	+	}


@@ -7,11 +7,15 @@ import (
7		"strconv"	7		"strconv"
8		"strings"	8		"strings"
9		"time"	9		"time"
			10	+
			11	+	// The time zones of schedules do not depend on the host's zoneinfo (the
			12	+	// image has none).
			13	+	_ "time/tzdata"
10		)	14		)
11			15
12		// Cron is a parsed "cron:" of a schedule trigger: five fields (minute, hour, day	16		// Cron is a parsed "cron:" of a schedule trigger: five fields (minute, hour, day
13	-	// of month, month, day of week), evaluated in UTC (CI-3 D5; the "timezone" key	17	+	// of month, month, day of week), matched against a wall clock (UTC, or the
14	-	// is ignored). The syntax is that of GitHub's schedule: "*", numbers, lists	18	+	// "timezone" of the entry: CronSchedule). The syntax is that of GitHub's schedule: "*", numbers, lists
15		// "a,b", ranges "a-b", steps "*/n", "a/n" and "a-b/n", the names jan-dec and	19		// "a,b", ranges "a-b", steps "*/n", "a/n" and "a-b/n", the names jan-dec and
16		// sun-sat in any case; the day of week is 0-6 (Sunday is 0).	20		// sun-sat in any case; the day of week is 0-6 (Sunday is 0).
17		//	21		//

@@ -132,3 +136,83 @@ func (c Cron) Matches(t time.Time) bool {
132		}	136		}
133		return domOK && dowOK	137		return domOK && dowOK
134		}	138		}
			139	+
			140	+	// CronSchedule is a cron entry with its time zone (CI-4 D6). GitHub runs the
			141	+	// entry at the wall clock of the zone, UTC when it has none. In a zone with
			142	+	// daylight saving time, an entry whose time is skipped by a spring-forward
			143	+	// transition runs at the next valid time (docs.github.com, "Events that trigger
			144	+	// workflows", schedule: "a 2:30 AM schedule advances to 3:00 AM"). The docs do
			145	+	// not say what a fall-back transition does: a wall clock that happens twice
			146	+	// runs once, at its first occurrence, as cron does.
			147	+	type CronSchedule struct {
			148	+	Cron Cron
			149	+	Loc *time.Location
			150	+	}
			151	+
			152	+	// ParseCronTimezone reads the "timezone" of a schedule entry: an IANA time zone
			153	+	// name, UTC when empty. "Local" (the server's zone) is not one.
			154	+	func ParseCronTimezone(name string) (*time.Location, error) {
			155	+	if name == "" {
			156	+	return time.UTC, nil
			157	+	}
			158	+	if name == "Local" \|\| strings.HasPrefix(name, "/") \|\| strings.Contains(name, "..") {
			159	+	return nil, fmt.Errorf("%w: invalid schedule timezone %q: expected an IANA time zone name such as America/New_York", ErrInvalid, name)
			160	+	}
			161	+	loc, err := time.LoadLocation(name)
			162	+	if err != nil {
			163	+	return nil, fmt.Errorf("%w: invalid schedule timezone %q: expected an IANA time zone name such as America/New_York", ErrInvalid, name)
			164	+	}
			165	+	return loc, nil
			166	+	}
			167	+
			168	+	// cronGapLimit bounds how far back a transition is looked for: the largest
			169	+	// clock change of the time zone database (Samoa skipped a whole day in 2011).
			170	+	const cronGapLimit = 25 * time.Hour
			171	+
			172	+	// Due reports whether the entry fires at the minute that starts at t.
			173	+	func (s CronSchedule) Due(t time.Time) bool {
			174	+	loc := s.Loc
			175	+	if loc == nil {
			176	+	loc = time.UTC
			177	+	}
			178	+	t = t.Truncate(time.Minute)
			179	+	wall := wallMinute(t, loc)
			180	+	if s.Cron.Matches(wall) {
			181	+	// A wall clock already seen in the last hours (a fall-back
			182	+	// transition repeats it) does not fire again.
			183	+	return !s.repeated(t, wall, loc)
			184	+	}
			185	+	// The first minute after a spring-forward gap runs the entries whose wall
			186	+	// clock fell in the gap.
			187	+	prev := wallMinute(t.Add(-time.Minute), loc)
			188	+	for w := prev.Add(time.Minute); w.Before(wall); w = w.Add(time.Minute) {
			189	+	if s.Cron.Matches(w) {
			190	+	return true
			191	+	}
			192	+	}
			193	+	return false
			194	+	}
			195	+
			196	+	// repeated reports whether the wall clock of t was already shown earlier: an
			197	+	// instant t-d has it when the offset there is the offset at t plus d, so only
			198	+	// the larger offsets of the last hours are candidates.
			199	+	func (s CronSchedule) repeated(t, wall time.Time, loc *time.Location) bool {
			200	+	if loc == time.UTC {
			201	+	return false
			202	+	}
			203	+	_, now := t.In(loc).Zone()
			204	+	for h := time.Hour; h <= cronGapLimit; h += time.Hour {
			205	+	_, off := t.Add(-h).In(loc).Zone()
			206	+	if back := time.Duration(off-now) * time.Second; back > 0 && wallMinute(t.Add(-back), loc).Equal(wall) {
			207	+	return true
			208	+	}
			209	+	}
			210	+	return false
			211	+	}
			212	+
			213	+	// wallMinute is the wall clock of t in loc, as a UTC time (Matches reads its
			214	+	// fields).
			215	+	func wallMinute(t time.Time, loc *time.Location) time.Time {
			216	+	l := t.In(loc)
			217	+	return time.Date(l.Year(), l.Month(), l.Day(), l.Hour(), l.Minute(), 0, 0, time.UTC)
			218	+	}

M internal/domain/ci_cron_test.go +67 -0


@@ -105,3 +105,70 @@ func TestParseCronRefuses(t *testing.T) {
105	105		assert.ErrorIs(t, err, domain.ErrInvalid, bad)
106	106		}
107	107		}
	108	+
	109	+	func TestParseCronTimezone(t *testing.T) {
	110	+	for _, ok := range []string{"", "UTC", "America/New_York", "Asia/Kolkata", "Europe/Paris"} {
	111	+	_, err := domain.ParseCronTimezone(ok)
	112	+	assert.NoError(t, err, ok)
	113	+	}
	114	+	for _, bad := range []string{"Mars/Olympus", "Local", "/etc/localtime", "../zoneinfo/UTC", "new york"} {
	115	+	_, err := domain.ParseCronTimezone(bad)
	116	+	assert.ErrorIs(t, err, domain.ErrInvalid, bad)
	117	+	}
	118	+	}
	119	+
	120	+	// firesOn returns the UTC minutes of the day (from start, for span) at which
	121	+	// the schedule is due.
	122	+	func firesOn(t *testing.T, spec, zone string, start time.Time, span time.Duration) []string {
	123	+	t.Helper()
	124	+	c, err := domain.ParseCron(spec)
	125	+	require.NoError(t, err)
	126	+	loc, err := domain.ParseCronTimezone(zone)
	127	+	require.NoError(t, err)
	128	+	s := domain.CronSchedule{Cron: c, Loc: loc}
	129	+	var out []string
	130	+	for m := start; m.Before(start.Add(span)); m = m.Add(time.Minute) {
	131	+	if s.Due(m) {
	132	+	out = append(out, m.UTC().Format("2006-01-02 15:04"))
	133	+	}
	134	+	}
	135	+	return out
	136	+	}
	137	+
	138	+	func TestCronScheduleTimezone(t *testing.T) {
	139	+	day := func(s string) time.Time { return at(t, s+" 00:00") }
	140	+	// Paris in summer is UTC+2.
	141	+	assert.Equal(t, []string{"2026-06-01 07:00"}, firesOn(t, "0 9 * * ", "Europe/Paris", day("2026-06-01"), 24time.Hour))
	142	+	// No timezone: UTC.
	143	+	assert.Equal(t, []string{"2026-06-01 09:00"}, firesOn(t, "0 9 * * ", "", day("2026-06-01"), 24time.Hour))
	144	+	// The day of week is the zone's: Monday 00:30 in Tokyo is Sunday 15:30 UTC.
	145	+	assert.Equal(t, []string{"2026-05-31 15:30"}, firesOn(t, "30 0 * * mon", "Asia/Tokyo", day("2026-05-31"), 24*time.Hour))
	146	+	}
	147	+
	148	+	// New York springs forward on 2026-03-08 (02:00 EST becomes 03:00 EDT) and
	149	+	// falls back on 2026-11-01 (02:00 EDT becomes 01:00 EST).
	150	+	func TestCronScheduleDST(t *testing.T) {
	151	+	ny := "America/New_York"
	152	+	spring, fall := at(t, "2026-03-08 04:00"), at(t, "2026-11-01 03:00") // local midnight, UTC
	153	+
	154	+	// A time the transition skips runs at the next valid time, 03:00 EDT
	155	+	// (GitHub's docs: "a 2:30 AM schedule advances to 3:00 AM"), once.
	156	+	assert.Equal(t, []string{"2026-03-08 07:00"}, firesOn(t, "30 2 * * ", ny, spring, 24time.Hour))
	157	+	assert.Equal(t, []string{"2026-03-09 06:30"}, firesOn(t, "30 2 * * ", ny, spring.Add(24time.Hour), 24*time.Hour), "the day after: 02:30 EDT")
	158	+	// Times around the gap are not moved, and 03:00 runs once.
	159	+	assert.Equal(t, []string{"2026-03-08 06:59"}, firesOn(t, "59 1 * * ", ny, spring, 24time.Hour))
	160	+	assert.Equal(t, []string{"2026-03-08 07:00"}, firesOn(t, "0 3 * * ", ny, spring, 24time.Hour))
	161	+	assert.Equal(t, []string{"2026-03-08 07:00", "2026-03-08 07:15", "2026-03-08 07:30", "2026-03-08 07:45"},
	162	+	firesOn(t, "/15 2-3 * ", ny, spring, 24time.Hour), "the skipped 02:00-02:45 entries run once at 03:00, then 03:15-03:45")
	163	+	// Hourly: 23 hours that day.
	164	+	assert.Len(t, firesOn(t, "0 * * * ", ny, spring, 23time.Hour), 23)
	165	+
	166	+	// A time that happens twice runs once, at its first occurrence (EDT).
	167	+	assert.Equal(t, []string{"2026-11-01 05:30"}, firesOn(t, "30 1 * * ", ny, fall, 25time.Hour))
	168	+	assert.Equal(t, []string{"2026-11-01 07:00"}, firesOn(t, "0 2 * * ", ny, fall, 25time.Hour), "02:00 EST, once")
	169	+	// Hourly: 24 runs over the 25 hours of that day, none at 01:00 EST.
	170	+	hourly := firesOn(t, "0 * * * ", ny, fall, 25time.Hour)
	171	+	assert.Len(t, hourly, 24)
	172	+	assert.Contains(t, hourly, "2026-11-01 05:00")
	173	+	assert.NotContains(t, hourly, "2026-11-01 06:00")
	174	+	}


@@ -105,3 +105,70 @@ func TestParseCronRefuses(t *testing.T) {
105	assert.ErrorIs(t, err, domain.ErrInvalid, bad)	105		assert.ErrorIs(t, err, domain.ErrInvalid, bad)
106	}	106		}
107	}	107		}
		108	+
		109	+	func TestParseCronTimezone(t *testing.T) {
		110	+	for _, ok := range []string{"", "UTC", "America/New_York", "Asia/Kolkata", "Europe/Paris"} {
		111	+	_, err := domain.ParseCronTimezone(ok)
		112	+	assert.NoError(t, err, ok)
		113	+	}
		114	+	for _, bad := range []string{"Mars/Olympus", "Local", "/etc/localtime", "../zoneinfo/UTC", "new york"} {
		115	+	_, err := domain.ParseCronTimezone(bad)
		116	+	assert.ErrorIs(t, err, domain.ErrInvalid, bad)
		117	+	}
		118	+	}
		119	+
		120	+	// firesOn returns the UTC minutes of the day (from start, for span) at which
		121	+	// the schedule is due.
		122	+	func firesOn(t *testing.T, spec, zone string, start time.Time, span time.Duration) []string {
		123	+	t.Helper()
		124	+	c, err := domain.ParseCron(spec)
		125	+	require.NoError(t, err)
		126	+	loc, err := domain.ParseCronTimezone(zone)
		127	+	require.NoError(t, err)
		128	+	s := domain.CronSchedule{Cron: c, Loc: loc}
		129	+	var out []string
		130	+	for m := start; m.Before(start.Add(span)); m = m.Add(time.Minute) {
		131	+	if s.Due(m) {
		132	+	out = append(out, m.UTC().Format("2006-01-02 15:04"))
		133	+	}
		134	+	}
		135	+	return out
		136	+	}
		137	+
		138	+	func TestCronScheduleTimezone(t *testing.T) {
		139	+	day := func(s string) time.Time { return at(t, s+" 00:00") }
		140	+	// Paris in summer is UTC+2.
		141	+	assert.Equal(t, []string{"2026-06-01 07:00"}, firesOn(t, "0 9 * * ", "Europe/Paris", day("2026-06-01"), 24time.Hour))
		142	+	// No timezone: UTC.
		143	+	assert.Equal(t, []string{"2026-06-01 09:00"}, firesOn(t, "0 9 * * ", "", day("2026-06-01"), 24time.Hour))
		144	+	// The day of week is the zone's: Monday 00:30 in Tokyo is Sunday 15:30 UTC.
		145	+	assert.Equal(t, []string{"2026-05-31 15:30"}, firesOn(t, "30 0 * * mon", "Asia/Tokyo", day("2026-05-31"), 24*time.Hour))
		146	+	}
		147	+
		148	+	// New York springs forward on 2026-03-08 (02:00 EST becomes 03:00 EDT) and
		149	+	// falls back on 2026-11-01 (02:00 EDT becomes 01:00 EST).
		150	+	func TestCronScheduleDST(t *testing.T) {
		151	+	ny := "America/New_York"
		152	+	spring, fall := at(t, "2026-03-08 04:00"), at(t, "2026-11-01 03:00") // local midnight, UTC
		153	+
		154	+	// A time the transition skips runs at the next valid time, 03:00 EDT
		155	+	// (GitHub's docs: "a 2:30 AM schedule advances to 3:00 AM"), once.
		156	+	assert.Equal(t, []string{"2026-03-08 07:00"}, firesOn(t, "30 2 * * ", ny, spring, 24time.Hour))
		157	+	assert.Equal(t, []string{"2026-03-09 06:30"}, firesOn(t, "30 2 * * ", ny, spring.Add(24time.Hour), 24*time.Hour), "the day after: 02:30 EDT")
		158	+	// Times around the gap are not moved, and 03:00 runs once.
		159	+	assert.Equal(t, []string{"2026-03-08 06:59"}, firesOn(t, "59 1 * * ", ny, spring, 24time.Hour))
		160	+	assert.Equal(t, []string{"2026-03-08 07:00"}, firesOn(t, "0 3 * * ", ny, spring, 24time.Hour))
		161	+	assert.Equal(t, []string{"2026-03-08 07:00", "2026-03-08 07:15", "2026-03-08 07:30", "2026-03-08 07:45"},
		162	+	firesOn(t, "/15 2-3 * ", ny, spring, 24time.Hour), "the skipped 02:00-02:45 entries run once at 03:00, then 03:15-03:45")
		163	+	// Hourly: 23 hours that day.
		164	+	assert.Len(t, firesOn(t, "0 * * * ", ny, spring, 23time.Hour), 23)
		165	+
		166	+	// A time that happens twice runs once, at its first occurrence (EDT).
		167	+	assert.Equal(t, []string{"2026-11-01 05:30"}, firesOn(t, "30 1 * * ", ny, fall, 25time.Hour))
		168	+	assert.Equal(t, []string{"2026-11-01 07:00"}, firesOn(t, "0 2 * * ", ny, fall, 25time.Hour), "02:00 EST, once")
		169	+	// Hourly: 24 runs over the 25 hours of that day, none at 01:00 EST.
		170	+	hourly := firesOn(t, "0 * * * ", ny, fall, 25time.Hour)
		171	+	assert.Len(t, hourly, 24)
		172	+	assert.Contains(t, hourly, "2026-11-01 05:00")
		173	+	assert.NotContains(t, hourly, "2026-11-01 06:00")
		174	+	}

M internal/usecase/ci.go +7 -0


@@ -141,6 +141,10 @@ type CIConfig struct {
141	141		// transaction removes (default 256, far below the size limit of a kv batch);
142	142		// the others go with the next decisions and the reconciliation.
143	143		ConcurrencyPruneBatch int
	144	+	// ConcurrencyQueueMax is how many entries may wait in a queue: max group:
	145	+	// 0 is GitHub's limit (domain.CIMaxConcurrencyQueue); the tests lower it so
	146	+	// as not to start a hundred runs.
	147	+	ConcurrencyQueueMax int
144	148		// ScheduleTick is how often the scheduler looks at the clock (default one
145	149		// second) and ScheduleBudget the repositories it handles per tick (default
146	150		// 64); see ci_schedule.go.

@@ -192,6 +196,9 @@ func (c CIConfig) withDefaults() CIConfig {
192	196		if c.TriggerQueue <= 0 {
193	197		c.TriggerQueue = DefaultCITriggerQueue
194	198		}
	199	+	if c.ConcurrencyQueueMax <= 0 {
	200	+	c.ConcurrencyQueueMax = domain.CIMaxConcurrencyQueue
	201	+	}
195	202		if c.ConcurrencyPruneBatch <= 0 {
196	203		c.ConcurrencyPruneBatch = DefaultCIConcurrencyPruneBatch
197	204		}


@@ -141,6 +141,10 @@ type CIConfig struct {
141	// transaction removes (default 256, far below the size limit of a kv batch);	141		// transaction removes (default 256, far below the size limit of a kv batch);
142	// the others go with the next decisions and the reconciliation.	142		// the others go with the next decisions and the reconciliation.
143	ConcurrencyPruneBatch int	143		ConcurrencyPruneBatch int
		144	+	// ConcurrencyQueueMax is how many entries may wait in a queue: max group:
		145	+	// 0 is GitHub's limit (domain.CIMaxConcurrencyQueue); the tests lower it so
		146	+	// as not to start a hundred runs.
		147	+	ConcurrencyQueueMax int
144	// ScheduleTick is how often the scheduler looks at the clock (default one	148		// ScheduleTick is how often the scheduler looks at the clock (default one
145	// second) and ScheduleBudget the repositories it handles per tick (default	149		// second) and ScheduleBudget the repositories it handles per tick (default
146	// 64); see ci_schedule.go.	150		// 64); see ci_schedule.go.

@@ -192,6 +196,9 @@ func (c CIConfig) withDefaults() CIConfig {
192	if c.TriggerQueue <= 0 {	196		if c.TriggerQueue <= 0 {
193	c.TriggerQueue = DefaultCITriggerQueue	197		c.TriggerQueue = DefaultCITriggerQueue
194	}	198		}
		199	+	if c.ConcurrencyQueueMax <= 0 {
		200	+	c.ConcurrencyQueueMax = domain.CIMaxConcurrencyQueue
		201	+	}
195	if c.ConcurrencyPruneBatch <= 0 {	202		if c.ConcurrencyPruneBatch <= 0 {
196	c.ConcurrencyPruneBatch = DefaultCIConcurrencyPruneBatch	203		c.ConcurrencyPruneBatch = DefaultCIConcurrencyPruneBatch
197	}	204		}

M internal/usecase/ci_concurrency.go +122 -84


@@ -19,10 +19,16 @@ import (
19	19
20	20		// Concurrency groups (CI-3 phase E). A group is per repository and compared
21	21		// without regard to case; a run (workflow level) or a job (job level) is an
22		-	// entry. With queue: single, at most one entry runs and one waits: a newer
23		-	// waiting entry cancels the older one, and cancel-in-progress also cancels the
24		-	// running one. An entry holds its group from the moment it may run until the
25		-	// run or job has ended.
	22	+	// entry. At most one entry runs; the others wait in line, first in first out.
	23	+	// With queue: single (the default) a newer waiting entry cancels every waiting
	24	+	// one, and cancel-in-progress also cancels the running one. With queue: max a
	25	+	// newer entry joins the end of the line, up to domain.CIMaxConcurrencyQueue
	26	+	// waiting entries; one more is cancelled (domain.ConcurrencyQueueFullMessage).
	27	+	// GitHub does not document a group whose entries mix both modes: ermite applies
	28	+	// each entry's own rule when it starts waiting (domain.CIConcurrencyRecord.
	29	+	// Enqueue), so a single entry clears the whole line, max ones included. An
	30	+	// entry holds its group from the moment it may run until the run or job has
	31	+	// ended, and the head of the line takes it then.
26	32		//
27	33		// The record of a group lives under "concur/<repo>/<key(group)>" and is
28	34		// changed with a version-checked write (Store.UpdateConcurrency). Cancelling

@@ -41,10 +47,19 @@ var (
41	47		errLongGroup = fmt.Errorf("concurrency group name must be less than %d characters", domain.CIMaxConcurrencyGroup)
42	48		)
43	49
	50	+	// queueValue is the error of a queue that is neither single nor max. The schema
	51	+	// of workflow-parser allows only those ("concurrency-queue": {"allowed-values":
	52	+	// ["single", "max"]}), so a literal is refused when the workflow is parsed; an
	53	+	// expression that evaluates to another value gets the error of the template
	54	+	// reader of languageservices for a value not allowed
	55	+	// (workflow-parser/src/templates/template-reader.ts:444,
	56	+	// "this._context.error(literal, `Unexpected value '${literal.toString()}'`);").
	57	+	func queueValue(v string) error { return fmt.Errorf("Unexpected value '%s'", v) } //nolint:staticcheck // ST1005: the text of languageservices, kept as is
	58	+
44	59		// evalConcurrency evaluates the "concurrency:" token of a workflow or a job:
45	60		// a string (or an expression) is the group; a mapping has group,
46		-	// cancel-in-progress (an expression is allowed) and queue. An empty group is
47		-	// an error, as in GitHub's runner; queue: max is not supported yet.
	61	+	// cancel-in-progress (an expression is allowed) and queue (single or max). An
	62	+	// empty group is an error, as in GitHub's runner.
48	63		func evalConcurrency(ctx context.Context, tok template.Token, env template.Env) (domain.CIConcurrency, error) {
49	64		var out domain.CIConcurrency
50	65		if tok == nil {

@@ -57,13 +72,12 @@ func evalConcurrency(ctx context.Context, tok template.Token, env template.Env)
57	72		if err != nil {
58	73		return out, err
59	74		}
60		-	if queue == "max" {
61		-	// languageservices, languageservice/src/validate.ts:702-743: the
62		-	// literal pair is invalid (expressions are not checked there).
63		-	if b, ok := m.Find("cancel-in-progress").(*template.Boolean); ok && b.Value {
64		-	return out, errQueueMaxCancel
65		-	}
66		-	return out, errors.New("not supported yet: queue: max")
	75	+	switch queue {
	76	+	case "single":
	77	+	case "max":
	78	+	out.Max = true
	79	+	default:
	80	+	return out, queueValue(queue)
67	81		}
68	82		}
69	83		if groupTok = m.Find("group"); groupTok == nil {

@@ -76,6 +90,12 @@ func evalConcurrency(ctx context.Context, tok template.Token, env template.Env)
76	90		}
77	91		out.Cancel = cancel
78	92		}
	93	+	// languageservices, languageservice/src/validate.ts:702-743, refuses the
	94	+	// literal pair; ermite refuses it with the same message whatever wrote
	95	+	// either value (an expression that evaluates to true included).
	96	+	if out.Max && out.Cancel {
	97	+	return domain.CIConcurrency{}, errQueueMaxCancel
	98	+	}
79	99		}
80	100		group, err := template.EvaluateString(ctx, groupTok, env)
81	101		switch {

@@ -143,12 +163,11 @@ func mark(st *domain.CIConcurrencyGroupState, e domain.CIConcurrencyEntry, group
143	163		st.Marks[e.Key()] = domain.CIConcurrencyMark{Entry: e, Group: group, State: state}
144	164		}
145	165
146		-	// promote gives the running place to the waiting entry when it is free.
	166	+	// promote gives the running place, when it is free, to the head of the line.
147	167		func promote(st *domain.CIConcurrencyGroupState, group string) {
148		-	if st.Record.Running == nil && st.Record.Pending != nil {
149		-	st.Record.Running, st.Record.Pending = st.Record.Pending, nil
150		-	m := st.Marks[st.Record.Running.Key()]
151		-	mark(st, *st.Record.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
	168	+	if head := st.Record.Promote(); head != nil {
	169	+	m := st.Marks[head.Key()]
	170	+	mark(st, *head, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
152	171		}
153	172		}
154	173

@@ -162,29 +181,38 @@ func firstNonEmpty(a, b string) string {
162	181		// prune brings a group back to its invariant, in the transaction that is about to
163	182		// decide on it: the entries whose run or job is gone or over leave the record
164	183		// and lose their marks (a crash between the write of a job and the release of
165		-	// its group, a run that was deleted); the waiting entry takes a free running
166		-	// place; an entry named by the record has its mark; and an entry marked as
167		-	// pending or holding that the record does not name any more is marked replaced
168		-	// (it should not happen, the batch writes both, and it is cancelled if it does).
	184	+	// its group, a run that was deleted), the others waiting in the same order; the
	185	+	// head of the line takes a free running place; an entry named by the record has
	186	+	// its mark; and an entry marked as pending or holding that the record does not
	187	+	// name any more is marked replaced (it should not happen, the batch writes both,
	188	+	// and it is cancelled if it does).
169	189		func (c CI) prune(ctx context.Context, repo domain.RepoName, group string, st domain.CIConcurrencyGroupState) (more bool) {
170	190		rec := &st.Record
171	191		if rec.Running != nil && !c.entryAlive(ctx, repo, *rec.Running) {
172	192		delete(st.Marks, rec.Running.Key())
173	193		rec.Running = nil
174	194		}
175		-	if rec.Pending != nil && !c.entryAlive(ctx, repo, *rec.Pending) {
176		-	delete(st.Marks, rec.Pending.Key())
177		-	rec.Pending = nil
	195	+	alive := rec.Pending[:0:0]
	196	+	for _, p := range rec.Pending {
	197	+	if c.entryAlive(ctx, repo, p) {
	198	+	alive = append(alive, p)
	199	+	} else {
	200	+	delete(st.Marks, p.Key())
	201	+	}
	202	+	}
	203	+	if len(alive) == 0 {
	204	+	alive = nil
178	205		}
	206	+	rec.Pending = alive
179	207		promote(st, group)
180	208		if rec.Running != nil {
181	209		if m, ok := st.Marks[rec.Running.Key()]; !ok \|\| m.State != domain.CIConcurrencyHolding {
182	210		mark(st, *rec.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
183	211		}
184	212		}
185		-	if rec.Pending != nil {
186		-	if m, ok := st.Marks[rec.Pending.Key()]; !ok \|\| m.State != domain.CIConcurrencyPending {
187		-	mark(st, *rec.Pending, firstNonEmpty(m.Group, group), domain.CIConcurrencyPending)
	213	+	for _, p := range rec.Pending {
	214	+	if m, ok := st.Marks[p.Key()]; !ok \|\| m.State != domain.CIConcurrencyPending {
	215	+	mark(st, p, firstNonEmpty(m.Group, group), domain.CIConcurrencyPending)
188	216		}
189	217		}
190	218		// The other marks are pruned a bounded number per transaction, in key order:

@@ -198,7 +226,7 @@ func (c CI) prune(ctx context.Context, repo domain.RepoName, group string, st
198	226		switch {
199	227		case !c.entryAlive(ctx, repo, m.Entry):
200	228		prune = true
201		-	case m.State != domain.CIConcurrencyReplaced && !rec.Names(m.Entry):
	229	+	case !m.State.Owed() && !rec.Names(m.Entry):
202	230		replace = true
203	231		default:
204	232		continue

@@ -298,14 +326,16 @@ func (c *CI) wake(repo domain.RepoName, group string, e domain.CIConcurrencyEntr
298	326		})
299	327		}
300	328
301		-	// cancelLater cancels an entry that a newer one replaced, outside any run lock,
302		-	// and again if that failed. The decision that replaced it is in the group (the
303		-	// entry is marked replaced in the batch that unnamed it, or stays named with
304		-	// Canceling set): the reconciliation finds it from there if this cancellation is
305		-	// lost (crash, retries used up).
306		-	func (c *CI) cancelLater(group string, e domain.CIConcurrencyEntry) {
	329	+	// cancelLater cancels an entry that a newer one replaced (why is
	330	+	// domain.CIConcurrencyReplaced) or that found the line full (why is
	331	+	// domain.CIConcurrencyRejected), outside any run lock, and again if that
	332	+	// failed. The decision is in the group (the entry is marked replaced or
	333	+	// rejected in the batch that decided it, or stays named with Canceling set):
	334	+	// the reconciliation finds it from there if this cancellation is lost (crash,
	335	+	// retries used up).
	336	+	func (c *CI) cancelLater(group string, e domain.CIConcurrencyEntry, why domain.CIConcurrencyState) {
307	337		c.later("concurrency cancel", func(ctx context.Context) error {
308		-	return c.retry(ctx, func() error { return c.cancelEntry(ctx, e, group) })
	338	+	return c.retry(ctx, func() error { return c.cancelEntry(ctx, e, group, why) })
309	339		})
310	340		}
311	341

@@ -340,15 +370,15 @@ func (c *CI) entryNeeds(ctx context.Context, repo domain.RepoName, group string,
340	370		// ReconcileConcurrency repairs the groups. It runs at startup and every minute.
341	371		// For each stored group it prunes the entries that are gone or over, asks again
342	372		// the running entry a cancel-in-progress entry replaced, cancels every entry
343		-	// marked replaced (their cancellation is owed), and plans the entries the record
344		-	// names whose own record does not know it yet.
	373	+	// marked replaced or rejected (their cancellation is owed), and plans the
	374	+	// entries the record names whose own record does not know it yet.
345	375		//
346	376		// Cost, per pass: a scan of the stored groups and their marks, and for each
347	377		// group a version-checked transaction (a write only if something is repaired;
348	378		// at most CIConfig.ConcurrencyPruneBatch stale marks go in one, the group is
349	379		// taken again until none is left) plus a read of the run or job of each entry
350		-	// the record or the marks name. It grows with the groups in use, not with the
351		-	// open runs.
	380	+	// the record or the marks name (at most CIConfig.ConcurrencyQueueMax waiting
	381	+	// ones per group). It grows with the groups in use, not with the open runs.
352	382		func (c *CI) ReconcileConcurrency(ctx context.Context) {
353	383		log := ctxLog(ctx)
354	384		groups, err := c.cfg.Store.ListConcurrency(ctx)

@@ -363,15 +393,7 @@ func (c *CI) ReconcileConcurrency(ctx context.Context) {
363	393		// left to prune (each pass removes some, so this ends).
364	394		for more := true; more && err == nil && ctx.Err() == nil; {
365	395		_, more, err = c.updateGroupMore(ctx, g.Repo, g.Group, func(cur *domain.CIConcurrencyGroupState) error {
366		-	st = domain.CIConcurrencyGroupState{Marks: maps.Clone(cur.Marks)}
367		-	if cur.Record.Running != nil {
368		-	r := *cur.Record.Running
369		-	st.Record.Running = &r
370		-	}
371		-	if cur.Record.Pending != nil {
372		-	p := *cur.Record.Pending
373		-	st.Record.Pending = &p
374		-	}
	396	+	st = domain.CIConcurrencyGroupState{Record: cur.Record.Clone(), Marks: maps.Clone(cur.Marks)}
375	397		return nil
376	398		})
377	399		}

@@ -383,18 +405,20 @@ func (c *CI) ReconcileConcurrency(ctx context.Context) {
383	405		}
384	406		if r := st.Record.Running; r != nil {
385	407		if r.Canceling {
386		-	c.cancelLater(st.Marks[r.Key()].Group, *r)
	408	+	c.cancelLater(st.Marks[r.Key()].Group, *r, domain.CIConcurrencyReplaced)
387	409		}
388	410		if c.entryNeeds(ctx, g.Repo, g.Group, *r, domain.CIConcurrencyHolding) {
389	411		c.wake(g.Repo, g.Group, *r)
390	412		}
391	413		}
392		-	if p := st.Record.Pending; p != nil && c.entryNeeds(ctx, g.Repo, g.Group, *p, domain.CIConcurrencyPending) {
393		-	c.wake(g.Repo, g.Group, *p)
	414	+	for _, p := range st.Record.Pending {
	415	+	if c.entryNeeds(ctx, g.Repo, g.Group, p, domain.CIConcurrencyPending) {
	416	+	c.wake(g.Repo, g.Group, p)
	417	+	}
394	418		}
395	419		for _, m := range st.Marks {
396		-	if m.State == domain.CIConcurrencyReplaced {
397		-	c.cancelLater(m.Group, m.Entry)
	420	+	if m.State.Owed() {
	421	+	c.cancelLater(m.Group, m.Entry, m.State)
398	422		}
399	423		}
400	424		}

@@ -405,51 +429,60 @@ func (c *CI) ReconcileConcurrency(ctx context.Context) {
405	429		// marks of every entry it touches are one batch (Store.UpdateConcurrency), so an
406	430		// entry is marked pending or holding if and only if the record names it.
407	431		//
408		-	// An entry marked replaced never acquires, whatever the state of the group
409		-	// (empty, or its waiting place free): it is owed a cancellation. Otherwise a
410		-	// pending entry stays so; a newer one replaces the waiting entry, which is
411		-	// marked replaced and cancelled, and with cancel-in-progress the running entry
412		-	// is cancelled too (it stays named, with Canceling set, until it has ended).
	432	+	// An entry marked replaced or rejected never acquires, whatever the state of
	433	+	// the group (empty, or its line free): it is owed a cancellation. Otherwise a
	434	+	// pending entry stays so, at its place in the line; a newer one is queued
	435	+	// (domain.CIConcurrencyRecord.Enqueue): with queue: single it replaces every
	436	+	// waiting entry, each marked replaced and cancelled, and with
	437	+	// cancel-in-progress the running entry is cancelled too (it stays named, with
	438	+	// Canceling set, until it has ended); with queue: max it joins the end of the
	439	+	// line, or, when the line is full, is marked rejected and cancelled itself.
413	440		// The call is repeatable: an entry that holds or waits is told so again.
414	441		func (rs *runState) acquire(ctx context.Context, group string, e domain.CIConcurrencyEntry) (domain.CIConcurrencyState, error) {
	442	+	type victim struct {
	443	+	e domain.CIConcurrencyEntry
	444	+	why domain.CIConcurrencyState
	445	+	}
415	446		var state domain.CIConcurrencyState
416		-	var victims []domain.CIConcurrencyEntry
	447	+	var victims []victim
417	448		promoted, err := rs.c.updateGroup(ctx, rs.run.Repo, group, func(st *domain.CIConcurrencyGroupState) error {
418	449		victims = nil
419	450		rec := &st.Record
420		-	if m, ok := st.Marks[e.Key()]; ok && m.State == domain.CIConcurrencyReplaced {
	451	+	if m, ok := st.Marks[e.Key()]; ok && m.State.Owed() {
421	452		state = domain.CIConcurrencyReplaced
422		-	victims = append(victims, e) // its cancellation may have been lost
	453	+	victims = append(victims, victim{e, m.State}) // its cancellation may have been lost
423	454		return nil
424	455		}
425	456		switch {
426	457		case rec.Running == nil:
427	458		me := e
428	459		rec.Running = &me
429		-	if rec.Pending != nil && rec.Pending.Same(e) {
430		-	rec.Pending = nil
431		-	}
	460	+	rec.RemovePending(e)
432	461		mark(st, e, group, domain.CIConcurrencyHolding)
433	462		state = domain.CIConcurrencyHolding
434	463		case rec.Running.Same(e):
435	464		state = domain.CIConcurrencyHolding
436		-	case rec.Pending != nil && rec.Pending.Same(e):
	465	+	case rec.PendingIndex(e) >= 0:
437	466		state = domain.CIConcurrencyPending
438	467		default:
	468	+	replaced, rejected := rec.Enqueue(e, rs.c.cfg.ConcurrencyQueueMax)
	469	+	if rejected {
	470	+	state = domain.CIConcurrencyReplaced
	471	+	mark(st, e, group, domain.CIConcurrencyRejected)
	472	+	victims = append(victims, victim{e, domain.CIConcurrencyRejected})
	473	+	return nil
	474	+	}
439	475		state = domain.CIConcurrencyPending
440		-	if rec.Pending != nil {
441		-	old := *rec.Pending
	476	+	for _, old := range replaced {
442	477		mark(st, old, group, domain.CIConcurrencyReplaced)
443		-	victims = append(victims, old)
	478	+	victims = append(victims, victim{old, domain.CIConcurrencyReplaced})
444	479		}
445		-	me := e
446		-	rec.Pending = &me
447	480		mark(st, e, group, domain.CIConcurrencyPending)
448	481		if e.Cancel && !rec.Running.Canceling {
449	482		run := *rec.Running
450	483		run.Canceling = true
451	484		rec.Running = &run
452		-	victims = append(victims, run)
	485	+	victims = append(victims, victim{run, domain.CIConcurrencyReplaced})
453	486		}
454	487		}
455	488		return nil

@@ -461,7 +494,7 @@ func (rs *runState) acquire(ctx context.Context, group string, e domain.CIConcur
461	494		rs.c.wake(rs.run.Repo, group, *promoted)
462	495		}
463	496		for _, v := range victims {
464		-	rs.c.cancelLater(group, v)
	497	+	rs.c.cancelLater(group, v.e, v.why)
465	498		}
466	499		if state == domain.CIConcurrencyReplaced {
467	500		state = domain.CIConcurrencyPending // it shows as waiting until its cancellation lands

@@ -470,16 +503,17 @@ func (rs *runState) acquire(ctx context.Context, group string, e domain.CIConcur
470	503		}
471	504
472	505		// releaseEntry takes an entry out of its group, because its run or job has
473		-	// ended, and gives the turn to the waiting entry, which is planned again. The
474		-	// mark of the entry goes in the same batch.
	506	+	// ended, and gives the turn to the head of the line, which is planned again; a
	507	+	// waiting entry that leaves keeps the order of the others. The mark of the
	508	+	// entry goes in the same batch.
475	509		func (c *CI) releaseEntry(ctx context.Context, repo domain.RepoName, group string, e domain.CIConcurrencyEntry) error {
476	510		promoted, err := c.updateGroup(ctx, repo, group, func(st *domain.CIConcurrencyGroupState) error {
477	511		switch {
478	512		case st.Record.Running != nil && st.Record.Running.Same(e):
479	513		st.Record.Running = nil
480	514		promote(st, group)
481		-	case st.Record.Pending != nil && st.Record.Pending.Same(e):
482		-	st.Record.Pending = nil
	515	+	default:
	516	+	st.Record.RemovePending(e)
483	517		}
484	518		delete(st.Marks, e.Key())
485	519		return nil

@@ -491,10 +525,11 @@ func (c *CI) releaseEntry(ctx context.Context, repo domain.RepoName, group strin
491	525		return nil
492	526		}
493	527
494		-	// cancelEntry cancels a run or a job that a newer entry of the group
495		-	// replaced: it ends cancelled with GitHub's annotation, and a job a runner has
496		-	// gets the cancel request the owner's cancel sends.
497		-	func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group string) error {
	528	+	// cancelEntry cancels a run or a job that a newer entry of the group replaced,
	529	+	// or that found the line full (why is domain.CIConcurrencyRejected): it ends
	530	+	// cancelled with GitHub's annotation (ermite's for a full line), and a job a
	531	+	// runner has gets the cancel request the owner's cancel sends.
	532	+	func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group string, why domain.CIConcurrencyState) error {
498	533		return c.withRun(ctx, e.RunID, e.Attempt, func(rs *runState) error {
499	534		if group == "" { // no name in the mark: use the entry's own group
500	535		group = rs.run.Concurrency.Group

@@ -503,6 +538,9 @@ func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group
503	538		}
504	539		}
505	540		msg := domain.ConcurrencyCancelMessage(group)
	541	+	if why == domain.CIConcurrencyRejected {
	542	+	msg = domain.ConcurrencyQueueFullMessage(group)
	543	+	}
506	544		if e.JobID == "" {
507	545		if !rs.latest() \|\| rs.run.Status == domain.CICompleted \|\| rs.run.CancelRequested {
508	546		return nil

@@ -541,8 +579,8 @@ func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group
541	579		})
542	580		}
543	581
544		-	// cancelForGroup ends a job a newer entry of its group replaced, or asks its
545		-	// runner to stop. The annotation is GitHub's.
	582	+	// cancelForGroup ends a job a newer entry of its group replaced (or that found
	583	+	// the line full), or asks its runner to stop, with the annotation msg.
546	584		func (rs runState) cancelForGroup(j domain.CIJob, msg string) {
547	585		j.Concurrency.Preempted = true
548	586		rs.annotations = append(rs.annotations, ports.CIStepAnnotations{

@@ -582,7 +620,7 @@ func (rs runState) acquireRun(ctx context.Context, wf workflow.Workflow) (bool
582	620		if run.Concurrency.Group == "" {
583	621		return true, nil
584	622		}
585		-	st, err := rs.acquire(ctx, run.Concurrency.Group, domain.CIConcurrencyEntry{RunID: run.ID, Attempt: run.Attempt, Cancel: run.Concurrency.Cancel})
	623	+	st, err := rs.acquire(ctx, run.Concurrency.Group, domain.CIConcurrencyEntry{RunID: run.ID, Attempt: run.Attempt, Cancel: run.Concurrency.Cancel, Max: run.Concurrency.Max})
586	624		if err != nil {
587	625		return false, err
588	626		}

 // Concurrency groups (CI-3 phase E). A group is per repository and compared
 // without regard to case; a run (workflow level) or a job (job level) is an
-// entry. With queue: single, at most one entry runs and one waits: a newer
+// entry. At most one entry runs; the others wait in line, first in first out.
-// waiting entry cancels the older one, and cancel-in-progress also cancels the
+// With queue: single (the default) a newer waiting entry cancels every waiting
-// running one. An entry holds its group from the moment it may run until the
+// one, and cancel-in-progress also cancels the running one. With queue: max a
-// run or job has ended.
+// newer entry joins the end of the line, up to domain.CIMaxConcurrencyQueue
+// waiting entries; one more is cancelled (domain.ConcurrencyQueueFullMessage).
+// GitHub does not document a group whose entries mix both modes: ermite applies
+// each entry's own rule when it starts waiting (domain.CIConcurrencyRecord.
+// Enqueue), so a single entry clears the whole line, max ones included. An
+// entry holds its group from the moment it may run until the run or job has
+// ended, and the head of the line takes it then.
 //
 // The record of a group lives under "concur/<repo>/<key(group)>" and is
 // changed with a version-checked write (Store.UpdateConcurrency). Cancelling
 	errLongGroup      = fmt.Errorf("concurrency group name must be less than %d characters", domain.CIMaxConcurrencyGroup)
 )
+// queueValue is the error of a queue that is neither single nor max. The schema
+// of workflow-parser allows only those ("concurrency-queue": {"allowed-values":
+// ["single", "max"]}), so a literal is refused when the workflow is parsed; an
+// expression that evaluates to another value gets the error of the template
+// reader of languageservices for a value not allowed
+// (workflow-parser/src/templates/template-reader.ts:444,
+// "this._context.error(literal, `Unexpected value '${literal.toString()}'`);").
+func queueValue(v string) error { return fmt.Errorf("Unexpected value '%s'", v) } //nolint:staticcheck // ST1005: the text of languageservices, kept as is
 // evalConcurrency evaluates the "concurrency:" token of a workflow or a job:
 // a string (or an expression) is the group; a mapping has group,
-// cancel-in-progress (an expression is allowed) and queue. An empty group is
+// cancel-in-progress (an expression is allowed) and queue (single or max). An
-// an error, as in GitHub's runner; queue: max is not supported yet.
+// empty group is an error, as in GitHub's runner.
 func evalConcurrency(ctx context.Context, tok template.Token, env template.Env) (domain.CIConcurrency, error) {
 	var out domain.CIConcurrency
 	if tok == nil {
 			if err != nil {
 				return out, err
 			}
-			if queue == "max" {
+			switch queue {
-				// languageservices, languageservice/src/validate.ts:702-743: the
+			case "single":
-				// literal pair is invalid (expressions are not checked there).
+			case "max":
-				if b, ok := m.Find("cancel-in-progress").(*template.Boolean); ok && b.Value {
+				out.Max = true
-					return out, errQueueMaxCancel
+			default:
-				}
+				return out, queueValue(queue)
-				return out, errors.New("not supported yet: queue: max")
 			}
 		}
 		if groupTok = m.Find("group"); groupTok == nil {
 			}
 			out.Cancel = cancel
 		}
+		// languageservices, languageservice/src/validate.ts:702-743, refuses the
+		// literal pair; ermite refuses it with the same message whatever wrote
+		// either value (an expression that evaluates to true included).
+		if out.Max && out.Cancel {
+			return domain.CIConcurrency{}, errQueueMaxCancel
+		}
 	}
 	group, err := template.EvaluateString(ctx, groupTok, env)
 	switch {
 	st.Marks[e.Key()] = domain.CIConcurrencyMark{Entry: e, Group: group, State: state}
 }
-// promote gives the running place to the waiting entry when it is free.
+// promote gives the running place, when it is free, to the head of the line.
 func promote(st *domain.CIConcurrencyGroupState, group string) {
-	if st.Record.Running == nil && st.Record.Pending != nil {
+	if head := st.Record.Promote(); head != nil {
-		st.Record.Running, st.Record.Pending = st.Record.Pending, nil
+		m := st.Marks[head.Key()]
-		m := st.Marks[st.Record.Running.Key()]
+		mark(st, *head, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
-		mark(st, *st.Record.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
 	}
 }
 // prune brings a group back to its invariant, in the transaction that is about to
 // decide on it: the entries whose run or job is gone or over leave the record
 // and lose their marks (a crash between the write of a job and the release of
-// its group, a run that was deleted); the waiting entry takes a free running
+// its group, a run that was deleted), the others waiting in the same order; the
-// place; an entry named by the record has its mark; and an entry marked as
+// head of the line takes a free running place; an entry named by the record has
-// pending or holding that the record does not name any more is marked replaced
+// its mark; and an entry marked as pending or holding that the record does not
-// (it should not happen, the batch writes both, and it is cancelled if it does).
+// name any more is marked replaced (it should not happen, the batch writes both,
+// and it is cancelled if it does).
 func (c *CI) prune(ctx context.Context, repo domain.RepoName, group string, st *domain.CIConcurrencyGroupState) (more bool) {
 	rec := &st.Record
 	if rec.Running != nil && !c.entryAlive(ctx, repo, *rec.Running) {
 		delete(st.Marks, rec.Running.Key())
 		rec.Running = nil
 	}
-	if rec.Pending != nil && !c.entryAlive(ctx, repo, *rec.Pending) {
+	alive := rec.Pending[:0:0]
-		delete(st.Marks, rec.Pending.Key())
+	for _, p := range rec.Pending {
-		rec.Pending = nil
+		if c.entryAlive(ctx, repo, p) {
+			alive = append(alive, p)
+		} else {
+			delete(st.Marks, p.Key())
+		}
+	}
+	if len(alive) == 0 {
+		alive = nil
 	}
+	rec.Pending = alive
 	promote(st, group)
 	if rec.Running != nil {
 		if m, ok := st.Marks[rec.Running.Key()]; !ok || m.State != domain.CIConcurrencyHolding {
 			mark(st, *rec.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
 		}
 	}
-	if rec.Pending != nil {
+	for _, p := range rec.Pending {
-		if m, ok := st.Marks[rec.Pending.Key()]; !ok || m.State != domain.CIConcurrencyPending {
+		if m, ok := st.Marks[p.Key()]; !ok || m.State != domain.CIConcurrencyPending {
-			mark(st, *rec.Pending, firstNonEmpty(m.Group, group), domain.CIConcurrencyPending)
+			mark(st, p, firstNonEmpty(m.Group, group), domain.CIConcurrencyPending)
 		}
 	}
 	// The other marks are pruned a bounded number per transaction, in key order:
 		switch {
 		case !c.entryAlive(ctx, repo, m.Entry):
 			prune = true
-		case m.State != domain.CIConcurrencyReplaced && !rec.Names(m.Entry):
+		case !m.State.Owed() && !rec.Names(m.Entry):
 			replace = true
 		default:
 			continue
 	})
 }
-// cancelLater cancels an entry that a newer one replaced, outside any run lock,
+// cancelLater cancels an entry that a newer one replaced (why is
-// and again if that failed. The decision that replaced it is in the group (the
+// domain.CIConcurrencyReplaced) or that found the line full (why is
-// entry is marked replaced in the batch that unnamed it, or stays named with
+// domain.CIConcurrencyRejected), outside any run lock, and again if that
-// Canceling set): the reconciliation finds it from there if this cancellation is
+// failed. The decision is in the group (the entry is marked replaced or
-// lost (crash, retries used up).
+// rejected in the batch that decided it, or stays named with Canceling set):
-func (c *CI) cancelLater(group string, e domain.CIConcurrencyEntry) {
+// the reconciliation finds it from there if this cancellation is lost (crash,
+// retries used up).
+func (c *CI) cancelLater(group string, e domain.CIConcurrencyEntry, why domain.CIConcurrencyState) {
 	c.later("concurrency cancel", func(ctx context.Context) error {
-		return c.retry(ctx, func() error { return c.cancelEntry(ctx, e, group) })
+		return c.retry(ctx, func() error { return c.cancelEntry(ctx, e, group, why) })
 	})
 }
 // ReconcileConcurrency repairs the groups. It runs at startup and every minute.
 // For each stored group it prunes the entries that are gone or over, asks again
 // the running entry a cancel-in-progress entry replaced, cancels every entry
-// marked replaced (their cancellation is owed), and plans the entries the record
+// marked replaced or rejected (their cancellation is owed), and plans the
-// names whose own record does not know it yet.
+// entries the record names whose own record does not know it yet.
 //
 // Cost, per pass: a scan of the stored groups and their marks, and for each
 // group a version-checked transaction (a write only if something is repaired;
 // at most CIConfig.ConcurrencyPruneBatch stale marks go in one, the group is
 // taken again until none is left) plus a read of the run or job of each entry
-// the record or the marks name. It grows with the groups in use, not with the
+// the record or the marks name (at most CIConfig.ConcurrencyQueueMax waiting
-// open runs.
+// ones per group). It grows with the groups in use, not with the open runs.
 func (c *CI) ReconcileConcurrency(ctx context.Context) {
 	log := ctxLog(ctx)
 	groups, err := c.cfg.Store.ListConcurrency(ctx)
 		// left to prune (each pass removes some, so this ends).
 		for more := true; more && err == nil && ctx.Err() == nil; {
 			_, more, err = c.updateGroupMore(ctx, g.Repo, g.Group, func(cur *domain.CIConcurrencyGroupState) error {
-				st = domain.CIConcurrencyGroupState{Marks: maps.Clone(cur.Marks)}
+				st = domain.CIConcurrencyGroupState{Record: cur.Record.Clone(), Marks: maps.Clone(cur.Marks)}
-				if cur.Record.Running != nil {
-					r := *cur.Record.Running
-					st.Record.Running = &r
-				}
-				if cur.Record.Pending != nil {
-					p := *cur.Record.Pending
-					st.Record.Pending = &p
-				}
 				return nil
 			})
 		}
 		}
 		if r := st.Record.Running; r != nil {
 			if r.Canceling {
-				c.cancelLater(st.Marks[r.Key()].Group, *r)
+				c.cancelLater(st.Marks[r.Key()].Group, *r, domain.CIConcurrencyReplaced)
 			}
 			if c.entryNeeds(ctx, g.Repo, g.Group, *r, domain.CIConcurrencyHolding) {
 				c.wake(g.Repo, g.Group, *r)
 			}
 		}
-		if p := st.Record.Pending; p != nil && c.entryNeeds(ctx, g.Repo, g.Group, *p, domain.CIConcurrencyPending) {
+		for _, p := range st.Record.Pending {
-			c.wake(g.Repo, g.Group, *p)
+			if c.entryNeeds(ctx, g.Repo, g.Group, p, domain.CIConcurrencyPending) {
+				c.wake(g.Repo, g.Group, p)
+			}
 		}
 		for _, m := range st.Marks {
-			if m.State == domain.CIConcurrencyReplaced {
+			if m.State.Owed() {
-				c.cancelLater(m.Group, m.Entry)
+				c.cancelLater(m.Group, m.Entry, m.State)
 			}
 		}
 	}
 // marks of every entry it touches are one batch (Store.UpdateConcurrency), so an
 // entry is marked pending or holding if and only if the record names it.
 //
-// An entry marked replaced never acquires, whatever the state of the group
+// An entry marked replaced or rejected never acquires, whatever the state of
-// (empty, or its waiting place free): it is owed a cancellation. Otherwise a
+// the group (empty, or its line free): it is owed a cancellation. Otherwise a
-// pending entry stays so; a newer one replaces the waiting entry, which is
+// pending entry stays so, at its place in the line; a newer one is queued
-// marked replaced and cancelled, and with cancel-in-progress the running entry
+// (domain.CIConcurrencyRecord.Enqueue): with queue: single it replaces every
-// is cancelled too (it stays named, with Canceling set, until it has ended).
+// waiting entry, each marked replaced and cancelled, and with
+// cancel-in-progress the running entry is cancelled too (it stays named, with
+// Canceling set, until it has ended); with queue: max it joins the end of the
+// line, or, when the line is full, is marked rejected and cancelled itself.
 // The call is repeatable: an entry that holds or waits is told so again.
 func (rs *runState) acquire(ctx context.Context, group string, e domain.CIConcurrencyEntry) (domain.CIConcurrencyState, error) {
+	type victim struct {
+		e   domain.CIConcurrencyEntry
+		why domain.CIConcurrencyState
+	}
 	var state domain.CIConcurrencyState
-	var victims []domain.CIConcurrencyEntry
+	var victims []victim
 	promoted, err := rs.c.updateGroup(ctx, rs.run.Repo, group, func(st *domain.CIConcurrencyGroupState) error {
 		victims = nil
 		rec := &st.Record
-		if m, ok := st.Marks[e.Key()]; ok && m.State == domain.CIConcurrencyReplaced {
+		if m, ok := st.Marks[e.Key()]; ok && m.State.Owed() {
 			state = domain.CIConcurrencyReplaced
-			victims = append(victims, e) // its cancellation may have been lost
+			victims = append(victims, victim{e, m.State}) // its cancellation may have been lost
 			return nil
 		}
 		switch {
 		case rec.Running == nil:
 			me := e
 			rec.Running = &me
-			if rec.Pending != nil && rec.Pending.Same(e) {
+			rec.RemovePending(e)
-				rec.Pending = nil
-			}
 			mark(st, e, group, domain.CIConcurrencyHolding)
 			state = domain.CIConcurrencyHolding
 		case rec.Running.Same(e):
 			state = domain.CIConcurrencyHolding
-		case rec.Pending != nil && rec.Pending.Same(e):
+		case rec.PendingIndex(e) >= 0:
 			state = domain.CIConcurrencyPending
 		default:
+			replaced, rejected := rec.Enqueue(e, rs.c.cfg.ConcurrencyQueueMax)
+			if rejected {
+				state = domain.CIConcurrencyReplaced
+				mark(st, e, group, domain.CIConcurrencyRejected)
+				victims = append(victims, victim{e, domain.CIConcurrencyRejected})
+				return nil
+			}
 			state = domain.CIConcurrencyPending
-			if rec.Pending != nil {
+			for _, old := range replaced {
-				old := *rec.Pending
 				mark(st, old, group, domain.CIConcurrencyReplaced)
-				victims = append(victims, old)
+				victims = append(victims, victim{old, domain.CIConcurrencyReplaced})
 			}
-			me := e
-			rec.Pending = &me
 			mark(st, e, group, domain.CIConcurrencyPending)
 			if e.Cancel && !rec.Running.Canceling {
 				run := *rec.Running
 				run.Canceling = true
 				rec.Running = &run
-				victims = append(victims, run)
+				victims = append(victims, victim{run, domain.CIConcurrencyReplaced})
 			}
 		}
 		return nil
 		rs.c.wake(rs.run.Repo, group, *promoted)
 	}
 	for _, v := range victims {
-		rs.c.cancelLater(group, v)
+		rs.c.cancelLater(group, v.e, v.why)
 	}
 	if state == domain.CIConcurrencyReplaced {
 		state = domain.CIConcurrencyPending // it shows as waiting until its cancellation lands
 }
 // releaseEntry takes an entry out of its group, because its run or job has
-// ended, and gives the turn to the waiting entry, which is planned again. The
+// ended, and gives the turn to the head of the line, which is planned again; a
-// mark of the entry goes in the same batch.
+// waiting entry that leaves keeps the order of the others. The mark of the
+// entry goes in the same batch.
 func (c *CI) releaseEntry(ctx context.Context, repo domain.RepoName, group string, e domain.CIConcurrencyEntry) error {
 	promoted, err := c.updateGroup(ctx, repo, group, func(st *domain.CIConcurrencyGroupState) error {
 		switch {
 		case st.Record.Running != nil && st.Record.Running.Same(e):
 			st.Record.Running = nil
 			promote(st, group)
-		case st.Record.Pending != nil && st.Record.Pending.Same(e):
+		default:
-			st.Record.Pending = nil
+			st.Record.RemovePending(e)
 		}
 		delete(st.Marks, e.Key())
 		return nil
 	return nil
 }
-// cancelEntry cancels a run or a job that a newer entry of the group
+// cancelEntry cancels a run or a job that a newer entry of the group replaced,
-// replaced: it ends cancelled with GitHub's annotation, and a job a runner has
+// or that found the line full (why is domain.CIConcurrencyRejected): it ends
-// gets the cancel request the owner's cancel sends.
+// cancelled with GitHub's annotation (ermite's for a full line), and a job a
-func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group string) error {
+// runner has gets the cancel request the owner's cancel sends.
+func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group string, why domain.CIConcurrencyState) error {
 	return c.withRun(ctx, e.RunID, e.Attempt, func(rs *runState) error {
 		if group == "" { // no name in the mark: use the entry's own group
 			group = rs.run.Concurrency.Group
 			}
 		}
 		msg := domain.ConcurrencyCancelMessage(group)
+		if why == domain.CIConcurrencyRejected {
+			msg = domain.ConcurrencyQueueFullMessage(group)
+		}
 		if e.JobID == "" {
 			if !rs.latest() || rs.run.Status == domain.CICompleted || rs.run.CancelRequested {
 				return nil
 	})
 }
-// cancelForGroup ends a job a newer entry of its group replaced, or asks its
+// cancelForGroup ends a job a newer entry of its group replaced (or that found
-// runner to stop. The annotation is GitHub's.
+// the line full), or asks its runner to stop, with the annotation msg.
 func (rs *runState) cancelForGroup(j *domain.CIJob, msg string) {
 	j.Concurrency.Preempted = true
 	rs.annotations = append(rs.annotations, ports.CIStepAnnotations{
 	if run.Concurrency.Group == "" {
 		return true, nil
 	}
-	st, err := rs.acquire(ctx, run.Concurrency.Group, domain.CIConcurrencyEntry{RunID: run.ID, Attempt: run.Attempt, Cancel: run.Concurrency.Cancel})
+	st, err := rs.acquire(ctx, run.Concurrency.Group, domain.CIConcurrencyEntry{RunID: run.ID, Attempt: run.Attempt, Cancel: run.Concurrency.Cancel, Max: run.Concurrency.Max})
 	if err != nil {
 		return false, err
 	}

M internal/usecase/ci_concurrency_test.go +15 -2


@@ -51,8 +51,13 @@ func TestEvalConcurrency(t *testing.T) {
51	51		{"cancel expression", "{group: g, cancel-in-progress: \"${{ github.ref == 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g", Cancel: true}, ""},
52	52		{"cancel expression false", "{group: g, cancel-in-progress: \"${{ github.ref != 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g"}, ""},
53	53		{"queue single", "{group: g, queue: single}", domain.CIConcurrency{Group: "g"}, ""},
54		-	{"queue max", "{group: g, queue: max}", domain.CIConcurrency{}, "not supported yet: queue: max"},
	54	+	{"queue max", "{group: g, queue: max}", domain.CIConcurrency{Group: "g", Max: true}, ""},
	55	+	{"queue max with cancel false", "{group: g, queue: max, cancel-in-progress: false}", domain.CIConcurrency{Group: "g", Max: true}, ""},
	56	+	{"queue max from an expression", "{group: g, queue: \"${{ 'max' }}\"}", domain.CIConcurrency{Group: "g", Max: true}, ""},
55	57		{"queue max with cancel true", "{group: g, queue: max, cancel-in-progress: true}", domain.CIConcurrency{}, "'queue: max' cannot be combined with 'cancel-in-progress: true'."},
	58	+	{"queue max with cancel expression true", "{group: g, queue: max, cancel-in-progress: \"${{ github.ref == 'refs/heads/main' }}\"}", domain.CIConcurrency{}, "'queue: max' cannot be combined with 'cancel-in-progress: true'."},
	59	+	{"queue max with cancel expression false", "{group: g, queue: max, cancel-in-progress: \"${{ github.ref != 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g", Max: true}, ""},
	60	+	{"queue expression of another value", "{group: g, queue: \"${{ github.workflow }}\"}", domain.CIConcurrency{}, "Unexpected value 'ci'"},
56	61		{"empty group", `${{ vars.NOPE }}`, domain.CIConcurrency{}, "concurrency group name cannot be empty"},
57	62		}
58	63		for _, tt := range tests {

@@ -61,7 +66,7 @@ func TestEvalConcurrency(t *testing.T) {
61	66		got, err := evalGroup(t, tt.yaml)
62	67		if tt.err != "" {
63	68		require.Error(t, err)
64		-	assert.Contains(t, err.Error(), tt.err)
	69	+	assert.Equal(t, tt.err, err.Error())
65	70		return
66	71		}
67	72		require.NoError(t, err)

@@ -109,3 +114,11 @@ func TestConcurrencyIdentity(t *testing.T) {
109	114		assert.Equal(t, domain.CICompleted, pending.Status(), "an ended job is not pending any more")
110	115		assert.Equal(t, domain.CIQueued, domain.CIJob{Phase: domain.CIPhaseWaiting}.Status())
111	116		}
	117	+
	118	+	// A literal queue other than single or max is refused by the schema when the
	119	+	// workflow is parsed ("concurrency-queue": allowed-values single, max).
	120	+	func TestConcurrencyQueueLiteralIsChecked(t *testing.T) {
	121	+	t.Parallel()
	122	+	_, errs := workflow.Parse("w.yml", []byte(fmt.Sprintf(concurrencyYAML, "{group: g, queue: often}")))
	123	+	require.NotEmpty(t, errs)
	124	+	}

 		{"cancel expression", "{group: g, cancel-in-progress: \"${{ github.ref == 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g", Cancel: true}, ""},
 		{"cancel expression false", "{group: g, cancel-in-progress: \"${{ github.ref != 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g"}, ""},
 		{"queue single", "{group: g, queue: single}", domain.CIConcurrency{Group: "g"}, ""},
-		{"queue max", "{group: g, queue: max}", domain.CIConcurrency{}, "not supported yet: queue: max"},
+		{"queue max", "{group: g, queue: max}", domain.CIConcurrency{Group: "g", Max: true}, ""},
+		{"queue max with cancel false", "{group: g, queue: max, cancel-in-progress: false}", domain.CIConcurrency{Group: "g", Max: true}, ""},
+		{"queue max from an expression", "{group: g, queue: \"${{ 'max' }}\"}", domain.CIConcurrency{Group: "g", Max: true}, ""},
 		{"queue max with cancel true", "{group: g, queue: max, cancel-in-progress: true}", domain.CIConcurrency{}, "'queue: max' cannot be combined with 'cancel-in-progress: true'."},
+		{"queue max with cancel expression true", "{group: g, queue: max, cancel-in-progress: \"${{ github.ref == 'refs/heads/main' }}\"}", domain.CIConcurrency{}, "'queue: max' cannot be combined with 'cancel-in-progress: true'."},
+		{"queue max with cancel expression false", "{group: g, queue: max, cancel-in-progress: \"${{ github.ref != 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g", Max: true}, ""},
+		{"queue expression of another value", "{group: g, queue: \"${{ github.workflow }}\"}", domain.CIConcurrency{}, "Unexpected value 'ci'"},
 		{"empty group", `${{ vars.NOPE }}`, domain.CIConcurrency{}, "concurrency group name cannot be empty"},
 	}
 	for _, tt := range tests {
 			got, err := evalGroup(t, tt.yaml)
 			if tt.err != "" {
 				require.Error(t, err)
-				assert.Contains(t, err.Error(), tt.err)
+				assert.Equal(t, tt.err, err.Error())
 				return
 			}
 			require.NoError(t, err)
 	assert.Equal(t, domain.CICompleted, pending.Status(), "an ended job is not pending any more")
 	assert.Equal(t, domain.CIQueued, domain.CIJob{Phase: domain.CIPhaseWaiting}.Status())
 }
+// A literal queue other than single or max is refused by the schema when the
+// workflow is parsed ("concurrency-queue": allowed-values single, max).
+func TestConcurrencyQueueLiteralIsChecked(t *testing.T) {
+	t.Parallel()
+	_, errs := workflow.Parse("w.yml", []byte(fmt.Sprintf(concurrencyYAML, "{group: g, queue: often}")))
+	require.NotEmpty(t, errs)
+}

M internal/usecase/ci_dispatch.go +1 -2


@@ -10,7 +10,6 @@ import (
10	10		"path"
11	11		"strings"
12	12
13		-	"bnema.dev/ermite-actions/workflow"
14	13		"bnema.dev/ermite/internal/domain"
15	14		"bnema.dev/ermite/internal/ports"
16	15		)

@@ -57,7 +56,7 @@ func (c *CI) DispatchWorkflow(ctx context.Context, p domain.Principal, name doma
57	56		if err != nil {
58	57		return domain.CIRun{}, err
59	58		}
60		-	wf, perrs := workflow.Parse(wfPath, content)
	59	+	wf, perrs := parseWorkflow(wfPath, content)
61	60		if len(perrs) > 0 {
62	61		lines := make([]string, 0, len(perrs))
63	62		for _, e := range perrs {

 	"path"
 	"strings"
-	"bnema.dev/ermite-actions/workflow"
 	"bnema.dev/ermite/internal/domain"
 	"bnema.dev/ermite/internal/ports"
 )
 	if err != nil {
 		return domain.CIRun{}, err
 	}
-	wf, perrs := workflow.Parse(wfPath, content)
+	wf, perrs := parseWorkflow(wfPath, content)
 	if len(perrs) > 0 {
 		lines := make([]string, 0, len(perrs))
 		for _, e := range perrs {

M internal/usecase/ci_events.go +10 -5


@@ -14,7 +14,8 @@ import (
14	14		"bnema.dev/ermite/internal/domain"
15	15		)
16	16
17		-	// The events workflow_dispatch, schedule and workflow_run (CI-3, D5). They share
	17	+	// The events workflow_dispatch, schedule and workflow_run (CI-3, D5), and create
	18	+	// and delete (ci_refevents.go). They share
18	19		// the run machinery of the push (startRun); what differs is the github context
19	20		// and, for workflow_dispatch, the typed inputs.
20	21

@@ -33,8 +34,8 @@ const (
33	34		maxDispatchPayload = 65535
34	35		)
35	36
36		-	// customEvent is what a run of workflow_dispatch, schedule or workflow_run is
37		-	// for, beside the commit (pushFacts.custom).
	37	+	// customEvent is what a run of workflow_dispatch, schedule, workflow_run,
	38	+	// create or delete is for, beside the commit (pushFacts.custom).
38	39		type customEvent struct {
39	40		name string
40	41		title string

@@ -45,6 +46,8 @@ type customEvent struct {
45	46		cron string
46	47		// triggered is the run a workflow_run is for.
47	48		triggered *workflowRunFacts
	49	+	// ref is the branch or tag created or deleted (create, delete).
	50	+	ref string
48	51		}
49	52
50	53		// workflowRunFacts is the triggering run of a workflow_run.

@@ -200,8 +203,8 @@ func eventInputs(wf workflow.Workflow, githubJSON string) expr.Dictionary {
200	203		return typedInputs(ev.Inputs, values)
201	204		}
202	205
203		-	// customGithubJSON is the github context of a workflow_dispatch, schedule or
204		-	// workflow_run run.
	206	+	// customGithubJSON is the github context of a workflow_dispatch, schedule,
	207	+	// workflow_run, create or delete run.
205	208		func (c *CI) customGithubJSON(repo domain.Repo, run domain.CIRun, f pushFacts) string {
206	209		u := c.urls(repo)
207	210		id := c.identity(run.Actor)

@@ -226,6 +229,8 @@ func (c *CI) customGithubJSON(repo domain.Repo, run domain.CIRun, f pushFacts) s
226	229		event["action"] = t.action
227	230		event["workflow_run"] = c.workflowRunObj(repo, u, t)
228	231		event["workflow"] = c.workflowObj(u, t.run.WorkflowPath, t.run.WorkflowName, t.run.CreatedAt)
	232	+	case eventCreate, eventDelete:
	233	+	refEventPayload(cu.name, cu.ref, f.defaultBranch, event)
229	234		}
230	235		refName, refType := strings.TrimPrefix(run.Ref, "refs/heads/"), "branch"
231	236		if tag, ok := strings.CutPrefix(run.Ref, "refs/tags/"); ok {

 	"bnema.dev/ermite/internal/domain"
 )
-// The events workflow_dispatch, schedule and workflow_run (CI-3, D5). They share
+// The events workflow_dispatch, schedule and workflow_run (CI-3, D5), and create
+// and delete (ci_refevents.go). They share
 // the run machinery of the push (startRun); what differs is the github context
 // and, for workflow_dispatch, the typed inputs.
 	maxDispatchPayload = 65535
 )
-// customEvent is what a run of workflow_dispatch, schedule or workflow_run is
+// customEvent is what a run of workflow_dispatch, schedule, workflow_run,
-// for, beside the commit (pushFacts.custom).
+// create or delete is for, beside the commit (pushFacts.custom).
 type customEvent struct {
 	name  string
 	title string
 	cron string
 	// triggered is the run a workflow_run is for.
 	triggered *workflowRunFacts
+	// ref is the branch or tag created or deleted (create, delete).
+	ref string
 }
 // workflowRunFacts is the triggering run of a workflow_run.
 	return typedInputs(ev.Inputs, values)
 }
-// customGithubJSON is the github context of a workflow_dispatch, schedule or
+// customGithubJSON is the github context of a workflow_dispatch, schedule,
-// workflow_run run.
+// workflow_run, create or delete run.
 func (c *CI) customGithubJSON(repo domain.Repo, run domain.CIRun, f pushFacts) string {
 	u := c.urls(repo)
 	id := c.identity(run.Actor)
 		event["action"] = t.action
 		event["workflow_run"] = c.workflowRunObj(repo, u, t)
 		event["workflow"] = c.workflowObj(u, t.run.WorkflowPath, t.run.WorkflowName, t.run.CreatedAt)
+	case eventCreate, eventDelete:
+		refEventPayload(cu.name, cu.ref, f.defaultBranch, event)
 	}
 	refName, refType := strings.TrimPrefix(run.Ref, "refs/heads/"), "branch"
 	if tag, ok := strings.CutPrefix(run.Ref, "refs/tags/"); ok {

M internal/usecase/ci_plan.go +1 -1


@@ -372,7 +372,7 @@ func (rs *runState) promote(ctx context.Context) error {
372	372		continue
373	373		}
374	374		if g := j.Concurrency; g.Group != "" && !g.FromRun {
375		-	st, err := rs.acquire(ctx, g.Group, domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID, Cancel: g.Cancel})
	375	+	st, err := rs.acquire(ctx, g.Group, domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID, Cancel: g.Cancel, Max: g.Max})
376	376		if err != nil {
377	377		return err
378	378		}


@@ -372,7 +372,7 @@ func (rs *runState) promote(ctx context.Context) error {
372		continue	372		continue
373		}	373		}
374		if g := j.Concurrency; g.Group != "" && !g.FromRun {	374		if g := j.Concurrency; g.Group != "" && !g.FromRun {
375	-	st, err := rs.acquire(ctx, g.Group, domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID, Cancel: g.Cancel})	375	+	st, err := rs.acquire(ctx, g.Group, domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID, Cancel: g.Cancel, Max: g.Max})
376		if err != nil {	376		if err != nil {
377		return err	377		return err
378		}	378		}

M internal/usecase/ci_pull.go +1 -1


@@ -993,7 +993,7 @@ func (c *CI) startPullRuns(ctx context.Context, e pullEvent, mr domain.MergeRequ
993	993		if file.problem != "" {
994	994		continue // reported by the push of the branch
995	995		}
996		-	wf, perrs := workflow.Parse(file.path, file.content)
	996	+	wf, perrs := parseWorkflow(file.path, file.content)
997	997		if len(perrs) > 0 \|\| !matchPullRequest(wf, f.eventName(), e.action, mr.Target, f.files, f.filesTruncated) {
998	998		continue
999	999		}


@@ -993,7 +993,7 @@ func (c *CI) startPullRuns(ctx context.Context, e pullEvent, mr domain.MergeRequ
993		if file.problem != "" {	993		if file.problem != "" {
994		continue // reported by the push of the branch	994		continue // reported by the push of the branch
995		}	995		}
996	-	wf, perrs := workflow.Parse(file.path, file.content)	996	+	wf, perrs := parseWorkflow(file.path, file.content)
997		if len(perrs) > 0 \|\| !matchPullRequest(wf, f.eventName(), e.action, mr.Target, f.files, f.filesTruncated) {	997		if len(perrs) > 0 \|\| !matchPullRequest(wf, f.eventName(), e.action, mr.Target, f.files, f.filesTruncated) {
998		continue	998		continue
999		}	999		}

A internal/usecase/ci_refevents.go +131 -0

@@ -0,0 +1,131 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package usecase
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"fmt"
	9	+	"strings"
	10	+
	11	+	"bnema.dev/ermite-actions/workflow"
	12	+	"bnema.dev/ermite/internal/domain"
	13	+	)
	14	+
	15	+	// The events create and delete (CI-4, D6): a branch or a tag is created or
	16	+	// deleted. As on GitHub ("Events that trigger workflows"), a create run is on
	17	+	// the created ref, its last commit, and runs the workflows of that commit; a
	18	+	// delete run is on the default branch, its last commit, and runs the workflows
	19	+	// of that commit: the deleted ref is gone. Neither has filters, and GitHub's
	20	+	// skip directives do not apply to them. A push that creates or deletes more
	21	+	// than three tags creates no event for them.
	22	+	//
	23	+	// Trust: a create run is trusted when the pusher is and the record of the
	24	+	// created ref (ci_tips.go) names the commit the run executes; a delete run when
	25	+	// the deleter is and a trusted principal set the default branch to the commit
	26	+	// it executes. The deleted ref's record went with the ref and is never looked
	27	+	// at. Job tokens start neither (OnPush).
	28	+
	29	+	const (
	30	+	eventCreate = "create"
	31	+	eventDelete = "delete"
	32	+	)
	33	+
	34	+	// startCreateRuns starts the create workflows of a ref a push created, from
	35	+	// the files of the commit it points to (files; an invalid file is reported by
	36	+	// the push run).
	37	+	func (c *CI) startCreateRuns(ctx context.Context, repo domain.Repo, t ciTrigger, f pushFacts, files []workflowFile) error {
	38	+	newID, err := domain.ParseObjectID(f.update.New)
	39	+	if err != nil {
	40	+	return err
	41	+	}
	42	+	ct := t
	43	+	ct.trusted = t.trusted && c.refTrusted(ctx, repo.Name, pinnedRef{ref: f.update.Ref, commit: f.commit, object: newID})
	44	+	if t.trusted && !ct.trusted {
	45	+	ctxLog(ctx).Info().Str("repo", repo.Name.String()).Str("ref", f.update.Ref).
	46	+	Msg("ci create runs are not trusted: the created ref is not at a commit a trusted principal pushed")
	47	+	}
	48	+	cf := f
	49	+	cf.payload = nil
	50	+	cf.custom = &customEvent{name: eventCreate, ref: f.update.Ref}
	51	+	return c.startRefEventRuns(ctx, repo, ct, cf, files)
	52	+	}
	53	+
	54	+	// handleDelete starts the delete workflows of the default branch for a deleted
	55	+	// branch or tag; deletedTags is the number of tags the push deletes.
	56	+	func (c *CI) handleDelete(ctx context.Context, t ciTrigger, u domain.RefUpdate, deletedTags int) error {
	57	+	if strings.HasPrefix(u.Ref, "refs/tags/") && deletedTags > workflow.MaxTagsPerPush {
	58	+	return nil
	59	+	}
	60	+	repo, err := c.cfg.Meta.Get(ctx, t.repo)
	61	+	if err != nil {
	62	+	return nil // deleted meanwhile
	63	+	}
	64	+	tip := c.defaultTipOf(ctx, t.repo)
	65	+	if tip.commit.IsZero() {
	66	+	return nil // no default branch to run from
	67	+	}
	68	+	head, _, err := c.cfg.Git.ListRange(ctx, t.repo, domain.ObjectID{}, tip.commit, 1)
	69	+	if err != nil {
	70	+	return fmt.Errorf("read the head commit of %s: %w", tip.branch, err)
	71	+	}
	72	+	if len(head) == 0 {
	73	+	return fmt.Errorf("read the head commit of %s: %w", tip.branch, domain.ErrNotFound)
	74	+	}
	75	+	files, err := c.workflowFiles(ctx, t.repo, tip.commit)
	76	+	if err != nil \|\| len(files) == 0 {
	77	+	return err
	78	+	}
	79	+	dt := t
	80	+	dt.trusted = t.trusted && tip.trusted
	81	+	if t.trusted && !tip.trusted {
	82	+	ctxLog(ctx).Info().Str("repo", t.repo.String()).Str("ref", u.Ref).Str("tip", tip.commit.String()).
	83	+	Msg("ci delete runs are not trusted: the default branch tip was not pushed by a trusted principal")
	84	+	}
	85	+	f := pushFacts{
	86	+	update: domain.RefUpdate{Ref: "refs/heads/" + tip.branch, Old: domain.ZeroOID, New: tip.commit.String()},
	87	+	commit: tip.commit, head: head[0], defaultBranch: tip.branch,
	88	+	custom: &customEvent{name: eventDelete, ref: u.Ref},
	89	+	}
	90	+	return c.startRefEventRuns(ctx, repo, dt, f, files)
	91	+	}
	92	+
	93	+	// startRefEventRuns starts the workflows of files that are on the event of
	94	+	// f.custom (create or delete).
	95	+	func (c *CI) startRefEventRuns(ctx context.Context, repo domain.Repo, t ciTrigger, f pushFacts, files []workflowFile) error {
	96	+	var errs []error
	97	+	for _, file := range files {
	98	+	if file.problem != "" {
	99	+	continue
	100	+	}
	101	+	wf, perrs := parseWorkflow(file.path, file.content)
	102	+	if len(perrs) > 0 \|\| wf.On == nil \|\| !wf.On.Has(f.custom.name) {
	103	+	continue
	104	+	}
	105	+	if problem := c.matrixProblem(ctx, repo, t, f, file.path, wf); problem.text != "" {
	106	+	name := file.path
	107	+	if problem.named {
	108	+	name = workflowName(wf, file.path)
	109	+	}
	110	+	errs = append(errs, c.startInvalid(ctx, repo, t, f, file.path, name, problem.text))
	111	+	continue
	112	+	}
	113	+	errs = append(errs, c.startRun(ctx, repo, t, f, file.path, string(file.content), wf))
	114	+	}
	115	+	return errors.Join(errs...)
	116	+	}
	117	+
	118	+	// refEventPayload is the payload of a create or a delete event: the ref by its
	119	+	// short name and its type; create adds the default branch and the description
	120	+	// of the repository (ermite has none: null). pusher_type is "user": a job
	121	+	// token, GitHub's "deploy key" case aside, starts no run.
	122	+	func refEventPayload(name, ref, defaultBranch string, event map[string]any) {
	123	+	refName, refType := strings.TrimPrefix(ref, "refs/heads/"), "branch"
	124	+	if tag, ok := strings.CutPrefix(ref, "refs/tags/"); ok {
	125	+	refName, refType = tag, "tag"
	126	+	}
	127	+	event["ref"], event["ref_type"], event["pusher_type"] = refName, refType, "user"
	128	+	if name == eventCreate {
	129	+	event["master_branch"], event["description"] = defaultBranch, nil
	130	+	}
	131	+	}

@@ -0,0 +1,131 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package usecase
	4	+
	5	+	import (
	6	+	"context"
	7	+	"errors"
	8	+	"fmt"
	9	+	"strings"
	10	+
	11	+	"bnema.dev/ermite-actions/workflow"
	12	+	"bnema.dev/ermite/internal/domain"
	13	+	)
	14	+
	15	+	// The events create and delete (CI-4, D6): a branch or a tag is created or
	16	+	// deleted. As on GitHub ("Events that trigger workflows"), a create run is on
	17	+	// the created ref, its last commit, and runs the workflows of that commit; a
	18	+	// delete run is on the default branch, its last commit, and runs the workflows
	19	+	// of that commit: the deleted ref is gone. Neither has filters, and GitHub's
	20	+	// skip directives do not apply to them. A push that creates or deletes more
	21	+	// than three tags creates no event for them.
	22	+	//
	23	+	// Trust: a create run is trusted when the pusher is and the record of the
	24	+	// created ref (ci_tips.go) names the commit the run executes; a delete run when
	25	+	// the deleter is and a trusted principal set the default branch to the commit
	26	+	// it executes. The deleted ref's record went with the ref and is never looked
	27	+	// at. Job tokens start neither (OnPush).
	28	+
	29	+	const (
	30	+	eventCreate = "create"
	31	+	eventDelete = "delete"
	32	+	)
	33	+
	34	+	// startCreateRuns starts the create workflows of a ref a push created, from
	35	+	// the files of the commit it points to (files; an invalid file is reported by
	36	+	// the push run).
	37	+	func (c *CI) startCreateRuns(ctx context.Context, repo domain.Repo, t ciTrigger, f pushFacts, files []workflowFile) error {
	38	+	newID, err := domain.ParseObjectID(f.update.New)
	39	+	if err != nil {
	40	+	return err
	41	+	}
	42	+	ct := t
	43	+	ct.trusted = t.trusted && c.refTrusted(ctx, repo.Name, pinnedRef{ref: f.update.Ref, commit: f.commit, object: newID})
	44	+	if t.trusted && !ct.trusted {
	45	+	ctxLog(ctx).Info().Str("repo", repo.Name.String()).Str("ref", f.update.Ref).
	46	+	Msg("ci create runs are not trusted: the created ref is not at a commit a trusted principal pushed")
	47	+	}
	48	+	cf := f
	49	+	cf.payload = nil
	50	+	cf.custom = &customEvent{name: eventCreate, ref: f.update.Ref}
	51	+	return c.startRefEventRuns(ctx, repo, ct, cf, files)
	52	+	}
	53	+
	54	+	// handleDelete starts the delete workflows of the default branch for a deleted
	55	+	// branch or tag; deletedTags is the number of tags the push deletes.
	56	+	func (c *CI) handleDelete(ctx context.Context, t ciTrigger, u domain.RefUpdate, deletedTags int) error {
	57	+	if strings.HasPrefix(u.Ref, "refs/tags/") && deletedTags > workflow.MaxTagsPerPush {
	58	+	return nil
	59	+	}
	60	+	repo, err := c.cfg.Meta.Get(ctx, t.repo)
	61	+	if err != nil {
	62	+	return nil // deleted meanwhile
	63	+	}
	64	+	tip := c.defaultTipOf(ctx, t.repo)
	65	+	if tip.commit.IsZero() {
	66	+	return nil // no default branch to run from
	67	+	}
	68	+	head, _, err := c.cfg.Git.ListRange(ctx, t.repo, domain.ObjectID{}, tip.commit, 1)
	69	+	if err != nil {
	70	+	return fmt.Errorf("read the head commit of %s: %w", tip.branch, err)
	71	+	}
	72	+	if len(head) == 0 {
	73	+	return fmt.Errorf("read the head commit of %s: %w", tip.branch, domain.ErrNotFound)
	74	+	}
	75	+	files, err := c.workflowFiles(ctx, t.repo, tip.commit)
	76	+	if err != nil \|\| len(files) == 0 {
	77	+	return err
	78	+	}
	79	+	dt := t
	80	+	dt.trusted = t.trusted && tip.trusted
	81	+	if t.trusted && !tip.trusted {
	82	+	ctxLog(ctx).Info().Str("repo", t.repo.String()).Str("ref", u.Ref).Str("tip", tip.commit.String()).
	83	+	Msg("ci delete runs are not trusted: the default branch tip was not pushed by a trusted principal")
	84	+	}
	85	+	f := pushFacts{
	86	+	update: domain.RefUpdate{Ref: "refs/heads/" + tip.branch, Old: domain.ZeroOID, New: tip.commit.String()},
	87	+	commit: tip.commit, head: head[0], defaultBranch: tip.branch,
	88	+	custom: &customEvent{name: eventDelete, ref: u.Ref},
	89	+	}
	90	+	return c.startRefEventRuns(ctx, repo, dt, f, files)
	91	+	}
	92	+
	93	+	// startRefEventRuns starts the workflows of files that are on the event of
	94	+	// f.custom (create or delete).
	95	+	func (c *CI) startRefEventRuns(ctx context.Context, repo domain.Repo, t ciTrigger, f pushFacts, files []workflowFile) error {
	96	+	var errs []error
	97	+	for _, file := range files {
	98	+	if file.problem != "" {
	99	+	continue
	100	+	}
	101	+	wf, perrs := parseWorkflow(file.path, file.content)
	102	+	if len(perrs) > 0 \|\| wf.On == nil \|\| !wf.On.Has(f.custom.name) {
	103	+	continue
	104	+	}
	105	+	if problem := c.matrixProblem(ctx, repo, t, f, file.path, wf); problem.text != "" {
	106	+	name := file.path
	107	+	if problem.named {
	108	+	name = workflowName(wf, file.path)
	109	+	}
	110	+	errs = append(errs, c.startInvalid(ctx, repo, t, f, file.path, name, problem.text))
	111	+	continue
	112	+	}
	113	+	errs = append(errs, c.startRun(ctx, repo, t, f, file.path, string(file.content), wf))
	114	+	}
	115	+	return errors.Join(errs...)
	116	+	}
	117	+
	118	+	// refEventPayload is the payload of a create or a delete event: the ref by its
	119	+	// short name and its type; create adds the default branch and the description
	120	+	// of the repository (ermite has none: null). pusher_type is "user": a job
	121	+	// token, GitHub's "deploy key" case aside, starts no run.
	122	+	func refEventPayload(name, ref, defaultBranch string, event map[string]any) {
	123	+	refName, refType := strings.TrimPrefix(ref, "refs/heads/"), "branch"
	124	+	if tag, ok := strings.CutPrefix(ref, "refs/tags/"); ok {
	125	+	refName, refType = tag, "tag"
	126	+	}
	127	+	event["ref"], event["ref_type"], event["pusher_type"] = refName, refType, "user"
	128	+	if name == eventCreate {
	129	+	event["master_branch"], event["description"] = defaultBranch, nil
	130	+	}
	131	+	}

M internal/usecase/ci_reusable.go +3 -3


@@ -545,7 +545,7 @@ func (rs runState) planCall(ctx context.Context, wf workflow.Workflow, wj *wor
545	545		return err
546	546		}
547	547		prefix := callerPrefix(wj)
548		-	cw, errs := workflow.Parse(file.path, file.yaml)
	548	+	cw, errs := parseWorkflow(file.path, file.yaml)
549	549		if len(errs) > 0 {
550	550		return callFailure{prefix + " " + errs[0].Error()}
551	551		}

@@ -999,7 +999,7 @@ func (rs *runState) holdCalls(ctx context.Context) (bool, error) {
999	999		entry := domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID}
1000	1000		if g := j.Concurrency; g.Group != "" && !g.FromRun && g.State != domain.CIConcurrencyHolding {
1001	1001		e := entry
1002		-	e.Cancel = g.Cancel
	1002	+	e.Cancel, e.Max = g.Cancel, g.Max
1003	1003		st, err := rs.acquire(ctx, g.Group, e)
1004	1004		if err != nil {
1005	1005		return changed, err

@@ -1014,7 +1014,7 @@ func (rs *runState) holdCalls(ctx context.Context) (bool, error) {
1014	1014		}
1015	1015		if g := j.Call.Concurrency; g.Group != "" && g.State != domain.CIConcurrencyHolding {
1016	1016		e := entry
1017		-	e.Cancel = g.Cancel
	1017	+	e.Cancel, e.Max = g.Cancel, g.Max
1018	1018		st, err := rs.acquire(ctx, g.Group, e)
1019	1019		if err != nil {
1020	1020		return changed, err


@@ -545,7 +545,7 @@ func (rs runState) planCall(ctx context.Context, wf workflow.Workflow, wj *wor
545		return err	545		return err
546		}	546		}
547		prefix := callerPrefix(wj)	547		prefix := callerPrefix(wj)
548	-	cw, errs := workflow.Parse(file.path, file.yaml)	548	+	cw, errs := parseWorkflow(file.path, file.yaml)
549		if len(errs) > 0 {	549		if len(errs) > 0 {
550		return callFailure{prefix + " " + errs[0].Error()}	550		return callFailure{prefix + " " + errs[0].Error()}
551		}	551		}

@@ -999,7 +999,7 @@ func (rs *runState) holdCalls(ctx context.Context) (bool, error) {
999		entry := domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID}	999		entry := domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID}
1000		if g := j.Concurrency; g.Group != "" && !g.FromRun && g.State != domain.CIConcurrencyHolding {	1000		if g := j.Concurrency; g.Group != "" && !g.FromRun && g.State != domain.CIConcurrencyHolding {
1001		e := entry	1001		e := entry
1002	-	e.Cancel = g.Cancel	1002	+	e.Cancel, e.Max = g.Cancel, g.Max
1003		st, err := rs.acquire(ctx, g.Group, e)	1003		st, err := rs.acquire(ctx, g.Group, e)
1004		if err != nil {	1004		if err != nil {
1005		return changed, err	1005		return changed, err

@@ -1014,7 +1014,7 @@ func (rs *runState) holdCalls(ctx context.Context) (bool, error) {
1014		}	1014		}
1015		if g := j.Call.Concurrency; g.Group != "" && g.State != domain.CIConcurrencyHolding {	1015		if g := j.Call.Concurrency; g.Group != "" && g.State != domain.CIConcurrencyHolding {
1016		e := entry	1016		e := entry
1017	-	e.Cancel = g.Cancel	1017	+	e.Cancel, e.Max = g.Cancel, g.Max
1018		st, err := rs.acquire(ctx, g.Group, e)	1018		st, err := rs.acquire(ctx, g.Group, e)
1019		if err != nil {	1019		if err != nil {
1020		return changed, err	1020		return changed, err

M internal/usecase/ci_schedule.go +10 -7


@@ -14,7 +14,6 @@ import (
14	14
15	15		"github.com/bnema/zerowrap"
16	16
17		-	"bnema.dev/ermite-actions/workflow"
18	17		"bnema.dev/ermite/internal/domain"
19	18		)
20	19

@@ -52,8 +51,8 @@ const (
52	51
53	52		// scheduledCron is one cron entry of a workflow.
54	53		type scheduledCron struct {
55		-	spec string
56		-	cron domain.Cron
	54	+	spec string
	55	+	sched domain.CronSchedule
57	56		}
58	57
59	58		// scheduledWorkflow is a workflow with at least one valid cron entry.

@@ -338,7 +337,7 @@ func (c CI) loadSchedules(ctx context.Context, repo domain.RepoName) (schedule
338	337		if f.problem != "" {
339	338		continue
340	339		}
341		-	wf, perrs := workflow.Parse(f.path, f.content)
	340	+	wf, perrs := parseWorkflow(f.path, f.content)
342	341		if len(perrs) > 0 \|\| wf.On == nil {
343	342		continue
344	343		}

@@ -352,7 +351,11 @@ func (c CI) loadSchedules(ctx context.Context, repo domain.RepoName) (schedule
352	351		if err != nil {
353	352		continue
354	353		}
355		-	sw.crons = append(sw.crons, scheduledCron{spec: sc.Cron, cron: cron})
	354	+	loc, err := domain.ParseCronTimezone(sc.Timezone)
	355	+	if err != nil {
	356	+	continue // the push of the file reported it (parseWorkflow)
	357	+	}
	358	+	sw.crons = append(sw.crons, scheduledCron{spec: sc.Cron, sched: domain.CronSchedule{Cron: cron, Loc: loc}})
356	359		}
357	360		if len(sw.crons) > 0 {
358	361		e.files = append(e.files, sw)

@@ -366,7 +369,7 @@ func (c CI) loadSchedules(ctx context.Context, repo domain.RepoName) (schedule
366	369		func dueFor(e *scheduleEntry, minute time.Time) (idx []int, specs []string) {
367	370		for i, f := range e.files {
368	371		for _, sc := range f.crons {
369		-	if sc.cron.Matches(minute) {
	372	+	if sc.sched.Due(minute) {
370	373		idx, specs = append(idx, i), append(specs, sc.spec)
371	374		break
372	375		}

@@ -419,7 +422,7 @@ func (c *CI) scheduleRepo(ctx context.Context, repo domain.RepoName, minute time
419	422
420	423		// fireSchedule starts the run of one scheduled workflow.
421	424		func (c CI) fireSchedule(ctx context.Context, repo domain.Repo, def string, e scheduleEntry, head domain.Commit, sw scheduledWorkflow, spec string) error {
422		-	wf, perrs := workflow.Parse(sw.path, sw.yaml)
	425	+	wf, perrs := parseWorkflow(sw.path, sw.yaml)
423	426		if len(perrs) > 0 {
424	427		return nil
425	428		}

 	"github.com/bnema/zerowrap"
-	"bnema.dev/ermite-actions/workflow"
 	"bnema.dev/ermite/internal/domain"
 )
 // scheduledCron is one cron entry of a workflow.
 type scheduledCron struct {
-	spec string
+	spec  string
-	cron domain.Cron
+	sched domain.CronSchedule
 }
 // scheduledWorkflow is a workflow with at least one valid cron entry.
 		if f.problem != "" {
 			continue
 		}
-		wf, perrs := workflow.Parse(f.path, f.content)
+		wf, perrs := parseWorkflow(f.path, f.content)
 		if len(perrs) > 0 || wf.On == nil {
 			continue
 		}
 			if err != nil {
 				continue
 			}
-			sw.crons = append(sw.crons, scheduledCron{spec: sc.Cron, cron: cron})
+			loc, err := domain.ParseCronTimezone(sc.Timezone)
+			if err != nil {
+				continue // the push of the file reported it (parseWorkflow)
+			}
+			sw.crons = append(sw.crons, scheduledCron{spec: sc.Cron, sched: domain.CronSchedule{Cron: cron, Loc: loc}})
 		}
 		if len(sw.crons) > 0 {
 			e.files = append(e.files, sw)
 func dueFor(e *scheduleEntry, minute time.Time) (idx []int, specs []string) {
 	for i, f := range e.files {
 		for _, sc := range f.crons {
-			if sc.cron.Matches(minute) {
+			if sc.sched.Due(minute) {
 				idx, specs = append(idx, i), append(specs, sc.spec)
 				break
 			}
 // fireSchedule starts the run of one scheduled workflow.
 func (c *CI) fireSchedule(ctx context.Context, repo domain.Repo, def string, e *scheduleEntry, head domain.Commit, sw scheduledWorkflow, spec string) error {
-	wf, perrs := workflow.Parse(sw.path, sw.yaml)
+	wf, perrs := parseWorkflow(sw.path, sw.yaml)
 	if len(perrs) > 0 {
 		return nil
 	}

M internal/usecase/ci_trigger.go +40 -15


@@ -91,7 +91,7 @@ func (c *CI) OnMerge(ctx context.Context, repo domain.RepoName, updates []domain
91	91		func (c *CI) enqueue(ctx context.Context, repo domain.RepoName, updates []domain.RefUpdate, actor string, pusher domain.CIPusher, trusted bool) {
92	92		var kept []domain.RefUpdate
93	93		for _, u := range updates {
94		-	if !u.IsDelete() && (strings.HasPrefix(u.Ref, "refs/heads/") \|\| strings.HasPrefix(u.Ref, "refs/tags/")) {
	94	+	if strings.HasPrefix(u.Ref, "refs/heads/") \|\| strings.HasPrefix(u.Ref, "refs/tags/") {
95	95		kept = append(kept, u)
96	96		}
97	97		}

@@ -148,15 +148,29 @@ func (c *CI) handlePush(ctx context.Context, t ciTrigger) error {
148	148		if len(runners) == 0 {
149	149		return nil
150	150		}
151		-	tags := 0
	151	+	var tags, created, deleted int
152	152		for _, u := range t.updates {
153	153		if strings.HasPrefix(u.Ref, "refs/tags/") {
154		-	tags++
	154	+	switch {
	155	+	case u.IsDelete():
	156	+	deleted++
	157	+	case u.IsCreate():
	158	+	tags++
	159	+	created++
	160	+	default:
	161	+	tags++
	162	+	}
155	163		}
156	164		}
157	165		var errs []error
158	166		for _, u := range t.updates {
159		-	if err := c.handleRef(ctx, t, u, tags); err != nil {
	167	+	var err error
	168	+	if u.IsDelete() {
	169	+	err = c.handleDelete(ctx, t, u, deleted)
	170	+	} else {
	171	+	err = c.handleRef(ctx, t, u, tags, created)
	172	+	}
	173	+	if err != nil {
160	174		errs = append(errs, fmt.Errorf("%s: %w", u.Ref, err))
161	175		}
162	176		}

@@ -187,7 +201,7 @@ type pushFacts struct {
187	201		// (pull.target) update.Ref is the default branch and commit its tip.
188	202		pull *pullFacts
189	203		// custom is set for the events that are not a push or a merge request:
190		-	// workflow_dispatch, schedule and workflow_run (ci_events.go).
	204	+	// workflow_dispatch, schedule, workflow_run, create and delete (ci_events.go).
191	205		custom *customEvent
192	206		}
193	207

@@ -234,10 +248,17 @@ func (f pushFacts) headSha() string {
234	248		return ""
235	249		}
236	250
237		-	func (c *CI) handleRef(ctx context.Context, t ciTrigger, u domain.RefUpdate, tagCount int) error {
	251	+	// handleRef starts the push workflows of a pushed ref and, for a created one,
	252	+	// its create workflows. tagCount is the number of tags the push sets,
	253	+	// tagsCreated the number it creates.
	254	+	func (c *CI) handleRef(ctx context.Context, t ciTrigger, u domain.RefUpdate, tagCount, tagsCreated int) error {
238	255		isTag := strings.HasPrefix(u.Ref, "refs/tags/")
239		-	if isTag && tagCount > workflow.MaxTagsPerPush {
240		-	return nil // GitHub creates no event for the tags of such a push
	256	+	// GitHub creates no push event for the tags of a push that sets more than
	257	+	// three, and no create event when it creates more than three.
	258	+	push := !isTag \|\| tagCount <= workflow.MaxTagsPerPush
	259	+	create := u.IsCreate() && (!isTag \|\| tagsCreated <= workflow.MaxTagsPerPush)
	260	+	if !push && !create {
	261	+	return nil
241	262		}
242	263		repo, err := c.cfg.Meta.Get(ctx, t.repo)
243	264		if err != nil {

@@ -254,16 +275,17 @@ func (c *CI) handleRef(ctx context.Context, t ciTrigger, u domain.RefUpdate, tag
254	275		skip := workflow.SkipCI(facts.head.Message)
255	276		var errs []error
256	277		for _, f := range files {
257		-	wf, perrs := workflow.Parse(f.path, f.content)
	278	+	wf, perrs := parseWorkflow(f.path, f.content)
	279	+	var problems []string
	280	+	for _, e := range perrs {
	281	+	problems = append(problems, e.Error())
	282	+	}
258	283		switch {
	284	+	case !push:
259	285		case f.problem != "":
260	286		errs = append(errs, c.startInvalid(ctx, repo, t, facts, f.path, f.path, f.problem))
261		-	case len(perrs) > 0:
262		-	var lines []string
263		-	for _, e := range perrs {
264		-	lines = append(lines, e.Error())
265		-	}
266		-	errs = append(errs, c.startInvalid(ctx, repo, t, facts, f.path, f.path, strings.Join(lines, "\n")))
	287	+	case len(problems) > 0:
	288	+	errs = append(errs, c.startInvalid(ctx, repo, t, facts, f.path, f.path, strings.Join(problems, "\n")))
267	289		case skip \|\| !wf.MatchPush(workflow.PushInfo{
268	290		Ref: u.Ref, Created: facts.created, ChangedFiles: facts.files, FilesTruncated: facts.filesTruncated, TagCount: tagCount,
269	291		}):

@@ -282,6 +304,9 @@ func (c *CI) handleRef(ctx context.Context, t ciTrigger, u domain.RefUpdate, tag
282	304		errs = append(errs, c.startRun(ctx, repo, t, facts, f.path, string(f.content), wf))
283	305		}
284	306		}
	307	+	if create {
	308	+	errs = append(errs, c.startCreateRuns(ctx, repo, t, facts, files))
	309	+	}
285	310		return errors.Join(errs...)
286	311		}
287	312

 func (c *CI) enqueue(ctx context.Context, repo domain.RepoName, updates []domain.RefUpdate, actor string, pusher domain.CIPusher, trusted bool) {
 	var kept []domain.RefUpdate
 	for _, u := range updates {
-		if !u.IsDelete() && (strings.HasPrefix(u.Ref, "refs/heads/") || strings.HasPrefix(u.Ref, "refs/tags/")) {
+		if strings.HasPrefix(u.Ref, "refs/heads/") || strings.HasPrefix(u.Ref, "refs/tags/") {
 			kept = append(kept, u)
 		}
 	}
 	if len(runners) == 0 {
 		return nil
 	}
-	tags := 0
+	var tags, created, deleted int
 	for _, u := range t.updates {
 		if strings.HasPrefix(u.Ref, "refs/tags/") {
-			tags++
+			switch {
+			case u.IsDelete():
+				deleted++
+			case u.IsCreate():
+				tags++
+				created++
+			default:
+				tags++
+			}
 		}
 	}
 	var errs []error
 	for _, u := range t.updates {
-		if err := c.handleRef(ctx, t, u, tags); err != nil {
+		var err error
+		if u.IsDelete() {
+			err = c.handleDelete(ctx, t, u, deleted)
+		} else {
+			err = c.handleRef(ctx, t, u, tags, created)
+		}
+		if err != nil {
 			errs = append(errs, fmt.Errorf("%s: %w", u.Ref, err))
 		}
 	}
 	// (pull.target) update.Ref is the default branch and commit its tip.
 	pull *pullFacts
 	// custom is set for the events that are not a push or a merge request:
-	// workflow_dispatch, schedule and workflow_run (ci_events.go).
+	// workflow_dispatch, schedule, workflow_run, create and delete (ci_events.go).
 	custom *customEvent
 }
 	return ""
 }
-func (c *CI) handleRef(ctx context.Context, t ciTrigger, u domain.RefUpdate, tagCount int) error {
+// handleRef starts the push workflows of a pushed ref and, for a created one,
+// its create workflows. tagCount is the number of tags the push sets,
+// tagsCreated the number it creates.
+func (c *CI) handleRef(ctx context.Context, t ciTrigger, u domain.RefUpdate, tagCount, tagsCreated int) error {
 	isTag := strings.HasPrefix(u.Ref, "refs/tags/")
-	if isTag && tagCount > workflow.MaxTagsPerPush {
+	// GitHub creates no push event for the tags of a push that sets more than
-		return nil // GitHub creates no event for the tags of such a push
+	// three, and no create event when it creates more than three.
+	push := !isTag || tagCount <= workflow.MaxTagsPerPush
+	create := u.IsCreate() && (!isTag || tagsCreated <= workflow.MaxTagsPerPush)
+	if !push && !create {
+		return nil
 	}
 	repo, err := c.cfg.Meta.Get(ctx, t.repo)
 	if err != nil {
 	skip := workflow.SkipCI(facts.head.Message)
 	var errs []error
 	for _, f := range files {
-		wf, perrs := workflow.Parse(f.path, f.content)
+		wf, perrs := parseWorkflow(f.path, f.content)
+		var problems []string
+		for _, e := range perrs {
+			problems = append(problems, e.Error())
+		}
 		switch {
+		case !push:
 		case f.problem != "":
 			errs = append(errs, c.startInvalid(ctx, repo, t, facts, f.path, f.path, f.problem))
-		case len(perrs) > 0:
+		case len(problems) > 0:
-			var lines []string
+			errs = append(errs, c.startInvalid(ctx, repo, t, facts, f.path, f.path, strings.Join(problems, "\n")))
-			for _, e := range perrs {
-				lines = append(lines, e.Error())
-			}
-			errs = append(errs, c.startInvalid(ctx, repo, t, facts, f.path, f.path, strings.Join(lines, "\n")))
 		case skip || !wf.MatchPush(workflow.PushInfo{
 			Ref: u.Ref, Created: facts.created, ChangedFiles: facts.files, FilesTruncated: facts.filesTruncated, TagCount: tagCount,
 		}):
 			errs = append(errs, c.startRun(ctx, repo, t, facts, f.path, string(f.content), wf))
 		}
 	}
+	if create {
+		errs = append(errs, c.startCreateRuns(ctx, repo, t, facts, files))
+	}
 	return errors.Join(errs...)
 }

A internal/usecase/ci_validate.go +69 -0

@@ -0,0 +1,69 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package usecase
	4	+
	5	+	import (
	6	+	"strings"
	7	+
	8	+	"bnema.dev/ermite-actions/template"
	9	+	"bnema.dev/ermite-actions/workflow"
	10	+	"bnema.dev/ermite/internal/domain"
	11	+	)
	12	+
	13	+	// parseWorkflow is workflow.Parse plus the checks GitHub makes when it reads a
	14	+	// workflow that the parser of ermite-actions does not: a schedule timezone that
	15	+	// is no IANA time zone name, and queue: max with cancel-in-progress: true (CI-4
	16	+	// D6). A workflow with such an error is invalid like one that does not parse.
	17	+	func parseWorkflow(path string, content []byte) (*workflow.Workflow, []workflow.Error) {
	18	+	wf, errs := workflow.Parse(path, content)
	19	+	if len(errs) > 0 \|\| wf == nil {
	20	+	return wf, errs
	21	+	}
	22	+	return wf, validateWorkflow(path, wf)
	23	+	}
	24	+
	25	+	func validateWorkflow(path string, wf *workflow.Workflow) []workflow.Error {
	26	+	var errs []workflow.Error
	27	+	if wf.On != nil {
	28	+	if ev := wf.On.Event(eventSchedule); ev != nil {
	29	+	for _, sc := range ev.Schedule {
	30	+	if _, err := domain.ParseCronTimezone(sc.Timezone); err != nil {
	31	+	errs = append(errs, workflow.Error{File: path, Raw: strings.TrimPrefix(err.Error(), domain.ErrInvalid.Error()+": ")})
	32	+	}
	33	+	}
	34	+	}
	35	+	}
	36	+	check := func(tok template.Token) {
	37	+	if pos, bad := queueMaxWithCancel(tok); bad {
	38	+	errs = append(errs, workflow.Error{File: path, Pos: pos, Raw: errQueueMaxCancel.Error()})
	39	+	}
	40	+	}
	41	+	check(wf.Concurrency)
	42	+	for _, j := range wf.Jobs {
	43	+	check(j.Concurrency)
	44	+	}
	45	+	return errs
	46	+	}
	47	+
	48	+	// queueMaxWithCancel reports a concurrency mapping that writes queue: max and
	49	+	// cancel-in-progress: true as literals, the pair GitHub refuses as a workflow
	50	+	// validation error (docs.github.com, "Control the concurrency of workflows and
	51	+	// jobs": "The combination of `queue: max` and `cancel-in-progress: true` is not
	52	+	// allowed and will result in a workflow validation error"; languageservices
	53	+	// languageservice/src/validate.ts:702-743 checks the literal pair). An
	54	+	// expression is checked when it is evaluated (evalConcurrency).
	55	+	func queueMaxWithCancel(tok template.Token) (template.Pos, bool) {
	56	+	m, ok := tok.(*template.Mapping)
	57	+	if !ok {
	58	+	return template.Pos{}, false
	59	+	}
	60	+	q, ok := m.Find("queue").(*template.String)
	61	+	if !ok \|\| q.Value != "max" {
	62	+	return template.Pos{}, false
	63	+	}
	64	+	c, ok := m.Find("cancel-in-progress").(*template.Boolean)
	65	+	if !ok \|\| !c.Value {
	66	+	return template.Pos{}, false
	67	+	}
	68	+	return c.Pos(), true
	69	+	}

@@ -0,0 +1,69 @@
	1	+	// SPDX-License-Identifier: AGPL-3.0-or-later
	2	+
	3	+	package usecase
	4	+
	5	+	import (
	6	+	"strings"
	7	+
	8	+	"bnema.dev/ermite-actions/template"
	9	+	"bnema.dev/ermite-actions/workflow"
	10	+	"bnema.dev/ermite/internal/domain"
	11	+	)
	12	+
	13	+	// parseWorkflow is workflow.Parse plus the checks GitHub makes when it reads a
	14	+	// workflow that the parser of ermite-actions does not: a schedule timezone that
	15	+	// is no IANA time zone name, and queue: max with cancel-in-progress: true (CI-4
	16	+	// D6). A workflow with such an error is invalid like one that does not parse.
	17	+	func parseWorkflow(path string, content []byte) (*workflow.Workflow, []workflow.Error) {
	18	+	wf, errs := workflow.Parse(path, content)
	19	+	if len(errs) > 0 \|\| wf == nil {
	20	+	return wf, errs
	21	+	}
	22	+	return wf, validateWorkflow(path, wf)
	23	+	}
	24	+
	25	+	func validateWorkflow(path string, wf *workflow.Workflow) []workflow.Error {
	26	+	var errs []workflow.Error
	27	+	if wf.On != nil {
	28	+	if ev := wf.On.Event(eventSchedule); ev != nil {
	29	+	for _, sc := range ev.Schedule {
	30	+	if _, err := domain.ParseCronTimezone(sc.Timezone); err != nil {
	31	+	errs = append(errs, workflow.Error{File: path, Raw: strings.TrimPrefix(err.Error(), domain.ErrInvalid.Error()+": ")})
	32	+	}
	33	+	}
	34	+	}
	35	+	}
	36	+	check := func(tok template.Token) {
	37	+	if pos, bad := queueMaxWithCancel(tok); bad {
	38	+	errs = append(errs, workflow.Error{File: path, Pos: pos, Raw: errQueueMaxCancel.Error()})
	39	+	}
	40	+	}
	41	+	check(wf.Concurrency)
	42	+	for _, j := range wf.Jobs {
	43	+	check(j.Concurrency)
	44	+	}
	45	+	return errs
	46	+	}
	47	+
	48	+	// queueMaxWithCancel reports a concurrency mapping that writes queue: max and
	49	+	// cancel-in-progress: true as literals, the pair GitHub refuses as a workflow
	50	+	// validation error (docs.github.com, "Control the concurrency of workflows and
	51	+	// jobs": "The combination of `queue: max` and `cancel-in-progress: true` is not
	52	+	// allowed and will result in a workflow validation error"; languageservices
	53	+	// languageservice/src/validate.ts:702-743 checks the literal pair). An
	54	+	// expression is checked when it is evaluated (evalConcurrency).
	55	+	func queueMaxWithCancel(tok template.Token) (template.Pos, bool) {
	56	+	m, ok := tok.(*template.Mapping)
	57	+	if !ok {
	58	+	return template.Pos{}, false
	59	+	}
	60	+	q, ok := m.Find("queue").(*template.String)
	61	+	if !ok \|\| q.Value != "max" {
	62	+	return template.Pos{}, false
	63	+	}
	64	+	c, ok := m.Find("cancel-in-progress").(*template.Boolean)
	65	+	if !ok \|\| !c.Value {
	66	+	return template.Pos{}, false
	67	+	}
	68	+	return c.Pos(), true
	69	+	}

M internal/usecase/ci_workflowrun.go +1 -1


@@ -213,7 +213,7 @@ func (c *CI) handleRunEvent(ctx context.Context, e runEvent) error {
213	213		if f.problem != "" {
214	214		continue
215	215		}
216		-	wf, perrs := workflow.Parse(f.path, f.content)
	216	+	wf, perrs := parseWorkflow(f.path, f.content)
217	217		if len(perrs) > 0 \|\| !matchWorkflowRun(wf, e.run.WorkflowName, e.action, head) {
218	218		continue
219	219		}


@@ -213,7 +213,7 @@ func (c *CI) handleRunEvent(ctx context.Context, e runEvent) error {
213		if f.problem != "" {	213		if f.problem != "" {
214		continue	214		continue
215		}	215		}
216	-	wf, perrs := workflow.Parse(f.path, f.content)	216	+	wf, perrs := parseWorkflow(f.path, f.content)
217		if len(perrs) > 0 \|\| !matchWorkflowRun(wf, e.run.WorkflowName, e.action, head) {	217		if len(perrs) > 0 \|\| !matchWorkflowRun(wf, e.run.WorkflowName, e.action, head) {
218		continue	218		continue
219		}	219		}

M proto/ermite/storage/ci.proto +16 -2


@@ -395,6 +395,8 @@ message CIConcurrency {
395	395		// preempted: cancelled because a newer entry of the group asked for it; the
396	396		// job ends with GitHub's annotation.
397	397		bool preempted = 6;
	398	+	// queue_max: queue: max (false is queue: single, the default).
	399	+	bool queue_max = 7;
398	400		}
399	401
400	402		// CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.

@@ -406,17 +408,25 @@ message CIConcurrencyEntry {
406	408		// canceling: a waiting entry with cancel-in-progress asked this running one
407	409		// to stop; the entry stays until it has ended.
408	410		bool canceling = 5;
	411	+	// queue_max: the entry was queued with queue: max.
	412	+	bool queue_max = 6;
409	413		}
410	414
411	415		// CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
412		-	// running entry and the pending one of a group (queue: single). A group with
413		-	// neither has no key.
	416	+	// running entry and the waiting ones of a group. A group with neither has no
	417	+	// key.
414	418		message CIConcurrencyRecord {
415	419		// 3 held the cancellations owed to replaced entries; a replaced entry is now
416	420		// found from its own record (it waits in a group that does not name it).
417	421		reserved 3;
418	422		CIConcurrencyEntry running = 1;
	423	+	// pending is the head of queue, written for the binaries that predate queue
	424	+	// (a downgrade keeps the first waiting entry; the others, which their record
	425	+	// no longer names, are cancelled as replaced). It is read when queue is
	426	+	// empty: a record stored by such a binary.
419	427		CIConcurrencyEntry pending = 2;
	428	+	// queue holds the waiting entries, first in first out.
	429	+	repeated CIConcurrencyEntry queue = 4;
420	430		}
421	431
422	432		// CIRefTip is stored under "cireftip/<repo>/<full ref name>" (a branch or a

@@ -446,4 +456,8 @@ message CIConcurrencyMark {
446	456		// group as the newest request wrote it, for the cancel annotation.
447	457		string group = 2;
448	458		CIConcurrencyState state = 3;
	459	+	// queue_full: with state REPLACED, the entry was a queue: max entry that
	460	+	// found the waiting line full (its annotation says so). Stored as REPLACED so
	461	+	// that a binary that predates it still cancels the entry.
	462	+	bool queue_full = 4;
449	463		}

   // preempted: cancelled because a newer entry of the group asked for it; the
   // job ends with GitHub's annotation.
   bool preempted = 6;
+  // queue_max: queue: max (false is queue: single, the default).
+  bool queue_max = 7;
 }
 // CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.
   // canceling: a waiting entry with cancel-in-progress asked this running one
   // to stop; the entry stays until it has ended.
   bool canceling = 5;
+  // queue_max: the entry was queued with queue: max.
+  bool queue_max = 6;
 }
 // CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
-// running entry and the pending one of a group (queue: single). A group with
+// running entry and the waiting ones of a group. A group with neither has no
-// neither has no key.
+// key.
 message CIConcurrencyRecord {
   // 3 held the cancellations owed to replaced entries; a replaced entry is now
   // found from its own record (it waits in a group that does not name it).
   reserved 3;
   CIConcurrencyEntry running = 1;
+  // pending is the head of queue, written for the binaries that predate queue
+  // (a downgrade keeps the first waiting entry; the others, which their record
+  // no longer names, are cancelled as replaced). It is read when queue is
+  // empty: a record stored by such a binary.
   CIConcurrencyEntry pending = 2;
+  // queue holds the waiting entries, first in first out.
+  repeated CIConcurrencyEntry queue = 4;
 }
 // CIRefTip is stored under "cireftip/<repo>/<full ref name>" (a branch or a
   // group as the newest request wrote it, for the cancel annotation.
   string group = 2;
   CIConcurrencyState state = 3;
+  // queue_full: with state REPLACED, the entry was a queue: max entry that
+  // found the waiting line full (its annotation says so). Stored as REPLACED so
+  // that a binary that predates it still cancels the entry.
+  bool queue_full = 4;
 }