detached dc42452

feat(ci): workflow and job concurrency groups (#31)

commit
dc42452c545afb7bf28069c49f46471f110494c0
author
brice · · 2026-10-15 02:32 UTC
committer
bnema · · 2026-10-15 02:32 UTC
parent
7328df5
signature
SSH UnverifiedSHA256:VOCA5lYARHcbKDZ7GhTV45VmcjWnJ6ZQIBazj17LivA The key is unknown
refs
tag: v0.13.0
27 files · +3450 -95
Unified
UnifiedSplit
M README.md +1 -1
@@ -127,7 +127,7 @@ The Actions that run here call a small part of GitHub's REST API, served at `git
127127
128128 Actions from github.com are resolved with `git ls-remote https://github.com/<owner>/<repo>` (no credentials, 20 s, 16 MiB of output, at most four at once, 60 s for a whole call). ermite hands git only the proxy settings of its own environment (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, `ALL_PROXY` and their lower-case forms), so a server that reaches the Internet through a proxy sets them for `ermite serve`; nothing else of its environment, no credential, reaches git.
129129
130-`ermite run list|view|watch|cancel|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. A workflow that uses a feature ermite does not support yet (reusable workflows, `environment:`, `concurrency`) fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
130+`ermite run list|view|watch|cancel|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. `concurrency:` is supported at the workflow level (evaluated when the run starts) and at the job level (evaluated when the job is planned, once per matrix leg): a group is per repository and compared without regard to case, `queue: single` (the default) lets one run or job go and keeps one waiting, a newer waiting entry cancels the older one (`Canceling since a higher priority waiting request for '<group>' exists`), and `cancel-in-progress` (a boolean or an expression) also cancels the running one. A job held back by a group shows `pending` in the API, the web pages and the CLI. A group that evaluates to nothing fails the job, and `queue: max` is refused with `not supported yet: queue: max`. A workflow that uses another feature ermite does not support yet (reusable workflows) fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
131131
132132 ## Web sign-in
133133
@@ -127,7 +127,7 @@ The Actions that run here call a small part of GitHub's REST API, served at `git
127 127
128 Actions from github.com are resolved with `git ls-remote https://github.com/<owner>/<repo>` (no credentials, 20 s, 16 MiB of output, at most four at once, 60 s for a whole call). ermite hands git only the proxy settings of its own environment (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, `ALL_PROXY` and their lower-case forms), so a server that reaches the Internet through a proxy sets them for `ermite serve`; nothing else of its environment, no credential, reaches git.128 Actions from github.com are resolved with `git ls-remote https://github.com/<owner>/<repo>` (no credentials, 20 s, 16 MiB of output, at most four at once, 60 s for a whole call). ermite hands git only the proxy settings of its own environment (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, `ALL_PROXY` and their lower-case forms), so a server that reaches the Internet through a proxy sets them for `ermite serve`; nothing else of its environment, no credential, reaches git.
129 129
130-`ermite run list|view|watch|cancel|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. A workflow that uses a feature ermite does not support yet (reusable workflows, `environment:`, `concurrency`) fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.130+`ermite run list|view|watch|cancel|rerun` and the `Actions` tab of a repository read and control the runs; `cancel` and `rerun` need the `ci:write` scope. With `failing_checks_block` on (the default, `ermite repo edit --failing-checks-block=false`), a merge is refused while a check of the head failed or has not finished. `concurrency:` is supported at the workflow level (evaluated when the run starts) and at the job level (evaluated when the job is planned, once per matrix leg): a group is per repository and compared without regard to case, `queue: single` (the default) lets one run or job go and keeps one waiting, a newer waiting entry cancels the older one (`Canceling since a higher priority waiting request for '<group>' exists`), and `cancel-in-progress` (a boolean or an expression) also cancels the running one. A job held back by a group shows `pending` in the API, the web pages and the CLI. A group that evaluates to nothing fails the job, and `queue: max` is refused with `not supported yet: queue: max`. A workflow that uses another feature ermite does not support yet (reusable workflows) fails with `not supported yet: <feature>`, never silently. Logs are files under `$ERMITE_DATA_DIR/ci/logs`, and the job summaries (the Markdown a step writes to `GITHUB_STEP_SUMMARY`) under `$ERMITE_DATA_DIR/ci/summaries`; both go with their run (retention, repository deletion). Annotations (`::notice`, `::warning`, `::error`) are kept in the metadata store. The run page shows them in an Annotations box and the job page at their step; the summaries are rendered under the run with the README renderer (raw HTML dropped, `javascript:` links dropped). `ermite run view` prints an ANNOTATIONS section, and `ermite run view --summary` prints the summaries as Markdown.
131 131
132 ## Web sign-in132 ## Web sign-in
133 133
M gen/ermite/storage/ci.pb.go +480 -68
@@ -36,6 +36,8 @@ const (
3636 CIStatus_CI_STATUS_QUEUED CIStatus = 1
3737 CIStatus_CI_STATUS_IN_PROGRESS CIStatus = 2
3838 CIStatus_CI_STATUS_COMPLETED CIStatus = 3
39+ // Held back by a concurrency group (jobs and checks only).
40+ CIStatus_CI_STATUS_PENDING CIStatus = 4
3941 )
4042
4143 // Enum value maps for CIStatus.
@@ -45,12 +47,14 @@ var (
4547 1: "CI_STATUS_QUEUED",
4648 2: "CI_STATUS_IN_PROGRESS",
4749 3: "CI_STATUS_COMPLETED",
50+ 4: "CI_STATUS_PENDING",
4851 }
4952 CIStatus_value = map[string]int32{
5053 "CI_STATUS_UNSPECIFIED": 0,
5154 "CI_STATUS_QUEUED": 1,
5255 "CI_STATUS_IN_PROGRESS": 2,
5356 "CI_STATUS_COMPLETED": 3,
57+ "CI_STATUS_PENDING": 4,
5458 }
5559 )
5660
@@ -422,6 +426,63 @@ func (CIAnnotationLevel) EnumDescriptor() ([]byte, []int) {
422426 return file_ermite_storage_ci_proto_rawDescGZIP(), []int{6}
423427 }
424428
429+// CIConcurrencyState is where a run or a job is in its concurrency group.
430+type CIConcurrencyState int32
431+
432+const (
433+ CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED CIConcurrencyState = 0
434+ // Another entry of the group runs: this one waits.
435+ CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING CIConcurrencyState = 1
436+ // This entry is the running one of its group.
437+ CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING CIConcurrencyState = 2
438+ // Only in a CIConcurrencyMark: a newer entry replaced this one, which is owed
439+ // a cancellation and may never acquire the group.
440+ CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED CIConcurrencyState = 3
441+)
442+
443+// Enum value maps for CIConcurrencyState.
444+var (
445+ CIConcurrencyState_name = map[int32]string{
446+ 0: "CI_CONCURRENCY_STATE_UNSPECIFIED",
447+ 1: "CI_CONCURRENCY_STATE_PENDING",
448+ 2: "CI_CONCURRENCY_STATE_HOLDING",
449+ 3: "CI_CONCURRENCY_STATE_REPLACED",
450+ }
451+ CIConcurrencyState_value = map[string]int32{
452+ "CI_CONCURRENCY_STATE_UNSPECIFIED": 0,
453+ "CI_CONCURRENCY_STATE_PENDING": 1,
454+ "CI_CONCURRENCY_STATE_HOLDING": 2,
455+ "CI_CONCURRENCY_STATE_REPLACED": 3,
456+ }
457+)
458+
459+func (x CIConcurrencyState) Enum() *CIConcurrencyState {
460+ p := new(CIConcurrencyState)
461+ *p = x
462+ return p
463+}
464+
465+func (x CIConcurrencyState) String() string {
466+ return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
467+}
468+
469+func (CIConcurrencyState) Descriptor() protoreflect.EnumDescriptor {
470+ return file_ermite_storage_ci_proto_enumTypes[7].Descriptor()
471+}
472+
473+func (CIConcurrencyState) Type() protoreflect.EnumType {
474+ return &file_ermite_storage_ci_proto_enumTypes[7]
475+}
476+
477+func (x CIConcurrencyState) Number() protoreflect.EnumNumber {
478+ return protoreflect.EnumNumber(x)
479+}
480+
481+// Deprecated: Use CIConcurrencyState.Descriptor instead.
482+func (CIConcurrencyState) EnumDescriptor() ([]byte, []int) {
483+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{7}
484+}
485+
425486 // CIRun is stored under "run/<repo>/<id:020d>".
426487 type CIRun struct {
427488 state protoimpl.MessageState `protogen:"open.v1"`
@@ -450,7 +511,9 @@ type CIRun struct {
450511 Name string `protobuf:"bytes,22,opt,name=name,proto3" json:"name,omitempty"`
451512 // trusted: the actor of the run is the owner, so its jobs get secrets. False
452513 // in runs stored before it existed (the safe reading).
453- Trusted bool `protobuf:"varint,23,opt,name=trusted,proto3" json:"trusted,omitempty"`
514+ Trusted bool `protobuf:"varint,23,opt,name=trusted,proto3" json:"trusted,omitempty"`
515+ // The evaluated workflow-level concurrency of the attempt.
516+ Concurrency *CIConcurrency `protobuf:"bytes,24,opt,name=concurrency,proto3" json:"concurrency,omitempty"`
454517 unknownFields protoimpl.UnknownFields
455518 sizeCache protoimpl.SizeCache
456519 }
@@ -632,6 +695,13 @@ func (x *CIRun) GetTrusted() bool {
632695 return false
633696 }
634697
698+func (x *CIRun) GetConcurrency() *CIConcurrency {
699+ if x != nil {
700+ return x.Concurrency
701+ }
702+ return nil
703+}
704+
635705 // CIRunSource is stored under "runsrc/<run id:020d>": what a run was planned
636706 // from, kept apart so that listing runs does not decode it.
637707 type CIRunSource struct {
@@ -900,7 +970,10 @@ type CIJob struct {
900970 // The effective permissions of the job's repository token (D3), computed when
901971 // the job is planned: permission key to "read", "write" or "none". Absent on
902972 // a job planned before the field existed: the restricted default.
903- Permissions map[string]string `protobuf:"bytes,42,rep,name=permissions,proto3" json:"permissions,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
973+ Permissions map[string]string `protobuf:"bytes,42,rep,name=permissions,proto3" json:"permissions,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
974+ // The job's concurrency group (its own, else the one of its run) and where
975+ // it is in it.
976+ Concurrency *CIConcurrency `protobuf:"bytes,43,opt,name=concurrency,proto3" json:"concurrency,omitempty"`
904977 unknownFields protoimpl.UnknownFields
905978 sizeCache protoimpl.SizeCache
906979 }
@@ -1222,6 +1295,13 @@ func (x *CIJob) GetPermissions() map[string]string {
12221295 return nil
12231296 }
12241297
1298+func (x *CIJob) GetConcurrency() *CIConcurrency {
1299+ if x != nil {
1300+ return x.Concurrency
1301+ }
1302+ return nil
1303+}
1304+
12251305 // CICheck is stored under "check/<repo>/<sha>/<id>": the id of its job, or
12261306 // "run-<run id>" for a run that failed without a job (invalid workflow file).
12271307 type CICheck struct {
@@ -1833,11 +1913,303 @@ func (x *CIStepAnnotations) GetAnnotations() []*CIAnnotation {
18331913 return nil
18341914 }
18351915
1916+// CIConcurrency is the evaluated "concurrency:" of a run or a job, kept on its
1917+// record (CIRun.concurrency, CIJob.concurrency).
1918+type CIConcurrency struct {
1919+ state protoimpl.MessageState `protogen:"open.v1"`
1920+ Group string `protobuf:"bytes,1,opt,name=group,proto3" json:"group,omitempty"`
1921+ Cancel bool `protobuf:"varint,2,opt,name=cancel,proto3" json:"cancel,omitempty"`
1922+ State CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
1923+ // evaluated: the expressions were evaluated, whatever the result.
1924+ Evaluated bool `protobuf:"varint,4,opt,name=evaluated,proto3" json:"evaluated,omitempty"`
1925+ // from_run: a job that shows the group of its run and holds nothing itself.
1926+ FromRun bool `protobuf:"varint,5,opt,name=from_run,json=fromRun,proto3" json:"from_run,omitempty"`
1927+ // preempted: cancelled because a newer entry of the group asked for it; the
1928+ // job ends with GitHub's annotation.
1929+ Preempted bool `protobuf:"varint,6,opt,name=preempted,proto3" json:"preempted,omitempty"`
1930+ unknownFields protoimpl.UnknownFields
1931+ sizeCache protoimpl.SizeCache
1932+}
1933+
1934+func (x *CIConcurrency) Reset() {
1935+ *x = CIConcurrency{}
1936+ mi := &file_ermite_storage_ci_proto_msgTypes[10]
1937+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
1938+ ms.StoreMessageInfo(mi)
1939+}
1940+
1941+func (x *CIConcurrency) String() string {
1942+ return protoimpl.X.MessageStringOf(x)
1943+}
1944+
1945+func (*CIConcurrency) ProtoMessage() {}
1946+
1947+func (x *CIConcurrency) ProtoReflect() protoreflect.Message {
1948+ mi := &file_ermite_storage_ci_proto_msgTypes[10]
1949+ if x != nil {
1950+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
1951+ if ms.LoadMessageInfo() == nil {
1952+ ms.StoreMessageInfo(mi)
1953+ }
1954+ return ms
1955+ }
1956+ return mi.MessageOf(x)
1957+}
1958+
1959+// Deprecated: Use CIConcurrency.ProtoReflect.Descriptor instead.
1960+func (*CIConcurrency) Descriptor() ([]byte, []int) {
1961+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{10}
1962+}
1963+
1964+func (x *CIConcurrency) GetGroup() string {
1965+ if x != nil {
1966+ return x.Group
1967+ }
1968+ return ""
1969+}
1970+
1971+func (x *CIConcurrency) GetCancel() bool {
1972+ if x != nil {
1973+ return x.Cancel
1974+ }
1975+ return false
1976+}
1977+
1978+func (x *CIConcurrency) GetState() CIConcurrencyState {
1979+ if x != nil {
1980+ return x.State
1981+ }
1982+ return CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED
1983+}
1984+
1985+func (x *CIConcurrency) GetEvaluated() bool {
1986+ if x != nil {
1987+ return x.Evaluated
1988+ }
1989+ return false
1990+}
1991+
1992+func (x *CIConcurrency) GetFromRun() bool {
1993+ if x != nil {
1994+ return x.FromRun
1995+ }
1996+ return false
1997+}
1998+
1999+func (x *CIConcurrency) GetPreempted() bool {
2000+ if x != nil {
2001+ return x.Preempted
2002+ }
2003+ return false
2004+}
2005+
2006+// CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.
2007+type CIConcurrencyEntry struct {
2008+ state protoimpl.MessageState `protogen:"open.v1"`
2009+ RunId int64 `protobuf:"varint,1,opt,name=run_id,json=runId,proto3" json:"run_id,omitempty"`
2010+ Attempt int32 `protobuf:"varint,2,opt,name=attempt,proto3" json:"attempt,omitempty"`
2011+ JobId string `protobuf:"bytes,3,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
2012+ Cancel bool `protobuf:"varint,4,opt,name=cancel,proto3" json:"cancel,omitempty"`
2013+ // canceling: a waiting entry with cancel-in-progress asked this running one
2014+ // to stop; the entry stays until it has ended.
2015+ Canceling bool `protobuf:"varint,5,opt,name=canceling,proto3" json:"canceling,omitempty"`
2016+ unknownFields protoimpl.UnknownFields
2017+ sizeCache protoimpl.SizeCache
2018+}
2019+
2020+func (x *CIConcurrencyEntry) Reset() {
2021+ *x = CIConcurrencyEntry{}
2022+ mi := &file_ermite_storage_ci_proto_msgTypes[11]
2023+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2024+ ms.StoreMessageInfo(mi)
2025+}
2026+
2027+func (x *CIConcurrencyEntry) String() string {
2028+ return protoimpl.X.MessageStringOf(x)
2029+}
2030+
2031+func (*CIConcurrencyEntry) ProtoMessage() {}
2032+
2033+func (x *CIConcurrencyEntry) ProtoReflect() protoreflect.Message {
2034+ mi := &file_ermite_storage_ci_proto_msgTypes[11]
2035+ if x != nil {
2036+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2037+ if ms.LoadMessageInfo() == nil {
2038+ ms.StoreMessageInfo(mi)
2039+ }
2040+ return ms
2041+ }
2042+ return mi.MessageOf(x)
2043+}
2044+
2045+// Deprecated: Use CIConcurrencyEntry.ProtoReflect.Descriptor instead.
2046+func (*CIConcurrencyEntry) Descriptor() ([]byte, []int) {
2047+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{11}
2048+}
2049+
2050+func (x *CIConcurrencyEntry) GetRunId() int64 {
2051+ if x != nil {
2052+ return x.RunId
2053+ }
2054+ return 0
2055+}
2056+
2057+func (x *CIConcurrencyEntry) GetAttempt() int32 {
2058+ if x != nil {
2059+ return x.Attempt
2060+ }
2061+ return 0
2062+}
2063+
2064+func (x *CIConcurrencyEntry) GetJobId() string {
2065+ if x != nil {
2066+ return x.JobId
2067+ }
2068+ return ""
2069+}
2070+
2071+func (x *CIConcurrencyEntry) GetCancel() bool {
2072+ if x != nil {
2073+ return x.Cancel
2074+ }
2075+ return false
2076+}
2077+
2078+func (x *CIConcurrencyEntry) GetCanceling() bool {
2079+ if x != nil {
2080+ return x.Canceling
2081+ }
2082+ return false
2083+}
2084+
2085+// CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
2086+// running entry and the pending one of a group (queue: single). A group with
2087+// neither has no key.
2088+type CIConcurrencyRecord struct {
2089+ state protoimpl.MessageState `protogen:"open.v1"`
2090+ Running *CIConcurrencyEntry `protobuf:"bytes,1,opt,name=running,proto3" json:"running,omitempty"`
2091+ Pending *CIConcurrencyEntry `protobuf:"bytes,2,opt,name=pending,proto3" json:"pending,omitempty"`
2092+ unknownFields protoimpl.UnknownFields
2093+ sizeCache protoimpl.SizeCache
2094+}
2095+
2096+func (x *CIConcurrencyRecord) Reset() {
2097+ *x = CIConcurrencyRecord{}
2098+ mi := &file_ermite_storage_ci_proto_msgTypes[12]
2099+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2100+ ms.StoreMessageInfo(mi)
2101+}
2102+
2103+func (x *CIConcurrencyRecord) String() string {
2104+ return protoimpl.X.MessageStringOf(x)
2105+}
2106+
2107+func (*CIConcurrencyRecord) ProtoMessage() {}
2108+
2109+func (x *CIConcurrencyRecord) ProtoReflect() protoreflect.Message {
2110+ mi := &file_ermite_storage_ci_proto_msgTypes[12]
2111+ if x != nil {
2112+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2113+ if ms.LoadMessageInfo() == nil {
2114+ ms.StoreMessageInfo(mi)
2115+ }
2116+ return ms
2117+ }
2118+ return mi.MessageOf(x)
2119+}
2120+
2121+// Deprecated: Use CIConcurrencyRecord.ProtoReflect.Descriptor instead.
2122+func (*CIConcurrencyRecord) Descriptor() ([]byte, []int) {
2123+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{12}
2124+}
2125+
2126+func (x *CIConcurrencyRecord) GetRunning() *CIConcurrencyEntry {
2127+ if x != nil {
2128+ return x.Running
2129+ }
2130+ return nil
2131+}
2132+
2133+func (x *CIConcurrencyRecord) GetPending() *CIConcurrencyEntry {
2134+ if x != nil {
2135+ return x.Pending
2136+ }
2137+ return nil
2138+}
2139+
2140+// CIConcurrencyMark is stored under
2141+// "concurmark/<repo>/<escaped key(group)>/<run:020d>/<attempt:010d>/<job id or ->":
2142+// where an entry stands in its group. It is written in the same batch as the
2143+// group record ("concur/..."), so that an entry is marked pending or holding if
2144+// and only if the record names it; an entry the record stops naming because a
2145+// newer one replaced it is marked replaced in that very batch. The mark is
2146+// removed with the entry (it ended, was released, or is gone).
2147+type CIConcurrencyMark struct {
2148+ state protoimpl.MessageState `protogen:"open.v1"`
2149+ Entry *CIConcurrencyEntry `protobuf:"bytes,1,opt,name=entry,proto3" json:"entry,omitempty"`
2150+ // group as the newest request wrote it, for the cancel annotation.
2151+ Group string `protobuf:"bytes,2,opt,name=group,proto3" json:"group,omitempty"`
2152+ State CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
2153+ unknownFields protoimpl.UnknownFields
2154+ sizeCache protoimpl.SizeCache
2155+}
2156+
2157+func (x *CIConcurrencyMark) Reset() {
2158+ *x = CIConcurrencyMark{}
2159+ mi := &file_ermite_storage_ci_proto_msgTypes[13]
2160+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2161+ ms.StoreMessageInfo(mi)
2162+}
2163+
2164+func (x *CIConcurrencyMark) String() string {
2165+ return protoimpl.X.MessageStringOf(x)
2166+}
2167+
2168+func (*CIConcurrencyMark) ProtoMessage() {}
2169+
2170+func (x *CIConcurrencyMark) ProtoReflect() protoreflect.Message {
2171+ mi := &file_ermite_storage_ci_proto_msgTypes[13]
2172+ if x != nil {
2173+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2174+ if ms.LoadMessageInfo() == nil {
2175+ ms.StoreMessageInfo(mi)
2176+ }
2177+ return ms
2178+ }
2179+ return mi.MessageOf(x)
2180+}
2181+
2182+// Deprecated: Use CIConcurrencyMark.ProtoReflect.Descriptor instead.
2183+func (*CIConcurrencyMark) Descriptor() ([]byte, []int) {
2184+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{13}
2185+}
2186+
2187+func (x *CIConcurrencyMark) GetEntry() *CIConcurrencyEntry {
2188+ if x != nil {
2189+ return x.Entry
2190+ }
2191+ return nil
2192+}
2193+
2194+func (x *CIConcurrencyMark) GetGroup() string {
2195+ if x != nil {
2196+ return x.Group
2197+ }
2198+ return ""
2199+}
2200+
2201+func (x *CIConcurrencyMark) GetState() CIConcurrencyState {
2202+ if x != nil {
2203+ return x.State
2204+ }
2205+ return CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED
2206+}
2207+
18362208 var File_ermite_storage_ci_proto protoreflect.FileDescriptor
18372209
18382210 const file_ermite_storage_ci_proto_rawDesc = "" +
18392211 "\n" +
1840- "\x17ermite/storage/ci.proto\x12\x0eermite.storage\x1a\x1fgoogle/protobuf/timestamp.proto\"\xf9\x05\n" +
2212+ "\x17ermite/storage/ci.proto\x12\x0eermite.storage\x1a\x1fgoogle/protobuf/timestamp.proto\"\xba\x06\n" +
18412213 "\x05CIRun\x12\x12\n" +
18422214 "\x04repo\x18\x01 \x01(\tR\x04repo\x12\x0e\n" +
18432215 "\x02id\x18\x02 \x01(\x03R\x02id\x12\x16\n" +
@@ -1864,7 +2236,8 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
18642236 "\x10cancel_requested\x18\x14 \x01(\bR\x0fcancelRequested\x12J\n" +
18652237 "\x13cancel_requested_at\x18\x15 \x01(\v2\x1a.google.protobuf.TimestampR\x11cancelRequestedAt\x12\x12\n" +
18662238 "\x04name\x18\x16 \x01(\tR\x04name\x12\x18\n" +
1867- "\atrusted\x18\x17 \x01(\bR\atrustedJ\x04\b\x12\x10\x13J\x04\b\x13\x10\x14\"\xc7\x01\n" +
2239+ "\atrusted\x18\x17 \x01(\bR\atrusted\x12?\n" +
2240+ "\vconcurrency\x18\x18 \x01(\v2\x1d.ermite.storage.CIConcurrencyR\vconcurrencyJ\x04\b\x12\x10\x13J\x04\b\x13\x10\x14\"\xc7\x01\n" +
18682241 "\vCIRunSource\x12\x15\n" +
18692242 "\x06run_id\x18\x04 \x01(\x03R\x05runId\x12#\n" +
18702243 "\rworkflow_yaml\x18\x01 \x01(\tR\fworkflowYaml\x12\x1f\n" +
@@ -1887,7 +2260,7 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
18872260 "\tlog_start\x18\a \x01(\x03R\blogStart\x12\x1b\n" +
18882261 "\thas_lines\x18\t \x01(\bR\bhasLines\x126\n" +
18892262 "\aoutcome\x18\v \x01(\x0e2\x1c.ermite.storage.CIConclusionR\aoutcomeJ\x04\b\b\x10\tJ\x04\b\n" +
1890- "\x10\v\"\x92\x0e\n" +
2263+ "\x10\v\"\xd3\x0e\n" +
18912264 "\x05CIJob\x12\x0e\n" +
18922265 "\x02id\x18\x01 \x01(\tR\x02id\x12\x15\n" +
18932266 "\x06run_id\x18\x02 \x01(\x03R\x05runId\x12\x18\n" +
@@ -1940,7 +2313,8 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
19402313 "\rcontexts_json\x18' \x01(\tR\fcontextsJson\x12*\n" +
19412314 "\x11continue_on_error\x18( \x01(\bR\x0fcontinueOnError\x12 \n" +
19422315 "\venvironment\x18) \x01(\tR\venvironment\x12H\n" +
1943- "\vpermissions\x18* \x03(\v2&.ermite.storage.CIJob.PermissionsEntryR\vpermissions\x1a:\n" +
2316+ "\vpermissions\x18* \x03(\v2&.ermite.storage.CIJob.PermissionsEntryR\vpermissions\x12?\n" +
2317+ "\vconcurrency\x18+ \x01(\v2\x1d.ermite.storage.CIConcurrencyR\vconcurrency\x1a:\n" +
19442318 "\fOutputsEntry\x12\x10\n" +
19452319 "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
19462320 "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1a>\n" +
@@ -2013,12 +2387,33 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
20132387 "\n" +
20142388 "end_column\x18\b \x01(\x05R\tendColumn\"S\n" +
20152389 "\x11CIStepAnnotations\x12>\n" +
2016- "\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations*o\n" +
2390+ "\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations\"\xce\x01\n" +
2391+ "\rCIConcurrency\x12\x14\n" +
2392+ "\x05group\x18\x01 \x01(\tR\x05group\x12\x16\n" +
2393+ "\x06cancel\x18\x02 \x01(\bR\x06cancel\x128\n" +
2394+ "\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state\x12\x1c\n" +
2395+ "\tevaluated\x18\x04 \x01(\bR\tevaluated\x12\x19\n" +
2396+ "\bfrom_run\x18\x05 \x01(\bR\afromRun\x12\x1c\n" +
2397+ "\tpreempted\x18\x06 \x01(\bR\tpreempted\"\x92\x01\n" +
2398+ "\x12CIConcurrencyEntry\x12\x15\n" +
2399+ "\x06run_id\x18\x01 \x01(\x03R\x05runId\x12\x18\n" +
2400+ "\aattempt\x18\x02 \x01(\x05R\aattempt\x12\x15\n" +
2401+ "\x06job_id\x18\x03 \x01(\tR\x05jobId\x12\x16\n" +
2402+ "\x06cancel\x18\x04 \x01(\bR\x06cancel\x12\x1c\n" +
2403+ "\tcanceling\x18\x05 \x01(\bR\tcanceling\"\x97\x01\n" +
2404+ "\x13CIConcurrencyRecord\x12<\n" +
2405+ "\arunning\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\arunning\x12<\n" +
2406+ "\apending\x18\x02 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\apendingJ\x04\b\x03\x10\x04\"\x9d\x01\n" +
2407+ "\x11CIConcurrencyMark\x128\n" +
2408+ "\x05entry\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\x05entry\x12\x14\n" +
2409+ "\x05group\x18\x02 \x01(\tR\x05group\x128\n" +
2410+ "\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state*\x86\x01\n" +
20172411 "\bCIStatus\x12\x19\n" +
20182412 "\x15CI_STATUS_UNSPECIFIED\x10\x00\x12\x14\n" +
20192413 "\x10CI_STATUS_QUEUED\x10\x01\x12\x19\n" +
20202414 "\x15CI_STATUS_IN_PROGRESS\x10\x02\x12\x17\n" +
2021- "\x13CI_STATUS_COMPLETED\x10\x03*\xf6\x01\n" +
2415+ "\x13CI_STATUS_COMPLETED\x10\x03\x12\x15\n" +
2416+ "\x11CI_STATUS_PENDING\x10\x04*\xf6\x01\n" +
20222417 "\fCIConclusion\x12\x1d\n" +
20232418 "\x19CI_CONCLUSION_UNSPECIFIED\x10\x00\x12\x19\n" +
20242419 "\x15CI_CONCLUSION_SUCCESS\x10\x01\x12\x19\n" +
@@ -2055,7 +2450,12 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
20552450 "\x1fCI_ANNOTATION_LEVEL_UNSPECIFIED\x10\x00\x12\x1e\n" +
20562451 "\x1aCI_ANNOTATION_LEVEL_NOTICE\x10\x01\x12\x1f\n" +
20572452 "\x1bCI_ANNOTATION_LEVEL_WARNING\x10\x02\x12\x1f\n" +
2058- "\x1bCI_ANNOTATION_LEVEL_FAILURE\x10\x03B/Z-bnema.dev/ermite/gen/ermite/storage;storagepbb\x06proto3"
2453+ "\x1bCI_ANNOTATION_LEVEL_FAILURE\x10\x03*\xa1\x01\n" +
2454+ "\x12CIConcurrencyState\x12$\n" +
2455+ " CI_CONCURRENCY_STATE_UNSPECIFIED\x10\x00\x12 \n" +
2456+ "\x1cCI_CONCURRENCY_STATE_PENDING\x10\x01\x12 \n" +
2457+ "\x1cCI_CONCURRENCY_STATE_HOLDING\x10\x02\x12!\n" +
2458+ "\x1dCI_CONCURRENCY_STATE_REPLACED\x10\x03B/Z-bnema.dev/ermite/gen/ermite/storage;storagepbb\x06proto3"
20592459
20602460 var (
20612461 file_ermite_storage_ci_proto_rawDescOnce sync.Once
@@ -2069,8 +2469,8 @@ func file_ermite_storage_ci_proto_rawDescGZIP() []byte {
20692469 return file_ermite_storage_ci_proto_rawDescData
20702470 }
20712471
2072-var file_ermite_storage_ci_proto_enumTypes = make([]protoimpl.EnumInfo, 7)
2073-var file_ermite_storage_ci_proto_msgTypes = make([]protoimpl.MessageInfo, 13)
2472+var file_ermite_storage_ci_proto_enumTypes = make([]protoimpl.EnumInfo, 8)
2473+var file_ermite_storage_ci_proto_msgTypes = make([]protoimpl.MessageInfo, 17)
20742474 var file_ermite_storage_ci_proto_goTypes = []any{
20752475 (CIStatus)(0), // 0: ermite.storage.CIStatus
20762476 (CIConclusion)(0), // 1: ermite.storage.CIConclusion
@@ -2079,65 +2479,77 @@ var file_ermite_storage_ci_proto_goTypes = []any{
20792479 (CIIsolation)(0), // 4: ermite.storage.CIIsolation
20802480 (CICancelReason)(0), // 5: ermite.storage.CICancelReason
20812481 (CIAnnotationLevel)(0), // 6: ermite.storage.CIAnnotationLevel
2082- (*CIRun)(nil), // 7: ermite.storage.CIRun
2083- (*CIRunSource)(nil), // 8: ermite.storage.CIRunSource
2084- (*CIStep)(nil), // 9: ermite.storage.CIStep
2085- (*CIJob)(nil), // 10: ermite.storage.CIJob
2086- (*CICheck)(nil), // 11: ermite.storage.CICheck
2087- (*CIRunner)(nil), // 12: ermite.storage.CIRunner
2088- (*CIRepoToken)(nil), // 13: ermite.storage.CIRepoToken
2089- (*CIActionResolution)(nil), // 14: ermite.storage.CIActionResolution
2090- (*CIAnnotation)(nil), // 15: ermite.storage.CIAnnotation
2091- (*CIStepAnnotations)(nil), // 16: ermite.storage.CIStepAnnotations
2092- nil, // 17: ermite.storage.CIJob.OutputsEntry
2093- nil, // 18: ermite.storage.CIJob.PermissionsEntry
2094- nil, // 19: ermite.storage.CIRepoToken.PermissionsEntry
2095- (*timestamppb.Timestamp)(nil), // 20: google.protobuf.Timestamp
2482+ (CIConcurrencyState)(0), // 7: ermite.storage.CIConcurrencyState
2483+ (*CIRun)(nil), // 8: ermite.storage.CIRun
2484+ (*CIRunSource)(nil), // 9: ermite.storage.CIRunSource
2485+ (*CIStep)(nil), // 10: ermite.storage.CIStep
2486+ (*CIJob)(nil), // 11: ermite.storage.CIJob
2487+ (*CICheck)(nil), // 12: ermite.storage.CICheck
2488+ (*CIRunner)(nil), // 13: ermite.storage.CIRunner
2489+ (*CIRepoToken)(nil), // 14: ermite.storage.CIRepoToken
2490+ (*CIActionResolution)(nil), // 15: ermite.storage.CIActionResolution
2491+ (*CIAnnotation)(nil), // 16: ermite.storage.CIAnnotation
2492+ (*CIStepAnnotations)(nil), // 17: ermite.storage.CIStepAnnotations
2493+ (*CIConcurrency)(nil), // 18: ermite.storage.CIConcurrency
2494+ (*CIConcurrencyEntry)(nil), // 19: ermite.storage.CIConcurrencyEntry
2495+ (*CIConcurrencyRecord)(nil), // 20: ermite.storage.CIConcurrencyRecord
2496+ (*CIConcurrencyMark)(nil), // 21: ermite.storage.CIConcurrencyMark
2497+ nil, // 22: ermite.storage.CIJob.OutputsEntry
2498+ nil, // 23: ermite.storage.CIJob.PermissionsEntry
2499+ nil, // 24: ermite.storage.CIRepoToken.PermissionsEntry
2500+ (*timestamppb.Timestamp)(nil), // 25: google.protobuf.Timestamp
20962501 }
20972502 var file_ermite_storage_ci_proto_depIdxs = []int32{
20982503 0, // 0: ermite.storage.CIRun.status:type_name -> ermite.storage.CIStatus
20992504 1, // 1: ermite.storage.CIRun.conclusion:type_name -> ermite.storage.CIConclusion
2100- 20, // 2: ermite.storage.CIRun.created_at:type_name -> google.protobuf.Timestamp
2101- 20, // 3: ermite.storage.CIRun.started_at:type_name -> google.protobuf.Timestamp
2102- 20, // 4: ermite.storage.CIRun.completed_at:type_name -> google.protobuf.Timestamp
2103- 20, // 5: ermite.storage.CIRun.cancel_requested_at:type_name -> google.protobuf.Timestamp
2104- 2, // 6: ermite.storage.CIStep.state:type_name -> ermite.storage.CIStepState
2105- 1, // 7: ermite.storage.CIStep.conclusion:type_name -> ermite.storage.CIConclusion
2106- 20, // 8: ermite.storage.CIStep.started_at:type_name -> google.protobuf.Timestamp
2107- 20, // 9: ermite.storage.CIStep.completed_at:type_name -> google.protobuf.Timestamp
2108- 1, // 10: ermite.storage.CIStep.outcome:type_name -> ermite.storage.CIConclusion
2109- 3, // 11: ermite.storage.CIJob.phase:type_name -> ermite.storage.CIJobPhase
2110- 1, // 12: ermite.storage.CIJob.conclusion:type_name -> ermite.storage.CIConclusion
2111- 4, // 13: ermite.storage.CIJob.isolation:type_name -> ermite.storage.CIIsolation
2112- 9, // 14: ermite.storage.CIJob.steps:type_name -> ermite.storage.CIStep
2113- 20, // 15: ermite.storage.CIJob.created_at:type_name -> google.protobuf.Timestamp
2114- 20, // 16: ermite.storage.CIJob.started_at:type_name -> google.protobuf.Timestamp
2115- 20, // 17: ermite.storage.CIJob.completed_at:type_name -> google.protobuf.Timestamp
2116- 17, // 18: ermite.storage.CIJob.outputs:type_name -> ermite.storage.CIJob.OutputsEntry
2117- 20, // 19: ermite.storage.CIJob.ready_at:type_name -> google.protobuf.Timestamp
2118- 20, // 20: ermite.storage.CIJob.assigned_at:type_name -> google.protobuf.Timestamp
2119- 20, // 21: ermite.storage.CIJob.acked_at:type_name -> google.protobuf.Timestamp
2120- 20, // 22: ermite.storage.CIJob.token_valid_until:type_name -> google.protobuf.Timestamp
2121- 5, // 23: ermite.storage.CIJob.cancel_reason:type_name -> ermite.storage.CICancelReason
2122- 20, // 24: ermite.storage.CIJob.cancel_requested_at:type_name -> google.protobuf.Timestamp
2123- 18, // 25: ermite.storage.CIJob.permissions:type_name -> ermite.storage.CIJob.PermissionsEntry
2124- 0, // 26: ermite.storage.CICheck.status:type_name -> ermite.storage.CIStatus
2125- 1, // 27: ermite.storage.CICheck.conclusion:type_name -> ermite.storage.CIConclusion
2126- 20, // 28: ermite.storage.CICheck.created_at:type_name -> google.protobuf.Timestamp
2127- 20, // 29: ermite.storage.CICheck.started_at:type_name -> google.protobuf.Timestamp
2128- 20, // 30: ermite.storage.CICheck.completed_at:type_name -> google.protobuf.Timestamp
2129- 4, // 31: ermite.storage.CIRunner.isolation:type_name -> ermite.storage.CIIsolation
2130- 20, // 32: ermite.storage.CIRunner.registered_at:type_name -> google.protobuf.Timestamp
2131- 20, // 33: ermite.storage.CIRepoToken.expires_at:type_name -> google.protobuf.Timestamp
2132- 20, // 34: ermite.storage.CIRepoToken.created_at:type_name -> google.protobuf.Timestamp
2133- 19, // 35: ermite.storage.CIRepoToken.permissions:type_name -> ermite.storage.CIRepoToken.PermissionsEntry
2134- 6, // 36: ermite.storage.CIAnnotation.level:type_name -> ermite.storage.CIAnnotationLevel
2135- 15, // 37: ermite.storage.CIStepAnnotations.annotations:type_name -> ermite.storage.CIAnnotation
2136- 38, // [38:38] is the sub-list for method output_type
2137- 38, // [38:38] is the sub-list for method input_type
2138- 38, // [38:38] is the sub-list for extension type_name
2139- 38, // [38:38] is the sub-list for extension extendee
2140- 0, // [0:38] is the sub-list for field type_name
2505+ 25, // 2: ermite.storage.CIRun.created_at:type_name -> google.protobuf.Timestamp
2506+ 25, // 3: ermite.storage.CIRun.started_at:type_name -> google.protobuf.Timestamp
2507+ 25, // 4: ermite.storage.CIRun.completed_at:type_name -> google.protobuf.Timestamp
2508+ 25, // 5: ermite.storage.CIRun.cancel_requested_at:type_name -> google.protobuf.Timestamp
2509+ 18, // 6: ermite.storage.CIRun.concurrency:type_name -> ermite.storage.CIConcurrency
2510+ 2, // 7: ermite.storage.CIStep.state:type_name -> ermite.storage.CIStepState
2511+ 1, // 8: ermite.storage.CIStep.conclusion:type_name -> ermite.storage.CIConclusion
2512+ 25, // 9: ermite.storage.CIStep.started_at:type_name -> google.protobuf.Timestamp
2513+ 25, // 10: ermite.storage.CIStep.completed_at:type_name -> google.protobuf.Timestamp
2514+ 1, // 11: ermite.storage.CIStep.outcome:type_name -> ermite.storage.CIConclusion
2515+ 3, // 12: ermite.storage.CIJob.phase:type_name -> ermite.storage.CIJobPhase
2516+ 1, // 13: ermite.storage.CIJob.conclusion:type_name -> ermite.storage.CIConclusion
2517+ 4, // 14: ermite.storage.CIJob.isolation:type_name -> ermite.storage.CIIsolation
2518+ 10, // 15: ermite.storage.CIJob.steps:type_name -> ermite.storage.CIStep
2519+ 25, // 16: ermite.storage.CIJob.created_at:type_name -> google.protobuf.Timestamp
2520+ 25, // 17: ermite.storage.CIJob.started_at:type_name -> google.protobuf.Timestamp
2521+ 25, // 18: ermite.storage.CIJob.completed_at:type_name -> google.protobuf.Timestamp
2522+ 22, // 19: ermite.storage.CIJob.outputs:type_name -> ermite.storage.CIJob.OutputsEntry
2523+ 25, // 20: ermite.storage.CIJob.ready_at:type_name -> google.protobuf.Timestamp
2524+ 25, // 21: ermite.storage.CIJob.assigned_at:type_name -> google.protobuf.Timestamp
2525+ 25, // 22: ermite.storage.CIJob.acked_at:type_name -> google.protobuf.Timestamp
2526+ 25, // 23: ermite.storage.CIJob.token_valid_until:type_name -> google.protobuf.Timestamp
2527+ 5, // 24: ermite.storage.CIJob.cancel_reason:type_name -> ermite.storage.CICancelReason
2528+ 25, // 25: ermite.storage.CIJob.cancel_requested_at:type_name -> google.protobuf.Timestamp
2529+ 23, // 26: ermite.storage.CIJob.permissions:type_name -> ermite.storage.CIJob.PermissionsEntry
2530+ 18, // 27: ermite.storage.CIJob.concurrency:type_name -> ermite.storage.CIConcurrency
2531+ 0, // 28: ermite.storage.CICheck.status:type_name -> ermite.storage.CIStatus
2532+ 1, // 29: ermite.storage.CICheck.conclusion:type_name -> ermite.storage.CIConclusion
2533+ 25, // 30: ermite.storage.CICheck.created_at:type_name -> google.protobuf.Timestamp
2534+ 25, // 31: ermite.storage.CICheck.started_at:type_name -> google.protobuf.Timestamp
2535+ 25, // 32: ermite.storage.CICheck.completed_at:type_name -> google.protobuf.Timestamp
2536+ 4, // 33: ermite.storage.CIRunner.isolation:type_name -> ermite.storage.CIIsolation
2537+ 25, // 34: ermite.storage.CIRunner.registered_at:type_name -> google.protobuf.Timestamp
2538+ 25, // 35: ermite.storage.CIRepoToken.expires_at:type_name -> google.protobuf.Timestamp
2539+ 25, // 36: ermite.storage.CIRepoToken.created_at:type_name -> google.protobuf.Timestamp
2540+ 24, // 37: ermite.storage.CIRepoToken.permissions:type_name -> ermite.storage.CIRepoToken.PermissionsEntry
2541+ 6, // 38: ermite.storage.CIAnnotation.level:type_name -> ermite.storage.CIAnnotationLevel
2542+ 16, // 39: ermite.storage.CIStepAnnotations.annotations:type_name -> ermite.storage.CIAnnotation
2543+ 7, // 40: ermite.storage.CIConcurrency.state:type_name -> ermite.storage.CIConcurrencyState
2544+ 19, // 41: ermite.storage.CIConcurrencyRecord.running:type_name -> ermite.storage.CIConcurrencyEntry
2545+ 19, // 42: ermite.storage.CIConcurrencyRecord.pending:type_name -> ermite.storage.CIConcurrencyEntry
2546+ 19, // 43: ermite.storage.CIConcurrencyMark.entry:type_name -> ermite.storage.CIConcurrencyEntry
2547+ 7, // 44: ermite.storage.CIConcurrencyMark.state:type_name -> ermite.storage.CIConcurrencyState
2548+ 45, // [45:45] is the sub-list for method output_type
2549+ 45, // [45:45] is the sub-list for method input_type
2550+ 45, // [45:45] is the sub-list for extension type_name
2551+ 45, // [45:45] is the sub-list for extension extendee
2552+ 0, // [0:45] is the sub-list for field type_name
21412553 }
21422554
21432555 func init() { file_ermite_storage_ci_proto_init() }
@@ -2150,8 +2562,8 @@ func file_ermite_storage_ci_proto_init() {
21502562 File: protoimpl.DescBuilder{
21512563 GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
21522564 RawDescriptor: unsafe.Slice(unsafe.StringData(file_ermite_storage_ci_proto_rawDesc), len(file_ermite_storage_ci_proto_rawDesc)),
2153- NumEnums: 7,
2154- NumMessages: 13,
2565+ NumEnums: 8,
2566+ NumMessages: 17,
21552567 NumExtensions: 0,
21562568 NumServices: 0,
21572569 },
@@ -36,6 +36,8 @@ const (
36 CIStatus_CI_STATUS_QUEUED CIStatus = 136 CIStatus_CI_STATUS_QUEUED CIStatus = 1
37 CIStatus_CI_STATUS_IN_PROGRESS CIStatus = 237 CIStatus_CI_STATUS_IN_PROGRESS CIStatus = 2
38 CIStatus_CI_STATUS_COMPLETED CIStatus = 338 CIStatus_CI_STATUS_COMPLETED CIStatus = 3
39+ // Held back by a concurrency group (jobs and checks only).
40+ CIStatus_CI_STATUS_PENDING CIStatus = 4
39 )41 )
40 42
41 // Enum value maps for CIStatus.43 // Enum value maps for CIStatus.
@@ -45,12 +47,14 @@ var (
45 1: "CI_STATUS_QUEUED",47 1: "CI_STATUS_QUEUED",
46 2: "CI_STATUS_IN_PROGRESS",48 2: "CI_STATUS_IN_PROGRESS",
47 3: "CI_STATUS_COMPLETED",49 3: "CI_STATUS_COMPLETED",
50+ 4: "CI_STATUS_PENDING",
48 }51 }
49 CIStatus_value = map[string]int32{52 CIStatus_value = map[string]int32{
50 "CI_STATUS_UNSPECIFIED": 0,53 "CI_STATUS_UNSPECIFIED": 0,
51 "CI_STATUS_QUEUED": 1,54 "CI_STATUS_QUEUED": 1,
52 "CI_STATUS_IN_PROGRESS": 2,55 "CI_STATUS_IN_PROGRESS": 2,
53 "CI_STATUS_COMPLETED": 3,56 "CI_STATUS_COMPLETED": 3,
57+ "CI_STATUS_PENDING": 4,
54 }58 }
55 )59 )
56 60
@@ -422,6 +426,63 @@ func (CIAnnotationLevel) EnumDescriptor() ([]byte, []int) {
422 return file_ermite_storage_ci_proto_rawDescGZIP(), []int{6}426 return file_ermite_storage_ci_proto_rawDescGZIP(), []int{6}
423 }427 }
424 428
429+// CIConcurrencyState is where a run or a job is in its concurrency group.
430+type CIConcurrencyState int32
431+
432+const (
433+ CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED CIConcurrencyState = 0
434+ // Another entry of the group runs: this one waits.
435+ CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING CIConcurrencyState = 1
436+ // This entry is the running one of its group.
437+ CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING CIConcurrencyState = 2
438+ // Only in a CIConcurrencyMark: a newer entry replaced this one, which is owed
439+ // a cancellation and may never acquire the group.
440+ CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED CIConcurrencyState = 3
441+)
442+
443+// Enum value maps for CIConcurrencyState.
444+var (
445+ CIConcurrencyState_name = map[int32]string{
446+ 0: "CI_CONCURRENCY_STATE_UNSPECIFIED",
447+ 1: "CI_CONCURRENCY_STATE_PENDING",
448+ 2: "CI_CONCURRENCY_STATE_HOLDING",
449+ 3: "CI_CONCURRENCY_STATE_REPLACED",
450+ }
451+ CIConcurrencyState_value = map[string]int32{
452+ "CI_CONCURRENCY_STATE_UNSPECIFIED": 0,
453+ "CI_CONCURRENCY_STATE_PENDING": 1,
454+ "CI_CONCURRENCY_STATE_HOLDING": 2,
455+ "CI_CONCURRENCY_STATE_REPLACED": 3,
456+ }
457+)
458+
459+func (x CIConcurrencyState) Enum() *CIConcurrencyState {
460+ p := new(CIConcurrencyState)
461+ *p = x
462+ return p
463+}
464+
465+func (x CIConcurrencyState) String() string {
466+ return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
467+}
468+
469+func (CIConcurrencyState) Descriptor() protoreflect.EnumDescriptor {
470+ return file_ermite_storage_ci_proto_enumTypes[7].Descriptor()
471+}
472+
473+func (CIConcurrencyState) Type() protoreflect.EnumType {
474+ return &file_ermite_storage_ci_proto_enumTypes[7]
475+}
476+
477+func (x CIConcurrencyState) Number() protoreflect.EnumNumber {
478+ return protoreflect.EnumNumber(x)
479+}
480+
481+// Deprecated: Use CIConcurrencyState.Descriptor instead.
482+func (CIConcurrencyState) EnumDescriptor() ([]byte, []int) {
483+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{7}
484+}
485+
425 // CIRun is stored under "run/<repo>/<id:020d>".486 // CIRun is stored under "run/<repo>/<id:020d>".
426 type CIRun struct {487 type CIRun struct {
427 state protoimpl.MessageState `protogen:"open.v1"`488 state protoimpl.MessageState `protogen:"open.v1"`
@@ -450,7 +511,9 @@ type CIRun struct {
450 Name string `protobuf:"bytes,22,opt,name=name,proto3" json:"name,omitempty"`511 Name string `protobuf:"bytes,22,opt,name=name,proto3" json:"name,omitempty"`
451 // trusted: the actor of the run is the owner, so its jobs get secrets. False512 // trusted: the actor of the run is the owner, so its jobs get secrets. False
452 // in runs stored before it existed (the safe reading).513 // in runs stored before it existed (the safe reading).
453- Trusted bool `protobuf:"varint,23,opt,name=trusted,proto3" json:"trusted,omitempty"`514+ Trusted bool `protobuf:"varint,23,opt,name=trusted,proto3" json:"trusted,omitempty"`
515+ // The evaluated workflow-level concurrency of the attempt.
516+ Concurrency *CIConcurrency `protobuf:"bytes,24,opt,name=concurrency,proto3" json:"concurrency,omitempty"`
454 unknownFields protoimpl.UnknownFields517 unknownFields protoimpl.UnknownFields
455 sizeCache protoimpl.SizeCache518 sizeCache protoimpl.SizeCache
456 }519 }
@@ -632,6 +695,13 @@ func (x *CIRun) GetTrusted() bool {
632 return false695 return false
633 }696 }
634 697
698+func (x *CIRun) GetConcurrency() *CIConcurrency {
699+ if x != nil {
700+ return x.Concurrency
701+ }
702+ return nil
703+}
704+
635 // CIRunSource is stored under "runsrc/<run id:020d>": what a run was planned705 // CIRunSource is stored under "runsrc/<run id:020d>": what a run was planned
636 // from, kept apart so that listing runs does not decode it.706 // from, kept apart so that listing runs does not decode it.
637 type CIRunSource struct {707 type CIRunSource struct {
@@ -900,7 +970,10 @@ type CIJob struct {
900 // The effective permissions of the job's repository token (D3), computed when970 // The effective permissions of the job's repository token (D3), computed when
901 // the job is planned: permission key to "read", "write" or "none". Absent on971 // the job is planned: permission key to "read", "write" or "none". Absent on
902 // a job planned before the field existed: the restricted default.972 // a job planned before the field existed: the restricted default.
903- Permissions map[string]string `protobuf:"bytes,42,rep,name=permissions,proto3" json:"permissions,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`973+ Permissions map[string]string `protobuf:"bytes,42,rep,name=permissions,proto3" json:"permissions,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
974+ // The job's concurrency group (its own, else the one of its run) and where
975+ // it is in it.
976+ Concurrency *CIConcurrency `protobuf:"bytes,43,opt,name=concurrency,proto3" json:"concurrency,omitempty"`
904 unknownFields protoimpl.UnknownFields977 unknownFields protoimpl.UnknownFields
905 sizeCache protoimpl.SizeCache978 sizeCache protoimpl.SizeCache
906 }979 }
@@ -1222,6 +1295,13 @@ func (x *CIJob) GetPermissions() map[string]string {
1222 return nil1295 return nil
1223 }1296 }
1224 1297
1298+func (x *CIJob) GetConcurrency() *CIConcurrency {
1299+ if x != nil {
1300+ return x.Concurrency
1301+ }
1302+ return nil
1303+}
1304+
1225 // CICheck is stored under "check/<repo>/<sha>/<id>": the id of its job, or1305 // CICheck is stored under "check/<repo>/<sha>/<id>": the id of its job, or
1226 // "run-<run id>" for a run that failed without a job (invalid workflow file).1306 // "run-<run id>" for a run that failed without a job (invalid workflow file).
1227 type CICheck struct {1307 type CICheck struct {
@@ -1833,11 +1913,303 @@ func (x *CIStepAnnotations) GetAnnotations() []*CIAnnotation {
1833 return nil1913 return nil
1834 }1914 }
1835 1915
1916+// CIConcurrency is the evaluated "concurrency:" of a run or a job, kept on its
1917+// record (CIRun.concurrency, CIJob.concurrency).
1918+type CIConcurrency struct {
1919+ state protoimpl.MessageState `protogen:"open.v1"`
1920+ Group string `protobuf:"bytes,1,opt,name=group,proto3" json:"group,omitempty"`
1921+ Cancel bool `protobuf:"varint,2,opt,name=cancel,proto3" json:"cancel,omitempty"`
1922+ State CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
1923+ // evaluated: the expressions were evaluated, whatever the result.
1924+ Evaluated bool `protobuf:"varint,4,opt,name=evaluated,proto3" json:"evaluated,omitempty"`
1925+ // from_run: a job that shows the group of its run and holds nothing itself.
1926+ FromRun bool `protobuf:"varint,5,opt,name=from_run,json=fromRun,proto3" json:"from_run,omitempty"`
1927+ // preempted: cancelled because a newer entry of the group asked for it; the
1928+ // job ends with GitHub's annotation.
1929+ Preempted bool `protobuf:"varint,6,opt,name=preempted,proto3" json:"preempted,omitempty"`
1930+ unknownFields protoimpl.UnknownFields
1931+ sizeCache protoimpl.SizeCache
1932+}
1933+
1934+func (x *CIConcurrency) Reset() {
1935+ *x = CIConcurrency{}
1936+ mi := &file_ermite_storage_ci_proto_msgTypes[10]
1937+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
1938+ ms.StoreMessageInfo(mi)
1939+}
1940+
1941+func (x *CIConcurrency) String() string {
1942+ return protoimpl.X.MessageStringOf(x)
1943+}
1944+
1945+func (*CIConcurrency) ProtoMessage() {}
1946+
1947+func (x *CIConcurrency) ProtoReflect() protoreflect.Message {
1948+ mi := &file_ermite_storage_ci_proto_msgTypes[10]
1949+ if x != nil {
1950+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
1951+ if ms.LoadMessageInfo() == nil {
1952+ ms.StoreMessageInfo(mi)
1953+ }
1954+ return ms
1955+ }
1956+ return mi.MessageOf(x)
1957+}
1958+
1959+// Deprecated: Use CIConcurrency.ProtoReflect.Descriptor instead.
1960+func (*CIConcurrency) Descriptor() ([]byte, []int) {
1961+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{10}
1962+}
1963+
1964+func (x *CIConcurrency) GetGroup() string {
1965+ if x != nil {
1966+ return x.Group
1967+ }
1968+ return ""
1969+}
1970+
1971+func (x *CIConcurrency) GetCancel() bool {
1972+ if x != nil {
1973+ return x.Cancel
1974+ }
1975+ return false
1976+}
1977+
1978+func (x *CIConcurrency) GetState() CIConcurrencyState {
1979+ if x != nil {
1980+ return x.State
1981+ }
1982+ return CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED
1983+}
1984+
1985+func (x *CIConcurrency) GetEvaluated() bool {
1986+ if x != nil {
1987+ return x.Evaluated
1988+ }
1989+ return false
1990+}
1991+
1992+func (x *CIConcurrency) GetFromRun() bool {
1993+ if x != nil {
1994+ return x.FromRun
1995+ }
1996+ return false
1997+}
1998+
1999+func (x *CIConcurrency) GetPreempted() bool {
2000+ if x != nil {
2001+ return x.Preempted
2002+ }
2003+ return false
2004+}
2005+
2006+// CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.
2007+type CIConcurrencyEntry struct {
2008+ state protoimpl.MessageState `protogen:"open.v1"`
2009+ RunId int64 `protobuf:"varint,1,opt,name=run_id,json=runId,proto3" json:"run_id,omitempty"`
2010+ Attempt int32 `protobuf:"varint,2,opt,name=attempt,proto3" json:"attempt,omitempty"`
2011+ JobId string `protobuf:"bytes,3,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
2012+ Cancel bool `protobuf:"varint,4,opt,name=cancel,proto3" json:"cancel,omitempty"`
2013+ // canceling: a waiting entry with cancel-in-progress asked this running one
2014+ // to stop; the entry stays until it has ended.
2015+ Canceling bool `protobuf:"varint,5,opt,name=canceling,proto3" json:"canceling,omitempty"`
2016+ unknownFields protoimpl.UnknownFields
2017+ sizeCache protoimpl.SizeCache
2018+}
2019+
2020+func (x *CIConcurrencyEntry) Reset() {
2021+ *x = CIConcurrencyEntry{}
2022+ mi := &file_ermite_storage_ci_proto_msgTypes[11]
2023+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2024+ ms.StoreMessageInfo(mi)
2025+}
2026+
2027+func (x *CIConcurrencyEntry) String() string {
2028+ return protoimpl.X.MessageStringOf(x)
2029+}
2030+
2031+func (*CIConcurrencyEntry) ProtoMessage() {}
2032+
2033+func (x *CIConcurrencyEntry) ProtoReflect() protoreflect.Message {
2034+ mi := &file_ermite_storage_ci_proto_msgTypes[11]
2035+ if x != nil {
2036+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2037+ if ms.LoadMessageInfo() == nil {
2038+ ms.StoreMessageInfo(mi)
2039+ }
2040+ return ms
2041+ }
2042+ return mi.MessageOf(x)
2043+}
2044+
2045+// Deprecated: Use CIConcurrencyEntry.ProtoReflect.Descriptor instead.
2046+func (*CIConcurrencyEntry) Descriptor() ([]byte, []int) {
2047+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{11}
2048+}
2049+
2050+func (x *CIConcurrencyEntry) GetRunId() int64 {
2051+ if x != nil {
2052+ return x.RunId
2053+ }
2054+ return 0
2055+}
2056+
2057+func (x *CIConcurrencyEntry) GetAttempt() int32 {
2058+ if x != nil {
2059+ return x.Attempt
2060+ }
2061+ return 0
2062+}
2063+
2064+func (x *CIConcurrencyEntry) GetJobId() string {
2065+ if x != nil {
2066+ return x.JobId
2067+ }
2068+ return ""
2069+}
2070+
2071+func (x *CIConcurrencyEntry) GetCancel() bool {
2072+ if x != nil {
2073+ return x.Cancel
2074+ }
2075+ return false
2076+}
2077+
2078+func (x *CIConcurrencyEntry) GetCanceling() bool {
2079+ if x != nil {
2080+ return x.Canceling
2081+ }
2082+ return false
2083+}
2084+
2085+// CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
2086+// running entry and the pending one of a group (queue: single). A group with
2087+// neither has no key.
2088+type CIConcurrencyRecord struct {
2089+ state protoimpl.MessageState `protogen:"open.v1"`
2090+ Running *CIConcurrencyEntry `protobuf:"bytes,1,opt,name=running,proto3" json:"running,omitempty"`
2091+ Pending *CIConcurrencyEntry `protobuf:"bytes,2,opt,name=pending,proto3" json:"pending,omitempty"`
2092+ unknownFields protoimpl.UnknownFields
2093+ sizeCache protoimpl.SizeCache
2094+}
2095+
2096+func (x *CIConcurrencyRecord) Reset() {
2097+ *x = CIConcurrencyRecord{}
2098+ mi := &file_ermite_storage_ci_proto_msgTypes[12]
2099+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2100+ ms.StoreMessageInfo(mi)
2101+}
2102+
2103+func (x *CIConcurrencyRecord) String() string {
2104+ return protoimpl.X.MessageStringOf(x)
2105+}
2106+
2107+func (*CIConcurrencyRecord) ProtoMessage() {}
2108+
2109+func (x *CIConcurrencyRecord) ProtoReflect() protoreflect.Message {
2110+ mi := &file_ermite_storage_ci_proto_msgTypes[12]
2111+ if x != nil {
2112+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2113+ if ms.LoadMessageInfo() == nil {
2114+ ms.StoreMessageInfo(mi)
2115+ }
2116+ return ms
2117+ }
2118+ return mi.MessageOf(x)
2119+}
2120+
2121+// Deprecated: Use CIConcurrencyRecord.ProtoReflect.Descriptor instead.
2122+func (*CIConcurrencyRecord) Descriptor() ([]byte, []int) {
2123+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{12}
2124+}
2125+
2126+func (x *CIConcurrencyRecord) GetRunning() *CIConcurrencyEntry {
2127+ if x != nil {
2128+ return x.Running
2129+ }
2130+ return nil
2131+}
2132+
2133+func (x *CIConcurrencyRecord) GetPending() *CIConcurrencyEntry {
2134+ if x != nil {
2135+ return x.Pending
2136+ }
2137+ return nil
2138+}
2139+
2140+// CIConcurrencyMark is stored under
2141+// "concurmark/<repo>/<escaped key(group)>/<run:020d>/<attempt:010d>/<job id or ->":
2142+// where an entry stands in its group. It is written in the same batch as the
2143+// group record ("concur/..."), so that an entry is marked pending or holding if
2144+// and only if the record names it; an entry the record stops naming because a
2145+// newer one replaced it is marked replaced in that very batch. The mark is
2146+// removed with the entry (it ended, was released, or is gone).
2147+type CIConcurrencyMark struct {
2148+ state protoimpl.MessageState `protogen:"open.v1"`
2149+ Entry *CIConcurrencyEntry `protobuf:"bytes,1,opt,name=entry,proto3" json:"entry,omitempty"`
2150+ // group as the newest request wrote it, for the cancel annotation.
2151+ Group string `protobuf:"bytes,2,opt,name=group,proto3" json:"group,omitempty"`
2152+ State CIConcurrencyState `protobuf:"varint,3,opt,name=state,proto3,enum=ermite.storage.CIConcurrencyState" json:"state,omitempty"`
2153+ unknownFields protoimpl.UnknownFields
2154+ sizeCache protoimpl.SizeCache
2155+}
2156+
2157+func (x *CIConcurrencyMark) Reset() {
2158+ *x = CIConcurrencyMark{}
2159+ mi := &file_ermite_storage_ci_proto_msgTypes[13]
2160+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2161+ ms.StoreMessageInfo(mi)
2162+}
2163+
2164+func (x *CIConcurrencyMark) String() string {
2165+ return protoimpl.X.MessageStringOf(x)
2166+}
2167+
2168+func (*CIConcurrencyMark) ProtoMessage() {}
2169+
2170+func (x *CIConcurrencyMark) ProtoReflect() protoreflect.Message {
2171+ mi := &file_ermite_storage_ci_proto_msgTypes[13]
2172+ if x != nil {
2173+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
2174+ if ms.LoadMessageInfo() == nil {
2175+ ms.StoreMessageInfo(mi)
2176+ }
2177+ return ms
2178+ }
2179+ return mi.MessageOf(x)
2180+}
2181+
2182+// Deprecated: Use CIConcurrencyMark.ProtoReflect.Descriptor instead.
2183+func (*CIConcurrencyMark) Descriptor() ([]byte, []int) {
2184+ return file_ermite_storage_ci_proto_rawDescGZIP(), []int{13}
2185+}
2186+
2187+func (x *CIConcurrencyMark) GetEntry() *CIConcurrencyEntry {
2188+ if x != nil {
2189+ return x.Entry
2190+ }
2191+ return nil
2192+}
2193+
2194+func (x *CIConcurrencyMark) GetGroup() string {
2195+ if x != nil {
2196+ return x.Group
2197+ }
2198+ return ""
2199+}
2200+
2201+func (x *CIConcurrencyMark) GetState() CIConcurrencyState {
2202+ if x != nil {
2203+ return x.State
2204+ }
2205+ return CIConcurrencyState_CI_CONCURRENCY_STATE_UNSPECIFIED
2206+}
2207+
1836 var File_ermite_storage_ci_proto protoreflect.FileDescriptor2208 var File_ermite_storage_ci_proto protoreflect.FileDescriptor
1837 2209
1838 const file_ermite_storage_ci_proto_rawDesc = "" +2210 const file_ermite_storage_ci_proto_rawDesc = "" +
1839 "\n" +2211 "\n" +
1840- "\x17ermite/storage/ci.proto\x12\x0eermite.storage\x1a\x1fgoogle/protobuf/timestamp.proto\"\xf9\x05\n" +2212+ "\x17ermite/storage/ci.proto\x12\x0eermite.storage\x1a\x1fgoogle/protobuf/timestamp.proto\"\xba\x06\n" +
1841 "\x05CIRun\x12\x12\n" +2213 "\x05CIRun\x12\x12\n" +
1842 "\x04repo\x18\x01 \x01(\tR\x04repo\x12\x0e\n" +2214 "\x04repo\x18\x01 \x01(\tR\x04repo\x12\x0e\n" +
1843 "\x02id\x18\x02 \x01(\x03R\x02id\x12\x16\n" +2215 "\x02id\x18\x02 \x01(\x03R\x02id\x12\x16\n" +
@@ -1864,7 +2236,8 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
1864 "\x10cancel_requested\x18\x14 \x01(\bR\x0fcancelRequested\x12J\n" +2236 "\x10cancel_requested\x18\x14 \x01(\bR\x0fcancelRequested\x12J\n" +
1865 "\x13cancel_requested_at\x18\x15 \x01(\v2\x1a.google.protobuf.TimestampR\x11cancelRequestedAt\x12\x12\n" +2237 "\x13cancel_requested_at\x18\x15 \x01(\v2\x1a.google.protobuf.TimestampR\x11cancelRequestedAt\x12\x12\n" +
1866 "\x04name\x18\x16 \x01(\tR\x04name\x12\x18\n" +2238 "\x04name\x18\x16 \x01(\tR\x04name\x12\x18\n" +
1867- "\atrusted\x18\x17 \x01(\bR\atrustedJ\x04\b\x12\x10\x13J\x04\b\x13\x10\x14\"\xc7\x01\n" +2239+ "\atrusted\x18\x17 \x01(\bR\atrusted\x12?\n" +
2240+ "\vconcurrency\x18\x18 \x01(\v2\x1d.ermite.storage.CIConcurrencyR\vconcurrencyJ\x04\b\x12\x10\x13J\x04\b\x13\x10\x14\"\xc7\x01\n" +
1868 "\vCIRunSource\x12\x15\n" +2241 "\vCIRunSource\x12\x15\n" +
1869 "\x06run_id\x18\x04 \x01(\x03R\x05runId\x12#\n" +2242 "\x06run_id\x18\x04 \x01(\x03R\x05runId\x12#\n" +
1870 "\rworkflow_yaml\x18\x01 \x01(\tR\fworkflowYaml\x12\x1f\n" +2243 "\rworkflow_yaml\x18\x01 \x01(\tR\fworkflowYaml\x12\x1f\n" +
@@ -1887,7 +2260,7 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
1887 "\tlog_start\x18\a \x01(\x03R\blogStart\x12\x1b\n" +2260 "\tlog_start\x18\a \x01(\x03R\blogStart\x12\x1b\n" +
1888 "\thas_lines\x18\t \x01(\bR\bhasLines\x126\n" +2261 "\thas_lines\x18\t \x01(\bR\bhasLines\x126\n" +
1889 "\aoutcome\x18\v \x01(\x0e2\x1c.ermite.storage.CIConclusionR\aoutcomeJ\x04\b\b\x10\tJ\x04\b\n" +2262 "\aoutcome\x18\v \x01(\x0e2\x1c.ermite.storage.CIConclusionR\aoutcomeJ\x04\b\b\x10\tJ\x04\b\n" +
1890- "\x10\v\"\x92\x0e\n" +2263+ "\x10\v\"\xd3\x0e\n" +
1891 "\x05CIJob\x12\x0e\n" +2264 "\x05CIJob\x12\x0e\n" +
1892 "\x02id\x18\x01 \x01(\tR\x02id\x12\x15\n" +2265 "\x02id\x18\x01 \x01(\tR\x02id\x12\x15\n" +
1893 "\x06run_id\x18\x02 \x01(\x03R\x05runId\x12\x18\n" +2266 "\x06run_id\x18\x02 \x01(\x03R\x05runId\x12\x18\n" +
@@ -1940,7 +2313,8 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
1940 "\rcontexts_json\x18' \x01(\tR\fcontextsJson\x12*\n" +2313 "\rcontexts_json\x18' \x01(\tR\fcontextsJson\x12*\n" +
1941 "\x11continue_on_error\x18( \x01(\bR\x0fcontinueOnError\x12 \n" +2314 "\x11continue_on_error\x18( \x01(\bR\x0fcontinueOnError\x12 \n" +
1942 "\venvironment\x18) \x01(\tR\venvironment\x12H\n" +2315 "\venvironment\x18) \x01(\tR\venvironment\x12H\n" +
1943- "\vpermissions\x18* \x03(\v2&.ermite.storage.CIJob.PermissionsEntryR\vpermissions\x1a:\n" +2316+ "\vpermissions\x18* \x03(\v2&.ermite.storage.CIJob.PermissionsEntryR\vpermissions\x12?\n" +
2317+ "\vconcurrency\x18+ \x01(\v2\x1d.ermite.storage.CIConcurrencyR\vconcurrency\x1a:\n" +
1944 "\fOutputsEntry\x12\x10\n" +2318 "\fOutputsEntry\x12\x10\n" +
1945 "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +2319 "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
1946 "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1a>\n" +2320 "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1a>\n" +
@@ -2013,12 +2387,33 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
2013 "\n" +2387 "\n" +
2014 "end_column\x18\b \x01(\x05R\tendColumn\"S\n" +2388 "end_column\x18\b \x01(\x05R\tendColumn\"S\n" +
2015 "\x11CIStepAnnotations\x12>\n" +2389 "\x11CIStepAnnotations\x12>\n" +
2016- "\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations*o\n" +2390+ "\vannotations\x18\x01 \x03(\v2\x1c.ermite.storage.CIAnnotationR\vannotations\"\xce\x01\n" +
2391+ "\rCIConcurrency\x12\x14\n" +
2392+ "\x05group\x18\x01 \x01(\tR\x05group\x12\x16\n" +
2393+ "\x06cancel\x18\x02 \x01(\bR\x06cancel\x128\n" +
2394+ "\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state\x12\x1c\n" +
2395+ "\tevaluated\x18\x04 \x01(\bR\tevaluated\x12\x19\n" +
2396+ "\bfrom_run\x18\x05 \x01(\bR\afromRun\x12\x1c\n" +
2397+ "\tpreempted\x18\x06 \x01(\bR\tpreempted\"\x92\x01\n" +
2398+ "\x12CIConcurrencyEntry\x12\x15\n" +
2399+ "\x06run_id\x18\x01 \x01(\x03R\x05runId\x12\x18\n" +
2400+ "\aattempt\x18\x02 \x01(\x05R\aattempt\x12\x15\n" +
2401+ "\x06job_id\x18\x03 \x01(\tR\x05jobId\x12\x16\n" +
2402+ "\x06cancel\x18\x04 \x01(\bR\x06cancel\x12\x1c\n" +
2403+ "\tcanceling\x18\x05 \x01(\bR\tcanceling\"\x97\x01\n" +
2404+ "\x13CIConcurrencyRecord\x12<\n" +
2405+ "\arunning\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\arunning\x12<\n" +
2406+ "\apending\x18\x02 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\apendingJ\x04\b\x03\x10\x04\"\x9d\x01\n" +
2407+ "\x11CIConcurrencyMark\x128\n" +
2408+ "\x05entry\x18\x01 \x01(\v2\".ermite.storage.CIConcurrencyEntryR\x05entry\x12\x14\n" +
2409+ "\x05group\x18\x02 \x01(\tR\x05group\x128\n" +
2410+ "\x05state\x18\x03 \x01(\x0e2\".ermite.storage.CIConcurrencyStateR\x05state*\x86\x01\n" +
2017 "\bCIStatus\x12\x19\n" +2411 "\bCIStatus\x12\x19\n" +
2018 "\x15CI_STATUS_UNSPECIFIED\x10\x00\x12\x14\n" +2412 "\x15CI_STATUS_UNSPECIFIED\x10\x00\x12\x14\n" +
2019 "\x10CI_STATUS_QUEUED\x10\x01\x12\x19\n" +2413 "\x10CI_STATUS_QUEUED\x10\x01\x12\x19\n" +
2020 "\x15CI_STATUS_IN_PROGRESS\x10\x02\x12\x17\n" +2414 "\x15CI_STATUS_IN_PROGRESS\x10\x02\x12\x17\n" +
2021- "\x13CI_STATUS_COMPLETED\x10\x03*\xf6\x01\n" +2415+ "\x13CI_STATUS_COMPLETED\x10\x03\x12\x15\n" +
2416+ "\x11CI_STATUS_PENDING\x10\x04*\xf6\x01\n" +
2022 "\fCIConclusion\x12\x1d\n" +2417 "\fCIConclusion\x12\x1d\n" +
2023 "\x19CI_CONCLUSION_UNSPECIFIED\x10\x00\x12\x19\n" +2418 "\x19CI_CONCLUSION_UNSPECIFIED\x10\x00\x12\x19\n" +
2024 "\x15CI_CONCLUSION_SUCCESS\x10\x01\x12\x19\n" +2419 "\x15CI_CONCLUSION_SUCCESS\x10\x01\x12\x19\n" +
@@ -2055,7 +2450,12 @@ const file_ermite_storage_ci_proto_rawDesc = "" +
2055 "\x1fCI_ANNOTATION_LEVEL_UNSPECIFIED\x10\x00\x12\x1e\n" +2450 "\x1fCI_ANNOTATION_LEVEL_UNSPECIFIED\x10\x00\x12\x1e\n" +
2056 "\x1aCI_ANNOTATION_LEVEL_NOTICE\x10\x01\x12\x1f\n" +2451 "\x1aCI_ANNOTATION_LEVEL_NOTICE\x10\x01\x12\x1f\n" +
2057 "\x1bCI_ANNOTATION_LEVEL_WARNING\x10\x02\x12\x1f\n" +2452 "\x1bCI_ANNOTATION_LEVEL_WARNING\x10\x02\x12\x1f\n" +
2058- "\x1bCI_ANNOTATION_LEVEL_FAILURE\x10\x03B/Z-bnema.dev/ermite/gen/ermite/storage;storagepbb\x06proto3"2453+ "\x1bCI_ANNOTATION_LEVEL_FAILURE\x10\x03*\xa1\x01\n" +
2454+ "\x12CIConcurrencyState\x12$\n" +
2455+ " CI_CONCURRENCY_STATE_UNSPECIFIED\x10\x00\x12 \n" +
2456+ "\x1cCI_CONCURRENCY_STATE_PENDING\x10\x01\x12 \n" +
2457+ "\x1cCI_CONCURRENCY_STATE_HOLDING\x10\x02\x12!\n" +
2458+ "\x1dCI_CONCURRENCY_STATE_REPLACED\x10\x03B/Z-bnema.dev/ermite/gen/ermite/storage;storagepbb\x06proto3"
2059 2459
2060 var (2460 var (
2061 file_ermite_storage_ci_proto_rawDescOnce sync.Once2461 file_ermite_storage_ci_proto_rawDescOnce sync.Once
@@ -2069,8 +2469,8 @@ func file_ermite_storage_ci_proto_rawDescGZIP() []byte {
2069 return file_ermite_storage_ci_proto_rawDescData2469 return file_ermite_storage_ci_proto_rawDescData
2070 }2470 }
2071 2471
2072-var file_ermite_storage_ci_proto_enumTypes = make([]protoimpl.EnumInfo, 7)2472+var file_ermite_storage_ci_proto_enumTypes = make([]protoimpl.EnumInfo, 8)
2073-var file_ermite_storage_ci_proto_msgTypes = make([]protoimpl.MessageInfo, 13)2473+var file_ermite_storage_ci_proto_msgTypes = make([]protoimpl.MessageInfo, 17)
2074 var file_ermite_storage_ci_proto_goTypes = []any{2474 var file_ermite_storage_ci_proto_goTypes = []any{
2075 (CIStatus)(0), // 0: ermite.storage.CIStatus2475 (CIStatus)(0), // 0: ermite.storage.CIStatus
2076 (CIConclusion)(0), // 1: ermite.storage.CIConclusion2476 (CIConclusion)(0), // 1: ermite.storage.CIConclusion
@@ -2079,65 +2479,77 @@ var file_ermite_storage_ci_proto_goTypes = []any{
2079 (CIIsolation)(0), // 4: ermite.storage.CIIsolation2479 (CIIsolation)(0), // 4: ermite.storage.CIIsolation
2080 (CICancelReason)(0), // 5: ermite.storage.CICancelReason2480 (CICancelReason)(0), // 5: ermite.storage.CICancelReason
2081 (CIAnnotationLevel)(0), // 6: ermite.storage.CIAnnotationLevel2481 (CIAnnotationLevel)(0), // 6: ermite.storage.CIAnnotationLevel
2082- (*CIRun)(nil), // 7: ermite.storage.CIRun2482+ (CIConcurrencyState)(0), // 7: ermite.storage.CIConcurrencyState
2083- (*CIRunSource)(nil), // 8: ermite.storage.CIRunSource2483+ (*CIRun)(nil), // 8: ermite.storage.CIRun
2084- (*CIStep)(nil), // 9: ermite.storage.CIStep2484+ (*CIRunSource)(nil), // 9: ermite.storage.CIRunSource
2085- (*CIJob)(nil), // 10: ermite.storage.CIJob2485+ (*CIStep)(nil), // 10: ermite.storage.CIStep
2086- (*CICheck)(nil), // 11: ermite.storage.CICheck2486+ (*CIJob)(nil), // 11: ermite.storage.CIJob
2087- (*CIRunner)(nil), // 12: ermite.storage.CIRunner2487+ (*CICheck)(nil), // 12: ermite.storage.CICheck
2088- (*CIRepoToken)(nil), // 13: ermite.storage.CIRepoToken2488+ (*CIRunner)(nil), // 13: ermite.storage.CIRunner
2089- (*CIActionResolution)(nil), // 14: ermite.storage.CIActionResolution2489+ (*CIRepoToken)(nil), // 14: ermite.storage.CIRepoToken
2090- (*CIAnnotation)(nil), // 15: ermite.storage.CIAnnotation2490+ (*CIActionResolution)(nil), // 15: ermite.storage.CIActionResolution
2091- (*CIStepAnnotations)(nil), // 16: ermite.storage.CIStepAnnotations2491+ (*CIAnnotation)(nil), // 16: ermite.storage.CIAnnotation
2092- nil, // 17: ermite.storage.CIJob.OutputsEntry2492+ (*CIStepAnnotations)(nil), // 17: ermite.storage.CIStepAnnotations
2093- nil, // 18: ermite.storage.CIJob.PermissionsEntry2493+ (*CIConcurrency)(nil), // 18: ermite.storage.CIConcurrency
2094- nil, // 19: ermite.storage.CIRepoToken.PermissionsEntry2494+ (*CIConcurrencyEntry)(nil), // 19: ermite.storage.CIConcurrencyEntry
2095- (*timestamppb.Timestamp)(nil), // 20: google.protobuf.Timestamp2495+ (*CIConcurrencyRecord)(nil), // 20: ermite.storage.CIConcurrencyRecord
2496+ (*CIConcurrencyMark)(nil), // 21: ermite.storage.CIConcurrencyMark
2497+ nil, // 22: ermite.storage.CIJob.OutputsEntry
2498+ nil, // 23: ermite.storage.CIJob.PermissionsEntry
2499+ nil, // 24: ermite.storage.CIRepoToken.PermissionsEntry
2500+ (*timestamppb.Timestamp)(nil), // 25: google.protobuf.Timestamp
2096 }2501 }
2097 var file_ermite_storage_ci_proto_depIdxs = []int32{2502 var file_ermite_storage_ci_proto_depIdxs = []int32{
2098 0, // 0: ermite.storage.CIRun.status:type_name -> ermite.storage.CIStatus2503 0, // 0: ermite.storage.CIRun.status:type_name -> ermite.storage.CIStatus
2099 1, // 1: ermite.storage.CIRun.conclusion:type_name -> ermite.storage.CIConclusion2504 1, // 1: ermite.storage.CIRun.conclusion:type_name -> ermite.storage.CIConclusion
2100- 20, // 2: ermite.storage.CIRun.created_at:type_name -> google.protobuf.Timestamp2505+ 25, // 2: ermite.storage.CIRun.created_at:type_name -> google.protobuf.Timestamp
2101- 20, // 3: ermite.storage.CIRun.started_at:type_name -> google.protobuf.Timestamp2506+ 25, // 3: ermite.storage.CIRun.started_at:type_name -> google.protobuf.Timestamp
2102- 20, // 4: ermite.storage.CIRun.completed_at:type_name -> google.protobuf.Timestamp2507+ 25, // 4: ermite.storage.CIRun.completed_at:type_name -> google.protobuf.Timestamp
2103- 20, // 5: ermite.storage.CIRun.cancel_requested_at:type_name -> google.protobuf.Timestamp2508+ 25, // 5: ermite.storage.CIRun.cancel_requested_at:type_name -> google.protobuf.Timestamp
2104- 2, // 6: ermite.storage.CIStep.state:type_name -> ermite.storage.CIStepState2509+ 18, // 6: ermite.storage.CIRun.concurrency:type_name -> ermite.storage.CIConcurrency
2105- 1, // 7: ermite.storage.CIStep.conclusion:type_name -> ermite.storage.CIConclusion2510+ 2, // 7: ermite.storage.CIStep.state:type_name -> ermite.storage.CIStepState
2106- 20, // 8: ermite.storage.CIStep.started_at:type_name -> google.protobuf.Timestamp2511+ 1, // 8: ermite.storage.CIStep.conclusion:type_name -> ermite.storage.CIConclusion
2107- 20, // 9: ermite.storage.CIStep.completed_at:type_name -> google.protobuf.Timestamp2512+ 25, // 9: ermite.storage.CIStep.started_at:type_name -> google.protobuf.Timestamp
2108- 1, // 10: ermite.storage.CIStep.outcome:type_name -> ermite.storage.CIConclusion2513+ 25, // 10: ermite.storage.CIStep.completed_at:type_name -> google.protobuf.Timestamp
2109- 3, // 11: ermite.storage.CIJob.phase:type_name -> ermite.storage.CIJobPhase2514+ 1, // 11: ermite.storage.CIStep.outcome:type_name -> ermite.storage.CIConclusion
2110- 1, // 12: ermite.storage.CIJob.conclusion:type_name -> ermite.storage.CIConclusion2515+ 3, // 12: ermite.storage.CIJob.phase:type_name -> ermite.storage.CIJobPhase
2111- 4, // 13: ermite.storage.CIJob.isolation:type_name -> ermite.storage.CIIsolation2516+ 1, // 13: ermite.storage.CIJob.conclusion:type_name -> ermite.storage.CIConclusion
2112- 9, // 14: ermite.storage.CIJob.steps:type_name -> ermite.storage.CIStep2517+ 4, // 14: ermite.storage.CIJob.isolation:type_name -> ermite.storage.CIIsolation
2113- 20, // 15: ermite.storage.CIJob.created_at:type_name -> google.protobuf.Timestamp2518+ 10, // 15: ermite.storage.CIJob.steps:type_name -> ermite.storage.CIStep
2114- 20, // 16: ermite.storage.CIJob.started_at:type_name -> google.protobuf.Timestamp2519+ 25, // 16: ermite.storage.CIJob.created_at:type_name -> google.protobuf.Timestamp
2115- 20, // 17: ermite.storage.CIJob.completed_at:type_name -> google.protobuf.Timestamp2520+ 25, // 17: ermite.storage.CIJob.started_at:type_name -> google.protobuf.Timestamp
2116- 17, // 18: ermite.storage.CIJob.outputs:type_name -> ermite.storage.CIJob.OutputsEntry2521+ 25, // 18: ermite.storage.CIJob.completed_at:type_name -> google.protobuf.Timestamp
2117- 20, // 19: ermite.storage.CIJob.ready_at:type_name -> google.protobuf.Timestamp2522+ 22, // 19: ermite.storage.CIJob.outputs:type_name -> ermite.storage.CIJob.OutputsEntry
2118- 20, // 20: ermite.storage.CIJob.assigned_at:type_name -> google.protobuf.Timestamp2523+ 25, // 20: ermite.storage.CIJob.ready_at:type_name -> google.protobuf.Timestamp
2119- 20, // 21: ermite.storage.CIJob.acked_at:type_name -> google.protobuf.Timestamp2524+ 25, // 21: ermite.storage.CIJob.assigned_at:type_name -> google.protobuf.Timestamp
2120- 20, // 22: ermite.storage.CIJob.token_valid_until:type_name -> google.protobuf.Timestamp2525+ 25, // 22: ermite.storage.CIJob.acked_at:type_name -> google.protobuf.Timestamp
2121- 5, // 23: ermite.storage.CIJob.cancel_reason:type_name -> ermite.storage.CICancelReason2526+ 25, // 23: ermite.storage.CIJob.token_valid_until:type_name -> google.protobuf.Timestamp
2122- 20, // 24: ermite.storage.CIJob.cancel_requested_at:type_name -> google.protobuf.Timestamp2527+ 5, // 24: ermite.storage.CIJob.cancel_reason:type_name -> ermite.storage.CICancelReason
2123- 18, // 25: ermite.storage.CIJob.permissions:type_name -> ermite.storage.CIJob.PermissionsEntry2528+ 25, // 25: ermite.storage.CIJob.cancel_requested_at:type_name -> google.protobuf.Timestamp
2124- 0, // 26: ermite.storage.CICheck.status:type_name -> ermite.storage.CIStatus2529+ 23, // 26: ermite.storage.CIJob.permissions:type_name -> ermite.storage.CIJob.PermissionsEntry
2125- 1, // 27: ermite.storage.CICheck.conclusion:type_name -> ermite.storage.CIConclusion2530+ 18, // 27: ermite.storage.CIJob.concurrency:type_name -> ermite.storage.CIConcurrency
2126- 20, // 28: ermite.storage.CICheck.created_at:type_name -> google.protobuf.Timestamp2531+ 0, // 28: ermite.storage.CICheck.status:type_name -> ermite.storage.CIStatus
2127- 20, // 29: ermite.storage.CICheck.started_at:type_name -> google.protobuf.Timestamp2532+ 1, // 29: ermite.storage.CICheck.conclusion:type_name -> ermite.storage.CIConclusion
2128- 20, // 30: ermite.storage.CICheck.completed_at:type_name -> google.protobuf.Timestamp2533+ 25, // 30: ermite.storage.CICheck.created_at:type_name -> google.protobuf.Timestamp
2129- 4, // 31: ermite.storage.CIRunner.isolation:type_name -> ermite.storage.CIIsolation2534+ 25, // 31: ermite.storage.CICheck.started_at:type_name -> google.protobuf.Timestamp
2130- 20, // 32: ermite.storage.CIRunner.registered_at:type_name -> google.protobuf.Timestamp2535+ 25, // 32: ermite.storage.CICheck.completed_at:type_name -> google.protobuf.Timestamp
2131- 20, // 33: ermite.storage.CIRepoToken.expires_at:type_name -> google.protobuf.Timestamp2536+ 4, // 33: ermite.storage.CIRunner.isolation:type_name -> ermite.storage.CIIsolation
2132- 20, // 34: ermite.storage.CIRepoToken.created_at:type_name -> google.protobuf.Timestamp2537+ 25, // 34: ermite.storage.CIRunner.registered_at:type_name -> google.protobuf.Timestamp
2133- 19, // 35: ermite.storage.CIRepoToken.permissions:type_name -> ermite.storage.CIRepoToken.PermissionsEntry2538+ 25, // 35: ermite.storage.CIRepoToken.expires_at:type_name -> google.protobuf.Timestamp
2134- 6, // 36: ermite.storage.CIAnnotation.level:type_name -> ermite.storage.CIAnnotationLevel2539+ 25, // 36: ermite.storage.CIRepoToken.created_at:type_name -> google.protobuf.Timestamp
2135- 15, // 37: ermite.storage.CIStepAnnotations.annotations:type_name -> ermite.storage.CIAnnotation2540+ 24, // 37: ermite.storage.CIRepoToken.permissions:type_name -> ermite.storage.CIRepoToken.PermissionsEntry
2136- 38, // [38:38] is the sub-list for method output_type2541+ 6, // 38: ermite.storage.CIAnnotation.level:type_name -> ermite.storage.CIAnnotationLevel
2137- 38, // [38:38] is the sub-list for method input_type2542+ 16, // 39: ermite.storage.CIStepAnnotations.annotations:type_name -> ermite.storage.CIAnnotation
2138- 38, // [38:38] is the sub-list for extension type_name2543+ 7, // 40: ermite.storage.CIConcurrency.state:type_name -> ermite.storage.CIConcurrencyState
2139- 38, // [38:38] is the sub-list for extension extendee2544+ 19, // 41: ermite.storage.CIConcurrencyRecord.running:type_name -> ermite.storage.CIConcurrencyEntry
2140- 0, // [0:38] is the sub-list for field type_name2545+ 19, // 42: ermite.storage.CIConcurrencyRecord.pending:type_name -> ermite.storage.CIConcurrencyEntry
2546+ 19, // 43: ermite.storage.CIConcurrencyMark.entry:type_name -> ermite.storage.CIConcurrencyEntry
2547+ 7, // 44: ermite.storage.CIConcurrencyMark.state:type_name -> ermite.storage.CIConcurrencyState
2548+ 45, // [45:45] is the sub-list for method output_type
2549+ 45, // [45:45] is the sub-list for method input_type
2550+ 45, // [45:45] is the sub-list for extension type_name
2551+ 45, // [45:45] is the sub-list for extension extendee
2552+ 0, // [0:45] is the sub-list for field type_name
2141 }2553 }
2142 2554
2143 func init() { file_ermite_storage_ci_proto_init() }2555 func init() { file_ermite_storage_ci_proto_init() }
@@ -2150,8 +2562,8 @@ func file_ermite_storage_ci_proto_init() {
2150 File: protoimpl.DescBuilder{2562 File: protoimpl.DescBuilder{
2151 GoPackagePath: reflect.TypeOf(x{}).PkgPath(),2563 GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
2152 RawDescriptor: unsafe.Slice(unsafe.StringData(file_ermite_storage_ci_proto_rawDesc), len(file_ermite_storage_ci_proto_rawDesc)),2564 RawDescriptor: unsafe.Slice(unsafe.StringData(file_ermite_storage_ci_proto_rawDesc), len(file_ermite_storage_ci_proto_rawDesc)),
2153- NumEnums: 7,2565+ NumEnums: 8,
2154- NumMessages: 13,2566+ NumMessages: 17,
2155 NumExtensions: 0,2567 NumExtensions: 0,
2156 NumServices: 0,2568 NumServices: 0,
2157 },2569 },
M internal/adapters/kvstore/ci.go +10 -3
@@ -37,6 +37,7 @@ import (
3737 // ciready/<ready_at:020d>/<job key> job waiting for a runner
3838 // ciactive/<job key> job assigned or running
3939 // jobtoken/<hash> job token (value: job key)
40+// concur/<repo>/<key(group)> concurrency group (ci_concurrency.go)
4041 //
4142 // The job repository tokens and the action resolutions are in ci_token.go, the
4243 // annotations in ci_annotation.go.
@@ -110,6 +111,8 @@ func encodeStatus(s domain.CIStatus) storagepb.CIStatus {
110111 return storagepb.CIStatus_CI_STATUS_IN_PROGRESS
111112 case domain.CICompleted:
112113 return storagepb.CIStatus_CI_STATUS_COMPLETED
114+ case domain.CIPending:
115+ return storagepb.CIStatus_CI_STATUS_PENDING
113116 }
114117 return storagepb.CIStatus_CI_STATUS_UNSPECIFIED
115118 }
@@ -120,6 +123,8 @@ func decodeStatus(s storagepb.CIStatus) domain.CIStatus {
120123 return domain.CIInProgress
121124 case storagepb.CIStatus_CI_STATUS_COMPLETED:
122125 return domain.CICompleted
126+ case storagepb.CIStatus_CI_STATUS_PENDING:
127+ return domain.CIPending
123128 }
124129 return domain.CIQueued
125130 }
@@ -276,6 +281,7 @@ func encodeRun(r domain.CIRun) ([]byte, error) {
276281 Status: encodeStatus(r.Status), Conclusion: encodeConclusion(r.Conclusion), Error: r.Error,
277282 CreatedAt: encodeTime(r.CreatedAt), StartedAt: encodeTime(r.StartedAt), CompletedAt: encodeTime(r.CompletedAt),
278283 CancelRequested: r.CancelRequested, CancelRequestedAt: encodeTime(r.CancelRequestedAt), Trusted: r.Trusted,
284+ Concurrency: encodeConcurrency(r.Concurrency),
279285 }), nil
280286 }
281287
@@ -299,7 +305,7 @@ func decodeRun(item kv.Item) (domain.CIRun, error) {
299305 Status: decodeStatus(pb.GetStatus()), Conclusion: decodeConclusion(pb.GetConclusion()), Error: pb.GetError(),
300306 CreatedAt: decodeTime(pb.GetCreatedAt()), StartedAt: decodeTime(pb.GetStartedAt()), CompletedAt: decodeTime(pb.GetCompletedAt()),
301307 CancelRequested: pb.GetCancelRequested(), CancelRequestedAt: decodeTime(pb.GetCancelRequestedAt()),
302- Trusted: pb.GetTrusted(), Version: item.Version,
308+ Trusted: pb.GetTrusted(), Concurrency: decodeConcurrency(pb.GetConcurrency()), Version: item.Version,
303309 }, nil
304310 }
305311
@@ -331,6 +337,7 @@ func encodeJob(j domain.CIJob) ([]byte, error) {
331337 AckedAt: encodeTime(j.AckedAt), AssignCancels: int32(j.AssignCancels), //nolint:gosec // at most 3
332338 TokenValidUntil: encodeTime(j.TokenValidUntil), CancelReason: encodeCancelReason(j.CancelReason), CancelMessage: j.CancelMessage,
333339 CancelRequestedAt: encodeTime(j.CancelRequestedAt), Planned: j.Planned, Environment: j.Environment, Permissions: j.Permissions,
340+ Concurrency: encodeConcurrency(j.Concurrency),
334341 }
335342 if j.HasToken() {
336343 pb.TokenHash = j.TokenHash[:]
@@ -360,7 +367,7 @@ func decodeJob(item kv.Item) (domain.CIJob, error) {
360367 TokenValidUntil: decodeTime(pb.GetTokenValidUntil()), ReadyAt: decodeTime(pb.GetReadyAt()),
361368 AssignedAt: decodeTime(pb.GetAssignedAt()), AckedAt: decodeTime(pb.GetAckedAt()), AssignCancels: int(pb.GetAssignCancels()),
362369 CancelMessage: pb.GetCancelMessage(), CancelRequestedAt: decodeTime(pb.GetCancelRequestedAt()),
363- Environment: pb.GetEnvironment(), Permissions: pb.GetPermissions(), Version: item.Version,
370+ Environment: pb.GetEnvironment(), Permissions: pb.GetPermissions(), Concurrency: decodeConcurrency(pb.GetConcurrency()), Version: item.Version,
364371 }
365372 j.CancelReason = decodeCancelReason(pb.GetCancelReason())
366373 copy(j.TokenHash[:], pb.GetTokenHash())
@@ -877,7 +884,7 @@ func ciKeys(ctx context.Context, snap *kv.Snapshot, repo domain.RepoName) [][]by
877884 keys = append(keys, slices.Clone(key))
878885 }
879886 keys = append(keys, repoReaderKeys(ctx, snap, repo)...)
880- for _, p := range []string{checkPrefix + repo.String() + "/", runNumPrefix + repo.String() + "/"} {
887+ for _, p := range []string{checkPrefix + repo.String() + "/", runNumPrefix + repo.String() + "/", concurPrefix + repo.String() + "/", concurMarkPrefixAll + repo.String() + "/"} {
881888 for k := range snap.Prefix(ctx, []byte(p)) {
882889 keys = append(keys, slices.Clone(k))
883890 }
@@ -37,6 +37,7 @@ import (
37 // ciready/<ready_at:020d>/<job key> job waiting for a runner37 // ciready/<ready_at:020d>/<job key> job waiting for a runner
38 // ciactive/<job key> job assigned or running38 // ciactive/<job key> job assigned or running
39 // jobtoken/<hash> job token (value: job key)39 // jobtoken/<hash> job token (value: job key)
40+// concur/<repo>/<key(group)> concurrency group (ci_concurrency.go)
40 //41 //
41 // The job repository tokens and the action resolutions are in ci_token.go, the42 // The job repository tokens and the action resolutions are in ci_token.go, the
42 // annotations in ci_annotation.go.43 // annotations in ci_annotation.go.
@@ -110,6 +111,8 @@ func encodeStatus(s domain.CIStatus) storagepb.CIStatus {
110 return storagepb.CIStatus_CI_STATUS_IN_PROGRESS111 return storagepb.CIStatus_CI_STATUS_IN_PROGRESS
111 case domain.CICompleted:112 case domain.CICompleted:
112 return storagepb.CIStatus_CI_STATUS_COMPLETED113 return storagepb.CIStatus_CI_STATUS_COMPLETED
114+ case domain.CIPending:
115+ return storagepb.CIStatus_CI_STATUS_PENDING
113 }116 }
114 return storagepb.CIStatus_CI_STATUS_UNSPECIFIED117 return storagepb.CIStatus_CI_STATUS_UNSPECIFIED
115 }118 }
@@ -120,6 +123,8 @@ func decodeStatus(s storagepb.CIStatus) domain.CIStatus {
120 return domain.CIInProgress123 return domain.CIInProgress
121 case storagepb.CIStatus_CI_STATUS_COMPLETED:124 case storagepb.CIStatus_CI_STATUS_COMPLETED:
122 return domain.CICompleted125 return domain.CICompleted
126+ case storagepb.CIStatus_CI_STATUS_PENDING:
127+ return domain.CIPending
123 }128 }
124 return domain.CIQueued129 return domain.CIQueued
125 }130 }
@@ -276,6 +281,7 @@ func encodeRun(r domain.CIRun) ([]byte, error) {
276 Status: encodeStatus(r.Status), Conclusion: encodeConclusion(r.Conclusion), Error: r.Error,281 Status: encodeStatus(r.Status), Conclusion: encodeConclusion(r.Conclusion), Error: r.Error,
277 CreatedAt: encodeTime(r.CreatedAt), StartedAt: encodeTime(r.StartedAt), CompletedAt: encodeTime(r.CompletedAt),282 CreatedAt: encodeTime(r.CreatedAt), StartedAt: encodeTime(r.StartedAt), CompletedAt: encodeTime(r.CompletedAt),
278 CancelRequested: r.CancelRequested, CancelRequestedAt: encodeTime(r.CancelRequestedAt), Trusted: r.Trusted,283 CancelRequested: r.CancelRequested, CancelRequestedAt: encodeTime(r.CancelRequestedAt), Trusted: r.Trusted,
284+ Concurrency: encodeConcurrency(r.Concurrency),
279 }), nil285 }), nil
280 }286 }
281 287
@@ -299,7 +305,7 @@ func decodeRun(item kv.Item) (domain.CIRun, error) {
299 Status: decodeStatus(pb.GetStatus()), Conclusion: decodeConclusion(pb.GetConclusion()), Error: pb.GetError(),305 Status: decodeStatus(pb.GetStatus()), Conclusion: decodeConclusion(pb.GetConclusion()), Error: pb.GetError(),
300 CreatedAt: decodeTime(pb.GetCreatedAt()), StartedAt: decodeTime(pb.GetStartedAt()), CompletedAt: decodeTime(pb.GetCompletedAt()),306 CreatedAt: decodeTime(pb.GetCreatedAt()), StartedAt: decodeTime(pb.GetStartedAt()), CompletedAt: decodeTime(pb.GetCompletedAt()),
301 CancelRequested: pb.GetCancelRequested(), CancelRequestedAt: decodeTime(pb.GetCancelRequestedAt()),307 CancelRequested: pb.GetCancelRequested(), CancelRequestedAt: decodeTime(pb.GetCancelRequestedAt()),
302- Trusted: pb.GetTrusted(), Version: item.Version,308+ Trusted: pb.GetTrusted(), Concurrency: decodeConcurrency(pb.GetConcurrency()), Version: item.Version,
303 }, nil309 }, nil
304 }310 }
305 311
@@ -331,6 +337,7 @@ func encodeJob(j domain.CIJob) ([]byte, error) {
331 AckedAt: encodeTime(j.AckedAt), AssignCancels: int32(j.AssignCancels), //nolint:gosec // at most 3337 AckedAt: encodeTime(j.AckedAt), AssignCancels: int32(j.AssignCancels), //nolint:gosec // at most 3
332 TokenValidUntil: encodeTime(j.TokenValidUntil), CancelReason: encodeCancelReason(j.CancelReason), CancelMessage: j.CancelMessage,338 TokenValidUntil: encodeTime(j.TokenValidUntil), CancelReason: encodeCancelReason(j.CancelReason), CancelMessage: j.CancelMessage,
333 CancelRequestedAt: encodeTime(j.CancelRequestedAt), Planned: j.Planned, Environment: j.Environment, Permissions: j.Permissions,339 CancelRequestedAt: encodeTime(j.CancelRequestedAt), Planned: j.Planned, Environment: j.Environment, Permissions: j.Permissions,
340+ Concurrency: encodeConcurrency(j.Concurrency),
334 }341 }
335 if j.HasToken() {342 if j.HasToken() {
336 pb.TokenHash = j.TokenHash[:]343 pb.TokenHash = j.TokenHash[:]
@@ -360,7 +367,7 @@ func decodeJob(item kv.Item) (domain.CIJob, error) {
360 TokenValidUntil: decodeTime(pb.GetTokenValidUntil()), ReadyAt: decodeTime(pb.GetReadyAt()),367 TokenValidUntil: decodeTime(pb.GetTokenValidUntil()), ReadyAt: decodeTime(pb.GetReadyAt()),
361 AssignedAt: decodeTime(pb.GetAssignedAt()), AckedAt: decodeTime(pb.GetAckedAt()), AssignCancels: int(pb.GetAssignCancels()),368 AssignedAt: decodeTime(pb.GetAssignedAt()), AckedAt: decodeTime(pb.GetAckedAt()), AssignCancels: int(pb.GetAssignCancels()),
362 CancelMessage: pb.GetCancelMessage(), CancelRequestedAt: decodeTime(pb.GetCancelRequestedAt()),369 CancelMessage: pb.GetCancelMessage(), CancelRequestedAt: decodeTime(pb.GetCancelRequestedAt()),
363- Environment: pb.GetEnvironment(), Permissions: pb.GetPermissions(), Version: item.Version,370+ Environment: pb.GetEnvironment(), Permissions: pb.GetPermissions(), Concurrency: decodeConcurrency(pb.GetConcurrency()), Version: item.Version,
364 }371 }
365 j.CancelReason = decodeCancelReason(pb.GetCancelReason())372 j.CancelReason = decodeCancelReason(pb.GetCancelReason())
366 copy(j.TokenHash[:], pb.GetTokenHash())373 copy(j.TokenHash[:], pb.GetTokenHash())
@@ -877,7 +884,7 @@ func ciKeys(ctx context.Context, snap *kv.Snapshot, repo domain.RepoName) [][]by
877 keys = append(keys, slices.Clone(key))884 keys = append(keys, slices.Clone(key))
878 }885 }
879 keys = append(keys, repoReaderKeys(ctx, snap, repo)...)886 keys = append(keys, repoReaderKeys(ctx, snap, repo)...)
880- for _, p := range []string{checkPrefix + repo.String() + "/", runNumPrefix + repo.String() + "/"} {887+ for _, p := range []string{checkPrefix + repo.String() + "/", runNumPrefix + repo.String() + "/", concurPrefix + repo.String() + "/", concurMarkPrefixAll + repo.String() + "/"} {
881 for k := range snap.Prefix(ctx, []byte(p)) {888 for k := range snap.Prefix(ctx, []byte(p)) {
882 keys = append(keys, slices.Clone(k))889 keys = append(keys, slices.Clone(k))
883 }890 }
A internal/adapters/kvstore/ci_concurrency.go +296 -0
@@ -0,0 +1,296 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package kvstore
4+
5+import (
6+ "bytes"
7+ "context"
8+ "errors"
9+ "fmt"
10+ "maps"
11+ "net/url"
12+ "slices"
13+ "strings"
14+
15+ kv "bnema.dev/ermite-kv"
16+ "google.golang.org/protobuf/proto"
17+
18+ storagepb "bnema.dev/ermite/gen/ermite/storage"
19+ "bnema.dev/ermite/internal/domain"
20+ "bnema.dev/ermite/internal/ports"
21+)
22+
23+// Key layout of the concurrency groups:
24+//
25+// concur/<repo>/<key(group)> CIConcurrencyRecord (running and pending entry)
26+// concurmark/<repo>/<escaped key(group)>/<run>/<attempt>/<job id>
27+// CIConcurrencyMark, written in the same batch
28+//
29+// The group key disappears only when both the record and its marks are empty.
30+// Both prefixes are removed with their repository (ciKeys).
31+const (
32+ concurPrefix = "concur/"
33+ concurMarkPrefixAll = "concurmark/"
34+)
35+
36+// updateConcurrencyTries bounds the retries of UpdateConcurrency under
37+// contention.
38+const updateConcurrencyTries = 32
39+
40+func concurKey(repo domain.RepoName, group string) []byte {
41+ return []byte(concurPrefix + repo.String() + "/" + domain.ConcurrencyKey(group))
42+}
43+
44+var concurrencyStates = []struct {
45+ d domain.CIConcurrencyState
46+ p storagepb.CIConcurrencyState
47+}{
48+ {domain.CIConcurrencyPending, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING},
49+ {domain.CIConcurrencyHolding, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING},
50+}
51+
52+func encodeConcurrency(c domain.CIConcurrency) *storagepb.CIConcurrency {
53+ if c == (domain.CIConcurrency{}) {
54+ return nil
55+ }
56+ out := &storagepb.CIConcurrency{Group: c.Group, Cancel: c.Cancel, Evaluated: c.Evaluated, FromRun: c.FromRun, Preempted: c.Preempted}
57+ for _, e := range concurrencyStates {
58+ if e.d == c.State {
59+ out.State = e.p
60+ }
61+ }
62+ return out
63+}
64+
65+func decodeConcurrency(pb *storagepb.CIConcurrency) domain.CIConcurrency {
66+ if pb == nil {
67+ return domain.CIConcurrency{}
68+ }
69+ c := domain.CIConcurrency{Group: pb.GetGroup(), Cancel: pb.GetCancel(), Evaluated: pb.GetEvaluated(), FromRun: pb.GetFromRun(), Preempted: pb.GetPreempted()}
70+ for _, e := range concurrencyStates {
71+ if e.p == pb.GetState() {
72+ c.State = e.d
73+ }
74+ }
75+ return c
76+}
77+
78+func encodeConcurrencyEntry(e *domain.CIConcurrencyEntry) *storagepb.CIConcurrencyEntry {
79+ if e == nil {
80+ return nil
81+ }
82+ return &storagepb.CIConcurrencyEntry{RunId: e.RunID, Attempt: e.Attempt, JobId: e.JobID, Cancel: e.Cancel, Canceling: e.Canceling}
83+}
84+
85+func encodeConcurrencyRecord(r domain.CIConcurrencyRecord) []byte {
86+ return mustMarshal(&storagepb.CIConcurrencyRecord{Running: encodeConcurrencyEntry(r.Running), Pending: encodeConcurrencyEntry(r.Pending)})
87+}
88+
89+//hexcheck:ignore suspicious-business-logic-in-adapter maps the stored record to the domain record field by field: no rule is decided here
90+func decodeConcurrencyRecord(val []byte) (domain.CIConcurrencyRecord, error) {
91+ var pb storagepb.CIConcurrencyRecord
92+ if err := proto.Unmarshal(val, &pb); err != nil {
93+ return domain.CIConcurrencyRecord{}, fmt.Errorf("decode concurrency group: %w", err)
94+ }
95+ return domain.CIConcurrencyRecord{Running: decodeConcurrencyEntry(pb.GetRunning()), Pending: decodeConcurrencyEntry(pb.GetPending())}, nil
96+}
97+
98+func decodeConcurrencyEntry(pb *storagepb.CIConcurrencyEntry) *domain.CIConcurrencyEntry {
99+ if pb == nil {
100+ return nil
101+ }
102+ return &domain.CIConcurrencyEntry{RunID: pb.GetRunId(), Attempt: pb.GetAttempt(), JobID: pb.GetJobId(), Cancel: pb.GetCancel(), Canceling: pb.GetCanceling()}
103+}
104+
105+// concurMarkPrefix is where the marks of a group are: the group key is escaped,
106+// so that the marks of "a" never fall under the prefix of "a/b".
107+func concurMarkPrefix(repo domain.RepoName, group string) string {
108+ return concurMarkPrefixAll + repo.String() + "/" + url.PathEscape(domain.ConcurrencyKey(group)) + "/"
109+}
110+
111+func encodeConcurrencyMark(m domain.CIConcurrencyMark) []byte {
112+ pb := &storagepb.CIConcurrencyMark{Entry: encodeConcurrencyEntry(&m.Entry), Group: m.Group}
113+ for _, e := range concurrencyStates {
114+ if e.d == m.State {
115+ pb.State = e.p
116+ }
117+ }
118+ if m.State == domain.CIConcurrencyReplaced {
119+ pb.State = storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED
120+ }
121+ return mustMarshal(pb)
122+}
123+
124+//hexcheck:ignore suspicious-business-logic-in-adapter maps the stored mark to the domain mark field by field: no rule is decided here
125+func decodeConcurrencyMark(val []byte) (domain.CIConcurrencyMark, error) {
126+ var pb storagepb.CIConcurrencyMark
127+ if err := proto.Unmarshal(val, &pb); err != nil {
128+ return domain.CIConcurrencyMark{}, fmt.Errorf("decode concurrency mark: %w", err)
129+ }
130+ m := domain.CIConcurrencyMark{Group: pb.GetGroup()}
131+ if e := decodeConcurrencyEntry(pb.GetEntry()); e != nil {
132+ m.Entry = *e
133+ }
134+ switch pb.GetState() {
135+ case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING:
136+ m.State = domain.CIConcurrencyPending
137+ case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING:
138+ m.State = domain.CIConcurrencyHolding
139+ case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED:
140+ m.State = domain.CIConcurrencyReplaced
141+ }
142+ return m, nil
143+}
144+
145+// encodeConcurrencyState is a canonical image of a group: the record, then the
146+// marks in key order. Two states are the same when their images are.
147+func encodeConcurrencyState(s domain.CIConcurrencyGroupState) []byte {
148+ out := encodeConcurrencyRecord(s.Record)
149+ for _, k := range slices.Sorted(maps.Keys(s.Marks)) {
150+ out = append(out, 0)
151+ out = append(out, k...)
152+ out = append(out, 0)
153+ out = append(out, encodeConcurrencyMark(s.Marks[k])...)
154+ }
155+ return out
156+}
157+
158+// UpdateConcurrency reads a group with its marks, calls fn on it and stores the
159+// result in ONE batch: the record (key "concur/..."), every mark that changed
160+// ("concurmark/...", written or deleted) and, when something changed, the guard
161+// of the repository. The batch requires the version the record had when it was
162+// read (or its absence) whatever fn does, and every change rewrites the record,
163+// so of two transactions that read the same state only one commits; the other
164+// calls fn again on the new state.
165+//
166+//hexcheck:ignore suspicious-business-logic-in-adapter compare-and-swap loop over a record and its marks in one batch: the rule is fn, decided by the caller
167+func (c *CI) UpdateConcurrency(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error {
168+ key, mprefix := concurKey(repo, group), concurMarkPrefix(repo, group)
169+ for range updateConcurrencyTries {
170+ var state domain.CIConcurrencyGroupState
171+ var version uint64
172+ if item, ok := c.db.Get(ctx, key); ok {
173+ var err error
174+ if state.Record, err = decodeConcurrencyRecord(item.Value); err != nil {
175+ return err
176+ }
177+ version = item.Version
178+ }
179+ // The marks are read after the record: a transaction that commits in
180+ // between has changed the record, and the batch below conflicts.
181+ state.Marks = map[string]domain.CIConcurrencyMark{}
182+ for _, item := range c.db.Snapshot(ctx).Prefix(ctx, []byte(mprefix)) {
183+ m, err := decodeConcurrencyMark(item.Value)
184+ if err != nil {
185+ return err
186+ }
187+ state.Marks[m.Entry.Key()] = m
188+ }
189+ // The state as read, in bytes, before fn sees it: fn may change the
190+ // entries it is given in place, and a copy of the struct shares them.
191+ before, stored := encodeConcurrencyState(state), maps.Clone(state.Marks)
192+ if err := fn(&state); err != nil {
193+ return err
194+ }
195+ var b kv.Batch
196+ if version == 0 {
197+ b.RequireAbsent(key)
198+ } else {
199+ b.RequireVersion(key, version)
200+ }
201+ // A state left as it was is not written, but the batch still holds the
202+ // precondition: the decision was made on the state that was read.
203+ if !bytes.Equal(before, encodeConcurrencyState(state)) {
204+ for k, m := range state.Marks {
205+ if old, ok := stored[k]; !ok || old != m {
206+ b.Put([]byte(mprefix+k), encodeConcurrencyMark(m))
207+ }
208+ }
209+ for k := range stored {
210+ if _, ok := state.Marks[k]; !ok {
211+ b.Delete([]byte(mprefix + k))
212+ }
213+ }
214+ if state.IsEmpty() {
215+ b.Delete(key)
216+ } else {
217+ b.Put(key, encodeConcurrencyRecord(state.Record)) // rewritten even if equal: it is the version gate
218+ if err := c.guardRepo(ctx, &b, repo); err != nil {
219+ return err
220+ }
221+ }
222+ }
223+ if _, err := c.db.Apply(ctx, &b); err != nil {
224+ if errors.Is(err, kv.ErrConflict) {
225+ continue
226+ }
227+ return fmt.Errorf("write concurrency group: %w", err)
228+ }
229+ return nil
230+ }
231+ return fmt.Errorf("%w: concurrency group is busy", domain.ErrConflict)
232+}
233+
234+// ListConcurrency returns every stored group with its marks.
235+//
236+//hexcheck:ignore suspicious-business-logic-in-adapter lists and decodes records and marks: no rule is decided here
237+func (c *CI) ListConcurrency(ctx context.Context) ([]ports.CIConcurrencyGroup, error) {
238+ type id struct{ repo, group string }
239+ snap := c.db.Snapshot(ctx)
240+ byID := map[id]*ports.CIConcurrencyGroup{}
241+ get := func(repoName, group string) *ports.CIConcurrencyGroup {
242+ repo, err := domain.ParseRepoName(repoName)
243+ if err != nil {
244+ return nil
245+ }
246+ g, ok := byID[id{repoName, group}]
247+ if !ok {
248+ g = &ports.CIConcurrencyGroup{Repo: repo, Group: group, State: domain.CIConcurrencyGroupState{Marks: map[string]domain.CIConcurrencyMark{}}}
249+ byID[id{repoName, group}] = g
250+ }
251+ return g
252+ }
253+ for key, item := range snap.Prefix(ctx, []byte(concurPrefix)) {
254+ repoName, group, ok := strings.Cut(strings.TrimPrefix(string(key), concurPrefix), "/")
255+ if !ok {
256+ continue
257+ }
258+ g := get(repoName, group)
259+ if g == nil {
260+ continue
261+ }
262+ rec, err := decodeConcurrencyRecord(item.Value)
263+ if err != nil {
264+ return nil, err
265+ }
266+ g.State.Record = rec
267+ }
268+ for key, item := range snap.Prefix(ctx, []byte(concurMarkPrefixAll)) {
269+ repoName, rest, ok := strings.Cut(strings.TrimPrefix(string(key), concurMarkPrefixAll), "/")
270+ if !ok {
271+ continue
272+ }
273+ escaped, _, ok := strings.Cut(rest, "/")
274+ if !ok {
275+ continue
276+ }
277+ group, err := url.PathUnescape(escaped)
278+ if err != nil {
279+ continue
280+ }
281+ g := get(repoName, group)
282+ if g == nil {
283+ continue
284+ }
285+ m, err := decodeConcurrencyMark(item.Value)
286+ if err != nil {
287+ return nil, err
288+ }
289+ g.State.Marks[m.Entry.Key()] = m
290+ }
291+ out := make([]ports.CIConcurrencyGroup, 0, len(byID))
292+ for _, g := range byID {
293+ out = append(out, *g)
294+ }
295+ return out, nil
296+}
@@ -0,0 +1,296 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package kvstore
4+
5+import (
6+ "bytes"
7+ "context"
8+ "errors"
9+ "fmt"
10+ "maps"
11+ "net/url"
12+ "slices"
13+ "strings"
14+
15+ kv "bnema.dev/ermite-kv"
16+ "google.golang.org/protobuf/proto"
17+
18+ storagepb "bnema.dev/ermite/gen/ermite/storage"
19+ "bnema.dev/ermite/internal/domain"
20+ "bnema.dev/ermite/internal/ports"
21+)
22+
23+// Key layout of the concurrency groups:
24+//
25+// concur/<repo>/<key(group)> CIConcurrencyRecord (running and pending entry)
26+// concurmark/<repo>/<escaped key(group)>/<run>/<attempt>/<job id>
27+// CIConcurrencyMark, written in the same batch
28+//
29+// The group key disappears only when both the record and its marks are empty.
30+// Both prefixes are removed with their repository (ciKeys).
31+const (
32+ concurPrefix = "concur/"
33+ concurMarkPrefixAll = "concurmark/"
34+)
35+
36+// updateConcurrencyTries bounds the retries of UpdateConcurrency under
37+// contention.
38+const updateConcurrencyTries = 32
39+
40+func concurKey(repo domain.RepoName, group string) []byte {
41+ return []byte(concurPrefix + repo.String() + "/" + domain.ConcurrencyKey(group))
42+}
43+
44+var concurrencyStates = []struct {
45+ d domain.CIConcurrencyState
46+ p storagepb.CIConcurrencyState
47+}{
48+ {domain.CIConcurrencyPending, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING},
49+ {domain.CIConcurrencyHolding, storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING},
50+}
51+
52+func encodeConcurrency(c domain.CIConcurrency) *storagepb.CIConcurrency {
53+ if c == (domain.CIConcurrency{}) {
54+ return nil
55+ }
56+ out := &storagepb.CIConcurrency{Group: c.Group, Cancel: c.Cancel, Evaluated: c.Evaluated, FromRun: c.FromRun, Preempted: c.Preempted}
57+ for _, e := range concurrencyStates {
58+ if e.d == c.State {
59+ out.State = e.p
60+ }
61+ }
62+ return out
63+}
64+
65+func decodeConcurrency(pb *storagepb.CIConcurrency) domain.CIConcurrency {
66+ if pb == nil {
67+ return domain.CIConcurrency{}
68+ }
69+ c := domain.CIConcurrency{Group: pb.GetGroup(), Cancel: pb.GetCancel(), Evaluated: pb.GetEvaluated(), FromRun: pb.GetFromRun(), Preempted: pb.GetPreempted()}
70+ for _, e := range concurrencyStates {
71+ if e.p == pb.GetState() {
72+ c.State = e.d
73+ }
74+ }
75+ return c
76+}
77+
78+func encodeConcurrencyEntry(e *domain.CIConcurrencyEntry) *storagepb.CIConcurrencyEntry {
79+ if e == nil {
80+ return nil
81+ }
82+ return &storagepb.CIConcurrencyEntry{RunId: e.RunID, Attempt: e.Attempt, JobId: e.JobID, Cancel: e.Cancel, Canceling: e.Canceling}
83+}
84+
85+func encodeConcurrencyRecord(r domain.CIConcurrencyRecord) []byte {
86+ return mustMarshal(&storagepb.CIConcurrencyRecord{Running: encodeConcurrencyEntry(r.Running), Pending: encodeConcurrencyEntry(r.Pending)})
87+}
88+
89+//hexcheck:ignore suspicious-business-logic-in-adapter maps the stored record to the domain record field by field: no rule is decided here
90+func decodeConcurrencyRecord(val []byte) (domain.CIConcurrencyRecord, error) {
91+ var pb storagepb.CIConcurrencyRecord
92+ if err := proto.Unmarshal(val, &pb); err != nil {
93+ return domain.CIConcurrencyRecord{}, fmt.Errorf("decode concurrency group: %w", err)
94+ }
95+ return domain.CIConcurrencyRecord{Running: decodeConcurrencyEntry(pb.GetRunning()), Pending: decodeConcurrencyEntry(pb.GetPending())}, nil
96+}
97+
98+func decodeConcurrencyEntry(pb *storagepb.CIConcurrencyEntry) *domain.CIConcurrencyEntry {
99+ if pb == nil {
100+ return nil
101+ }
102+ return &domain.CIConcurrencyEntry{RunID: pb.GetRunId(), Attempt: pb.GetAttempt(), JobID: pb.GetJobId(), Cancel: pb.GetCancel(), Canceling: pb.GetCanceling()}
103+}
104+
105+// concurMarkPrefix is where the marks of a group are: the group key is escaped,
106+// so that the marks of "a" never fall under the prefix of "a/b".
107+func concurMarkPrefix(repo domain.RepoName, group string) string {
108+ return concurMarkPrefixAll + repo.String() + "/" + url.PathEscape(domain.ConcurrencyKey(group)) + "/"
109+}
110+
111+func encodeConcurrencyMark(m domain.CIConcurrencyMark) []byte {
112+ pb := &storagepb.CIConcurrencyMark{Entry: encodeConcurrencyEntry(&m.Entry), Group: m.Group}
113+ for _, e := range concurrencyStates {
114+ if e.d == m.State {
115+ pb.State = e.p
116+ }
117+ }
118+ if m.State == domain.CIConcurrencyReplaced {
119+ pb.State = storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED
120+ }
121+ return mustMarshal(pb)
122+}
123+
124+//hexcheck:ignore suspicious-business-logic-in-adapter maps the stored mark to the domain mark field by field: no rule is decided here
125+func decodeConcurrencyMark(val []byte) (domain.CIConcurrencyMark, error) {
126+ var pb storagepb.CIConcurrencyMark
127+ if err := proto.Unmarshal(val, &pb); err != nil {
128+ return domain.CIConcurrencyMark{}, fmt.Errorf("decode concurrency mark: %w", err)
129+ }
130+ m := domain.CIConcurrencyMark{Group: pb.GetGroup()}
131+ if e := decodeConcurrencyEntry(pb.GetEntry()); e != nil {
132+ m.Entry = *e
133+ }
134+ switch pb.GetState() {
135+ case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_PENDING:
136+ m.State = domain.CIConcurrencyPending
137+ case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_HOLDING:
138+ m.State = domain.CIConcurrencyHolding
139+ case storagepb.CIConcurrencyState_CI_CONCURRENCY_STATE_REPLACED:
140+ m.State = domain.CIConcurrencyReplaced
141+ }
142+ return m, nil
143+}
144+
145+// encodeConcurrencyState is a canonical image of a group: the record, then the
146+// marks in key order. Two states are the same when their images are.
147+func encodeConcurrencyState(s domain.CIConcurrencyGroupState) []byte {
148+ out := encodeConcurrencyRecord(s.Record)
149+ for _, k := range slices.Sorted(maps.Keys(s.Marks)) {
150+ out = append(out, 0)
151+ out = append(out, k...)
152+ out = append(out, 0)
153+ out = append(out, encodeConcurrencyMark(s.Marks[k])...)
154+ }
155+ return out
156+}
157+
158+// UpdateConcurrency reads a group with its marks, calls fn on it and stores the
159+// result in ONE batch: the record (key "concur/..."), every mark that changed
160+// ("concurmark/...", written or deleted) and, when something changed, the guard
161+// of the repository. The batch requires the version the record had when it was
162+// read (or its absence) whatever fn does, and every change rewrites the record,
163+// so of two transactions that read the same state only one commits; the other
164+// calls fn again on the new state.
165+//
166+//hexcheck:ignore suspicious-business-logic-in-adapter compare-and-swap loop over a record and its marks in one batch: the rule is fn, decided by the caller
167+func (c *CI) UpdateConcurrency(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error {
168+ key, mprefix := concurKey(repo, group), concurMarkPrefix(repo, group)
169+ for range updateConcurrencyTries {
170+ var state domain.CIConcurrencyGroupState
171+ var version uint64
172+ if item, ok := c.db.Get(ctx, key); ok {
173+ var err error
174+ if state.Record, err = decodeConcurrencyRecord(item.Value); err != nil {
175+ return err
176+ }
177+ version = item.Version
178+ }
179+ // The marks are read after the record: a transaction that commits in
180+ // between has changed the record, and the batch below conflicts.
181+ state.Marks = map[string]domain.CIConcurrencyMark{}
182+ for _, item := range c.db.Snapshot(ctx).Prefix(ctx, []byte(mprefix)) {
183+ m, err := decodeConcurrencyMark(item.Value)
184+ if err != nil {
185+ return err
186+ }
187+ state.Marks[m.Entry.Key()] = m
188+ }
189+ // The state as read, in bytes, before fn sees it: fn may change the
190+ // entries it is given in place, and a copy of the struct shares them.
191+ before, stored := encodeConcurrencyState(state), maps.Clone(state.Marks)
192+ if err := fn(&state); err != nil {
193+ return err
194+ }
195+ var b kv.Batch
196+ if version == 0 {
197+ b.RequireAbsent(key)
198+ } else {
199+ b.RequireVersion(key, version)
200+ }
201+ // A state left as it was is not written, but the batch still holds the
202+ // precondition: the decision was made on the state that was read.
203+ if !bytes.Equal(before, encodeConcurrencyState(state)) {
204+ for k, m := range state.Marks {
205+ if old, ok := stored[k]; !ok || old != m {
206+ b.Put([]byte(mprefix+k), encodeConcurrencyMark(m))
207+ }
208+ }
209+ for k := range stored {
210+ if _, ok := state.Marks[k]; !ok {
211+ b.Delete([]byte(mprefix + k))
212+ }
213+ }
214+ if state.IsEmpty() {
215+ b.Delete(key)
216+ } else {
217+ b.Put(key, encodeConcurrencyRecord(state.Record)) // rewritten even if equal: it is the version gate
218+ if err := c.guardRepo(ctx, &b, repo); err != nil {
219+ return err
220+ }
221+ }
222+ }
223+ if _, err := c.db.Apply(ctx, &b); err != nil {
224+ if errors.Is(err, kv.ErrConflict) {
225+ continue
226+ }
227+ return fmt.Errorf("write concurrency group: %w", err)
228+ }
229+ return nil
230+ }
231+ return fmt.Errorf("%w: concurrency group is busy", domain.ErrConflict)
232+}
233+
234+// ListConcurrency returns every stored group with its marks.
235+//
236+//hexcheck:ignore suspicious-business-logic-in-adapter lists and decodes records and marks: no rule is decided here
237+func (c *CI) ListConcurrency(ctx context.Context) ([]ports.CIConcurrencyGroup, error) {
238+ type id struct{ repo, group string }
239+ snap := c.db.Snapshot(ctx)
240+ byID := map[id]*ports.CIConcurrencyGroup{}
241+ get := func(repoName, group string) *ports.CIConcurrencyGroup {
242+ repo, err := domain.ParseRepoName(repoName)
243+ if err != nil {
244+ return nil
245+ }
246+ g, ok := byID[id{repoName, group}]
247+ if !ok {
248+ g = &ports.CIConcurrencyGroup{Repo: repo, Group: group, State: domain.CIConcurrencyGroupState{Marks: map[string]domain.CIConcurrencyMark{}}}
249+ byID[id{repoName, group}] = g
250+ }
251+ return g
252+ }
253+ for key, item := range snap.Prefix(ctx, []byte(concurPrefix)) {
254+ repoName, group, ok := strings.Cut(strings.TrimPrefix(string(key), concurPrefix), "/")
255+ if !ok {
256+ continue
257+ }
258+ g := get(repoName, group)
259+ if g == nil {
260+ continue
261+ }
262+ rec, err := decodeConcurrencyRecord(item.Value)
263+ if err != nil {
264+ return nil, err
265+ }
266+ g.State.Record = rec
267+ }
268+ for key, item := range snap.Prefix(ctx, []byte(concurMarkPrefixAll)) {
269+ repoName, rest, ok := strings.Cut(strings.TrimPrefix(string(key), concurMarkPrefixAll), "/")
270+ if !ok {
271+ continue
272+ }
273+ escaped, _, ok := strings.Cut(rest, "/")
274+ if !ok {
275+ continue
276+ }
277+ group, err := url.PathUnescape(escaped)
278+ if err != nil {
279+ continue
280+ }
281+ g := get(repoName, group)
282+ if g == nil {
283+ continue
284+ }
285+ m, err := decodeConcurrencyMark(item.Value)
286+ if err != nil {
287+ return nil, err
288+ }
289+ g.State.Marks[m.Entry.Key()] = m
290+ }
291+ out := make([]ports.CIConcurrencyGroup, 0, len(byID))
292+ for _, g := range byID {
293+ out = append(out, *g)
294+ }
295+ return out, nil
296+}
A internal/adapters/kvstore/ci_concurrency_test.go +294 -0
@@ -0,0 +1,294 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package kvstore_test
4+
5+import (
6+ "context"
7+ "sync"
8+ "testing"
9+
10+ "github.com/stretchr/testify/assert"
11+ "github.com/stretchr/testify/require"
12+
13+ "bnema.dev/ermite/internal/domain"
14+)
15+
16+func entry(run int64) *domain.CIConcurrencyEntry {
17+ return &domain.CIConcurrencyEntry{RunID: run, Attempt: 1}
18+}
19+
20+// update runs one transaction on a group.
21+func update(t *testing.T, ci interface {
22+ UpdateConcurrency(context.Context, domain.RepoName, string, func(*domain.CIConcurrencyGroupState) error) error
23+}, n domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState)) {
24+ t.Helper()
25+ require.NoError(t, ci.UpdateConcurrency(context.Background(), n, group, func(st *domain.CIConcurrencyGroupState) error {
26+ fn(st)
27+ return nil
28+ }))
29+}
30+
31+type txer interface {
32+ UpdateConcurrency(context.Context, domain.RepoName, string, func(*domain.CIConcurrencyGroupState) error) error
33+}
34+
35+func read(t *testing.T, ci txer, n domain.RepoName, group string) (st domain.CIConcurrencyGroupState) {
36+ t.Helper()
37+ update(t, ci, n, group, func(s *domain.CIConcurrencyGroupState) { st = *s })
38+ return st
39+}
40+
41+func markOf(e domain.CIConcurrencyEntry, state domain.CIConcurrencyState) domain.CIConcurrencyMark {
42+ return domain.CIConcurrencyMark{Entry: e, Group: "Deploy", State: state}
43+}
44+
45+// A group is one record per repository, compared with domain.ConcurrencyKey, and
46+// an empty group (no record, no mark) leaves no key.
47+func TestCIConcurrencyGroupRecord(t *testing.T) {
48+ _, ci, n := ciRepo(t)
49+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
50+ s.Record.Running, s.Record.Pending = entry(1), &domain.CIConcurrencyEntry{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}
51+ })
52+ got := read(t, ci, n, "DEPLOY")
53+ require.NotNil(t, got.Record.Running)
54+ require.NotNil(t, got.Record.Pending)
55+ assert.EqualValues(t, 1, got.Record.Running.RunID)
56+ assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
57+ assert.True(t, got.Record.Pending.Cancel)
58+ assert.True(t, read(t, ci, n, "other").IsEmpty(), "another group is another record")
59+
60+ update(t, ci, n, "deploy", func(s *domain.CIConcurrencyGroupState) { *s = domain.CIConcurrencyGroupState{} })
61+ assert.True(t, read(t, ci, n, "Deploy").IsEmpty())
62+ groups, err := ci.ListConcurrency(context.Background())
63+ require.NoError(t, err)
64+ assert.Empty(t, groups, "no key is left")
65+}
66+
67+// The record and the marks of its entries are written by one batch, and a mark
68+// that a transaction removes goes in the same batch.
69+func TestCIConcurrencyMarksAreWrittenWithTheRecord(t *testing.T) {
70+ _, ci, n := ciRepo(t)
71+ run, wait := *entry(1), *entry(2)
72+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
73+ s.Record.Running, s.Record.Pending = &run, &wait
74+ s.Marks = map[string]domain.CIConcurrencyMark{
75+ run.Key(): markOf(run, domain.CIConcurrencyHolding),
76+ wait.Key(): markOf(wait, domain.CIConcurrencyPending),
77+ }
78+ })
79+ got := read(t, ci, n, "deploy")
80+ assert.Equal(t, domain.CIConcurrencyHolding, got.Marks[run.Key()].State)
81+ assert.Equal(t, domain.CIConcurrencyPending, got.Marks[wait.Key()].State)
82+ assert.Equal(t, "Deploy", got.Marks[wait.Key()].Group)
83+
84+ // The waiting entry is replaced: unnamed and marked in one transaction.
85+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
86+ s.Record.Pending = entry(3)
87+ s.Marks[wait.Key()] = markOf(wait, domain.CIConcurrencyReplaced)
88+ s.Marks[entry(3).Key()] = markOf(*entry(3), domain.CIConcurrencyPending)
89+ })
90+ got = read(t, ci, n, "Deploy")
91+ assert.EqualValues(t, 3, got.Record.Pending.RunID)
92+ assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[wait.Key()].State)
93+ assert.Len(t, got.Marks, 3)
94+
95+ // A group with marks only (nothing named) is kept, and listed with them.
96+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
97+ s.Record = domain.CIConcurrencyRecord{}
98+ delete(s.Marks, run.Key())
99+ delete(s.Marks, entry(3).Key())
100+ })
101+ groups, err := ci.ListConcurrency(context.Background())
102+ require.NoError(t, err)
103+ require.Len(t, groups, 1)
104+ assert.Equal(t, domain.ConcurrencyKey("Deploy"), groups[0].Group)
105+ assert.Equal(t, n, groups[0].Repo)
106+ assert.True(t, groups[0].State.Record.IsEmpty())
107+ assert.Equal(t, domain.CIConcurrencyReplaced, groups[0].State.Marks[wait.Key()].State)
108+
109+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) { delete(s.Marks, wait.Key()) })
110+ groups, err = ci.ListConcurrency(context.Background())
111+ require.NoError(t, err)
112+ assert.Empty(t, groups, "no record and no mark: no key")
113+}
114+
115+// The marks of "a" are not the marks of "a/b": a group may hold a slash.
116+func TestCIConcurrencyMarksBelongToTheirOwnGroup(t *testing.T) {
117+ _, ci, n := ciRepo(t)
118+ for _, g := range []string{"a", "a/b"} {
119+ e := *entry(int64(len(g)))
120+ update(t, ci, n, g, func(s *domain.CIConcurrencyGroupState) {
121+ s.Record.Running = &e
122+ s.Marks = map[string]domain.CIConcurrencyMark{e.Key(): {Entry: e, Group: g, State: domain.CIConcurrencyHolding}}
123+ })
124+ }
125+ a, ab := read(t, ci, n, "a"), read(t, ci, n, "a/b")
126+ assert.Len(t, a.Marks, 1)
127+ assert.Len(t, ab.Marks, 1)
128+ assert.Equal(t, "a", a.Marks["1/1/"].Group)
129+ assert.Equal(t, "a/b", ab.Marks["3/1/"].Group)
130+}
131+
132+// Two writers that both read an empty group never both claim it: the batch of
133+// the second carries the precondition of the state it read, so it conflicts and
134+// its callback runs again on the winner's record.
135+func TestCIConcurrencyGroupCompareAndSwap(t *testing.T) {
136+ ctx := context.Background()
137+ _, ci, n := ciRepo(t)
138+
139+ // Both first callbacks see the empty group before either write happens.
140+ var arrived sync.WaitGroup
141+ arrived.Add(2)
142+ type writer struct {
143+ claimed bool // reset by every call: it is the outcome of the last one
144+ calls int
145+ }
146+ ws := []*writer{{}, {}}
147+ var wg sync.WaitGroup
148+ for i, w := range ws {
149+ wg.Add(1)
150+ go func() {
151+ defer wg.Done()
152+ err := ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
153+ w.calls++
154+ w.claimed = false
155+ if w.calls == 1 {
156+ assert.Nil(t, s.Record.Running, "the first call sees the empty group")
157+ arrived.Done()
158+ arrived.Wait()
159+ }
160+ if s.Record.Running == nil {
161+ e := *entry(int64(i + 1))
162+ s.Record.Running, w.claimed = &e, true
163+ s.Marks = map[string]domain.CIConcurrencyMark{e.Key(): markOf(e, domain.CIConcurrencyHolding)}
164+ }
165+ return nil
166+ })
167+ assert.NoError(t, err)
168+ }()
169+ }
170+ wg.Wait()
171+
172+ claimed, winner := 0, int64(0)
173+ for i, w := range ws {
174+ if w.claimed {
175+ claimed, winner = claimed+1, int64(i+1)
176+ }
177+ }
178+ assert.Equal(t, 1, claimed, "exactly one writer holds the group")
179+ assert.Equal(t, 3, ws[0].calls+ws[1].calls, "the loser's callback ran again on the winner's record")
180+ final := read(t, ci, n, "g")
181+ require.NotNil(t, final.Record.Running)
182+ assert.Equal(t, winner, final.Record.Running.RunID, "the stored owner is the one that claimed")
183+ assert.Len(t, final.Marks, 1, "and it alone is marked")
184+}
185+
186+// A transaction that only changes a mark still conflicts with one that changed
187+// the record meanwhile: the mark rewrites the record, the version gate.
188+func TestCIConcurrencyAMarkChangeIsGatedByTheRecord(t *testing.T) {
189+ ctx := context.Background()
190+ _, ci, n := ciRepo(t)
191+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
192+ s.Record.Running = entry(1)
193+ s.Marks = map[string]domain.CIConcurrencyMark{entry(1).Key(): markOf(*entry(1), domain.CIConcurrencyHolding)}
194+ })
195+ calls := 0
196+ require.NoError(t, ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
197+ calls++
198+ if calls == 1 {
199+ update(t, ci, n, "g", func(o *domain.CIConcurrencyGroupState) { o.Record.Pending = entry(2) })
200+ }
201+ m := s.Marks[entry(1).Key()]
202+ m.State = domain.CIConcurrencyReplaced
203+ s.Marks[entry(1).Key()] = m
204+ return nil
205+ }))
206+ assert.Equal(t, 2, calls, "the first attempt was made on a stale record")
207+ got := read(t, ci, n, "g")
208+ require.NotNil(t, got.Record.Pending, "the concurrent change is kept")
209+ assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[entry(1).Key()].State)
210+}
211+
212+// A callback that leaves the group as it was writes nothing, but its batch
213+// still carries the version precondition: a writer that got in meanwhile makes
214+// it run again.
215+func TestCIConcurrencyUnchangedRecordKeepsItsPrecondition(t *testing.T) {
216+ ctx := context.Background()
217+ _, ci, n := ciRepo(t)
218+ calls := 0
219+ var second domain.CIConcurrencyGroupState
220+ err := ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
221+ calls++
222+ if calls == 1 {
223+ // Another writer creates the group while this decision is made.
224+ update(t, ci, n, "g", func(o *domain.CIConcurrencyGroupState) { o.Record.Running = entry(7) })
225+ } else {
226+ second = *s
227+ }
228+ return nil // unchanged
229+ })
230+ require.NoError(t, err)
231+ assert.Equal(t, 2, calls, "the unchanged decision was made on stale state: it ran again")
232+ require.NotNil(t, second.Record.Running)
233+ assert.EqualValues(t, 7, second.Record.Running.RunID)
234+}
235+
236+// fn may change the entries it is given in place: the update is still written,
237+// because the state is compared with its bytes as read, not with a shallow
238+// copy that shares the entries.
239+func TestCIConcurrencyInPlaceMutationIsWritten(t *testing.T) {
240+ _, ci, n := ciRepo(t)
241+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) { s.Record.Running, s.Record.Pending = entry(1), entry(2) })
242+
243+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
244+ s.Record.Running.Canceling = true // through the pointer, the struct is returned as given
245+ })
246+ got := read(t, ci, n, "g")
247+ require.NotNil(t, got.Record.Running)
248+ assert.True(t, got.Record.Running.Canceling, "the in-place change of the running entry is stored")
249+
250+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
251+ s.Record.Pending.JobID, s.Record.Pending.Cancel = "2-0-0", true
252+ })
253+ got = read(t, ci, n, "g")
254+ require.NotNil(t, got.Record.Pending)
255+ assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
256+ assert.True(t, got.Record.Pending.Cancel)
257+}
258+
259+// ListConcurrency returns every group with its repository and canonical key.
260+func TestCIConcurrencyListGroups(t *testing.T) {
261+ _, ci, n := ciRepo(t)
262+ for i, g := range []string{"Deploy", "Build/Linux"} {
263+ update(t, ci, n, g, func(s *domain.CIConcurrencyGroupState) { s.Record.Running = entry(int64(i + 1)) })
264+ }
265+ got, err := ci.ListConcurrency(context.Background())
266+ require.NoError(t, err)
267+ require.Len(t, got, 2)
268+ keys := []string{got[0].Group, got[1].Group}
269+ assert.ElementsMatch(t, []string{"DEPLOY", "BUILD/LINUX"}, keys, "a group may hold a slash")
270+ assert.Equal(t, n, got[0].Repo)
271+ assert.NotNil(t, got[0].State.Record.Running)
272+}
273+
274+// Groups and their marks go with their repository, and nothing is created under
275+// one that is gone.
276+func TestCIConcurrencyGroupsAreDeletedWithTheRepository(t *testing.T) {
277+ ctx := context.Background()
278+ s, ci, n := ciRepo(t)
279+ update(t, ci, n, "g", func(st *domain.CIConcurrencyGroupState) {
280+ st.Record.Running = entry(1)
281+ st.Marks = map[string]domain.CIConcurrencyMark{entry(1).Key(): markOf(*entry(1), domain.CIConcurrencyHolding)}
282+ })
283+ require.NoError(t, s.Delete(ctx, n))
284+ err := ci.UpdateConcurrency(ctx, n, "g", func(st *domain.CIConcurrencyGroupState) error {
285+ st.Record.Running = entry(2)
286+ return nil
287+ })
288+ assert.ErrorIs(t, err, domain.ErrNotFound, "no key is created under a repository that is gone")
289+ require.NoError(t, s.Create(ctx, domain.Repo{Name: n, Visibility: domain.VisibilityPrivate, CreatedAt: created}))
290+ assert.True(t, read(t, ci, n, "g").IsEmpty(), "the repository created again starts without groups or marks")
291+ groups, err := ci.ListConcurrency(ctx)
292+ require.NoError(t, err)
293+ assert.Empty(t, groups)
294+}
@@ -0,0 +1,294 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package kvstore_test
4+
5+import (
6+ "context"
7+ "sync"
8+ "testing"
9+
10+ "github.com/stretchr/testify/assert"
11+ "github.com/stretchr/testify/require"
12+
13+ "bnema.dev/ermite/internal/domain"
14+)
15+
16+func entry(run int64) *domain.CIConcurrencyEntry {
17+ return &domain.CIConcurrencyEntry{RunID: run, Attempt: 1}
18+}
19+
20+// update runs one transaction on a group.
21+func update(t *testing.T, ci interface {
22+ UpdateConcurrency(context.Context, domain.RepoName, string, func(*domain.CIConcurrencyGroupState) error) error
23+}, n domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState)) {
24+ t.Helper()
25+ require.NoError(t, ci.UpdateConcurrency(context.Background(), n, group, func(st *domain.CIConcurrencyGroupState) error {
26+ fn(st)
27+ return nil
28+ }))
29+}
30+
31+type txer interface {
32+ UpdateConcurrency(context.Context, domain.RepoName, string, func(*domain.CIConcurrencyGroupState) error) error
33+}
34+
35+func read(t *testing.T, ci txer, n domain.RepoName, group string) (st domain.CIConcurrencyGroupState) {
36+ t.Helper()
37+ update(t, ci, n, group, func(s *domain.CIConcurrencyGroupState) { st = *s })
38+ return st
39+}
40+
41+func markOf(e domain.CIConcurrencyEntry, state domain.CIConcurrencyState) domain.CIConcurrencyMark {
42+ return domain.CIConcurrencyMark{Entry: e, Group: "Deploy", State: state}
43+}
44+
45+// A group is one record per repository, compared with domain.ConcurrencyKey, and
46+// an empty group (no record, no mark) leaves no key.
47+func TestCIConcurrencyGroupRecord(t *testing.T) {
48+ _, ci, n := ciRepo(t)
49+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
50+ s.Record.Running, s.Record.Pending = entry(1), &domain.CIConcurrencyEntry{RunID: 2, Attempt: 1, JobID: "2-0-0", Cancel: true}
51+ })
52+ got := read(t, ci, n, "DEPLOY")
53+ require.NotNil(t, got.Record.Running)
54+ require.NotNil(t, got.Record.Pending)
55+ assert.EqualValues(t, 1, got.Record.Running.RunID)
56+ assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
57+ assert.True(t, got.Record.Pending.Cancel)
58+ assert.True(t, read(t, ci, n, "other").IsEmpty(), "another group is another record")
59+
60+ update(t, ci, n, "deploy", func(s *domain.CIConcurrencyGroupState) { *s = domain.CIConcurrencyGroupState{} })
61+ assert.True(t, read(t, ci, n, "Deploy").IsEmpty())
62+ groups, err := ci.ListConcurrency(context.Background())
63+ require.NoError(t, err)
64+ assert.Empty(t, groups, "no key is left")
65+}
66+
67+// The record and the marks of its entries are written by one batch, and a mark
68+// that a transaction removes goes in the same batch.
69+func TestCIConcurrencyMarksAreWrittenWithTheRecord(t *testing.T) {
70+ _, ci, n := ciRepo(t)
71+ run, wait := *entry(1), *entry(2)
72+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
73+ s.Record.Running, s.Record.Pending = &run, &wait
74+ s.Marks = map[string]domain.CIConcurrencyMark{
75+ run.Key(): markOf(run, domain.CIConcurrencyHolding),
76+ wait.Key(): markOf(wait, domain.CIConcurrencyPending),
77+ }
78+ })
79+ got := read(t, ci, n, "deploy")
80+ assert.Equal(t, domain.CIConcurrencyHolding, got.Marks[run.Key()].State)
81+ assert.Equal(t, domain.CIConcurrencyPending, got.Marks[wait.Key()].State)
82+ assert.Equal(t, "Deploy", got.Marks[wait.Key()].Group)
83+
84+ // The waiting entry is replaced: unnamed and marked in one transaction.
85+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
86+ s.Record.Pending = entry(3)
87+ s.Marks[wait.Key()] = markOf(wait, domain.CIConcurrencyReplaced)
88+ s.Marks[entry(3).Key()] = markOf(*entry(3), domain.CIConcurrencyPending)
89+ })
90+ got = read(t, ci, n, "Deploy")
91+ assert.EqualValues(t, 3, got.Record.Pending.RunID)
92+ assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[wait.Key()].State)
93+ assert.Len(t, got.Marks, 3)
94+
95+ // A group with marks only (nothing named) is kept, and listed with them.
96+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) {
97+ s.Record = domain.CIConcurrencyRecord{}
98+ delete(s.Marks, run.Key())
99+ delete(s.Marks, entry(3).Key())
100+ })
101+ groups, err := ci.ListConcurrency(context.Background())
102+ require.NoError(t, err)
103+ require.Len(t, groups, 1)
104+ assert.Equal(t, domain.ConcurrencyKey("Deploy"), groups[0].Group)
105+ assert.Equal(t, n, groups[0].Repo)
106+ assert.True(t, groups[0].State.Record.IsEmpty())
107+ assert.Equal(t, domain.CIConcurrencyReplaced, groups[0].State.Marks[wait.Key()].State)
108+
109+ update(t, ci, n, "Deploy", func(s *domain.CIConcurrencyGroupState) { delete(s.Marks, wait.Key()) })
110+ groups, err = ci.ListConcurrency(context.Background())
111+ require.NoError(t, err)
112+ assert.Empty(t, groups, "no record and no mark: no key")
113+}
114+
115+// The marks of "a" are not the marks of "a/b": a group may hold a slash.
116+func TestCIConcurrencyMarksBelongToTheirOwnGroup(t *testing.T) {
117+ _, ci, n := ciRepo(t)
118+ for _, g := range []string{"a", "a/b"} {
119+ e := *entry(int64(len(g)))
120+ update(t, ci, n, g, func(s *domain.CIConcurrencyGroupState) {
121+ s.Record.Running = &e
122+ s.Marks = map[string]domain.CIConcurrencyMark{e.Key(): {Entry: e, Group: g, State: domain.CIConcurrencyHolding}}
123+ })
124+ }
125+ a, ab := read(t, ci, n, "a"), read(t, ci, n, "a/b")
126+ assert.Len(t, a.Marks, 1)
127+ assert.Len(t, ab.Marks, 1)
128+ assert.Equal(t, "a", a.Marks["1/1/"].Group)
129+ assert.Equal(t, "a/b", ab.Marks["3/1/"].Group)
130+}
131+
132+// Two writers that both read an empty group never both claim it: the batch of
133+// the second carries the precondition of the state it read, so it conflicts and
134+// its callback runs again on the winner's record.
135+func TestCIConcurrencyGroupCompareAndSwap(t *testing.T) {
136+ ctx := context.Background()
137+ _, ci, n := ciRepo(t)
138+
139+ // Both first callbacks see the empty group before either write happens.
140+ var arrived sync.WaitGroup
141+ arrived.Add(2)
142+ type writer struct {
143+ claimed bool // reset by every call: it is the outcome of the last one
144+ calls int
145+ }
146+ ws := []*writer{{}, {}}
147+ var wg sync.WaitGroup
148+ for i, w := range ws {
149+ wg.Add(1)
150+ go func() {
151+ defer wg.Done()
152+ err := ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
153+ w.calls++
154+ w.claimed = false
155+ if w.calls == 1 {
156+ assert.Nil(t, s.Record.Running, "the first call sees the empty group")
157+ arrived.Done()
158+ arrived.Wait()
159+ }
160+ if s.Record.Running == nil {
161+ e := *entry(int64(i + 1))
162+ s.Record.Running, w.claimed = &e, true
163+ s.Marks = map[string]domain.CIConcurrencyMark{e.Key(): markOf(e, domain.CIConcurrencyHolding)}
164+ }
165+ return nil
166+ })
167+ assert.NoError(t, err)
168+ }()
169+ }
170+ wg.Wait()
171+
172+ claimed, winner := 0, int64(0)
173+ for i, w := range ws {
174+ if w.claimed {
175+ claimed, winner = claimed+1, int64(i+1)
176+ }
177+ }
178+ assert.Equal(t, 1, claimed, "exactly one writer holds the group")
179+ assert.Equal(t, 3, ws[0].calls+ws[1].calls, "the loser's callback ran again on the winner's record")
180+ final := read(t, ci, n, "g")
181+ require.NotNil(t, final.Record.Running)
182+ assert.Equal(t, winner, final.Record.Running.RunID, "the stored owner is the one that claimed")
183+ assert.Len(t, final.Marks, 1, "and it alone is marked")
184+}
185+
186+// A transaction that only changes a mark still conflicts with one that changed
187+// the record meanwhile: the mark rewrites the record, the version gate.
188+func TestCIConcurrencyAMarkChangeIsGatedByTheRecord(t *testing.T) {
189+ ctx := context.Background()
190+ _, ci, n := ciRepo(t)
191+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
192+ s.Record.Running = entry(1)
193+ s.Marks = map[string]domain.CIConcurrencyMark{entry(1).Key(): markOf(*entry(1), domain.CIConcurrencyHolding)}
194+ })
195+ calls := 0
196+ require.NoError(t, ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
197+ calls++
198+ if calls == 1 {
199+ update(t, ci, n, "g", func(o *domain.CIConcurrencyGroupState) { o.Record.Pending = entry(2) })
200+ }
201+ m := s.Marks[entry(1).Key()]
202+ m.State = domain.CIConcurrencyReplaced
203+ s.Marks[entry(1).Key()] = m
204+ return nil
205+ }))
206+ assert.Equal(t, 2, calls, "the first attempt was made on a stale record")
207+ got := read(t, ci, n, "g")
208+ require.NotNil(t, got.Record.Pending, "the concurrent change is kept")
209+ assert.Equal(t, domain.CIConcurrencyReplaced, got.Marks[entry(1).Key()].State)
210+}
211+
212+// A callback that leaves the group as it was writes nothing, but its batch
213+// still carries the version precondition: a writer that got in meanwhile makes
214+// it run again.
215+func TestCIConcurrencyUnchangedRecordKeepsItsPrecondition(t *testing.T) {
216+ ctx := context.Background()
217+ _, ci, n := ciRepo(t)
218+ calls := 0
219+ var second domain.CIConcurrencyGroupState
220+ err := ci.UpdateConcurrency(ctx, n, "g", func(s *domain.CIConcurrencyGroupState) error {
221+ calls++
222+ if calls == 1 {
223+ // Another writer creates the group while this decision is made.
224+ update(t, ci, n, "g", func(o *domain.CIConcurrencyGroupState) { o.Record.Running = entry(7) })
225+ } else {
226+ second = *s
227+ }
228+ return nil // unchanged
229+ })
230+ require.NoError(t, err)
231+ assert.Equal(t, 2, calls, "the unchanged decision was made on stale state: it ran again")
232+ require.NotNil(t, second.Record.Running)
233+ assert.EqualValues(t, 7, second.Record.Running.RunID)
234+}
235+
236+// fn may change the entries it is given in place: the update is still written,
237+// because the state is compared with its bytes as read, not with a shallow
238+// copy that shares the entries.
239+func TestCIConcurrencyInPlaceMutationIsWritten(t *testing.T) {
240+ _, ci, n := ciRepo(t)
241+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) { s.Record.Running, s.Record.Pending = entry(1), entry(2) })
242+
243+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
244+ s.Record.Running.Canceling = true // through the pointer, the struct is returned as given
245+ })
246+ got := read(t, ci, n, "g")
247+ require.NotNil(t, got.Record.Running)
248+ assert.True(t, got.Record.Running.Canceling, "the in-place change of the running entry is stored")
249+
250+ update(t, ci, n, "g", func(s *domain.CIConcurrencyGroupState) {
251+ s.Record.Pending.JobID, s.Record.Pending.Cancel = "2-0-0", true
252+ })
253+ got = read(t, ci, n, "g")
254+ require.NotNil(t, got.Record.Pending)
255+ assert.Equal(t, "2-0-0", got.Record.Pending.JobID)
256+ assert.True(t, got.Record.Pending.Cancel)
257+}
258+
259+// ListConcurrency returns every group with its repository and canonical key.
260+func TestCIConcurrencyListGroups(t *testing.T) {
261+ _, ci, n := ciRepo(t)
262+ for i, g := range []string{"Deploy", "Build/Linux"} {
263+ update(t, ci, n, g, func(s *domain.CIConcurrencyGroupState) { s.Record.Running = entry(int64(i + 1)) })
264+ }
265+ got, err := ci.ListConcurrency(context.Background())
266+ require.NoError(t, err)
267+ require.Len(t, got, 2)
268+ keys := []string{got[0].Group, got[1].Group}
269+ assert.ElementsMatch(t, []string{"DEPLOY", "BUILD/LINUX"}, keys, "a group may hold a slash")
270+ assert.Equal(t, n, got[0].Repo)
271+ assert.NotNil(t, got[0].State.Record.Running)
272+}
273+
274+// Groups and their marks go with their repository, and nothing is created under
275+// one that is gone.
276+func TestCIConcurrencyGroupsAreDeletedWithTheRepository(t *testing.T) {
277+ ctx := context.Background()
278+ s, ci, n := ciRepo(t)
279+ update(t, ci, n, "g", func(st *domain.CIConcurrencyGroupState) {
280+ st.Record.Running = entry(1)
281+ st.Marks = map[string]domain.CIConcurrencyMark{entry(1).Key(): markOf(*entry(1), domain.CIConcurrencyHolding)}
282+ })
283+ require.NoError(t, s.Delete(ctx, n))
284+ err := ci.UpdateConcurrency(ctx, n, "g", func(st *domain.CIConcurrencyGroupState) error {
285+ st.Record.Running = entry(2)
286+ return nil
287+ })
288+ assert.ErrorIs(t, err, domain.ErrNotFound, "no key is created under a repository that is gone")
289+ require.NoError(t, s.Create(ctx, domain.Repo{Name: n, Visibility: domain.VisibilityPrivate, CreatedAt: created}))
290+ assert.True(t, read(t, ci, n, "g").IsEmpty(), "the repository created again starts without groups or marks")
291+ groups, err := ci.ListConcurrency(ctx)
292+ require.NoError(t, err)
293+ assert.Empty(t, groups)
294+}
M internal/adapters/rpc/api.go +1 -1
@@ -34,7 +34,7 @@ const RunnerMaxRequestBytes = MaxRequestBytes + 64<<10
3434 // always implements, as GetVersion lists them. FeaturePasskeys comes on top
3535 // when Config.Passkeys is set, and FeatureCISecrets when the settings services
3636 // are.
37-var Features = []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries"}
37+var Features = []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency"}
3838
3939 // FeaturePasskeys is listed by GetVersion when passkeys are available.
4040 const FeaturePasskeys = "passkeys"
@@ -34,7 +34,7 @@ const RunnerMaxRequestBytes = MaxRequestBytes + 64<<10
34 // always implements, as GetVersion lists them. FeaturePasskeys comes on top34 // always implements, as GetVersion lists them. FeaturePasskeys comes on top
35 // when Config.Passkeys is set, and FeatureCISecrets when the settings services35 // when Config.Passkeys is set, and FeatureCISecrets when the settings services
36 // are.36 // are.
37-var Features = []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries"}37+var Features = []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency"}
38 38
39 // FeaturePasskeys is listed by GetVersion when passkeys are available.39 // FeaturePasskeys is listed by GetVersion when passkeys are available.
40 const FeaturePasskeys = "passkeys"40 const FeaturePasskeys = "passkeys"
M internal/adapters/rpc/ci_convert.go +3 -1
@@ -18,6 +18,8 @@ func statusToPB(s domain.CIStatus) pb.CiStatus {
1818 return pb.CiStatus_CI_STATUS_IN_PROGRESS
1919 case domain.CICompleted:
2020 return pb.CiStatus_CI_STATUS_COMPLETED
21+ case domain.CIPending:
22+ return pb.CiStatus_CI_STATUS_PENDING
2123 }
2224 return pb.CiStatus_CI_STATUS_UNSPECIFIED
2325 }
@@ -181,7 +183,7 @@ func jobToPB(j domain.CIJob, lines int64) *pb.RunJob {
181183 Id: j.ID, RunId: j.RunID, Attempt: j.Attempt, Repo: j.Repo.String(), Key: j.Key, Name: j.Name,
182184 Status: statusToPB(j.Status()), Conclusion: conclusionToPB(j.Conclusion), Error: j.Error,
183185 Labels: j.Labels, Needs: j.Needs, RunnerId: j.RunnerID, RunnerName: j.RunnerName, Isolation: isolationToPB(j.Isolation),
184- Steps: stepsToPB(j.Steps, lines), CreatedAt: timestamp(j.CreatedAt), StartedAt: timestamp(j.StartedAt),
186+ ConcurrencyGroup: j.Concurrency.Group, Steps: stepsToPB(j.Steps, lines), CreatedAt: timestamp(j.CreatedAt), StartedAt: timestamp(j.StartedAt),
185187 CompletedAt: timestamp(j.CompletedAt),
186188 Version: int64(j.Version), //nolint:gosec // a store sequence, far below 2^63
187189 }
@@ -18,6 +18,8 @@ func statusToPB(s domain.CIStatus) pb.CiStatus {
18 return pb.CiStatus_CI_STATUS_IN_PROGRESS18 return pb.CiStatus_CI_STATUS_IN_PROGRESS
19 case domain.CICompleted:19 case domain.CICompleted:
20 return pb.CiStatus_CI_STATUS_COMPLETED20 return pb.CiStatus_CI_STATUS_COMPLETED
21+ case domain.CIPending:
22+ return pb.CiStatus_CI_STATUS_PENDING
21 }23 }
22 return pb.CiStatus_CI_STATUS_UNSPECIFIED24 return pb.CiStatus_CI_STATUS_UNSPECIFIED
23 }25 }
@@ -181,7 +183,7 @@ func jobToPB(j domain.CIJob, lines int64) *pb.RunJob {
181 Id: j.ID, RunId: j.RunID, Attempt: j.Attempt, Repo: j.Repo.String(), Key: j.Key, Name: j.Name,183 Id: j.ID, RunId: j.RunID, Attempt: j.Attempt, Repo: j.Repo.String(), Key: j.Key, Name: j.Name,
182 Status: statusToPB(j.Status()), Conclusion: conclusionToPB(j.Conclusion), Error: j.Error,184 Status: statusToPB(j.Status()), Conclusion: conclusionToPB(j.Conclusion), Error: j.Error,
183 Labels: j.Labels, Needs: j.Needs, RunnerId: j.RunnerID, RunnerName: j.RunnerName, Isolation: isolationToPB(j.Isolation),185 Labels: j.Labels, Needs: j.Needs, RunnerId: j.RunnerID, RunnerName: j.RunnerName, Isolation: isolationToPB(j.Isolation),
184- Steps: stepsToPB(j.Steps, lines), CreatedAt: timestamp(j.CreatedAt), StartedAt: timestamp(j.StartedAt),186+ ConcurrencyGroup: j.Concurrency.Group, Steps: stepsToPB(j.Steps, lines), CreatedAt: timestamp(j.CreatedAt), StartedAt: timestamp(j.StartedAt),
185 CompletedAt: timestamp(j.CompletedAt),187 CompletedAt: timestamp(j.CompletedAt),
186 Version: int64(j.Version), //nolint:gosec // a store sequence, far below 2^63188 Version: int64(j.Version), //nolint:gosec // a store sequence, far below 2^63
187 }189 }
M internal/adapters/web/actions_pages.go +3 -1
@@ -50,6 +50,8 @@ func ciState(status domain.CIStatus, c domain.CIConclusion) string {
5050 switch status {
5151 case domain.CIQueued:
5252 return "queued"
53+ case domain.CIPending:
54+ return "pending"
5355 case domain.CIInProgress:
5456 return "running"
5557 }
@@ -75,7 +77,7 @@ func ciState(status domain.CIStatus, c domain.CIConclusion) string {
7577 // ciKind groups the states into the few that have an icon and a colour.
7678 func ciKind(status domain.CIStatus, c domain.CIConclusion) string {
7779 switch {
78- case status == domain.CIQueued:
80+ case status == domain.CIQueued, status == domain.CIPending:
7981 return "queued"
8082 case status == domain.CIInProgress:
8183 return "running"
@@ -50,6 +50,8 @@ func ciState(status domain.CIStatus, c domain.CIConclusion) string {
50 switch status {50 switch status {
51 case domain.CIQueued:51 case domain.CIQueued:
52 return "queued"52 return "queued"
53+ case domain.CIPending:
54+ return "pending"
53 case domain.CIInProgress:55 case domain.CIInProgress:
54 return "running"56 return "running"
55 }57 }
@@ -75,7 +77,7 @@ func ciState(status domain.CIStatus, c domain.CIConclusion) string {
75 // ciKind groups the states into the few that have an icon and a colour.77 // ciKind groups the states into the few that have an icon and a colour.
76 func ciKind(status domain.CIStatus, c domain.CIConclusion) string {78 func ciKind(status domain.CIStatus, c domain.CIConclusion) string {
77 switch {79 switch {
78- case status == domain.CIQueued:80+ case status == domain.CIQueued, status == domain.CIPending:
79 return "queued"81 return "queued"
80 case status == domain.CIInProgress:82 case status == domain.CIInProgress:
81 return "running"83 return "running"
M internal/app/api_test.go +1 -1
@@ -196,7 +196,7 @@ func TestAC2ReadAccess(t *testing.T) {
196196 for _, who := range allPrincipals {
197197 resp, err := r.who[who].server.GetVersion(ctx, connect.NewRequest(&pb.GetVersionRequest{}))
198198 require.NoError(t, err, who)
199- assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.secrets"}, resp.Msg.GetFeatures(), who)
199+ assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.secrets"}, resp.Msg.GetFeatures(), who)
200200 assert.NotEmpty(t, resp.Msg.GetVersion())
201201 }
202202 })
@@ -196,7 +196,7 @@ func TestAC2ReadAccess(t *testing.T) {
196 for _, who := range allPrincipals {196 for _, who := range allPrincipals {
197 resp, err := r.who[who].server.GetVersion(ctx, connect.NewRequest(&pb.GetVersionRequest{}))197 resp, err := r.who[who].server.GetVersion(ctx, connect.NewRequest(&pb.GetVersionRequest{}))
198 require.NoError(t, err, who)198 require.NoError(t, err, who)
199- assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.secrets"}, resp.Msg.GetFeatures(), who)199+ assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.secrets"}, resp.Msg.GetFeatures(), who)
200 assert.NotEmpty(t, resp.Msg.GetVersion())200 assert.NotEmpty(t, resp.Msg.GetVersion())
201 }201 }
202 })202 })
A internal/app/ci_concurrency_test.go +1112 -0
@@ -0,0 +1,1112 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package app
4+
5+import (
6+ "context"
7+ "fmt"
8+ "net/http/httptest"
9+ "strings"
10+ "sync/atomic"
11+ "testing"
12+ "time"
13+
14+ "bnema.dev/ermite/internal/domain"
15+
16+ "connectrpc.com/connect"
17+ "github.com/stretchr/testify/assert"
18+ "github.com/stretchr/testify/require"
19+
20+ pb "bnema.dev/ermite-api/gen/ermite"
21+ "bnema.dev/ermite/internal/ports"
22+ "bnema.dev/ermite/internal/usecase"
23+)
24+
25+// concurrencyHarness is a CI harness with fast deadlines and one runner.
26+func concurrencyHarness(t *testing.T) (*ciHarness, *fakeRunner) {
27+ t.Helper()
28+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
29+ cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
30+ })
31+ return c, c.registerRunner("alpha", "ubuntu-latest")
32+}
33+
34+// workflowAt is a one-job workflow on main with extra top-level and job lines.
35+func workflowAt(top, job string) string {
36+ return "name: ci\non:\n push:\n branches: [main]\n" + top +
37+ "jobs:\n build:\n runs-on: ubuntu-latest\n" + job +
38+ " steps:\n - run: echo hello\n"
39+}
40+
41+// pushN pushes a new commit with the workflow and returns the new run.
42+func (c *ciHarness) pushN(n int, wf string) *pb.Run {
43+ c.t.Helper()
44+ before := len(c.runs())
45+ // A comment makes each push a new commit even when the workflow repeats.
46+ c.push("main", map[string]string{".github/workflows/ci.yml": fmt.Sprintf("# push %d\n%s", n, wf)}, "")
47+ c.waitFor("the run exists", func() bool { return len(c.runs()) == before+1 })
48+ return c.runs()[0] // newest first
49+}
50+
51+func (c *ciHarness) jobsOf(run *pb.Run) []*pb.RunJob { return c.getRun(run.GetId()).GetJobs() }
52+
53+func (c *ciHarness) waitJobs(what string, run *pb.Run, cond func([]*pb.RunJob) bool) {
54+ c.t.Helper()
55+ c.waitFor(what, func() bool { return cond(c.jobsOf(run)) })
56+}
57+
58+func allStatus(s pb.CiStatus) func([]*pb.RunJob) bool {
59+ return func(js []*pb.RunJob) bool {
60+ for _, j := range js {
61+ if j.GetStatus() != s {
62+ return false
63+ }
64+ }
65+ return len(js) > 0
66+ }
67+}
68+
69+func (c *ciHarness) conclusionOf(run *pb.Run) pb.CiConclusion {
70+ return c.getRun(run.GetId()).GetRun().GetConclusion()
71+}
72+
73+func (c *ciHarness) cancelAnnotation(run *pb.Run, group string) bool {
74+ for _, j := range c.jobsOf(run) {
75+ for _, a := range c.annotations(run.GetId(), j.GetId()).GetAnnotations() {
76+ if a.GetAnnotation().GetMessage() == "Canceling since a higher priority waiting request for '"+group+"' exists" {
77+ return true
78+ }
79+ }
80+ }
81+ return false
82+}
83+
84+const wfGroup = "concurrency:\n group: Deploy\n"
85+
86+// AC4: a workflow-level group lets one run go and one wait; a newer waiting run
87+// replaces the older one, which ends cancelled with GitHub's annotation; the
88+// group is handed over when the running one completes.
89+func TestCIWorkflowConcurrencyPendingReplacement(t *testing.T) {
90+ c, runner := concurrencyHarness(t)
91+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
92+ job1 := runner.await()
93+ require.Equal(t, r1.GetId(), job1.GetRunId())
94+
95+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
96+ c.waitJobs("the second run is pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
97+ assert.Equal(t, "Deploy", c.jobsOf(r2)[0].GetConcurrencyGroup())
98+ assert.Equal(t, pb.CiStatus_CI_STATUS_QUEUED, c.getRun(r2.GetId()).GetRun().GetStatus(), "a run is queued, only its jobs are pending")
99+ // The pending job is not assignable.
100+ j, err := runner.fetch("")
101+ require.NoError(t, err)
102+ assert.Nil(t, j)
103+
104+ r3 := c.pushN(3, workflowAt(wfGroup, ""))
105+ c.waitFor("the older pending run is cancelled", func() bool {
106+ return c.getRun(r2.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
107+ })
108+ assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2))
109+ assert.True(t, c.cancelAnnotation(r2, "Deploy"))
110+ c.waitJobs("the third run is pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
111+ assert.NotEqual(t, pb.CiStatus_CI_STATUS_COMPLETED, c.getRun(r1.GetId()).GetRun().GetStatus(), "the running run is left alone")
112+
113+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
114+ job3 := runner.await()
115+ assert.Equal(t, r3.GetId(), job3.GetRunId(), "the group goes to the waiting run")
116+ runner.runJob(job3, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
117+ c.waitFor("the third run completes", func() bool {
118+ return c.getRun(r3.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
119+ })
120+
121+ // The group is free: a new run takes it at once.
122+ r4 := c.pushN(4, workflowAt(wfGroup, ""))
123+ job4 := runner.await()
124+ assert.Equal(t, r4.GetId(), job4.GetRunId())
125+}
126+
127+// AC4: cancel-in-progress cancels the running run, whose runner is told through
128+// the cancel mechanism of a user cancel; an expression decides it, and groups
129+// are compared without regard to case.
130+func TestCIWorkflowConcurrencyCancelInProgress(t *testing.T) {
131+ for _, tt := range []struct{ name, cancel string }{
132+ {"true", "true"},
133+ {"expression", "${{ github.ref == 'refs/heads/main' }}"},
134+ } {
135+ t.Run(tt.name, func(t *testing.T) {
136+ c, runner := concurrencyHarness(t)
137+ r1 := c.pushN(1, workflowAt("concurrency:\n group: Release\n cancel-in-progress: "+tt.cancel+"\n", ""))
138+ job1 := runner.await()
139+ watch, err := runner.jobAPI(job1).runner.WatchJob(c.ctx, connect.NewRequest(&pb.WatchJobRequest{JobId: job1.GetJobId()}))
140+ require.NoError(t, err)
141+
142+ r2 := c.pushN(2, workflowAt("concurrency:\n group: RELEASE\n cancel-in-progress: "+tt.cancel+"\n", ""))
143+ var cancel *pb.CancelRequest
144+ for watch.Receive() {
145+ if m := watch.Msg().GetCancel(); m != nil {
146+ cancel = m
147+ break
148+ }
149+ }
150+ require.NotNil(t, cancel, "the running job is asked to stop")
151+ assert.Equal(t, pb.CancelReason_CANCEL_REASON_USER, cancel.GetReason())
152+ assert.Contains(t, cancel.GetMessage(), "higher priority waiting request for 'RELEASE'")
153+
154+ _, err = runner.jobAPI(job1).runner.CompleteJob(c.ctx, connect.NewRequest(&pb.CompleteJobRequest{
155+ JobId: job1.GetJobId(), Conclusion: pb.CiConclusion_CI_CONCLUSION_CANCELLED,
156+ }))
157+ require.NoError(t, err)
158+ c.waitFor("the first run is cancelled", func() bool { return c.conclusionOf(r1) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
159+ job2 := runner.await()
160+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "the waiting run takes the group")
161+ })
162+ }
163+ t.Run("false expression only queues", func(t *testing.T) {
164+ c, runner := concurrencyHarness(t)
165+ top := "concurrency:\n group: G\n cancel-in-progress: ${{ github.ref == 'refs/heads/other' }}\n"
166+ c.pushN(1, workflowAt(top, ""))
167+ job1 := runner.await()
168+ r2 := c.pushN(2, workflowAt(top, ""))
169+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
170+ runner.heartbeat(job1)
171+ time.Sleep(200 * time.Millisecond)
172+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(c.runs()[1]))
173+ })
174+}
175+
176+const matrixJob = ` strategy:
177+ matrix:
178+ n: [1, 2]
179+ concurrency:
180+ group: shared-${{ 'X' }}
181+`
182+
183+// AC4: the legs of a matrix share a job-level group: one runs, one waits, and
184+// a leg is freed when the other ends.
185+func TestCIJobConcurrencyMatrixLegsShareAGroup(t *testing.T) {
186+ c, runner := concurrencyHarness(t)
187+ r := c.pushN(1, workflowAt("", matrixJob))
188+ job1 := runner.await()
189+ c.waitJobs("one leg is pending", r, func(js []*pb.RunJob) bool {
190+ pending, other := 0, 0
191+ for _, j := range js {
192+ switch j.GetStatus() {
193+ case pb.CiStatus_CI_STATUS_PENDING:
194+ pending++
195+ default:
196+ other++
197+ }
198+ }
199+ return len(js) == 2 && pending == 1 && other == 1
200+ })
201+ j, err := runner.fetch("")
202+ require.NoError(t, err)
203+ assert.Nil(t, j, "the pending leg is not assignable")
204+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
205+ job2 := runner.await()
206+ assert.NotEqual(t, job1.GetJobId(), job2.GetJobId())
207+ runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
208+ c.waitFor("the run completes", func() bool { return c.getRun(r.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
209+ assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_SUCCESS, c.conclusionOf(r))
210+}
211+
212+// AC4: a job-level group is shared between runs, case-insensitively; the third
213+// waiting job replaces the second, which ends cancelled with the annotation.
214+func TestCIJobConcurrencyAcrossRuns(t *testing.T) {
215+ c, runner := concurrencyHarness(t)
216+ group := func(g string) string { return " concurrency: " + g + "\n" }
217+ r1 := c.pushN(1, workflowAt("", group("Build-Lock")))
218+ job1 := runner.await()
219+ r2 := c.pushN(2, workflowAt("", group("build-lock")))
220+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
221+ assert.Equal(t, "build-lock", c.jobsOf(r2)[0].GetConcurrencyGroup())
222+ r3 := c.pushN(3, workflowAt("", group("BUILD-LOCK")))
223+ c.waitFor("the second job is cancelled", func() bool { return c.conclusionOf(r2) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
224+ assert.True(t, c.cancelAnnotation(r2, "BUILD-LOCK"), "the group is named as the newer request wrote it")
225+ c.waitJobs("pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
226+ _ = r1
227+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
228+ job3 := runner.await()
229+ assert.Equal(t, r3.GetId(), job3.GetRunId())
230+}
231+
232+// AC4: the group is freed when its run is cancelled, and when a re-run joins
233+// the group again it waits behind the run that took it meanwhile.
234+func TestCIConcurrencyFreedByCancelAndRerunRejoins(t *testing.T) {
235+ c, runner := concurrencyHarness(t)
236+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
237+ job1 := runner.await()
238+ _, err := c.root.run.CancelRun(c.ctx, connect.NewRequest(&pb.CancelRunRequest{Repo: "proj", RunId: r1.GetId()}))
239+ require.NoError(t, err)
240+ runner.heartbeat(job1)
241+ c.waitFor("the cancelled run completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
242+
243+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
244+ job2 := runner.await()
245+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "a cancelled run freed the group")
246+
247+ _, err = c.root.run.RerunRun(c.ctx, connect.NewRequest(&pb.RerunRunRequest{Repo: "proj", RunId: r1.GetId()}))
248+ require.NoError(t, err)
249+ c.waitJobs("the re-run waits for the group", r1, allStatus(pb.CiStatus_CI_STATUS_PENDING))
250+ runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
251+ job1b := runner.await()
252+ assert.Equal(t, r1.GetId(), job1b.GetRunId(), "the re-run takes the group once it is free")
253+}
254+
255+// AC4: a job lost to a missing heartbeat frees its group.
256+func TestCIConcurrencyFreedByLostJob(t *testing.T) {
257+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
258+ cfg.HeartbeatDeadline, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
259+ })
260+ runner := c.registerRunner("alpha", "ubuntu-latest")
261+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
262+ job1 := runner.await()
263+ runner.heartbeat(job1)
264+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
265+ c.waitFor("the lost job fails", func() bool { return c.conclusionOf(r1) == pb.CiConclusion_CI_CONCLUSION_FAILURE })
266+ job2 := runner.await()
267+ assert.Equal(t, r2.GetId(), job2.GetRunId())
268+}
269+
270+// A group that evaluates to nothing fails the job as GitHub does, and
271+// queue: max is refused.
272+func TestCIConcurrencyErrors(t *testing.T) {
273+ c, runner := concurrencyHarness(t)
274+ _ = runner
275+ r1 := c.pushN(1, workflowAt("", " concurrency:\n group: ${{ vars.NOPE }}\n"))
276+ c.waitJobs("failed", r1, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
277+ assert.Contains(t, c.jobsOf(r1)[0].GetError(), "concurrency group name cannot be empty")
278+
279+ r3 := c.pushN(3, workflowAt("", " concurrency:\n group: g\n queue: max\n cancel-in-progress: true\n"))
280+ c.waitJobs("failed", r3, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
281+ assert.Equal(t, "'queue: max' cannot be combined with 'cancel-in-progress: true'.", c.jobsOf(r3)[0].GetError())
282+
283+ r2 := c.pushN(2, workflowAt("", " concurrency:\n group: g\n queue: max\n"))
284+ c.waitJobs("failed", r2, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
285+ assert.Equal(t, "not supported yet: queue: max", strings.TrimSpace(c.jobsOf(r2)[0].GetError()))
286+}
287+
288+// holdGroup writes a group behind the use case's back, as a crash would leave
289+// it: the entries are named and marked whatever the state of their runs.
290+func (c *ciHarness) holdGroup(group string, running, pending *domain.CIConcurrencyEntry) {
291+ c.t.Helper()
292+ require.NoError(c.t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), group, func(st *domain.CIConcurrencyGroupState) error {
293+ st.Record.Running, st.Record.Pending = running, pending
294+ st.Marks = map[string]domain.CIConcurrencyMark{}
295+ if running != nil {
296+ st.Marks[running.Key()] = domain.CIConcurrencyMark{Entry: *running, Group: group, State: domain.CIConcurrencyHolding}
297+ }
298+ if pending != nil {
299+ st.Marks[pending.Key()] = domain.CIConcurrencyMark{Entry: *pending, Group: group, State: domain.CIConcurrencyPending}
300+ }
301+ return nil
302+ }))
303+}
304+
305+func (c *ciHarness) groupState(group string) (st domain.CIConcurrencyGroupState) {
306+ c.t.Helper()
307+ require.NoError(c.t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), group, func(s *domain.CIConcurrencyGroupState) error {
308+ st = *s
309+ return nil
310+ }))
311+ return st
312+}
313+
314+func (c *ciHarness) groupRecord(group string) domain.CIConcurrencyRecord {
315+ return c.groupState(group).Record
316+}
317+
318+// markState is the mark of an entry in a group, "" when it has none.
319+func (c *ciHarness) markState(group string, e domain.CIConcurrencyEntry) domain.CIConcurrencyState {
320+ return c.groupState(group).Marks[e.Key()].State
321+}
322+
323+// A crash between the write of an ended job and the release of its group leaves
324+// the record holding it: the next run to ask finds the entry dead and takes the
325+// group.
326+func TestCIConcurrencyHealsAfterACrash(t *testing.T) {
327+ c, runner := concurrencyHarness(t)
328+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
329+ runner.runJob(runner.await(), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
330+ c.waitFor("the first run completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
331+ c.waitFor("the group is released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
332+ c.holdGroup("Deploy", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil) // the release that did not happen
333+
334+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
335+ job2 := runner.await()
336+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "the dead entry does not hold the group")
337+ rec := c.groupRecord("Deploy")
338+ require.NotNil(t, rec.Running)
339+ assert.Equal(t, r2.GetId(), rec.Running.RunID)
340+}
341+
342+// A run that is deleted (the retention sweep, not its repository) frees the
343+// group it held and the one it waited in.
344+func TestCIConcurrencyHealsAfterRunDeletion(t *testing.T) {
345+ c, runner := concurrencyHarness(t)
346+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
347+ runner.runJob(runner.await(), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
348+ c.waitFor("completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
349+ c.waitFor("released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
350+ ghost := &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}
351+ c.holdGroup("Deploy", ghost, nil)
352+ ids, err := c.forge.store.CI().Sweep(c.ctx, time.Now().Add(time.Hour))
353+ require.NoError(t, err)
354+ require.Contains(t, ids, r1.GetId(), "the run is deleted")
355+
356+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
357+ job2 := runner.await()
358+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "a deleted run does not hold the group")
359+}
360+
361+// flakyStore fails the writes that match, as many times as fails says. The
362+// failure is returned before anything is written.
363+type flakyStore struct {
364+ ports.CIStore
365+ match func(ports.CIWrite) bool
366+ fails atomic.Int32
367+ hits atomic.Int32
368+}
369+
370+func (f *flakyStore) Apply(ctx context.Context, w ports.CIWrite) (uint64, error) {
371+ if f.match != nil && f.match(w) && f.fails.Add(-1) >= 0 {
372+ f.hits.Add(1)
373+ return 0, domain.ErrConflict
374+ }
375+ return f.CIStore.Apply(ctx, w)
376+}
377+
378+// flakyConcurrencyHarness is a harness whose store fails the writes fs.match
379+// selects. retries is CIConfig.ConcurrencyRetries: 0 keeps the default, a
380+// negative number leaves the recovery of a lost delivery to the reconciliation.
381+func flakyConcurrencyHarness(t *testing.T, retries int) (*ciHarness, *fakeRunner, *flakyStore) {
382+ t.Helper()
383+ fs := &flakyStore{}
384+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
385+ cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
386+ cfg.ConcurrencyRetries = retries
387+ fs.CIStore, cfg.Store = cfg.Store, fs
388+ })
389+ return c, c.registerRunner("alpha", "ubuntu-latest"), fs
390+}
391+
392+// endsCancelled matches the write that ends a job of the run as cancelled.
393+func endsCancelled(runID int64) func(ports.CIWrite) bool {
394+ return func(w ports.CIWrite) bool {
395+ for _, j := range w.Jobs {
396+ if j.RunID == runID && j.Ended() && j.Conclusion == domain.CICancelled {
397+ return true
398+ }
399+ }
400+ return false
401+ }
402+}
403+
404+// A replaced pending run whose cancel write fails once is cancelled by the
405+// retry.
406+func TestCIConcurrencyCancelSurvivesAFailedDelivery(t *testing.T) {
407+ c, runner, fs := flakyConcurrencyHarness(t, 0)
408+ c.pushN(1, workflowAt(wfGroup, ""))
409+ job1 := runner.await()
410+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
411+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
412+
413+ fs.match = endsCancelled(r2.GetId())
414+ fs.fails.Store(1)
415+ r3 := c.pushN(3, workflowAt(wfGroup, ""))
416+ c.waitFor("the first delivery failed", func() bool { return fs.hits.Load() == 1 })
417+ rec := c.groupRecord("Deploy")
418+ require.NotNil(t, rec.Pending)
419+ assert.Equal(t, r3.GetId(), rec.Pending.RunID)
420+ c.waitFor("the replaced run is cancelled by the retry", func() bool { return c.conclusionOf(r2) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
421+ assert.True(t, c.cancelAnnotation(r2, "Deploy"))
422+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
423+ assert.Equal(t, r3.GetId(), runner.await().GetRunId())
424+}
425+
426+// Nothing is bounded: more replacements than any list could hold, every cancel
427+// write failing, and after the store works again the reconciliation discovers
428+// each victim through its "replaced" mark in the group, and cancels it.
429+func TestCIConcurrencyReconcileCancelsEveryReplacedVictim(t *testing.T) {
430+ const replaced = 35
431+ c, runner, fs := flakyConcurrencyHarness(t, -1)
432+ c.pushN(1, workflowAt(wfGroup, ""))
433+ runner.await()
434+
435+ // Each new run replaces the one that waited before it; no cancel lands.
436+ fs.match = func(w ports.CIWrite) bool {
437+ for _, j := range w.Jobs {
438+ if j.Ended() && j.Conclusion == domain.CICancelled {
439+ return true
440+ }
441+ }
442+ return false
443+ }
444+ fs.fails.Store(1 << 20)
445+ var victims []*pb.Run
446+ prev := c.pushN(2, workflowAt(wfGroup, ""))
447+ c.waitJobs("pending", prev, allStatus(pb.CiStatus_CI_STATUS_PENDING))
448+ for i := range replaced {
449+ next := c.pushN(3+i, workflowAt(wfGroup, ""))
450+ c.waitFor("the newer run takes the waiting place", func() bool {
451+ rec := c.groupRecord("Deploy")
452+ return rec.Pending != nil && rec.Pending.RunID == next.GetId()
453+ })
454+ c.waitJobs("the newer run is pending", next, allStatus(pb.CiStatus_CI_STATUS_PENDING))
455+ victims = append(victims, prev)
456+ prev = next
457+ }
458+ for _, v := range victims {
459+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(v), "no cancel landed yet")
460+ }
461+ // The test has no in-process retry (ConcurrencyRetries is negative): only
462+ // the reconciliation can recover the victims.
463+ fs.fails.Store(0)
464+ c.forge.CI.ReconcileConcurrency(c.ctx)
465+ for _, v := range victims {
466+ c.waitFor("a replaced run is cancelled", func() bool { return c.conclusionOf(v) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
467+ assert.True(t, c.cancelAnnotation(v, "Deploy"))
468+ }
469+ c.waitJobs("the last one still waits", prev, allStatus(pb.CiStatus_CI_STATUS_PENDING))
470+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(prev))
471+}
472+
473+// A running entry that cancel-in-progress replaced stays named with Canceling
474+// set; when its cancel is lost, the reconciliation asks again.
475+func TestCIConcurrencyReconcileAsksARunningVictimAgain(t *testing.T) {
476+ c, runner, fs := flakyConcurrencyHarness(t, -1)
477+ top := "concurrency:\n group: Deploy\n cancel-in-progress: true\n"
478+ r1 := c.pushN(1, workflowAt(top, ""))
479+ job1 := runner.await()
480+ watch, err := runner.jobAPI(job1).runner.WatchJob(c.ctx, connect.NewRequest(&pb.WatchJobRequest{JobId: job1.GetJobId()}))
481+ require.NoError(t, err)
482+
483+ // The cancel of the running run cannot be written.
484+ fs.match = func(w ports.CIWrite) bool {
485+ for _, r := range w.Runs {
486+ if r.ID == r1.GetId() && r.CancelRequested {
487+ return true
488+ }
489+ }
490+ return false
491+ }
492+ fs.fails.Store(1 << 20)
493+ c.pushN(2, workflowAt(top, ""))
494+ c.waitFor("the cancel fails", func() bool { return fs.hits.Load() >= 1 })
495+ rec := c.groupRecord("Deploy")
496+ require.NotNil(t, rec.Running)
497+ assert.True(t, rec.Running.Canceling, "the victim stays named, marked")
498+ assert.Equal(t, r1.GetId(), rec.Running.RunID)
499+
500+ fs.fails.Store(0)
501+ c.forge.CI.ReconcileConcurrency(c.ctx)
502+ var cancel *pb.CancelRequest
503+ for watch.Receive() {
504+ if m := watch.Msg().GetCancel(); m != nil {
505+ cancel = m
506+ break
507+ }
508+ }
509+ require.NotNil(t, cancel, "the runner is told after the reconciliation")
510+ // The runner stops: its watch ends with the job.
511+ _, err = runner.jobAPI(job1).runner.CompleteJob(c.ctx, connect.NewRequest(&pb.CompleteJobRequest{
512+ JobId: job1.GetJobId(), Conclusion: pb.CiConclusion_CI_CONCLUSION_CANCELLED,
513+ }))
514+ require.NoError(t, err)
515+}
516+
517+// Crash between the write of the group and the write of the entry: the group
518+// names a pending run whose own record does not say it waits yet. It is not
519+// stale and not cancelled; the reconciliation plans it, which marks it
520+// pending, and the group goes on to it when the holder ends.
521+func TestCIConcurrencyCrashBetweenGroupAndEntry(t *testing.T) {
522+ c, runner := concurrencyHarness(t)
523+ c.pushN(1, workflowAt(wfGroup, ""))
524+ job1 := runner.await()
525+
526+ // Stop the planning of the second run from marking it, as a crash would:
527+ // name it in the group by hand, behind the first.
528+ store := c.forge.store.CI()
529+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
530+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
531+ run, err := store.GetRun(c.ctx, c.h("proj"), r2.GetId())
532+ require.NoError(t, err)
533+ run.Concurrency.State = domain.CIConcurrencyIdle
534+ jobs, err := store.ListJobs(c.ctx, r2.GetId(), 1)
535+ require.NoError(t, err)
536+ for i := range jobs {
537+ jobs[i].Concurrency.State = domain.CIConcurrencyIdle
538+ }
539+ _, err = store.Apply(c.ctx, ports.CIWrite{Runs: []domain.CIRun{run}, Jobs: jobs})
540+ require.NoError(t, err)
541+ rec := c.groupRecord("Deploy")
542+ require.NotNil(t, rec.Pending)
543+ require.Equal(t, r2.GetId(), rec.Pending.RunID, "the group names the run")
544+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", *rec.Pending), "and marks it, in the same batch")
545+
546+ c.forge.CI.ReconcileConcurrency(c.ctx)
547+ c.waitFor("the entry is marked pending again", func() bool {
548+ r, err := store.GetRun(c.ctx, c.h("proj"), r2.GetId())
549+ return err == nil && r.Concurrency.State == domain.CIConcurrencyPending
550+ })
551+ time.Sleep(200 * time.Millisecond)
552+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2), "a named entry is not a replaced one")
553+ rec = c.groupRecord("Deploy")
554+ require.NotNil(t, rec.Pending)
555+ assert.Equal(t, r2.GetId(), rec.Pending.RunID)
556+
557+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
558+ assert.Equal(t, r2.GetId(), runner.await().GetRunId())
559+}
560+
561+// holdsGroup matches the write that tells a run it holds its group.
562+func holdsGroup(runID int64) func(ports.CIWrite) bool {
563+ return func(w ports.CIWrite) bool {
564+ for _, r := range w.Runs {
565+ if r.ID == runID && r.Concurrency.State == domain.CIConcurrencyHolding {
566+ return true
567+ }
568+ }
569+ return false
570+ }
571+}
572+
573+// HIGH-2: a promotion whose planning fails once is tried again.
574+func TestCIConcurrencyPromotionIsRetried(t *testing.T) {
575+ c, runner, fs := flakyConcurrencyHarness(t, 0)
576+ c.pushN(1, workflowAt(wfGroup, ""))
577+ job1 := runner.await()
578+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
579+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
580+
581+ fs.match = holdsGroup(r2.GetId())
582+ fs.fails.Store(1)
583+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
584+ c.waitFor("the first planning failed", func() bool { return fs.hits.Load() == 1 })
585+ job2 := runner.await()
586+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "the retry planned the promoted run")
587+}
588+
589+// HIGH-2: a holder that was never told it holds the group is planned by the
590+// reconciliation, whatever happened to the retries.
591+func TestCIConcurrencyReconcilePlansAStaleHolder(t *testing.T) {
592+ c, runner, fs := flakyConcurrencyHarness(t, -1)
593+ c.pushN(1, workflowAt(wfGroup, ""))
594+ job1 := runner.await()
595+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
596+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
597+
598+ fs.match = holdsGroup(r2.GetId())
599+ fs.fails.Store(1000)
600+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
601+ c.waitFor("the planning fails", func() bool { return fs.hits.Load() >= 1 })
602+ rec := c.groupRecord("Deploy")
603+ require.NotNil(t, rec.Running)
604+ assert.Equal(t, r2.GetId(), rec.Running.RunID, "the group names the promoted run")
605+ c.waitJobs("which still shows pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
606+
607+ fs.fails.Store(0)
608+ c.forge.CI.ReconcileConcurrency(c.ctx)
609+ job2 := runner.await()
610+ assert.Equal(t, r2.GetId(), job2.GetRunId())
611+}
612+
613+const siblingJobs = ` steps:
614+ - run: echo grouped
615+ sibling:
616+ runs-on: ubuntu-latest
617+ steps:
618+ - run: echo sibling
619+`
620+
621+// jobWorkflow is a workflow of a grouped job with the given concurrency lines
622+// and an unrelated sibling.
623+func jobWorkflow(concurrency string) string {
624+ return "name: ci\non:\n push:\n branches: [main]\njobs:\n grouped:\n runs-on: ubuntu-latest\n" +
625+ concurrency + siblingJobs
626+}
627+
628+// awaitKey fetches jobs until the one of the run with the key comes.
629+func (r *fakeRunner) awaitKey(runID int64, key string) *pb.Job {
630+ r.c.t.Helper()
631+ for range 20 {
632+ if j := r.await(); j.GetRunId() == runID && j.GetJobKey() == key {
633+ return j
634+ }
635+ }
636+ r.c.t.Fatalf("no job %s of run %d", key, runID)
637+ return nil
638+}
639+
640+// T-2: a job-level cancel-in-progress (true, or an expression) cancels the
641+// running job of the group only: its sibling goes on, and the replacement
642+// becomes assignable.
643+func TestCIJobConcurrencyCancelInProgress(t *testing.T) {
644+ for _, tt := range []struct{ name, cancel string }{
645+ {"true", "true"},
646+ {"expression", "${{ github.ref == 'refs/heads/main' }}"},
647+ } {
648+ t.Run(tt.name, func(t *testing.T) {
649+ c, runner := concurrencyHarness(t)
650+ wf := jobWorkflow(" concurrency:\n group: jobgroup\n cancel-in-progress: " + tt.cancel + "\n")
651+ r1 := c.pushN(1, wf)
652+ grouped1, sibling1 := runner.awaitKey(r1.GetId(), "grouped"), runner.awaitKey(r1.GetId(), "sibling")
653+ watch, err := runner.jobAPI(grouped1).runner.WatchJob(c.ctx, connect.NewRequest(&pb.WatchJobRequest{JobId: grouped1.GetJobId()}))
654+ require.NoError(t, err)
655+
656+ r2 := c.pushN(2, wf)
657+ var cancel *pb.CancelRequest
658+ for watch.Receive() {
659+ if m := watch.Msg().GetCancel(); m != nil {
660+ cancel = m
661+ break
662+ }
663+ }
664+ require.NotNil(t, cancel, "the grouped job is asked to stop")
665+ assert.Contains(t, cancel.GetMessage(), "higher priority waiting request for 'jobgroup'")
666+
667+ // The sibling of the first run was not touched.
668+ runner.heartbeat(sibling1)
669+ for _, j := range c.jobsOf(r1) {
670+ if j.GetKey() == "sibling" {
671+ assert.NotEqual(t, pb.CiStatus_CI_STATUS_COMPLETED, j.GetStatus())
672+ assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_UNSPECIFIED, j.GetConclusion())
673+ }
674+ }
675+ _, err = runner.jobAPI(grouped1).runner.CompleteJob(c.ctx, connect.NewRequest(&pb.CompleteJobRequest{
676+ JobId: grouped1.GetJobId(), Conclusion: pb.CiConclusion_CI_CONCLUSION_CANCELLED,
677+ }))
678+ require.NoError(t, err)
679+ runner.runJob(sibling1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
680+
681+ grouped2 := runner.awaitKey(r2.GetId(), "grouped")
682+ assert.Equal(t, r2.GetId(), grouped2.GetRunId(), "the replacement is assignable")
683+ c.waitFor("the first run is over", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
684+ for _, j := range c.jobsOf(r1) {
685+ want := pb.CiConclusion_CI_CONCLUSION_CANCELLED
686+ if j.GetKey() == "sibling" {
687+ want = pb.CiConclusion_CI_CONCLUSION_SUCCESS
688+ }
689+ assert.Equal(t, want, j.GetConclusion(), j.GetKey())
690+ }
691+ })
692+ }
693+}
694+
695+// T-3: the server restarts: the groups are repaired at startup. A group held by
696+// an ended run is freed and the run behind it is planned; a group whose only
697+// run is over is removed.
698+func TestCIConcurrencyReconcileAtRestart(t *testing.T) {
699+ c, runner := concurrencyHarness(t)
700+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
701+ runner.await()
702+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
703+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
704+
705+ // The first run ends in the store without the release (the crash), and a
706+ // second group is held by a run that is over.
707+ store := c.forge.store.CI()
708+ jobs, err := store.ListJobs(c.ctx, r1.GetId(), 1)
709+ require.NoError(t, err)
710+ run, err := store.GetRun(c.ctx, c.h("proj"), r1.GetId())
711+ require.NoError(t, err)
712+ jobs[0].Phase, jobs[0].Conclusion = domain.CIPhaseEnded, domain.CISuccess
713+ run.Status, run.Conclusion, run.CompletedAt = domain.CICompleted, domain.CISuccess, time.Now()
714+ _, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs, Runs: []domain.CIRun{run}})
715+ require.NoError(t, err)
716+ c.holdGroup("Stale", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil)
717+ rec := c.groupRecord("Deploy")
718+ require.NotNil(t, rec.Running)
719+ require.Equal(t, r1.GetId(), rec.Running.RunID, "the record still holds the ended run")
720+
721+ // Restart on the same data.
722+ cfg, hookExe := c.forge.cfg, binary(t)
723+ c.srv.Close()
724+ c.forge.Close(c.ctx)
725+ f2, err := newForge(c.ctx, cfg, options{hookExe: hookExe, ci: func(cc *usecase.CIConfig) {
726+ cc.CancelGrace, cc.Tick, cc.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
727+ }})
728+ require.NoError(t, err)
729+ t.Cleanup(func() { f2.Close(context.WithoutCancel(c.ctx)) })
730+ srv := httptest.NewServer(f2.server.Handler())
731+ t.Cleanup(srv.Close)
732+
733+ var got *pb.Job
734+ c.waitFor("the waiting run is planned by the startup repair", func() bool {
735+ res, err := newClients(srv.URL+"/-/api", runner.secret).runner.FetchJob(c.ctx, connect.NewRequest(&pb.FetchJobRequest{}))
736+ if err == nil && res.Msg.GetJob() != nil {
737+ got = res.Msg.GetJob()
738+ }
739+ return got != nil
740+ })
741+ assert.Equal(t, r2.GetId(), got.GetRunId())
742+ groups, err := f2.store.CI().ListConcurrency(c.ctx)
743+ require.NoError(t, err)
744+ require.Len(t, groups, 1, "the stale group is gone")
745+ assert.Equal(t, "DEPLOY", groups[0].Group)
746+ require.NotNil(t, groups[0].State.Record.Running)
747+ assert.Equal(t, r2.GetId(), groups[0].State.Record.Running.RunID)
748+ assert.Nil(t, groups[0].State.Record.Pending)
749+}
750+
751+// Both writes of a run's own record fail when it queues (its "pending" state),
752+// and the cancel of the run fails for good: the group alone knows B. A newer run
753+// replaces B; B's mark is "replaced", written in the batch that unnamed it. After
754+// the store works again the reconciliation finds B from that mark and cancels it.
755+func TestCIConcurrencyReplacedRunWithLostWritesIsCancelled(t *testing.T) {
756+ c, runner, fs := flakyConcurrencyHarness(t, -1)
757+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
758+ runner.await()
759+
760+ // B queues, but no run after the first gets the pending state in its own
761+ // record, and no cancel lands.
762+ fs.match = func(w ports.CIWrite) bool {
763+ for _, r := range w.Runs {
764+ if r.ID > r1.GetId() && r.Concurrency.State == domain.CIConcurrencyPending {
765+ return true
766+ }
767+ }
768+ for _, j := range w.Jobs {
769+ if j.RunID > r1.GetId() && j.Ended() && j.Conclusion == domain.CICancelled {
770+ return true
771+ }
772+ }
773+ return false
774+ }
775+ fs.fails.Store(1 << 20)
776+ b := c.pushN(2, workflowAt(wfGroup, ""))
777+ c.waitFor("B is named by the group", func() bool {
778+ rec := c.groupRecord("Deploy")
779+ return rec.Pending != nil && rec.Pending.RunID == b.GetId()
780+ })
781+ bEntry := domain.CIConcurrencyEntry{RunID: b.GetId(), Attempt: 1}
782+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", bEntry), "the group batch marked it")
783+
784+ // C replaces B: B is unnamed and marked replaced in one batch.
785+ c.pushN(3, workflowAt(wfGroup, ""))
786+ c.waitFor("B is marked replaced", func() bool { return c.markState("Deploy", bEntry) == domain.CIConcurrencyReplaced })
787+ rec := c.groupRecord("Deploy")
788+ require.NotNil(t, rec.Pending)
789+ assert.NotEqual(t, b.GetId(), rec.Pending.RunID, "B is no longer named")
790+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(b))
791+
792+ fs.fails.Store(0)
793+ c.forge.CI.ReconcileConcurrency(c.ctx)
794+ c.waitFor("B ends cancelled", func() bool { return c.conclusionOf(b) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
795+ assert.True(t, c.cancelAnnotation(b, "Deploy"))
796+ c.waitFor("its mark goes with it", func() bool { return c.markState("Deploy", bEntry) == "" })
797+}
798+
799+// A grouped job B is replaced by C, the cancel never lands, C is promoted when
800+// the running job ends, and an unrelated sibling of B completes and advances B:
801+// B's grouped job must not take the free place, it ends cancelled and never runs.
802+func TestCIJobConcurrencyReplacedJobNeverAcquiresAgain(t *testing.T) {
803+ c, runner, fs := flakyConcurrencyHarness(t, -1)
804+ wf := jobWorkflow(" concurrency: jobgroup\n")
805+ r1 := c.pushN(1, wf)
806+ grouped1, sibling1 := runner.awaitKey(r1.GetId(), "grouped"), runner.awaitKey(r1.GetId(), "sibling")
807+
808+ rB := c.pushN(2, wf)
809+ siblingB := runner.awaitKey(rB.GetId(), "sibling") // B's unrelated job is running
810+ c.waitJobs("B's grouped job waits", rB, func(js []*pb.RunJob) bool {
811+ for _, j := range js {
812+ if j.GetKey() == "grouped" {
813+ return j.GetStatus() == pb.CiStatus_CI_STATUS_PENDING
814+ }
815+ }
816+ return false
817+ })
818+
819+ fs.match = endsCancelled(rB.GetId())
820+ fs.fails.Store(1 << 20)
821+ rC := c.pushN(3, wf)
822+ bEntry := domain.CIConcurrencyEntry{RunID: rB.GetId(), Attempt: 1, JobID: domain.NewJobID(rB.GetId(), 0, 0)}
823+ c.waitFor("B's grouped job is replaced", func() bool { return c.markState("jobgroup", bEntry) == domain.CIConcurrencyReplaced })
824+
825+ // The running job ends: C's grouped job is promoted.
826+ runner.runJob(grouped1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
827+ runner.runJob(sibling1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
828+ groupedC := runner.awaitKey(rC.GetId(), "grouped")
829+ rec := c.groupRecord("jobgroup")
830+ require.NotNil(t, rec.Running)
831+ assert.Equal(t, rC.GetId(), rec.Running.RunID)
832+
833+ // B's sibling completes: B is advanced, its grouped job asks for the group.
834+ runner.runJob(siblingB, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
835+ c.waitFor("B was advanced", func() bool {
836+ for _, j := range c.jobsOf(rB) {
837+ if j.GetKey() == "sibling" {
838+ return j.GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
839+ }
840+ }
841+ return false
842+ })
843+ time.Sleep(300 * time.Millisecond)
844+ rec = c.groupRecord("jobgroup")
845+ require.NotNil(t, rec.Running)
846+ assert.Equal(t, rC.GetId(), rec.Running.RunID, "C still holds the group")
847+ if rec.Pending != nil {
848+ assert.NotEqual(t, rB.GetId(), rec.Pending.RunID, "B did not take the waiting place back")
849+ }
850+ // Nothing of B's grouped job is given to a runner.
851+ for range 5 {
852+ j, err := runner.fetch("")
853+ require.NoError(t, err)
854+ if j != nil {
855+ assert.False(t, j.GetRunId() == rB.GetId() && j.GetJobKey() == "grouped", "B's grouped job must not run")
856+ }
857+ }
858+
859+ // Once the store works, the reconciliation cancels it.
860+ fs.fails.Store(0)
861+ c.forge.CI.ReconcileConcurrency(c.ctx)
862+ c.waitFor("B's grouped job ends cancelled", func() bool {
863+ for _, j := range c.jobsOf(rB) {
864+ if j.GetKey() == "grouped" {
865+ return j.GetConclusion() == pb.CiConclusion_CI_CONCLUSION_CANCELLED
866+ }
867+ }
868+ return false
869+ })
870+ runner.runJob(groupedC, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
871+}
872+
873+// The same, with an empty group: B's grouped job is replaced and cancelled
874+// before its replacement ends too; a replaced job never runs even when the
875+// group is free.
876+func TestCIJobConcurrencyReplacedJobDoesNotTakeAnEmptyGroup(t *testing.T) {
877+ c, runner, fs := flakyConcurrencyHarness(t, -1)
878+ wf := jobWorkflow(" concurrency: jobgroup\n")
879+ r1 := c.pushN(1, wf)
880+ grouped1, sibling1 := runner.awaitKey(r1.GetId(), "grouped"), runner.awaitKey(r1.GetId(), "sibling")
881+ rB := c.pushN(2, wf)
882+ siblingB := runner.awaitKey(rB.GetId(), "sibling")
883+ c.waitJobs("pending", rB, func(js []*pb.RunJob) bool {
884+ for _, j := range js {
885+ if j.GetKey() == "grouped" {
886+ return j.GetStatus() == pb.CiStatus_CI_STATUS_PENDING
887+ }
888+ }
889+ return false
890+ })
891+ fs.match = endsCancelled(rB.GetId())
892+ fs.fails.Store(1 << 20)
893+ rC := c.pushN(3, wf)
894+ bEntry := domain.CIConcurrencyEntry{RunID: rB.GetId(), Attempt: 1, JobID: domain.NewJobID(rB.GetId(), 0, 0)}
895+ c.waitFor("replaced", func() bool { return c.markState("jobgroup", bEntry) == domain.CIConcurrencyReplaced })
896+
897+ // Both the running job and the promoted one finish: the group is empty.
898+ runner.runJob(grouped1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
899+ runner.runJob(sibling1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
900+ runner.runJob(runner.awaitKey(rC.GetId(), "grouped"), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
901+ c.waitFor("the group is free", func() bool { return c.groupRecord("jobgroup").Running == nil })
902+
903+ runner.runJob(siblingB, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil) // advances B
904+ time.Sleep(300 * time.Millisecond)
905+ assert.Nil(t, c.groupRecord("jobgroup").Running, "B's replaced job did not take the empty group")
906+ for range 5 {
907+ j, err := runner.fetch("")
908+ require.NoError(t, err)
909+ if j != nil {
910+ assert.False(t, j.GetRunId() == rB.GetId() && j.GetJobKey() == "grouped", "B's grouped job must not run")
911+ }
912+ }
913+ fs.fails.Store(0)
914+ c.forge.CI.ReconcileConcurrency(c.ctx)
915+ c.waitFor("it ends cancelled", func() bool {
916+ for _, j := range c.jobsOf(rB) {
917+ if j.GetKey() == "grouped" {
918+ return j.GetConclusion() == pb.CiConclusion_CI_CONCLUSION_CANCELLED
919+ }
920+ }
921+ return false
922+ })
923+}
924+
925+// Crash between the group batch and the write of a JOB's own state: the group
926+// names (and marks) the grouped job as waiting, the job's record does not say so
927+// yet. It is not replaced and not cancelled; reconciliation tells it, and it runs
928+// when its turn comes.
929+func TestCIJobConcurrencyCrashBetweenGroupAndJob(t *testing.T) {
930+ c, runner := concurrencyHarness(t)
931+ group := " concurrency: jobgroup\n"
932+ c.pushN(1, workflowAt("", group))
933+ job1 := runner.await()
934+ r2 := c.pushN(2, workflowAt("", group))
935+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
936+
937+ store := c.forge.store.CI()
938+ jobs, err := store.ListJobs(c.ctx, r2.GetId(), 1)
939+ require.NoError(t, err)
940+ require.Len(t, jobs, 1)
941+ entry := domain.CIConcurrencyEntry{RunID: r2.GetId(), Attempt: 1, JobID: jobs[0].ID}
942+ jobs[0].Concurrency.State = domain.CIConcurrencyIdle // the write that was lost
943+ _, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs})
944+ require.NoError(t, err)
945+ rec := c.groupRecord("jobgroup")
946+ require.NotNil(t, rec.Pending)
947+ assert.Equal(t, entry.JobID, rec.Pending.JobID, "the group names the job")
948+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("jobgroup", entry), "and its mark says so")
949+
950+ c.forge.CI.ReconcileConcurrency(c.ctx)
951+ c.waitFor("the job is told it waits", func() bool {
952+ js, err := store.ListJobs(c.ctx, r2.GetId(), 1)
953+ return err == nil && js[0].Concurrency.State == domain.CIConcurrencyPending
954+ })
955+ time.Sleep(200 * time.Millisecond)
956+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2), "a named job is not a replaced one")
957+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("jobgroup", entry))
958+
959+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
960+ assert.Equal(t, r2.GetId(), runner.await().GetRunId(), "it runs when the group is free")
961+}
962+
963+// plannedWrite matches a write of a run after the first that stores planned jobs.
964+func plannedWrite(after int64) func(ports.CIWrite) bool {
965+ return func(w ports.CIWrite) bool {
966+ for _, j := range w.Jobs {
967+ if j.RunID > after && j.Planned {
968+ return true
969+ }
970+ }
971+ return false
972+ }
973+}
974+
975+const matrixGroupWorkflow = `name: b
976+on:
977+ push:
978+ branches: [other]
979+jobs:
980+ build:
981+ runs-on: ubuntu-latest
982+ strategy:
983+ matrix:
984+ n: [1, 2, 3]
985+ concurrency: shared
986+ steps:
987+ - run: echo ${{ matrix.n }}
988+`
989+
990+const holderWorkflow = `name: a
991+on:
992+ push:
993+ branches: [main]
994+jobs:
995+ hold:
996+ runs-on: ubuntu-latest
997+ concurrency: shared
998+ steps:
999+ - run: echo hold
1000+`
1001+
1002+// Run B plans three legs of a matrix that share a group held by A. The legs
1003+// replace one another (leg 0 by leg 1 by leg 2: only one waits), the group
1004+// batches commit, and the write of B's planned jobs fails, leaving the
1005+// placeholder of the job in the store. After the store recovers, the
1006+// reconciliation must plan B before it cancels anything: only legs 0 and 1
1007+// end cancelled, and leg 2, the legitimate waiting one, runs when A ends.
1008+func TestCIJobConcurrencyRecoveryPlansTheMatrixBeforeCancellingALeg(t *testing.T) {
1009+ c, runner, fs := flakyConcurrencyHarness(t, -1)
1010+ a := c.push("main", map[string]string{".github/workflows/a.yml": holderWorkflow}, "")
1011+ _ = a
1012+ holder := runner.await()
1013+ require.Equal(t, "hold", holder.GetJobKey())
1014+ aRun := holder.GetRunId()
1015+
1016+ fs.match = plannedWrite(aRun)
1017+ fs.fails.Store(1 << 20)
1018+ c.push("other", map[string]string{".github/workflows/b.yml": matrixGroupWorkflow}, "")
1019+ c.waitFor("B's run exists", func() bool {
1020+ for _, r := range c.runs() {
1021+ if r.GetId() > aRun {
1022+ return true
1023+ }
1024+ }
1025+ return false
1026+ })
1027+ var bRun *pb.Run
1028+ for _, r := range c.runs() {
1029+ if r.GetId() > aRun {
1030+ bRun = r
1031+ }
1032+ }
1033+ require.NotNil(t, bRun)
1034+ g := domain.ConcurrencyKey("shared")
1035+ c.waitFor("the group batches committed: two legs are marked replaced", func() bool {
1036+ replaced := 0
1037+ for _, m := range c.groupState(g).Marks {
1038+ if m.Entry.RunID == bRun.GetId() && m.State == domain.CIConcurrencyReplaced {
1039+ replaced++
1040+ }
1041+ }
1042+ return replaced == 2
1043+ })
1044+ assert.Equal(t, 1, len(c.jobsOf(bRun)), "B's job is still the placeholder: its planned legs were never stored")
1045+
1046+ fs.fails.Store(0)
1047+ c.forge.CI.ReconcileConcurrency(c.ctx)
1048+ c.waitFor("B has its three legs", func() bool { return len(c.jobsOf(bRun)) == 3 })
1049+ c.waitFor("legs 0 and 1 end cancelled", func() bool {
1050+ n := 0
1051+ for _, j := range c.jobsOf(bRun) {
1052+ if j.GetConclusion() == pb.CiConclusion_CI_CONCLUSION_CANCELLED {
1053+ n++
1054+ }
1055+ }
1056+ return n == 2
1057+ })
1058+ byCombo := map[string]*pb.RunJob{}
1059+ for _, j := range c.jobsOf(bRun) {
1060+ byCombo[j.GetName()] = j
1061+ }
1062+ var waiting []*pb.RunJob
1063+ for _, j := range c.jobsOf(bRun) {
1064+ if j.GetConclusion() != pb.CiConclusion_CI_CONCLUSION_CANCELLED {
1065+ waiting = append(waiting, j)
1066+ }
1067+ }
1068+ require.Len(t, waiting, 1, "exactly one leg is not cancelled")
1069+ assert.Equal(t, pb.CiStatus_CI_STATUS_PENDING, waiting[0].GetStatus(), "the final leg waits for A")
1070+ assert.Contains(t, waiting[0].GetName(), "3", "and it is the last leg")
1071+
1072+ runner.runJob(holder, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1073+ last := runner.await()
1074+ assert.Equal(t, bRun.GetId(), last.GetRunId())
1075+ assert.Equal(t, waiting[0].GetId(), last.GetJobId(), "the final leg runs when A ends")
1076+ runner.runJob(last, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1077+}
1078+
1079+// Stale marks (of runs that are gone) are pruned a bounded number per
1080+// transaction: a decision on a group with a long backlog succeeds at once, and
1081+// the reconciliation takes the others, a bounded batch at a time.
1082+func TestCIConcurrencyPrunesStaleMarksInBoundedBatches(t *testing.T) {
1083+ const stale, bound = 10, 3
1084+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
1085+ cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
1086+ cfg.ConcurrencyPruneBatch = bound
1087+ })
1088+ runner := c.registerRunner("alpha", "ubuntu-latest")
1089+ require.NoError(t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), "Deploy", func(st *domain.CIConcurrencyGroupState) error {
1090+ st.Marks = map[string]domain.CIConcurrencyMark{}
1091+ for i := range stale {
1092+ e := domain.CIConcurrencyEntry{RunID: int64(9000 + i), Attempt: 1, JobID: "9000-0-0"}
1093+ st.Marks[e.Key()] = domain.CIConcurrencyMark{Entry: e, Group: "Deploy", State: domain.CIConcurrencyPending}
1094+ }
1095+ return nil
1096+ }))
1097+ require.Len(t, c.groupState("Deploy").Marks, stale)
1098+
1099+ r := c.pushN(1, workflowAt(wfGroup, ""))
1100+ job := runner.await()
1101+ assert.Equal(t, r.GetId(), job.GetRunId(), "the decision succeeded with the backlog in place")
1102+ st := c.groupState("Deploy")
1103+ require.NotNil(t, st.Record.Running)
1104+ assert.Equal(t, r.GetId(), st.Record.Running.RunID)
1105+ assert.Len(t, st.Marks, stale-bound+1, "one transaction pruned a bounded number of the stale marks, and marked the run")
1106+
1107+ c.forge.CI.ReconcileConcurrency(c.ctx)
1108+ st = c.groupState("Deploy")
1109+ assert.Len(t, st.Marks, 1, "the reconciliation prunes the rest, a batch at a time")
1110+ assert.Contains(t, st.Marks, domain.CIConcurrencyEntry{RunID: r.GetId(), Attempt: 1}.Key())
1111+ runner.runJob(job, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1112+}
@@ -0,0 +1,1112 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package app
4+
5+import (
6+ "context"
7+ "fmt"
8+ "net/http/httptest"
9+ "strings"
10+ "sync/atomic"
11+ "testing"
12+ "time"
13+
14+ "bnema.dev/ermite/internal/domain"
15+
16+ "connectrpc.com/connect"
17+ "github.com/stretchr/testify/assert"
18+ "github.com/stretchr/testify/require"
19+
20+ pb "bnema.dev/ermite-api/gen/ermite"
21+ "bnema.dev/ermite/internal/ports"
22+ "bnema.dev/ermite/internal/usecase"
23+)
24+
25+// concurrencyHarness is a CI harness with fast deadlines and one runner.
26+func concurrencyHarness(t *testing.T) (*ciHarness, *fakeRunner) {
27+ t.Helper()
28+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
29+ cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
30+ })
31+ return c, c.registerRunner("alpha", "ubuntu-latest")
32+}
33+
34+// workflowAt is a one-job workflow on main with extra top-level and job lines.
35+func workflowAt(top, job string) string {
36+ return "name: ci\non:\n push:\n branches: [main]\n" + top +
37+ "jobs:\n build:\n runs-on: ubuntu-latest\n" + job +
38+ " steps:\n - run: echo hello\n"
39+}
40+
41+// pushN pushes a new commit with the workflow and returns the new run.
42+func (c *ciHarness) pushN(n int, wf string) *pb.Run {
43+ c.t.Helper()
44+ before := len(c.runs())
45+ // A comment makes each push a new commit even when the workflow repeats.
46+ c.push("main", map[string]string{".github/workflows/ci.yml": fmt.Sprintf("# push %d\n%s", n, wf)}, "")
47+ c.waitFor("the run exists", func() bool { return len(c.runs()) == before+1 })
48+ return c.runs()[0] // newest first
49+}
50+
51+func (c *ciHarness) jobsOf(run *pb.Run) []*pb.RunJob { return c.getRun(run.GetId()).GetJobs() }
52+
53+func (c *ciHarness) waitJobs(what string, run *pb.Run, cond func([]*pb.RunJob) bool) {
54+ c.t.Helper()
55+ c.waitFor(what, func() bool { return cond(c.jobsOf(run)) })
56+}
57+
58+func allStatus(s pb.CiStatus) func([]*pb.RunJob) bool {
59+ return func(js []*pb.RunJob) bool {
60+ for _, j := range js {
61+ if j.GetStatus() != s {
62+ return false
63+ }
64+ }
65+ return len(js) > 0
66+ }
67+}
68+
69+func (c *ciHarness) conclusionOf(run *pb.Run) pb.CiConclusion {
70+ return c.getRun(run.GetId()).GetRun().GetConclusion()
71+}
72+
73+func (c *ciHarness) cancelAnnotation(run *pb.Run, group string) bool {
74+ for _, j := range c.jobsOf(run) {
75+ for _, a := range c.annotations(run.GetId(), j.GetId()).GetAnnotations() {
76+ if a.GetAnnotation().GetMessage() == "Canceling since a higher priority waiting request for '"+group+"' exists" {
77+ return true
78+ }
79+ }
80+ }
81+ return false
82+}
83+
84+const wfGroup = "concurrency:\n group: Deploy\n"
85+
86+// AC4: a workflow-level group lets one run go and one wait; a newer waiting run
87+// replaces the older one, which ends cancelled with GitHub's annotation; the
88+// group is handed over when the running one completes.
89+func TestCIWorkflowConcurrencyPendingReplacement(t *testing.T) {
90+ c, runner := concurrencyHarness(t)
91+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
92+ job1 := runner.await()
93+ require.Equal(t, r1.GetId(), job1.GetRunId())
94+
95+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
96+ c.waitJobs("the second run is pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
97+ assert.Equal(t, "Deploy", c.jobsOf(r2)[0].GetConcurrencyGroup())
98+ assert.Equal(t, pb.CiStatus_CI_STATUS_QUEUED, c.getRun(r2.GetId()).GetRun().GetStatus(), "a run is queued, only its jobs are pending")
99+ // The pending job is not assignable.
100+ j, err := runner.fetch("")
101+ require.NoError(t, err)
102+ assert.Nil(t, j)
103+
104+ r3 := c.pushN(3, workflowAt(wfGroup, ""))
105+ c.waitFor("the older pending run is cancelled", func() bool {
106+ return c.getRun(r2.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
107+ })
108+ assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2))
109+ assert.True(t, c.cancelAnnotation(r2, "Deploy"))
110+ c.waitJobs("the third run is pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
111+ assert.NotEqual(t, pb.CiStatus_CI_STATUS_COMPLETED, c.getRun(r1.GetId()).GetRun().GetStatus(), "the running run is left alone")
112+
113+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
114+ job3 := runner.await()
115+ assert.Equal(t, r3.GetId(), job3.GetRunId(), "the group goes to the waiting run")
116+ runner.runJob(job3, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
117+ c.waitFor("the third run completes", func() bool {
118+ return c.getRun(r3.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
119+ })
120+
121+ // The group is free: a new run takes it at once.
122+ r4 := c.pushN(4, workflowAt(wfGroup, ""))
123+ job4 := runner.await()
124+ assert.Equal(t, r4.GetId(), job4.GetRunId())
125+}
126+
127+// AC4: cancel-in-progress cancels the running run, whose runner is told through
128+// the cancel mechanism of a user cancel; an expression decides it, and groups
129+// are compared without regard to case.
130+func TestCIWorkflowConcurrencyCancelInProgress(t *testing.T) {
131+ for _, tt := range []struct{ name, cancel string }{
132+ {"true", "true"},
133+ {"expression", "${{ github.ref == 'refs/heads/main' }}"},
134+ } {
135+ t.Run(tt.name, func(t *testing.T) {
136+ c, runner := concurrencyHarness(t)
137+ r1 := c.pushN(1, workflowAt("concurrency:\n group: Release\n cancel-in-progress: "+tt.cancel+"\n", ""))
138+ job1 := runner.await()
139+ watch, err := runner.jobAPI(job1).runner.WatchJob(c.ctx, connect.NewRequest(&pb.WatchJobRequest{JobId: job1.GetJobId()}))
140+ require.NoError(t, err)
141+
142+ r2 := c.pushN(2, workflowAt("concurrency:\n group: RELEASE\n cancel-in-progress: "+tt.cancel+"\n", ""))
143+ var cancel *pb.CancelRequest
144+ for watch.Receive() {
145+ if m := watch.Msg().GetCancel(); m != nil {
146+ cancel = m
147+ break
148+ }
149+ }
150+ require.NotNil(t, cancel, "the running job is asked to stop")
151+ assert.Equal(t, pb.CancelReason_CANCEL_REASON_USER, cancel.GetReason())
152+ assert.Contains(t, cancel.GetMessage(), "higher priority waiting request for 'RELEASE'")
153+
154+ _, err = runner.jobAPI(job1).runner.CompleteJob(c.ctx, connect.NewRequest(&pb.CompleteJobRequest{
155+ JobId: job1.GetJobId(), Conclusion: pb.CiConclusion_CI_CONCLUSION_CANCELLED,
156+ }))
157+ require.NoError(t, err)
158+ c.waitFor("the first run is cancelled", func() bool { return c.conclusionOf(r1) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
159+ job2 := runner.await()
160+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "the waiting run takes the group")
161+ })
162+ }
163+ t.Run("false expression only queues", func(t *testing.T) {
164+ c, runner := concurrencyHarness(t)
165+ top := "concurrency:\n group: G\n cancel-in-progress: ${{ github.ref == 'refs/heads/other' }}\n"
166+ c.pushN(1, workflowAt(top, ""))
167+ job1 := runner.await()
168+ r2 := c.pushN(2, workflowAt(top, ""))
169+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
170+ runner.heartbeat(job1)
171+ time.Sleep(200 * time.Millisecond)
172+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(c.runs()[1]))
173+ })
174+}
175+
176+const matrixJob = ` strategy:
177+ matrix:
178+ n: [1, 2]
179+ concurrency:
180+ group: shared-${{ 'X' }}
181+`
182+
183+// AC4: the legs of a matrix share a job-level group: one runs, one waits, and
184+// a leg is freed when the other ends.
185+func TestCIJobConcurrencyMatrixLegsShareAGroup(t *testing.T) {
186+ c, runner := concurrencyHarness(t)
187+ r := c.pushN(1, workflowAt("", matrixJob))
188+ job1 := runner.await()
189+ c.waitJobs("one leg is pending", r, func(js []*pb.RunJob) bool {
190+ pending, other := 0, 0
191+ for _, j := range js {
192+ switch j.GetStatus() {
193+ case pb.CiStatus_CI_STATUS_PENDING:
194+ pending++
195+ default:
196+ other++
197+ }
198+ }
199+ return len(js) == 2 && pending == 1 && other == 1
200+ })
201+ j, err := runner.fetch("")
202+ require.NoError(t, err)
203+ assert.Nil(t, j, "the pending leg is not assignable")
204+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
205+ job2 := runner.await()
206+ assert.NotEqual(t, job1.GetJobId(), job2.GetJobId())
207+ runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
208+ c.waitFor("the run completes", func() bool { return c.getRun(r.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
209+ assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_SUCCESS, c.conclusionOf(r))
210+}
211+
212+// AC4: a job-level group is shared between runs, case-insensitively; the third
213+// waiting job replaces the second, which ends cancelled with the annotation.
214+func TestCIJobConcurrencyAcrossRuns(t *testing.T) {
215+ c, runner := concurrencyHarness(t)
216+ group := func(g string) string { return " concurrency: " + g + "\n" }
217+ r1 := c.pushN(1, workflowAt("", group("Build-Lock")))
218+ job1 := runner.await()
219+ r2 := c.pushN(2, workflowAt("", group("build-lock")))
220+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
221+ assert.Equal(t, "build-lock", c.jobsOf(r2)[0].GetConcurrencyGroup())
222+ r3 := c.pushN(3, workflowAt("", group("BUILD-LOCK")))
223+ c.waitFor("the second job is cancelled", func() bool { return c.conclusionOf(r2) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
224+ assert.True(t, c.cancelAnnotation(r2, "BUILD-LOCK"), "the group is named as the newer request wrote it")
225+ c.waitJobs("pending", r3, allStatus(pb.CiStatus_CI_STATUS_PENDING))
226+ _ = r1
227+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
228+ job3 := runner.await()
229+ assert.Equal(t, r3.GetId(), job3.GetRunId())
230+}
231+
232+// AC4: the group is freed when its run is cancelled, and when a re-run joins
233+// the group again it waits behind the run that took it meanwhile.
234+func TestCIConcurrencyFreedByCancelAndRerunRejoins(t *testing.T) {
235+ c, runner := concurrencyHarness(t)
236+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
237+ job1 := runner.await()
238+ _, err := c.root.run.CancelRun(c.ctx, connect.NewRequest(&pb.CancelRunRequest{Repo: "proj", RunId: r1.GetId()}))
239+ require.NoError(t, err)
240+ runner.heartbeat(job1)
241+ c.waitFor("the cancelled run completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
242+
243+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
244+ job2 := runner.await()
245+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "a cancelled run freed the group")
246+
247+ _, err = c.root.run.RerunRun(c.ctx, connect.NewRequest(&pb.RerunRunRequest{Repo: "proj", RunId: r1.GetId()}))
248+ require.NoError(t, err)
249+ c.waitJobs("the re-run waits for the group", r1, allStatus(pb.CiStatus_CI_STATUS_PENDING))
250+ runner.runJob(job2, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
251+ job1b := runner.await()
252+ assert.Equal(t, r1.GetId(), job1b.GetRunId(), "the re-run takes the group once it is free")
253+}
254+
255+// AC4: a job lost to a missing heartbeat frees its group.
256+func TestCIConcurrencyFreedByLostJob(t *testing.T) {
257+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
258+ cfg.HeartbeatDeadline, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
259+ })
260+ runner := c.registerRunner("alpha", "ubuntu-latest")
261+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
262+ job1 := runner.await()
263+ runner.heartbeat(job1)
264+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
265+ c.waitFor("the lost job fails", func() bool { return c.conclusionOf(r1) == pb.CiConclusion_CI_CONCLUSION_FAILURE })
266+ job2 := runner.await()
267+ assert.Equal(t, r2.GetId(), job2.GetRunId())
268+}
269+
270+// A group that evaluates to nothing fails the job as GitHub does, and
271+// queue: max is refused.
272+func TestCIConcurrencyErrors(t *testing.T) {
273+ c, runner := concurrencyHarness(t)
274+ _ = runner
275+ r1 := c.pushN(1, workflowAt("", " concurrency:\n group: ${{ vars.NOPE }}\n"))
276+ c.waitJobs("failed", r1, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
277+ assert.Contains(t, c.jobsOf(r1)[0].GetError(), "concurrency group name cannot be empty")
278+
279+ r3 := c.pushN(3, workflowAt("", " concurrency:\n group: g\n queue: max\n cancel-in-progress: true\n"))
280+ c.waitJobs("failed", r3, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
281+ assert.Equal(t, "'queue: max' cannot be combined with 'cancel-in-progress: true'.", c.jobsOf(r3)[0].GetError())
282+
283+ r2 := c.pushN(2, workflowAt("", " concurrency:\n group: g\n queue: max\n"))
284+ c.waitJobs("failed", r2, allStatus(pb.CiStatus_CI_STATUS_COMPLETED))
285+ assert.Equal(t, "not supported yet: queue: max", strings.TrimSpace(c.jobsOf(r2)[0].GetError()))
286+}
287+
288+// holdGroup writes a group behind the use case's back, as a crash would leave
289+// it: the entries are named and marked whatever the state of their runs.
290+func (c *ciHarness) holdGroup(group string, running, pending *domain.CIConcurrencyEntry) {
291+ c.t.Helper()
292+ require.NoError(c.t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), group, func(st *domain.CIConcurrencyGroupState) error {
293+ st.Record.Running, st.Record.Pending = running, pending
294+ st.Marks = map[string]domain.CIConcurrencyMark{}
295+ if running != nil {
296+ st.Marks[running.Key()] = domain.CIConcurrencyMark{Entry: *running, Group: group, State: domain.CIConcurrencyHolding}
297+ }
298+ if pending != nil {
299+ st.Marks[pending.Key()] = domain.CIConcurrencyMark{Entry: *pending, Group: group, State: domain.CIConcurrencyPending}
300+ }
301+ return nil
302+ }))
303+}
304+
305+func (c *ciHarness) groupState(group string) (st domain.CIConcurrencyGroupState) {
306+ c.t.Helper()
307+ require.NoError(c.t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), group, func(s *domain.CIConcurrencyGroupState) error {
308+ st = *s
309+ return nil
310+ }))
311+ return st
312+}
313+
314+func (c *ciHarness) groupRecord(group string) domain.CIConcurrencyRecord {
315+ return c.groupState(group).Record
316+}
317+
318+// markState is the mark of an entry in a group, "" when it has none.
319+func (c *ciHarness) markState(group string, e domain.CIConcurrencyEntry) domain.CIConcurrencyState {
320+ return c.groupState(group).Marks[e.Key()].State
321+}
322+
323+// A crash between the write of an ended job and the release of its group leaves
324+// the record holding it: the next run to ask finds the entry dead and takes the
325+// group.
326+func TestCIConcurrencyHealsAfterACrash(t *testing.T) {
327+ c, runner := concurrencyHarness(t)
328+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
329+ runner.runJob(runner.await(), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
330+ c.waitFor("the first run completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
331+ c.waitFor("the group is released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
332+ c.holdGroup("Deploy", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil) // the release that did not happen
333+
334+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
335+ job2 := runner.await()
336+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "the dead entry does not hold the group")
337+ rec := c.groupRecord("Deploy")
338+ require.NotNil(t, rec.Running)
339+ assert.Equal(t, r2.GetId(), rec.Running.RunID)
340+}
341+
342+// A run that is deleted (the retention sweep, not its repository) frees the
343+// group it held and the one it waited in.
344+func TestCIConcurrencyHealsAfterRunDeletion(t *testing.T) {
345+ c, runner := concurrencyHarness(t)
346+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
347+ runner.runJob(runner.await(), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
348+ c.waitFor("completes", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
349+ c.waitFor("released", func() bool { return c.groupRecord("Deploy").IsEmpty() })
350+ ghost := &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}
351+ c.holdGroup("Deploy", ghost, nil)
352+ ids, err := c.forge.store.CI().Sweep(c.ctx, time.Now().Add(time.Hour))
353+ require.NoError(t, err)
354+ require.Contains(t, ids, r1.GetId(), "the run is deleted")
355+
356+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
357+ job2 := runner.await()
358+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "a deleted run does not hold the group")
359+}
360+
361+// flakyStore fails the writes that match, as many times as fails says. The
362+// failure is returned before anything is written.
363+type flakyStore struct {
364+ ports.CIStore
365+ match func(ports.CIWrite) bool
366+ fails atomic.Int32
367+ hits atomic.Int32
368+}
369+
370+func (f *flakyStore) Apply(ctx context.Context, w ports.CIWrite) (uint64, error) {
371+ if f.match != nil && f.match(w) && f.fails.Add(-1) >= 0 {
372+ f.hits.Add(1)
373+ return 0, domain.ErrConflict
374+ }
375+ return f.CIStore.Apply(ctx, w)
376+}
377+
378+// flakyConcurrencyHarness is a harness whose store fails the writes fs.match
379+// selects. retries is CIConfig.ConcurrencyRetries: 0 keeps the default, a
380+// negative number leaves the recovery of a lost delivery to the reconciliation.
381+func flakyConcurrencyHarness(t *testing.T, retries int) (*ciHarness, *fakeRunner, *flakyStore) {
382+ t.Helper()
383+ fs := &flakyStore{}
384+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
385+ cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
386+ cfg.ConcurrencyRetries = retries
387+ fs.CIStore, cfg.Store = cfg.Store, fs
388+ })
389+ return c, c.registerRunner("alpha", "ubuntu-latest"), fs
390+}
391+
392+// endsCancelled matches the write that ends a job of the run as cancelled.
393+func endsCancelled(runID int64) func(ports.CIWrite) bool {
394+ return func(w ports.CIWrite) bool {
395+ for _, j := range w.Jobs {
396+ if j.RunID == runID && j.Ended() && j.Conclusion == domain.CICancelled {
397+ return true
398+ }
399+ }
400+ return false
401+ }
402+}
403+
404+// A replaced pending run whose cancel write fails once is cancelled by the
405+// retry.
406+func TestCIConcurrencyCancelSurvivesAFailedDelivery(t *testing.T) {
407+ c, runner, fs := flakyConcurrencyHarness(t, 0)
408+ c.pushN(1, workflowAt(wfGroup, ""))
409+ job1 := runner.await()
410+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
411+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
412+
413+ fs.match = endsCancelled(r2.GetId())
414+ fs.fails.Store(1)
415+ r3 := c.pushN(3, workflowAt(wfGroup, ""))
416+ c.waitFor("the first delivery failed", func() bool { return fs.hits.Load() == 1 })
417+ rec := c.groupRecord("Deploy")
418+ require.NotNil(t, rec.Pending)
419+ assert.Equal(t, r3.GetId(), rec.Pending.RunID)
420+ c.waitFor("the replaced run is cancelled by the retry", func() bool { return c.conclusionOf(r2) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
421+ assert.True(t, c.cancelAnnotation(r2, "Deploy"))
422+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
423+ assert.Equal(t, r3.GetId(), runner.await().GetRunId())
424+}
425+
426+// Nothing is bounded: more replacements than any list could hold, every cancel
427+// write failing, and after the store works again the reconciliation discovers
428+// each victim through its "replaced" mark in the group, and cancels it.
429+func TestCIConcurrencyReconcileCancelsEveryReplacedVictim(t *testing.T) {
430+ const replaced = 35
431+ c, runner, fs := flakyConcurrencyHarness(t, -1)
432+ c.pushN(1, workflowAt(wfGroup, ""))
433+ runner.await()
434+
435+ // Each new run replaces the one that waited before it; no cancel lands.
436+ fs.match = func(w ports.CIWrite) bool {
437+ for _, j := range w.Jobs {
438+ if j.Ended() && j.Conclusion == domain.CICancelled {
439+ return true
440+ }
441+ }
442+ return false
443+ }
444+ fs.fails.Store(1 << 20)
445+ var victims []*pb.Run
446+ prev := c.pushN(2, workflowAt(wfGroup, ""))
447+ c.waitJobs("pending", prev, allStatus(pb.CiStatus_CI_STATUS_PENDING))
448+ for i := range replaced {
449+ next := c.pushN(3+i, workflowAt(wfGroup, ""))
450+ c.waitFor("the newer run takes the waiting place", func() bool {
451+ rec := c.groupRecord("Deploy")
452+ return rec.Pending != nil && rec.Pending.RunID == next.GetId()
453+ })
454+ c.waitJobs("the newer run is pending", next, allStatus(pb.CiStatus_CI_STATUS_PENDING))
455+ victims = append(victims, prev)
456+ prev = next
457+ }
458+ for _, v := range victims {
459+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(v), "no cancel landed yet")
460+ }
461+ // The test has no in-process retry (ConcurrencyRetries is negative): only
462+ // the reconciliation can recover the victims.
463+ fs.fails.Store(0)
464+ c.forge.CI.ReconcileConcurrency(c.ctx)
465+ for _, v := range victims {
466+ c.waitFor("a replaced run is cancelled", func() bool { return c.conclusionOf(v) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
467+ assert.True(t, c.cancelAnnotation(v, "Deploy"))
468+ }
469+ c.waitJobs("the last one still waits", prev, allStatus(pb.CiStatus_CI_STATUS_PENDING))
470+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(prev))
471+}
472+
473+// A running entry that cancel-in-progress replaced stays named with Canceling
474+// set; when its cancel is lost, the reconciliation asks again.
475+func TestCIConcurrencyReconcileAsksARunningVictimAgain(t *testing.T) {
476+ c, runner, fs := flakyConcurrencyHarness(t, -1)
477+ top := "concurrency:\n group: Deploy\n cancel-in-progress: true\n"
478+ r1 := c.pushN(1, workflowAt(top, ""))
479+ job1 := runner.await()
480+ watch, err := runner.jobAPI(job1).runner.WatchJob(c.ctx, connect.NewRequest(&pb.WatchJobRequest{JobId: job1.GetJobId()}))
481+ require.NoError(t, err)
482+
483+ // The cancel of the running run cannot be written.
484+ fs.match = func(w ports.CIWrite) bool {
485+ for _, r := range w.Runs {
486+ if r.ID == r1.GetId() && r.CancelRequested {
487+ return true
488+ }
489+ }
490+ return false
491+ }
492+ fs.fails.Store(1 << 20)
493+ c.pushN(2, workflowAt(top, ""))
494+ c.waitFor("the cancel fails", func() bool { return fs.hits.Load() >= 1 })
495+ rec := c.groupRecord("Deploy")
496+ require.NotNil(t, rec.Running)
497+ assert.True(t, rec.Running.Canceling, "the victim stays named, marked")
498+ assert.Equal(t, r1.GetId(), rec.Running.RunID)
499+
500+ fs.fails.Store(0)
501+ c.forge.CI.ReconcileConcurrency(c.ctx)
502+ var cancel *pb.CancelRequest
503+ for watch.Receive() {
504+ if m := watch.Msg().GetCancel(); m != nil {
505+ cancel = m
506+ break
507+ }
508+ }
509+ require.NotNil(t, cancel, "the runner is told after the reconciliation")
510+ // The runner stops: its watch ends with the job.
511+ _, err = runner.jobAPI(job1).runner.CompleteJob(c.ctx, connect.NewRequest(&pb.CompleteJobRequest{
512+ JobId: job1.GetJobId(), Conclusion: pb.CiConclusion_CI_CONCLUSION_CANCELLED,
513+ }))
514+ require.NoError(t, err)
515+}
516+
517+// Crash between the write of the group and the write of the entry: the group
518+// names a pending run whose own record does not say it waits yet. It is not
519+// stale and not cancelled; the reconciliation plans it, which marks it
520+// pending, and the group goes on to it when the holder ends.
521+func TestCIConcurrencyCrashBetweenGroupAndEntry(t *testing.T) {
522+ c, runner := concurrencyHarness(t)
523+ c.pushN(1, workflowAt(wfGroup, ""))
524+ job1 := runner.await()
525+
526+ // Stop the planning of the second run from marking it, as a crash would:
527+ // name it in the group by hand, behind the first.
528+ store := c.forge.store.CI()
529+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
530+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
531+ run, err := store.GetRun(c.ctx, c.h("proj"), r2.GetId())
532+ require.NoError(t, err)
533+ run.Concurrency.State = domain.CIConcurrencyIdle
534+ jobs, err := store.ListJobs(c.ctx, r2.GetId(), 1)
535+ require.NoError(t, err)
536+ for i := range jobs {
537+ jobs[i].Concurrency.State = domain.CIConcurrencyIdle
538+ }
539+ _, err = store.Apply(c.ctx, ports.CIWrite{Runs: []domain.CIRun{run}, Jobs: jobs})
540+ require.NoError(t, err)
541+ rec := c.groupRecord("Deploy")
542+ require.NotNil(t, rec.Pending)
543+ require.Equal(t, r2.GetId(), rec.Pending.RunID, "the group names the run")
544+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", *rec.Pending), "and marks it, in the same batch")
545+
546+ c.forge.CI.ReconcileConcurrency(c.ctx)
547+ c.waitFor("the entry is marked pending again", func() bool {
548+ r, err := store.GetRun(c.ctx, c.h("proj"), r2.GetId())
549+ return err == nil && r.Concurrency.State == domain.CIConcurrencyPending
550+ })
551+ time.Sleep(200 * time.Millisecond)
552+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2), "a named entry is not a replaced one")
553+ rec = c.groupRecord("Deploy")
554+ require.NotNil(t, rec.Pending)
555+ assert.Equal(t, r2.GetId(), rec.Pending.RunID)
556+
557+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
558+ assert.Equal(t, r2.GetId(), runner.await().GetRunId())
559+}
560+
561+// holdsGroup matches the write that tells a run it holds its group.
562+func holdsGroup(runID int64) func(ports.CIWrite) bool {
563+ return func(w ports.CIWrite) bool {
564+ for _, r := range w.Runs {
565+ if r.ID == runID && r.Concurrency.State == domain.CIConcurrencyHolding {
566+ return true
567+ }
568+ }
569+ return false
570+ }
571+}
572+
573+// HIGH-2: a promotion whose planning fails once is tried again.
574+func TestCIConcurrencyPromotionIsRetried(t *testing.T) {
575+ c, runner, fs := flakyConcurrencyHarness(t, 0)
576+ c.pushN(1, workflowAt(wfGroup, ""))
577+ job1 := runner.await()
578+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
579+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
580+
581+ fs.match = holdsGroup(r2.GetId())
582+ fs.fails.Store(1)
583+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
584+ c.waitFor("the first planning failed", func() bool { return fs.hits.Load() == 1 })
585+ job2 := runner.await()
586+ assert.Equal(t, r2.GetId(), job2.GetRunId(), "the retry planned the promoted run")
587+}
588+
589+// HIGH-2: a holder that was never told it holds the group is planned by the
590+// reconciliation, whatever happened to the retries.
591+func TestCIConcurrencyReconcilePlansAStaleHolder(t *testing.T) {
592+ c, runner, fs := flakyConcurrencyHarness(t, -1)
593+ c.pushN(1, workflowAt(wfGroup, ""))
594+ job1 := runner.await()
595+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
596+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
597+
598+ fs.match = holdsGroup(r2.GetId())
599+ fs.fails.Store(1000)
600+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
601+ c.waitFor("the planning fails", func() bool { return fs.hits.Load() >= 1 })
602+ rec := c.groupRecord("Deploy")
603+ require.NotNil(t, rec.Running)
604+ assert.Equal(t, r2.GetId(), rec.Running.RunID, "the group names the promoted run")
605+ c.waitJobs("which still shows pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
606+
607+ fs.fails.Store(0)
608+ c.forge.CI.ReconcileConcurrency(c.ctx)
609+ job2 := runner.await()
610+ assert.Equal(t, r2.GetId(), job2.GetRunId())
611+}
612+
613+const siblingJobs = ` steps:
614+ - run: echo grouped
615+ sibling:
616+ runs-on: ubuntu-latest
617+ steps:
618+ - run: echo sibling
619+`
620+
621+// jobWorkflow is a workflow of a grouped job with the given concurrency lines
622+// and an unrelated sibling.
623+func jobWorkflow(concurrency string) string {
624+ return "name: ci\non:\n push:\n branches: [main]\njobs:\n grouped:\n runs-on: ubuntu-latest\n" +
625+ concurrency + siblingJobs
626+}
627+
628+// awaitKey fetches jobs until the one of the run with the key comes.
629+func (r *fakeRunner) awaitKey(runID int64, key string) *pb.Job {
630+ r.c.t.Helper()
631+ for range 20 {
632+ if j := r.await(); j.GetRunId() == runID && j.GetJobKey() == key {
633+ return j
634+ }
635+ }
636+ r.c.t.Fatalf("no job %s of run %d", key, runID)
637+ return nil
638+}
639+
640+// T-2: a job-level cancel-in-progress (true, or an expression) cancels the
641+// running job of the group only: its sibling goes on, and the replacement
642+// becomes assignable.
643+func TestCIJobConcurrencyCancelInProgress(t *testing.T) {
644+ for _, tt := range []struct{ name, cancel string }{
645+ {"true", "true"},
646+ {"expression", "${{ github.ref == 'refs/heads/main' }}"},
647+ } {
648+ t.Run(tt.name, func(t *testing.T) {
649+ c, runner := concurrencyHarness(t)
650+ wf := jobWorkflow(" concurrency:\n group: jobgroup\n cancel-in-progress: " + tt.cancel + "\n")
651+ r1 := c.pushN(1, wf)
652+ grouped1, sibling1 := runner.awaitKey(r1.GetId(), "grouped"), runner.awaitKey(r1.GetId(), "sibling")
653+ watch, err := runner.jobAPI(grouped1).runner.WatchJob(c.ctx, connect.NewRequest(&pb.WatchJobRequest{JobId: grouped1.GetJobId()}))
654+ require.NoError(t, err)
655+
656+ r2 := c.pushN(2, wf)
657+ var cancel *pb.CancelRequest
658+ for watch.Receive() {
659+ if m := watch.Msg().GetCancel(); m != nil {
660+ cancel = m
661+ break
662+ }
663+ }
664+ require.NotNil(t, cancel, "the grouped job is asked to stop")
665+ assert.Contains(t, cancel.GetMessage(), "higher priority waiting request for 'jobgroup'")
666+
667+ // The sibling of the first run was not touched.
668+ runner.heartbeat(sibling1)
669+ for _, j := range c.jobsOf(r1) {
670+ if j.GetKey() == "sibling" {
671+ assert.NotEqual(t, pb.CiStatus_CI_STATUS_COMPLETED, j.GetStatus())
672+ assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_UNSPECIFIED, j.GetConclusion())
673+ }
674+ }
675+ _, err = runner.jobAPI(grouped1).runner.CompleteJob(c.ctx, connect.NewRequest(&pb.CompleteJobRequest{
676+ JobId: grouped1.GetJobId(), Conclusion: pb.CiConclusion_CI_CONCLUSION_CANCELLED,
677+ }))
678+ require.NoError(t, err)
679+ runner.runJob(sibling1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
680+
681+ grouped2 := runner.awaitKey(r2.GetId(), "grouped")
682+ assert.Equal(t, r2.GetId(), grouped2.GetRunId(), "the replacement is assignable")
683+ c.waitFor("the first run is over", func() bool { return c.getRun(r1.GetId()).GetRun().GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
684+ for _, j := range c.jobsOf(r1) {
685+ want := pb.CiConclusion_CI_CONCLUSION_CANCELLED
686+ if j.GetKey() == "sibling" {
687+ want = pb.CiConclusion_CI_CONCLUSION_SUCCESS
688+ }
689+ assert.Equal(t, want, j.GetConclusion(), j.GetKey())
690+ }
691+ })
692+ }
693+}
694+
695+// T-3: the server restarts: the groups are repaired at startup. A group held by
696+// an ended run is freed and the run behind it is planned; a group whose only
697+// run is over is removed.
698+func TestCIConcurrencyReconcileAtRestart(t *testing.T) {
699+ c, runner := concurrencyHarness(t)
700+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
701+ runner.await()
702+ r2 := c.pushN(2, workflowAt(wfGroup, ""))
703+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
704+
705+ // The first run ends in the store without the release (the crash), and a
706+ // second group is held by a run that is over.
707+ store := c.forge.store.CI()
708+ jobs, err := store.ListJobs(c.ctx, r1.GetId(), 1)
709+ require.NoError(t, err)
710+ run, err := store.GetRun(c.ctx, c.h("proj"), r1.GetId())
711+ require.NoError(t, err)
712+ jobs[0].Phase, jobs[0].Conclusion = domain.CIPhaseEnded, domain.CISuccess
713+ run.Status, run.Conclusion, run.CompletedAt = domain.CICompleted, domain.CISuccess, time.Now()
714+ _, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs, Runs: []domain.CIRun{run}})
715+ require.NoError(t, err)
716+ c.holdGroup("Stale", &domain.CIConcurrencyEntry{RunID: r1.GetId(), Attempt: 1}, nil)
717+ rec := c.groupRecord("Deploy")
718+ require.NotNil(t, rec.Running)
719+ require.Equal(t, r1.GetId(), rec.Running.RunID, "the record still holds the ended run")
720+
721+ // Restart on the same data.
722+ cfg, hookExe := c.forge.cfg, binary(t)
723+ c.srv.Close()
724+ c.forge.Close(c.ctx)
725+ f2, err := newForge(c.ctx, cfg, options{hookExe: hookExe, ci: func(cc *usecase.CIConfig) {
726+ cc.CancelGrace, cc.Tick, cc.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
727+ }})
728+ require.NoError(t, err)
729+ t.Cleanup(func() { f2.Close(context.WithoutCancel(c.ctx)) })
730+ srv := httptest.NewServer(f2.server.Handler())
731+ t.Cleanup(srv.Close)
732+
733+ var got *pb.Job
734+ c.waitFor("the waiting run is planned by the startup repair", func() bool {
735+ res, err := newClients(srv.URL+"/-/api", runner.secret).runner.FetchJob(c.ctx, connect.NewRequest(&pb.FetchJobRequest{}))
736+ if err == nil && res.Msg.GetJob() != nil {
737+ got = res.Msg.GetJob()
738+ }
739+ return got != nil
740+ })
741+ assert.Equal(t, r2.GetId(), got.GetRunId())
742+ groups, err := f2.store.CI().ListConcurrency(c.ctx)
743+ require.NoError(t, err)
744+ require.Len(t, groups, 1, "the stale group is gone")
745+ assert.Equal(t, "DEPLOY", groups[0].Group)
746+ require.NotNil(t, groups[0].State.Record.Running)
747+ assert.Equal(t, r2.GetId(), groups[0].State.Record.Running.RunID)
748+ assert.Nil(t, groups[0].State.Record.Pending)
749+}
750+
751+// Both writes of a run's own record fail when it queues (its "pending" state),
752+// and the cancel of the run fails for good: the group alone knows B. A newer run
753+// replaces B; B's mark is "replaced", written in the batch that unnamed it. After
754+// the store works again the reconciliation finds B from that mark and cancels it.
755+func TestCIConcurrencyReplacedRunWithLostWritesIsCancelled(t *testing.T) {
756+ c, runner, fs := flakyConcurrencyHarness(t, -1)
757+ r1 := c.pushN(1, workflowAt(wfGroup, ""))
758+ runner.await()
759+
760+ // B queues, but no run after the first gets the pending state in its own
761+ // record, and no cancel lands.
762+ fs.match = func(w ports.CIWrite) bool {
763+ for _, r := range w.Runs {
764+ if r.ID > r1.GetId() && r.Concurrency.State == domain.CIConcurrencyPending {
765+ return true
766+ }
767+ }
768+ for _, j := range w.Jobs {
769+ if j.RunID > r1.GetId() && j.Ended() && j.Conclusion == domain.CICancelled {
770+ return true
771+ }
772+ }
773+ return false
774+ }
775+ fs.fails.Store(1 << 20)
776+ b := c.pushN(2, workflowAt(wfGroup, ""))
777+ c.waitFor("B is named by the group", func() bool {
778+ rec := c.groupRecord("Deploy")
779+ return rec.Pending != nil && rec.Pending.RunID == b.GetId()
780+ })
781+ bEntry := domain.CIConcurrencyEntry{RunID: b.GetId(), Attempt: 1}
782+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("Deploy", bEntry), "the group batch marked it")
783+
784+ // C replaces B: B is unnamed and marked replaced in one batch.
785+ c.pushN(3, workflowAt(wfGroup, ""))
786+ c.waitFor("B is marked replaced", func() bool { return c.markState("Deploy", bEntry) == domain.CIConcurrencyReplaced })
787+ rec := c.groupRecord("Deploy")
788+ require.NotNil(t, rec.Pending)
789+ assert.NotEqual(t, b.GetId(), rec.Pending.RunID, "B is no longer named")
790+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(b))
791+
792+ fs.fails.Store(0)
793+ c.forge.CI.ReconcileConcurrency(c.ctx)
794+ c.waitFor("B ends cancelled", func() bool { return c.conclusionOf(b) == pb.CiConclusion_CI_CONCLUSION_CANCELLED })
795+ assert.True(t, c.cancelAnnotation(b, "Deploy"))
796+ c.waitFor("its mark goes with it", func() bool { return c.markState("Deploy", bEntry) == "" })
797+}
798+
799+// A grouped job B is replaced by C, the cancel never lands, C is promoted when
800+// the running job ends, and an unrelated sibling of B completes and advances B:
801+// B's grouped job must not take the free place, it ends cancelled and never runs.
802+func TestCIJobConcurrencyReplacedJobNeverAcquiresAgain(t *testing.T) {
803+ c, runner, fs := flakyConcurrencyHarness(t, -1)
804+ wf := jobWorkflow(" concurrency: jobgroup\n")
805+ r1 := c.pushN(1, wf)
806+ grouped1, sibling1 := runner.awaitKey(r1.GetId(), "grouped"), runner.awaitKey(r1.GetId(), "sibling")
807+
808+ rB := c.pushN(2, wf)
809+ siblingB := runner.awaitKey(rB.GetId(), "sibling") // B's unrelated job is running
810+ c.waitJobs("B's grouped job waits", rB, func(js []*pb.RunJob) bool {
811+ for _, j := range js {
812+ if j.GetKey() == "grouped" {
813+ return j.GetStatus() == pb.CiStatus_CI_STATUS_PENDING
814+ }
815+ }
816+ return false
817+ })
818+
819+ fs.match = endsCancelled(rB.GetId())
820+ fs.fails.Store(1 << 20)
821+ rC := c.pushN(3, wf)
822+ bEntry := domain.CIConcurrencyEntry{RunID: rB.GetId(), Attempt: 1, JobID: domain.NewJobID(rB.GetId(), 0, 0)}
823+ c.waitFor("B's grouped job is replaced", func() bool { return c.markState("jobgroup", bEntry) == domain.CIConcurrencyReplaced })
824+
825+ // The running job ends: C's grouped job is promoted.
826+ runner.runJob(grouped1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
827+ runner.runJob(sibling1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
828+ groupedC := runner.awaitKey(rC.GetId(), "grouped")
829+ rec := c.groupRecord("jobgroup")
830+ require.NotNil(t, rec.Running)
831+ assert.Equal(t, rC.GetId(), rec.Running.RunID)
832+
833+ // B's sibling completes: B is advanced, its grouped job asks for the group.
834+ runner.runJob(siblingB, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
835+ c.waitFor("B was advanced", func() bool {
836+ for _, j := range c.jobsOf(rB) {
837+ if j.GetKey() == "sibling" {
838+ return j.GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED
839+ }
840+ }
841+ return false
842+ })
843+ time.Sleep(300 * time.Millisecond)
844+ rec = c.groupRecord("jobgroup")
845+ require.NotNil(t, rec.Running)
846+ assert.Equal(t, rC.GetId(), rec.Running.RunID, "C still holds the group")
847+ if rec.Pending != nil {
848+ assert.NotEqual(t, rB.GetId(), rec.Pending.RunID, "B did not take the waiting place back")
849+ }
850+ // Nothing of B's grouped job is given to a runner.
851+ for range 5 {
852+ j, err := runner.fetch("")
853+ require.NoError(t, err)
854+ if j != nil {
855+ assert.False(t, j.GetRunId() == rB.GetId() && j.GetJobKey() == "grouped", "B's grouped job must not run")
856+ }
857+ }
858+
859+ // Once the store works, the reconciliation cancels it.
860+ fs.fails.Store(0)
861+ c.forge.CI.ReconcileConcurrency(c.ctx)
862+ c.waitFor("B's grouped job ends cancelled", func() bool {
863+ for _, j := range c.jobsOf(rB) {
864+ if j.GetKey() == "grouped" {
865+ return j.GetConclusion() == pb.CiConclusion_CI_CONCLUSION_CANCELLED
866+ }
867+ }
868+ return false
869+ })
870+ runner.runJob(groupedC, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
871+}
872+
873+// The same, with an empty group: B's grouped job is replaced and cancelled
874+// before its replacement ends too; a replaced job never runs even when the
875+// group is free.
876+func TestCIJobConcurrencyReplacedJobDoesNotTakeAnEmptyGroup(t *testing.T) {
877+ c, runner, fs := flakyConcurrencyHarness(t, -1)
878+ wf := jobWorkflow(" concurrency: jobgroup\n")
879+ r1 := c.pushN(1, wf)
880+ grouped1, sibling1 := runner.awaitKey(r1.GetId(), "grouped"), runner.awaitKey(r1.GetId(), "sibling")
881+ rB := c.pushN(2, wf)
882+ siblingB := runner.awaitKey(rB.GetId(), "sibling")
883+ c.waitJobs("pending", rB, func(js []*pb.RunJob) bool {
884+ for _, j := range js {
885+ if j.GetKey() == "grouped" {
886+ return j.GetStatus() == pb.CiStatus_CI_STATUS_PENDING
887+ }
888+ }
889+ return false
890+ })
891+ fs.match = endsCancelled(rB.GetId())
892+ fs.fails.Store(1 << 20)
893+ rC := c.pushN(3, wf)
894+ bEntry := domain.CIConcurrencyEntry{RunID: rB.GetId(), Attempt: 1, JobID: domain.NewJobID(rB.GetId(), 0, 0)}
895+ c.waitFor("replaced", func() bool { return c.markState("jobgroup", bEntry) == domain.CIConcurrencyReplaced })
896+
897+ // Both the running job and the promoted one finish: the group is empty.
898+ runner.runJob(grouped1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
899+ runner.runJob(sibling1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
900+ runner.runJob(runner.awaitKey(rC.GetId(), "grouped"), pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
901+ c.waitFor("the group is free", func() bool { return c.groupRecord("jobgroup").Running == nil })
902+
903+ runner.runJob(siblingB, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil) // advances B
904+ time.Sleep(300 * time.Millisecond)
905+ assert.Nil(t, c.groupRecord("jobgroup").Running, "B's replaced job did not take the empty group")
906+ for range 5 {
907+ j, err := runner.fetch("")
908+ require.NoError(t, err)
909+ if j != nil {
910+ assert.False(t, j.GetRunId() == rB.GetId() && j.GetJobKey() == "grouped", "B's grouped job must not run")
911+ }
912+ }
913+ fs.fails.Store(0)
914+ c.forge.CI.ReconcileConcurrency(c.ctx)
915+ c.waitFor("it ends cancelled", func() bool {
916+ for _, j := range c.jobsOf(rB) {
917+ if j.GetKey() == "grouped" {
918+ return j.GetConclusion() == pb.CiConclusion_CI_CONCLUSION_CANCELLED
919+ }
920+ }
921+ return false
922+ })
923+}
924+
925+// Crash between the group batch and the write of a JOB's own state: the group
926+// names (and marks) the grouped job as waiting, the job's record does not say so
927+// yet. It is not replaced and not cancelled; reconciliation tells it, and it runs
928+// when its turn comes.
929+func TestCIJobConcurrencyCrashBetweenGroupAndJob(t *testing.T) {
930+ c, runner := concurrencyHarness(t)
931+ group := " concurrency: jobgroup\n"
932+ c.pushN(1, workflowAt("", group))
933+ job1 := runner.await()
934+ r2 := c.pushN(2, workflowAt("", group))
935+ c.waitJobs("pending", r2, allStatus(pb.CiStatus_CI_STATUS_PENDING))
936+
937+ store := c.forge.store.CI()
938+ jobs, err := store.ListJobs(c.ctx, r2.GetId(), 1)
939+ require.NoError(t, err)
940+ require.Len(t, jobs, 1)
941+ entry := domain.CIConcurrencyEntry{RunID: r2.GetId(), Attempt: 1, JobID: jobs[0].ID}
942+ jobs[0].Concurrency.State = domain.CIConcurrencyIdle // the write that was lost
943+ _, err = store.Apply(c.ctx, ports.CIWrite{Jobs: jobs})
944+ require.NoError(t, err)
945+ rec := c.groupRecord("jobgroup")
946+ require.NotNil(t, rec.Pending)
947+ assert.Equal(t, entry.JobID, rec.Pending.JobID, "the group names the job")
948+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("jobgroup", entry), "and its mark says so")
949+
950+ c.forge.CI.ReconcileConcurrency(c.ctx)
951+ c.waitFor("the job is told it waits", func() bool {
952+ js, err := store.ListJobs(c.ctx, r2.GetId(), 1)
953+ return err == nil && js[0].Concurrency.State == domain.CIConcurrencyPending
954+ })
955+ time.Sleep(200 * time.Millisecond)
956+ assert.NotEqual(t, pb.CiConclusion_CI_CONCLUSION_CANCELLED, c.conclusionOf(r2), "a named job is not a replaced one")
957+ assert.Equal(t, domain.CIConcurrencyPending, c.markState("jobgroup", entry))
958+
959+ runner.runJob(job1, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
960+ assert.Equal(t, r2.GetId(), runner.await().GetRunId(), "it runs when the group is free")
961+}
962+
963+// plannedWrite matches a write of a run after the first that stores planned jobs.
964+func plannedWrite(after int64) func(ports.CIWrite) bool {
965+ return func(w ports.CIWrite) bool {
966+ for _, j := range w.Jobs {
967+ if j.RunID > after && j.Planned {
968+ return true
969+ }
970+ }
971+ return false
972+ }
973+}
974+
975+const matrixGroupWorkflow = `name: b
976+on:
977+ push:
978+ branches: [other]
979+jobs:
980+ build:
981+ runs-on: ubuntu-latest
982+ strategy:
983+ matrix:
984+ n: [1, 2, 3]
985+ concurrency: shared
986+ steps:
987+ - run: echo ${{ matrix.n }}
988+`
989+
990+const holderWorkflow = `name: a
991+on:
992+ push:
993+ branches: [main]
994+jobs:
995+ hold:
996+ runs-on: ubuntu-latest
997+ concurrency: shared
998+ steps:
999+ - run: echo hold
1000+`
1001+
1002+// Run B plans three legs of a matrix that share a group held by A. The legs
1003+// replace one another (leg 0 by leg 1 by leg 2: only one waits), the group
1004+// batches commit, and the write of B's planned jobs fails, leaving the
1005+// placeholder of the job in the store. After the store recovers, the
1006+// reconciliation must plan B before it cancels anything: only legs 0 and 1
1007+// end cancelled, and leg 2, the legitimate waiting one, runs when A ends.
1008+func TestCIJobConcurrencyRecoveryPlansTheMatrixBeforeCancellingALeg(t *testing.T) {
1009+ c, runner, fs := flakyConcurrencyHarness(t, -1)
1010+ a := c.push("main", map[string]string{".github/workflows/a.yml": holderWorkflow}, "")
1011+ _ = a
1012+ holder := runner.await()
1013+ require.Equal(t, "hold", holder.GetJobKey())
1014+ aRun := holder.GetRunId()
1015+
1016+ fs.match = plannedWrite(aRun)
1017+ fs.fails.Store(1 << 20)
1018+ c.push("other", map[string]string{".github/workflows/b.yml": matrixGroupWorkflow}, "")
1019+ c.waitFor("B's run exists", func() bool {
1020+ for _, r := range c.runs() {
1021+ if r.GetId() > aRun {
1022+ return true
1023+ }
1024+ }
1025+ return false
1026+ })
1027+ var bRun *pb.Run
1028+ for _, r := range c.runs() {
1029+ if r.GetId() > aRun {
1030+ bRun = r
1031+ }
1032+ }
1033+ require.NotNil(t, bRun)
1034+ g := domain.ConcurrencyKey("shared")
1035+ c.waitFor("the group batches committed: two legs are marked replaced", func() bool {
1036+ replaced := 0
1037+ for _, m := range c.groupState(g).Marks {
1038+ if m.Entry.RunID == bRun.GetId() && m.State == domain.CIConcurrencyReplaced {
1039+ replaced++
1040+ }
1041+ }
1042+ return replaced == 2
1043+ })
1044+ assert.Equal(t, 1, len(c.jobsOf(bRun)), "B's job is still the placeholder: its planned legs were never stored")
1045+
1046+ fs.fails.Store(0)
1047+ c.forge.CI.ReconcileConcurrency(c.ctx)
1048+ c.waitFor("B has its three legs", func() bool { return len(c.jobsOf(bRun)) == 3 })
1049+ c.waitFor("legs 0 and 1 end cancelled", func() bool {
1050+ n := 0
1051+ for _, j := range c.jobsOf(bRun) {
1052+ if j.GetConclusion() == pb.CiConclusion_CI_CONCLUSION_CANCELLED {
1053+ n++
1054+ }
1055+ }
1056+ return n == 2
1057+ })
1058+ byCombo := map[string]*pb.RunJob{}
1059+ for _, j := range c.jobsOf(bRun) {
1060+ byCombo[j.GetName()] = j
1061+ }
1062+ var waiting []*pb.RunJob
1063+ for _, j := range c.jobsOf(bRun) {
1064+ if j.GetConclusion() != pb.CiConclusion_CI_CONCLUSION_CANCELLED {
1065+ waiting = append(waiting, j)
1066+ }
1067+ }
1068+ require.Len(t, waiting, 1, "exactly one leg is not cancelled")
1069+ assert.Equal(t, pb.CiStatus_CI_STATUS_PENDING, waiting[0].GetStatus(), "the final leg waits for A")
1070+ assert.Contains(t, waiting[0].GetName(), "3", "and it is the last leg")
1071+
1072+ runner.runJob(holder, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1073+ last := runner.await()
1074+ assert.Equal(t, bRun.GetId(), last.GetRunId())
1075+ assert.Equal(t, waiting[0].GetId(), last.GetJobId(), "the final leg runs when A ends")
1076+ runner.runJob(last, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1077+}
1078+
1079+// Stale marks (of runs that are gone) are pruned a bounded number per
1080+// transaction: a decision on a group with a long backlog succeeds at once, and
1081+// the reconciliation takes the others, a bounded batch at a time.
1082+func TestCIConcurrencyPrunesStaleMarksInBoundedBatches(t *testing.T) {
1083+ const stale, bound = 10, 3
1084+ c := newCIHarness(t, func(cfg *usecase.CIConfig) {
1085+ cfg.CancelGrace, cfg.Tick, cfg.PollTimeout = 300*time.Millisecond, 20*time.Millisecond, 100*time.Millisecond
1086+ cfg.ConcurrencyPruneBatch = bound
1087+ })
1088+ runner := c.registerRunner("alpha", "ubuntu-latest")
1089+ require.NoError(t, c.forge.store.CI().UpdateConcurrency(c.ctx, c.h("proj"), "Deploy", func(st *domain.CIConcurrencyGroupState) error {
1090+ st.Marks = map[string]domain.CIConcurrencyMark{}
1091+ for i := range stale {
1092+ e := domain.CIConcurrencyEntry{RunID: int64(9000 + i), Attempt: 1, JobID: "9000-0-0"}
1093+ st.Marks[e.Key()] = domain.CIConcurrencyMark{Entry: e, Group: "Deploy", State: domain.CIConcurrencyPending}
1094+ }
1095+ return nil
1096+ }))
1097+ require.Len(t, c.groupState("Deploy").Marks, stale)
1098+
1099+ r := c.pushN(1, workflowAt(wfGroup, ""))
1100+ job := runner.await()
1101+ assert.Equal(t, r.GetId(), job.GetRunId(), "the decision succeeded with the backlog in place")
1102+ st := c.groupState("Deploy")
1103+ require.NotNil(t, st.Record.Running)
1104+ assert.Equal(t, r.GetId(), st.Record.Running.RunID)
1105+ assert.Len(t, st.Marks, stale-bound+1, "one transaction pruned a bounded number of the stale marks, and marked the run")
1106+
1107+ c.forge.CI.ReconcileConcurrency(c.ctx)
1108+ st = c.groupState("Deploy")
1109+ assert.Len(t, st.Marks, 1, "the reconciliation prunes the rest, a batch at a time")
1110+ assert.Contains(t, st.Marks, domain.CIConcurrencyEntry{RunID: r.GetId(), Attempt: 1}.Key())
1111+ runner.runJob(job, pb.CiConclusion_CI_CONCLUSION_SUCCESS, nil)
1112+}
M internal/app/ci_flow_test.go +2 -5
@@ -323,15 +323,12 @@ func TestCIUnsupportedFeatureFailsTheJob(t *testing.T) {
323323 c.push("main", map[string]string{".github/workflows/u.yml": `on: push
324324 jobs:
325325 j:
326- runs-on: ubuntu-latest
327- concurrency: group
328- steps:
329- - run: echo
326+ uses: ./.github/workflows/callee.yml
330327 `}, "")
331328 c.waitFor("the run fails", func() bool { return len(c.runs()) == 1 && c.runs()[0].GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
332329 j := c.getRun(c.runs()[0].GetId()).GetJobs()[0]
333330 assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, j.GetConclusion())
334- assert.Equal(t, "not supported yet: concurrency", j.GetError())
331+ assert.Equal(t, "not supported yet: reusable workflows (uses at the job level)", j.GetError())
335332 }
336333
337334 // A job for a label no runner has is "waiting for a runner" and fails after the
@@ -323,15 +323,12 @@ func TestCIUnsupportedFeatureFailsTheJob(t *testing.T) {
323 c.push("main", map[string]string{".github/workflows/u.yml": `on: push323 c.push("main", map[string]string{".github/workflows/u.yml": `on: push
324 jobs:324 jobs:
325 j:325 j:
326- runs-on: ubuntu-latest326+ uses: ./.github/workflows/callee.yml
327- concurrency: group
328- steps:
329- - run: echo
330 `}, "")327 `}, "")
331 c.waitFor("the run fails", func() bool { return len(c.runs()) == 1 && c.runs()[0].GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })328 c.waitFor("the run fails", func() bool { return len(c.runs()) == 1 && c.runs()[0].GetStatus() == pb.CiStatus_CI_STATUS_COMPLETED })
332 j := c.getRun(c.runs()[0].GetId()).GetJobs()[0]329 j := c.getRun(c.runs()[0].GetId()).GetJobs()[0]
333 assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, j.GetConclusion())330 assert.Equal(t, pb.CiConclusion_CI_CONCLUSION_FAILURE, j.GetConclusion())
334- assert.Equal(t, "not supported yet: concurrency", j.GetError())331+ assert.Equal(t, "not supported yet: reusable workflows (uses at the job level)", j.GetError())
335 }332 }
336 333
337 // A job for a label no runner has is "waiting for a runner" and fails after the334 // A job for a label no runner has is "waiting for a runner" and fails after the
M internal/app/ci_parity_test.go +1 -3
@@ -308,8 +308,7 @@ jobs:
308308 }
309309
310310 // CI-2: planning runs what CI-1 refused (uses, container, services, a step
311-// timeout) and CI-3 environments, and still refuses job-level uses and
312-// concurrency.
311+// timeout) and CI-3 environments, and still refuses job-level uses.
313312 func TestCIPlanningAcceptsCI2Features(t *testing.T) {
314313 c := newCIHarness(t, nil)
315314 runner := c.registerRunner("alpha", "ubuntu-latest")
@@ -331,7 +330,6 @@ jobs:
331330
332331 for _, tc := range []struct{ name, job, want string }{
333332 {"reusable", "uses: ./.github/workflows/x.yml", "not supported yet: reusable workflows (uses at the job level)"},
334- {"concurrency", "runs-on: ubuntu-latest\n concurrency: g\n steps:\n - run: echo", "not supported yet: concurrency"},
335333 } {
336334 t.Run(tc.name, func(t *testing.T) {
337335 body := "on: push\njobs:\n r:\n " + tc.job + "\n"
@@ -308,8 +308,7 @@ jobs:
308 }308 }
309 309
310 // CI-2: planning runs what CI-1 refused (uses, container, services, a step310 // CI-2: planning runs what CI-1 refused (uses, container, services, a step
311-// timeout) and CI-3 environments, and still refuses job-level uses and311+// timeout) and CI-3 environments, and still refuses job-level uses.
312-// concurrency.
313 func TestCIPlanningAcceptsCI2Features(t *testing.T) {312 func TestCIPlanningAcceptsCI2Features(t *testing.T) {
314 c := newCIHarness(t, nil)313 c := newCIHarness(t, nil)
315 runner := c.registerRunner("alpha", "ubuntu-latest")314 runner := c.registerRunner("alpha", "ubuntu-latest")
@@ -331,7 +330,6 @@ jobs:
331 330
332 for _, tc := range []struct{ name, job, want string }{331 for _, tc := range []struct{ name, job, want string }{
333 {"reusable", "uses: ./.github/workflows/x.yml", "not supported yet: reusable workflows (uses at the job level)"},332 {"reusable", "uses: ./.github/workflows/x.yml", "not supported yet: reusable workflows (uses at the job level)"},
334- {"concurrency", "runs-on: ubuntu-latest\n concurrency: g\n steps:\n - run: echo", "not supported yet: concurrency"},
335 } {333 } {
336 t.Run(tc.name, func(t *testing.T) {334 t.Run(tc.name, func(t *testing.T) {
337 body := "on: push\njobs:\n r:\n " + tc.job + "\n"335 body := "on: push\njobs:\n r:\n " + tc.job + "\n"
M internal/app/cli_e2e_test.go +1 -1
@@ -326,7 +326,7 @@ func TestAC6CLIEndToEnd(t *testing.T) {
326326 ver := decode[versionOut](t, res.stdout)
327327 assert.Equal(t, c.url, ver.Server.URL)
328328 assert.NotEmpty(t, ver.Server.Version)
329- assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.secrets"}, ver.Server.Features)
329+ assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.secrets"}, ver.Server.Features)
330330 assert.Equal(t, []string{"passkeys"}, ver.Server.MissingFeatures)
331331 stdout, stderr, code = c.bin(t, secret, "version")
332332 require.Equal(t, 0, code, stderr)
@@ -326,7 +326,7 @@ func TestAC6CLIEndToEnd(t *testing.T) {
326 ver := decode[versionOut](t, res.stdout)326 ver := decode[versionOut](t, res.stdout)
327 assert.Equal(t, c.url, ver.Server.URL)327 assert.Equal(t, c.url, ver.Server.URL)
328 assert.NotEmpty(t, ver.Server.Version)328 assert.NotEmpty(t, ver.Server.Version)
329- assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.secrets"}, ver.Server.Features)329+ assert.ElementsMatch(t, []string{"repo.update", "events.resume", "merge_requests", "merge_requests.delete_branch", "signing_key", "session", "git.list_paths", "token.branch_prefixes", "ci.runs", "ci.runs.filters", "ci.runners", "ci.actions", "ci.annotations", "ci.summaries", "ci.concurrency", "ci.secrets"}, ver.Server.Features)
330 assert.Equal(t, []string{"passkeys"}, ver.Server.MissingFeatures)330 assert.Equal(t, []string{"passkeys"}, ver.Server.MissingFeatures)
331 stdout, stderr, code = c.bin(t, secret, "version")331 stdout, stderr, code = c.bin(t, secret, "version")
332 require.Equal(t, 0, code, stderr)332 require.Equal(t, 0, code, stderr)
M internal/app/cli_run.go +2 -2
@@ -505,7 +505,7 @@ func printRunSummaries(ctx context.Context, w, errw io.Writer, c *client.Client,
505505 var all []found
506506 truncated := false
507507 for _, j := range jobs {
508- if j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED {
508+ if j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED || j.GetStatus() == ermite.CiStatus_CI_STATUS_PENDING {
509509 continue
510510 }
511511 res, err := c.Runs().GetStepSummaries(ctx, connect.NewRequest(&ermite.GetStepSummariesRequest{Repo: repo, JobId: j.GetId(), Attempt: j.GetAttempt()}))
@@ -601,7 +601,7 @@ func printRunLog(ctx context.Context, w io.Writer, c *client.Client, repo string
601601 // streamJobLog writes the lines of one job. Continued pieces are joined to the
602602 // line before them, as the raw log has them.
603603 func streamJobLog(ctx context.Context, w io.Writer, c *client.Client, repo string, j *ermite.RunJob, follow bool) error {
604- if j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED && !follow {
604+ if (j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED || j.GetStatus() == ermite.CiStatus_CI_STATUS_PENDING) && !follow {
605605 noticef2(w, "job %q has not started: no log yet", j.GetName())
606606 return nil
607607 }
@@ -505,7 +505,7 @@ func printRunSummaries(ctx context.Context, w, errw io.Writer, c *client.Client,
505 var all []found505 var all []found
506 truncated := false506 truncated := false
507 for _, j := range jobs {507 for _, j := range jobs {
508- if j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED {508+ if j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED || j.GetStatus() == ermite.CiStatus_CI_STATUS_PENDING {
509 continue509 continue
510 }510 }
511 res, err := c.Runs().GetStepSummaries(ctx, connect.NewRequest(&ermite.GetStepSummariesRequest{Repo: repo, JobId: j.GetId(), Attempt: j.GetAttempt()}))511 res, err := c.Runs().GetStepSummaries(ctx, connect.NewRequest(&ermite.GetStepSummariesRequest{Repo: repo, JobId: j.GetId(), Attempt: j.GetAttempt()}))
@@ -601,7 +601,7 @@ func printRunLog(ctx context.Context, w io.Writer, c *client.Client, repo string
601 // streamJobLog writes the lines of one job. Continued pieces are joined to the601 // streamJobLog writes the lines of one job. Continued pieces are joined to the
602 // line before them, as the raw log has them.602 // line before them, as the raw log has them.
603 func streamJobLog(ctx context.Context, w io.Writer, c *client.Client, repo string, j *ermite.RunJob, follow bool) error {603 func streamJobLog(ctx context.Context, w io.Writer, c *client.Client, repo string, j *ermite.RunJob, follow bool) error {
604- if j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED && !follow {604+ if (j.GetStatus() == ermite.CiStatus_CI_STATUS_QUEUED || j.GetStatus() == ermite.CiStatus_CI_STATUS_PENDING) && !follow {
605 noticef2(w, "job %q has not started: no log yet", j.GetName())605 noticef2(w, "job %q has not started: no log yet", j.GetName())
606 return nil606 return nil
607 }607 }
M internal/domain/ci.go +12 -0
@@ -20,6 +20,9 @@ const (
2020 CIQueued CIStatus = "queued"
2121 CIInProgress CIStatus = "in_progress"
2222 CICompleted CIStatus = "completed"
23+ // CIPending is a job or a check held back by a concurrency group: another
24+ // entry of the group runs, or a newer one is waiting. Never a run's status.
25+ CIPending CIStatus = "pending"
2326 )
2427
2528 // CIConclusion is how a run, a job, a step or a check ended, with GitHub's
@@ -183,6 +186,8 @@ type CIRun struct {
183186 // keeps or lowers it, never raises it (RerunTrusted). False for a run
184187 // stored before the field existed: the safe reading.
185188 Trusted bool
189+ // Concurrency is the workflow-level concurrency of the latest attempt.
190+ Concurrency CIConcurrency
186191 // Version is the version of the stored record, which grows at every
187192 // change and never goes back.
188193 Version uint64
@@ -279,6 +284,9 @@ type CIJob struct {
279284 // MaxParallel is the max-parallel of the strategy, 0 for no limit.
280285 MaxParallel int
281286 Outputs map[string]string
287+ // Concurrency is the concurrency group of the job: its own, evaluated when
288+ // the job is planned, else the one of its run (FromRun).
289+ Concurrency CIConcurrency
282290
283291 // TokenHash is the hash of the job token of the current assignment; zero
284292 // when there is none.
@@ -305,6 +313,10 @@ func (j CIJob) Status() CIStatus {
305313 return CIInProgress
306314 case CIPhaseEnded:
307315 return CICompleted
316+ case CIPhaseWaiting:
317+ if j.Concurrency.State == CIConcurrencyPending {
318+ return CIPending
319+ }
308320 }
309321 return CIQueued
310322 }
@@ -20,6 +20,9 @@ const (
20 CIQueued CIStatus = "queued"20 CIQueued CIStatus = "queued"
21 CIInProgress CIStatus = "in_progress"21 CIInProgress CIStatus = "in_progress"
22 CICompleted CIStatus = "completed"22 CICompleted CIStatus = "completed"
23+ // CIPending is a job or a check held back by a concurrency group: another
24+ // entry of the group runs, or a newer one is waiting. Never a run's status.
25+ CIPending CIStatus = "pending"
23 )26 )
24 27
25 // CIConclusion is how a run, a job, a step or a check ended, with GitHub's28 // CIConclusion is how a run, a job, a step or a check ended, with GitHub's
@@ -183,6 +186,8 @@ type CIRun struct {
183 // keeps or lowers it, never raises it (RerunTrusted). False for a run186 // keeps or lowers it, never raises it (RerunTrusted). False for a run
184 // stored before the field existed: the safe reading.187 // stored before the field existed: the safe reading.
185 Trusted bool188 Trusted bool
189+ // Concurrency is the workflow-level concurrency of the latest attempt.
190+ Concurrency CIConcurrency
186 // Version is the version of the stored record, which grows at every191 // Version is the version of the stored record, which grows at every
187 // change and never goes back.192 // change and never goes back.
188 Version uint64193 Version uint64
@@ -279,6 +284,9 @@ type CIJob struct {
279 // MaxParallel is the max-parallel of the strategy, 0 for no limit.284 // MaxParallel is the max-parallel of the strategy, 0 for no limit.
280 MaxParallel int285 MaxParallel int
281 Outputs map[string]string286 Outputs map[string]string
287+ // Concurrency is the concurrency group of the job: its own, evaluated when
288+ // the job is planned, else the one of its run (FromRun).
289+ Concurrency CIConcurrency
282 290
283 // TokenHash is the hash of the job token of the current assignment; zero291 // TokenHash is the hash of the job token of the current assignment; zero
284 // when there is none.292 // when there is none.
@@ -305,6 +313,10 @@ func (j CIJob) Status() CIStatus {
305 return CIInProgress313 return CIInProgress
306 case CIPhaseEnded:314 case CIPhaseEnded:
307 return CICompleted315 return CICompleted
316+ case CIPhaseWaiting:
317+ if j.Concurrency.State == CIConcurrencyPending {
318+ return CIPending
319+ }
308 }320 }
309 return CIQueued321 return CIQueued
310 }322 }
A internal/domain/ci_concurrency.go +149 -0
@@ -0,0 +1,149 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package domain
4+
5+import (
6+ "strconv"
7+ "unicode"
8+ "unicode/utf16"
9+)
10+
11+// CIConcurrencyState is where a run or a job is in its concurrency group.
12+type CIConcurrencyState string
13+
14+// Concurrency states.
15+const (
16+ // CIConcurrencyIdle: the group is evaluated but the entry has not asked for
17+ // its turn yet.
18+ CIConcurrencyIdle CIConcurrencyState = ""
19+ // CIConcurrencyPending: the group record names this entry as the waiting
20+ // one, behind the running entry.
21+ CIConcurrencyPending CIConcurrencyState = "pending"
22+ // CIConcurrencyHolding: this entry is the running one of its group.
23+ CIConcurrencyHolding CIConcurrencyState = "holding"
24+ // CIConcurrencyReplaced: only in a mark. A newer entry replaced this one:
25+ // it is owed a cancellation and never acquires the group.
26+ CIConcurrencyReplaced CIConcurrencyState = "replaced"
27+)
28+
29+// CIMaxConcurrencyGroup is the longest concurrency group name GitHub accepts
30+// (actions/runner WorkflowTemplateConverter.cs:679-687, "must be less than 400
31+// characters").
32+const CIMaxConcurrencyGroup = 400
33+
34+// CIConcurrency is the evaluated "concurrency:" of a run (workflow level) or of
35+// a job.
36+type CIConcurrency struct {
37+ // Group is the evaluated group as written; "" for none. Groups are compared
38+ // without regard to case (ConcurrencyKey).
39+ Group string
40+ // Cancel is the evaluated cancel-in-progress.
41+ Cancel bool
42+ // State is written to the record of the entry only after the group record
43+ // named it (acquire committed): an entry that waits in a group that does
44+ // not name it any more was replaced, and is cancelled.
45+ State CIConcurrencyState
46+ // Evaluated is set once the expressions were evaluated, whatever the result
47+ // (a run starts unevaluated, and a re-run evaluates again).
48+ Evaluated bool
49+ // FromRun is set on a job that shows the group of its run: the entry is the
50+ // run's, the job holds nothing of its own.
51+ FromRun bool
52+ // Preempted is set when a newer entry of the group cancelled this one.
53+ Preempted bool
54+}
55+
56+// ConcurrencyKey is the identity of a group within a repository. GitHub compares
57+// groups with .NET's OrdinalIgnoreCase, which compares strings upper-cased
58+// character by character with the simple case mapping (OrdinalCasing in
59+// dotnet/runtime, System.Private.CoreLib/src/System/Globalization/
60+// OrdinalCasing.cs; the source was not available here, so this follows its
61+// documented rule). A character that is not ASCII and whose upper case is ASCII
62+// keeps itself: ı (U+0131) and ſ (U+017F) are neither I nor S. So σ, ς and Σ are
63+// one letter, ı and İ are distinct from I, and i equals I. The key is upper
64+// case and is never lowered back.
65+func ConcurrencyKey(group string) string {
66+ out := make([]rune, 0, len(group))
67+ for _, r := range group {
68+ if u := unicode.ToUpper(r); r <= unicode.MaxASCII || u > unicode.MaxASCII {
69+ r = u
70+ }
71+ out = append(out, r)
72+ }
73+ return string(out)
74+}
75+
76+// ConcurrencyGroupLength is the length of a group as C#'s String.Length counts
77+// it: in UTF-16 code units.
78+func ConcurrencyGroupLength(group string) int { return len(utf16.Encode([]rune(group))) }
79+
80+// ConcurrencyCancelMessage is GitHub's annotation for an entry cancelled by a
81+// newer one (docs.github.com, "Control the concurrency of workflows and jobs").
82+func ConcurrencyCancelMessage(group string) string {
83+ return "Canceling since a higher priority waiting request for '" + group + "' exists"
84+}
85+
86+// CIConcurrencyEntry is a run (JobID empty) or a job waiting for or holding a
87+// concurrency group.
88+type CIConcurrencyEntry struct {
89+ RunID int64
90+ Attempt int32
91+ JobID string
92+ // Cancel is the cancel-in-progress of the entry: while it waits, the
93+ // running entry of the group is cancelled.
94+ Cancel bool
95+ // Canceling is set on a running entry that a waiting entry with
96+ // cancel-in-progress asked to stop; it stays named until it has ended, and
97+ // the reconciliation asks it again.
98+ Canceling bool
99+}
100+
101+// Key identifies the entry in the marks of its group.
102+func (e CIConcurrencyEntry) Key() string {
103+ return strconv.FormatInt(e.RunID, 10) + "/" + strconv.FormatInt(int64(e.Attempt), 10) + "/" + e.JobID
104+}
105+
106+// Same reports whether two entries are the same run or job.
107+func (e CIConcurrencyEntry) Same(o CIConcurrencyEntry) bool {
108+ return e.RunID == o.RunID && e.Attempt == o.Attempt && e.JobID == o.JobID
109+}
110+
111+// CIConcurrencyRecord is the state of a group: at most one running entry and
112+// one pending entry (queue: single).
113+type CIConcurrencyRecord struct {
114+ Running *CIConcurrencyEntry
115+ Pending *CIConcurrencyEntry
116+}
117+
118+// IsEmpty reports whether nothing runs or waits in the group.
119+func (r CIConcurrencyRecord) IsEmpty() bool { return r.Running == nil && r.Pending == nil }
120+
121+// Names reports whether the record names the entry as the running or the
122+// pending one.
123+func (r CIConcurrencyRecord) Names(e CIConcurrencyEntry) bool {
124+ return (r.Running != nil && r.Running.Same(e)) || (r.Pending != nil && r.Pending.Same(e))
125+}
126+
127+// CIConcurrencyMark says where an entry stands in its group. The marks of a group
128+// are written in the same batch as its record: an entry is marked pending or
129+// holding if and only if the record names it as the pending or the running
130+// entry, and an entry that a newer one replaced is marked replaced in the batch
131+// that unnames it. So a replaced entry is found from the marks alone, whatever
132+// failed after that batch.
133+type CIConcurrencyMark struct {
134+ Entry CIConcurrencyEntry
135+ // Group is the group as the newest request wrote it (the cancel annotation
136+ // names it).
137+ Group string
138+ State CIConcurrencyState
139+}
140+
141+// CIConcurrencyGroupState is a group as one transaction sees it: its record and
142+// the marks of the entries that belong to it, by CIConcurrencyEntry.Key.
143+type CIConcurrencyGroupState struct {
144+ Record CIConcurrencyRecord
145+ Marks map[string]CIConcurrencyMark
146+}
147+
148+// IsEmpty reports whether the group has no entry and no mark left.
149+func (s CIConcurrencyGroupState) IsEmpty() bool { return s.Record.IsEmpty() && len(s.Marks) == 0 }
@@ -0,0 +1,149 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package domain
4+
5+import (
6+ "strconv"
7+ "unicode"
8+ "unicode/utf16"
9+)
10+
11+// CIConcurrencyState is where a run or a job is in its concurrency group.
12+type CIConcurrencyState string
13+
14+// Concurrency states.
15+const (
16+ // CIConcurrencyIdle: the group is evaluated but the entry has not asked for
17+ // its turn yet.
18+ CIConcurrencyIdle CIConcurrencyState = ""
19+ // CIConcurrencyPending: the group record names this entry as the waiting
20+ // one, behind the running entry.
21+ CIConcurrencyPending CIConcurrencyState = "pending"
22+ // CIConcurrencyHolding: this entry is the running one of its group.
23+ CIConcurrencyHolding CIConcurrencyState = "holding"
24+ // CIConcurrencyReplaced: only in a mark. A newer entry replaced this one:
25+ // it is owed a cancellation and never acquires the group.
26+ CIConcurrencyReplaced CIConcurrencyState = "replaced"
27+)
28+
29+// CIMaxConcurrencyGroup is the longest concurrency group name GitHub accepts
30+// (actions/runner WorkflowTemplateConverter.cs:679-687, "must be less than 400
31+// characters").
32+const CIMaxConcurrencyGroup = 400
33+
34+// CIConcurrency is the evaluated "concurrency:" of a run (workflow level) or of
35+// a job.
36+type CIConcurrency struct {
37+ // Group is the evaluated group as written; "" for none. Groups are compared
38+ // without regard to case (ConcurrencyKey).
39+ Group string
40+ // Cancel is the evaluated cancel-in-progress.
41+ Cancel bool
42+ // State is written to the record of the entry only after the group record
43+ // named it (acquire committed): an entry that waits in a group that does
44+ // not name it any more was replaced, and is cancelled.
45+ State CIConcurrencyState
46+ // Evaluated is set once the expressions were evaluated, whatever the result
47+ // (a run starts unevaluated, and a re-run evaluates again).
48+ Evaluated bool
49+ // FromRun is set on a job that shows the group of its run: the entry is the
50+ // run's, the job holds nothing of its own.
51+ FromRun bool
52+ // Preempted is set when a newer entry of the group cancelled this one.
53+ Preempted bool
54+}
55+
56+// ConcurrencyKey is the identity of a group within a repository. GitHub compares
57+// groups with .NET's OrdinalIgnoreCase, which compares strings upper-cased
58+// character by character with the simple case mapping (OrdinalCasing in
59+// dotnet/runtime, System.Private.CoreLib/src/System/Globalization/
60+// OrdinalCasing.cs; the source was not available here, so this follows its
61+// documented rule). A character that is not ASCII and whose upper case is ASCII
62+// keeps itself: ı (U+0131) and ſ (U+017F) are neither I nor S. So σ, ς and Σ are
63+// one letter, ı and İ are distinct from I, and i equals I. The key is upper
64+// case and is never lowered back.
65+func ConcurrencyKey(group string) string {
66+ out := make([]rune, 0, len(group))
67+ for _, r := range group {
68+ if u := unicode.ToUpper(r); r <= unicode.MaxASCII || u > unicode.MaxASCII {
69+ r = u
70+ }
71+ out = append(out, r)
72+ }
73+ return string(out)
74+}
75+
76+// ConcurrencyGroupLength is the length of a group as C#'s String.Length counts
77+// it: in UTF-16 code units.
78+func ConcurrencyGroupLength(group string) int { return len(utf16.Encode([]rune(group))) }
79+
80+// ConcurrencyCancelMessage is GitHub's annotation for an entry cancelled by a
81+// newer one (docs.github.com, "Control the concurrency of workflows and jobs").
82+func ConcurrencyCancelMessage(group string) string {
83+ return "Canceling since a higher priority waiting request for '" + group + "' exists"
84+}
85+
86+// CIConcurrencyEntry is a run (JobID empty) or a job waiting for or holding a
87+// concurrency group.
88+type CIConcurrencyEntry struct {
89+ RunID int64
90+ Attempt int32
91+ JobID string
92+ // Cancel is the cancel-in-progress of the entry: while it waits, the
93+ // running entry of the group is cancelled.
94+ Cancel bool
95+ // Canceling is set on a running entry that a waiting entry with
96+ // cancel-in-progress asked to stop; it stays named until it has ended, and
97+ // the reconciliation asks it again.
98+ Canceling bool
99+}
100+
101+// Key identifies the entry in the marks of its group.
102+func (e CIConcurrencyEntry) Key() string {
103+ return strconv.FormatInt(e.RunID, 10) + "/" + strconv.FormatInt(int64(e.Attempt), 10) + "/" + e.JobID
104+}
105+
106+// Same reports whether two entries are the same run or job.
107+func (e CIConcurrencyEntry) Same(o CIConcurrencyEntry) bool {
108+ return e.RunID == o.RunID && e.Attempt == o.Attempt && e.JobID == o.JobID
109+}
110+
111+// CIConcurrencyRecord is the state of a group: at most one running entry and
112+// one pending entry (queue: single).
113+type CIConcurrencyRecord struct {
114+ Running *CIConcurrencyEntry
115+ Pending *CIConcurrencyEntry
116+}
117+
118+// IsEmpty reports whether nothing runs or waits in the group.
119+func (r CIConcurrencyRecord) IsEmpty() bool { return r.Running == nil && r.Pending == nil }
120+
121+// Names reports whether the record names the entry as the running or the
122+// pending one.
123+func (r CIConcurrencyRecord) Names(e CIConcurrencyEntry) bool {
124+ return (r.Running != nil && r.Running.Same(e)) || (r.Pending != nil && r.Pending.Same(e))
125+}
126+
127+// CIConcurrencyMark says where an entry stands in its group. The marks of a group
128+// are written in the same batch as its record: an entry is marked pending or
129+// holding if and only if the record names it as the pending or the running
130+// entry, and an entry that a newer one replaced is marked replaced in the batch
131+// that unnames it. So a replaced entry is found from the marks alone, whatever
132+// failed after that batch.
133+type CIConcurrencyMark struct {
134+ Entry CIConcurrencyEntry
135+ // Group is the group as the newest request wrote it (the cancel annotation
136+ // names it).
137+ Group string
138+ State CIConcurrencyState
139+}
140+
141+// CIConcurrencyGroupState is a group as one transaction sees it: its record and
142+// the marks of the entries that belong to it, by CIConcurrencyEntry.Key.
143+type CIConcurrencyGroupState struct {
144+ Record CIConcurrencyRecord
145+ Marks map[string]CIConcurrencyMark
146+}
147+
148+// IsEmpty reports whether the group has no entry and no mark left.
149+func (s CIConcurrencyGroupState) IsEmpty() bool { return s.Record.IsEmpty() && len(s.Marks) == 0 }
A internal/domain/ci_concurrency_test.go +47 -0
@@ -0,0 +1,47 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package domain_test
4+
5+import (
6+ "strings"
7+ "testing"
8+
9+ "github.com/stretchr/testify/assert"
10+
11+ "bnema.dev/ermite/internal/domain"
12+)
13+
14+// Groups compare like .NET's OrdinalIgnoreCase: upper-cased character by
15+// character, a non-ASCII character whose upper case is ASCII keeping itself.
16+func TestConcurrencyKeyFoldsLikeOrdinalIgnoreCase(t *testing.T) {
17+ t.Parallel()
18+ k := domain.ConcurrencyKey
19+ // The three sigmas are one letter.
20+ assert.Equal(t, k("Σ"), k("σ"))
21+ assert.Equal(t, k("σ"), k("ς"))
22+ assert.Equal(t, k("ΟΔΟΣ"), k("οδος"))
23+ // ASCII and ordinary letters.
24+ assert.Equal(t, k("Deploy"), k("DEPLOY"))
25+ assert.Equal(t, k("É"), k("é"))
26+ assert.NotEqual(t, k("a"), k("b"))
27+ // The Turkish i's: i == I, but ı (dotless) and İ (dotted capital) are not I.
28+ assert.Equal(t, k("i"), k("I"))
29+ assert.NotEqual(t, k("ı"), k("I"))
30+ assert.NotEqual(t, k("İ"), k("I"))
31+ assert.NotEqual(t, k("ı"), k("i"))
32+ assert.NotEqual(t, k("İ"), k("i"))
33+ assert.NotEqual(t, k("ı"), k("İ"))
34+ // ſ (long s) is not S.
35+ assert.NotEqual(t, k("ſ"), k("S"))
36+ assert.NotEqual(t, k("ſ"), k("s"))
37+ // One character stays one character.
38+ assert.Len(t, []rune(k("ß")), 1)
39+ assert.NotEqual(t, k("ß"), k("ss"))
40+}
41+
42+func TestConcurrencyGroupLengthCountsUTF16Units(t *testing.T) {
43+ t.Parallel()
44+ assert.Equal(t, 400, domain.ConcurrencyGroupLength(strings.Repeat("é", 400)), "a BMP character is one unit")
45+ assert.Equal(t, 2, domain.ConcurrencyGroupLength("😀"), "a supplementary character is two")
46+ assert.Equal(t, 400, domain.ConcurrencyGroupLength(strings.Repeat("😀", 200)))
47+}
@@ -0,0 +1,47 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package domain_test
4+
5+import (
6+ "strings"
7+ "testing"
8+
9+ "github.com/stretchr/testify/assert"
10+
11+ "bnema.dev/ermite/internal/domain"
12+)
13+
14+// Groups compare like .NET's OrdinalIgnoreCase: upper-cased character by
15+// character, a non-ASCII character whose upper case is ASCII keeping itself.
16+func TestConcurrencyKeyFoldsLikeOrdinalIgnoreCase(t *testing.T) {
17+ t.Parallel()
18+ k := domain.ConcurrencyKey
19+ // The three sigmas are one letter.
20+ assert.Equal(t, k("Σ"), k("σ"))
21+ assert.Equal(t, k("σ"), k("ς"))
22+ assert.Equal(t, k("ΟΔΟΣ"), k("οδος"))
23+ // ASCII and ordinary letters.
24+ assert.Equal(t, k("Deploy"), k("DEPLOY"))
25+ assert.Equal(t, k("É"), k("é"))
26+ assert.NotEqual(t, k("a"), k("b"))
27+ // The Turkish i's: i == I, but ı (dotless) and İ (dotted capital) are not I.
28+ assert.Equal(t, k("i"), k("I"))
29+ assert.NotEqual(t, k("ı"), k("I"))
30+ assert.NotEqual(t, k("İ"), k("I"))
31+ assert.NotEqual(t, k("ı"), k("i"))
32+ assert.NotEqual(t, k("İ"), k("i"))
33+ assert.NotEqual(t, k("ı"), k("İ"))
34+ // ſ (long s) is not S.
35+ assert.NotEqual(t, k("ſ"), k("S"))
36+ assert.NotEqual(t, k("ſ"), k("s"))
37+ // One character stays one character.
38+ assert.Len(t, []rune(k("ß")), 1)
39+ assert.NotEqual(t, k("ß"), k("ss"))
40+}
41+
42+func TestConcurrencyGroupLengthCountsUTF16Units(t *testing.T) {
43+ t.Parallel()
44+ assert.Equal(t, 400, domain.ConcurrencyGroupLength(strings.Repeat("é", 400)), "a BMP character is one unit")
45+ assert.Equal(t, 2, domain.ConcurrencyGroupLength("😀"), "a supplementary character is two")
46+ assert.Equal(t, 400, domain.ConcurrencyGroupLength(strings.Repeat("😀", 200)))
47+}
M internal/ports/ci.go +21 -0
@@ -47,6 +47,14 @@ type CIWrite struct {
4747 ClearAnnotations []CILogRef
4848 }
4949
50+// CIConcurrencyGroup is a stored concurrency group: its repository, its key (the
51+// group, canonical: ConcurrencyKey), its record and its marks.
52+type CIConcurrencyGroup struct {
53+ Repo domain.RepoName
54+ Group string
55+ State domain.CIConcurrencyGroupState
56+}
57+
5058 // CIStepAnnotations are all the annotations of one step of one execution of a
5159 // job, in the order the step produced them.
5260 type CIStepAnnotations struct {
@@ -123,6 +131,19 @@ type CIStore interface {
123131 // ListChecks returns the checks of a commit, at most limit; more is true
124132 // when there are more.
125133 ListChecks(ctx context.Context, repo domain.RepoName, sha string, limit int) (checks []domain.CICheck, more bool, err error)
134+ // UpdateConcurrency reads a concurrency group of a repository (compared
135+ // with domain.ConcurrencyKey) with the marks of its entries, calls fn on a
136+ // copy and stores what fn leaves in it, in one batch: the record, the marks
137+ // that changed and the marks that fn removed. A group left with neither
138+ // record nor marks is removed. The batch carries the version precondition of
139+ // the group whatever fn does, and every change of the marks rewrites the
140+ // record, so no two transactions of a group both succeed on the same state:
141+ // on a conflict fn is called again on the new state and must be repeatable.
142+ // A deleted repository is domain.ErrNotFound.
143+ UpdateConcurrency(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error
144+ // ListConcurrency returns every concurrency group that has an entry, for
145+ // the reconciliation at startup.
146+ ListConcurrency(ctx context.Context) ([]CIConcurrencyGroup, error)
126147 // Sweep removes the runs completed before the time, with their jobs,
127148 // checks and sources (not their numbers), and returns their ids so that
128149 // the logs can go too.
@@ -47,6 +47,14 @@ type CIWrite struct {
47 ClearAnnotations []CILogRef47 ClearAnnotations []CILogRef
48 }48 }
49 49
50+// CIConcurrencyGroup is a stored concurrency group: its repository, its key (the
51+// group, canonical: ConcurrencyKey), its record and its marks.
52+type CIConcurrencyGroup struct {
53+ Repo domain.RepoName
54+ Group string
55+ State domain.CIConcurrencyGroupState
56+}
57+
50 // CIStepAnnotations are all the annotations of one step of one execution of a58 // CIStepAnnotations are all the annotations of one step of one execution of a
51 // job, in the order the step produced them.59 // job, in the order the step produced them.
52 type CIStepAnnotations struct {60 type CIStepAnnotations struct {
@@ -123,6 +131,19 @@ type CIStore interface {
123 // ListChecks returns the checks of a commit, at most limit; more is true131 // ListChecks returns the checks of a commit, at most limit; more is true
124 // when there are more.132 // when there are more.
125 ListChecks(ctx context.Context, repo domain.RepoName, sha string, limit int) (checks []domain.CICheck, more bool, err error)133 ListChecks(ctx context.Context, repo domain.RepoName, sha string, limit int) (checks []domain.CICheck, more bool, err error)
134+ // UpdateConcurrency reads a concurrency group of a repository (compared
135+ // with domain.ConcurrencyKey) with the marks of its entries, calls fn on a
136+ // copy and stores what fn leaves in it, in one batch: the record, the marks
137+ // that changed and the marks that fn removed. A group left with neither
138+ // record nor marks is removed. The batch carries the version precondition of
139+ // the group whatever fn does, and every change of the marks rewrites the
140+ // record, so no two transactions of a group both succeed on the same state:
141+ // on a conflict fn is called again on the new state and must be repeatable.
142+ // A deleted repository is domain.ErrNotFound.
143+ UpdateConcurrency(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error
144+ // ListConcurrency returns every concurrency group that has an entry, for
145+ // the reconciliation at startup.
146+ ListConcurrency(ctx context.Context) ([]CIConcurrencyGroup, error)
126 // Sweep removes the runs completed before the time, with their jobs,147 // Sweep removes the runs completed before the time, with their jobs,
127 // checks and sources (not their numbers), and returns their ids so that148 // checks and sources (not their numbers), and returns their ids so that
128 // the logs can go too.149 // the logs can go too.
M internal/ports/mocks/CIStore.go +131 -0
@@ -993,6 +993,68 @@ func (_c *CIStore_ListChecks_Call) RunAndReturn(run func(ctx context.Context, re
993993 return _c
994994 }
995995
996+// ListConcurrency provides a mock function for the type CIStore
997+func (_mock *CIStore) ListConcurrency(ctx context.Context) ([]ports.CIConcurrencyGroup, error) {
998+ ret := _mock.Called(ctx)
999+
1000+ if len(ret) == 0 {
1001+ panic("no return value specified for ListConcurrency")
1002+ }
1003+
1004+ var r0 []ports.CIConcurrencyGroup
1005+ var r1 error
1006+ if returnFunc, ok := ret.Get(0).(func(context.Context) ([]ports.CIConcurrencyGroup, error)); ok {
1007+ return returnFunc(ctx)
1008+ }
1009+ if returnFunc, ok := ret.Get(0).(func(context.Context) []ports.CIConcurrencyGroup); ok {
1010+ r0 = returnFunc(ctx)
1011+ } else {
1012+ if ret.Get(0) != nil {
1013+ r0 = ret.Get(0).([]ports.CIConcurrencyGroup)
1014+ }
1015+ }
1016+ if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
1017+ r1 = returnFunc(ctx)
1018+ } else {
1019+ r1 = ret.Error(1)
1020+ }
1021+ return r0, r1
1022+}
1023+
1024+// CIStore_ListConcurrency_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ListConcurrency'
1025+type CIStore_ListConcurrency_Call struct {
1026+ *mock.Call
1027+}
1028+
1029+// ListConcurrency is a helper method to define mock.On call
1030+// - ctx context.Context
1031+func (_e *CIStore_Expecter) ListConcurrency(ctx any) *CIStore_ListConcurrency_Call {
1032+ return &CIStore_ListConcurrency_Call{Call: _e.mock.On("ListConcurrency", ctx)}
1033+}
1034+
1035+func (_c *CIStore_ListConcurrency_Call) Run(run func(ctx context.Context)) *CIStore_ListConcurrency_Call {
1036+ _c.Call.Run(func(args mock.Arguments) {
1037+ var arg0 context.Context
1038+ if args[0] != nil {
1039+ arg0 = args[0].(context.Context)
1040+ }
1041+ run(
1042+ arg0,
1043+ )
1044+ })
1045+ return _c
1046+}
1047+
1048+func (_c *CIStore_ListConcurrency_Call) Return(cIConcurrencyGroups []ports.CIConcurrencyGroup, err error) *CIStore_ListConcurrency_Call {
1049+ _c.Call.Return(cIConcurrencyGroups, err)
1050+ return _c
1051+}
1052+
1053+func (_c *CIStore_ListConcurrency_Call) RunAndReturn(run func(ctx context.Context) ([]ports.CIConcurrencyGroup, error)) *CIStore_ListConcurrency_Call {
1054+ _c.Call.Return(run)
1055+ return _c
1056+}
1057+
9961058 // ListJobs provides a mock function for the type CIStore
9971059 func (_mock *CIStore) ListJobs(ctx context.Context, runID int64, runAttempt int32) ([]domain.CIJob, error) {
9981060 ret := _mock.Called(ctx, runID, runAttempt)
@@ -1540,3 +1602,72 @@ func (_c *CIStore_Sweep_Call) RunAndReturn(run func(ctx context.Context, before
15401602 _c.Call.Return(run)
15411603 return _c
15421604 }
1605+
1606+// UpdateConcurrency provides a mock function for the type CIStore
1607+func (_mock *CIStore) UpdateConcurrency(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error {
1608+ ret := _mock.Called(ctx, repo, group, fn)
1609+
1610+ if len(ret) == 0 {
1611+ panic("no return value specified for UpdateConcurrency")
1612+ }
1613+
1614+ var r0 error
1615+ if returnFunc, ok := ret.Get(0).(func(context.Context, domain.RepoName, string, func(*domain.CIConcurrencyGroupState) error) error); ok {
1616+ r0 = returnFunc(ctx, repo, group, fn)
1617+ } else {
1618+ r0 = ret.Error(0)
1619+ }
1620+ return r0
1621+}
1622+
1623+// CIStore_UpdateConcurrency_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateConcurrency'
1624+type CIStore_UpdateConcurrency_Call struct {
1625+ *mock.Call
1626+}
1627+
1628+// UpdateConcurrency is a helper method to define mock.On call
1629+// - ctx context.Context
1630+// - repo domain.RepoName
1631+// - group string
1632+// - fn func(*domain.CIConcurrencyGroupState) error
1633+func (_e *CIStore_Expecter) UpdateConcurrency(ctx any, repo any, group any, fn any) *CIStore_UpdateConcurrency_Call {
1634+ return &CIStore_UpdateConcurrency_Call{Call: _e.mock.On("UpdateConcurrency", ctx, repo, group, fn)}
1635+}
1636+
1637+func (_c *CIStore_UpdateConcurrency_Call) Run(run func(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error)) *CIStore_UpdateConcurrency_Call {
1638+ _c.Call.Run(func(args mock.Arguments) {
1639+ var arg0 context.Context
1640+ if args[0] != nil {
1641+ arg0 = args[0].(context.Context)
1642+ }
1643+ var arg1 domain.RepoName
1644+ if args[1] != nil {
1645+ arg1 = args[1].(domain.RepoName)
1646+ }
1647+ var arg2 string
1648+ if args[2] != nil {
1649+ arg2 = args[2].(string)
1650+ }
1651+ var arg3 func(*domain.CIConcurrencyGroupState) error
1652+ if args[3] != nil {
1653+ arg3 = args[3].(func(*domain.CIConcurrencyGroupState) error)
1654+ }
1655+ run(
1656+ arg0,
1657+ arg1,
1658+ arg2,
1659+ arg3,
1660+ )
1661+ })
1662+ return _c
1663+}
1664+
1665+func (_c *CIStore_UpdateConcurrency_Call) Return(err error) *CIStore_UpdateConcurrency_Call {
1666+ _c.Call.Return(err)
1667+ return _c
1668+}
1669+
1670+func (_c *CIStore_UpdateConcurrency_Call) RunAndReturn(run func(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error) *CIStore_UpdateConcurrency_Call {
1671+ _c.Call.Return(run)
1672+ return _c
1673+}
@@ -993,6 +993,68 @@ func (_c *CIStore_ListChecks_Call) RunAndReturn(run func(ctx context.Context, re
993 return _c993 return _c
994 }994 }
995 995
996+// ListConcurrency provides a mock function for the type CIStore
997+func (_mock *CIStore) ListConcurrency(ctx context.Context) ([]ports.CIConcurrencyGroup, error) {
998+ ret := _mock.Called(ctx)
999+
1000+ if len(ret) == 0 {
1001+ panic("no return value specified for ListConcurrency")
1002+ }
1003+
1004+ var r0 []ports.CIConcurrencyGroup
1005+ var r1 error
1006+ if returnFunc, ok := ret.Get(0).(func(context.Context) ([]ports.CIConcurrencyGroup, error)); ok {
1007+ return returnFunc(ctx)
1008+ }
1009+ if returnFunc, ok := ret.Get(0).(func(context.Context) []ports.CIConcurrencyGroup); ok {
1010+ r0 = returnFunc(ctx)
1011+ } else {
1012+ if ret.Get(0) != nil {
1013+ r0 = ret.Get(0).([]ports.CIConcurrencyGroup)
1014+ }
1015+ }
1016+ if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
1017+ r1 = returnFunc(ctx)
1018+ } else {
1019+ r1 = ret.Error(1)
1020+ }
1021+ return r0, r1
1022+}
1023+
1024+// CIStore_ListConcurrency_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ListConcurrency'
1025+type CIStore_ListConcurrency_Call struct {
1026+ *mock.Call
1027+}
1028+
1029+// ListConcurrency is a helper method to define mock.On call
1030+// - ctx context.Context
1031+func (_e *CIStore_Expecter) ListConcurrency(ctx any) *CIStore_ListConcurrency_Call {
1032+ return &CIStore_ListConcurrency_Call{Call: _e.mock.On("ListConcurrency", ctx)}
1033+}
1034+
1035+func (_c *CIStore_ListConcurrency_Call) Run(run func(ctx context.Context)) *CIStore_ListConcurrency_Call {
1036+ _c.Call.Run(func(args mock.Arguments) {
1037+ var arg0 context.Context
1038+ if args[0] != nil {
1039+ arg0 = args[0].(context.Context)
1040+ }
1041+ run(
1042+ arg0,
1043+ )
1044+ })
1045+ return _c
1046+}
1047+
1048+func (_c *CIStore_ListConcurrency_Call) Return(cIConcurrencyGroups []ports.CIConcurrencyGroup, err error) *CIStore_ListConcurrency_Call {
1049+ _c.Call.Return(cIConcurrencyGroups, err)
1050+ return _c
1051+}
1052+
1053+func (_c *CIStore_ListConcurrency_Call) RunAndReturn(run func(ctx context.Context) ([]ports.CIConcurrencyGroup, error)) *CIStore_ListConcurrency_Call {
1054+ _c.Call.Return(run)
1055+ return _c
1056+}
1057+
996 // ListJobs provides a mock function for the type CIStore1058 // ListJobs provides a mock function for the type CIStore
997 func (_mock *CIStore) ListJobs(ctx context.Context, runID int64, runAttempt int32) ([]domain.CIJob, error) {1059 func (_mock *CIStore) ListJobs(ctx context.Context, runID int64, runAttempt int32) ([]domain.CIJob, error) {
998 ret := _mock.Called(ctx, runID, runAttempt)1060 ret := _mock.Called(ctx, runID, runAttempt)
@@ -1540,3 +1602,72 @@ func (_c *CIStore_Sweep_Call) RunAndReturn(run func(ctx context.Context, before
1540 _c.Call.Return(run)1602 _c.Call.Return(run)
1541 return _c1603 return _c
1542 }1604 }
1605+
1606+// UpdateConcurrency provides a mock function for the type CIStore
1607+func (_mock *CIStore) UpdateConcurrency(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error {
1608+ ret := _mock.Called(ctx, repo, group, fn)
1609+
1610+ if len(ret) == 0 {
1611+ panic("no return value specified for UpdateConcurrency")
1612+ }
1613+
1614+ var r0 error
1615+ if returnFunc, ok := ret.Get(0).(func(context.Context, domain.RepoName, string, func(*domain.CIConcurrencyGroupState) error) error); ok {
1616+ r0 = returnFunc(ctx, repo, group, fn)
1617+ } else {
1618+ r0 = ret.Error(0)
1619+ }
1620+ return r0
1621+}
1622+
1623+// CIStore_UpdateConcurrency_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateConcurrency'
1624+type CIStore_UpdateConcurrency_Call struct {
1625+ *mock.Call
1626+}
1627+
1628+// UpdateConcurrency is a helper method to define mock.On call
1629+// - ctx context.Context
1630+// - repo domain.RepoName
1631+// - group string
1632+// - fn func(*domain.CIConcurrencyGroupState) error
1633+func (_e *CIStore_Expecter) UpdateConcurrency(ctx any, repo any, group any, fn any) *CIStore_UpdateConcurrency_Call {
1634+ return &CIStore_UpdateConcurrency_Call{Call: _e.mock.On("UpdateConcurrency", ctx, repo, group, fn)}
1635+}
1636+
1637+func (_c *CIStore_UpdateConcurrency_Call) Run(run func(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error)) *CIStore_UpdateConcurrency_Call {
1638+ _c.Call.Run(func(args mock.Arguments) {
1639+ var arg0 context.Context
1640+ if args[0] != nil {
1641+ arg0 = args[0].(context.Context)
1642+ }
1643+ var arg1 domain.RepoName
1644+ if args[1] != nil {
1645+ arg1 = args[1].(domain.RepoName)
1646+ }
1647+ var arg2 string
1648+ if args[2] != nil {
1649+ arg2 = args[2].(string)
1650+ }
1651+ var arg3 func(*domain.CIConcurrencyGroupState) error
1652+ if args[3] != nil {
1653+ arg3 = args[3].(func(*domain.CIConcurrencyGroupState) error)
1654+ }
1655+ run(
1656+ arg0,
1657+ arg1,
1658+ arg2,
1659+ arg3,
1660+ )
1661+ })
1662+ return _c
1663+}
1664+
1665+func (_c *CIStore_UpdateConcurrency_Call) Return(err error) *CIStore_UpdateConcurrency_Call {
1666+ _c.Call.Return(err)
1667+ return _c
1668+}
1669+
1670+func (_c *CIStore_UpdateConcurrency_Call) RunAndReturn(run func(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) error) *CIStore_UpdateConcurrency_Call {
1671+ _c.Call.Return(run)
1672+ return _c
1673+}
M internal/usecase/ci.go +16 -0
@@ -34,6 +34,9 @@ const (
3434 DefaultCIRemoteConcurrency = 4
3535 // DefaultCIResolveTimeout bounds one ResolveActions call.
3636 DefaultCIResolveTimeout = 60 * time.Second
37+ // DefaultCIConcurrencyPruneBatch is the stale marks of a concurrency group a
38+ // transaction removes at most.
39+ DefaultCIConcurrencyPruneBatch = 256
3740 // MaxCIChecks is the most checks ListChecks returns.
3841 MaxCIChecks = 1000
3942 // MaxCILogStreams and MaxCILogStreamsPerClient bound the concurrent log
@@ -98,6 +101,15 @@ type CIConfig struct {
98101 TimeoutSlack time.Duration
99102 QueueTimeout time.Duration
100103 TriggerQueue int
104+ // ConcurrencyRetries bounds the retries of a delivery of the concurrency
105+ // groups that failed (a cancel, the planning of a promoted entry): 0 is the
106+ // default (8), a negative number none, which the tests use to leave the
107+ // recovery to the reconciliation.
108+ ConcurrencyRetries int
109+ // ConcurrencyPruneBatch bounds the stale marks of a concurrency group one
110+ // transaction removes (default 256, far below the size limit of a kv batch);
111+ // the others go with the next decisions and the reconciliation.
112+ ConcurrencyPruneBatch int
101113 }
102114
103115 func (c CIConfig) withDefaults() CIConfig {
@@ -122,6 +134,9 @@ func (c CIConfig) withDefaults() CIConfig {
122134 if c.TriggerQueue <= 0 {
123135 c.TriggerQueue = DefaultCITriggerQueue
124136 }
137+ if c.ConcurrencyPruneBatch <= 0 {
138+ c.ConcurrencyPruneBatch = DefaultCIConcurrencyPruneBatch
139+ }
125140 return c
126141 }
127142
@@ -405,6 +420,7 @@ func (rs *runState) flush(ctx context.Context) error {
405420 rs.run.Version = v
406421 }
407422 changedRun := rs.runDirty
423+ rs.releaseEnded(ctx, changedJobs, changedRun && rs.run.Status == domain.CICompleted)
408424 rs.dirty, rs.checks, rs.runDirty = map[string]bool{}, map[string]bool{}, false
409425 rs.tokens, rs.revoke, rs.resolutions = nil, nil, nil
410426 rs.annotations, rs.clear = nil, nil
@@ -34,6 +34,9 @@ const (
34 DefaultCIRemoteConcurrency = 434 DefaultCIRemoteConcurrency = 4
35 // DefaultCIResolveTimeout bounds one ResolveActions call.35 // DefaultCIResolveTimeout bounds one ResolveActions call.
36 DefaultCIResolveTimeout = 60 * time.Second36 DefaultCIResolveTimeout = 60 * time.Second
37+ // DefaultCIConcurrencyPruneBatch is the stale marks of a concurrency group a
38+ // transaction removes at most.
39+ DefaultCIConcurrencyPruneBatch = 256
37 // MaxCIChecks is the most checks ListChecks returns.40 // MaxCIChecks is the most checks ListChecks returns.
38 MaxCIChecks = 100041 MaxCIChecks = 1000
39 // MaxCILogStreams and MaxCILogStreamsPerClient bound the concurrent log42 // MaxCILogStreams and MaxCILogStreamsPerClient bound the concurrent log
@@ -98,6 +101,15 @@ type CIConfig struct {
98 TimeoutSlack time.Duration101 TimeoutSlack time.Duration
99 QueueTimeout time.Duration102 QueueTimeout time.Duration
100 TriggerQueue int103 TriggerQueue int
104+ // ConcurrencyRetries bounds the retries of a delivery of the concurrency
105+ // groups that failed (a cancel, the planning of a promoted entry): 0 is the
106+ // default (8), a negative number none, which the tests use to leave the
107+ // recovery to the reconciliation.
108+ ConcurrencyRetries int
109+ // ConcurrencyPruneBatch bounds the stale marks of a concurrency group one
110+ // transaction removes (default 256, far below the size limit of a kv batch);
111+ // the others go with the next decisions and the reconciliation.
112+ ConcurrencyPruneBatch int
101 }113 }
102 114
103 func (c CIConfig) withDefaults() CIConfig {115 func (c CIConfig) withDefaults() CIConfig {
@@ -122,6 +134,9 @@ func (c CIConfig) withDefaults() CIConfig {
122 if c.TriggerQueue <= 0 {134 if c.TriggerQueue <= 0 {
123 c.TriggerQueue = DefaultCITriggerQueue135 c.TriggerQueue = DefaultCITriggerQueue
124 }136 }
137+ if c.ConcurrencyPruneBatch <= 0 {
138+ c.ConcurrencyPruneBatch = DefaultCIConcurrencyPruneBatch
139+ }
125 return c140 return c
126 }141 }
127 142
@@ -405,6 +420,7 @@ func (rs *runState) flush(ctx context.Context) error {
405 rs.run.Version = v420 rs.run.Version = v
406 }421 }
407 changedRun := rs.runDirty422 changedRun := rs.runDirty
423+ rs.releaseEnded(ctx, changedJobs, changedRun && rs.run.Status == domain.CICompleted)
408 rs.dirty, rs.checks, rs.runDirty = map[string]bool{}, map[string]bool{}, false424 rs.dirty, rs.checks, rs.runDirty = map[string]bool{}, map[string]bool{}, false
409 rs.tokens, rs.revoke, rs.resolutions = nil, nil, nil425 rs.tokens, rs.revoke, rs.resolutions = nil, nil, nil
410 rs.annotations, rs.clear = nil, nil426 rs.annotations, rs.clear = nil, nil
A internal/usecase/ci_concurrency.go +635 -0
@@ -0,0 +1,635 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package usecase
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "maps"
10+ "slices"
11+ "time"
12+
13+ "bnema.dev/ermite-actions/expr"
14+ "bnema.dev/ermite-actions/template"
15+ "bnema.dev/ermite-actions/workflow"
16+ "bnema.dev/ermite/internal/domain"
17+ "bnema.dev/ermite/internal/ports"
18+)
19+
20+// Concurrency groups (CI-3 phase E). A group is per repository and compared
21+// without regard to case; a run (workflow level) or a job (job level) is an
22+// entry. With queue: single, at most one entry runs and one waits: a newer
23+// waiting entry cancels the older one, and cancel-in-progress also cancels the
24+// running one. An entry holds its group from the moment it may run until the
25+// run or job has ended.
26+//
27+// The record of a group lives under "concur/<repo>/<key(group)>" and is
28+// changed with a version-checked write (Store.UpdateConcurrency). Cancelling
29+// another entry needs the lock of another run, which the caller of a change
30+// does not hold in order (two runs would wait for each other): those effects
31+// are queued with later and run after the lock is released.
32+
33+// errEmptyGroup and errLongGroup are GitHub's errors for a group that
34+// evaluates to nothing or too much (actions/runner,
35+// WorkflowTemplateConverter.cs ConvertToConcurrency, lines 679-687).
36+var (
37+ errEmptyGroup = errors.New("concurrency group name cannot be empty")
38+ // errQueueMaxCancel is the message of languageservices
39+ // (languageservice/src/validate.ts:743).
40+ errQueueMaxCancel = errors.New("'queue: max' cannot be combined with 'cancel-in-progress: true'.") //nolint:staticcheck // ST1005: the text of languageservices, kept as is
41+ errLongGroup = fmt.Errorf("concurrency group name must be less than %d characters", domain.CIMaxConcurrencyGroup)
42+)
43+
44+// evalConcurrency evaluates the "concurrency:" token of a workflow or a job:
45+// a string (or an expression) is the group; a mapping has group,
46+// cancel-in-progress (an expression is allowed) and queue. An empty group is
47+// an error, as in GitHub's runner; queue: max is not supported yet.
48+func evalConcurrency(ctx context.Context, tok template.Token, env template.Env) (domain.CIConcurrency, error) {
49+ var out domain.CIConcurrency
50+ if tok == nil {
51+ return out, nil
52+ }
53+ groupTok := tok
54+ if m, ok := tok.(*template.Mapping); ok {
55+ if q := m.Find("queue"); q != nil {
56+ queue, err := template.EvaluateString(ctx, q, env)
57+ if err != nil {
58+ return out, err
59+ }
60+ if queue == "max" {
61+ // languageservices, languageservice/src/validate.ts:702-743: the
62+ // literal pair is invalid (expressions are not checked there).
63+ if b, ok := m.Find("cancel-in-progress").(*template.Boolean); ok && b.Value {
64+ return out, errQueueMaxCancel
65+ }
66+ return out, errors.New("not supported yet: queue: max")
67+ }
68+ }
69+ if groupTok = m.Find("group"); groupTok == nil {
70+ return out, errEmptyGroup
71+ }
72+ if c := m.Find("cancel-in-progress"); c != nil {
73+ cancel, err := template.EvaluateBool(ctx, c, env)
74+ if err != nil {
75+ return out, err
76+ }
77+ out.Cancel = cancel
78+ }
79+ }
80+ group, err := template.EvaluateString(ctx, groupTok, env)
81+ switch {
82+ case err != nil:
83+ return out, err
84+ case group == "":
85+ return out, errEmptyGroup
86+ case domain.ConcurrencyGroupLength(group) > domain.CIMaxConcurrencyGroup:
87+ return out, errLongGroup
88+ }
89+ out.Group = group
90+ return out, nil
91+}
92+
93+// later runs fn in the background, after the lock of the caller's run is
94+// released. Close waits for it.
95+func (c *CI) later(what string, fn func(ctx context.Context) error) {
96+ c.mu.Lock()
97+ if c.closed {
98+ c.mu.Unlock()
99+ return
100+ }
101+ c.wg.Add(1)
102+ c.mu.Unlock()
103+ go func() {
104+ defer c.wg.Done()
105+ c.guard(what, func() {
106+ if err := fn(c.ctx); err != nil && c.ctx.Err() == nil && !errors.Is(err, domain.ErrNotFound) {
107+ ctxLog(c.ctx).Error().Err(err).Str("worker", what).Msg("ci concurrency")
108+ }
109+ })
110+ }()
111+}
112+
113+// entryAlive reports whether the run or job of an entry can still hold or want
114+// its group: it exists, and it has not ended (a run: it is on the attempt of the
115+// entry and has not completed). Run ids are never reused, so a missing run or
116+// job is gone for good. A read that fails says alive: nothing is dropped on a
117+// doubt.
118+func (c *CI) entryAlive(ctx context.Context, repo domain.RepoName, e domain.CIConcurrencyEntry) bool {
119+ if e.JobID == "" {
120+ run, err := c.cfg.Store.GetRun(ctx, repo, e.RunID)
121+ if err != nil {
122+ return !errors.Is(err, domain.ErrNotFound)
123+ }
124+ return run.Attempt == e.Attempt && run.Status != domain.CICompleted
125+ }
126+ j, err := c.cfg.Store.GetJob(ctx, e.RunID, e.Attempt, e.JobID)
127+ switch {
128+ case errors.Is(err, domain.ErrNotFound):
129+ // A matrix leg that took its group is written with the flush that
130+ // follows: without its record yet, it lives as long as its run's attempt.
131+ return c.entryAlive(ctx, repo, domain.CIConcurrencyEntry{RunID: e.RunID, Attempt: e.Attempt})
132+ case err != nil:
133+ return true
134+ }
135+ return !j.Ended()
136+}
137+
138+// mark sets the mark of an entry.
139+func mark(st *domain.CIConcurrencyGroupState, e domain.CIConcurrencyEntry, group string, state domain.CIConcurrencyState) {
140+ if st.Marks == nil {
141+ st.Marks = map[string]domain.CIConcurrencyMark{}
142+ }
143+ st.Marks[e.Key()] = domain.CIConcurrencyMark{Entry: e, Group: group, State: state}
144+}
145+
146+// promote gives the running place to the waiting entry when it is free.
147+func promote(st *domain.CIConcurrencyGroupState, group string) {
148+ if st.Record.Running == nil && st.Record.Pending != nil {
149+ st.Record.Running, st.Record.Pending = st.Record.Pending, nil
150+ m := st.Marks[st.Record.Running.Key()]
151+ mark(st, *st.Record.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
152+ }
153+}
154+
155+func firstNonEmpty(a, b string) string {
156+ if a != "" {
157+ return a
158+ }
159+ return b
160+}
161+
162+// prune brings a group back to its invariant, in the transaction that is about to
163+// decide on it: the entries whose run or job is gone or over leave the record
164+// and lose their marks (a crash between the write of a job and the release of
165+// its group, a run that was deleted); the waiting entry takes a free running
166+// place; an entry named by the record has its mark; and an entry marked as
167+// pending or holding that the record does not name any more is marked replaced
168+// (it should not happen, the batch writes both, and it is cancelled if it does).
169+func (c *CI) prune(ctx context.Context, repo domain.RepoName, group string, st *domain.CIConcurrencyGroupState) (more bool) {
170+ rec := &st.Record
171+ if rec.Running != nil && !c.entryAlive(ctx, repo, *rec.Running) {
172+ delete(st.Marks, rec.Running.Key())
173+ rec.Running = nil
174+ }
175+ if rec.Pending != nil && !c.entryAlive(ctx, repo, *rec.Pending) {
176+ delete(st.Marks, rec.Pending.Key())
177+ rec.Pending = nil
178+ }
179+ promote(st, group)
180+ if rec.Running != nil {
181+ if m, ok := st.Marks[rec.Running.Key()]; !ok || m.State != domain.CIConcurrencyHolding {
182+ mark(st, *rec.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
183+ }
184+ }
185+ if rec.Pending != nil {
186+ if m, ok := st.Marks[rec.Pending.Key()]; !ok || m.State != domain.CIConcurrencyPending {
187+ mark(st, *rec.Pending, firstNonEmpty(m.Group, group), domain.CIConcurrencyPending)
188+ }
189+ }
190+ // The other marks are pruned a bounded number per transaction, in key order:
191+ // a batch has a size limit, and a group with a long backlog of stale marks
192+ // must not wedge on one transaction that deletes them all. What is left is
193+ // reported, and the next decision or the reconciliation takes the next ones.
194+ left := c.cfg.ConcurrencyPruneBatch
195+ for _, k := range slices.Sorted(maps.Keys(st.Marks)) {
196+ m := st.Marks[k]
197+ var prune, replace bool
198+ switch {
199+ case !c.entryAlive(ctx, repo, m.Entry):
200+ prune = true
201+ case m.State != domain.CIConcurrencyReplaced && !rec.Names(m.Entry):
202+ replace = true
203+ default:
204+ continue
205+ }
206+ if left <= 0 {
207+ return true
208+ }
209+ left--
210+ if prune {
211+ delete(st.Marks, k)
212+ } else if replace {
213+ m.State = domain.CIConcurrencyReplaced
214+ st.Marks[k] = m
215+ }
216+ }
217+ return false
218+}
219+
220+// updateGroup is Store.UpdateConcurrency over the pruned group. It returns the
221+// entry that holds the group after the change and did not before (the waiting
222+// entry that was given the turn), or nil: its run is to be planned again.
223+func (c *CI) updateGroup(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) (*domain.CIConcurrencyEntry, error) {
224+ promoted, _, err := c.updateGroupMore(ctx, repo, group, fn)
225+ return promoted, err
226+}
227+
228+// updateGroupMore is updateGroup that also says whether the pruning stopped at
229+// its bound, with stale marks left for another transaction.
230+func (c *CI) updateGroupMore(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) (promoted *domain.CIConcurrencyEntry, more bool, err error) {
231+ err = c.cfg.Store.UpdateConcurrency(ctx, repo, group, func(st *domain.CIConcurrencyGroupState) error {
232+ promoted = nil
233+ var was *domain.CIConcurrencyEntry
234+ if r := st.Record.Running; r != nil {
235+ e := *r
236+ was = &e
237+ }
238+ more = c.prune(ctx, repo, group, st)
239+ if err := fn(st); err != nil {
240+ return err
241+ }
242+ if r := st.Record.Running; r != nil && (was == nil || !was.Same(*r)) {
243+ e := *r
244+ promoted = &e
245+ }
246+ return nil
247+ })
248+ return promoted, more, err
249+}
250+
251+// concurrencyRetries bounds the retries of a delivery that failed; the delay
252+// doubles from the watchdog tick up to maxConcurrencyBackoff.
253+const (
254+ defaultConcurrencyRetries = 8
255+ maxConcurrencyBackoff = 30 * time.Second
256+)
257+
258+// retry calls fn until it succeeds, at most concurrencyRetries more times, with
259+// a doubling delay that ends with ctx. A domain.ErrNotFound is not retried.
260+func (c *CI) retry(ctx context.Context, fn func() error) error {
261+ delay := c.cfg.Tick
262+ err := fn()
263+ retries := c.cfg.ConcurrencyRetries
264+ switch {
265+ case retries == 0:
266+ retries = defaultConcurrencyRetries
267+ case retries < 0:
268+ retries = 0
269+ }
270+ for range retries {
271+ if err == nil || errors.Is(err, domain.ErrNotFound) {
272+ return err
273+ }
274+ select {
275+ case <-ctx.Done():
276+ return errors.Join(err, ctx.Err())
277+ case <-time.After(delay):
278+ }
279+ delay = min(delay*2, maxConcurrencyBackoff)
280+ err = fn()
281+ }
282+ return err
283+}
284+
285+// wake plans again the run of an entry that was given its group, outside any
286+// run lock, and again if that failed. A run that is gone passes the group on.
287+// What the retries do not reach, the reconciliation (startup, and every
288+// minute) does: it advances every holder whose own record still says pending.
289+func (c *CI) wake(repo domain.RepoName, group string, e domain.CIConcurrencyEntry) {
290+ c.later("concurrency promote", func(ctx context.Context) error {
291+ return c.retry(ctx, func() error {
292+ err := c.advanceRun(ctx, e.RunID)
293+ if errors.Is(err, domain.ErrNotFound) {
294+ return c.releaseEntry(ctx, repo, group, e)
295+ }
296+ return err
297+ })
298+ })
299+}
300+
301+// cancelLater cancels an entry that a newer one replaced, outside any run lock,
302+// and again if that failed. The decision that replaced it is in the group (the
303+// entry is marked replaced in the batch that unnamed it, or stays named with
304+// Canceling set): the reconciliation finds it from there if this cancellation is
305+// lost (crash, retries used up).
306+func (c *CI) cancelLater(group string, e domain.CIConcurrencyEntry) {
307+ c.later("concurrency cancel", func(ctx context.Context) error {
308+ return c.retry(ctx, func() error { return c.cancelEntry(ctx, e, group) })
309+ })
310+}
311+
312+// entryNeeds reports whether the entry named by its group still has to be told
313+// where it stands: its own record does not say want yet (a promotion whose
314+// planning failed or was lost, or a crash between the write of the group and the
315+// write of the entry). An entry that is gone or over is the pruning's business.
316+func (c *CI) entryNeeds(ctx context.Context, repo domain.RepoName, e domain.CIConcurrencyEntry, want domain.CIConcurrencyState) bool {
317+ if e.JobID == "" {
318+ run, err := c.cfg.Store.GetRun(ctx, repo, e.RunID)
319+ return err == nil && run.Status != domain.CICompleted && run.Attempt == e.Attempt && run.Concurrency.State != want
320+ }
321+ j, err := c.cfg.Store.GetJob(ctx, e.RunID, e.Attempt, e.JobID)
322+ if errors.Is(err, domain.ErrNotFound) {
323+ // A leg the group names whose record is not written: the run has to be
324+ // planned, which expands its matrix, while its attempt is open.
325+ run, err := c.cfg.Store.GetRun(ctx, repo, e.RunID)
326+ return err == nil && run.Status != domain.CICompleted && run.Attempt == e.Attempt
327+ }
328+ return err == nil && !j.Ended() && j.Phase == domain.CIPhaseWaiting && j.Concurrency.State != want
329+}
330+
331+// ReconcileConcurrency repairs the groups. It runs at startup and every minute.
332+// For each stored group it prunes the entries that are gone or over, asks again
333+// the running entry a cancel-in-progress entry replaced, cancels every entry
334+// marked replaced (their cancellation is owed), and plans the entries the record
335+// names whose own record does not know it yet.
336+//
337+// Cost, per pass: a scan of the stored groups and their marks, and for each
338+// group a version-checked transaction (a write only if something is repaired;
339+// at most CIConfig.ConcurrencyPruneBatch stale marks go in one, the group is
340+// taken again until none is left) plus a read of the run or job of each entry
341+// the record or the marks name. It grows with the groups in use, not with the
342+// open runs.
343+func (c *CI) ReconcileConcurrency(ctx context.Context) {
344+ log := ctxLog(ctx)
345+ groups, err := c.cfg.Store.ListConcurrency(ctx)
346+ if err != nil {
347+ log.Error().Err(err).Msg("ci: list the concurrency groups")
348+ return
349+ }
350+ for _, g := range groups {
351+ var st domain.CIConcurrencyGroupState
352+ var err error
353+ // One transaction prunes a bounded number of marks: go on until none is
354+ // left to prune (each pass removes some, so this ends).
355+ for more := true; more && err == nil && ctx.Err() == nil; {
356+ _, more, err = c.updateGroupMore(ctx, g.Repo, g.Group, func(cur *domain.CIConcurrencyGroupState) error {
357+ st = domain.CIConcurrencyGroupState{Marks: maps.Clone(cur.Marks)}
358+ if cur.Record.Running != nil {
359+ r := *cur.Record.Running
360+ st.Record.Running = &r
361+ }
362+ if cur.Record.Pending != nil {
363+ p := *cur.Record.Pending
364+ st.Record.Pending = &p
365+ }
366+ return nil
367+ })
368+ }
369+ if err != nil {
370+ if !errors.Is(err, domain.ErrNotFound) && ctx.Err() == nil {
371+ log.Error().Err(err).Str("group", g.Group).Msg("ci: repair a concurrency group")
372+ }
373+ continue
374+ }
375+ if r := st.Record.Running; r != nil {
376+ if r.Canceling {
377+ c.cancelLater(st.Marks[r.Key()].Group, *r)
378+ }
379+ if c.entryNeeds(ctx, g.Repo, *r, domain.CIConcurrencyHolding) {
380+ c.wake(g.Repo, g.Group, *r)
381+ }
382+ }
383+ if p := st.Record.Pending; p != nil && c.entryNeeds(ctx, g.Repo, *p, domain.CIConcurrencyPending) {
384+ c.wake(g.Repo, g.Group, *p)
385+ }
386+ for _, m := range st.Marks {
387+ if m.State == domain.CIConcurrencyReplaced {
388+ c.cancelLater(m.Group, m.Entry)
389+ }
390+ }
391+ }
392+}
393+
394+// acquire asks for the turn of an entry in a group and returns where it stands:
395+// holding, pending behind the running entry, or replaced. The decision and the
396+// marks of every entry it touches are one batch (Store.UpdateConcurrency), so an
397+// entry is marked pending or holding if and only if the record names it.
398+//
399+// An entry marked replaced never acquires, whatever the state of the group
400+// (empty, or its waiting place free): it is owed a cancellation. Otherwise a
401+// pending entry stays so; a newer one replaces the waiting entry, which is
402+// marked replaced and cancelled, and with cancel-in-progress the running entry
403+// is cancelled too (it stays named, with Canceling set, until it has ended).
404+// The call is repeatable: an entry that holds or waits is told so again.
405+func (rs *runState) acquire(ctx context.Context, group string, e domain.CIConcurrencyEntry) (domain.CIConcurrencyState, error) {
406+ var state domain.CIConcurrencyState
407+ var victims []domain.CIConcurrencyEntry
408+ promoted, err := rs.c.updateGroup(ctx, rs.run.Repo, group, func(st *domain.CIConcurrencyGroupState) error {
409+ victims = nil
410+ rec := &st.Record
411+ if m, ok := st.Marks[e.Key()]; ok && m.State == domain.CIConcurrencyReplaced {
412+ state = domain.CIConcurrencyReplaced
413+ victims = append(victims, e) // its cancellation may have been lost
414+ return nil
415+ }
416+ switch {
417+ case rec.Running == nil:
418+ me := e
419+ rec.Running = &me
420+ if rec.Pending != nil && rec.Pending.Same(e) {
421+ rec.Pending = nil
422+ }
423+ mark(st, e, group, domain.CIConcurrencyHolding)
424+ state = domain.CIConcurrencyHolding
425+ case rec.Running.Same(e):
426+ state = domain.CIConcurrencyHolding
427+ case rec.Pending != nil && rec.Pending.Same(e):
428+ state = domain.CIConcurrencyPending
429+ default:
430+ state = domain.CIConcurrencyPending
431+ if rec.Pending != nil {
432+ old := *rec.Pending
433+ mark(st, old, group, domain.CIConcurrencyReplaced)
434+ victims = append(victims, old)
435+ }
436+ me := e
437+ rec.Pending = &me
438+ mark(st, e, group, domain.CIConcurrencyPending)
439+ if e.Cancel && !rec.Running.Canceling {
440+ run := *rec.Running
441+ run.Canceling = true
442+ rec.Running = &run
443+ victims = append(victims, run)
444+ }
445+ }
446+ return nil
447+ })
448+ if err != nil {
449+ return "", err
450+ }
451+ if promoted != nil && !promoted.Same(e) {
452+ rs.c.wake(rs.run.Repo, group, *promoted)
453+ }
454+ for _, v := range victims {
455+ rs.c.cancelLater(group, v)
456+ }
457+ if state == domain.CIConcurrencyReplaced {
458+ state = domain.CIConcurrencyPending // it shows as waiting until its cancellation lands
459+ }
460+ return state, nil
461+}
462+
463+// releaseEntry takes an entry out of its group, because its run or job has
464+// ended, and gives the turn to the waiting entry, which is planned again. The
465+// mark of the entry goes in the same batch.
466+func (c *CI) releaseEntry(ctx context.Context, repo domain.RepoName, group string, e domain.CIConcurrencyEntry) error {
467+ promoted, err := c.updateGroup(ctx, repo, group, func(st *domain.CIConcurrencyGroupState) error {
468+ switch {
469+ case st.Record.Running != nil && st.Record.Running.Same(e):
470+ st.Record.Running = nil
471+ promote(st, group)
472+ case st.Record.Pending != nil && st.Record.Pending.Same(e):
473+ st.Record.Pending = nil
474+ }
475+ delete(st.Marks, e.Key())
476+ return nil
477+ })
478+ if err != nil || promoted == nil {
479+ return err
480+ }
481+ c.wake(repo, group, *promoted)
482+ return nil
483+}
484+
485+// cancelEntry cancels a run or a job that a newer entry of the group
486+// replaced: it ends cancelled with GitHub's annotation, and a job a runner has
487+// gets the cancel request the owner's cancel sends.
488+func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group string) error {
489+ return c.withRun(ctx, e.RunID, e.Attempt, func(rs *runState) error {
490+ if group == "" { // no name in the mark: use the entry's own group
491+ group = rs.run.Concurrency.Group
492+ if j := rs.find(e.JobID); e.JobID != "" && j != nil {
493+ group = j.Concurrency.Group
494+ }
495+ }
496+ msg := domain.ConcurrencyCancelMessage(group)
497+ if e.JobID == "" {
498+ if !rs.latest() || rs.run.Status == domain.CICompleted || rs.run.CancelRequested {
499+ return nil
500+ }
501+ rs.run.CancelRequested, rs.run.CancelRequestedAt, rs.runDirty = true, c.now(), true
502+ rs.run.Concurrency.Preempted = true
503+ for i := range rs.jobs {
504+ if !rs.jobs[i].Ended() {
505+ rs.cancelForGroup(&rs.jobs[i], msg)
506+ }
507+ }
508+ return rs.flush(ctx)
509+ }
510+ j := rs.find(e.JobID)
511+ if j == nil || !j.Planned {
512+ // The group names a leg whose record the run has not written yet (the
513+ // flush that followed the planning was lost): the record found under
514+ // the id of leg 0 is the placeholder of the whole job, not the leg.
515+ // Plan the job first, so that every leg exists, and only then act on
516+ // the one that was targeted.
517+ if err := rs.advance(ctx); err != nil {
518+ return err
519+ }
520+ if j = rs.find(e.JobID); j == nil || !j.Planned {
521+ return rs.flush(ctx) // not plannable (yet): nothing to cancel, keep what advance did
522+ }
523+ }
524+ if j.Ended() || j.Concurrency.Preempted {
525+ return rs.flush(ctx) // over, or cancelled by an earlier delivery of the same item
526+ }
527+ rs.cancelForGroup(j, msg)
528+ if err := rs.advance(ctx); err != nil {
529+ return err
530+ }
531+ return rs.flush(ctx)
532+ })
533+}
534+
535+// cancelForGroup ends a job a newer entry of its group replaced, or asks its
536+// runner to stop. The annotation is GitHub's.
537+func (rs *runState) cancelForGroup(j *domain.CIJob, msg string) {
538+ j.Concurrency.Preempted = true
539+ rs.annotations = append(rs.annotations, ports.CIStepAnnotations{
540+ Repo: j.Repo, Exec: logRef(*j), Step: 1,
541+ Items: []domain.CIAnnotation{{Level: domain.AnnotationWarning, Message: msg, Step: 1}},
542+ })
543+ if j.Active() {
544+ rs.requestCancel(j, domain.CICancelUser, msg)
545+ rs.touch(j)
546+ return
547+ }
548+ rs.end(j, domain.CICancelled, msg)
549+}
550+
551+// acquireRun evaluates the workflow-level concurrency of the attempt once and
552+// asks for its turn. It returns whether the run may go on: a pending run
553+// plans nothing, and its jobs show the pending status.
554+func (rs *runState) acquireRun(ctx context.Context, wf *workflow.Workflow) (bool, error) {
555+ run := &rs.run
556+ if !slices.ContainsFunc(rs.jobs, func(j domain.CIJob) bool { return !j.Ended() }) {
557+ return true, nil // nothing left to run: the group is freed at the flush
558+ }
559+ if !run.Concurrency.Evaluated {
560+ conc, err := rs.evalRunConcurrency(ctx, wf)
561+ if err != nil {
562+ for i := range rs.jobs {
563+ if !rs.jobs[i].Ended() && !rs.jobs[i].Active() {
564+ rs.end(&rs.jobs[i], domain.CIFailure, err.Error())
565+ }
566+ }
567+ run.Concurrency, rs.runDirty = domain.CIConcurrency{Evaluated: true}, true
568+ return false, nil
569+ }
570+ conc.Evaluated = true
571+ run.Concurrency, rs.runDirty = conc, true
572+ }
573+ if run.Concurrency.Group == "" {
574+ return true, nil
575+ }
576+ st, err := rs.acquire(ctx, run.Concurrency.Group, domain.CIConcurrencyEntry{RunID: run.ID, Attempt: run.Attempt, Cancel: run.Concurrency.Cancel})
577+ if err != nil {
578+ return false, err
579+ }
580+ if st != run.Concurrency.State {
581+ run.Concurrency.State, rs.runDirty = st, true
582+ }
583+ want := run.Concurrency
584+ want.FromRun, want.Preempted = true, false
585+ for i := range rs.jobs {
586+ j := &rs.jobs[i]
587+ if j.Ended() || (j.Concurrency.Group != "" && !j.Concurrency.FromRun) {
588+ continue // a job with a group of its own shows that one
589+ }
590+ if j.Concurrency != want {
591+ j.Concurrency = want
592+ rs.touch(j)
593+ }
594+ }
595+ return st == domain.CIConcurrencyHolding, nil
596+}
597+
598+func (rs *runState) evalRunConcurrency(ctx context.Context, wf *workflow.Workflow) (domain.CIConcurrency, error) {
599+ if wf.Concurrency == nil {
600+ return domain.CIConcurrency{}, nil
601+ }
602+ gh, err := rs.githubValue("", rs.run.Attempt)
603+ if err != nil {
604+ return domain.CIConcurrency{}, err
605+ }
606+ vars, err := rs.c.varsContext(ctx, rs.run.Repo)
607+ if err != nil {
608+ return domain.CIConcurrency{}, err
609+ }
610+ env := template.Env{Contexts: map[string]expr.Value{"github": gh, "inputs": expr.NewDictionary(), "vars": vars}}
611+ return evalConcurrency(ctx, wf.Concurrency, env)
612+}
613+
614+// releaseEnded frees the groups of what ended in the write that was just
615+// applied: the jobs that hold one of their own, and the run itself when it
616+// completed.
617+func (rs *runState) releaseEnded(ctx context.Context, changed []int, runDone bool) {
618+ c := rs.c
619+ for _, i := range changed {
620+ j := rs.jobs[i]
621+ if !j.Ended() || j.Concurrency.Group == "" || j.Concurrency.FromRun {
622+ continue
623+ }
624+ e := domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID}
625+ if err := c.releaseEntry(ctx, rs.run.Repo, j.Concurrency.Group, e); err != nil && !errors.Is(err, domain.ErrNotFound) {
626+ ctxLog(ctx).Error().Err(err).Str("job_id", j.ID).Msg("ci: free a concurrency group")
627+ }
628+ }
629+ if runDone && rs.run.Concurrency.Group != "" {
630+ e := domain.CIConcurrencyEntry{RunID: rs.run.ID, Attempt: rs.run.Attempt}
631+ if err := c.releaseEntry(ctx, rs.run.Repo, rs.run.Concurrency.Group, e); err != nil && !errors.Is(err, domain.ErrNotFound) {
632+ ctxLog(ctx).Error().Err(err).Int64("run_id", rs.run.ID).Msg("ci: free a concurrency group")
633+ }
634+ }
635+}
@@ -0,0 +1,635 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package usecase
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "maps"
10+ "slices"
11+ "time"
12+
13+ "bnema.dev/ermite-actions/expr"
14+ "bnema.dev/ermite-actions/template"
15+ "bnema.dev/ermite-actions/workflow"
16+ "bnema.dev/ermite/internal/domain"
17+ "bnema.dev/ermite/internal/ports"
18+)
19+
20+// Concurrency groups (CI-3 phase E). A group is per repository and compared
21+// without regard to case; a run (workflow level) or a job (job level) is an
22+// entry. With queue: single, at most one entry runs and one waits: a newer
23+// waiting entry cancels the older one, and cancel-in-progress also cancels the
24+// running one. An entry holds its group from the moment it may run until the
25+// run or job has ended.
26+//
27+// The record of a group lives under "concur/<repo>/<key(group)>" and is
28+// changed with a version-checked write (Store.UpdateConcurrency). Cancelling
29+// another entry needs the lock of another run, which the caller of a change
30+// does not hold in order (two runs would wait for each other): those effects
31+// are queued with later and run after the lock is released.
32+
33+// errEmptyGroup and errLongGroup are GitHub's errors for a group that
34+// evaluates to nothing or too much (actions/runner,
35+// WorkflowTemplateConverter.cs ConvertToConcurrency, lines 679-687).
36+var (
37+ errEmptyGroup = errors.New("concurrency group name cannot be empty")
38+ // errQueueMaxCancel is the message of languageservices
39+ // (languageservice/src/validate.ts:743).
40+ errQueueMaxCancel = errors.New("'queue: max' cannot be combined with 'cancel-in-progress: true'.") //nolint:staticcheck // ST1005: the text of languageservices, kept as is
41+ errLongGroup = fmt.Errorf("concurrency group name must be less than %d characters", domain.CIMaxConcurrencyGroup)
42+)
43+
44+// evalConcurrency evaluates the "concurrency:" token of a workflow or a job:
45+// a string (or an expression) is the group; a mapping has group,
46+// cancel-in-progress (an expression is allowed) and queue. An empty group is
47+// an error, as in GitHub's runner; queue: max is not supported yet.
48+func evalConcurrency(ctx context.Context, tok template.Token, env template.Env) (domain.CIConcurrency, error) {
49+ var out domain.CIConcurrency
50+ if tok == nil {
51+ return out, nil
52+ }
53+ groupTok := tok
54+ if m, ok := tok.(*template.Mapping); ok {
55+ if q := m.Find("queue"); q != nil {
56+ queue, err := template.EvaluateString(ctx, q, env)
57+ if err != nil {
58+ return out, err
59+ }
60+ if queue == "max" {
61+ // languageservices, languageservice/src/validate.ts:702-743: the
62+ // literal pair is invalid (expressions are not checked there).
63+ if b, ok := m.Find("cancel-in-progress").(*template.Boolean); ok && b.Value {
64+ return out, errQueueMaxCancel
65+ }
66+ return out, errors.New("not supported yet: queue: max")
67+ }
68+ }
69+ if groupTok = m.Find("group"); groupTok == nil {
70+ return out, errEmptyGroup
71+ }
72+ if c := m.Find("cancel-in-progress"); c != nil {
73+ cancel, err := template.EvaluateBool(ctx, c, env)
74+ if err != nil {
75+ return out, err
76+ }
77+ out.Cancel = cancel
78+ }
79+ }
80+ group, err := template.EvaluateString(ctx, groupTok, env)
81+ switch {
82+ case err != nil:
83+ return out, err
84+ case group == "":
85+ return out, errEmptyGroup
86+ case domain.ConcurrencyGroupLength(group) > domain.CIMaxConcurrencyGroup:
87+ return out, errLongGroup
88+ }
89+ out.Group = group
90+ return out, nil
91+}
92+
93+// later runs fn in the background, after the lock of the caller's run is
94+// released. Close waits for it.
95+func (c *CI) later(what string, fn func(ctx context.Context) error) {
96+ c.mu.Lock()
97+ if c.closed {
98+ c.mu.Unlock()
99+ return
100+ }
101+ c.wg.Add(1)
102+ c.mu.Unlock()
103+ go func() {
104+ defer c.wg.Done()
105+ c.guard(what, func() {
106+ if err := fn(c.ctx); err != nil && c.ctx.Err() == nil && !errors.Is(err, domain.ErrNotFound) {
107+ ctxLog(c.ctx).Error().Err(err).Str("worker", what).Msg("ci concurrency")
108+ }
109+ })
110+ }()
111+}
112+
113+// entryAlive reports whether the run or job of an entry can still hold or want
114+// its group: it exists, and it has not ended (a run: it is on the attempt of the
115+// entry and has not completed). Run ids are never reused, so a missing run or
116+// job is gone for good. A read that fails says alive: nothing is dropped on a
117+// doubt.
118+func (c *CI) entryAlive(ctx context.Context, repo domain.RepoName, e domain.CIConcurrencyEntry) bool {
119+ if e.JobID == "" {
120+ run, err := c.cfg.Store.GetRun(ctx, repo, e.RunID)
121+ if err != nil {
122+ return !errors.Is(err, domain.ErrNotFound)
123+ }
124+ return run.Attempt == e.Attempt && run.Status != domain.CICompleted
125+ }
126+ j, err := c.cfg.Store.GetJob(ctx, e.RunID, e.Attempt, e.JobID)
127+ switch {
128+ case errors.Is(err, domain.ErrNotFound):
129+ // A matrix leg that took its group is written with the flush that
130+ // follows: without its record yet, it lives as long as its run's attempt.
131+ return c.entryAlive(ctx, repo, domain.CIConcurrencyEntry{RunID: e.RunID, Attempt: e.Attempt})
132+ case err != nil:
133+ return true
134+ }
135+ return !j.Ended()
136+}
137+
138+// mark sets the mark of an entry.
139+func mark(st *domain.CIConcurrencyGroupState, e domain.CIConcurrencyEntry, group string, state domain.CIConcurrencyState) {
140+ if st.Marks == nil {
141+ st.Marks = map[string]domain.CIConcurrencyMark{}
142+ }
143+ st.Marks[e.Key()] = domain.CIConcurrencyMark{Entry: e, Group: group, State: state}
144+}
145+
146+// promote gives the running place to the waiting entry when it is free.
147+func promote(st *domain.CIConcurrencyGroupState, group string) {
148+ if st.Record.Running == nil && st.Record.Pending != nil {
149+ st.Record.Running, st.Record.Pending = st.Record.Pending, nil
150+ m := st.Marks[st.Record.Running.Key()]
151+ mark(st, *st.Record.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
152+ }
153+}
154+
155+func firstNonEmpty(a, b string) string {
156+ if a != "" {
157+ return a
158+ }
159+ return b
160+}
161+
162+// prune brings a group back to its invariant, in the transaction that is about to
163+// decide on it: the entries whose run or job is gone or over leave the record
164+// and lose their marks (a crash between the write of a job and the release of
165+// its group, a run that was deleted); the waiting entry takes a free running
166+// place; an entry named by the record has its mark; and an entry marked as
167+// pending or holding that the record does not name any more is marked replaced
168+// (it should not happen, the batch writes both, and it is cancelled if it does).
169+func (c *CI) prune(ctx context.Context, repo domain.RepoName, group string, st *domain.CIConcurrencyGroupState) (more bool) {
170+ rec := &st.Record
171+ if rec.Running != nil && !c.entryAlive(ctx, repo, *rec.Running) {
172+ delete(st.Marks, rec.Running.Key())
173+ rec.Running = nil
174+ }
175+ if rec.Pending != nil && !c.entryAlive(ctx, repo, *rec.Pending) {
176+ delete(st.Marks, rec.Pending.Key())
177+ rec.Pending = nil
178+ }
179+ promote(st, group)
180+ if rec.Running != nil {
181+ if m, ok := st.Marks[rec.Running.Key()]; !ok || m.State != domain.CIConcurrencyHolding {
182+ mark(st, *rec.Running, firstNonEmpty(m.Group, group), domain.CIConcurrencyHolding)
183+ }
184+ }
185+ if rec.Pending != nil {
186+ if m, ok := st.Marks[rec.Pending.Key()]; !ok || m.State != domain.CIConcurrencyPending {
187+ mark(st, *rec.Pending, firstNonEmpty(m.Group, group), domain.CIConcurrencyPending)
188+ }
189+ }
190+ // The other marks are pruned a bounded number per transaction, in key order:
191+ // a batch has a size limit, and a group with a long backlog of stale marks
192+ // must not wedge on one transaction that deletes them all. What is left is
193+ // reported, and the next decision or the reconciliation takes the next ones.
194+ left := c.cfg.ConcurrencyPruneBatch
195+ for _, k := range slices.Sorted(maps.Keys(st.Marks)) {
196+ m := st.Marks[k]
197+ var prune, replace bool
198+ switch {
199+ case !c.entryAlive(ctx, repo, m.Entry):
200+ prune = true
201+ case m.State != domain.CIConcurrencyReplaced && !rec.Names(m.Entry):
202+ replace = true
203+ default:
204+ continue
205+ }
206+ if left <= 0 {
207+ return true
208+ }
209+ left--
210+ if prune {
211+ delete(st.Marks, k)
212+ } else if replace {
213+ m.State = domain.CIConcurrencyReplaced
214+ st.Marks[k] = m
215+ }
216+ }
217+ return false
218+}
219+
220+// updateGroup is Store.UpdateConcurrency over the pruned group. It returns the
221+// entry that holds the group after the change and did not before (the waiting
222+// entry that was given the turn), or nil: its run is to be planned again.
223+func (c *CI) updateGroup(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) (*domain.CIConcurrencyEntry, error) {
224+ promoted, _, err := c.updateGroupMore(ctx, repo, group, fn)
225+ return promoted, err
226+}
227+
228+// updateGroupMore is updateGroup that also says whether the pruning stopped at
229+// its bound, with stale marks left for another transaction.
230+func (c *CI) updateGroupMore(ctx context.Context, repo domain.RepoName, group string, fn func(*domain.CIConcurrencyGroupState) error) (promoted *domain.CIConcurrencyEntry, more bool, err error) {
231+ err = c.cfg.Store.UpdateConcurrency(ctx, repo, group, func(st *domain.CIConcurrencyGroupState) error {
232+ promoted = nil
233+ var was *domain.CIConcurrencyEntry
234+ if r := st.Record.Running; r != nil {
235+ e := *r
236+ was = &e
237+ }
238+ more = c.prune(ctx, repo, group, st)
239+ if err := fn(st); err != nil {
240+ return err
241+ }
242+ if r := st.Record.Running; r != nil && (was == nil || !was.Same(*r)) {
243+ e := *r
244+ promoted = &e
245+ }
246+ return nil
247+ })
248+ return promoted, more, err
249+}
250+
251+// concurrencyRetries bounds the retries of a delivery that failed; the delay
252+// doubles from the watchdog tick up to maxConcurrencyBackoff.
253+const (
254+ defaultConcurrencyRetries = 8
255+ maxConcurrencyBackoff = 30 * time.Second
256+)
257+
258+// retry calls fn until it succeeds, at most concurrencyRetries more times, with
259+// a doubling delay that ends with ctx. A domain.ErrNotFound is not retried.
260+func (c *CI) retry(ctx context.Context, fn func() error) error {
261+ delay := c.cfg.Tick
262+ err := fn()
263+ retries := c.cfg.ConcurrencyRetries
264+ switch {
265+ case retries == 0:
266+ retries = defaultConcurrencyRetries
267+ case retries < 0:
268+ retries = 0
269+ }
270+ for range retries {
271+ if err == nil || errors.Is(err, domain.ErrNotFound) {
272+ return err
273+ }
274+ select {
275+ case <-ctx.Done():
276+ return errors.Join(err, ctx.Err())
277+ case <-time.After(delay):
278+ }
279+ delay = min(delay*2, maxConcurrencyBackoff)
280+ err = fn()
281+ }
282+ return err
283+}
284+
285+// wake plans again the run of an entry that was given its group, outside any
286+// run lock, and again if that failed. A run that is gone passes the group on.
287+// What the retries do not reach, the reconciliation (startup, and every
288+// minute) does: it advances every holder whose own record still says pending.
289+func (c *CI) wake(repo domain.RepoName, group string, e domain.CIConcurrencyEntry) {
290+ c.later("concurrency promote", func(ctx context.Context) error {
291+ return c.retry(ctx, func() error {
292+ err := c.advanceRun(ctx, e.RunID)
293+ if errors.Is(err, domain.ErrNotFound) {
294+ return c.releaseEntry(ctx, repo, group, e)
295+ }
296+ return err
297+ })
298+ })
299+}
300+
301+// cancelLater cancels an entry that a newer one replaced, outside any run lock,
302+// and again if that failed. The decision that replaced it is in the group (the
303+// entry is marked replaced in the batch that unnamed it, or stays named with
304+// Canceling set): the reconciliation finds it from there if this cancellation is
305+// lost (crash, retries used up).
306+func (c *CI) cancelLater(group string, e domain.CIConcurrencyEntry) {
307+ c.later("concurrency cancel", func(ctx context.Context) error {
308+ return c.retry(ctx, func() error { return c.cancelEntry(ctx, e, group) })
309+ })
310+}
311+
312+// entryNeeds reports whether the entry named by its group still has to be told
313+// where it stands: its own record does not say want yet (a promotion whose
314+// planning failed or was lost, or a crash between the write of the group and the
315+// write of the entry). An entry that is gone or over is the pruning's business.
316+func (c *CI) entryNeeds(ctx context.Context, repo domain.RepoName, e domain.CIConcurrencyEntry, want domain.CIConcurrencyState) bool {
317+ if e.JobID == "" {
318+ run, err := c.cfg.Store.GetRun(ctx, repo, e.RunID)
319+ return err == nil && run.Status != domain.CICompleted && run.Attempt == e.Attempt && run.Concurrency.State != want
320+ }
321+ j, err := c.cfg.Store.GetJob(ctx, e.RunID, e.Attempt, e.JobID)
322+ if errors.Is(err, domain.ErrNotFound) {
323+ // A leg the group names whose record is not written: the run has to be
324+ // planned, which expands its matrix, while its attempt is open.
325+ run, err := c.cfg.Store.GetRun(ctx, repo, e.RunID)
326+ return err == nil && run.Status != domain.CICompleted && run.Attempt == e.Attempt
327+ }
328+ return err == nil && !j.Ended() && j.Phase == domain.CIPhaseWaiting && j.Concurrency.State != want
329+}
330+
331+// ReconcileConcurrency repairs the groups. It runs at startup and every minute.
332+// For each stored group it prunes the entries that are gone or over, asks again
333+// the running entry a cancel-in-progress entry replaced, cancels every entry
334+// marked replaced (their cancellation is owed), and plans the entries the record
335+// names whose own record does not know it yet.
336+//
337+// Cost, per pass: a scan of the stored groups and their marks, and for each
338+// group a version-checked transaction (a write only if something is repaired;
339+// at most CIConfig.ConcurrencyPruneBatch stale marks go in one, the group is
340+// taken again until none is left) plus a read of the run or job of each entry
341+// the record or the marks name. It grows with the groups in use, not with the
342+// open runs.
343+func (c *CI) ReconcileConcurrency(ctx context.Context) {
344+ log := ctxLog(ctx)
345+ groups, err := c.cfg.Store.ListConcurrency(ctx)
346+ if err != nil {
347+ log.Error().Err(err).Msg("ci: list the concurrency groups")
348+ return
349+ }
350+ for _, g := range groups {
351+ var st domain.CIConcurrencyGroupState
352+ var err error
353+ // One transaction prunes a bounded number of marks: go on until none is
354+ // left to prune (each pass removes some, so this ends).
355+ for more := true; more && err == nil && ctx.Err() == nil; {
356+ _, more, err = c.updateGroupMore(ctx, g.Repo, g.Group, func(cur *domain.CIConcurrencyGroupState) error {
357+ st = domain.CIConcurrencyGroupState{Marks: maps.Clone(cur.Marks)}
358+ if cur.Record.Running != nil {
359+ r := *cur.Record.Running
360+ st.Record.Running = &r
361+ }
362+ if cur.Record.Pending != nil {
363+ p := *cur.Record.Pending
364+ st.Record.Pending = &p
365+ }
366+ return nil
367+ })
368+ }
369+ if err != nil {
370+ if !errors.Is(err, domain.ErrNotFound) && ctx.Err() == nil {
371+ log.Error().Err(err).Str("group", g.Group).Msg("ci: repair a concurrency group")
372+ }
373+ continue
374+ }
375+ if r := st.Record.Running; r != nil {
376+ if r.Canceling {
377+ c.cancelLater(st.Marks[r.Key()].Group, *r)
378+ }
379+ if c.entryNeeds(ctx, g.Repo, *r, domain.CIConcurrencyHolding) {
380+ c.wake(g.Repo, g.Group, *r)
381+ }
382+ }
383+ if p := st.Record.Pending; p != nil && c.entryNeeds(ctx, g.Repo, *p, domain.CIConcurrencyPending) {
384+ c.wake(g.Repo, g.Group, *p)
385+ }
386+ for _, m := range st.Marks {
387+ if m.State == domain.CIConcurrencyReplaced {
388+ c.cancelLater(m.Group, m.Entry)
389+ }
390+ }
391+ }
392+}
393+
394+// acquire asks for the turn of an entry in a group and returns where it stands:
395+// holding, pending behind the running entry, or replaced. The decision and the
396+// marks of every entry it touches are one batch (Store.UpdateConcurrency), so an
397+// entry is marked pending or holding if and only if the record names it.
398+//
399+// An entry marked replaced never acquires, whatever the state of the group
400+// (empty, or its waiting place free): it is owed a cancellation. Otherwise a
401+// pending entry stays so; a newer one replaces the waiting entry, which is
402+// marked replaced and cancelled, and with cancel-in-progress the running entry
403+// is cancelled too (it stays named, with Canceling set, until it has ended).
404+// The call is repeatable: an entry that holds or waits is told so again.
405+func (rs *runState) acquire(ctx context.Context, group string, e domain.CIConcurrencyEntry) (domain.CIConcurrencyState, error) {
406+ var state domain.CIConcurrencyState
407+ var victims []domain.CIConcurrencyEntry
408+ promoted, err := rs.c.updateGroup(ctx, rs.run.Repo, group, func(st *domain.CIConcurrencyGroupState) error {
409+ victims = nil
410+ rec := &st.Record
411+ if m, ok := st.Marks[e.Key()]; ok && m.State == domain.CIConcurrencyReplaced {
412+ state = domain.CIConcurrencyReplaced
413+ victims = append(victims, e) // its cancellation may have been lost
414+ return nil
415+ }
416+ switch {
417+ case rec.Running == nil:
418+ me := e
419+ rec.Running = &me
420+ if rec.Pending != nil && rec.Pending.Same(e) {
421+ rec.Pending = nil
422+ }
423+ mark(st, e, group, domain.CIConcurrencyHolding)
424+ state = domain.CIConcurrencyHolding
425+ case rec.Running.Same(e):
426+ state = domain.CIConcurrencyHolding
427+ case rec.Pending != nil && rec.Pending.Same(e):
428+ state = domain.CIConcurrencyPending
429+ default:
430+ state = domain.CIConcurrencyPending
431+ if rec.Pending != nil {
432+ old := *rec.Pending
433+ mark(st, old, group, domain.CIConcurrencyReplaced)
434+ victims = append(victims, old)
435+ }
436+ me := e
437+ rec.Pending = &me
438+ mark(st, e, group, domain.CIConcurrencyPending)
439+ if e.Cancel && !rec.Running.Canceling {
440+ run := *rec.Running
441+ run.Canceling = true
442+ rec.Running = &run
443+ victims = append(victims, run)
444+ }
445+ }
446+ return nil
447+ })
448+ if err != nil {
449+ return "", err
450+ }
451+ if promoted != nil && !promoted.Same(e) {
452+ rs.c.wake(rs.run.Repo, group, *promoted)
453+ }
454+ for _, v := range victims {
455+ rs.c.cancelLater(group, v)
456+ }
457+ if state == domain.CIConcurrencyReplaced {
458+ state = domain.CIConcurrencyPending // it shows as waiting until its cancellation lands
459+ }
460+ return state, nil
461+}
462+
463+// releaseEntry takes an entry out of its group, because its run or job has
464+// ended, and gives the turn to the waiting entry, which is planned again. The
465+// mark of the entry goes in the same batch.
466+func (c *CI) releaseEntry(ctx context.Context, repo domain.RepoName, group string, e domain.CIConcurrencyEntry) error {
467+ promoted, err := c.updateGroup(ctx, repo, group, func(st *domain.CIConcurrencyGroupState) error {
468+ switch {
469+ case st.Record.Running != nil && st.Record.Running.Same(e):
470+ st.Record.Running = nil
471+ promote(st, group)
472+ case st.Record.Pending != nil && st.Record.Pending.Same(e):
473+ st.Record.Pending = nil
474+ }
475+ delete(st.Marks, e.Key())
476+ return nil
477+ })
478+ if err != nil || promoted == nil {
479+ return err
480+ }
481+ c.wake(repo, group, *promoted)
482+ return nil
483+}
484+
485+// cancelEntry cancels a run or a job that a newer entry of the group
486+// replaced: it ends cancelled with GitHub's annotation, and a job a runner has
487+// gets the cancel request the owner's cancel sends.
488+func (c *CI) cancelEntry(ctx context.Context, e domain.CIConcurrencyEntry, group string) error {
489+ return c.withRun(ctx, e.RunID, e.Attempt, func(rs *runState) error {
490+ if group == "" { // no name in the mark: use the entry's own group
491+ group = rs.run.Concurrency.Group
492+ if j := rs.find(e.JobID); e.JobID != "" && j != nil {
493+ group = j.Concurrency.Group
494+ }
495+ }
496+ msg := domain.ConcurrencyCancelMessage(group)
497+ if e.JobID == "" {
498+ if !rs.latest() || rs.run.Status == domain.CICompleted || rs.run.CancelRequested {
499+ return nil
500+ }
501+ rs.run.CancelRequested, rs.run.CancelRequestedAt, rs.runDirty = true, c.now(), true
502+ rs.run.Concurrency.Preempted = true
503+ for i := range rs.jobs {
504+ if !rs.jobs[i].Ended() {
505+ rs.cancelForGroup(&rs.jobs[i], msg)
506+ }
507+ }
508+ return rs.flush(ctx)
509+ }
510+ j := rs.find(e.JobID)
511+ if j == nil || !j.Planned {
512+ // The group names a leg whose record the run has not written yet (the
513+ // flush that followed the planning was lost): the record found under
514+ // the id of leg 0 is the placeholder of the whole job, not the leg.
515+ // Plan the job first, so that every leg exists, and only then act on
516+ // the one that was targeted.
517+ if err := rs.advance(ctx); err != nil {
518+ return err
519+ }
520+ if j = rs.find(e.JobID); j == nil || !j.Planned {
521+ return rs.flush(ctx) // not plannable (yet): nothing to cancel, keep what advance did
522+ }
523+ }
524+ if j.Ended() || j.Concurrency.Preempted {
525+ return rs.flush(ctx) // over, or cancelled by an earlier delivery of the same item
526+ }
527+ rs.cancelForGroup(j, msg)
528+ if err := rs.advance(ctx); err != nil {
529+ return err
530+ }
531+ return rs.flush(ctx)
532+ })
533+}
534+
535+// cancelForGroup ends a job a newer entry of its group replaced, or asks its
536+// runner to stop. The annotation is GitHub's.
537+func (rs *runState) cancelForGroup(j *domain.CIJob, msg string) {
538+ j.Concurrency.Preempted = true
539+ rs.annotations = append(rs.annotations, ports.CIStepAnnotations{
540+ Repo: j.Repo, Exec: logRef(*j), Step: 1,
541+ Items: []domain.CIAnnotation{{Level: domain.AnnotationWarning, Message: msg, Step: 1}},
542+ })
543+ if j.Active() {
544+ rs.requestCancel(j, domain.CICancelUser, msg)
545+ rs.touch(j)
546+ return
547+ }
548+ rs.end(j, domain.CICancelled, msg)
549+}
550+
551+// acquireRun evaluates the workflow-level concurrency of the attempt once and
552+// asks for its turn. It returns whether the run may go on: a pending run
553+// plans nothing, and its jobs show the pending status.
554+func (rs *runState) acquireRun(ctx context.Context, wf *workflow.Workflow) (bool, error) {
555+ run := &rs.run
556+ if !slices.ContainsFunc(rs.jobs, func(j domain.CIJob) bool { return !j.Ended() }) {
557+ return true, nil // nothing left to run: the group is freed at the flush
558+ }
559+ if !run.Concurrency.Evaluated {
560+ conc, err := rs.evalRunConcurrency(ctx, wf)
561+ if err != nil {
562+ for i := range rs.jobs {
563+ if !rs.jobs[i].Ended() && !rs.jobs[i].Active() {
564+ rs.end(&rs.jobs[i], domain.CIFailure, err.Error())
565+ }
566+ }
567+ run.Concurrency, rs.runDirty = domain.CIConcurrency{Evaluated: true}, true
568+ return false, nil
569+ }
570+ conc.Evaluated = true
571+ run.Concurrency, rs.runDirty = conc, true
572+ }
573+ if run.Concurrency.Group == "" {
574+ return true, nil
575+ }
576+ st, err := rs.acquire(ctx, run.Concurrency.Group, domain.CIConcurrencyEntry{RunID: run.ID, Attempt: run.Attempt, Cancel: run.Concurrency.Cancel})
577+ if err != nil {
578+ return false, err
579+ }
580+ if st != run.Concurrency.State {
581+ run.Concurrency.State, rs.runDirty = st, true
582+ }
583+ want := run.Concurrency
584+ want.FromRun, want.Preempted = true, false
585+ for i := range rs.jobs {
586+ j := &rs.jobs[i]
587+ if j.Ended() || (j.Concurrency.Group != "" && !j.Concurrency.FromRun) {
588+ continue // a job with a group of its own shows that one
589+ }
590+ if j.Concurrency != want {
591+ j.Concurrency = want
592+ rs.touch(j)
593+ }
594+ }
595+ return st == domain.CIConcurrencyHolding, nil
596+}
597+
598+func (rs *runState) evalRunConcurrency(ctx context.Context, wf *workflow.Workflow) (domain.CIConcurrency, error) {
599+ if wf.Concurrency == nil {
600+ return domain.CIConcurrency{}, nil
601+ }
602+ gh, err := rs.githubValue("", rs.run.Attempt)
603+ if err != nil {
604+ return domain.CIConcurrency{}, err
605+ }
606+ vars, err := rs.c.varsContext(ctx, rs.run.Repo)
607+ if err != nil {
608+ return domain.CIConcurrency{}, err
609+ }
610+ env := template.Env{Contexts: map[string]expr.Value{"github": gh, "inputs": expr.NewDictionary(), "vars": vars}}
611+ return evalConcurrency(ctx, wf.Concurrency, env)
612+}
613+
614+// releaseEnded frees the groups of what ended in the write that was just
615+// applied: the jobs that hold one of their own, and the run itself when it
616+// completed.
617+func (rs *runState) releaseEnded(ctx context.Context, changed []int, runDone bool) {
618+ c := rs.c
619+ for _, i := range changed {
620+ j := rs.jobs[i]
621+ if !j.Ended() || j.Concurrency.Group == "" || j.Concurrency.FromRun {
622+ continue
623+ }
624+ e := domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID}
625+ if err := c.releaseEntry(ctx, rs.run.Repo, j.Concurrency.Group, e); err != nil && !errors.Is(err, domain.ErrNotFound) {
626+ ctxLog(ctx).Error().Err(err).Str("job_id", j.ID).Msg("ci: free a concurrency group")
627+ }
628+ }
629+ if runDone && rs.run.Concurrency.Group != "" {
630+ e := domain.CIConcurrencyEntry{RunID: rs.run.ID, Attempt: rs.run.Attempt}
631+ if err := c.releaseEntry(ctx, rs.run.Repo, rs.run.Concurrency.Group, e); err != nil && !errors.Is(err, domain.ErrNotFound) {
632+ ctxLog(ctx).Error().Err(err).Int64("run_id", rs.run.ID).Msg("ci: free a concurrency group")
633+ }
634+ }
635+}
A internal/usecase/ci_concurrency_test.go +111 -0
@@ -0,0 +1,111 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package usecase_test
4+
5+import (
6+ "context"
7+ "fmt"
8+ "strings"
9+ "testing"
10+
11+ "github.com/stretchr/testify/assert"
12+ "github.com/stretchr/testify/require"
13+
14+ "bnema.dev/ermite-actions/expr"
15+ "bnema.dev/ermite-actions/template"
16+ "bnema.dev/ermite-actions/workflow"
17+ "bnema.dev/ermite/internal/domain"
18+ "bnema.dev/ermite/internal/usecase"
19+)
20+
21+const concurrencyYAML = `on: push
22+concurrency: %s
23+jobs:
24+ j:
25+ runs-on: ubuntu-latest
26+ steps:
27+ - run: echo
28+`
29+
30+func evalGroup(t *testing.T, value string) (domain.CIConcurrency, error) {
31+ t.Helper()
32+ wf, errs := workflow.Parse("w.yml", []byte(fmt.Sprintf(concurrencyYAML, value)))
33+ require.Empty(t, errs)
34+ gh, err := expr.FromJSON([]byte(`{"ref":"refs/heads/main","workflow":"ci"}`))
35+ require.NoError(t, err)
36+ env := template.Env{Contexts: map[string]expr.Value{"github": gh, "inputs": expr.NewDictionary(), "vars": expr.NewDictionary()}}
37+ return usecase.EvalConcurrency(context.Background(), wf.Concurrency, env)
38+}
39+
40+func TestEvalConcurrency(t *testing.T) {
41+ t.Parallel()
42+ tests := []struct {
43+ name, yaml string
44+ want domain.CIConcurrency
45+ err string
46+ }{
47+ {"a string is the group", `Deploy`, domain.CIConcurrency{Group: "Deploy"}, ""},
48+ {"an expression is the group", `${{ github.workflow }}-${{ github.ref }}`, domain.CIConcurrency{Group: "ci-refs/heads/main"}, ""},
49+ {"mapping", "{group: g}", domain.CIConcurrency{Group: "g"}, ""},
50+ {"cancel true", "{group: g, cancel-in-progress: true}", domain.CIConcurrency{Group: "g", Cancel: true}, ""},
51+ {"cancel expression", "{group: g, cancel-in-progress: \"${{ github.ref == 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g", Cancel: true}, ""},
52+ {"cancel expression false", "{group: g, cancel-in-progress: \"${{ github.ref != 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g"}, ""},
53+ {"queue single", "{group: g, queue: single}", domain.CIConcurrency{Group: "g"}, ""},
54+ {"queue max", "{group: g, queue: max}", domain.CIConcurrency{}, "not supported yet: queue: max"},
55+ {"queue max with cancel true", "{group: g, queue: max, cancel-in-progress: true}", domain.CIConcurrency{}, "'queue: max' cannot be combined with 'cancel-in-progress: true'."},
56+ {"empty group", `${{ vars.NOPE }}`, domain.CIConcurrency{}, "concurrency group name cannot be empty"},
57+ }
58+ for _, tt := range tests {
59+ t.Run(tt.name, func(t *testing.T) {
60+ t.Parallel()
61+ got, err := evalGroup(t, tt.yaml)
62+ if tt.err != "" {
63+ require.Error(t, err)
64+ assert.Contains(t, err.Error(), tt.err)
65+ return
66+ }
67+ require.NoError(t, err)
68+ assert.Equal(t, tt.want, got)
69+ })
70+ }
71+ // GitHub's limit counts UTF-16 code units, like C#'s String.Length.
72+ for _, tc := range []struct {
73+ name, group string
74+ ok bool
75+ }{
76+ {"400 BMP characters", strings.Repeat("é", 400), true},
77+ {"401 BMP characters", strings.Repeat("é", 401), false},
78+ {"200 emoji are 400 units", strings.Repeat("😀", 200), true},
79+ {"201 emoji are 402 units", strings.Repeat("😀", 201), false},
80+ {"199 emoji and 2 letters", strings.Repeat("😀", 199) + "ab", true},
81+ {"199 emoji and 3 letters", strings.Repeat("😀", 199) + "abc", false},
82+ } {
83+ t.Run(tc.name, func(t *testing.T) {
84+ t.Parallel()
85+ _, err := evalGroup(t, `"`+tc.group+`"`)
86+ if tc.ok {
87+ assert.NoError(t, err)
88+ return
89+ }
90+ require.Error(t, err)
91+ assert.Contains(t, err.Error(), "less than 400 characters")
92+ })
93+ }
94+ t.Run("too long", func(t *testing.T) {
95+ t.Parallel()
96+ _, err := evalGroup(t, strings.Repeat("a", 401))
97+ require.Error(t, err)
98+ assert.Contains(t, err.Error(), "less than 400 characters")
99+ })
100+}
101+
102+func TestConcurrencyIdentity(t *testing.T) {
103+ t.Parallel()
104+ assert.Equal(t, domain.ConcurrencyKey("Deploy"), domain.ConcurrencyKey("DEPLOY"))
105+ assert.Equal(t, "Canceling since a higher priority waiting request for 'g' exists", domain.ConcurrencyCancelMessage("g"))
106+ pending := domain.CIJob{Phase: domain.CIPhaseWaiting, Concurrency: domain.CIConcurrency{State: domain.CIConcurrencyPending}}
107+ assert.Equal(t, domain.CIPending, pending.Status())
108+ pending.Phase = domain.CIPhaseEnded
109+ assert.Equal(t, domain.CICompleted, pending.Status(), "an ended job is not pending any more")
110+ assert.Equal(t, domain.CIQueued, domain.CIJob{Phase: domain.CIPhaseWaiting}.Status())
111+}
@@ -0,0 +1,111 @@
1+// SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+package usecase_test
4+
5+import (
6+ "context"
7+ "fmt"
8+ "strings"
9+ "testing"
10+
11+ "github.com/stretchr/testify/assert"
12+ "github.com/stretchr/testify/require"
13+
14+ "bnema.dev/ermite-actions/expr"
15+ "bnema.dev/ermite-actions/template"
16+ "bnema.dev/ermite-actions/workflow"
17+ "bnema.dev/ermite/internal/domain"
18+ "bnema.dev/ermite/internal/usecase"
19+)
20+
21+const concurrencyYAML = `on: push
22+concurrency: %s
23+jobs:
24+ j:
25+ runs-on: ubuntu-latest
26+ steps:
27+ - run: echo
28+`
29+
30+func evalGroup(t *testing.T, value string) (domain.CIConcurrency, error) {
31+ t.Helper()
32+ wf, errs := workflow.Parse("w.yml", []byte(fmt.Sprintf(concurrencyYAML, value)))
33+ require.Empty(t, errs)
34+ gh, err := expr.FromJSON([]byte(`{"ref":"refs/heads/main","workflow":"ci"}`))
35+ require.NoError(t, err)
36+ env := template.Env{Contexts: map[string]expr.Value{"github": gh, "inputs": expr.NewDictionary(), "vars": expr.NewDictionary()}}
37+ return usecase.EvalConcurrency(context.Background(), wf.Concurrency, env)
38+}
39+
40+func TestEvalConcurrency(t *testing.T) {
41+ t.Parallel()
42+ tests := []struct {
43+ name, yaml string
44+ want domain.CIConcurrency
45+ err string
46+ }{
47+ {"a string is the group", `Deploy`, domain.CIConcurrency{Group: "Deploy"}, ""},
48+ {"an expression is the group", `${{ github.workflow }}-${{ github.ref }}`, domain.CIConcurrency{Group: "ci-refs/heads/main"}, ""},
49+ {"mapping", "{group: g}", domain.CIConcurrency{Group: "g"}, ""},
50+ {"cancel true", "{group: g, cancel-in-progress: true}", domain.CIConcurrency{Group: "g", Cancel: true}, ""},
51+ {"cancel expression", "{group: g, cancel-in-progress: \"${{ github.ref == 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g", Cancel: true}, ""},
52+ {"cancel expression false", "{group: g, cancel-in-progress: \"${{ github.ref != 'refs/heads/main' }}\"}", domain.CIConcurrency{Group: "g"}, ""},
53+ {"queue single", "{group: g, queue: single}", domain.CIConcurrency{Group: "g"}, ""},
54+ {"queue max", "{group: g, queue: max}", domain.CIConcurrency{}, "not supported yet: queue: max"},
55+ {"queue max with cancel true", "{group: g, queue: max, cancel-in-progress: true}", domain.CIConcurrency{}, "'queue: max' cannot be combined with 'cancel-in-progress: true'."},
56+ {"empty group", `${{ vars.NOPE }}`, domain.CIConcurrency{}, "concurrency group name cannot be empty"},
57+ }
58+ for _, tt := range tests {
59+ t.Run(tt.name, func(t *testing.T) {
60+ t.Parallel()
61+ got, err := evalGroup(t, tt.yaml)
62+ if tt.err != "" {
63+ require.Error(t, err)
64+ assert.Contains(t, err.Error(), tt.err)
65+ return
66+ }
67+ require.NoError(t, err)
68+ assert.Equal(t, tt.want, got)
69+ })
70+ }
71+ // GitHub's limit counts UTF-16 code units, like C#'s String.Length.
72+ for _, tc := range []struct {
73+ name, group string
74+ ok bool
75+ }{
76+ {"400 BMP characters", strings.Repeat("é", 400), true},
77+ {"401 BMP characters", strings.Repeat("é", 401), false},
78+ {"200 emoji are 400 units", strings.Repeat("😀", 200), true},
79+ {"201 emoji are 402 units", strings.Repeat("😀", 201), false},
80+ {"199 emoji and 2 letters", strings.Repeat("😀", 199) + "ab", true},
81+ {"199 emoji and 3 letters", strings.Repeat("😀", 199) + "abc", false},
82+ } {
83+ t.Run(tc.name, func(t *testing.T) {
84+ t.Parallel()
85+ _, err := evalGroup(t, `"`+tc.group+`"`)
86+ if tc.ok {
87+ assert.NoError(t, err)
88+ return
89+ }
90+ require.Error(t, err)
91+ assert.Contains(t, err.Error(), "less than 400 characters")
92+ })
93+ }
94+ t.Run("too long", func(t *testing.T) {
95+ t.Parallel()
96+ _, err := evalGroup(t, strings.Repeat("a", 401))
97+ require.Error(t, err)
98+ assert.Contains(t, err.Error(), "less than 400 characters")
99+ })
100+}
101+
102+func TestConcurrencyIdentity(t *testing.T) {
103+ t.Parallel()
104+ assert.Equal(t, domain.ConcurrencyKey("Deploy"), domain.ConcurrencyKey("DEPLOY"))
105+ assert.Equal(t, "Canceling since a higher priority waiting request for 'g' exists", domain.ConcurrencyCancelMessage("g"))
106+ pending := domain.CIJob{Phase: domain.CIPhaseWaiting, Concurrency: domain.CIConcurrency{State: domain.CIConcurrencyPending}}
107+ assert.Equal(t, domain.CIPending, pending.Status())
108+ pending.Phase = domain.CIPhaseEnded
109+ assert.Equal(t, domain.CICompleted, pending.Status(), "an ended job is not pending any more")
110+ assert.Equal(t, domain.CIQueued, domain.CIJob{Phase: domain.CIPhaseWaiting}.Status())
111+}
M internal/usecase/ci_plan.go +32 -6
@@ -89,8 +89,6 @@ func unsupported(j *workflow.Job) string {
8989 switch {
9090 case j.IsReusable():
9191 return "reusable workflows (uses at the job level)"
92- case j.Concurrency != nil:
93- return "concurrency"
9492 }
9593 return ""
9694 }
@@ -332,6 +330,9 @@ func (rs *runState) advance(ctx context.Context) error {
332330 }
333331 return nil
334332 }
333+ if ok, err := rs.acquireRun(ctx, wf); err != nil || !ok {
334+ return err // waits for its group, or was failed by it
335+ }
335336 for changed := true; changed; {
336337 changed = false
337338 for _, wj := range wf.Jobs {
@@ -351,12 +352,13 @@ func (rs *runState) advance(ctx context.Context) error {
351352 changed = true
352353 }
353354 }
354- rs.promote()
355- return nil
355+ return rs.promote(ctx)
356356 }
357357
358-// promote makes planned instances ready while max-parallel allows it.
359-func (rs *runState) promote() {
358+// promote makes planned instances ready while max-parallel allows it and the
359+// concurrency group of the job, if it has one of its own, gives it the turn.
360+// An instance that waits for its group is pending.
361+func (rs *runState) promote(ctx context.Context) error {
360362 now := rs.c.now()
361363 counts := map[string]int{}
362364 for _, j := range rs.jobs {
@@ -372,10 +374,24 @@ func (rs *runState) promote() {
372374 if j.MaxParallel > 0 && counts[j.Key] >= j.MaxParallel {
373375 continue
374376 }
377+ if g := j.Concurrency; g.Group != "" && !g.FromRun {
378+ st, err := rs.acquire(ctx, g.Group, domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID, Cancel: g.Cancel})
379+ if err != nil {
380+ return err
381+ }
382+ if st != g.State {
383+ j.Concurrency.State = st
384+ rs.touch(j)
385+ }
386+ if st != domain.CIConcurrencyHolding {
387+ continue
388+ }
389+ }
375390 j.Phase, j.ReadyAt = domain.CIPhaseReady, now
376391 counts[j.Key]++
377392 rs.touch(j)
378393 }
394+ return nil
379395 }
380396
381397 // plan evaluates a job whose needs have ended: its condition, its strategy and
@@ -520,6 +536,16 @@ func (rs *runState) instance(ctx context.Context, wf *workflow.Workflow, wj *wor
520536 }
521537 j.ContinueOnError = coe
522538 }
539+ if wj.Concurrency != nil {
540+ // Evaluated with github, needs, strategy, matrix, inputs and vars;
541+ // this leg's own group, which it takes before it may run.
542+ conc, err := evalConcurrency(ctx, wj.Concurrency, env)
543+ if err != nil {
544+ return j, err
545+ }
546+ conc.Evaluated = true
547+ j.Concurrency = conc
548+ }
523549 j.Planned, j.Phase = true, domain.CIPhaseWaiting
524550 return j, nil
525551 }
@@ -89,8 +89,6 @@ func unsupported(j *workflow.Job) string {
89 switch {89 switch {
90 case j.IsReusable():90 case j.IsReusable():
91 return "reusable workflows (uses at the job level)"91 return "reusable workflows (uses at the job level)"
92- case j.Concurrency != nil:
93- return "concurrency"
94 }92 }
95 return ""93 return ""
96 }94 }
@@ -332,6 +330,9 @@ func (rs *runState) advance(ctx context.Context) error {
332 }330 }
333 return nil331 return nil
334 }332 }
333+ if ok, err := rs.acquireRun(ctx, wf); err != nil || !ok {
334+ return err // waits for its group, or was failed by it
335+ }
335 for changed := true; changed; {336 for changed := true; changed; {
336 changed = false337 changed = false
337 for _, wj := range wf.Jobs {338 for _, wj := range wf.Jobs {
@@ -351,12 +352,13 @@ func (rs *runState) advance(ctx context.Context) error {
351 changed = true352 changed = true
352 }353 }
353 }354 }
354- rs.promote()355+ return rs.promote(ctx)
355- return nil
356 }356 }
357 357
358-// promote makes planned instances ready while max-parallel allows it.358+// promote makes planned instances ready while max-parallel allows it and the
359-func (rs *runState) promote() {359+// concurrency group of the job, if it has one of its own, gives it the turn.
360+// An instance that waits for its group is pending.
361+func (rs *runState) promote(ctx context.Context) error {
360 now := rs.c.now()362 now := rs.c.now()
361 counts := map[string]int{}363 counts := map[string]int{}
362 for _, j := range rs.jobs {364 for _, j := range rs.jobs {
@@ -372,10 +374,24 @@ func (rs *runState) promote() {
372 if j.MaxParallel > 0 && counts[j.Key] >= j.MaxParallel {374 if j.MaxParallel > 0 && counts[j.Key] >= j.MaxParallel {
373 continue375 continue
374 }376 }
377+ if g := j.Concurrency; g.Group != "" && !g.FromRun {
378+ st, err := rs.acquire(ctx, g.Group, domain.CIConcurrencyEntry{RunID: j.RunID, Attempt: j.RunAttempt, JobID: j.ID, Cancel: g.Cancel})
379+ if err != nil {
380+ return err
381+ }
382+ if st != g.State {
383+ j.Concurrency.State = st
384+ rs.touch(j)
385+ }
386+ if st != domain.CIConcurrencyHolding {
387+ continue
388+ }
389+ }
375 j.Phase, j.ReadyAt = domain.CIPhaseReady, now390 j.Phase, j.ReadyAt = domain.CIPhaseReady, now
376 counts[j.Key]++391 counts[j.Key]++
377 rs.touch(j)392 rs.touch(j)
378 }393 }
394+ return nil
379 }395 }
380 396
381 // plan evaluates a job whose needs have ended: its condition, its strategy and397 // plan evaluates a job whose needs have ended: its condition, its strategy and
@@ -520,6 +536,16 @@ func (rs *runState) instance(ctx context.Context, wf *workflow.Workflow, wj *wor
520 }536 }
521 j.ContinueOnError = coe537 j.ContinueOnError = coe
522 }538 }
539+ if wj.Concurrency != nil {
540+ // Evaluated with github, needs, strategy, matrix, inputs and vars;
541+ // this leg's own group, which it takes before it may run.
542+ conc, err := evalConcurrency(ctx, wj.Concurrency, env)
543+ if err != nil {
544+ return j, err
545+ }
546+ conc.Evaluated = true
547+ j.Concurrency = conc
548+ }
523 j.Planned, j.Phase = true, domain.CIPhaseWaiting549 j.Planned, j.Phase = true, domain.CIPhaseWaiting
524 return j, nil550 return j, nil
525 }551 }
M internal/usecase/ci_runs.go +1 -0
@@ -399,6 +399,7 @@ func (c *CI) RerunRun(ctx context.Context, p domain.Principal, name domain.RepoN
399399 run.Attempt, run.Status, run.Conclusion = next, domain.CIQueued, domain.CINoConclusion
400400 run.StartedAt, run.CompletedAt, run.CreatedAt = time.Time{}, time.Time{}, now
401401 run.CancelRequested, run.CancelRequestedAt = false, time.Time{}
402+ run.Concurrency = domain.CIConcurrency{} // evaluated again: the re-run joins its group again
402403 // Kept or lowered, never raised: the owner who runs an agent's run
403404 // again does not give it secrets (D1).
404405 run.Trusted = domain.RerunTrusted(rs.run.Trusted, p)
@@ -399,6 +399,7 @@ func (c *CI) RerunRun(ctx context.Context, p domain.Principal, name domain.RepoN
399 run.Attempt, run.Status, run.Conclusion = next, domain.CIQueued, domain.CINoConclusion399 run.Attempt, run.Status, run.Conclusion = next, domain.CIQueued, domain.CINoConclusion
400 run.StartedAt, run.CompletedAt, run.CreatedAt = time.Time{}, time.Time{}, now400 run.StartedAt, run.CompletedAt, run.CreatedAt = time.Time{}, time.Time{}, now
401 run.CancelRequested, run.CancelRequestedAt = false, time.Time{}401 run.CancelRequested, run.CancelRequestedAt = false, time.Time{}
402+ run.Concurrency = domain.CIConcurrency{} // evaluated again: the re-run joins its group again
402 // Kept or lowered, never raised: the owner who runs an agent's run403 // Kept or lowered, never raised: the owner who runs an agent's run
403 // again does not give it secrets (D1).404 // again does not give it secrets (D1).
404 run.Trusted = domain.RerunTrusted(rs.run.Trusted, p)405 run.Trusted = domain.RerunTrusted(rs.run.Trusted, p)
M internal/usecase/ci_watch.go +12 -2
@@ -12,7 +12,12 @@ import (
1212 "bnema.dev/ermite/internal/domain"
1313 )
1414
15-const sweepEvery = 24 * time.Hour
15+const (
16+ sweepEvery = 24 * time.Hour
17+ // reconcileEvery is how often the concurrency groups are repaired (see
18+ // ReconcileConcurrency for what a pass costs).
19+ reconcileEvery = time.Minute
20+)
1621
1722 // advanceRun plans what can start in the latest attempt of a run.
1823 func (c *CI) advanceRun(ctx context.Context, runID int64) error {
@@ -35,6 +40,7 @@ func (c *CI) resume() {
3540 ctxLog(ctx).Error().Err(err).Msg("ci: list the open runs")
3641 return
3742 }
43+ c.ReconcileConcurrency(ctx)
3844 for _, r := range runs {
3945 if err := c.advanceRun(ctx, r.ID); err != nil && !errors.Is(err, domain.ErrNotFound) && ctx.Err() == nil {
4046 ctxLog(ctx).Error().Err(err).Int64("run_id", r.ID).Msg("ci: plan an open run again")
@@ -49,7 +55,7 @@ func (c *CI) watchdog() {
4955 defer c.wg.Done()
5056 tick := time.NewTicker(c.cfg.Tick)
5157 defer tick.Stop()
52- lastSweep := time.Time{}
58+ lastSweep, lastReconcile := time.Time{}, c.now()
5359 for {
5460 select {
5561 case <-c.ctx.Done():
@@ -59,6 +65,10 @@ func (c *CI) watchdog() {
5965 c.guard("watchdog", func() {
6066 c.checkActive(c.ctx)
6167 c.checkQueue(c.ctx)
68+ if now := c.now(); now.Sub(lastReconcile) >= reconcileEvery {
69+ lastReconcile = now
70+ c.ReconcileConcurrency(c.ctx)
71+ }
6272 if now := c.now(); now.Sub(lastSweep) >= sweepEvery {
6373 lastSweep = now
6474 c.sweep(c.ctx)
@@ -12,7 +12,12 @@ import (
12 "bnema.dev/ermite/internal/domain"12 "bnema.dev/ermite/internal/domain"
13 )13 )
14 14
15-const sweepEvery = 24 * time.Hour15+const (
16+ sweepEvery = 24 * time.Hour
17+ // reconcileEvery is how often the concurrency groups are repaired (see
18+ // ReconcileConcurrency for what a pass costs).
19+ reconcileEvery = time.Minute
20+)
16 21
17 // advanceRun plans what can start in the latest attempt of a run.22 // advanceRun plans what can start in the latest attempt of a run.
18 func (c *CI) advanceRun(ctx context.Context, runID int64) error {23 func (c *CI) advanceRun(ctx context.Context, runID int64) error {
@@ -35,6 +40,7 @@ func (c *CI) resume() {
35 ctxLog(ctx).Error().Err(err).Msg("ci: list the open runs")40 ctxLog(ctx).Error().Err(err).Msg("ci: list the open runs")
36 return41 return
37 }42 }
43+ c.ReconcileConcurrency(ctx)
38 for _, r := range runs {44 for _, r := range runs {
39 if err := c.advanceRun(ctx, r.ID); err != nil && !errors.Is(err, domain.ErrNotFound) && ctx.Err() == nil {45 if err := c.advanceRun(ctx, r.ID); err != nil && !errors.Is(err, domain.ErrNotFound) && ctx.Err() == nil {
40 ctxLog(ctx).Error().Err(err).Int64("run_id", r.ID).Msg("ci: plan an open run again")46 ctxLog(ctx).Error().Err(err).Int64("run_id", r.ID).Msg("ci: plan an open run again")
@@ -49,7 +55,7 @@ func (c *CI) watchdog() {
49 defer c.wg.Done()55 defer c.wg.Done()
50 tick := time.NewTicker(c.cfg.Tick)56 tick := time.NewTicker(c.cfg.Tick)
51 defer tick.Stop()57 defer tick.Stop()
52- lastSweep := time.Time{}58+ lastSweep, lastReconcile := time.Time{}, c.now()
53 for {59 for {
54 select {60 select {
55 case <-c.ctx.Done():61 case <-c.ctx.Done():
@@ -59,6 +65,10 @@ func (c *CI) watchdog() {
59 c.guard("watchdog", func() {65 c.guard("watchdog", func() {
60 c.checkActive(c.ctx)66 c.checkActive(c.ctx)
61 c.checkQueue(c.ctx)67 c.checkQueue(c.ctx)
68+ if now := c.now(); now.Sub(lastReconcile) >= reconcileEvery {
69+ lastReconcile = now
70+ c.ReconcileConcurrency(c.ctx)
71+ }
62 if now := c.now(); now.Sub(lastSweep) >= sweepEvery {72 if now := c.now(); now.Sub(lastSweep) >= sweepEvery {
63 lastSweep = now73 lastSweep = now
64 c.sweep(c.ctx)74 c.sweep(c.ctx)
M internal/usecase/export_test.go +6 -0
@@ -6,6 +6,7 @@ import (
66 "context"
77
88 "bnema.dev/ermite-actions/expr"
9+ "bnema.dev/ermite-actions/template"
910 "bnema.dev/ermite-actions/workflow"
1011 "bnema.dev/ermite/internal/domain"
1112 )
@@ -54,3 +55,8 @@ func (s *SecretUseCache) Use(runID int64, yaml, jobKey string) SecretUse {
5455
5556 // Drop is wfCache.drop.
5657 func (s *SecretUseCache) Drop(runID int64) { s.c.drop(runID) }
58+
59+// EvalConcurrency exposes evalConcurrency to the tests.
60+func EvalConcurrency(ctx context.Context, tok template.Token, env template.Env) (domain.CIConcurrency, error) {
61+ return evalConcurrency(ctx, tok, env)
62+}
@@ -6,6 +6,7 @@ import (
6 "context"6 "context"
7 7
8 "bnema.dev/ermite-actions/expr"8 "bnema.dev/ermite-actions/expr"
9+ "bnema.dev/ermite-actions/template"
9 "bnema.dev/ermite-actions/workflow"10 "bnema.dev/ermite-actions/workflow"
10 "bnema.dev/ermite/internal/domain"11 "bnema.dev/ermite/internal/domain"
11 )12 )
@@ -54,3 +55,8 @@ func (s *SecretUseCache) Use(runID int64, yaml, jobKey string) SecretUse {
54 55
55 // Drop is wfCache.drop.56 // Drop is wfCache.drop.
56 func (s *SecretUseCache) Drop(runID int64) { s.c.drop(runID) }57 func (s *SecretUseCache) Drop(runID int64) { s.c.drop(runID) }
58+
59+// EvalConcurrency exposes evalConcurrency to the tests.
60+func EvalConcurrency(ctx context.Context, tok template.Token, env template.Env) (domain.CIConcurrency, error) {
61+ return evalConcurrency(ctx, tok, env)
62+}
M proto/ermite/storage/ci.proto +70 -0
@@ -18,6 +18,8 @@ enum CIStatus {
1818 CI_STATUS_QUEUED = 1;
1919 CI_STATUS_IN_PROGRESS = 2;
2020 CI_STATUS_COMPLETED = 3;
21+ // Held back by a concurrency group (jobs and checks only).
22+ CI_STATUS_PENDING = 4;
2123 }
2224
2325 // CIConclusion is how a run, a job, a step or a check ended.
@@ -102,6 +104,8 @@ message CIRun {
102104 // trusted: the actor of the run is the owner, so its jobs get secrets. False
103105 // in runs stored before it existed (the safe reading).
104106 bool trusted = 23;
107+ // The evaluated workflow-level concurrency of the attempt.
108+ CIConcurrency concurrency = 24;
105109 }
106110
107111 // CIRunSource is stored under "runsrc/<run id:020d>": what a run was planned
@@ -206,6 +210,9 @@ message CIJob {
206210 // the job is planned: permission key to "read", "write" or "none". Absent on
207211 // a job planned before the field existed: the restricted default.
208212 map<string, string> permissions = 42;
213+ // The job's concurrency group (its own, else the one of its run) and where
214+ // it is in it.
215+ CIConcurrency concurrency = 43;
209216 }
210217
211218 // CICheck is stored under "check/<repo>/<sha>/<id>": the id of its job, or
@@ -306,3 +313,66 @@ message CIAnnotation {
306313 message CIStepAnnotations {
307314 repeated CIAnnotation annotations = 1;
308315 }
316+
317+// CIConcurrencyState is where a run or a job is in its concurrency group.
318+enum CIConcurrencyState {
319+ CI_CONCURRENCY_STATE_UNSPECIFIED = 0;
320+ // Another entry of the group runs: this one waits.
321+ CI_CONCURRENCY_STATE_PENDING = 1;
322+ // This entry is the running one of its group.
323+ CI_CONCURRENCY_STATE_HOLDING = 2;
324+ // Only in a CIConcurrencyMark: a newer entry replaced this one, which is owed
325+ // a cancellation and may never acquire the group.
326+ CI_CONCURRENCY_STATE_REPLACED = 3;
327+}
328+
329+// CIConcurrency is the evaluated "concurrency:" of a run or a job, kept on its
330+// record (CIRun.concurrency, CIJob.concurrency).
331+message CIConcurrency {
332+ string group = 1;
333+ bool cancel = 2;
334+ CIConcurrencyState state = 3;
335+ // evaluated: the expressions were evaluated, whatever the result.
336+ bool evaluated = 4;
337+ // from_run: a job that shows the group of its run and holds nothing itself.
338+ bool from_run = 5;
339+ // preempted: cancelled because a newer entry of the group asked for it; the
340+ // job ends with GitHub's annotation.
341+ bool preempted = 6;
342+}
343+
344+// CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.
345+message CIConcurrencyEntry {
346+ int64 run_id = 1;
347+ int32 attempt = 2;
348+ string job_id = 3;
349+ bool cancel = 4;
350+ // canceling: a waiting entry with cancel-in-progress asked this running one
351+ // to stop; the entry stays until it has ended.
352+ bool canceling = 5;
353+}
354+
355+// CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
356+// running entry and the pending one of a group (queue: single). A group with
357+// neither has no key.
358+message CIConcurrencyRecord {
359+ // 3 held the cancellations owed to replaced entries; a replaced entry is now
360+ // found from its own record (it waits in a group that does not name it).
361+ reserved 3;
362+ CIConcurrencyEntry running = 1;
363+ CIConcurrencyEntry pending = 2;
364+}
365+
366+// CIConcurrencyMark is stored under
367+// "concurmark/<repo>/<escaped key(group)>/<run:020d>/<attempt:010d>/<job id or ->":
368+// where an entry stands in its group. It is written in the same batch as the
369+// group record ("concur/..."), so that an entry is marked pending or holding if
370+// and only if the record names it; an entry the record stops naming because a
371+// newer one replaced it is marked replaced in that very batch. The mark is
372+// removed with the entry (it ended, was released, or is gone).
373+message CIConcurrencyMark {
374+ CIConcurrencyEntry entry = 1;
375+ // group as the newest request wrote it, for the cancel annotation.
376+ string group = 2;
377+ CIConcurrencyState state = 3;
378+}
@@ -18,6 +18,8 @@ enum CIStatus {
18 CI_STATUS_QUEUED = 1;18 CI_STATUS_QUEUED = 1;
19 CI_STATUS_IN_PROGRESS = 2;19 CI_STATUS_IN_PROGRESS = 2;
20 CI_STATUS_COMPLETED = 3;20 CI_STATUS_COMPLETED = 3;
21+ // Held back by a concurrency group (jobs and checks only).
22+ CI_STATUS_PENDING = 4;
21 }23 }
22 24
23 // CIConclusion is how a run, a job, a step or a check ended.25 // CIConclusion is how a run, a job, a step or a check ended.
@@ -102,6 +104,8 @@ message CIRun {
102 // trusted: the actor of the run is the owner, so its jobs get secrets. False104 // trusted: the actor of the run is the owner, so its jobs get secrets. False
103 // in runs stored before it existed (the safe reading).105 // in runs stored before it existed (the safe reading).
104 bool trusted = 23;106 bool trusted = 23;
107+ // The evaluated workflow-level concurrency of the attempt.
108+ CIConcurrency concurrency = 24;
105 }109 }
106 110
107 // CIRunSource is stored under "runsrc/<run id:020d>": what a run was planned111 // CIRunSource is stored under "runsrc/<run id:020d>": what a run was planned
@@ -206,6 +210,9 @@ message CIJob {
206 // the job is planned: permission key to "read", "write" or "none". Absent on210 // the job is planned: permission key to "read", "write" or "none". Absent on
207 // a job planned before the field existed: the restricted default.211 // a job planned before the field existed: the restricted default.
208 map<string, string> permissions = 42;212 map<string, string> permissions = 42;
213+ // The job's concurrency group (its own, else the one of its run) and where
214+ // it is in it.
215+ CIConcurrency concurrency = 43;
209 }216 }
210 217
211 // CICheck is stored under "check/<repo>/<sha>/<id>": the id of its job, or218 // CICheck is stored under "check/<repo>/<sha>/<id>": the id of its job, or
@@ -306,3 +313,66 @@ message CIAnnotation {
306 message CIStepAnnotations {313 message CIStepAnnotations {
307 repeated CIAnnotation annotations = 1;314 repeated CIAnnotation annotations = 1;
308 }315 }
316+
317+// CIConcurrencyState is where a run or a job is in its concurrency group.
318+enum CIConcurrencyState {
319+ CI_CONCURRENCY_STATE_UNSPECIFIED = 0;
320+ // Another entry of the group runs: this one waits.
321+ CI_CONCURRENCY_STATE_PENDING = 1;
322+ // This entry is the running one of its group.
323+ CI_CONCURRENCY_STATE_HOLDING = 2;
324+ // Only in a CIConcurrencyMark: a newer entry replaced this one, which is owed
325+ // a cancellation and may never acquire the group.
326+ CI_CONCURRENCY_STATE_REPLACED = 3;
327+}
328+
329+// CIConcurrency is the evaluated "concurrency:" of a run or a job, kept on its
330+// record (CIRun.concurrency, CIJob.concurrency).
331+message CIConcurrency {
332+ string group = 1;
333+ bool cancel = 2;
334+ CIConcurrencyState state = 3;
335+ // evaluated: the expressions were evaluated, whatever the result.
336+ bool evaluated = 4;
337+ // from_run: a job that shows the group of its run and holds nothing itself.
338+ bool from_run = 5;
339+ // preempted: cancelled because a newer entry of the group asked for it; the
340+ // job ends with GitHub's annotation.
341+ bool preempted = 6;
342+}
343+
344+// CIConcurrencyEntry is a run (job_id empty) or a job in a concurrency group.
345+message CIConcurrencyEntry {
346+ int64 run_id = 1;
347+ int32 attempt = 2;
348+ string job_id = 3;
349+ bool cancel = 4;
350+ // canceling: a waiting entry with cancel-in-progress asked this running one
351+ // to stop; the entry stays until it has ended.
352+ bool canceling = 5;
353+}
354+
355+// CIConcurrencyRecord is stored under "concur/<repo>/<key(group)>" (ConcurrencyKey): the
356+// running entry and the pending one of a group (queue: single). A group with
357+// neither has no key.
358+message CIConcurrencyRecord {
359+ // 3 held the cancellations owed to replaced entries; a replaced entry is now
360+ // found from its own record (it waits in a group that does not name it).
361+ reserved 3;
362+ CIConcurrencyEntry running = 1;
363+ CIConcurrencyEntry pending = 2;
364+}
365+
366+// CIConcurrencyMark is stored under
367+// "concurmark/<repo>/<escaped key(group)>/<run:020d>/<attempt:010d>/<job id or ->":
368+// where an entry stands in its group. It is written in the same batch as the
369+// group record ("concur/..."), so that an entry is marked pending or holding if
370+// and only if the record names it; an entry the record stops naming because a
371+// newer one replaced it is marked replaced in that very batch. The mark is
372+// removed with the entry (it ended, was released, or is gone).
373+message CIConcurrencyMark {
374+ CIConcurrencyEntry entry = 1;
375+ // group as the newest request wrote it, for the cancel annotation.
376+ string group = 2;
377+ CIConcurrencyState state = 3;
378+}

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 2

Tags 45

↑↓ moveenter openesc close